Data ferrying method, image ferrying method, equipment and storage medium

Abstract

The invention discloses a data ferrying method, an image ferrying method, equipment and a storage medium, and the data ferrying method comprises the steps: determining a selected candidate input operator, a selected candidate data conversion operator and a selected candidate output operator, and obtaining a target input operator, a target data conversion operator and a target output operator; determining an execution sequence among the target input operator, the target data conversion operator and the target output operator, and generating an operator execution link; and the execution operator execution link is used for converting data of the data source end corresponding to the target input operator and then transmitting the data to the data storage end corresponding to the target output operator for storage. Through flexible configuration of a complete data ferry link of the input operator, the conversion operator and the output operator, mixed data processing of different data source ends and data storage ends can be supported, processing of unstructured data can be achieved, more application scenes can be covered, the development efficiency is improved, and maintenance is convenient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data ferry method, an image ferry method, a device and a storage medium. BACKGROUND

[0002] Data ferry refers to the transmission and exchange of data between different computers under the premise of ensuring data security.

[0003] At present, data platforms can be used to realize data ferry, but data platforms generally rely on full data extraction to data warehouses, and then subsequent standardized management is performed through structured query language (Structured Query Language, SQL) engines (such as Spark / Flink), which leads to long processing links, large resource consumption, low development efficiency, high maintenance cost, and poor processing capacity for unstructured data. SUMMARY

[0004] To solve the above technical problems, the present application at least provides a data ferry method, an image ferry method, a device and a storage medium.

[0005] The first aspect of the present application provides a data ferry method, which comprises: obtaining a candidate input operator set, a candidate data conversion operator set and a candidate output operator set; wherein the candidate input operator is used to obtain data from a data source end, the candidate data conversion operator is used to convert data, and the candidate output operator is used to transmit data to a data storage end; receiving operator selection instructions for the candidate input operator set, the candidate data conversion operator set and the candidate output operator set, determining the selected candidate input operator, the candidate data conversion operator and the candidate output operator, obtaining the target input operator, the target data conversion operator and the target output operator; receiving execution order configuration instructions for the target input operator, the target data conversion operator and the target output operator, determining the execution order between the target input operator, the target data conversion operator and the target output operator, and generating an operator execution link; executing the operator execution link to transmit the data of the data source end corresponding to the target input operator to the data storage end corresponding to the target output operator after conversion.

[0006] In an embodiment, the method further comprises: displaying a task editing page, the task editing page containing an input operator display area, a data conversion operator display area, an output operator display area and an operator editing area, the input operator display area being used to display the candidate input operator, the data conversion operator display area being used to display the candidate data conversion operator, the output operator display area being used to display the candidate output operator, and the operator editing area being used to display the target input operator, the target data conversion operator, the target output operator and their execution order.

[0007] In one embodiment, receiving an operator selection instruction for a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators, determining the selected candidate input operators, candidate data transformation operators, and candidate output operators, and obtaining the target input operator, target data transformation operator, and target output operator includes: if it is detected that a candidate input operator in the input operator display area, a candidate data transformation operator in the data transformation operator display area, or a candidate output operator in the output operator display area is dragged to the operator editing area, then it is determined that the candidate input operator, candidate data transformation operator, or candidate output operator dragged to the operator editing area is selected, and the target input operator, target data transformation operator, and target output operator are obtained. The system defines a target data conversion operator and a target output operator; it receives an execution order configuration instruction for the target input operator, target data conversion operator, and target output operator, and obtains the execution order among the target input operator, target data conversion operator, and target output operator, including: in response to a received connection line creation operation for the target input operator, target data conversion operator, and target output operator within the operator editing area, generating connection lines among the target input operator, target data conversion operator, and target output operator; and determining the execution order among the target input operator, target data conversion operator, and target output operator based on the connection lines among the target input operator, target data conversion operator, and target output operator.

[0008] In one embodiment, executing the operator execution link includes: verifying the operator execution link using a preset verification rule; if the verification passes, then executing the operator execution link.

[0009] In one embodiment, the operator execution link is verified using a preset verification rule, including: verifying the internal parameters corresponding to the operators contained in the operator execution link; and / or verifying the number of operators corresponding to the operators contained in the operator execution link; and / or verifying the link type corresponding to the operator execution link; and / or verifying whether the operator execution link contains all the selected operators.

[0010] In one embodiment, the candidate data transformation operator includes a field value mapping operator, and / or a field increment operator, and / or a data format conversion operator, and / or a data filtering operator.

[0011] A second aspect of this application provides an image transfer method, comprising: acquiring a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; wherein, candidate input operators are used to acquire image data from a data source, candidate data transformation operators are used to transform the image data, and candidate output operators are used to transmit the transformed image data to a data storage end; receiving operator selection instructions for the set of candidate input operators, the set of candidate data transformation operators, and the set of candidate output operators, determining the selected candidate input operators, candidate data transformation operators, and candidate output operators, and obtaining target input operators, target data transformation operators, and target output operators; receiving execution order configuration instructions for the target input operators, target data transformation operators, and target output operators, obtaining the execution order between the target input operators, target data transformation operators, and target output operators, and generating an operator execution chain; and executing the operator execution chain to transform the image data from the data source corresponding to the target input operator and then transmit it to the data storage end corresponding to the target output operator for storage.

[0012] In one embodiment, the execution operator execution chain includes: a target input operator obtaining image data from a data source, the image data containing an image download address and an image type; a target data conversion operator obtaining the service to which the image belongs based on the image type mapping, and downloading the image to be stored from the data source based on the image download address; and a target output operator transmitting the service to which the image belongs and the image to be stored to a data storage terminal, so that the data storage terminal stores the image to be stored in the specified location corresponding to the service to which the image belongs.

[0013] A third aspect of this application provides a data transfer device, comprising: a data transfer operator acquisition module, used to acquire a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; wherein, candidate input operators are used to acquire data from a data source, candidate data transformation operators are used to transform the data, and candidate output operators are used to transmit the data to a data storage end; a data transfer operator selection module, used to receive operator selection instructions for the set of candidate input operators, the set of candidate data transformation operators, and the set of candidate output operators, determine the selected candidate input operators, candidate data transformation operators, and candidate output operators, and obtain target input operators, target data transformation operators, and target output operators; a data transfer operator configuration module, used to receive execution order configuration instructions for the target input operators, target data transformation operators, and target output operators, determine the execution order between the target input operators, target data transformation operators, and target output operators, and generate an operator execution link; and a data transmission module, used to execute the operator execution link to transform the data from the data source corresponding to the target input operator and transmit it to the data storage end corresponding to the target output operator for storage.

[0014] A fourth aspect of this application provides an image ferrying device, comprising: an image ferrying operator acquisition module, configured to acquire a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; wherein, candidate input operators are used to acquire image data from a data source, candidate data transformation operators are used to transform the image data, and candidate output operators are used to transmit the transformed image data to a data storage end; and an image ferrying operator selection module, configured to receive operator selection instructions for the set of candidate input operators, the set of candidate data transformation operators, and the set of candidate output operators, and determine the selected candidate input... The system comprises three parts: an operator, a candidate data transformation operator, and a candidate output operator; an image ferry operator configuration module, which receives execution order configuration instructions for the target input operator, target data transformation operator, and target output operator, obtains the execution order between the target input operator, target data transformation operator, and target output operator, and generates an operator execution chain; and an image transmission module, which executes the operator execution chain to transform the image data from the data source corresponding to the target input operator and transmit it to the data storage end corresponding to the target output operator for storage.

[0015] The fifth aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-described data transfer method or image transfer method.

[0016] The sixth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described data transfer method or image transfer method.

[0017] The above scheme obtains a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; determines the selected candidate input operators, candidate data transformation operators, and candidate output operators to obtain the target input operator, target data transformation operator, and target output operator; determines the execution order among the target input operator, target data transformation operator, and target output operator to generate an operator execution link; and executes the operator execution link to transform the data from the data source corresponding to the target input operator and transmit it to the data storage end corresponding to the target output operator for storage. This application achieves a modular design by flexibly configuring the complete data transfer link of input operators, transformation operators, and output operators. Data input, transformation, and output can be realized by directly executing the configured operator execution link. Its overall processing link is simpler and consumes less resources. Different input operators and output operators can support mixed data processing from different data sources and different data storage ends. Different transformation operators can process unstructured or structured data, thereby covering more application scenarios. Furthermore, the operator combination configuration method can also improve development efficiency and facilitate maintenance.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a schematic diagram illustrating the implementation environment of the solution in an exemplary embodiment of this application;

[0021] Figure 2 This is a flowchart illustrating a data transfer method in an exemplary embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a task editing page shown in an exemplary embodiment of this application;

[0023] Figure 4 This is a schematic diagram illustrating an operator execution chain in an exemplary embodiment of this application;

[0024] Figure 5 This is a block diagram illustrating a data transfer device in an exemplary embodiment of this application;

[0025] Figure 6 This is a block diagram illustrating an image transfer device in an exemplary embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application;

[0027] Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0028] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0030] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0031] The data transfer method provided in the embodiments of this application will be described below.

[0032] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment of the scheme in an exemplary embodiment of this application. The implementation environment may include a data source 110, a data transfer terminal 120, and a data storage terminal 130, with the data transfer terminal 120 communicatively connected to both the data source 110 and the data storage terminal 130.

[0033] The data source 110 is used to store or generate data, and the data storage 130 is used to store data. The data source 110 and the data storage 130 can be cloud storage, database clusters, network cameras, smartphones, tablets, laptops, desktop computers, etc., but are not limited to these.

[0034] The data transfer terminal 120 is used to receive data from the data source terminal 110, process the data, and send it to the data storage terminal 130.

[0035] For example, the data transfer terminal 120 can be a data platform, a data integration platform, etc., and this application does not limit it.

[0036] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data transfer method in an exemplary embodiment of this application. This data transfer method can be applied to... Figure 1 The implementation environment shown is specifically executed by the data transfer terminal within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0037] like Figure 2 As shown, the data transfer method includes at least steps S210 to S240, which are described in detail below:

[0038] Step S210: Obtain the set of candidate input operators, the set of candidate data transformation operators, and the set of candidate output operators.

[0039] Among them, the candidate input operator is used to obtain data from the data source, the candidate data transformation operator is used to transform the data, and the candidate output operator is used to transmit the data to the data storage end.

[0040] Different candidate input operators correspond to different data sources. For example, candidate input operators can correspond to relational database management systems (such as MySQL), distributed publish-subscribe messaging systems (such as Kafka), message queues (such as RabbitMQ), etc.

[0041] Candidate data transformation operators are used to transform data. These include field value mapping operators, and / or field addition operators, and / or data format transformation operators, and / or data filtering operators.

[0042] The field value mapping operator is used to map the value of a specified field. For example, if the specified field is a status field and the original value is "0" or "1", the original value "0" will be mapped to "normal" and the original value "1" will be mapped to "abnormal".

[0043] The field addition operator is used to add one or more new fields. The value of the added field can be a default setting or can be flexibly calculated based on other fields in the original data. It can be flexibly set according to the actual situation.

[0044] Data format conversion operators are used to convert data formats, such as converting binary data to decimal data.

[0045] Data filtering operators are used to filter data. The filtering conditions can be flexibly set according to specific situations, such as filtering data based on the value of a specified field in the data.

[0046] Different candidate output operators correspond to different data storage terminals, such as MySQL, Kafka, RabbitMQ, etc.

[0047] Step S220: Receive operator selection instructions for the candidate input operator set, candidate data transformation operator set, and candidate output operator set; determine the selected candidate input operator, candidate data transformation operator, and candidate output operator; and obtain the target input operator, target data transformation operator, and target output operator.

[0048] The operator selection instruction is used to indicate the selected candidate input operator, candidate data transformation operator, and candidate output operator.

[0049] The selected candidate input operator, candidate data transformation operator, and candidate output operator are used as the target input operator, target data transformation operator, and target output operator, respectively.

[0050] For example, the operator selection instruction can be generated by the user terminal after selecting operators from the candidate input operator set, candidate data transformation operator set, and candidate output operator set according to the actual situation.

[0051] For example, operator selection instructions can also be automatically generated. This involves obtaining task prompt text, parsing the text, determining the matching data source, data storage, and data processing operation based on the parsing results, selecting candidate input operators from the candidate input operator set corresponding to the matching data source, candidate data transformation operators from the candidate data transformation operator set corresponding to the matching data processing operation, and candidate output operators from the candidate output operator set corresponding to the matching data storage. This generates the corresponding operator selection instructions. Alternatively, the selected candidate input operators, candidate data transformation operators, and candidate output operators can be directly used as the target input operator, target data transformation operator, and target output operator, respectively. Or, the selected candidate input operators, candidate data transformation operators, and candidate output operators can be sent to the user terminal to provide operator selection prompts, facilitating operator selection by the user.

[0052] It should be noted that the number of target input operators, target data transformation operators, and target output operators can be one or more.

[0053] Step S230: Receive the execution order configuration instruction for the target input operator, target data conversion operator and target output operator, obtain the execution order between the target input operator, target data conversion operator and target output operator, and generate the operator execution chain.

[0054] The sequence configuration instruction contains information indicating the execution order of the target input operator, the target data transformation operator, and the target output operator.

[0055] The sequential configuration instructions are parsed to obtain the execution order between the target input operator, the target data transformation operator, and the target output operator, and an operator execution chain is generated.

[0056] For example, the sequence configuration instruction can be generated by the user end after setting the execution order between each target input operator, target data transformation operator and target output operator according to the actual situation.

[0057] For example, the sequence configuration instructions can also be automatically generated, such as obtaining the task prompt text, parsing the task prompt text, and obtaining the execution order between the target input operator, the target data transformation operator, and the target output operator based on the text parsing results, so as to generate the corresponding sequence configuration instructions.

[0058] Step S240: Execute the operator execution link to transform the data from the data source corresponding to the target input operator and then transmit it to the data storage end corresponding to the target output operator for storage.

[0059] The operator execution chain indicates the target input operator, target data transformation operator, and target output operator, as well as the execution order among them. When the operator execution chain is executed, the target data transformation operator transforms the data from the data source corresponding to the target input operator and then transmits it to the data storage end corresponding to the target output operator for storage.

[0060] The target input operator determines whether to perform batch processing or stream processing based on the specific situation of the data source. If the data source sends data that needs to be processed in real time, then stream processing is performed; if the data source sends data that does not need to be processed in real time, then batch processing is performed.

[0061] This application achieves a modular design by flexibly configuring a complete data transfer link of input operators, transformation operators, and output operators. Data input, transformation, and output can be realized by directly executing the operator execution link. Its overall processing link is simpler and consumes less resources. Different input and output operators can support mixed data processing from different data sources and different data storage ends. Different transformation operators can process unstructured or structured data, thereby covering more application scenarios. Furthermore, the operator combination configuration method can also improve development efficiency and facilitate maintenance.

[0062] The following describes some embodiments of this application in detail.

[0063] In some implementations, an operator execution chain is obtained by providing a visual interface to the user and detecting the operations performed on the visual interface. The user interface can be a terminal that communicates with the data transfer terminal, such as a laptop, desktop computer, or smartphone.

[0064] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of a task editing page shown in an exemplary embodiment of this application, as follows: Figure 3As shown, the task editing page is displayed. The task editing page contains an input operator display area, a data transformation operator display area, an output operator display area, and an operator editing area. The input operator display area is used to display candidate input operators, the data transformation operator display area is used to display candidate data transformation operators, the output operator display area is used to display candidate output operators, and the operator editing area is used to display the target input operator, the target data transformation operator, the target output operator, and their execution order.

[0065] If a candidate input operator in the input operator display area, a candidate data conversion operator in the data conversion operator display area, or a candidate output operator in the output operator display area is dragged into the operator editing area, it is determined that the candidate input operator, candidate data conversion operator, or candidate output operator dragged into the operator editing area has been selected. The user terminal generates the corresponding operator selection instruction and sends the operator selection instruction to the data transfer terminal. The data transfer terminal obtains the corresponding target input operator, target data conversion operator, and target output operator based on the operator selection instruction.

[0066] You can also quickly search for and select operators by their names.

[0067] In addition, in response to the received connection line creation operation between the target input operator, target data transformation operator and target output operator within the operator editing area, connection lines are generated between the target input operator, target data transformation operator and target output operator, and the execution order between the target input operator, target data transformation operator and target output operator is determined based on the connection lines between the target input operator, target data transformation operator and target output operator.

[0068] Similarly, after the user terminal detects the connection creation operation, it generates a corresponding execution order configuration instruction and sends the execution order configuration instruction to the data transfer terminal. The data transfer terminal then uses this execution order configuration instruction to determine the execution order between the target input operator, the target data transformation operator, and the target output operator.

[0069] In addition, specific parameters can be configured for each operator. For example, if the target data transformation operator contains a field addition operator, the name of the field to be added and the filling rules for the value of the added field can be configured. Alternatively, the task prompt text can be parsed, and the parameters to be configured for each operator and the configuration method can be determined based on the text parsing results to achieve automatic parameter configuration. This application does not limit this.

[0070] After obtaining the target input operator, target data transformation operator, and target output operator, as well as the execution order among the target input operator, target data transformation operator, and target output operator, an operator execution chain is generated.

[0071] For example, please see Figure 4 , Figure 4 This is a schematic diagram illustrating an operator execution chain as shown in an exemplary embodiment of this application, such as... Figure 4 As shown, each operator is treated as a node in the operator execution chain. The target input operator, as the input node, obtains data from the data source and transmits the data to the subsequent transformation node group. The transformation node group has one or more transformation nodes, and each transformation node corresponds to a target data transformation operator. The transformation nodes in the transformation node group process the data in the execution order, and then send the processed data to the output node. The target output operator in the output node transmits the data to the data storage end.

[0072] For example, the target input operator is a MySQL operator, the target data transformation operator contains a field addition operator and a data format conversion operator, and the target output operator is a RabbitMQ operator. Based on the execution order of each operator, the operator execution chain is as follows: MySQL operator, field addition operator, data format conversion operator, and RabbitMQ operator are executed in sequence.

[0073] In some implementations, after obtaining the operator execution link, the operator execution link is verified using preset verification rules; if the verification passes, the operator execution link is executed; if the verification fails, an operator configuration error message is generated.

[0074] For example, the operator execution link is verified using preset verification rules, including: verifying the internal parameters corresponding to the operators contained in the operator execution link; and / or verifying the number of operators corresponding to the operators contained in the operator execution link; and / or verifying the link type corresponding to the operator execution link; and / or verifying whether the operator execution link contains all the selected operators.

[0075] Internal parameters include, but are not limited to, field names and types; the number of operators refers to the number of operators of the same type; and the link type includes, but is not limited to, single link and ring link.

[0076] For example, the process of verifying the operator execution chain includes:

[0077] Step S301: Determine whether there is one and only one target input operator. If yes, proceed to step S302; otherwise, proceed to step S308.

[0078] Step S302: Determine whether there is one and only one target output operator. If yes, proceed to step S303; otherwise, proceed to step S308.

[0079] Step S303: Determine whether the parameters of all selected operators have been configured. If yes, proceed to step S304; otherwise, proceed to step S308.

[0080] Step S304: Traverse each operator in the operator execution chain, record the field information of each operator (such as field name and type), and obtain the operator field list;

[0081] Step S305: Determine whether there is duplicate field information for any operator based on the operator field list. If not, proceed to step S306; if yes, proceed to step S308.

[0082] Step S306: Determine whether the operator execution link is a single link. If yes, proceed to step S307; otherwise, proceed to step S308.

[0083] Step S307: Determine whether all selected operators are connected to the operator execution chain. If yes, proceed to step S309; ​​otherwise, proceed to step S308.

[0084] Step S308: Verification failed.

[0085] Step S309: Verification passed.

[0086] Optionally, the start time of the operator execution chain, the execution method (such as one-time execution, periodic execution, real-time execution (if data exists at the data source end), concurrency, etc. can also be configured, and error data can be logged.

[0087] In this case, considering the differences between different data sources and data storage terminals, the configuration of the corresponding operator execution chain will differ. For example, MySQL can only configure the number of error records and does not support concurrency configuration, while Kafka can configure real-time execution tasks and supports concurrency configuration.

[0088] In addition, multiple operator execution links can be started, stopped, paused, and resumed in batches, and the execution records of the operator execution links can be statistically analyzed to obtain execution logs, such as start execution time, end execution time, amount of data processed, execution status, and trend charts (statistics of the amount of data stored at the data storage terminal at each time point).

[0089] The above embodiments support complete, end-to-end, and error-correctable task management capabilities.

[0090] This application also provides an image ferrying method, which can be applied to... Figure 1The implementation environment shown is specifically executed by the data transfer terminal within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0091] The image transfer method includes at least steps S410 to S440, which are described in detail below:

[0092] Step S410: Obtain the set of candidate input operators, the set of candidate data transformation operators, and the set of candidate output operators.

[0093] Among them, the candidate input operator is used to obtain image data from the data source, the candidate data transformation operator is used to transform the image data, and the candidate output operator is used to transmit the transformed image data to the data storage end.

[0094] For example, the data source is the upper-level image storage end, and the data storage end is the lower-level image storage end.

[0095] Step S420: Receive operator selection instructions for the candidate input operator set, candidate data transformation operator set, and candidate output operator set; determine the selected candidate input operator, candidate data transformation operator, and candidate output operator; and obtain the target input operator, target data transformation operator, and target output operator.

[0096] Step S430: Receive the execution order configuration instruction for the target input operator, target data conversion operator and target output operator, obtain the execution order between the target input operator, target data conversion operator and target output operator, and generate the operator execution chain.

[0097] Step S440: Execute the operator execution link to convert the image data from the data source corresponding to the target input operator and then transmit it to the data storage end corresponding to the target output operator for storage.

[0098] Optionally, pre-connection to the data source and data storage ends can be performed. This can involve pre-storing database connection information for each data source and data storage end. This database connection information includes the address, account name, account password, and domain identity document (ID) of the data source or data storage end. The domain includes primary and secondary domains. Selecting one domain yields the associated image gateway address and image storage address. Interface authentication is performed using this database connection information to achieve pre-connection and obtain a list of domains. Each domain contains an image gateway address and image storage address, enabling the download or upload of image data.

[0099] Alternatively, the above information can be configured directly in the input or output operator to achieve authentication connection.

[0100] In the image ferry scenario, the target input operator is the RabbitMQ input operator, the target data transformation operator is the image ferry operator, and the target output operator is the RabbitMQ output operator.

[0101] Image data is generated in real time at the data source. RabbitMQ input operators subscribe to the image data generated by the data source (e.g., a maximum of 3000 image data per batch). The image data contains the image download address and image type. The image ferry operator obtains the business to which the image belongs based on the image type mapping and downloads the image to be stored from the data source based on the image download address. The RabbitMQ output operator transmits the business to which the image belongs and the image to be stored to the data storage end, so that the data storage end stores the image to be stored in the specified location corresponding to the business to which the image belongs.

[0102] For example, image data contains an `event` field and an `imageList` field. The image download address is obtained from the `imgUrl` field in `imageList`. Based on this address, the image is requested from and downloaded from the image gateway at the data source, resulting in the image to be stored. Additionally, the image type is obtained from the `imgType` field in `imageList`. Following a predefined field value mapping logic, the image type is mapped to the corresponding service (`fileType`). Based on the `fileType`, the image upload interface (such as a Hypertext Transfer Protocol (HTTP) interface) of the data storage service on the data storage terminal is called to store the image to be stored in the specified location corresponding to the image's service on the data storage terminal.

[0103] The business category to which an image belongs is calculated through internal business mapping logic. For example, if the value of the event field is trafficJunction (vehicle) and the image type imgType is 0, the business category to which the image belongs is determined to be trafficJunction through the mapping table.

[0104] In addition, the data storage end provides feedback on the location where the image to be stored is located, obtains the new image storage address, replaces the original value of imgUrl in the image data (i.e. the original image download address) with the new image storage address, and then transfers the image data to the RabbitMQ queue on the data storage end.

[0105] Here's an example illustrating a portion of the content of the imageList field of a single image data received by the target input operator:

[0106] [{

[0107] "imgHeight":0,

[0108] "imgIdx":1,

[0109] "imgPixel":"0*0",

[0110] "imgSize":1063944,

[0111] "imgType":0,

[0112] "imgUrl":

[0113] " / image / efs_EVx3rgAh_001 / f3803aacc5aa7ee05433649c_trafficJunction_9_1 / archivefile1-2025-01-07-082503-3137C2DA2705200A:1264766976 / 1063944.jpg",

[0114] "imgWidth":0

[0115] }]

[0116] Among them, "imgHeight" indicates the height of the image, "imgWidth" indicates the width of the image, "imgIdx" indicates the index of the image, "imgPixel" indicates the pixel information of the image, "imgSize" indicates the size of the image, "imgType" indicates the image type, and "imgUrl" indicates the download address of the image.

[0117] Here's an example illustrating part of the content of the imageList field in an image data set output after address replacement by the target output operator:

[0118] [{

[0119] "imgHeight":0,

[0120] "imgIdx":1,

[0121] "imgPixel":"0*0",

[0122] "imgSize":1063944,

[0123] "imgType":0,

[0124] "imgUrl":

[0125] " / image / efs_EVx3rgAh_001 / f3803aacc5aa7ee05433649c_trafficJunction / archivefile1-2025-01-07-082503-3137C2DA2705200A:1264766976 / 1063944.jpg",

[0126] "imgWidth":0

[0127] }]

[0128] The data storage end consumes data from the RabbitMQ queue to visualize the data in the RabbitMQ queue, displaying images and their corresponding related information to facilitate user access to or understanding of image storage status.

[0129] Through the above embodiments, unstructured image data can be automatically transmitted, reducing transmission time, ensuring transmission security, and improving the transmission efficiency of image data.

[0130] Figure 5 This is a block diagram illustrating a data transfer device in an exemplary embodiment of this application. Figure 5 As shown, the exemplary data transfer device 500 includes:

[0131] The data transfer operator acquisition module 510 is used to acquire a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; wherein, the candidate input operators are used to acquire data from the data source, the candidate data transformation operators are used to transform the data, and the candidate output operators are used to transmit the data to the data storage end;

[0132] The data transfer operator selection module 520 is used to receive operator selection instructions for the candidate input operator set, the candidate data transformation operator set, and the candidate output operator set, determine the selected candidate input operator, candidate data transformation operator, and candidate output operator, and obtain the target input operator, target data transformation operator, and target output operator.

[0133] The data transfer operator configuration module 530 is used to receive execution order configuration instructions for the target input operator, target data conversion operator and target output operator, determine the execution order between the target input operator, target data conversion operator and target output operator, and generate operator execution links;

[0134] The data transmission module 540 is used to execute the operator execution link, so as to convert the data from the data source end corresponding to the target input operator and transmit it to the data storage end corresponding to the target output operator for storage.

[0135] Figure 6 This is a block diagram illustrating an image transfer device according to an exemplary embodiment of this application. Figure 6 As shown, the exemplary image transfer device 600 includes:

[0136] The image ferry operator acquisition module 610 is used to acquire a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; wherein, the candidate input operators are used to acquire image data from the data source, the candidate data transformation operators are used to transform the image data, and the candidate output operators are used to transmit the transformed image data to the data storage end.

[0137] The image ferry operator selection module 620 is used to receive operator selection instructions for the candidate input operator set, the candidate data transformation operator set, and the candidate output operator set, determine the selected candidate input operator, candidate data transformation operator, and candidate output operator, and obtain the target input operator, target data transformation operator, and target output operator;

[0138] The image ferry operator configuration module 630 is used to receive the execution order configuration instruction for the target input operator, the target data conversion operator and the target output operator, obtain the execution order between the target input operator, the target data conversion operator and the target output operator, and generate the operator execution link;

[0139] The image transmission module 640 is used to execute the operator execution link, so as to convert the image data at the data source end corresponding to the target input operator and transmit it to the data storage end corresponding to the target output operator for storage.

[0140] It should be noted that the data transfer device and image transfer device provided in the above embodiments belong to the same concept as the data transfer method and image transfer method provided in the above embodiments. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data transfer device and image transfer device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation.

[0141] Please see Figure 7 , Figure 7This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 700 includes a memory 701 and a processor 702. The processor 702 is used to execute program instructions stored in the memory 701 to implement the steps in any of the above-described data transfer method or image transfer method embodiments. In a specific implementation scenario, the electronic device 700 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 700 may also include mobile devices such as laptops and tablets, which are not limited here.

[0142] Specifically, processor 702 controls itself and memory 701 to implement the steps in any of the above-described data transfer method or image transfer method embodiments. Processor 702 can also be referred to as a Central Processing Unit (CPU). Processor 702 may be an integrated circuit chip with signal processing capabilities. Processor 702 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 702 can be implemented using integrated circuit chips.

[0143] Please see Figure 8 , Figure 8 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor. The program instructions 810 are used to implement the steps in any of the above-described data transfer method or image transfer method embodiments.

[0144] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0145] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data transfer method, characterized in that, The method includes: Obtain a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; wherein, candidate input operators are used to obtain data from the data source, candidate data transformation operators are used to transform the data, and candidate output operators are used to transmit the data to the data storage end; Receive operator selection instructions for the candidate input operator set, the candidate data transformation operator set, and the candidate output operator set; determine the selected candidate input operator, candidate data transformation operator, and candidate output operator; and obtain the target input operator, target data transformation operator, and target output operator. Receive an execution order configuration instruction for the target input operator, the target data conversion operator, and the target output operator; determine the execution order among the target input operator, the target data conversion operator, and the target output operator; and generate an operator execution chain. The operator execution chain is executed to transform the data from the data source corresponding to the target input operator and then transmit it to the data storage end corresponding to the target output operator for storage.

2. The method according to claim 1, characterized in that, The method further includes: The task editing page includes an input operator display area, a data transformation operator display area, an output operator display area, and an operator editing area. The input operator display area displays candidate input operators, the data transformation operator display area displays candidate data transformation operators, the output operator display area displays candidate output operators, and the operator editing area displays the target input operator, the target data transformation operator, the target output operator, and their execution order.

3. The method according to claim 2, characterized in that, The step of receiving operator selection instructions for the candidate input operator set, the candidate data transformation operator set, and the candidate output operator set, determining the selected candidate input operator, candidate data transformation operator, and candidate output operator, and obtaining the target input operator, target data transformation operator, and target output operator includes: If a candidate input operator in the input operator display area, a candidate data conversion operator in the data conversion operator display area, or a candidate output operator in the output operator display area is dragged to the operator editing area, it is determined that the candidate input operator, candidate data conversion operator, or candidate output operator dragged to the operator editing area is selected, and the target input operator, target data conversion operator, and target output operator are obtained. The step of receiving the execution order configuration instruction for the target input operator, the target data transformation operator, and the target output operator, and determining the execution order among the target input operator, the target data transformation operator, and the target output operator, includes: In response to a received connection line creation operation for the target input operator, the target data conversion operator, and the target output operator within the operator editing region, a connection line is generated between the target input operator, the target data conversion operator, and the target output operator; The execution order of the target input operator, the target data conversion operator, and the target output operator is determined based on the connection lines between them.

4. The method according to claim 1, characterized in that, The execution of the operator execution chain includes: The operator execution link is verified using preset verification rules; If the verification passes, the operator execution chain is executed.

5. The method according to claim 4, characterized in that, The step of verifying the operator execution link using preset verification rules includes: The internal parameters corresponding to the operators contained in the operator execution chain are verified; And / or, verify the number of operators corresponding to the operators contained in the operator execution link; And / or, verify the link type corresponding to the operator execution link; And / or, verify whether the operator execution chain contains all selected operators.

6. The method according to claim 1, characterized in that, The candidate data transformation operators include field value mapping operators, and / or field addition operators, and / or data format conversion operators, and / or data filtering operators.

7. An image transfer method for a data platform, characterized in that, The method includes: Obtain a set of candidate input operators, a set of candidate data transformation operators, and a set of candidate output operators; wherein, candidate input operators are used to obtain image data from the data source, candidate data transformation operators are used to transform the image data, and candidate output operators are used to transmit the transformed image data to the data storage end; Receive operator selection instructions for the candidate input operator set, the candidate data transformation operator set, and the candidate output operator set; determine the selected candidate input operator, candidate data transformation operator, and candidate output operator; and obtain the target input operator, target data transformation operator, and target output operator. Receive an execution order configuration instruction for the target input operator, the target data conversion operator, and the target output operator, obtain the execution order between the target input operator, the target data conversion operator, and the target output operator, and generate an operator execution chain; The operator execution chain is executed to convert the image data from the data source corresponding to the target input operator and then transmit it to the data storage end corresponding to the target output operator for storage.

8. The method according to claim 7, characterized in that, The execution of the operator execution chain includes: The target input operator obtains image data from the data source, and the image data contains the image download address and image type. The target data conversion operator obtains the service to which the image belongs based on the image type mapping, and downloads the image to be stored from the data source based on the image download address; The target output operator transmits the service to which the image belongs and the image to be stored to the data storage terminal, so that the data storage terminal stores the image to be stored in the designated location corresponding to the service to which the image belongs.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps in the data transfer method as described in any one of claims 1-6 or the steps in the image transfer method of the data platform as described in any one of claims 7-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps in the data transfer method as described in any one of claims 1-6 or the steps in the image transfer method of the data platform as described in any one of claims 7-8.

Images