Data processing method and device, storage medium and product

Abstract

The data processing method comprises the following steps: acquiring a to-be-processed array from a data source through a preset iterator node; wherein the preset iterator node is used for executing loop iteration processing on multiple pieces of data in the array to be processed; the preset iterator node sequentially inputs the data in the array to be processed into a preset sub-process processing module through the first transmission channel, and the preset sub-process processing module executes corresponding service logic processing on the data to obtain a processing result corresponding to the data; a processing result corresponding to the data is sent to a preset iterator node through a second transmission channel; and inputting a processing result corresponding to the data in the to-be-processed array into an output module through a preset iterator node, and outputting the processing result. Due to the fact that the preset iterator node can execute loop iteration processing on a large amount of data, the preset sub-process processing module can execute corresponding service logic processing on the large amount of data, and the processing efficiency of the large amount of data is improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a data processing method, device, storage medium and product. Background Art

[0002] At present, simple forms and process processing can be achieved through a small amount of drag and drop, while complex business logic such as iterative loops to process large amounts of data generally requires code development or multiple drag and drop nodes, and the entire implementation process is relatively complicated. At present, many OT personnel are not familiar with programming languages. When they use drag and drop editors to iterate large amounts of data multiple times, they need to ask programmers to develop code, which lengthens the business delivery cycle.

[0003] Therefore, how to improve the efficiency of processing large amounts of data is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main technical problem solved by the present invention is to improve the efficiency of processing large amounts of data.

[0005] According to the first aspect, an embodiment provides a data processing method, the method comprising: Obtaining an array to be processed from a data source through a preset iterator node; wherein the preset iterator node is used to perform loop iterative processing on multiple data in the array to be processed; The preset iterator node sequentially inputs the data in the array to be processed into a preset sub-process processing module through the first transmission channel, and the preset sub-process processing module performs corresponding business logic processing on the data to obtain processing results corresponding to the data; Sending the processing result corresponding to the data to the preset iterator node through a second transmission channel; wherein the second transmission channel is different from the first transmission channel; The processing results corresponding to the data in the array to be processed are input into an output module through the preset iterator node for output.

[0006] According to the second aspect, an embodiment provides a data processing device, the data processing device comprising: An acquisition unit, used for acquiring an array to be processed from a data source through a preset iterator node; wherein the preset iterator node is used for performing loop iterative processing on multiple data in the array to be processed; A processing unit is used for the preset iterator node to input the data in the array to be processed into a preset sub-process processing module in sequence through a first transmission channel, and the preset sub-process processing module performs corresponding business logic processing on the data to obtain a processing result corresponding to the data; sends the processing result corresponding to the data to the preset iterator node through a second transmission channel; wherein the second transmission channel is different from the first transmission channel; and inputs the processing result corresponding to the data in the array to be processed into an output module through the preset iterator node for output.

[0007] According to a third aspect, an embodiment provides a computer-readable storage medium, comprising a program, wherein the program can be executed by a processor to implement a data processing method.

[0008] According to a fourth aspect, an embodiment provides a computer program product, comprising a computer program and / or instructions, wherein the computer program and / or instructions implement a data processing method when executed by a processor.

[0009] According to the data processing method of the above embodiment, since the preset iterator node can realize loop iterative processing on a large amount of data, and the preset sub-process processing module can realize corresponding business logic processing on a large amount of data, the present invention improves the processing efficiency of a large amount of data by distributing the processing of a large amount of data to the preset iterator node and the preset sub-process processing module. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a data processing method provided in this embodiment; Figure 2 Another schematic diagram of a flow chart of a data processing method provided in this embodiment; Figure 3 This is a structural block diagram of a data processing device provided in this embodiment. DETAILED DESCRIPTION

[0011] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0012] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0013] The serial numbers assigned to the components herein, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0014] At present, simple forms and process processing can be achieved through a small amount of drag and drop, while complex business logic such as iterative loops to process large amounts of data generally requires code development or multiple drag and drop nodes, and the entire implementation process is relatively complicated. At present, many OT personnel are not familiar with programming languages. When they use drag and drop editors to iterate large amounts of data multiple times, they need to ask programmers to develop code, which lengthens the business delivery cycle. Therefore, how to improve the efficiency of processing large amounts of data has become increasingly critical.

[0015] Based on this, the solution of the present invention is proposed. In an embodiment of the present invention, an array to be processed is obtained from a data source through a preset iterator node; wherein the preset iterator node is used to perform loop iterative processing on multiple data in the array to be processed; the preset iterator node sequentially inputs the data in the array to be processed into a preset sub-process processing module through a first transmission channel, and the preset sub-process processing module performs corresponding business logic processing on the data to obtain processing results corresponding to the data; the processing results corresponding to the data are sent to the preset iterator node through a second transmission channel; the processing results corresponding to the data in the array to be processed are input into an output module through the preset iterator node for output. Since the preset iterator node can realize loop iterative processing on a large amount of data, the preset sub-process processing module can realize corresponding business logic processing on a large amount of data, so that the processing efficiency of a large amount of data is improved. This is described in detail below through some embodiments.

[0016] Example 1, please refer to Figure 1 , Figure 1 is a flow chart of a data processing method provided in this embodiment, such as Figure 1 As shown, the data processing method specifically includes the following steps 101-104: Step 101: Obtain an array to be processed from a data source through a preset iterator node; wherein the preset iterator node is used to perform loop iterative processing on multiple data in the array to be processed.

[0017] Here, the array to be processed is classified by element type, including: value array, character array, pointer array or structure array. The array to be processed is classified by dimension, including: one-dimensional array, two-dimensional array or multi-dimensional array.

[0018] It should be noted that the array to be processed is used to store a group of data elements of the same type.

[0019] It should be noted that the elements in a numeric array are all numeric types, such as integers, floating-point numbers, etc. The elements in a character array are all character types, which are used to store strings or character sequences. The elements in a pointer array are all pointer types, which are used to store the addresses of variables. The elements in a structure array are all structure types. A structure is a custom data type that can contain multiple fields of different types. A structure array is used to store data sets with the same structure.

[0020] It should be noted that a one-dimensional array contains a series of elements of the same data type, which are accessed through a single index. A two-dimensional array consists of a one-dimensional array, which can be regarded as a table or matrix; the elements in a two-dimensional array are accessed through two indexes (usually a row index and a column index). A multidimensional array consists of a one-dimensional or two-dimensional array with more than two dimensions. For example, a three-dimensional array can be regarded as consisting of a series of two-dimensional arrays.

[0021] It should be noted that in actual applications, users can determine the maximum number of iterative processing times for this round of data based on the subscript of the array. 10 , it can be determined that the maximum number of iterative processing times for this round of data is 10.

[0022] Here, before obtaining the array to be processed from the data source through the preset iterator node, the preset iterator node can be set in the preset drag editor through the preset code.

[0023] It should be noted that the preset drag-and-drop editor may be any type of editor. For example, the preset drag-and-drop editor may be a form design editor or a content management system editor, and this application does not make any limitation on this.

[0024] It should be noted that the data source may be data output by a front-end subprocess or a preset database; wherein the preset database may be a database that stores arrays as field types.

[0025] Step 102: The preset iterator node sequentially inputs the data in the array to be processed into the preset sub-process processing module through the first transmission channel. The preset sub-process processing module performs corresponding business logic processing on the data to obtain processing results corresponding to the data.

[0026] It should be noted that, assuming that the array to be processed contains N data, when performing business logic processing on the N data, the preset iterator node can input the first data in the array to be processed into the preset sub-process processing module for corresponding business logic processing; then input the second data in the array to be processed into the preset sub-process processing module for corresponding business logic processing;... then input the Nth data in the array to be processed into the preset sub-process processing module for corresponding business logic processing, so as to realize iterative loop processing.

[0027] It should be noted that business logic processing can represent the part of the application program that processes data required for business execution.

[0028] Taking the flash sale system of an e-commerce platform as an example, its business logic processing flow may include the following steps: User Verification: Verify whether the user is logged in and whether the user is eligible to participate in the flash sale.

[0029] Inventory check: Check whether the inventory of flash sale products is sufficient.

[0030] Order processing: If the inventory is sufficient, a flash sale order will be generated for the user and the inventory will be deducted.

[0031] Payment processing: Users pay the amount of the flash sale order.

[0032] Order confirmation: After successful payment, the order status is confirmed and the user is notified of the success of the flash sale.

[0033] Step 103: Send the processing result corresponding to the data to the preset iterator node through the second transmission channel; wherein the second transmission channel may be the same as or different from the first transmission channel.

[0034] It should be noted that the information transmitted by the second transmission channel is different from that transmitted by the first transmission channel. The first transmission channel transmits a certain data in the array to be processed; the second transmission channel transmits the processing result corresponding to the data in the array to be processed, or the second transmission channel transmits the processing result corresponding to the data in the array to be processed and the end iteration field carried by the data.

[0035] Step 104: Input the processing results corresponding to the data in the array to be processed into the output module through the preset iterator node for output.

[0036] Here, when outputting the processing results, the processing results corresponding to all the data in the array to be processed can be input into the output module through the preset iterator node for output; or, the processing results corresponding to part of the data in the array to be processed can be input into the output module through the preset iterator node for output.

[0037] Here, when outputting the processing results corresponding to all data in the array to be processed, when the preset sub-process processing module determines that all data in the array to be processed are valid data, the processing results corresponding to all data in the preset iterator node can be input into the output module for output.

[0038] It should be noted that when the preset sub-process processing module determines whether the data in the array to be processed is valid data, it can obtain a data from the preset iterator node, determine its validity, and then perform business logic processing, and store the processing result in the preset iterator node after obtaining it; and then process the next data. In this way, when it is found that a certain data in the array to be processed is invalid, the data processing process can be exited in time to avoid wasting system resources.

[0039] Here, when outputting the processing results corresponding to part of the data in the array to be processed, when the preset sub-process processing module determines that part of the data in the array to be processed is valid data, the processing results corresponding to the valid data in the preset iterator node can be input into the output module for output.

[0040] It should be noted that in actual applications, not all data in the array to be processed are valid data. When the preset sub-process processing module identifies a data as invalid data through its index, the preset sub-process processing module stops processing the invalid data for business logic and sends the processing result and the end iteration field corresponding to the last valid data to the preset iterator node. In this way, the data processing process can be exited in time to avoid wasting system resources.

[0041] It should be noted that when the preset iterator node recognizes the end iteration field, it will stop sending data to the preset sub-process processing module.

[0042] In actual applications, the preset iterator node provides the preset sub-process processing module with the function of ending data processing in advance. First, in the processing flow of the preset sub-process processing module, the value of msg.iteratorEnd is assigned to on; when the business sub-process processing logic finds that there is no need to process subsequent messages (data), the subsequent data processing of this round can be ended in advance. It should be noted that not all sub-processes need to end the processing of subsequent data in advance.

[0043] Here, when outputting the processing results corresponding to the valid data in the array to be processed, the processing results and the end iteration field corresponding to the last data in the valid data input into the preset sub-process processing module can be sent to the preset iterator node through the second transmission channel; the end iteration field is identified by the preset iterator node, and when the end iteration field is true, the current round of data processing is ended, and subsequent data processing is not performed, and the processing results corresponding to the valid data are input into the output module for output.

[0044] It should be noted that the preset iterator node inputs the processing results corresponding to all valid data into the output module for output.

[0045] In practical applications, the output fields can be: { "id": "6fc4657619de69f6", "type": "subflow", "name": "Data Processing", "info": "", "category": "", "in": [ { "x": 60, "y": 80, "wires": [ { "id": "5ac43eb5dbbf0802" } ] } ], "out": [ { "x": 520, "y": 140, "wires": [ { "id": "ae1d3564cfaed922", "port": 0 } ] } ], "env": [], "meta": {}, "color": "#DDAA99" }. Here, after the processing results corresponding to the data in the array to be processed are input into the output module through the preset iterator node and output, the index field, the end iteration field and the data field of the current iteration corresponding to the data of the current iteration in the preset iterator node can be deleted.

[0046] It should be noted that after the processing result is output, the index field, the end iteration field and the data field of the current iteration corresponding to the data of the current iteration in the preset iterator node are deleted to avoid affecting the next round of data iteration processing.

[0047] In actual applications, the preset iterator node obtains the data to be processed from a data source such as a database, and iteratively processes the data to be processed, and then transmits the data to be processed to the preset sub-process processing module for specific business logic processing. After processing one piece of data, the preset sub-process processing module returns the processing result to the preset iterator node, and the preset iterator node inputs the next piece of data to the preset sub-process processing module, and performs business logic processing until all input data is processed and then output. If the preset sub-process processing module finds that it needs to end the processing early in the middle of processing the data (that is, the remaining data does not need to be processed further), the preset sub-process processing module can send an early end flag (end iteration field) to the preset iterator node, and the preset iterator node cancels the subsequent data iterative loop processing based on the early end flag. Figure 2 Another flowchart of a data processing method is shown.

[0048] The detailed implementation scheme of getting the preset iterator node by dragging and dropping the editor according to the preset is as follows: The programmer fills the iteration field into the preset drag-and-drop editor to obtain the preset iterator node, where the iteration field can be of Array type (i.e., arrays of various types); if the field type is not Array, this node will not do any processing; one input port and two output ports are set for the preset iterator node; the first output port of the preset iterator node is a single entity iteration port, which is connected to the iteration sub-process (preset sub-process processing module), and the data element of the current iteration can be obtained through msg.element; the second output port of the preset iterator node is connected to the output module, which is used to output the processing results.

[0049] It should be noted that after the preset sub-process processing module has processed the data, the node line can be connected back to the input end of the preset iterator node, and then the next data element iteration can be performed; It should be noted that the preset iterator node does not configure options for the end of iteration. If you need to end the iteration early, you can make a judgment in the preset sub-process processing module and assign the msg.iteratorEnd field to on at the end of the iteration loop (msg.iteratorEnd = 'on', which can be operated through the "Adjust Message Properties" node).

[0050] It should be noted that the iteratorIndex field is a field reserved by the preset iterator node. When the iteration loop ends, iteratorIndex, iteratorEnd, and element in msg are deleted synchronously. Among them, IteratorIndex is used to indicate the index corresponding to the current iteration round of data; iteratorEnd is used to indicate whether it is necessary to end the current round of data processing in advance; element is used to indicate the data that needs to be processed in the current iteration.

[0051] It should be noted that the preset iterator node supports dragging and dropping to process multiple data in a loop, without the need for OT personnel to develop code; it supports the function of ending data processing early, and supports the processing of multi-layer loop nested data (two-dimensional arrays, multi-dimensional arrays).

[0052] In this embodiment, a data processing method is provided, which obtains an array to be processed from a data source through a preset iterator node; wherein the preset iterator node is used to perform loop iterative processing on multiple data in the array to be processed; the preset iterator node sequentially inputs the data in the array to be processed into a preset sub-process processing module through a first transmission channel, and the preset sub-process processing module performs corresponding business logic processing on the data to obtain processing results corresponding to the data; the processing results corresponding to the data are sent to the preset iterator node through a second transmission channel; wherein the second transmission channel is different from the first transmission channel; and the processing results corresponding to the data in the array to be processed are input into an output module through the preset iterator node for output. Since the preset iterator node can perform loop iterative processing on a large amount of data, the preset sub-process processing module can perform corresponding business logic processing on a large amount of data, so that the processing efficiency of a large amount of data is improved.

[0053] The present invention also provides a data processing device 30, such as Figure 3 The structure block diagram of a data processing device shown in FIG. 30 includes an acquisition unit 301 and a processing unit 302. Among them: The acquisition unit 301 is used to acquire the array to be processed from the data source through a preset iterator node; wherein the preset iterator node is used to perform loop iterative processing on multiple data in the array to be processed, that is, to execute the above step 101.

[0054] The processing unit 302 is used for the preset iterator node to input the data in the array to be processed into the preset sub-process processing module in sequence through the first transmission channel, and the preset sub-process processing module performs corresponding business logic processing on the data to obtain the processing result corresponding to the data; sends the processing result corresponding to the data to the preset iterator node through the second transmission channel; wherein the second transmission channel is different from the first transmission channel; and inputs the processing result corresponding to the data in the array to be processed into the output module through the preset iterator node for output, that is, for executing the above steps 102-104. The specific functions of the processing unit 302 have been described in detail in steps 102-104 and will not be repeated here. The processing unit 302 is connected to a display to display the processing result corresponding to the data.

[0055] An embodiment of the present application provides a data processing device, the data processing device comprising: an acquisition unit for acquiring an array to be processed from a data source through a preset iterator node; wherein the preset iterator node is used to perform loop iterative processing on multiple data in the array to be processed; a processing unit is used for the preset iterator node to sequentially input the data in the array to be processed into a preset sub-process processing module through a first transmission channel, the preset sub-process processing module performs corresponding business logic processing on the data to obtain the processing result corresponding to the data; the processing result corresponding to the data is sent to the preset iterator node through a second transmission channel; wherein the second transmission channel is different from the first transmission channel; the processing result corresponding to the data in the array to be processed is input into an output module through the preset iterator node for output. Since the preset iterator node can implement loop iterative processing on a large amount of data, the preset sub-process processing module can implement corresponding business logic processing on a large amount of data, so that the processing efficiency of a large amount of data is improved.

[0056] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0057] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining an array to be processed from a data source through a preset iterator node; wherein the preset iterator node is used to perform loop iterative processing on multiple data in the array to be processed; The preset iterator node sequentially inputs the data in the array to be processed into a preset sub-process processing module through the first transmission channel, and the preset sub-process processing module performs corresponding business logic processing on the data to obtain processing results corresponding to the data; Sending the processing result corresponding to the data to the preset iterator node through a second transmission channel; wherein the second transmission channel is different from the first transmission channel; The processing results corresponding to the data in the array to be processed are input into an output module through the preset iterator node for output.

2. The method according to claim 1, characterized in that Before obtaining the array to be processed from the data source through the preset iterator node, the method further includes: The preset iterator node is set in the preset drag editor through the preset code.

3. The method according to claim 1 or 2, characterized in that The array to be processed is classified according to element type, including: numerical array, character array, pointer array or structure array; The array to be processed is classified according to the dimension, including: one-dimensional array, two-dimensional array or multi-dimensional array.

4. The method according to claim 1 or 2, characterized in that: The step of inputting the processing result corresponding to the data in the to-be-processed array into an output module through the preset iterator node for output includes: The processing results corresponding to all the data in the array to be processed are input into the output module through the preset iterator node for output; Alternatively, the processing results corresponding to part of the data in the array to be processed are input into the output module through the preset iterator node for output.

5. The method according to claim 4, characterized in that The step of inputting the processing results corresponding to all the data in the to-be-processed array into the output module through the preset iterator node for output includes: When the preset sub-process processing module determines that all the data in the array to be processed are valid data, the processing results corresponding to all the data in the preset iterator node are input into the output module for output; Alternatively, the processing result corresponding to part of the data in the to-be-processed array is input into the output module through the preset iterator node for output, including: When the preset sub-process processing module determines that part of the data in the array to be processed is valid data, the processing result corresponding to the valid data in the preset iterator node is input into the output module for output.

6. The method according to claim 5, characterized in that The step of inputting the processing result corresponding to the valid data in the preset iterator node into the output module for output includes: Sending the processing result and the end iteration field corresponding to the last data in the valid data input into the preset sub-process processing module to the preset iterator node through the second transmission channel; The end iteration field is identified through the preset iterator node, and the processing result corresponding to the valid data is input into the output module for output.

7. The method according to claim 6, characterized in that After the processing result corresponding to the data in the to-be-processed array is input into an output module through the preset iterator node for output, the method further includes: The index field, the end iteration field and the data field of the current iteration corresponding to the data of the current iteration in the preset iterator node are deleted.

8. A data processing device, characterized in that: The data processing device comprises: An acquisition unit, used for acquiring an array to be processed from a data source through a preset iterator node; wherein the preset iterator node is used for performing loop iterative processing on multiple data in the array to be processed; A processing unit is used for the preset iterator node to input the data in the array to be processed into a preset sub-process processing module in sequence through a first transmission channel, and the preset sub-process processing module performs corresponding business logic processing on the data to obtain a processing result corresponding to the data; sends the processing result corresponding to the data to the preset iterator node through a second transmission channel; wherein the second transmission channel is different from the first transmission channel; and inputs the processing result corresponding to the data in the array to be processed into an output module through the preset iterator node for output.

9. A computer-readable storage medium, characterized in that: The method comprises a program which can be executed by a processor to implement the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images