Efficient execution data segmentation and flow transfer method of low-code data processing platform

Abstract

The application discloses a kind of low-code data processing platform's efficient execution data division and flow method, comprising the following steps: S1, user creates data processing component by drag operation;S2, create a node object for each node;S3, system constructs complete node network graph according to the front and rear relationship between node;S4, identify and divide out the sublink that can be executed independently from other nodes;S5, further stage division is carried out to the sublink identified and divided out in step S4;S6, the sublink in each stage is carried out parallel scheduling and execution;S7, the output result of each sublink is integrated by system;S8, system ends entire data processing procedure.The application has the following beneficial effects: 1, enhanced data processing and analysis capability;2, improved data visualization;3, ability to deal with complex data flow;4, efficient data division and flow;5, improve data processing speed and efficiency;6, optimized resource utilization;7, simplified maintenance and upgrade.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an efficient data splitting and flow method for a low-code data processing platform. BACKGROUND

[0002] With the rapid development of information technology, data processing has become an indispensable part of daily operations for enterprises and organizations. In modern enterprises, the demand for data processing platforms is growing, as they need to process and analyze large amounts of data to support decision-making, business process optimization, and customer experience improvement. Traditional data processing platforms often require specialized programming knowledge and complex configuration, which limits the use of non-technical users and increases the cost of development and maintenance.

[0003] In the field of low-code development, there are more and more low-code development platforms for enterprises. They use visual methods to develop applications, allowing developers with different levels of experience to create web pages and mobile applications through graphical user interfaces, using drag-and-drop components and model-driven logic. These platforms establish data storage and reading driving by studying basic processes, forms, etc., to realize the display of data in visual applications.

[0004] For example, patent CN115826971A provides a method for pushing and receiving device data through a separate data processing engine, solving the problem of relying on underlying Internet of Things data processing, improving the flexibility of developing and deploying device visualization applications, patent CN117555935B discloses a data processing method and device for a low-code platform, aiming to improve the efficiency and flexibility of data processing, suitable for the management and analysis of multiple data sources, and patent CN118012963B provides a low-code data exploration device that can realize lightweight and fast custom exploration and analysis of multiple data sources, suitable for fine operation and business reporting and other application scenarios.

[0005] However, the above data processing methods have the following defects:

[0006] (1) Limitations of data source access and simple analysis: Many existing low-code platforms can only perform simple access and analysis of data sources, lacking the ability to handle complex data processing and analysis. These platforms often cannot perform in-depth data processing and analysis, resulting in low efficiency when handling large-scale or complex data sets, and failing to meet the needs of enterprises for deep data mining.

[0007] (2) Insufficient data visualization: Current technologies are difficult to achieve visualization of data at each node in a low-code platform. This limits user monitoring and understanding of data flow processes, affecting decision-making and business process optimization.

[0008] (3) Data flow complexity is not well addressed: Existing low-code platforms mostly support simple one-way data flow, but when faced with complex data flow forms such as one-to-many and many-to-one, they often cannot effectively address them. This requires the platform to have more advanced data routing and processing capabilities to adapt to changing business needs.

[0009] (4) Lack of efficient data splitting and flow method: Current low-code platforms do not explicitly provide an efficient data splitting method and data flow form. When dealing with large-scale data, data splitting and intelligent routing are key to improving data processing efficiency. The lack of such mechanisms can lead to data processing bottlenecks, affecting the overall system performance.

[0010] Therefore, it is necessary to make further improvements. SUMMARY

[0011] The purpose of the present application is to provide an efficient execution data splitting and flow method for a low-code data processing platform that has high data processing efficiency, high data visualization, strong ability to handle complex data flow, efficient data splitting and flow, high data processing speed and efficiency, high resource utilization, and simple maintenance and upgrading, to overcome the shortcomings of the prior art.

[0012] An efficient execution data splitting and flow method for a low-code data processing platform designed to achieve this purpose, characterized by the following steps:

[0013] S1, according to the data processing components selected and placed by dragging and the connection relationship between components, the user creates data processing components through dragging operation, after the dragging is completed, the system obtains the parameters and related information of all nodes;

[0014] S2: The system traverses all nodes involved in data processing and creates a node object for each node;

[0015] S3: The system constructs a complete node network graph according to the front and back relationship between nodes;

[0016] S4: The system uses a breadth-first search strategy to traverse the constructed node network graph and identifies sub-links that can be executed independently of other nodes;

[0017] S5: Further stage division is performed on the sub-links identified in step S4, and the system assigns them to different execution stages according to their dependency relationships;

[0018] S6: According to the new stage division result, the system uses a parallel scheduling algorithm to perform parallel scheduling and execution of sub-links in each stage in order;

[0019] S7: After each stage of sub-link completes its respective execution task, the system will integrate the output results of each sub-link and pass the results to the next stage of sub-link according to the dependency relationship;

[0020] S8: After all sub-links are executed and the data flow is completed, the system will end the entire data processing process and verify and post-process the final results.

[0021] In step S1, the relevant information of the node includes node ID, node name, node pointing ID, node pointed ID, and node execution method.

[0022] In step S2, the steps of creating a node object are as follows:

[0023] S21, initially traverse all nodes to complete node object class initialization;

[0024] S22, according to the connection pointing relationship of the node, determine the node type and assign the node type attribute.

[0025] In step S21, the node object includes the following attributes: node ID, node name, child node ID, parent node ID, node execution method, node type, child node object list, parent node object list, and node execution result.

[0026] In step S3, the steps of constructing a node network diagram are as follows:

[0027] S31, establish the connection relationship between nodes according to their front node ID and back node ID to construct a node network diagram;

[0028] S32, after the node network diagram is constructed, the system will verify the diagram to ensure that there is no loop.

[0029] In step S32, the system checks for loops when verifying the node network diagram. The loop detection method is: the system traverses the node network through algorithms such as depth-first search, checks the front and back relationships of each node, and ensures that there is no repeated access to nodes in the diagram. If a loop is detected, the system will issue a warning and require the user to check the connection relationship. If no loop is detected, the system will continue to the next step.

[0030] In step S4, the sub-link identification and segmentation steps are as follows:

[0031] S41, find all nodes of type "one-way pointing" as starting points for searching;

[0032] S42, traverse all starting points and search for simple sub-links;

[0033] S43, traverse the entire network diagram to complete sub-link segmentation.

[0034] In step S42, the retrieval step of simple sub-links is as follows:

[0035] S421, search from the starting point: for each found starting point node, the system will start a breadth-first search, traversing its subsequent nodes until a complete sub-link is found;

[0036] S422, during the retrieval process, the system will record each complete sub-link;

[0037] S423, when a complete sub-link is found, the last node of the sub-link will be taken as a new starting point to continue searching for the next simple sub-link.

[0038] In step S5, the phase division step of the sub-link is as follows:

[0039] S51, the system will traverse all sub-links to find those links whose end nodes have no subsequent nodes, and mark them as "terminal links";

[0040] S52, taking the terminal links defined in step S51 as the starting point, the parent links are retrieved in reverse to divide new phases.

[0041] In step S8, when the system ends the entire data processing flow, the system will record the execution process of the entire data processing flow in detail, including the execution time of each node, data flow, resource consumption, etc., and the system will inform the user through a message or notification mechanism that the data processing has been completed, and provide access methods and download links for the final results.

[0042] The efficient data splitting and flow method of the low-code data processing platform of the present application has the following beneficial effects:

[0043] (1) Enhanced data processing and analysis capabilities:

[0044] By constructing node objects and parsing the connection and dependency relationships between nodes, the present scheme provides a mechanism that enables the low-code platform to handle more complex data sets and analysis tasks, which overcomes the low efficiency of existing platforms in handling large-scale or complex data sets, and meets the needs of enterprises for deep data mining;

[0045] (2) Improved data visualization:

[0046] The present scheme allows users to more clearly monitor and understand the data flow process through efficient data flow, and this improved data visualization capability helps users make better decisions and optimize business processes, thereby solving the deficiencies of existing technologies in data visualization;

[0047] (3) Ability to handle complex data flow:

[0048] By stage segmentation and parallel execution of sub-links in the same stage, the scheme provides a method for effectively dealing with one-to-many, many-to-one and other complex data flow forms. This advanced data routing and processing capability enables the platform to adapt to changing business needs, solving the shortcomings of existing platforms in dealing with complex data flow;

[0049] (4) Efficient data segmentation and flow:

[0050] The scheme provides an efficient data segmentation method and data flow form through stage segmentation and intelligent routing mechanism, which helps to improve the efficiency of large-scale data processing and solve the bottleneck problem of existing platforms in data segmentation and flow, thereby improving the overall system performance;

[0051] (5) Improve data processing speed and efficiency:

[0052] By parallel execution of sub-links in the same stage, the scheme significantly improves the speed and efficiency of data processing, which allows multiple sub-links to be performed simultaneously, reducing waiting time and speeding up the overall data processing process;

[0053] (6) Optimized resource utilization:

[0054] By identifying sub-links that can be executed independently, the scheme allows resources to be more effectively allocated and utilized among multiple sub-links, which helps to improve the performance and response speed of the entire platform.

[0055] (7) Simplified maintenance and upgrade:

[0056] Since the scheme provides clear dependencies between nodes and sub-links, it makes maintenance and upgrade of the platform much simpler, allowing maintenance personnel to more easily identify and solve potential problems, thereby improving the stability and reliability of the platform. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The flowchart of the data segmentation and flow method in an embodiment of the present application.

[0058] Figure 2 The node connection diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application will be further described below in conjunction with the drawings and examples.

[0060] Referring to Figures 1-2 , the efficient data segmentation and flow method of the low-code data processing platform includes the following steps:

[0061] S1. Based on the data processing components selected and placed by dragging and dropping and the connection relationship between the components, the user creates the data processing components by dragging and dropping. The data processing components are as follows: Figure 1 As shown, after dragging is completed, the system obtains the parameters and related information of all nodes; a brief example of specific node information is as follows:

[0062] {

[0063] "nodeList": [

[0064] {

[0065] "name": "Taxi Data",

[0066] "nodeType": "jdbc",

[0067] "nodeId": "211ukci8w1694071821570",

[0068] "preNodeId": [],

[0069] "nextNodeId": ["fzztb5hwdo1694072985930"]

[0070] },

[0071] {

[0072] "name": "District",

[0073] "nodeType": "jdbc",

[0074] "nodeId": "cz9qpo3qoa1694071996912",

[0075] "preNodeId": [],

[0076] "nextNodeId": ["uzhrhitl0n1694074170505"]

[0077] },

[0078] {

[0079] "name": "Date Format",

[0080] "nodeType": "date_format",

[0081] "nodeId": "fzztb5hwdo1694072985930",

[0082] "preNodeId": ["211ukci8w1694071821570"],

[0083] "nextNodeId": ["9fgte19s381694073011362"]

[0084] },

[0085] {

[0086] "name": "Deduplication Component",

[0087] "nodeType": "drop_duplicates",

[0088] "nodeId": "9fgte19s381694073011362",

[0089] "preNodeId": ["fzztb5hwdo1694072985930"],

[0090] "nextNodeId": ["p4tjgcsmee1694073021049"]

[0091] },

[0092] {

[0093] "name": "Aggregation component",

[0094] "nodeType": "aggregate",

[0095] "nodeId": "p4tjgcsmee1694073021049",

[0096] "preNodeId": ["9fgte19s381694073011362"],

[0097] "nextNodeId": ["qksblwe7361694074224254"]

[0098] },

[0099] {

[0100] "name": "Database",

[0101] "nodeType": "jdbc",

[0102] "nodeId": "fno7yccvhb1694073885056",

[0103] "preNodeId": [],

[0104] "nextNodeId": ["umrebv4s91694074155487"]

[0105] },

[0106] {

[0107] "name": "Field rename",

[0108] "nodeType": "rename_columns",

[0109] "nodeId": "umrebv4s91694074155487",

[0110] "preNodeId": ["fno7yccvhb1694073885056"],

[0111] "nextNodeId": ["uzhrhitl0n1694074170505"]

[0112] },

[0113] {

[0114] "name": "Merge Component",

[0115] "nodeType": "row_concat",

[0116] "nodeId": "uzhrhitl0n1694074170505",

[0117] "preNodeId": ["cz9qpo3qoa1694071996912",

[0118] "umrebv4s91694074155487"],

[0119] "nextNodeId": ["41v7qx4p691694074203079"]

[0120] },

[0121] {

[0122] "name": "Complete Division",

[0123] "nodeType": "drop_duplicates",

[0124] "nodeId": "41v7qx4p691694074203079",

[0125] "preNodeId": ["uzhrhitl0n1694074170505"],

[0126] "nextNodeId": ["qksblwe7361694074224254"],

[0127] "params": {

[0128] "input": "uzhrhitl0n1694074170505",

[0129] "keep": "first",

[0130] "subset": []

[0131] }

[0132] },

[0133] {

[0134] "name": "Statistical results",

[0135] "nodeType": "inner_join",

[0136] "nodeId": "qksblwe7361694074224254",

[0137] "preNodeId": ["p4tjgcsmee1694073021049","41v7qx4p691694074203079"],

[0138] "nextNodeId": []

[0139] } ]

[0141] }

[0142] The specific information of the above code is as follows:

[0143] (1) Node ID: Each node has a unique identification code. For example, the ID of the taxi data node is 211ukci8w1694071821570.

[0144] (2) Node name: Each node has a corresponding name, such as "Taxi Data" or "Date Formatting";

[0145] (3) Node pointing ID: Describes the connection relationship of a node pointing to other nodes. For example, the "Taxi Data" node points to the "Date Formatting" node through a connection;

[0146] (4) Node pointed ID: Describes the connection relationship of a node being pointed to by other nodes. For example, the "Date Formatting" node is pointed to by the "Taxi Data" node;

[0147] (5) Node execution method: The specific function performed by each node, such as JDBC connection, deduplication, aggregation, etc.

[0148] S2, after obtaining the node information in step S1, the system traverses all nodes participating in data processing, and creates a node object for each node, as follows:

[0149] S21, initially traverse all nodes to complete node object class initialization; the node object includes the following attributes:

[0150] (1) Node ID: Each node has a unique identification code (for example, UUID4 format), such as the ID of the "Taxi Data" node is 211ukci8w1694071821570.

[0151] (2) Node name: Identifies the name of the node, such as "zone" or "deduplication component".

[0152] (3) Sub-node ID: The ID of the other sub-node pointed to by this node, such as the sub-node of the "Taxi Data" node is the "Date Formatting" node.

[0153] (4) Parent node ID: The parent node ID pointing to this node, such as the parent node of the "deduplication component" node is the "Date Formatting" node.

[0154] (5) Node execution method: The corresponding data processing method, such as JDBC reading, field renaming, deduplication, or aggregation.

[0155] In addition, in addition to the information provided by the node itself, the system will automatically generate additional attributes to facilitate the construction of a complete graph network structure. During initialization, the following attributes will be left blank and will be supplemented in the next step.

[0156] (6) Node type: Such as bidirectional type, single-pointing type, or single-pointed-in type.

[0157] (7) Sub-node object list: a list of sub-node objects built according to the sub-node IDs, for example, the sub-node pointed by the "aggregation component" is the "statistical result".

[0158] (8) Parent node object list: a list of parent node objects built according to the parent node IDs.

[0159] (9) Node execution result: when a node is executed, if there is an output result, it is stored in this attribute.

[0160] S22, according to the connection direction relationship of the nodes, the node type is determined and the node type attribute is assigned. For example:

[0161] The "taxi data" node has no dependent node and serves as a data source, so it is the head node of the entire link. This node is a one-way pointing type;

[0162] The "date formatting" node depends on the "taxi data" node to provide data as its input, and the processing result of this node also needs to be used as the input of other nodes, so this node is a bidirectional type;

[0163] The "statistical result" node is located at the end, which depends on the results of multiple parent nodes, but its result does not need to be passed to other nodes, so this node is a one-way pointing type;

[0164] After the system judges, the type of each node is determined and the node type is labeled.

[0165] S3: In this step, the system will traverse each node dragged and configured from the user interface, and build a complete node network graph according to the front and back relationship between nodes; the following is a detailed description of the specific steps:

[0166] S31, according to its preNodeId (predecessor node ID) and nextNodeId (successor node ID), the connection relationship between nodes is established to build a node network graph; for example:

[0167] For the "date formatting" node, its predecessor node is "taxi data", and the system will add the node object of "date formatting" to the sub-node object list of "taxi data"; similarly, the system will also add the ID of each node to the parent node object list of its successor node to form a complete node linked list structure.

[0168] Through this process, the final node network graph will show the dependency relationship and execution order between nodes.

[0169] S32, after building the node network graph, the system will verify the graph to ensure that there is no loop; the specific operation includes:

[0170] (1) Loop detection: The system traverses the node network using algorithms such as Depth-First Search (DFS) to check the pre- and post-relationships of each node, ensuring that there is no repeated access to nodes in the graph. This step is crucial to ensure smooth and efficient data processing flow, avoiding potential infinite loop problems.

[0171] (2) Result confirmation: If a loop is detected, the system will issue a warning and require the user to check the connection relationship; if the network graph is valid (i.e., no loop is detected), proceed to the next step.

[0172] S4: The system uses a Breadth-First Search (BFS) strategy to traverse the constructed node network graph, identifying sub-links that can be executed independently of other nodes. These sub-links are simple sub-links that can be processed in parallel without pre-dependencies, thus speeding up the overall data processing flow. A simple sub-link is defined as a unidirectional link with only the start node having multiple parent nodes and the end node having multiple child nodes.

[0173] The identification and segmentation of sub-links is as follows:

[0174] S41, Find all nodes of type "unidirectional point" as the starting point of the search; in the example, nodes such as "taxi data" and "district" are of the unidirectional point type.

[0175] S42, Traverse all starting points and search for simple sub-links; the specific steps are as follows:

[0176] S43, Traverse the entire network graph to complete the sub-link segmentation.

[0177] During the traversal of the entire node network graph, the system ensures that each starting point is searched until all possible sub-links are identified. Finally, the system will aggregate all identified simple sub-links and prepare for subsequent stage division; for example, the identified sub-links may include:

[0178] Sub-link 1: ["taxi data", "date formatting", "de-duplication component", "aggregation component"]

[0179] Sub-link 2: ["district"]

[0180] Sub-link 3: ["database", "field renaming"]

[0181] Sub-link 4: ["merge component", "complete district"]

[0182] Sub-link 5: ["statistical results"]

[0183] In step S42, the search steps for simple sub-links are as follows:

[0184] S421、Search from start: For each found start node, the system will launch a breadth-first search (BFS) to traverse its subsequent nodes until a complete sublink is found;

[0185] S422、Record sublinks: During the search process, the system will record each complete sublink; for example, starting from the "Taxi Data" node, sequentially passing through the "Date Formatting" and "De-duplication Component" nodes, a complete sublink is formed:

[0186] Sublink example: ["Taxi Data", "Date Formatting", "De-duplication Component", "Aggregation Component"]

[0187] S423、Update search start: When a complete sublink is found, the last node of the sublink will be taken as the new start to continue searching for the next simple sublink; in this way, the system will continuously search for other potential sublinks.

[0188] S5: Further stage division is performed on the sublinks identified in step S4, and the system assigns them to different execution stages according to their dependency relationships; the specific steps are as follows:

[0189] S51、The system will traverse all sublinks to find those links with no subsequent nodes at the end node, and mark them as "terminal links"; according to the sublink division in step S4, "Statistical Results" is the end node, so it belongs to stage 1, indicating that it will be executed as the final sublink;

[0190] S52、Take the terminal link defined in step S51 as the starting point to reverse search the parent link and divide new stages; the specific steps are as follows:

[0191] (1) Reverse search: Starting from the "Statistical Results" node, search its parent nodes forward to find the sublinks that depend on it; according to the data, "Statistical Results" depends on "Aggregation Component" and "Complete Zone" nodes, and the sublinks corresponding to these two nodes are divided into the second stage. That is, sublink 1: ["Taxi Data", "Date Formatting", "De-duplication Component", "Aggregation Component"] and sublink 4: ["Merge Component", "Complete Zone"] are in the second stage;

[0192] (2) Gradual division: The system continues to reverse search the parent nodes of the sublinks to find the preceding nodes in turn and continue to divide new stages; the final stages are as follows:

[0193] Stage 1: Sublink 5

[0194] Stage 2: Sublink 1, Sublink 4

[0195] Stage 3: Sublink 2, Sublink 3

[0196] S53: Reverse the order of the stages divided in S52, complete the final stage division. The final stage division is:

[0197] Stage 1: Sublink 2, Sublink 3

[0198] Stage 2: Sublink 1, Sublink 4

[0199] Stage 3: Sublink 5

[0200] Through the S5 step, the system completes the stage division of all sublinks; the sublinks in each stage are independent of each other and can be executed in parallel, while the stages are executed in series according to the dependency relationship.

[0201] S6: According to the new stage division result, the system uses a parallel scheduling algorithm to schedule and execute the sublinks in each stage in order;

[0202] (1) Stage 1: ["Districting"] and ["Database", "Field Renaming"]

[0203] In the first stage, the "Districting" sublink and the "Database" sublink can be executed in parallel; since they have no pre-dependency relationship, the system will allocate computing resources for these two sublinks at the same time for parallel processing.

[0204] (2) Stage 2: ["Taxi Data", "Date Formatting", "De-duplication Component", "Aggregation Component"] and ["Merge Component", "Complete Districting"]

[0205] In the second stage, the "Taxi Data" link and the "Merge Component" link depend on the completion of the sublinks in the first stage; when the first stage is completed, the system will schedule these two links in parallel and dynamically optimize the allocation of resources according to the resource situation.

[0206] (3) Stage 3: ["Statistical Results"]

[0207] The last stage, "Statistical Results", as the end point of the process, depends on the output of the second stage; when all the pre-link is completed, the system will schedule and execute the link.

[0208] S7: After the sublinks of each stage complete their respective execution tasks, the system will integrate the output results of each sublink and pass the results to the sublinks of the next stage according to the dependency relationship, ensuring the coherence and accuracy of the entire data processing process.

[0209] (1) Stage 1 Integration:

[0210] After the execution of sublink 2 (["division"]) and sublink 3 (["database", "field renaming"]), the system will pass their output results to the sublinks in stage 2, and the output of sublink 2 will be passed to sublink 4 (["merge component", "complete division"]).

[0211] (2) Integration of stage 2:

[0212] After the execution of sublink 1 (["taxi data", "date formatting", "de-duplication component", "aggregation component"]) and sublink 4 (["merge component", "complete division"]), the system will integrate the results of the two sublinks. At this time, the processed results of taxi data will be associated with the data of complete division in sublink 5 (["statistical results"]) of the next stage.

[0213] S8: After all sublinks are executed and data flow is completed, the system will end the entire data processing process and perform verification and post-processing on the final results to ensure the quality and usability of the results.

[0214] When sublink 5 (["statistical results"]) of stage 3 is executed, the system marks the completion of the entire data processing process; the execution status of all data processing tasks will be updated to "completed" for subsequent operations or user queries.

[0215] When the system ends the entire data processing process, the system will record detailed logs of the execution process of the entire data processing process, including the execution time of each node, data flow, resource consumption, etc. These logs can be used for subsequent performance optimization, debugging, and tracking of data processing paths.

[0216] Finally, the system will inform the user through messages or notification mechanisms that the data processing has been completed, and provide access methods and download links for the final results.

[0217] The above is the preferred scheme of the present application, which shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for efficiently executing data segmentation and circulation on a low-code data processing platform, characterized by: The following steps are involved: S1. Based on the data processing components selected and placed by dragging and dropping and the connection relationships between the components, the user creates the data processing components by dragging and dropping. After the dragging is completed, the system obtains the parameters and related information of all nodes; S2. The system traverses all nodes involved in data processing and creates a node object for each node; S3. The system builds a complete node network graph based on the previous and next relationships between nodes; S4. The system uses a breadth-first search strategy to traverse the constructed node network graph and identify and segment sub-links that can be executed independently of other nodes; S5. Further stage division is performed on the sub-links identified and segmented in step S4. The system assigns them to different execution stages based on their dependencies. S6. Based on the new stage division results, the system uses a parallel scheduling algorithm to parallel schedule and execute the sub-links in each stage in the order of the stages. S7. After each sub-link completes its respective execution task, the system will integrate the output results of each sub-link and pass the results to the sub-link of the next stage according to the dependency relationship; S8. After all sub-links are executed and data transfer is completed, the system will end the entire data processing process and verify and post-process the final results; In step S4, the steps of identifying and segmenting the sub-links are as follows: S41. Find all nodes of type "one-way pointing" as the starting point of the search; S42, traverse all starting points and retrieve simple sub-links; S43, traverse the entire network graph to complete sub-link cutting; In step S42, the steps for retrieving a simple sub-link are as follows: S421. Start searching from the starting point: For each starting point node found, the system will start a breadth-first search and traverse its subsequent nodes until a complete sub-link is found; S422. During the retrieval process, the system will record each complete sub-link; S423: After a complete sub-link is found, the last node of the sub-link is used as a new starting point to continue searching for the next simple sub-link. In step S5, the sub-link phase division steps are as follows: S51, the system will traverse all sub-links, find those links whose end nodes have no subsequent nodes, and mark them as "end links"; S52: Starting from the end link defined in step S51, reversely search for the parent link to divide the new stage.

2. The method for efficiently executing data segmentation and circulation on a low-code data processing platform according to claim 1 is characterized in that: In step S1, the relevant information of the node includes the node ID, node name, node pointing ID, node pointed ID and node execution method.

3. The method for efficiently executing data segmentation and circulation on a low-code data processing platform according to claim 1 is characterized in that: In step S2, the steps for creating a node object are as follows: S21, traverse all nodes for the first time and complete the node object class initialization; S22. According to the connection direction relationship of the nodes, the node type is determined and the node type attribute is assigned.

4. The method for efficiently executing data segmentation and circulation on a low-code data processing platform according to claim 3 is characterized by: In step S21 , the node object includes the following attributes: node ID, node name, child node ID, parent node ID, node execution method, node type, child node object list, parent node object list and node execution result.

5. The method for efficiently executing data segmentation and circulation on a low-code data processing platform according to claim 1 is characterized in that: In step S3, the steps for constructing the node network graph are as follows: S31, establishing a connection relationship between nodes according to their preceding node IDs and succeeding node IDs to construct a node network graph; S32. After constructing the node network graph, the system will verify the graph to ensure that there are no loops.

6. The method for efficiently executing data segmentation and circulation on a low-code data processing platform according to claim 5 is characterized in that: In step S32, the system performs loop detection when verifying the node network graph. The loop detection method is: the system traverses the node network through the depth-first search algorithm, checks the predecessor and successor relationships of each node, and ensures that there is no repeated access to nodes in the graph; if a loop is detected, the system will issue a warning and ask the user to check the connection relationship; if no loop is detected, the system proceeds to the next step.

7. The method for efficiently executing data segmentation and circulation on a low-code data processing platform according to claim 1 is characterized in that: In step S8, when the system ends the entire data processing process, the system will log the execution process of the entire data processing process in detail, including the execution time of each node, data flow, and resource consumption. The system will inform the user that the data processing is completed through a message or notification mechanism, and provide access methods and download links for the final results.

Images