Data link optimization processing method and data link optimization processing device

Abstract

The invention provides a data link optimization processing method and a data link optimization processing device, a directed acyclic graph is generated according to data link configuration, a data relationship in a data framework is represented based on the directed acyclic graph, the logic and sequence of data execution are completely controllable, and the implementation of the directed acyclic graph supports asynchronous and parallel execution of data. And after the directed acyclic graph is generated, inputting to-be-processed data of the data processing service into a head node of the directed acyclic graph, so that each node executes corresponding data processing on the received data according to a streaming sequence. Only when the data arriving at the node triggers the operation condition of the node, the relay unit controls the node to execute data processing, so that each node can keep the streaming sequence of the directed acyclic graph in the data processing process, the data processing sequence and rule can be ensured to conform to the directed acyclic graph configuration even under the condition of multi-data concurrence, and the data processing efficiency is improved. And finally, obtaining a final data processing result through a tail node of the directed acyclic graph.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method and device for optimizing data link processing. Background Art

[0002] A Directed Acyclic Graph (DAG) is a data structure widely used in data processing and task scheduling. Its core feature is to represent the dependency relationships between tasks or data through the combination of nodes and edges, and there are no loops in the graph, thus ensuring that tasks or data can be executed or processed in a certain order. In scenarios such as distributed computing, data processing, and parallel computing, DAGs are widely used to describe complex data processing flows and task dependencies.

[0003] In the prior art, DAGs are usually used to process large-scale data streams or high-frequency events. For example, in a streaming data processing system, data is divided into multiple events or tasks and processed sequentially through the nodes in the DAG. Each node may depend on the output result of the previous node to ensure the correctness, stability, and consistency of the data. However, when the data volume is particularly large or the processing frequency is particularly high, especially when the acquisition frequencies of each input data of the nodes vary greatly, the conventional DAG may have data out-of-order problems. If the data order is incorrect or the intermediate values corresponding to different input data are confused with each other, the downstream nodes may receive incorrect or incomplete input data, resulting in incorrect calculation results or data link crashes. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and device for optimizing data link processing, which can enable each node to maintain the streaming order of the DAG during data processing. Even in the case of multi-data concurrency, it can ensure that the data processing order and rules conform to the DAG configuration, and finally obtain the final data processing result through the tail node of the DAG.

[0005] In a first aspect, an embodiment of this application provides a method for optimizing data link processing. The method for optimizing data link processing is implemented based on a DAG, and the method for optimizing data link processing includes: Generate at least one directed acyclic graph according to at least one data link configuration; wherein, each directed acyclic graph includes at least two nodes, each pair of nodes is connected by a directed edge, and each node includes at least one relay unit, and the relay unit is used to store and manage the data arriving at the node. When the data arriving at the node triggers the operating condition of the node, the relay unit controls the node to perform data processing. If the data arriving at the node does not meet the operating condition of the node, the relay unit controls the node not to perform data processing.

[0006] Further, each relay unit of each node includes at least one input data queue and at least one output data queue, and the operating condition of the node includes at least: there is preset input data in all input data queues corresponding to the node.

[0007] Further, the number of input data queues of the node is determined according to the number of input directed edges connected to the node, and the number of output data queues of the node is determined according to the number of output directed edges connected to the node, wherein the data in each output data queue is the same.

[0008] Further, in response to the existence of a data execution condition configuration in the data link configuration, a data processing expression is configured for the corresponding directed edge. The data link optimization processing method further includes: In response to transmission data passing through a first directed edge, the transmission data is processed based on the data processing expression, and the first directed edge is a directed edge with a data processing expression.

[0009] Further, in response to the existence of a data output condition configuration in the data link configuration, a data output condition is configured for the corresponding node. The data link optimization processing method further includes: In response to a first node outputting a data processing result, based on the data processing result and the data output condition, a target node is determined from at least one downstream node of the first node, and the processed data processing result is transmitted to the target node only through the output data queue. The first node is a node with a data output condition.

[0010] Further, the data link optimization processing method further includes: The processed data processing result is transmitted to the target node only through the output data queue, and a second signal is transmitted to other downstream nodes, and the second signal is an empty signal.

[0011] Further, at least one basic node is included in the nodes of the directed acyclic graph; the data link optimization processing method further includes: In response to an exception occurring in the base node, a first exception signal is transmitted to the downstream nodes of the base node where the exception occurs, and the downstream nodes continue to transmit the first exception signal in the direction of the directed edge; the base node represents a necessary node for the directed acyclic graph to implement the main data processing function, and the first exception signal is an exception signal indicating that an exception has occurred in the base node.

[0012] Further, in response to the existence of a data enhancement processing step in the data link configuration, enhancement nodes are further included in the directed acyclic graph; the data link optimization processing method further includes: In response to an exception occurring in the enhancement node, a second exception signal is transmitted to the downstream enhancement nodes of the enhancement node where the exception occurs, and the downstream enhancement nodes continue to transmit the second exception signal in the direction of the directed edge; the enhancement node represents a node in the directed acyclic graph that implements the data enhancement processing step, and the second exception signal is an exception signal indicating that an exception has occurred in the enhancement node.

[0013] Further, the data link optimization processing method further includes: In response to a node receiving a third indication signal, the node interrupts its operation and transmits the third indication signal to the downstream nodes according to the order of the directed edges of the directed acyclic graph; Moreover, in response to the management unit detecting that any node has received the third indication signal, the third indication signal is input to the head node, so as to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph to interrupt the operation of the directed acyclic graph; wherein, the management unit is used to control and manage the directed acyclic graph, the third indication signal is used to indicate at least one node to interrupt its operation, and the head node represents the node in the directed acyclic graph that receives the initial input.

[0014] In a second aspect, an embodiment of the present application further provides a data link optimization processing device, which is implemented based on a directed acyclic graph, and the data link optimization processing device includes: A directed acyclic graph generation module, configured to generate at least one directed acyclic graph according to at least one data link configuration; wherein, each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit, and the relay unit is used to store and manage the data arriving at the node. When the data arriving at the node triggers the operating condition of the node, the relay unit controls the node to perform data processing. If the data arriving at the node does not meet the operating condition of the node, the relay unit controls the node not to perform data processing.

[0015] A data link optimization processing method and a data link optimization processing device provided by an embodiment of the present application generate a directed acyclic graph according to a data link configuration, represent the data relationship within a data framework based on the directed acyclic graph, and the logic and sequence of data execution are completely controllable. Moreover, the implementation of the directed acyclic graph supports asynchronous and parallel execution of data. After the directed acyclic graph is generated, the data to be processed in a data processing service is input into the head node of the directed acyclic graph. The relay unit can ensure that each node and each directed edge stably execute corresponding data processing on the received data in a streaming order. Whether the data to be processed is in a streaming form or a non-streaming form, only when the data reaching a node triggers the running condition of the node, the relay unit controls the node to execute data processing. In this way, it can be ensured that each node maintains the streaming order of the directed acyclic graph during the data processing process. Even in the case of multi-data concurrency, it can also ensure that the order and rules of data processing conform to the directed acyclic graph configuration. Finally, the final data processing result is obtained through the tail node of the directed acyclic graph.

[0016] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0018] Figure 1 Schematic diagrams of two common directed acyclic graphs provided by an embodiment of the present application; Figure 2 Schematic diagram of a directed acyclic graph with a relay unit provided by an embodiment of the present application; Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part rather than all of the embodiments of this application. The components of the embodiments of this application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application that is required to be protected, but merely represents the selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without creative efforts belongs to the scope of protection of this application.

[0020] First, an application scenario applicable to this application will be introduced. This application can be applied to the field of data processing technology.

[0021] A directed acyclic graph (DAG) is a data structure widely used in fields such as data processing and task scheduling. Its core feature is that it represents the dependency relationships between tasks or data through the combination of nodes and edges, and there are no loops in the graph, thus ensuring that tasks or data can be executed or processed in a certain order. In scenarios such as distributed computing, data processing, and parallel computing, DAGs are widely used to describe complex data processing flows and task dependency relationships.

[0022] Please refer to Figure 1 , Figure 1 which are schematic diagrams of two common directed acyclic graphs provided in the embodiments of this application. As Figure 1 shown, a directed acyclic graph includes several nodes, and each node is connected by a directed edge. A node can be a function or a function. The directed edge between nodes represents the dependency relationship between each node. For example, it can be the order or relationship of data transfer. For example, if the processing of step B must depend on the data processed by A, then its directed edge represents A→B. Several nodes and the dependency relationships between the nodes form a data processing link in at least one scenario. According to the actual business situation, a node can establish dependency relationships with several nodes. For each dependency relationship (that is, each edge), each node needs to be configured with at least one input relationship and at least one output relationship. Among them, there are no loops in the directed acyclic graph, that is, there is no path that starts from a certain node, passes through several nodes, and then returns to the original node. This ensures that there is no cycle in the graph and can be processed or calculated in a certain order.

[0023] The data processing forms of a directed acyclic graph are divided into a streaming form and a non-streaming form. Suppose in a data logic, the dependency relationship of each node is A→B→C→D→E→F. If it is a non-streaming scenario, there is only one data to be processed, and this data to be processed only needs to go through the process from node A to node F once; if it is a streaming scenario, there will be a large amount of data that repetitively executes the process of A→B→C→D→E→F in parallel until all data is processed. Moreover, in the streaming scenario, the processes between each data are independent and asynchronous. When node A receives input data 1, after being processed by node A, it forms output A1 and is passed to node B for processing. Subsequently, node A receives input data 2 and continues to execute A→B→C→D→E→F. The execution of input data 2 is independent of that of input data 1, that is, the processing process of each input data in the streaming scenario is independent of each other. Non-streaming can be regarded as streaming data that only needs to run once. Therefore, a directed acyclic graph configuration that supports streaming data processing must support the data processing of non-streaming scenarios.

[0024] It has been found through research that in the prior art, DAGs are usually used to process large-scale data streams or high-frequency events. For example, in a streaming data processing system, data is divided into multiple events or tasks and processed sequentially through the nodes in the DAG. Each node may depend on the output result of the previous node to ensure the correctness, stability, and consistency of the data. However, when the data volume is particularly large or the processing frequency is particularly high, especially when the acquisition frequencies of each input data of the nodes vary greatly, the directed acyclic graph with a conventional configuration may have data out-of-order problems. If the data order is incorrect or the intermediate values corresponding to different input data are confused with each other, the downstream node may receive incorrect or incomplete input data, resulting in incorrect calculation results or data link crashes.

[0025] Based on this, the embodiments of the present application provide a data link optimization processing method to keep the data processing link order of each input data stable, so that even in the case of multi-data concurrency, it can ensure that the data processing order and rules conform to the directed acyclic graph configuration, and finally obtain the final data processing result through the tail node of the directed acyclic graph.

[0026] The data link optimization processing method provided by the embodiments of the present application is implemented based on a directed acyclic graph. The data link optimization processing method includes: Generating at least one directed acyclic graph according to at least one data link configuration.

[0027] Here, in a directed acyclic graph, each data execution step is a node. The data link configuration includes at least each data execution step included in the data processing flow, and converts the configuration into the data transmission relationship between each node, the input parameter type and output parameter type of each node, node information, and node type. Specifically, the input parameter type and output parameter type refer to the data types of the input and output values of each node, such as common data types like int, string, float, bool, etc. The present application places no restrictions on the data type. Node information may include: node name, the specific implementation algorithm of the node. According to the embodiments provided by the present application, the node type may include: head node, output node, and tail node. Specifically, the head node represents the node in the directed acyclic graph that receives the initial input and is the node that needs to obtain the service input. The output node indicates that at least one output parameter of the output node is output to other nodes downstream. The tail node represents the node in the directed acyclic graph that finally outputs the processing result of the entire data processing flow. In the directed acyclic graph, the nodes can be divided into head nodes and non-head nodes. Among them, for each non-head node, it must have a parameter output by other nodes as its input parameter. Therefore, for each non-head node, the data type of the input data of each non-head node must be configured in the data link configuration, while the input of the head node can be determined by the data configuration or by the actual data processing service. In the directed acyclic graph, there are also output nodes and non-output nodes. Only the nodes with the output parameter type configured can be used as output nodes, and the output nodes can output their processing results outward.

[0028] In the embodiments provided by the present application, each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit. The relay unit is used to store and manage the data arriving at the node. When the data arriving at the node triggers the operating condition of the node, the relay unit controls the node to execute data processing. If the data arriving at the node does not meet the operating condition of the node, the relay unit controls the node not to execute data processing.

[0029] Thus, according to the data link optimization processing method provided by the embodiments of the present application, a directed acyclic graph is first generated according to the data link configuration. Here, the method of generating a directed acyclic graph according to the data link configuration has been described in detail in the prior art and will not be elaborated here. After the directed acyclic graph is generated, the data to be processed of the data processing service is input into the head node of the directed acyclic graph, so that each node executes corresponding data processing on the received data in a streaming order. When the data reaching a node triggers the running condition of the node, the relay unit controls the node to execute data processing, so that each node can maintain the streaming order of the directed acyclic graph during the data processing. Even in the case of multi-data concurrency, it can ensure that the order of data processing conforms to the directed acyclic graph. Finally, the final data processing result is obtained through the tail node of the directed acyclic graph.

[0030] Further, at least one input data queue and at least one output data queue are included in the relay unit of each node. The running conditions of the node at least include: there is preset input data in all input data queues corresponding to the node. Here, the preset input data refers to the input data required for the node to perform data processing. For example, if the preset input data of node A is X, Y, and Z, then it is necessary that the input X, Y, and Z all meet the preset conditions. For example, the preset condition can be that the first data in each input data queue has passed the integrity verification before it can be regarded as the preset input data X, Y, and Z. The input data queue of the node is used to store the preset input data transmitted to the node and required to execute the function of the node, and the output data queue is used to store the data that needs to be transmitted to the downstream node. As an example, when a certain intermediate node includes 3 input data queues, it means that the node needs to receive three types of data transmitted from 3 upstream nodes. When there is preset input data transmitted from the upstream nodes in all these 3 input data queues and all meet the preset conditions, it is considered that the data of the node meets the running conditions. At this time, the relay unit can control the node to execute data processing.

[0031] Please refer to Figure 2 , Figure 2 which is a schematic diagram of a directed acyclic graph with a relay unit provided by the embodiments of the present application. As shown in Figure 2 shown, Figure 2Each node in [the system] is configured with a relay unit, which includes an ingress coroutine and an egress coroutine. Here, the ingress coroutine represents the input coroutine, and the egress coroutine represents the output coroutine. The ingress coroutine includes at least one input data queue, and the number of input data queues of a node is determined according to the number of incoming directed edges connected to the node. The egress coroutine includes at least one output data queue, and the number of output data queues of a node is determined according to the number of outgoing directed edges connected to the node. The data in each output data queue is usually the same. Even if the calculation frequencies of each input data of the node are different, the node configured with the ingress coroutine can ensure that the calculations are executed in order. For example, Figure 2 The input data of node G in [the system] comes from the output of node A1, the output of node C, and the output of node D. Therefore, the relay unit of node G includes three data input queues. In actual business, the output frequencies of each node are very likely to be different and vary greatly. Suppose the output frequency of A1 is once per second, the output frequency of node C is once every 5 seconds, and the output frequency of node D is once every 30 seconds. Then, when waiting for the output of node D, the output data of node A1 and the output data of node C need to enter the input data queues corresponding to node A1 and node C in the relay unit of node G for storage. The relay unit is used to ensure that node G will only execute data processing when the output data of node A1, the output data of node C, and the output data of node D have all entered their respective queues. At this time, node G responds to the executable signal sent by the relay unit, calls the corresponding processing function, and executes the data processing steps corresponding to the first data in each queue (each queue is in a first-in-first-out mode, and the first data is the data that entered the queue first), so as to achieve the preservation of the order of data. If the relay unit, data queues, and node operating conditions are not set, it may cause the node to start executing the function before obtaining the data with a lower output frequency, resulting in an error, or it may cause node G to execute calculations based on the first data of node D and the 30th data of node A1 and the 6th data of node C, thus leading to subsequent data errors.

[0032] As an optional embodiment, in response to the existence of data execution condition configuration in the data link configuration, the corresponding directed edge is configured with a data processing expression, and the data link optimization processing method further includes: In response to the transmission data passing through the first directed edge, the transmission data is processed based on the data processing expression.

[0033] Here, the first directed edge is the directed edge with a data processing expression. As an example, data processing may include calculating data, adding a prefix and / or suffix to data, hiding plaintext of data, and performing format conversion on data, etc., which is not specifically limited in this application.

[0034] For the above steps, in specific implementation, when there is a data execution condition configuration in the data link configuration, the corresponding directed edge is configured with a data processing expression. When transmission data passes through the first directed edge with a data processing expression, the transmission data is processed based on this data processing expression. Here, as an example, when the data processing expression configured for the first directed edge is data * 2, it means that the transmission data of the output node connected to the first directed edge is multiplied by 2 and then transmitted to the input node connected to the first directed edge.

[0035] As an alternative embodiment, in response to the existence of a data output condition configuration in the data link configuration, the corresponding node is configured with a data output condition, and the data link optimization processing method further includes: In response to the first node outputting a data processing result, based on the data processing result and the data output condition, a target node is determined from at least one downstream node of the first node, and only the processed data processing result is transmitted to the target node through the output data queue.

[0036] Here, the first node is the node with a data output condition. The data output condition may be a condition for selecting the transmission path of the data processing result. The downstream node refers to the input ends of N nodes directly connected by the output edge of a node, and the N nodes are the downstream nodes of this node. For example, continuing Figure 2 the example above, at node A2, a data output condition can be configured: when the data processing result output by node A2 is odd, the data processing result is transmitted to node C; when the data processing result output by node A2 is even, the data processing result is transmitted to node D. The data output condition may also be a condition for filtering the data processing result. For example, at node A2, a data output condition can be configured: only transmit downward when the data processing result is greater than 1.

[0037] For the above steps, in specific implementation, in response to the existence of a data output condition configuration in the data link configuration, the corresponding node is configured with a data output condition. In response to the first node outputting a data processing result, based on the data processing result and the data output condition configured for the first node, a target node is determined from at least one downstream node of the first node, and only the processed data processing result is transmitted to the target node through the output data queue. Here, continuing the above example, when the data processing result output by node A2 is an odd number, based on this data processing result and the data output condition in the above example, node C is determined as the target node, and only the data processing result is transmitted to node C through the output data queue.

[0038] Furthermore, the data link optimization processing method further includes: Only the processed data processing result is transmitted to the target node through the output data queue, and a second signal is transmitted to other downstream nodes, and the second signal is an empty signal.

[0039] Here, the empty signal is a signal used to indicate canceling the data processing of the node in the current round.

[0040] For the above steps, in specific implementation, the first node only transmits the processed data processing result to the target node through the output data queue, and for other downstream nodes of the first node except the target node among at least one downstream node of the first node, the first node transmits an empty signal to other downstream nodes. Here, continuing the above example, when node A2 determines node C as the target node, node A2 needs to transmit the second signal to node D, and when node D receives the second signal, it stops executing the data processing in the current round, which can ensure the orderliness of node D.

[0041] Furthermore, at least one basic node is included in the nodes of the directed acyclic graph; the data link optimization processing method further includes: In response to the basic node having an abnormality, a first abnormal signal is transmitted to the downstream nodes of the basic node where the abnormality occurs, and the downstream nodes continue to transmit the first abnormal signal in the direction of the directed edge.

[0042] Here, the basic node represents a necessary node for implementing the main data processing function in the directed acyclic graph, such as a necessary node in the directed acyclic graph. The first abnormal signal is an abnormal signal indicating that the basic node has an abnormality. Having an abnormality can include situations such as the node execution failing, the node execution not meeting expectations, and the node execution timing out. Here, as an optional embodiment, the node execution timing out is usually due to network fluctuations and needs to be retried. If the retry is successful, the timeout problem is ignored and the normal operation continues. If the retry still times out, the retry continues until the number of retries exceeds the preset retry threshold and the timeout problem still occurs. At this time, the timeout error is converted into the first abnormal signal of the node.

[0043] Regarding the above steps, in specific implementation, when a basic node in the directed acyclic graph has an abnormality, a first abnormal signal is transmitted to the downstream nodes of the basic node with the abnormality, and the downstream nodes continue to transmit the first abnormal signal in the direction of the directed edge. In this way, when a basic node has an abnormality, starting from the current data, the operation of this basic node and the subsequent operations of this basic node are stopped, without affecting the processing of the previously transmitted data.

[0044] Furthermore, in response to the existence of a data enhancement processing step in the data link configuration, an enhancement node is further included in the directed acyclic graph; the data link optimization processing method further includes: In response to the enhancement node having an abnormality, a second abnormal signal is transmitted to the downstream enhancement nodes of the enhancement node with the abnormality, and the downstream enhancement nodes continue to transmit the second abnormal signal in the direction of the directed edge.

[0045] Here, the enhancement node represents a node in the directed acyclic graph that implements the data enhancement processing step. The enhancement node is a non-essential node in the directed acyclic graph. For example, a node in the directed acyclic graph that depends on the output of other basic nodes or other enhancement nodes and performs data enhancement processing on this basis. Among them, the downstream nodes of the enhancement node must also be enhancement nodes, and the upstream nodes of the enhancement node can be basic nodes or enhancement nodes. The second abnormal signal is the abnormal signal when the enhancement node has an abnormality.

[0046] Regarding the above steps, in specific implementation, when there is a data enhancement processing step in the data link configuration, an enhancement node is included in the directed acyclic graph. When the enhancement node has an abnormality, a second abnormal signal is transmitted to the downstream nodes of the enhancement node with the abnormality, and the downstream nodes continue to further transmit the second abnormal signal in the direction of the directed edge. In this way, when the enhancement node has an abnormality, the second abnormal signal is transmitted to the downstream enhancement nodes according to the order of the directed edges, which can achieve stopping the operation of the enhancement node and the downstream enhancement nodes starting from the current data, without affecting the data that has been processed before, nor affecting the data processing of the essential nodes and the enhancement nodes corresponding to other enhancement functions.

[0047] Among them, the data processing steps corresponding to the necessary nodes are used to implement necessary functions, while the data processing steps corresponding to the enhanced nodes are used to implement non-necessary enhanced functions. Moreover, a data link configuration may include one or more enhanced functions, that is, corresponding to one or more enhanced function branches. Here, according to the above two steps, when facing exception handling, the difference between the basic node and the enhanced node is that in this application, a relay node is set, and the data order-preserving design is achieved by setting the node running conditions. If an exception occurs during a certain data processing or data transfer of the basic node, that is, if an exception occurs when the Nth data is transferred to the enhanced node, the execution of the Nth and subsequent data will be stopped, but it will not affect the data of the 0th to N-1th times that have already run; for the enhanced node, if an exception occurs in a certain enhanced node, the enhanced function branch where the exception occurs and the downstream branches will be skipped in the current and subsequent processing, and it will not affect other necessary nodes and the nodes corresponding to other enhanced functions.

[0048] Further, the data link optimization processing method further includes: When a node receives a third indication signal, the node interrupts its operation and transmits the third indication signal to the downstream node according to the directed edge order of the directed acyclic graph; and, in response to the management unit detecting that any node has received the third indication signal, the management unit inputs the third indication signal to the head node to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph to interrupt the operation of the directed acyclic graph.

[0049] Here, the management unit is used to control and manage the directed acyclic graph, the third indication signal is used to indicate at least one node to interrupt its operation, and the head node represents the node in the directed acyclic graph that receives external input data of the directed acyclic graph.

[0050] For the above two steps, in specific implementation, when a node in the directed acyclic graph receives a third indication signal indicating that the node interrupts its operation, the node interrupts its operation and transmits the third indication signal to the downstream node according to the directed edge order of the root directed acyclic graph. Here, the transmission of the third indication signal is the same as the transmission of input data and output data, and the third indication signal is also transmitted downstream according to the directed edge order in the directed acyclic graph. The management unit monitors all nodes. When the management unit detects that any node has received the third indication signal, the management unit inputs the third indication signal to the head node, and the head node transmits the third indication signal to all nodes in the directed acyclic graph according to the directed edges of the directed acyclic graph to ensure that the operation of all nodes in the directed acyclic graph is interrupted in sequence.

[0051] Here, as an optional embodiment, after the node receives the third indication signal, when the relay unit of the interrupted node detects that the interrupted node meets the operating conditions, that is, when there is preset input data in all the input data queues corresponding to the interrupted node, the relay unit can restart the data processing work of the interrupted node.

[0052] The data link optimization processing method provided by the embodiments of the present application generates a directed acyclic graph according to the data link configuration, represents the data relationship within the data framework based on the directed acyclic graph, and the logic and sequence of its data execution are completely controllable. Moreover, the implementation of the directed acyclic graph supports the asynchronous and parallel execution of data. After the directed acyclic graph is generated, the data to be processed for the data processing service is input into the head node of the directed acyclic graph. The relay unit can ensure that each node and each directed edge stably execute the corresponding data processing on the received data in a streaming order. Whether the data to be processed is in a streaming form or a non-streaming form, only when the data arriving at the node triggers the operating conditions of the node, the relay unit controls the node to execute data processing. In this way, each node can maintain the streaming order of the directed acyclic graph during the data processing process. Even in the case of multi-data concurrency, it can ensure that the order and rules of data processing conform to the directed acyclic graph configuration. Finally, the final data processing result is obtained through the tail node of the directed acyclic graph.

[0053] Based on the same inventive concept, the embodiments of the present application also provide a data link optimization processing device. The data link optimization processing device is implemented based on a directed acyclic graph, and the data link optimization processing device includes: [[ID=P8]]A directed acyclic graph generation module, configured to generate at least one directed acyclic graph according to at least one data link configuration; wherein, each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit. The relay unit is used to store and manage the data arriving at the node. When the data arriving at the node triggers the operating conditions of the node, the relay unit controls the node to execute data processing. If the data arriving at the node does not meet the operating conditions of the node, the relay unit controls the node not to execute data processing.

[0054] Further, at least one input data queue and at least one output data queue are included in the relay unit of each node, and the operating conditions of the node at least include: there is preset input data in all the input data queues corresponding to the node.

[0055] Further, the number of input data queues of the node is determined according to the number of input directed edges connected to the node, and the number of output data queues of the node is determined according to the number of output directed edges connected to the node, wherein the data in each output data queue is the same.

[0056] Further, in response to the presence of a data execution condition configuration in the data link configuration, a corresponding directed edge configures a data processing expression. The data link optimization processing device further includes a first processing module, and the first processing module is configured to: In response to transmission data passing through a first directed edge, process the transmission data based on the data processing expression, where the first directed edge is a directed edge with a data processing expression.

[0057] Further, in response to the presence of a data output condition configuration in the data link configuration, a corresponding node configures a data output condition. The data link optimization processing device further includes a first transmission module, and the first transmission module is configured to: In response to a first node outputting a data processing result, determine a target node from at least one downstream node of the first node based on the data processing result and the data output condition, and transmit only the processed data processing result to the target node through an output data queue. The first node is a node with a data output condition.

[0058] Further, the data link optimization processing device further includes a second transmission module, and the second transmission module is configured to: Transmit only the processed data processing result to the target node through an output data queue, and transmit a second signal to other downstream nodes, where the second signal is an empty signal.

[0059] Further, at least one basic node is included in the nodes of the directed acyclic graph. The data link optimization processing device further includes a third transmission module, and the third transmission module is configured to: In response to an exception occurring in the basic node, transmit a first exception signal to the downstream nodes of the basic node where the exception occurs, and the downstream nodes continue to transmit the first exception signal in the direction of the directed edge. The basic node represents a necessary node for the directed acyclic graph to implement the main data processing function, and the first exception signal is an exception signal indicating that an exception has occurred in the basic node.

[0060] Further, in response to the presence of a data enhancement processing step in the data link configuration, an enhancement node is further included in the directed acyclic graph. The data link optimization processing device further includes a fourth transmission module, and the fourth transmission module is configured to: In response to an exception occurring in the enhancement node, transmit a second exception signal to the downstream enhancement nodes of the enhancement node where the exception occurs, and the downstream enhancement nodes continue to transmit the second exception signal in the direction of the directed edge. The enhancement node represents a node for the directed acyclic graph to implement the data enhancement processing step, and the second exception signal is an exception signal indicating that an exception has occurred in the enhancement node.

[0061] Further, the data link optimization processing device further includes a fifth transmission module, and the fifth transmission module is configured to: When a node receives a third indication signal, the node interrupts its operation and transmits the third indication signal to a downstream node according to the order of the directed edges of the directed acyclic graph; Moreover, in response to the management unit detecting that any node has received the third indication signal, the third indication signal is input into the head node, so as to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph, thereby interrupting the operation of the directed acyclic graph; Wherein, the management unit is configured to control and manage the directed acyclic graph, the third indication signal is used to indicate at least one node to interrupt its operation, and the head node represents the node that receives the initial input in the directed acyclic graph.

[0062] Please refer to Figure 3 , Figure 3 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 3 shown in

[0063] the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0064] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 runs, the processor 310 communicates with the memory 320 through the bus 330. When the machine-readable instructions are executed by the processor 310, the steps of the data link optimization processing method in the method embodiment can be executed. The specific implementation manner can refer to the method embodiment and will not be elaborated herein.

[0065] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0066] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0067] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0068] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0069] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0070] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present application, used to illustrate the technical solutions of the present application, rather than limiting it. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the technical field can still modify the technical solutions described in the foregoing embodiments, or easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data link optimization processing method, the data link optimization processing method is implemented based on a directed acyclic graph, and is characterized in that The data link optimization processing method includes: Generating at least one directed acyclic graph according to at least one data link configuration; wherein, each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, each node includes at least one relay unit, and the relay unit is used to store and manage the data arriving at the node. When the data arriving at the node triggers the operating condition of the node, the relay unit controls the node to execute data processing. If the data arriving at the node does not meet the operating condition of the node, the relay unit controls the node not to execute data processing.

2. The data link optimization processing method according to claim 1, wherein Each relay unit in each node includes at least one input data queue and at least one output data queue, and the operating condition of the node includes at least: there is preset input data in all input data queues corresponding to the node.

3. The data link optimization processing method according to claim 2, characterized in that The number of input data queues of the node is determined according to the number of input directed edges connected to the node, and the number of output data queues of the node is determined according to the number of output directed edges connected to the node, wherein the data in each output data queue is the same.

4. The data link optimization processing method according to claim 2, wherein In response to the existence of a data execution condition configuration in the data link configuration, a data processing expression is configured for the corresponding directed edge. The data link optimization processing method further includes: In response to the transmission data passing through the first directed edge, the transmission data is processed based on the data processing expression, and the first directed edge is a directed edge with a data processing expression.

5. The data link optimization processing method according to claim 2, wherein In response to the existence of a data output condition configuration in the data link configuration, a data output condition is configured for the corresponding node. The data link optimization processing method further includes: In response to the first node outputting a data processing result, based on the data processing result and the data output condition, a target node is determined from at least one downstream node of the first node, and the processed data processing result is transmitted to the target node only through the output data queue. The first node is a node with a data output condition.

6. The data link optimization processing method according to claim 5, wherein The data link optimization processing method further includes: The processed data processing result is transmitted to the target node only through the output data queue, and a second signal is transmitted to other downstream nodes, and the second signal is an empty signal.

7. The data link optimization processing method according to claim 1, wherein At least one basic node is included in the nodes of the directed acyclic graph; the data link optimization processing method further includes: In response to the basic node being abnormal, a first abnormal signal is transmitted to the downstream nodes of the abnormal basic node, and the downstream nodes continue to transmit the first abnormal signal in the direction of the directed edge; the basic node represents a necessary node for the directed acyclic graph to implement the main data processing function, and the first abnormal signal is an abnormal signal indicating that the basic node is abnormal.

8. The data link optimization processing method according to claim 7, wherein In response to the existence of a data enhancement processing step in the data link configuration, an enhancement node is further included in the directed acyclic graph; the data link optimization processing method further includes: In response to an exception occurring in the enhanced node, a second exception signal is transmitted to the downstream enhanced node of the enhanced node where the exception occurs, and the downstream enhanced node continues to transmit the second exception signal in the direction of the directed edge; the enhanced node represents a node that implements the data enhancement processing step of the directed acyclic graph, and the second exception signal is an exception signal indicating that an exception has occurred in the enhanced node.

9. The data link optimization processing method according to claim 1, wherein The data link optimization processing method further includes: When a node receives a third indication signal, the node interrupts its operation and transmits the third indication signal to the downstream node according to the order of the directed edges of the directed acyclic graph; Moreover, in response to the management unit detecting that any node has received the third indication signal, the third indication signal is input into the head node, so as to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph to interrupt the operation of the directed acyclic graph; Wherein, the management unit is used to control and manage the directed acyclic graph, the third indication signal is used to instruct at least one node to interrupt its operation, and the head node represents the node that receives the initial input in the directed acyclic graph.

10. A data link optimization processing device, which is implemented based on a directed acyclic graph, and is characterized in that The data link optimization processing device includes: A directed acyclic graph generation module, configured to generate at least one directed acyclic graph according to at least one data link configuration; wherein, each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, each node includes at least one relay unit, the relay unit is used to store and manage the data arriving at the node, and when the data arriving at the node triggers the operating condition of the node, the relay unit controls the node to execute data processing, and if the data arriving at the node does not meet the operating condition of the node, the relay unit controls the node not to execute data processing.

Images