A data link optimization processing method and a data link optimization processing device

Abstract

The application provides a data link optimization processing method and device. A directed acyclic graph is generated according to data link configuration, and data relationship in a data framework is represented based on the directed acyclic graph. The logic and order of data execution are completely controllable, and the implementation of the directed acyclic graph supports asynchronous and parallel execution of data. After the directed acyclic graph is generated, the to-be-processed data of a data processing service is input into a head node of the directed acyclic graph, so that each node performs corresponding data processing on the received data in a streaming order. Only when the data reaching the node triggers the running condition of the node, the relay unit controls the node to perform data processing. In this way, each node can maintain the streaming order of the directed acyclic graph in the data processing process. Even in the case of multiple data concurrency, the order of data processing can be ensured to comply with the directed acyclic graph configuration, and finally the final data processing result is obtained through a tail node of the directed acyclic graph.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data link optimization processing method and a data link optimization processing device. Background Art

[0002] A directed acyclic graph (DAG) is a data structure widely used in fields such as data processing and task scheduling. Its core feature is that it represents the dependencies between tasks or data through a combination of nodes and edges, without any loops in the graph, thus ensuring that tasks or data are executed or processed in a specific order. In scenarios such as distributed computing, data processing, and parallel computing, DAGs are widely used to describe complex data processing flows and task dependencies.

[0003] In the prior art, DAG is typically used to process large-scale data streams or high-frequency events. For example, in a streaming data processing system, data is divided into multiple events or tasks and processed sequentially through the nodes in the DAG. Each node may rely on the output result of the previous node to ensure the correctness, stability, and consistency of the data. However, when the amount of data is particularly large or the processing frequency is particularly high, especially when the acquisition frequency of each input data of the node varies greatly, the conventionally configured directed acyclic graph may have data disorder problems. If the data order is wrong or the intermediate values ​​corresponding to different input data are confused with each other, the downstream node may receive incorrect or incomplete input data, resulting in incorrect calculation results or data link crashes. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data link optimization processing method and a data link optimization processing device, which can enable each node to maintain the streaming order of the directed acyclic graph during the data processing process, and even in the case of multiple data concurrency, it can ensure that the order and rules of data processing conform to the directed acyclic graph configuration, and finally obtain the final data processing result through the tail node of the directed acyclic graph.

[0005] In a first aspect, an embodiment of the present application provides a data link optimization processing method, which is implemented based on a directed acyclic graph and includes:

[0006] At least one directed acyclic graph is generated based on at least one data link configuration; wherein each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit, and the relay unit is used to store and manage data arriving at the node. When the data arriving at the node triggers the operating conditions of the node, the relay unit controls the node to perform data processing; if the data arriving at the node does not meet the operating conditions of the node, the relay unit controls the node not to perform data processing.

[0007] Furthermore, the relay unit of each node includes at least one input data queue and at least one output data queue, and the operation condition of the node at least includes: preset input data exists in all input data queues corresponding to the node.

[0008] Furthermore, the number of input data queues of the node is determined according to the number of input directed edges connected to the node, and the number of output data queues of the node is determined according to the number of output directed edges connected to the node, wherein the data in each output data queue is the same.

[0009] Furthermore, in response to the presence of a data execution condition configuration in the data link configuration, a data processing expression is configured for the corresponding directed edge, and the data link optimization processing method further includes:

[0010] In response to transmission data passing through a first directed edge, the transmission data is processed based on the data processing expression, where the first directed edge is a directed edge having a data processing expression.

[0011] Furthermore, in response to the presence of a data output condition configuration in the data link configuration, the corresponding node configures the data output condition, and the data link optimization processing method further includes:

[0012] In response to the first node outputting the data processing result, a target node is determined from at least one downstream node of the first node based on the data processing result and the data output condition, and only the processed data processing result is transmitted to the target node through the output data queue, where the first node is a node for which the data output condition exists.

[0013] Furthermore, the data link optimization processing method further includes:

[0014] Only the processed data result is transmitted to the target node through the output data queue, and the second signal is transmitted to other downstream nodes, where the second signal is a null signal.

[0015] Furthermore, the nodes in the directed acyclic graph include at least one basic node; and the data link optimization processing method further includes:

[0016] In response to an abnormality occurring in the basic node, a first abnormality signal is transmitted to the downstream node of the basic node where the abnormality occurs, and the downstream node continues to transmit the first abnormality signal in the direction of the directed edge; the basic node represents a necessary node of the directed acyclic graph to implement the main data processing function, and the first abnormality signal is an abnormality signal of the basic node.

[0017] Furthermore, in response to the presence of a data enhancement processing step in the data link configuration, the directed acyclic graph further includes an enhancement node; and the data link optimization processing method further includes:

[0018] In response to an abnormality occurring in the enhancement node, a second abnormality signal is transmitted to the downstream enhancement node of the abnormal enhancement node, and the downstream enhancement node continues to transmit the second abnormality signal in the direction of the directed edge; the enhancement node represents a node in the directed acyclic graph that implements the data enhancement processing step, and the second abnormality signal is an abnormality signal indicating that the enhancement node has an abnormality.

[0019] Furthermore, the data link optimization processing method further includes:

[0020] In response to the node receiving the third indication signal, the node interrupts operation and transmits the third indication signal to a downstream node according to the directed edge sequence of the directed acyclic graph;

[0021] and, in response to the management unit detecting that any node receives a third indication signal, inputting the third indication signal into the head node, so as to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph, so as to interrupt the operation of the directed acyclic graph;

[0022] The management unit is used to control and manage the directed acyclic graph, the third indication signal is used to instruct at least one node to interrupt operation, and the head node represents a node in the directed acyclic graph that receives initial input.

[0023] In a second aspect, an embodiment of the present application further provides a data link optimization processing device, which is implemented based on a directed acyclic graph and includes:

[0024] A directed acyclic graph generation module is used to generate at least one directed acyclic graph based on at least one data link configuration; wherein each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit, which is used to store and manage data arriving at the node. When the data arriving at the node triggers the node's operating conditions, the relay unit controls the node to perform data processing. If the data arriving at the node does not meet the node's operating conditions, the relay unit controls the node not to perform data processing.

[0025] The data link optimization processing method and device provided by the embodiment of the present application generate a directed acyclic graph according to data link configuration, represent data relationship in a data framework based on the directed acyclic graph, and the logic and order of data execution are completely controllable, and the implementation of the directed acyclic graph supports asynchronous and parallel execution of data. After the directed acyclic graph is generated, the to-be-processed data of the data processing service is input into the head node of the directed acyclic graph, and the relay unit can ensure that each node and each directed edge perform corresponding data processing on the received data in a stable flow order. Whether the form of the to-be-processed data is a flow form or a non-flow form, the relay unit controls the node to perform data processing only when the data arriving at the node triggers the running condition of the node, so that the flow order of the directed acyclic graph can be maintained during data processing of each node. Even in the case of multiple data concurrency, the order of data processing can also be ensured to comply with the directed acyclic graph configuration, and finally the final data processing result is obtained through the tail node of the directed acyclic graph.

[0026] In order to make the above objectives, characteristics and advantages of the present application more apparent, clear and easy to understand, the following preferred embodiments are specifically described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 The schematic diagrams of two common directed acyclic graphs provided by the embodiments of the present application;

[0029] Figure 2 The schematic diagram of a directed acyclic graph with a relay unit provided by the embodiments of the present application;

[0030] Figure 3 The structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0032] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of data processing technology.

[0033] A directed acyclic graph (DAG) is a data structure widely used in fields such as data processing and task scheduling. Its core feature is that it represents the dependencies between tasks or data through a combination of nodes and edges, without any loops in the graph, thus ensuring that tasks or data are executed or processed in a specific order. In scenarios such as distributed computing, data processing, and parallel computing, DAGs are widely used to describe complex data processing flows and task dependencies.

[0034] See also Figure 1 , Figure 1 Schematic diagrams of two common directed acyclic graphs provided in the embodiments of this application. Figure 1 As shown, a directed acyclic graph includes several nodes, each connected by directed edges. A node can be a function or a feature, and the directed edges between nodes represent dependencies between them. For example, this can be the order or relationship of data transfer. For example, if the processing of step B requires data processed by step A, then the directed edge represents A→B. Several nodes and the dependencies between them form a data processing chain in at least one scenario. Depending on the actual business situation, a node can establish dependencies with multiple nodes. For each dependency (that is, each edge), each node must be configured with at least one input relationship and at least one output relationship. In a directed acyclic graph, there are no loops—that is, no path that starts at a node, passes through several nodes, and then returns to the original node. This ensures that there are no cycles in the graph, allowing processing or calculations to be performed in a certain order.

[0035] Directed acyclic graphs (DAGs) can be processed in streaming or non-streaming formats. For example, in a data logic, the dependencies between nodes are A→B→C→D→E→F. In a non-streaming scenario, there is only one piece of data to be processed, and this data only needs to go through the process from node A to node F once. In a streaming scenario, a large amount of data will be repeatedly processed in parallel, following the A→B→C→D→E→F process until all data is processed. In a streaming scenario, each data flow is independent and asynchronous. For example, node A receives input data 1, processes it, and generates output A1, which is then passed to node B for processing. Subsequently, node A receives input data 2 and continues the A→B→C→D→E→F process. The execution of input data 2 is independent of input data 1. In other words, in a streaming scenario, the processing flow for each input data is independent. Non-streaming can be considered streaming data that only needs to be run once. Therefore, a DAG configuration that supports streaming data processing must also support data processing in non-streaming scenarios.

[0036] Research has found that in the existing technology, DAG is usually used to process large-scale data streams or high-frequency events. For example, in a streaming data processing system, data is divided into multiple events or tasks and processed sequentially through the nodes in the DAG. Each node may rely on the output result of the previous node to ensure the correctness, stability and consistency of the data. However, when the amount of data is particularly large or the processing frequency is particularly high, especially when the acquisition frequency of each input data of the node varies greatly, the conventionally configured directed acyclic graph may have data disorder problems. If the data order is wrong or the intermediate values ​​corresponding to different input data are confused with each other, the downstream node may receive incorrect or incomplete input data, resulting in incorrect calculation results or data link crashes.

[0037] Based on this, an embodiment of the present application provides a data link optimization processing method to ensure that the data processing link order of each input data remains stable, and even in the case of multiple data concurrency, the order and rules of data processing can be guaranteed to comply with the directed acyclic graph configuration, and finally the final data processing result is obtained through the tail node of the directed acyclic graph.

[0038] The data link optimization processing method provided in the embodiment of the present application is implemented based on a directed acyclic graph, and the data link optimization processing method includes:

[0039] At least one directed acyclic graph is generated based on at least one data link configuration.

[0040] Here, in a directed acyclic graph, each data execution step is a node, and the data link configuration includes at least the various data execution steps contained in the data processing flow, which converts the configuration into the data transmission relationship between each node, the input parameter type and output parameter type of each node, node information, and node type. Specifically, the input parameter type and output parameter type refer to the data type of the input and output values ​​of each node, such as common data types such as int, string, float, bool, etc. This application does not limit the data type. Node information may include: node name, specific implementation algorithm of the node. According to the embodiment provided by this application, the node type may include: head node, output node, and tail node. Specifically, the head node represents the node that receives the initial input in the directed acyclic graph, and is the node that needs to obtain service input. The output node indicates that at least one output parameter of the output node is output to other downstream nodes. The tail node represents the node that ultimately outputs the processing result of the entire data processing flow in the directed acyclic graph. In a directed acyclic graph, nodes can be divided into head nodes and non-head nodes. Non-head nodes must have a parameter output by another node as their input parameter. Therefore, for each non-head node, the data link configuration must configure the data type of the input data of each non-head node. The input of the head node can be determined by the data configuration or the actual data processing business. A directed acyclic graph also includes output nodes and non-output nodes. Only nodes configured with an output parameter type can serve as output nodes. Output nodes can output their processing results.

[0041] In the embodiments provided herein, each directed acyclic graph includes at least two nodes, each of which is connected by a directed edge, and each node includes at least one relay unit. The relay unit is used to store and manage data arriving at the node. When data arriving at the node triggers the node's operating conditions, the relay unit controls the node to perform data processing. If the data arriving at the node does not meet the node's operating conditions, the relay unit controls the node not to perform data processing.

[0042] Like this, according to the data link optimization processing method provided by the embodiment of the present application, first generate a directed acyclic graph according to the data link configuration.Here, the method for generating a directed acyclic graph according to the data link configuration is described in detail in the prior art and will not be repeated here.After the directed acyclic graph is generated, the pending data of the data processing business is input into the head node of the directed acyclic graph so that each node performs corresponding data processing on the received data in a streaming order.When the data arriving at the node triggers the operating conditions of the node, the relay unit controls the node to perform data processing, so that each node can maintain the streaming order of the directed acyclic graph during the data processing process, even in the case of multiple data concurrency, it can be ensured that the order of data processing conforms to the directed acyclic graph.Finally, the final data processing result is obtained through the tail node of the directed acyclic graph.

[0043] Furthermore, each node's relay unit includes at least one input data queue and at least one output data queue. The node's operating conditions include at least the presence of preset input data in all input data queues corresponding to the node. Here, preset input data refers to the input data required for the node to process data. For example, if node A's preset input data is X, Y, and Z, then inputs X, Y, and Z must meet preset conditions. For example, the preset condition may be that the first data in each input data queue has passed integrity verification before it can be considered as preset input data X, Y, and Z. The node's input data queue is used to store the preset input data required for transmission to the node to perform its functions, and the output data queue is used to store data that needs to be further transmitted to downstream nodes. As an example, if an intermediate node includes three input data queues, it means that the node needs to receive three types of data transmitted from three upstream nodes. When all three input data queues contain the preset input data transmitted by the upstream nodes and all meet the preset conditions, the node's data is considered to meet the operating conditions, and the relay unit can control the node to perform data processing.

[0044] See also Figure 2 , Figure 2 A schematic diagram of a directed acyclic graph with a relay unit provided in an embodiment of the present application. Figure 2 As shown, Figure 2Each node in the is configured with a relay unit, which includes an ingress coroutine and an egress coroutine, where the ingress coroutine represents the input coroutine and the egress coroutine represents the output coroutine. The ingress coroutine includes at least one input data queue, and the number of input data queues of a node is determined by the number of input directed edges connected to the node. The egress coroutine includes at least one output data queue, and the number of output data queues of a node is determined by the number of output directed edges connected to the node. The data in each output data queue is usually the same. Even if the calculation frequency of each input data of the node is different, the node configured with the ingress coroutine can ensure that the calculation is performed in sequence. For example, Figure 2 In this example, the input data for node G originates from the outputs of nodes A1, C, and D. Therefore, node G's relay unit includes three data input queues. In actual operations, the output frequencies of each node are likely to differ significantly. For example, assume that A1's output frequency is once per second, C's is once every 5 seconds, and D's is once every 30 seconds. While waiting for the output of node D, the output data of nodes A1 and C must be stored in the corresponding input data queues of nodes A1 and C in node G's relay unit. The relay unit ensures that node G will only execute data processing after the output data of nodes A1, C, and D have all entered their respective queues. At this point, node G responds to the executable signal sent by the relay unit, calls the corresponding processing function, and executes the data processing steps corresponding to the first data in each queue (each queue operates in a first-in, first-out mode, meaning the first data is the first data to enter the queue), thereby preserving data order. If the relay unit, data queue, and node operating conditions are not set, the node may start executing the function before obtaining the data with a lower output frequency, resulting in an error, or node G may perform calculations based on the first data of node D, the 30th data of node A1, and the 6th data of node C, resulting in subsequent data errors.

[0045] As an optional embodiment, in response to the presence of a data execution condition configuration in the data link configuration, a data processing expression is configured for the corresponding directed edge, and the data link optimization processing method further includes:

[0046] In response to transmission data passing through the first directed edge, the transmission data is processed based on the data processing expression.

[0047] Here, the first directed edge is a directed edge having a data processing expression. As an example, data processing may include calculating data, adding prefixes and / or suffixes to data, hiding plaintext of data, and converting data formats, etc., which is not specifically limited in this application.

[0048] Regarding the above steps, in a specific implementation, when a data execution condition is configured in the data link configuration, the corresponding directed edge is configured with a data processing expression. When transmitted data passes through the first directed edge with a data processing expression, the transmitted data is processed based on the data processing expression. Here, as an example, if the data processing expression configured for the first directed edge is data * 2, this means that the transmitted data of the output node connected to the first directed edge is multiplied by 2 before being transmitted to the input node connected to the first directed edge.

[0049] As an optional embodiment, in response to the presence of a data output condition configuration in the data link configuration, the corresponding node configures the data output condition, and the data link optimization processing method further includes:

[0050] In response to the first node outputting the data processing result, a target node is determined from at least one downstream node of the first node based on the data processing result and the data output condition, and only the processed data processing result is transmitted to the target node through the output data queue.

[0051] Here, the first node is a node with a data output condition. The data output condition can be a condition for selecting a transmission path for the data processing result. The downstream node refers to the input end of N nodes directly connected to the output edge of a node, and the N nodes are the downstream nodes of the node. For example, Figure 2 In the example above, a data output condition can be configured at node A2: when the data processing result output by node A2 is an odd number, the data processing result is transmitted to node C; when the data processing result output by node A2 is an even number, the data processing result is transmitted to node D. Data output conditions can also be used to filter data processing results. For example, a data output condition can be configured at node A2: only transmit data when the data processing result is greater than 1.

[0052] Regarding the above steps, during specific implementation, in response to the presence of a data output condition configuration in the data link configuration, the corresponding node is configured with a data output condition. In response to the first node outputting a data processing result, based on the data processing result and the data output condition configured for the first node, a target node is determined from at least one downstream node of the first node, and only the processed data processing result is transmitted to the target node via the output data queue. Here, continuing with the above example, when the data processing result output by node A2 is an odd number, based on the data processing result and the data output condition in the above example, node C is determined as the target node, and only the data processing result is transmitted to node C via the output data queue.

[0053] Furthermore, the data link optimization processing method further includes:

[0054] Only the processed data result is transmitted to the target node through the output data queue, and the second signal is transmitted to other downstream nodes, where the second signal is a null signal.

[0055] Here, the null signal is a signal used to instruct the node to cancel data processing in the current round.

[0056] Regarding the above steps, during specific implementation, the first node transmits only the processed data results to the target node via the output data queue. For at least one downstream node of the first node other than the target node, the first node transmits a null signal to the other downstream nodes. Continuing with the above example, when node A2 identifies node C as the target node, node A2 needs to transmit a second signal to node D. Upon receiving the second signal, node D ceases executing the current round of data processing, thus ensuring the orderliness of node D.

[0057] Furthermore, the nodes in the directed acyclic graph include at least one basic node; and the data link optimization processing method further includes:

[0058] In response to an abnormality occurring at the basic node, a first abnormality signal is transmitted to a downstream node of the abnormal basic node, and the downstream node continues to transmit the first abnormality signal in a directed edge direction.

[0059] Here, the basic node represents a necessary node for implementing the main data processing function in the directed acyclic graph, such as a necessary node in the directed acyclic graph. The first abnormal signal is an abnormal signal of an abnormality in the basic node. The abnormality may include node execution failure, node execution not meeting expectations, node execution timeout, etc. Here, as an optional embodiment, node execution timeout is usually caused by network fluctuations, and repeated attempts are required. If the retry is successful, the timeout problem is ignored and normal operation continues. If the retry still times out, the retry continues until the number of retries exceeds the preset retry threshold, and a timeout problem still occurs. At this time, the timeout error is converted into the first abnormal signal of the node.

[0060] Regarding the above steps, in a specific implementation, when an exception occurs at a base node in the directed acyclic graph, a first exception signal is transmitted to the downstream node of the abnormal base node, and the downstream node continues to transmit the first exception signal along the directed edge direction. In this way, when an exception occurs at a base node, the base node and its subsequent operations are stopped starting from the current data, without affecting the processing of previously transmitted data.

[0061] Furthermore, in response to the presence of a data enhancement processing step in the data link configuration, the directed acyclic graph further includes an enhancement node; and the data link optimization processing method further includes:

[0062] In response to an abnormality occurring at the enhanced node, a second abnormality signal is transmitted to a downstream enhanced node of the enhanced node where the abnormality occurs, and the downstream enhanced node continues to transmit the second abnormality signal in the direction of the directed edge.

[0063] Here, the enhancement node represents a node in a directed acyclic graph that implements the data enhancement processing step. The enhancement node is a non-essential node in the directed acyclic graph, for example, a node that relies on the output of other basic nodes or other enhancement nodes in the directed acyclic graph and performs data enhancement processing on this basis. Among them, the downstream node of the enhancement node must also be an enhancement node, and the upstream node of the enhancement node can be a basic node or an enhancement node. The second abnormal signal is an abnormal signal that the enhancement node is abnormal.

[0064] Regarding the above steps, in a specific implementation, when there is a data enhancement processing step in the data link configuration, the directed acyclic graph includes an enhancement node. When an abnormality occurs in the enhancement node, a second abnormality signal is transmitted to the downstream node of the abnormal enhancement node, and the downstream node continues to further transmit the second abnormality signal in the direction of the directed edge. In this way, when an abnormality occurs in the enhancement node, the second abnormality signal is transmitted to the downstream enhancement node according to the order of the directed edges, which can stop the operation of the enhancement node and the downstream enhancement node starting from the current data, without affecting the data that has been run before, and without affecting the data processing of the necessary nodes and the enhancement nodes corresponding to other enhanced functions.

[0065] Among them, the data processing steps corresponding to the necessary nodes are used to implement necessary functions, while the data processing steps corresponding to the enhanced nodes are used to implement non-essential enhanced functions, and a data link configuration may include one or more enhanced functions, that is, corresponding to one or more enhanced function branches. Here, according to the above two steps, when facing exception processing, the difference between the basic node and the enhanced node is that the present application sets up a relay node, and realizes the data order preservation design by setting the node operation conditions. If the basic node has an exception during a certain data processing or data transmission, that is, if an exception occurs when the Nth data is transmitted to the enhanced node, the execution of the Nth and subsequent data is stopped, but it will not affect the data that has been run for the 0th to N-1th times; and for the enhanced node, if an exception occurs in a certain enhanced node, the enhanced function branch and downstream branches where the exception occurs are skipped in the current and subsequent processing, and other necessary nodes and nodes corresponding to other enhanced functions will not be affected.

[0066] Furthermore, the data link optimization processing method further includes:

[0067] In response to a node receiving a third indication signal, the node interrupts operation and transmits the third indication signal to a downstream node according to the directed edge sequence of the directed acyclic graph; and, in response to the management unit detecting that any node receives the third indication signal, the third indication signal is input into the head node to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph to interrupt the operation of the directed acyclic graph.

[0068] Here, the management unit is used to control and manage the directed acyclic graph, the third indication signal is used to instruct at least one node to interrupt operation, and the head node represents a node in the directed acyclic graph that receives external input data of the directed acyclic graph.

[0069] With respect to the above two steps, in specific implementation, when a node in a directed acyclic graph receives a third indication signal indicating that the node is to interrupt operation, the node interrupts operation and transmits the third indication signal to the downstream node according to the directed edge sequence of the root directed acyclic graph. Here, the transmission of the third indication signal is the same as the transmission of input data and output data, and the third indication signal is also transmitted downstream in sequence through the directed edges in the directed acyclic graph. The management unit monitors all nodes. When the management unit detects that any node receives the third indication signal, the management unit inputs the third indication signal into the head node. The head node transmits the third indication signal to all nodes in the directed acyclic graph according to the directed edges of the directed acyclic graph, ensuring that the operation of all nodes in the directed acyclic graph is interrupted in sequence.

[0070] Here, as an optional embodiment, after the node receives the third indication signal, when the relay unit of the interrupted node detects that the interrupted node meets the operating conditions, that is, when there is preset input data in all input data queues corresponding to the interrupted node, the relay unit can restart the data processing work of the interrupted node.

[0071] The data link optimization processing method provided by the embodiment of the present application generates a directed acyclic graph according to the data link configuration, and represents the data relationship within the data framework based on the directed acyclic graph. The logic and order of its data execution are fully controllable, and the implementation of the directed acyclic graph supports asynchronous and parallel execution of data. After the directed acyclic graph is generated, the to-be-processed data of the data processing business is input into the head node of the directed acyclic graph. The relay unit can ensure that each node and each directed edge stably performs corresponding data processing on the received data in a streaming order. Regardless of whether the form of the to-be-processed data is streaming or non-streaming, the relay unit controls the node to perform data processing only when the data arriving at the node triggers the operating conditions of the node. In this way, each node can maintain the streaming order of the directed acyclic graph during the data processing process. Even in the case of multiple data concurrency, the order and rules of data processing can be guaranteed to comply with the directed acyclic graph configuration, and finally the final data processing result is obtained through the tail node of the directed acyclic graph.

[0072] Based on the same inventive concept, an embodiment of the present application further provides a data link optimization processing device, which is implemented based on a directed acyclic graph and includes:

[0073] A directed acyclic graph generation module is used to generate at least one directed acyclic graph based on at least one data link configuration; wherein each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit, which is used to store and manage data arriving at the node. When the data arriving at the node triggers the node's operating conditions, the relay unit controls the node to perform data processing. If the data arriving at the node does not meet the node's operating conditions, the relay unit controls the node not to perform data processing.

[0074] Furthermore, the relay unit of each node includes at least one input data queue and at least one output data queue, and the operation condition of the node at least includes: preset input data exists in all input data queues corresponding to the node.

[0075] Furthermore, the number of input data queues of the node is determined according to the number of input directed edges connected to the node, and the number of output data queues of the node is determined according to the number of output directed edges connected to the node, wherein the data in each output data queue is the same.

[0076] Furthermore, in response to the presence of a data execution condition configuration in the data link configuration, a data processing expression is configured for the corresponding directed edge, and the data link optimization processing device further includes a first processing module, the first processing module being configured to:

[0077] In response to transmission data passing through a first directed edge, the transmission data is processed based on the data processing expression, where the first directed edge is a directed edge having a data processing expression.

[0078] Furthermore, in response to the presence of a data output condition configuration in the data link configuration, the corresponding node configures the data output condition, and the data link optimization processing device further includes a first transmission module, the first transmission module being configured to:

[0079] In response to the first node outputting the data processing result, a target node is determined from at least one downstream node of the first node based on the data processing result and the data output condition, and only the processed data processing result is transmitted to the target node through the output data queue, where the first node is a node for which the data output condition exists.

[0080] Furthermore, the data link optimization processing device further includes a second transmission module, and the second transmission module is configured to:

[0081] Only the processed data result is transmitted to the target node through the output data queue, and the second signal is transmitted to other downstream nodes, where the second signal is a null signal.

[0082] Furthermore, the nodes in the directed acyclic graph include at least one basic node; the data link optimization processing device further includes a third transmission module, and the third transmission module is used to:

[0083] In response to an abnormality occurring in the basic node, a first abnormality signal is transmitted to the downstream node of the basic node where the abnormality occurs, and the downstream node continues to transmit the first abnormality signal in the direction of the directed edge; the basic node represents a necessary node of the directed acyclic graph to implement the main data processing function, and the first abnormality signal is an abnormality signal of the basic node.

[0084] Furthermore, in response to the presence of a data enhancement processing step in the data link configuration, the directed acyclic graph further includes an enhancement node; and the data link optimization processing device further includes a fourth transmission module, the fourth transmission module being configured to:

[0085] In response to an abnormality occurring in the enhancement node, a second abnormality signal is transmitted to the downstream enhancement node of the abnormal enhancement node, and the downstream enhancement node continues to transmit the second abnormality signal in the direction of the directed edge; the enhancement node represents a node in the directed acyclic graph that implements the data enhancement processing step, and the second abnormality signal is an abnormality signal indicating that the enhancement node has an abnormality.

[0086] Furthermore, the data link optimization processing device further includes a fifth transmission module, and the fifth transmission module is configured to:

[0087] In response to the node receiving the third indication signal, the node interrupts operation and transmits the third indication signal to a downstream node according to the directed edge sequence of the directed acyclic graph;

[0088] and, in response to the management unit detecting that any node receives a third indication signal, inputting the third indication signal into the head node, so as to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph, so as to interrupt the operation of the directed acyclic graph;

[0089] The management unit is used to control and manage the directed acyclic graph, the third indication signal is used to instruct at least one node to interrupt operation, and the head node represents a node in the directed acyclic graph that receives initial input.

[0090] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown in FIG, the electronic device 300 includes a processor 310 , a memory 320 and a bus 330 .

[0091] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 through the bus 330. When the machine-readable instructions are executed by the processor 310, the steps of the data link optimization processing method in the method embodiment can be executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0092] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the data link optimization processing method in the method embodiment can be executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0093] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0094] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. The described device embodiments are merely schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0095] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0096] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0097] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solutions of the present application essentially or in part with respect to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0098] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data link optimization processing method, which is implemented based on a directed acyclic graph and is characterized in that: The data link optimization processing method includes: At least one directed acyclic graph is generated based on at least one data link configuration; wherein each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit, the relay unit is used to store and manage data arriving at the node, when the data arriving at the node triggers the node's operating condition, the relay unit controls the node to perform data processing, if the data arriving at the node does not meet the node's operating condition, the relay unit controls the node not to perform data processing; The relay unit of each node includes at least one input data queue and at least one output data queue. The operating conditions of the node include at least: there is preset input data in all input data queues corresponding to the node; The number of input data queues of the node is determined according to the number of input directed edges connected to the node, and the number of output data queues of the node is determined according to the number of output directed edges connected to the node, wherein the data in each output data queue is the same; In response to the presence of a data output condition configuration in the data link configuration, the corresponding node configures the data output condition, and the data link optimization processing method further includes: In response to the first node outputting a data processing result, determining a target node from at least one downstream node of the first node based on the data processing result and the data output condition, and transmitting only the processed data processing result to the target node through the output data queue, where the first node is a node for which the data output condition exists; The data link optimization processing method further includes: Only the processed data result is transmitted to the target node through the output data queue, and the second signal is transmitted to other downstream nodes, where the second signal is a null signal.

2. The data link optimization processing method according to claim 1, characterized in that: In response to the presence of a data execution condition configuration in the data link configuration, a data processing expression is configured for the corresponding directed edge, and the data link optimization processing method further includes: In response to transmission data passing through a first directed edge, the transmission data is processed based on the data processing expression, where the first directed edge is a directed edge having a data processing expression.

3. The data link optimization processing method according to claim 1, characterized in that: The nodes in the directed acyclic graph include at least one basic node; and the data link optimization processing method further includes: In response to an abnormality occurring in the basic node, a first abnormality signal is transmitted to a downstream node of the basic node where the abnormality occurs, and the downstream node continues to transmit the first abnormality signal in the direction of the directed edge; the basic node represents a necessary node of the directed acyclic graph to implement the data processing function, and the first abnormality signal is an abnormality signal indicating that an abnormality has occurred in the basic node.

4. The data link optimization processing method according to claim 3, characterized in that: In response to the presence of a data enhancement processing step in the data link configuration, the directed acyclic graph further includes an enhancement node; and the data link optimization processing method further includes: In response to an abnormality occurring in the enhancement node, a second abnormality signal is transmitted to the downstream enhancement node of the abnormal enhancement node, and the downstream enhancement node continues to transmit the second abnormality signal in the direction of the directed edge; the enhancement node represents a node in the directed acyclic graph that implements the data enhancement processing step, and the second abnormality signal is an abnormality signal indicating that the enhancement node has an abnormality.

5. The data link optimization processing method according to claim 1, characterized in that: The data link optimization processing method further includes: In response to the node receiving the third indication signal, the node interrupts operation and transmits the third indication signal to a downstream node according to the directed edge sequence of the directed acyclic graph; and, in response to the management unit detecting that any node receives a third indication signal, inputting the third indication signal into the head node, so as to transmit the third indication signal to all nodes according to the directed edges of the directed acyclic graph, so as to interrupt the operation of the directed acyclic graph; The management unit is used to control and manage the directed acyclic graph, the third indication signal is used to instruct at least one node to interrupt operation, and the head node represents a node in the directed acyclic graph that receives initial input.

6. A data link optimization processing device, which is implemented based on a directed acyclic graph and is characterized in that: The data link optimization processing device includes: A directed acyclic graph generation module, configured to generate at least one directed acyclic graph based on at least one data link configuration; wherein each directed acyclic graph includes at least two nodes, each node is connected by a directed edge, and each node includes at least one relay unit, the relay unit being configured to store and manage data arriving at the node, and when data arriving at the node triggers an operating condition of the node, the relay unit controls the node to perform data processing, and if the data arriving at the node does not meet the operating condition of the node, the relay unit controls the node not to perform data processing; The relay unit of each node includes at least one input data queue and at least one output data queue. The operating conditions of the node include at least: there is preset input data in all input data queues corresponding to the node; The number of input data queues of the node is determined according to the number of input directed edges connected to the node, and the number of output data queues of the node is determined according to the number of output directed edges connected to the node, wherein the data in each output data queue is the same; In response to the presence of a data output condition configuration in the data link configuration, the corresponding node configures the data output condition, and the data link optimization processing device further includes a first transmission module, the first transmission module being configured to: In response to the first node outputting a data processing result, determining a target node from at least one downstream node of the first node based on the data processing result and the data output condition, and transmitting only the processed data processing result to the target node through the output data queue, where the first node is a node for which the data output condition exists; The data link optimization processing device further includes a second transmission module, wherein the second transmission module is configured to: Only the processed data result is transmitted to the target node through the output data queue, and the second signal is transmitted to other downstream nodes, where the second signal is a null signal.

Images