Data processing device, method and system and readable storage medium

Abstract

The invention provides a data processing device, method and system and a readable storage medium, and the device comprises a configuration unit which comprises a logic unit which carries out the arrangement of an algorithm execution process based on an algorithm library, carries out the setting of algorithm parameters, and builds a data execution process; the DAG builder is used for carrying out data processing on the data execution process and determining a DAG; the DAG analyzer is used for analyzing the DAG and determining an execution sequence and a dependency relationship of algorithm nodes; the scheduler is used for grouping the algorithm nodes without the dependency relationship according to the execution sequence; and the processing unit is used for performing parallel processing on the algorithm nodes without the dependency relationship. According to the technical scheme, by analyzing the DAG and determining the execution sequence and the dependency relationship of the algorithm nodes, the algorithm scheduling graph can be quickly configured, synchronous data sources and data processing modes and algorithms can be flexibly combined, parallel processing can be carried out on the algorithm nodes without the dependency relationship, and the parallelism degree and the execution efficiency of algorithm operation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing device, method, system and readable storage medium. Background Art

[0002] The traditional algorithm execution process is usually a solidified execution process, single-process scheduling, serial execution, and low efficiency. Summary of the invention

[0003] In order to solve or improve at least one of the above technical problems, an object of the present invention is to provide a data processing device.

[0004] Another object of the present invention is to provide a data processing method.

[0005] Another object of the present invention is to provide a data processing system.

[0006] Another object of the present invention is to provide a readable storage medium.

[0007] To achieve the above-mentioned purpose, the first aspect of the present invention provides a data processing device for automatically building and scheduling execution of optimization algorithms. The data processing device includes a configuration unit and a processing unit. The configuration unit includes a logic unit, a DAG builder, a DAG parser and a scheduler. The logic unit is used to orchestrate the algorithm execution process based on the algorithm library, set the algorithm parameters, and build the data execution process. The DAG builder is used to process the data execution process and determine the DAG graph. The DAG parser is used to parse the DAG graph and determine the execution order and dependencies of the algorithm nodes. The scheduler is used to group the algorithm nodes that do not have dependencies according to the execution order. The processing unit is used to determine the algorithm nodes that are grouped and run in the business middle station and / or the data middle station and / or the intelligent middle station according to the algorithm definition of the algorithm library, and process the algorithm nodes that do not have dependencies in parallel.

[0008] It should be noted that DAG (Directed Acyclic Graph) is a directed acyclic graph. In the present invention, DAG and DAG graph both represent directed acyclic graphs.

[0009] According to the technical solution of the data processing device provided by the present invention, the execution order and dependencies of the algorithm nodes are determined by parsing the DAG graph, which is conducive to quickly configuring the algorithm scheduling graph, flexibly combining the synchronization data source and the data processing method and algorithm, and can also process in parallel the algorithm nodes without direct dependencies, which is conducive to significantly improving the parallelism and execution efficiency of the algorithm operation.

[0010] In addition, the above technical solution provided by the present invention may also have the following additional technical features: In some technical solutions, optionally, the DAG builder is used to determine the first data source according to the data execution process and obtain the second data source; perform data conversion on the first data source at least twice and normalize the second data source; perform data processing on the first data source after at least two data conversions and the second data source after normalization through a cubic exponential smoothing algorithm; and determine the DAG graph.

[0011] In this technical solution, the data execution process is processed to determine the DAG graph. The specific steps include: The first data source is determined according to the data execution process, and the second data source is obtained. A directed acyclic graph is determined based on the first data source and the second data source.

[0012] The first data source is subjected to at least two data conversions, and the second data source is subjected to normalization processing. Optionally, the number of data conversions is two. In this step, the first data source is subjected to data conversion, and after the data conversion again, the data is processed by a cubic exponential smoothing algorithm. It should be noted that normalization processing is an important method in data preprocessing, which aims to convert feature data of different dimensions to the same order of magnitude for comprehensive comparative evaluation.

[0013] The first data source after at least two data conversions and the second data source after normalization are processed by the triple exponential smoothing algorithm. It should be noted that the triple exponential smoothing algorithm is a time series prediction technology that uses the weighted average of the observations of each period in chronological order as the prediction value, showing that the influence of the future value of historical data decreases with time. This method is more practical and suitable for analyzing all application problems of time series.

[0014] Determine the DAG graph. After processing the data using the triple exponential smoothing algorithm, enter the model evaluation and determine the directed acyclic graph.

[0015] In some technical solutions, optionally, a DAG parser is used to process data outliers to correct the dependencies of algorithm nodes.

[0016] In this technical solution, after determining the DAG graph, the data in the DAG graph needs to be verified. If there are data outliers after verification, the algorithm nodes need to be adjusted and the dependencies of the algorithm nodes need to be corrected. By continuously correcting the dependencies of the algorithm nodes, it is beneficial to group the algorithm nodes that do not have dependencies in the subsequent steps, and the algorithm nodes that do not have direct dependencies can be processed in parallel, thereby improving the parallelism and execution efficiency of the algorithm operation.

[0017] In some technical solutions, optionally, a DAG parser is used to obtain input fields and output fields of algorithm nodes; and process data outliers according to the input fields and output fields to correct the dependency relationship of the algorithm nodes.

[0018] In this technical solution, each algorithm node declares the required input fields and output fields, which are fields in the database table. The input fields and output fields of each algorithm node are verified, and it is determined whether there are data outliers. If there are data outliers after verification, the algorithm node needs to be adjusted and the dependencies of the algorithm nodes need to be corrected. By continuously correcting the dependencies of the algorithm nodes, it is beneficial to group the algorithm nodes that do not have dependencies in the subsequent steps, and the algorithm nodes that do not have direct dependencies can be processed in parallel, thereby improving the parallelism and execution efficiency of the algorithm operation.

[0019] In some technical solutions, optionally, a DAG parser is used to parse a DAG graph, convert graph data of the DAG graph into relational data and perform data analysis and processing; and determine the execution order of algorithm nodes and the dependency relationship.

[0020] In this technical solution, graph technology is used to sort out the upstream and downstream dependencies (by parsing the DAG graph, the execution order and dependencies of the algorithm nodes are determined), which is conducive to quickly configuring the algorithm scheduling graph, flexibly combining synchronization data sources and data processing methods and algorithms, and parallel processing of algorithm nodes without direct dependencies, which is conducive to significantly improving the parallelism and execution efficiency of the algorithm operation.

[0021] Optionally, during the graph data loading process, the previous data is converted into relational data for processing, and the point data and edge data in the graph are recorded separately.

[0022] In some technical solutions, optionally, the processing unit is used to mark the running status of each algorithm node, marking the algorithm node that has been running as completed, marking the algorithm node that is running as executing, and marking the algorithm node that is not running as waiting; for multiple algorithm nodes with a dependency relationship, they are run sequentially in the execution order; for multiple algorithm nodes that do not have a dependency relationship, they are processed in parallel.

[0023] In this technical solution, after clicking the data run, the processing unit will pull the data and calculate according to the algorithm. After the calculation is completed, the current algorithm node will be marked as completed, and then the next algorithm node will be executed until all algorithm nodes are marked as completed.

[0024] During the data operation process, multiple algorithm nodes without dependencies are processed in parallel; multiple algorithm nodes with dependencies are run in sequence according to the execution order. This data processing method can ensure the accuracy of the data processing results and the efficiency of the data processing process.

[0025] It should be noted that during the data operation, if the machine crashes or the task is interrupted, the task status of the algorithm node needs to be verified after the machine is restored. For the algorithm nodes marked as executing or waiting, they continue to run according to the execution order and dependency relationship.

[0026] Optionally, the processing unit has a task merging function. Each algorithm node has an algorithm type. If the algorithm type of the current algorithm node and the algorithm type of the next algorithm node are both conversion types, the above two algorithm nodes can be merged. In other words, the processing unit can merge algorithm nodes with the same algorithm type. This data processing method can reduce useless serialization processes and improve execution efficiency.

[0027] In some technical solutions, optionally, the data processing device further includes a storage unit. The storage unit is used to collect the results after the algorithm nodes are executed and store them in a temporary table.

[0028] In this technical solution, the storage unit can store data after unifying the data type through a translator according to a semantic model.

[0029] The second aspect of the present invention provides a data processing method for automatically constructing and scheduling execution of an optimization algorithm. The steps of the data processing method include: The algorithm execution process is orchestrated based on the algorithm library, and the algorithm parameters are set to build the data execution process.

[0030] Perform data processing on the data execution flow and determine the DAG graph.

[0031] Parse the DAG graph and determine the execution order and dependencies of the algorithm nodes.

[0032] Algorithm nodes that do not have dependencies are grouped according to the execution order.

[0033] According to the algorithm definition of the algorithm library, determine the grouped algorithm nodes to be run in the business middle station and / or data middle station and / or intelligent middle station, and process the algorithm nodes that have no dependencies in parallel.

[0034] It should be noted that DAG (Directed Acyclic Graph) is a directed acyclic graph. In the present invention, DAG and DAG graph both represent directed acyclic graphs.

[0035] According to the technical solution of the data processing method of the present invention, the execution order and dependencies of the algorithm nodes are determined by parsing the DAG graph, which is conducive to quickly configuring the algorithm scheduling graph, flexibly combining the synchronization data source and the data processing method and algorithm, and can also process the algorithm nodes without direct dependencies in parallel, which is conducive to significantly improving the parallelism and execution efficiency of the algorithm operation.

[0036] The third aspect of the present invention provides a data processing system, including a memory and a processor, wherein the memory stores programs or instructions that can be run on the processor, and when the processor executes the program or instructions, the steps of the data processing method in the above technical solution are implemented, so it has the beneficial effects of the above technical solution, which will not be repeated here.

[0037] The fourth aspect of the present invention provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the data processing method in the above technical solution are implemented, so it has the beneficial effects of the above technical solution and will not be repeated here.

[0038] Additional aspects and advantages of the technical solutions of the present invention will become apparent in the following description or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A structural block diagram of a data processing device according to an embodiment of the present invention is shown; Figure 2 A structural block diagram of a data processing device according to another embodiment of the present invention is shown; Figure 3 A schematic diagram of constructing a DAG graph according to an embodiment of the present invention is shown; Figure 4 A schematic diagram showing marking of algorithm nodes according to an embodiment of the present invention is shown; Figure 5 A schematic diagram showing merging algorithm nodes according to an embodiment of the present invention is shown; Figure 6 A flow chart showing a data processing method according to an embodiment of the present invention is shown; Figure 7 A flow chart showing a data processing method according to another embodiment of the present invention is shown; Figure 8 A flow chart showing a data processing method according to another embodiment of the present invention is shown; Fig. 9 A flow chart showing a data processing method according to another embodiment of the present invention is shown; Fig.10A flow chart showing a data processing method according to another embodiment of the present invention is shown; Fig.11 A flow chart showing a data processing method according to another embodiment of the present invention is shown; Fig.12 A structural block diagram of a data processing system according to an embodiment of the present invention is shown.

[0040] in, Figure 1 , Figure 2 and Fig.12 The corresponding relationship between the reference numerals and the component names is as follows: 100: data processing device; 110: configuration unit; 111: logic unit; 112: DAG builder; 113: DAG parser; 114: scheduler; 120: processing unit; 130: storage unit; 300: data processing system; 310: memory; 320: processor. DETAILED DESCRIPTION

[0041] In order to more clearly understand the above-mentioned purposes, features and advantages of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the embodiments of the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0043] Refer to the following Figures 1 to 12 The data processing device 100, the data processing method, the data processing system 300 and the readable storage medium provided according to some embodiments of the present invention are described.

[0044] In one embodiment of the present invention, the data processing device 100 is used for automatically constructing and scheduling execution of optimization algorithms.

[0045] like Figure 1 and Figure 2 As shown, the data processing device 100 includes a configuration unit 110 and a processing unit 120 .

[0046] The configuration unit 110 includes a logic unit 111 , a DAG builder 112 , a DAG parser 113 , and a scheduler 114 .

[0047] It should be noted that DAG (Directed Acyclic Graph) is a directed acyclic graph. DAG and DAG graph in the present invention both represent directed acyclic graphs. A directed acyclic graph is a special directed graph that does not contain any loops.

[0048] The logic unit 111 is used to arrange the algorithm execution process based on the algorithm library, set the algorithm parameters, and build the data execution process. The algorithms in the algorithm library include one of the following or a combination thereof: evaluation algorithm, time series algorithm, machine learning algorithm, classification algorithm, data cleaning algorithm, custom algorithm, and outlier detection algorithm.

[0049] The corresponding input parameters and output parameters are configured in the algorithm library, that is, the input parameters and output parameters of the algorithm are determined. It should be noted that the input parameters and output parameters of the algorithm are the data received and returned by the algorithm function or process. The logic unit 111 uses at least one algorithm in the algorithm library to construct a data execution process, arrange the algorithm execution process, and set the algorithm parameters.

[0050] The logic unit 111 is also used to send or submit the data execution process to the DAG builder 112. The DAG builder 112 is used to perform data processing on the data execution process and determine the DAG graph. The DAG graph represents a directed acyclic graph. A directed acyclic graph is a special directed graph that does not contain any loops. In a DAG, each edge has a direction, and there cannot be a path that starts from a vertex and returns to the vertex through several edges. DAG is often used to describe some tasks or events with dependencies, where a vertex represents an algorithm node and an edge represents a dependency. For example, in a task scheduling system, each algorithm node can be represented as a vertex in the DAG, and the dependency between algorithm nodes can be represented as an edge. By topologically sorting the DAG, the execution order between multiple algorithm nodes can be determined.

[0051] The DAG parser 113 is used to parse the DAG graph and determine the execution order and dependency of the algorithm nodes. The dependency is usually defined by describing the relationship between tasks. When the output of one task is the input of another task, it means that there is a dependency between the two tasks.

[0052] For example, task A must be executed before task B, which means that the execution order of task A is before the execution order of task B. The output of task C is the input of task D, which means that there is a dependency relationship between task C and task D. Usually, algorithm nodes without dependencies are grouped to ensure that algorithm nodes without dependencies can be processed in parallel, thereby achieving efficient task scheduling and significantly improving the parallelism and execution efficiency of algorithm operation.

[0053] The scheduler 114 is used to group algorithm nodes that do not have dependencies according to the execution order. Algorithm nodes that are executed first are processed first, and algorithm nodes that do not have dependencies are processed in parallel. This data processing method can ensure efficient task scheduling while significantly improving the parallelism and execution efficiency of algorithm operation.

[0054] The processing unit 120 is used to determine the grouped algorithm nodes to be run in the business middle station and / or data middle station and / or intelligent middle station according to the algorithm definition of the algorithm library, and to process the algorithm nodes that have no dependencies in parallel.

[0055] It should be noted that the Business Middle Office is an important platform built by enterprises in the process of digital transformation. It carries the core business capabilities of the enterprise and provides shared services and support for the various business lines of the enterprise. The Business Middle Office provides strong support for the development of the enterprise through functions such as capability sharing and reuse, rapid response to business needs, data integration and analysis, and optimization of resource allocation. The Data Middle Platform is the sedimentation of the business and data of existing or newly built information systems. It is an intermediate and supporting platform for realizing data empowerment of new businesses and new applications. The Intelligent Middle Office is a platform based on the Data Middle Office that provides a series of intelligent services such as model or algorithm libraries, model design training, reuse annotation management, and monitoring services.

[0056] In other words, the processing unit 120 of the present invention includes one of the following or a combination thereof: a business middle station, a data middle station, and an intelligent middle station. The business middle station, the data middle station, and the intelligent middle station are execution units of the algorithm. The data middle station is partial to data processing, data cleaning, and simple data statistics; the intelligent middle station supports machine learning related algorithms; the business middle station adopts a simple algorithm formula with high execution efficiency. Optionally, the processing unit also includes other multidimensional engines in addition to the business middle station, the data middle station, and the intelligent middle station.

[0057] The present invention aims to provide a data processing device 100, which uses graph technology to sort out upstream and downstream dependencies (by parsing the DAG graph, determining the execution order and dependencies of algorithm nodes), which is conducive to quickly configuring the algorithm scheduling graph, flexibly combining synchronization data sources and data processing methods and algorithms, and can also process algorithm nodes without direct dependencies in parallel, which is conducive to significantly improving the parallelism and execution efficiency of the algorithm operation.

[0058] The data processing device 100 of the present invention can use the algorithm to automatically build the orchestration and scheduling technology in the supply chain demand forecasting and operation optimization scenarios, and realize the automatic construction and scheduling execution of the algorithm by flexibly configuring the input data set, algorithm input and output parameters, multi-algorithm routing, algorithm scheduling, and output indicator mapping of the algorithm scheme. The data processing device 100 of the present invention is mainly used in business fields such as planning and budgeting, data analysis, monitoring and early warning, and can be provided to various business fields as an independent application component for automatic construction and scheduling execution of optimization algorithms.

[0059] It should be emphasized that the data processing device 100 of the present invention can automatically construct and schedule algorithm processes in prediction and operation scenarios. By arranging the input and output of the algorithm solution and executing the process, it is conducive to flexible scheduling, and different business data can be quickly processed and predicted. The data processing device 100 of the present invention can support algorithm execution in a variety of environments, such as Java (a programming language) code, business middle platform, data middle platform, and intelligent middle platform. The data processing device 100 of the present invention can perform efficient data analysis, data detection, and abnormal data processing, and can be used in various business fields for automatic construction and scheduling of optimization algorithms.

[0060] In some embodiments, optionally, the DAG builder 112 is configured to determine the first data source according to the data execution process, and obtain the second data source.

[0061] A directed acyclic graph is determined based on the first data source and the second data source.

[0062] The DAG builder 112 is further configured to perform at least two data conversions on the first data source and perform normalization processing on the second data source.

[0063] Optionally, the number of data conversions is two. In this step, the first data source is converted, and after the data conversion, the data is processed by a cubic exponential smoothing algorithm. It should be noted that normalization is an important method in data preprocessing, which aims to convert feature data of different dimensions to the same order of magnitude for comprehensive comparative evaluation.

[0064] The DAG builder 112 is further used to process the first data source after at least two data conversions and the second data source after normalization through a cubic exponential smoothing algorithm.

[0065] It should be noted that the triple exponential smoothing algorithm is a time series forecasting technology that uses the weighted average of the observations of each period in chronological order as the forecast value, showing that the impact of the future value of historical data decreases with time. This method is more practical and suitable for analyzing all application problems of time series.

[0066] The DAG builder 112 is also used to determine the DAG graph. After the data is processed by the triple exponential smoothing algorithm, the model evaluation is entered and the directed acyclic graph (such as Figure 3 as shown).

[0067] In some embodiments, optionally, the DAG parser 113 is used to process data outliers to correct the dependencies of algorithm nodes.

[0068] After determining the DAG graph, the data in the DAG graph needs to be verified. If there are data outliers after verification, the algorithm nodes need to be adjusted and the dependencies of the algorithm nodes need to be corrected. By continuously correcting the dependencies of the algorithm nodes, it is beneficial to group the algorithm nodes that do not have dependencies in the subsequent steps. Algorithm nodes without direct dependencies can be processed in parallel, thereby improving the parallelism and execution efficiency of the algorithm operation.

[0069] In some embodiments, optionally, the DAG parser 113 is used to obtain input fields and output fields of the algorithm nodes; and process data outliers according to the input fields and output fields to correct the dependency relationship of the algorithm nodes.

[0070] Each algorithm node declares the required input fields and output fields, which are fields in the database table. Verify the input fields and output fields of each algorithm node and determine whether there are data outliers. If there are data outliers after verification, the algorithm node needs to be adjusted and the dependencies of the algorithm nodes need to be corrected. By continuously correcting the dependencies of the algorithm nodes, it is beneficial to group the algorithm nodes that do not have dependencies in the subsequent steps. Algorithm nodes that do not have direct dependencies can be processed in parallel, thereby improving the parallelism and execution efficiency of the algorithm operation.

[0071] In some embodiments, optionally, the DAG parser 113 is used to parse the DAG graph, convert the graph data of the DAG graph into relational data and perform data analysis and processing; determine the execution order of the algorithm nodes and the dependency relationship.

[0072] Using graph technology to sort out the upstream and downstream dependencies (determining the execution order and dependencies of algorithm nodes by parsing the DAG graph) is conducive to quickly configuring the algorithm scheduling graph, flexibly combining synchronization data sources and data processing methods and algorithms, and parallel processing of algorithm nodes without direct dependencies, which is conducive to significantly improving the parallelism and execution efficiency of algorithm operation.

[0073] Optionally, during the graph data loading process, the previous data is converted into relational data for processing, and the point data and edge data in the graph are recorded separately.

[0074] In some embodiments, optionally, Figure 4 As shown, the processing unit 120 is used to mark the running status of each algorithm node. The algorithm nodes that have been running are marked as completed, the algorithm nodes that are running are marked as executing, and the algorithm nodes that are not running are marked as waiting; for multiple algorithm nodes with dependencies, they are run sequentially in the execution order; for multiple algorithm nodes that do not have dependencies, they are processed in parallel.

[0075] After clicking the data run, the processing unit 120 will pull the data and perform calculations according to the algorithm. After the calculation is completed, the current algorithm node will be marked as executed, and then the next algorithm node will be executed until all algorithm nodes are marked as executed.

[0076] During the data operation process, multiple algorithm nodes without dependencies are processed in parallel; multiple algorithm nodes with dependencies are run in sequence according to the execution order. This data processing method can ensure the accuracy of the data processing results and the efficiency of the data processing process.

[0077] It should be noted that during the data operation, if the machine crashes or the task is interrupted, the task status of the algorithm node needs to be verified after the machine is restored. For the algorithm nodes marked as executing or waiting, they continue to run according to the execution order and dependency relationship.

[0078] Alternatively, if Figure 5 As shown, the processing unit 120 has the function of task merging. Each algorithm node will have an algorithm type. If the algorithm type of the current algorithm node and the algorithm type of the next algorithm node are both conversion types, the above two algorithm nodes can be merged. In other words, the processing unit 120 can merge algorithm nodes with the same algorithm type. This data processing method can reduce useless serialization processes and improve execution efficiency.

[0079] In some embodiments, optionally, Figure 2 As shown, the data processing device 100 further includes a storage unit 130. The storage unit 130 is used to collect the results after the algorithm nodes are executed and store them in a temporary table.

[0080] The storage unit 130 can store data after unifying the data type through a translator according to the semantic model.

[0081] It should be noted that Figure 2 mysql, oracle, postgreSQI, and hbase refer to storage platforms.

[0082] In one embodiment according to the present invention, a data processing method is used for automatic construction and scheduled execution of an optimization algorithm.

[0083] like Figure 6 As shown, the steps of the data processing method include: S302, choreographing the algorithm execution process based on the algorithm library, setting the algorithm parameters, and constructing the data execution process.

[0084] The algorithms in the algorithm library include one or a combination of the following: evaluation algorithms, time series algorithms, machine learning algorithms, classification algorithms, data cleaning algorithms, custom algorithms, and outlier detection algorithms.

[0085] Configure the corresponding input parameters and output parameters in the algorithm library, that is, determine the input and output parameters of the algorithm. It should be noted that the input and output parameters of the algorithm are the data received and returned by the algorithm function or process. The logic unit uses at least one algorithm in the algorithm library to build a data execution process, arrange the algorithm execution process, and set the algorithm parameters.

[0086] S304: Process the data execution flow to determine a DAG graph.

[0087] It should be noted that DAG (Directed Acyclic Graph) is a directed acyclic graph. DAG and DAG graph in the present invention both represent directed acyclic graphs. A directed acyclic graph is a special directed graph that does not contain any loops.

[0088] In DAG, each edge has a direction, and there cannot be a path from a vertex through several edges back to the vertex. DAG is often used to describe some tasks or events with dependencies, where the vertex represents an algorithm node and the edge represents the dependency. For example, in a task scheduling system, each algorithm node can be represented as a vertex in DAG, and the dependency between algorithm nodes can be represented as an edge. By topologically sorting DAG, the execution order between multiple algorithm nodes can be determined.

[0089] S306, parsing the DAG graph and determining the execution order and dependency relationships of the algorithm nodes.

[0090] Dependencies are usually defined by describing the relationship between tasks. When the output of one task is the input of another task, it means that there is a dependency between the two tasks.

[0091] For example, task A must be executed before task B, which means that the execution order of task A is before the execution order of task B. The output of task C is the input of task D, which means that there is a dependency relationship between task C and task D. Usually, algorithm nodes without dependencies are grouped to ensure that algorithm nodes without dependencies can be processed in parallel, thereby achieving efficient task scheduling and significantly improving the parallelism and execution efficiency of algorithm operation.

[0092] S308, grouping algorithm nodes that have no dependency relationship according to the execution order.

[0093] Algorithm nodes that are executed earlier in the order are processed first, and algorithm nodes that do not have dependencies are processed in parallel. This data processing method can ensure efficient task scheduling while significantly improving the parallelism and execution efficiency of algorithm operation.

[0094] S310, according to the algorithm definition of the algorithm library, determine the grouped algorithm nodes to be run in the business middle station and / or data middle station and / or intelligent middle station, and process the algorithm nodes that have no dependencies in parallel.

[0095] It should be noted that the Business Middle Office is an important platform built by enterprises in the process of digital transformation. It carries the core business capabilities of the enterprise and provides shared services and support for the various business lines of the enterprise. The Business Middle Office provides strong support for the development of the enterprise through functions such as capability sharing and reuse, rapid response to business needs, data integration and analysis, and optimization of resource allocation. The Data Middle Platform is the sedimentation of the business and data of existing or newly built information systems. It is an intermediate and supporting platform for realizing data empowerment of new businesses and new applications. The Intelligent Middle Office is a platform based on the Data Middle Office that provides a series of intelligent services such as model or algorithm libraries, model design training, reuse annotation management, and monitoring services.

[0096] In other words, the processing unit of the present invention includes one of the following or a combination thereof: business middle station, data middle station and intelligent middle station. Business middle station, data middle station and intelligent middle station are execution units of algorithms. The data middle station is partial to data processing, data cleaning, and simple data statistics; the intelligent middle station supports machine learning related algorithms; the business middle station adopts simple algorithm formulas and has high execution efficiency.

[0097] The present invention aims to provide a data processing method, which uses graph technology to sort out the upstream and downstream dependencies (by parsing the DAG graph, determining the execution order and dependencies of the algorithm nodes), which is conducive to quickly configuring the algorithm scheduling graph, flexibly combining the synchronization data source and the data processing method and algorithm, and can also process in parallel the algorithm nodes without direct dependencies, which is conducive to significantly improving the parallelism and execution efficiency of the algorithm operation.

[0098] The data processing method of the present invention can use the algorithm to automatically build the orchestration and scheduling technology in the supply chain demand forecasting and operation optimization scenarios, and realize the automatic construction and scheduling execution of the algorithm by flexibly configuring the input data set, algorithm input and output parameters, multi-algorithm routing, algorithm scheduling, and output indicator mapping of the algorithm scheme. The data processing method of the present invention is mainly used in business fields such as planning and budgeting, data analysis, monitoring and early warning, and can optimize the automatic construction and scheduling execution of the algorithm.

[0099] It should be emphasized that the data processing method of the present invention can automatically construct and schedule the algorithm flow in the prediction and operation scenarios. By arranging the input and output of the algorithm solution and executing the process, it is conducive to flexible scheduling, and different business data can be quickly processed and predicted. The data processing method of the present invention can support algorithm execution in a variety of environments, such as Java (a programming language) code, business middle platform, data middle platform, and intelligent middle platform. The data processing method of the present invention can perform efficient data analysis, data detection, and abnormal data processing, and can be used in various business fields for automatic construction and scheduling of optimization algorithms.

[0100] In some embodiments, optionally, Figure 7 As shown, S304 (processing the data execution process to determine the DAG graph) specifically includes the following steps: S3042, determine the first data source according to the data execution process, and obtain the second data source.

[0101] A directed acyclic graph is determined based on the first data source and the second data source.

[0102] S3044, perform data conversion on the first data source at least twice, and normalize the second data source.

[0103] Optionally, the number of data conversions is two. In this step, the first data source is converted, and after the data conversion, the data is processed by a cubic exponential smoothing algorithm. It should be noted that normalization is an important method in data preprocessing, which aims to convert feature data of different dimensions to the same order of magnitude for comprehensive comparative evaluation.

[0104] S3046, performing data processing on the first data source after at least two data conversions and the second data source after normalization by using a cubic exponential smoothing algorithm.

[0105] It should be noted that the triple exponential smoothing algorithm is a time series forecasting technology that uses the weighted average of the observations of each period in chronological order as the forecast value, showing that the impact of the future value of historical data decreases with time. This method is more practical and suitable for analyzing all application problems of time series.

[0106] S3048, determine the DAG graph.

[0107] After the data is processed by the triple exponential smoothing algorithm, it enters the model evaluation and determines the directed acyclic graph.

[0108] In some embodiments, optionally, Figure 8 As shown, before S308 (grouping algorithm nodes without dependency according to execution order), the steps of the data processing method further include: S307, processing the data outliers to correct the dependency relationship of the algorithm nodes.

[0109] After determining the DAG graph, the data in the DAG graph needs to be verified. If there are data outliers after verification, the algorithm nodes need to be adjusted and the dependencies of the algorithm nodes need to be corrected. By continuously correcting the dependencies of the algorithm nodes, it is beneficial to group the algorithm nodes that do not have dependencies in the subsequent steps. Algorithm nodes without direct dependencies can be processed in parallel, thereby improving the parallelism and execution efficiency of the algorithm operation.

[0110] In some embodiments, optionally, Fig. 9 As shown, S307 (processing data outliers to correct the dependency of algorithm nodes) specifically includes the following steps: S3072, obtain the input field and output field of the algorithm node.

[0111] S3074, processing data outliers according to input fields and output fields to correct dependency relationships of algorithm nodes.

[0112] Each algorithm node declares the required input fields and output fields, which are fields in the database table. Verify the input fields and output fields of each algorithm node and determine whether there are data outliers. If there are data outliers after verification, the algorithm node needs to be adjusted and the dependencies of the algorithm nodes need to be corrected. By continuously correcting the dependencies of the algorithm nodes, it is beneficial to group the algorithm nodes that do not have dependencies in the subsequent steps. Algorithm nodes that do not have direct dependencies can be processed in parallel, thereby improving the parallelism and execution efficiency of the algorithm operation.

[0113] In some embodiments, optionally, Fig.10 As shown, S306 (parsing the DAG graph and determining the execution order and dependency of the algorithm nodes) specifically includes the following steps: S3062, parse the DAG graph, convert the graph data of the DAG graph into relational data and perform data analysis and processing.

[0114] S3064, determine the execution order and dependency of the algorithm nodes.

[0115] Using graph technology to sort out the upstream and downstream dependencies (determining the execution order and dependencies of algorithm nodes by parsing the DAG graph) is conducive to quickly configuring the algorithm scheduling graph, flexibly combining synchronization data sources and data processing methods and algorithms, and parallel processing of algorithm nodes without direct dependencies, which is conducive to significantly improving the parallelism and execution efficiency of algorithm operation.

[0116] Optionally, during the graph data loading process, the previous data is converted into relational data for processing, and the point data and edge data in the graph are recorded separately.

[0117] In some embodiments, optionally, Fig.11 As shown, the steps of the data processing method include: S3122, mark the running status of each algorithm node, mark the algorithm node that has been running as completed, mark the algorithm node that is running as executing, and mark the algorithm node that is not running as waiting.

[0118] S3124, for multiple algorithm nodes with dependencies, they are run sequentially in the execution order; for multiple algorithm nodes without dependencies, they are processed in parallel.

[0119] After clicking the data run button, the processing unit will pull the data and perform calculations according to the algorithm. After the calculation is completed, the current algorithm node will be marked as completed, and then the next algorithm node will be executed until all algorithm nodes are marked as completed.

[0120] During the data operation process, multiple algorithm nodes without dependencies are processed in parallel; multiple algorithm nodes with dependencies are run in sequence according to the execution order. This data processing method can ensure the accuracy of the data processing results and the efficiency of the data processing process.

[0121] It should be noted that during the data operation, if the machine crashes or the task is interrupted, the task status of the algorithm node needs to be verified after the machine is restored. For the algorithm nodes marked as executing or waiting, they continue to run according to the execution order and dependency relationship.

[0122] In one embodiment according to the present invention, Fig.12 As shown, the data processing system 300 includes a memory 310 and a processor 320, wherein the memory 310 stores programs or instructions that can be run on the processor 320, and when the processor 320 executes the programs or instructions, the steps of the data processing method in any of the above-mentioned embodiments are implemented, so it has the beneficial effects of any of the above-mentioned embodiments, which will not be repeated here.

[0123] In one embodiment of the present invention, a readable storage medium stores a program or instruction thereon, and when the program or instruction is executed by a processor, the steps of the data processing method in any of the above embodiments are implemented, and thus the beneficial effects in any of the above embodiments are obtained, which will not be described in detail herein.

[0124] In the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0125] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by terms such as “upper”, “lower”, “left”, “right”, “front” and “back” are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, cannot be understood as a limitation on the present invention.

[0126] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0127] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing device, characterized in that: Automatically build and schedule execution of optimization algorithms, including: The configuration unit (110) includes: A logic unit (111) is used to arrange the algorithm execution process based on the algorithm library, set the algorithm parameters, and build the data execution process; A DAG builder (112), configured to perform data processing on the data execution process and determine a DAG graph; A DAG parser (113) is used to parse the DAG graph and determine the execution order and dependency relationship of the algorithm nodes; and A scheduler (114), configured to group the algorithm nodes that do not have the dependency relationship according to the execution order; The processing unit (120) is used to determine the algorithm nodes after grouping and running them in the business middle station and / or the data middle station and / or the intelligent middle station according to the algorithm definition of the algorithm library, and to process the algorithm nodes that do not have the dependency relationship in parallel.

2. The data processing device according to claim 1, characterized in that: The DAG builder (112) is used to determine a first data source according to the data execution process and obtain a second data source; perform at least two data conversions on the first data source and perform normalization processing on the second data source; Processing the first data source after at least two data conversions and the second data source after normalization by a cubic exponential smoothing algorithm; The DAG graph is determined.

3. The data processing device according to claim 1, characterized in that: The DAG parser (113) is used to process data outliers to correct the dependency relationship of the algorithm nodes.

4. The data processing device according to claim 3, characterized in that: The DAG parser (113) is used to obtain the input field and the output field of the algorithm node; and process the data abnormal value according to the input field and the output field to correct the dependency relationship of the algorithm node.

5. The data processing device according to claim 1, characterized in that: The DAG parser (113) is used to parse the DAG graph, convert the graph data of the DAG graph into relational data and perform data analysis and processing; and determine the execution order and dependency relationship of the algorithm nodes.

6. The data processing device according to any one of claims 1 to 5, characterized in that: The processing unit (120) is used to mark the running status of each algorithm node, marking the algorithm node that has been running as completed, marking the algorithm node that is running as executing, and marking the algorithm node that is not running as waiting; for multiple algorithm nodes that have the dependency relationship, they are run in sequence according to the execution order; for multiple algorithm nodes that do not have the dependency relationship, they are processed in parallel.

7. The data processing device according to any one of claims 1 to 5, characterized in that: Also includes: The storage unit (130) is used to collect the results after the algorithm nodes are executed and store them in a temporary table.

8. A data processing method, characterized in that: Automatically build and schedule execution of optimization algorithms, including: Arrange the algorithm execution process based on the algorithm library, set the algorithm parameters, and build the data execution process; Performing data processing on the data execution process to determine a DAG graph; Parse the DAG graph and determine the execution order and dependency of algorithm nodes; According to the execution order, grouping the algorithm nodes that do not have the dependency relationship; According to the algorithm definition of the algorithm library, determine the algorithm nodes after grouping to run in the business middle station and / or data middle station and / or intelligent middle station, and process the algorithm nodes that do not have the dependency relationship in parallel.

9. A data processing system, characterized in that: include: A memory (310) and a processor (320), wherein the memory (310) stores a program or instruction that can be run on the processor (320), and the processor (320) implements the steps of the data processing method according to claim 8 when executing the program or the instruction.

10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or the instruction is executed by a processor, the steps of the data processing method according to claim 8 are implemented.

Images