A data processing coordinator based on shared data processing flow technology

Abstract

The present invention provides a data processing coordinator based on shared data processing flow technology, which belongs to the field of data processing technology. The data processing coordinator of the present invention is connected to the upper-level business system, coordinates and optimizes data according to the current business processing requirements, matches it with appropriate data acquisition sensors, selects the optimal data processing flow and calls corresponding heterogeneous computing resources for data processing, thereby meeting the data requirements of business processing to the greatest extent and improving data processing efficiency. At the same time, the optimal data processing flow is obtained based on the shared data processing flow chart, which contains data processing flows for all businesses. For different businesses, some processing flows can be shared, and the data processing flows can be reused. The business system and the data processing coordinator can be dynamically adapted to ensure the maximum efficiency of data processing in different business environments.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a data processing coordinator based on shared data processing flow technology. Background Art

[0002] With the development of smart grids and the widespread penetration of information technology, the vast amount of power system information, encompassing large spatial scales, small temporal scales, and multiple data types, has significantly fueled the development and application of power big data. The power system is a complex, high-dimensional system, subject to harsh environments and unpredictable human factors. Distribution networks, especially overhead lines, are subject to a high probability of failure. Furthermore, during operation, various abnormal operating conditions, such as magnetizing inrush currents, overloads, lightning strikes, and power theft, are common. Proactively responding to emergencies requires not only accurate, reliable, comprehensive, and timely status information for the distribution network within its jurisdiction, but also professional analysis and diagnosis of this complex, multi-source, heterogeneous data.

[0003] Information mining and diagnosis are aimed at improving the safe operation and service quality of distribution networks, and at preventing and controlling safety risks. A wide variety of basic data types are available in distribution networks, including production management system data, fault repair data, and SCADA data. These data include statistical and operational data on fault type, fault distribution, fault-load relationships, seasonal correlations, power flows, and switch status. Furthermore, distribution network construction and renovation data, equipment maintenance data, and even operational assessment and tripping data can serve as important sources for analyzing and mining distribution network operational status. Due to the complex structure of distribution networks, basic data is diverse and voluminous. For example, the OPEN3000 system contains hundreds of tables with complex data types, inconsistent structures, decentralized storage, and poor correlation. Power grid fault information comprises a variety of structured and unstructured data. Effectively improving the processing capabilities and utilization efficiency of power operation and maintenance big data is crucial for ensuring reliable, safe, efficient, and economical operation of power companies.

[0004] In existing technologies, salespeople often build independent data processing flows for each independent business application. This approach fails to consider reusability across businesses, resulting in wasted computing resources and time-consuming data processing. Furthermore, independent data processing flows are often adapted to individual businesses and are static, making it impossible to coordinate and integrate data processing with business systems. Summary of the Invention

[0005] In view of this, the present invention aims to solve the problem in existing business processing that an independent data processing process needs to be built for each independent business application, resulting in waste of computing resources and the static processing process cannot be coordinated and linked with the business system.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A data processing coordinator based on shared data processing flow technology is used to achieve dynamic linkage between data processing flow and business processing requirements, including:

[0008] The data processing coordinator connects to the upper-level business system to obtain business processing requirements and convert them into sensor requirements, data processing requirements, and computing resource requirements. It also processes the business data required by the business system and returns it.

[0009] Among them, the data processing coordinator configures sensor parameters according to sensor requirements and configures appropriate data acquisition sensors for business processing needs with the goal of minimizing resource usage;

[0010] The data processing process is also constructed according to the data processing requirements. The optimal data processing process is obtained from the shared data processing flow chart with the goal of maximizing processing efficiency. The shared data processing flow chart includes data processing processes that meet all business processing requirements in the business system, and some data processing processes can be shared between different business processing requirements.

[0011] Heterogeneous resources are also scheduled based on computing resource requirements, and the called computing resources process the data required by the business according to the current data processing flow.

[0012] Furthermore, in the shared data processing flow chart, each data processing process is matched to different business processing requirements in a combination of a general method and a corresponding customized configuration.

[0013] Furthermore, the construction process of the shared data processing flowchart specifically includes:

[0014] Build a data processing method library according to general and customized processing procedures;

[0015] According to the data processing requirements of each business, the data processing process is divided into a general method and customized configuration corresponding to each business;

[0016] For each data processing process of the business, search whether the current shared data processing flow chart contains the corresponding general method;

[0017] If the corresponding general method is included and the customized parameter configuration is consistent, the general node in the process is reused;

[0018] If the corresponding general method is included and the customized parameter configuration is inconsistent, modify the general node configuration to the basic configuration that meets all current business processing requirements, and configure the customized node based on the data processing method library;

[0019] If the corresponding general method is not included, the general node is configured based on the data processing method library;

[0020] Combine all configured business data processing nodes to obtain a shared data processing flow chart for all businesses.

[0021] Furthermore, the shared data processing flow chart is in the form of a directed acyclic graph, where each link in the graph represents a data processing flow, each graph node is a specific data processing process, and each different graph node is deployed on different computing resources.

[0022] Furthermore, all graph nodes in the shared data processing flow chart can return data processing results to the business system. Each graph node returns a resource address for obtaining data processing results, and the business system obtains the data required for the business through the resource address.

[0023] Furthermore, the data processing coordinator constructs the data processing process according to the data processing requirements, specifically:

[0024] Based on the shared data processing flow chart, the data processing coordinator searches the graph to find the shortest link on the shared data process and takes the shortest link as the optimal data processing process.

[0025] Furthermore, the data processing coordinator uses a depth-first strategy to search the graph.

[0026] Furthermore, the data processing coordinator processes the business data required by the business system and returns it, specifically including:

[0027] Determine the corresponding sensor data acquisition node as the source node, perform a depth-first search of the shared data processing flow chart, and obtain the currently available data processing flow;

[0028] According to business processing requirements, determine whether the searched data processing flow meets the business processing requirements;

[0029] If satisfied, it is added to the list of available data flows;

[0030] Traverse all available data processing processes and obtain a list of available data processes;

[0031] If the list of available data processes is not empty, the shortest path is selected as the optimal data processing process for the current business;

[0032] If the available data flow list is empty, add a corresponding configuration node for the current business in the shared data processing flow chart to configure the data processing flow of the current business;

[0033] Based on the current utilization of heterogeneous resources, the least busy computing resources are selected for deployment and data processing;

[0034] The resource address after the optimal data processing process is executed is sent to the business system so that the business system can obtain the required business data.

[0035] Furthermore, the general methods include data acquisition, data loading, data cleaning, data format conversion, data downsampling, data pre-processing, data compression and data transmission.

[0036] Furthermore, the data processing coordinator follows the principle of global optimization when matching nodes according to different business processing requirements.

[0037] In summary, the present invention provides a data processing coordinator based on shared data processing flow technology. The coordinator is connected to the upper-level business system, coordinates and optimizes data according to the current business processing requirements, matches it with appropriate data acquisition sensors, selects the optimal data processing flow, and calls corresponding heterogeneous computing resources for data processing, thereby maximizing the satisfaction of the data requirements of business processing and improving data processing efficiency. At the same time, the optimal data processing flow is obtained based on a shared data processing flow chart. The shared data processing flow chart contains data processing flows for all businesses. For different businesses, some processing flows can be shared, and data processing flows can be reused. The business system and the data processing coordinator can dynamically adapt to maximize the efficiency of data processing in different business environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 An architectural diagram of a data processing coordinator based on shared data processing flow technology provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a data processing method library provided by an embodiment of the present invention;

[0041] Figure 3 A directional diagram of data processing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purposes, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] In existing data processing technologies for power grids, operators often build independent data processing processes for each independent business application. After the business is completed, the processed data is released or saved to hardware storage. However, different businesses may have similar data processing processes and reusable data processing processes, such as abnormal data cleaning, data downsampling, and data format conversion. For businesses with large data volumes and complex data processing, traditional processing methods do not consider reusability between businesses, resulting in wasted computing resources and time-consuming data processing.

[0044] Moreover, independent data processing processes are often adapted to independent businesses, and the processing processes are static, resulting in the inability to coordinate and link data processing with business systems. For example, during the business processing process, if the data processing process needs to be changed, the data processing process can only be rebuilt and recalculated, resulting in a significant waste of computing resources.

[0045] Based on this, the present invention provides a data processing coordinator based on shared data processing flow technology. In different business processing processes, the data processing coordinator is used to reuse the data processing flow. The business system and the data processing coordinator can be dynamically adapted to ensure the maximum efficiency of data processing in different business environments to the greatest extent.

[0046] An embodiment of a data processing coordinator based on the shared data processing flow technology of the present invention is described in detail below.

[0047] See also Figure 1 , this embodiment provides a data processing coordinator based on shared data processing flow technology, which is used to realize dynamic linkage between data processing flow and business processing requirements.

[0048] The data processing coordinator connects to the upper-level business system to obtain business processing requirements and convert them into sensor requirements, data processing requirements, and computing resource requirements. It also processes the business data required by the business system and returns it.

[0049] Among them, the data processing coordinator configures sensor parameters according to sensor requirements and configures appropriate data acquisition sensors for business processing needs with the goal of minimizing resource usage;

[0050] The data processing process is also constructed according to the data processing requirements. The optimal data processing process is obtained from the shared data processing flow chart with the goal of maximizing processing efficiency. The shared data processing flow chart includes data processing processes that meet all business processing requirements in the business system, and some data processing processes can be shared between different business processing requirements.

[0051] Heterogeneous resources are also scheduled based on computing resource requirements, and the called computing resources process the data required by the business according to the current data processing flow.

[0052] The data processing coordinator provided in this embodiment ensures the maximum efficiency of data processing in different business environments to the greatest extent possible, mainly including two aspects. First, by building a shared data processing process, the data processing coordinator can reuse common data processing processes under different business processing requirements, reducing the waste of computing resources; second, based on the data processing process orchestration, data acquisition parameters and heterogeneous computing environment configuration input, data coordination and optimization are performed through the data processing coordinator. Among them, the data processing process refers to the process of reading data in real time, then performing real-time calculations on the data, and providing the business with real-time feedback on relevant data results.

[0053] The specific implementation process of the above two aspects is introduced in detail below.

[0054] 1) Construction of shared data processing process

[0055] The shared data processing process includes data collection, data loading, data cleaning, data format conversion, data downsampling, data pre-processing, data compression, and data transmission. The shared process optimizes data processing parameters and steps based on the processing requirements of different business systems. The process then returns a URL for obtaining the data processing results. Different business systems use this URL to retrieve the data they need. It should be understood that the URL serves as a resource address for business systems to retrieve the required data. Other technical means that can achieve the same purpose are applicable and are not detailed here.

[0056] There is a high probability of a mismatch between shared data processing flows and business requirements. When different businesses require sensor data collected at different resolutions, formats, or frequencies, the shared data processing flow automatically resolves the mismatch between sensor capabilities and business requirements in the most efficient way, ensuring that all running models provide appropriate processed data without incurring excessive delays and overhead for the host device.

[0057] The shared data processing process is matched to different business needs, and is carried out in a way that combines general methods and customizes configurations. The entire processing process is presented in the form of a DAG (directed acyclic graph). All graph nodes can return data processing results to the business system to realize the sharing of data processing processes. The general method obtains general requirements in the requirements. For example, for size processing, the general process obtains the maximum size of all requirements. During the data acquisition process, the sensor is configured for the maximum size data, and other business needs derive the downsampling processing node on this node. The image format conversion configuration is based on the YUV format, and other business needs derive the format conversion node on the YUV node. When multiple businesses need to receive images from the camera at a frequency of 10Hz, but they all require different resolutions and color space combinations. By sharing the downsampling and resizing operations between all models, the system avoids repeating these operations when each business has a separate pipeline.

[0058] Specifically, the construction process of the shared data processing flowchart is as follows:

[0059] 101. According to the general and customized processing flow, build a data processing method library, which contains general methods and customized configurations, as follows: Figure 2 shown.

[0060] 102. In response to business data processing requirements, the data processing process is split into general methods and corresponding customized configurations.

[0061] 103. For each data processing process of the business, search whether the current global shared data DAG process contains this common method.

[0062] 104. If a general method is included and the customized parameter configuration is consistent, reuse the general node.

[0063] 105. If a general method is included and the customized parameter configuration is inconsistent, modify the general node configuration to meet the basic configuration of all current business requirements, and customize the business processing flow.

[0064] 106. If the general method is not included, configure the general node.

[0065] 107. Combine the above business data processing nodes to output the overall business data processing flow chart. In this case, the data processing flow is a link in the global shared data flow chart DAG.

[0066] 2) Data processing coordinator performs data coordination and optimization

[0067] The data processing coordinator abstracts away the details of each data acquisition sensor and data processing, presenting a unified interface to the upper layers of the system. Based on business data processing requirements, the data processing coordinator translates these requirements into sensor requirements, data processing requirements, and computing resource requirements. For example, requirements for image acquisition and model processing include sampling rate, resolution, and color space. These requirements define each business model's expectations for its input data in order to complete computations and produce valid inference results. Any deviation from these expectations will result in the model failing to execute or a significant degradation in its output quality (i.e., recognition accuracy).

[0068] The data processing coordinator automatically resolves mismatches between sensor capabilities and business requirements in the most efficient manner, ensuring that all running businesses receive appropriate sensor data without incurring excessive latency and overhead for host devices. The data processing coordinator manages and coordinates sensor instances and conversion protocols, configures shared processing parameters, and selects appropriate data processing flows to run shared data processing processes. This enables dynamic linkage between data processing and business processing.

[0069] The data processing coordinator is built on the basis of shared data processing processes, data sensor capabilities, and a heterogeneous computing environment to meet the data input requirements of the model. The shared data processing process provides data processing orchestration and parameter configuration, the data sensors provide data acquisition parameter configuration, and the heterogeneous processors provide the data computing environment. The data processing coordinator's role is to coordinate and optimize data based on the data processing orchestration, data acquisition parameters, and heterogeneous computing environment configuration inputs to maximize the data requirements of model processing and improve data processing efficiency.

[0070] Specifically: First, we model the data processing process. The shared data processing process can be viewed as a directed acyclic graph. The nodes on the graph are specific data processing processes, the edges are the connections between the data processing processes, and the computing environment is the configuration parameters of the nodes. Different nodes can be deployed on different computing resources, corresponding to different processing performance. A specific business data processing requirement can be viewed as a link in the DAG, which contains multiple nodes and the edges corresponding to the nodes. Figure 3 The case of directed graphs for data processing.

[0071] The data processing coordinator switches to searching the shared data flow DAG graph, finding the shortest data processing flow along the shared data flow. Based on computing resource usage, it selects the most optimal processing resources and configures the shared data flow. When the number of nodes is small, a depth-first strategy can be used to quickly traverse and read all nodes. This strategy is based on a recursive program that is easy to code and understand, and has a relatively low time overhead. The basic idea of ​​the depth-first strategy is that, once a path is selected for search at the current level, the current path is searched until no further is possible. With each level of search, the recursive program generates a context and saves it on the call stack. When the current path reaches its end, the search ends and the current level continues to explore whether other paths exist. If so, the search continues until all branch paths are traversed, all nodes are read, and all possible paths are saved.

[0072] The data processing coordinator coordinates and optimizes data so that the business system can obtain the required business data. The specific steps are as follows:

[0073] 201. Determine the corresponding sensor data acquisition node as the source node, perform a depth-first search of the shared data processing flow chart, and obtain the currently available data processing flow;

[0074] 202. Based on business requirements, determine whether the searched data processing flow meets the business requirements;

[0075] 203. If it meets the business requirements, add it to the list of available data processes;

[0076] 204. Traverse all available processes and obtain a list of available data processes;

[0077] 205. If the list of available data flows is not empty, select the shortest path as the data processing flow for this business;

[0078] 206. If the list of available data processes is empty, build the shared process in S1 and add the corresponding configuration node in the DAG to configure the data processing process according to business requirements;

[0079] 207. During the configuration process, the least busy computing resources are selected for deployment based on the utilization of heterogeneous computing resources;

[0080] 208. Output the URL of the data processing flow to the corresponding business system. When the data processing coordinator matches nodes according to different business requirements, the coordination process follows the principle of global optimization.

[0081] For example, if two vision models require images of 224x224 pixels and 512x512 pixels, respectively, the data processing coordinator will attempt to configure the camera to output images at 512x512 resolution, or larger if that resolution is not available. This avoids unnecessarily loading suboptimal sensor configurations for the current model being run (e.g., capturing 4K images when the service only uses 512x512 images).

[0082] This embodiment provides a data processing coordinator based on shared data processing flow technology. This coordinator interfaces with upper-level business systems, coordinates and optimizes data based on current business processing requirements, matches appropriate data acquisition sensors, selects the optimal data processing flow, and invokes corresponding heterogeneous computing resources for data processing, thereby maximizing the satisfaction of business processing data requirements and improving data processing efficiency. Furthermore, the optimal data processing flow is derived from a shared data processing flow chart, which contains data processing flows for all businesses. For different businesses, portions of the processing flow can be shared, allowing for reuse of data processing flows. Business systems and the data processing coordinator can dynamically adapt to maximize data processing efficiency across diverse business environments.

[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data processing coordinator based on shared data processing flow technology, characterized in that: The data processing coordinator is used to realize the dynamic linkage between data processing flow and business processing requirements, including: The data processing coordinator connects to the upper-layer business system to obtain business processing requirements, and converts the business processing requirements into sensor requirements, data processing requirements and computing resource requirements, and processes the business data required by the business system and returns it; The data processing coordinator configures sensor parameters according to the sensor requirements, and configures appropriate data acquisition sensors for the business processing requirements with the goal of minimizing resource usage; Furthermore, a data processing process is constructed according to the data processing requirements, and an optimal data processing process is obtained from a shared data processing flow chart with the goal of maximizing processing efficiency. The shared data processing flow chart includes data processing processes that meet all business processing requirements in the business system, and some of the data processing processes can be shared between different business processing requirements. Heterogeneous resource scheduling is also performed based on the computing resource requirements, and the called computing resources process the data required by the business according to the current data processing flow; In the shared data processing flow chart, each data processing flow is matched to different business processing requirements in a combination of a general method and a corresponding customized configuration; The construction process of the shared data processing flowchart specifically includes: Build a data processing method library according to general and customized processing procedures; According to the data processing requirements of each business, the data processing process is divided into a general method and customized configuration corresponding to each business; For each data processing process of the business, search whether the current shared data processing flow chart contains the corresponding general method; If the corresponding general method is included and the customized parameter configuration is consistent, the general node in the reuse process; If the corresponding general method is included and the customized parameter configuration is inconsistent, modify the general node configuration to a basic configuration that meets all current business processing requirements, and configure the customized node based on the data processing method library; If the corresponding general method is not included, configuring the general node based on the data processing method library; Combine all configured business data processing nodes to obtain a shared data processing flow chart for all businesses.

2. The data processing coordinator based on shared data processing flow technology according to claim 1, characterized in that: The shared data processing flow chart is in the form of a directed acyclic graph, where each link in the graph represents a data processing flow, each graph node is a specific data processing process, and different graph nodes are deployed on different computing resources.

3. The data processing coordinator based on the shared data processing flow technology according to claim 2, characterized in that: All graph nodes in the shared data processing flow chart can return data processing results to the business system. Each graph node returns a resource address for obtaining data processing results, and the business system obtains data required for the business through the resource address.

4. The data processing coordinator based on the shared data processing flow technology according to claim 3, characterized in that: The data processing coordinator constructs a data processing flow according to the data processing requirements, specifically: The data processing coordinator performs a graph search based on the shared data processing flow chart to find the shortest link on the shared data flow, and uses the shortest link as the optimal data processing flow.

5. The data processing coordinator based on the shared data processing flow technology according to claim 4, characterized in that: The data processing coordinator uses a depth-first strategy to search the graph.

6. The data processing coordinator based on the shared data processing flow technology according to claim 5, characterized in that: The data processing coordinator processes the business data required by the business system and returns it, specifically including: Determine the corresponding sensor data acquisition node as the source node, perform a depth-first search of the shared data processing flow chart, and obtain a currently available data processing flow; According to the business processing requirements, determine whether the searched data processing flow meets the business processing requirements; If satisfied, it is added to the list of available data flows; Traversing all available data processing processes to obtain a list of available data processes; If the available data flow list is not empty, the shortest path is selected as the optimal data processing flow for the current business; If the available data flow list is empty, then add a corresponding configuration node for the current business in the shared data processing flow chart to configure the data processing flow of the current business; Based on the current utilization of heterogeneous resources, the least busy computing resources are selected for deployment and data processing; The resource address after the optimal data processing flow is executed is sent to the business system so that the business system can obtain the required business data.

7. The data processing coordinator based on shared data processing flow technology according to claim 1, characterized in that: The general method includes data acquisition, data loading, data cleaning, data format conversion, data downsampling, data pre-processing, data compression and data transmission.

8. The data processing coordinator based on the shared data processing flow technology according to claim 6, characterized in that: The data processing coordinator follows the principle of global optimization when matching nodes according to different business processing requirements.

Images