Industrial algorithm model scheduling method and system of industrial big data

Abstract

The application relates to the technical field of industrial big data processing and algorithm scheduling, and particularly discloses an industrial algorithm model scheduling method and system for industrial big data, wherein the industrial algorithm model scheduling method for industrial big data comprises the following steps: constructing an algorithm dependency graph of an execution pipeline composed of multiple industrial algorithms, and identifying bottleneck nodes affecting overall performance through a critical path analysis algorithm; performing operation fusion on algorithm pairs with close dependency relationships; determining optimal data partitioning strategies for different algorithms, and realizing a flexible execution pipeline; allocating the most matching heterogeneous computing resources to algorithm operations on the critical path, and adjusting the execution priority of the algorithm operations in real time according to system load and running scenarios; and realizing a fine-grained synchronization system; the application improves the execution efficiency of industrial algorithms and meets the high-performance requirements of industrial big data processing scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial big data processing and algorithm scheduling, and more particularly to an industrial algorithm model scheduling method and system for industrial big data. BACKGROUND

[0002] With the development of industry and the popularity of intelligent manufacturing, the amount of data generated in industrial production processes is growing explosively, which puts higher requirements on data processing and algorithm scheduling systems. Industrial big data has the characteristics of large volume, multiple types, and strong timeliness, and requires multiple professional algorithms to work together to extract valuable information to assist decision-making and optimize production.

[0003] However, the existing industrial algorithm scheduling method mainly faces the following problems: the complex data dependency relationship in the industrial algorithm pipeline leads to the fact that single algorithm optimization cannot achieve end-to-end performance improvement, and there is a lack of global optimization perspective; multiple memory copies in the data transmission and format conversion process cause high system overhead and high delay; different algorithms have different data access modes, and a unified partitioning strategy leads to low parallel efficiency; there is a lack of intelligent matching and scheduling capability for heterogeneous computing resources, and the resource utilization rate is not high; the parallel bottleneck of the algorithm pipeline in the big data processing scenario limits the overall throughput.

[0004] Therefore, an industrial algorithm model scheduling method specifically for industrial big data environment is needed, which can optimize the multi-algorithm collaborative execution process from a global perspective, solve complex dependencies, reduce data transmission overhead, and improve parallel efficiency, so as to meet the high-performance requirements of industrial big data processing. SUMMARY

[0005] The present application provides an industrial algorithm model scheduling method and system for industrial big data, which solves the technical problems of complex data dependency, multiple memory copies, lack of global optimization, data access mode difference, and parallel bottleneck in the multi-algorithm collaborative execution process in related technologies.

[0006] The present application provides an industrial algorithm model scheduling method for industrial big data, which includes:

[0007] An algorithm dependency relationship graph of an execution pipeline composed of multiple industrial algorithms is constructed, wherein the graph nodes represent algorithm operations, and the edges represent data dependency relationships, and a bottleneck node affecting the overall performance is identified through a critical path analysis algorithm;

[0008] For the identified bottleneck node, the data exchange mode between adjacent algorithm operations is analyzed based on the data dependency characteristics, the operation fusion is performed on the algorithm pairs with close dependency relationships, and the data transmission and copy overhead is reduced;

[0009] According to the data access mode and parallel acceleration ratio of the algorithm, the optimal data partition strategy is determined for different algorithms, and a flexible execution pipeline is implemented to dynamically adjust the parallelism in the data stream processing process.

[0010] Based on the algorithm calculation feature vector and the calculation resource characteristic vector, the most matched heterogeneous calculation resource is allocated to the algorithm operation on the critical path, and the execution priority of the algorithm operation is adjusted in real time according to the system load and running scene.

[0011] A fine-grained synchronization system is implemented, and the data dependency between algorithms is decomposed into finer-grained partial dependencies, so that the downstream algorithm can start processing when the upstream algorithm produces partial results, reducing the waiting time and improving the concurrency.

[0012] In a preferred embodiment, the critical path analysis algorithm comprises the following steps:

[0013] An execution time weight is assigned to each algorithm node, and the time weight is obtained based on historical execution data or performance model estimation;

[0014] All possible paths from the starting node to the terminating node are calculated:

[0015] Path={Path1,Path2,...,Path n};

[0016] Wherein, Path represents all possible paths from the starting node to the terminating node, Path1, Path2, Path n represent the 1st, 2nd, n-th complete execution path respectively, and n represents the total number of paths;

[0017] For each path Path i , its total execution time is calculated:

[0018] T_path(Path i )=∑v∈Path i w(v);

[0019] Wherein, T_path(Path i ) represents the total execution time of the path Path i , v represents the algorithm node on the path Path i , w(v) represents the execution time weight of the node, and ∑ represents the sum of the weights of all nodes on the path;

[0020] The critical path is determined:

[0021] CirtPath=argmax(T_path(Path i ));

[0022] i.e., the path with the longest total execution time, where CritPath represents the critical path, T_path(Path i ) represents the total execution time of the ith path, and argmax is used to return the variable value that makes the function take the maximum value.

[0023] Identify the nodes on the critical path that are the bottleneck points affecting the overall performance.

[0024] In a preferred embodiment, the data dependency characteristics are quantified by a data dependency strength matrix DS, which is calculated as follows:

[0025]

[0026] where DS(i, j) represents the data dependency strength between algorithm i and algorithm j, D size (i, j) is the data exchange size, F access (i, j) is the access frequency, and T interval (i, j) is the execution time interval between the two algorithms.

[0027] In a preferred embodiment, the data partitioning strategy is based on the speedup ratio of the algorithm to different data characteristics, which is calculated as follows:

[0028]

[0029] where S(v, D, PS i ) represents the speedup ratio of algorithm v when processing data set D using partitioning strategy PS i , T serial (v, D) is the serial execution time of algorithm v processing data D, T parallel (v, D, PSi, n) is the execution time when using partitioning strategy PS i with parallelism degree n.

[0030] In a preferred embodiment, the allocation of the most matching heterogeneous computing resources is determined by an algorithm resource matching degree matrix ARM, which is calculated as follows:

[0031]

[0032] where ARM(v, r) represents the matching degree score between algorithm node v and computing resource r, k represents the number of dimensions of the feature vector, w i is the weight of the ith feature dimension, CF i (v) represents the ith computing feature component of algorithm node v, RF i (r) represents the ith resource feature component of computing resource r, and match(CF i (v), RFi (r) is a matching degree function of the algorithm feature vector and the resource feature vector.

[0033] In a preferred embodiment, the reducing data transmission and copy overhead includes the following steps:

[0034] Creating a shared memory region pool for data exchange among algorithms;

[0035] For each pair of algorithms that need data exchange, allocating a shared memory region;

[0036] Modifying the output interface of the algorithm to write data directly to the shared memory region instead of the local memory;

[0037] Modifying the input interface of the receiving algorithm to read data directly from the shared memory region, avoiding data copying.

[0038] In a preferred embodiment, the implementing a flexible execution pipeline dynamically adjusts the parallelism during data stream processing according to the following formula:

[0039] n t+1 = n t + Δn;

[0040] Δn = f(L sys , P cur , P target );

[0041] Where n t+1 is the adjusted parallelism at the next time point t+1, Δn is the change in parallelism, n t is the current parallelism, f is the adjustment function, L sys is the system load, P cur is the current performance, and P target is the target performance.

[0042] In a preferred embodiment, the real-time adjusting the execution priority of algorithm operations is calculated by the following formula:

[0043]

[0044] Where Priority(v, t) represents the real-time priority value of algorithm node v at time point t, BP(v) is the base priority of algorithm v, k represents the number of adjustment factors that affect the priority, represents the continuous multiplication operation from i=1 to k, α i is the adjustment factor weight, and af i (v, t) is the i-th adjustment factor value of algorithm v at time t.

[0045] In a preferred embodiment, the fine-grained synchronization system determines the algorithm execution opportunity based on data readiness conditions, and the condition expression is:

[0046] Ready(v)=∧ u∈pred(v) Complete(u,output(u,v));

[0047] where Ready(v) represents the readiness state of algorithm node v, ∧ represents the logical "and" operation, pred(v) is the set of predecessor nodes of v, output(u,v) is the output data of node u passed to v, and Complete(u,d) represents that node u has completed the generation of data d.

[0048] In a preferred embodiment, the industrial big data algorithm model scheduling system is used to implement the industrial big data algorithm model scheduling method, and comprises:

[0049] An algorithm dependency graph construction module is used to construct a dependency graph of the algorithm execution pipeline and identify bottleneck nodes.

[0050] A zero-copy data channel module is used to realize efficient data transmission between algorithms.

[0051] An adaptive partitioning and scheduling module is used to determine the optimal data partitioning strategy according to the algorithm characteristics and dynamically adjust the parallelism.

[0052] A heterogeneous resource matching and priority allocation module is used to allocate the most matching computing resources to the algorithm and adjust the execution priority.

[0053] A fine-grained synchronization module is used to realize partial data dependency and asynchronous execution mechanism between algorithms.

[0054] The beneficial effects of the present application are:

[0055] Performance improvement: by reducing data transmission overhead between algorithms, optimizing execution path and improving parallel efficiency, the data transmission overhead between algorithms is reduced by 65%, the end-to-end processing delay is reduced by 48%, the overall throughput of the system is improved by 52%, and the ability to process large-scale industrial data is significantly enhanced.

[0056] Resource utilization optimization: through global performance bottleneck identification and precise resource allocation, the situation of some resources being overloaded while others being idle is avoided, the utilization rate of computing resources is improved by 60%, and the system runs more balanced and efficiently.

[0057] Adaptability enhancement: by using adaptive data partitioning strategy and dynamic priority adjustment mechanism, the system can automatically select the best execution strategy according to the algorithm characteristics and runtime environment changes, adapt to various industrial big data processing scenarios, and realize the optimal execution scheme without manual intervention.

[0058] Scalability improvement: Based on modular design and standardized interface, the system supports dynamic addition of new algorithm operations to the existing pipeline, and can automatically perform global re-optimization, making the newly added algorithm work efficiently with the existing algorithm, facilitating the continuous evolution and functional expansion of industrial algorithm models.

[0059] Stability guarantee: Through fine-grained synchronization mechanism and data consistency management, even under high concurrency execution conditions, the accuracy and reliability of the calculation results can be ensured, meeting the strict requirements of industrial control and decision-making systems. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of the industrial algorithm model scheduling method of the industrial big data of the present application;

[0061] Figure 2 is a detailed flowchart of the algorithm dependency relationship graph of the execution pipeline composed of multiple industrial algorithms of the present application;

[0062] Figure 3 is a detailed flowchart of the operation fusion of algorithm pairs with close dependency relationships of the present application;

[0063] Figure 4 is a detailed flowchart of determining the optimal data partitioning strategy for different algorithms of the present application;

[0064] Figure 5 is a detailed flowchart of allocating the most matching heterogeneous computing resources for algorithm operations on the critical path of the present application;

[0065] Figure 6 is a detailed flowchart of decomposing the data dependency between algorithms into more fine-grained partial dependencies of the present application. DETAILED DESCRIPTION

[0066] The industrial algorithm model scheduling method of the industrial big data described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is merely to provide a better understanding of the subject matter described herein, and changes to the function and arrangement of the elements discussed can be made without departing from the scope of the present specification. Various processes or components can be omitted, replaced, or added according to need. In addition, the features described in some examples can also be combined in other examples.

[0067] The industrial algorithm model scheduling method of the industrial big data of at least one embodiment of the present application is disclosed, as shown in Figures 1 to 6 specifically includes the following steps:

[0068] Step 1, construct an algorithm dependency graph of the execution pipeline composed of multiple industrial algorithms, where graph nodes represent algorithm operations and edges represent data dependency relationships, and identify bottleneck nodes affecting overall performance through a critical path analysis algorithm;

[0069] Specifically, the following sub-steps are included:

[0070] Sub-step 1.1, construct an algorithm operation dependency graph;

[0071] Use a directed acyclic graph to represent the execution pipeline composed of multiple industrial algorithms, where graph nodes represent algorithm operations and edges represent data dependency relationships. The specific implementation process is as follows:

[0072] Collect all algorithm operation node information in the industrial environment, including algorithm identifier, algorithm type, input data requirement, output data type, etc.

[0073] Analyze the data flow relationship between algorithms to determine which algorithm output serves as the input of other algorithms.

[0074] Create directed edges to connect related algorithm nodes to form a complete algorithm dependency graph G(V, E), where V is the set of algorithm nodes and E is the set of data dependency edges.

[0075] Sub-step 1.2, perform critical path analysis;

[0076] Apply the critical path analysis algorithm to process the constructed dependency graph to identify bottleneck nodes affecting overall performance, and the calculation process is as follows:

[0077] Assign an execution time weight w(v) to each algorithm node, which is based on historical execution data or performance model estimation;

[0078] Calculate all possible paths from the starting node to the terminating node:

[0079] Path={Path1,Path2,...,Path n};

[0080] Where Path represents all possible paths from the starting node to the terminating node, Path1, Path2, Path n represent the 1st, 2nd, and nth complete execution paths, respectively, and n represents the total number of paths.

[0081] For each path Path i , calculate its total execution time:

[0082] T_path(Path i )=∑v∈Path i w(v);

[0083] where T_path(Path i ) denotes the total execution time of path Path i , v denotes an algorithm node on path Path i , w(v) denotes the execution time weight of the node, and ∑ denotes the sum of weights of all nodes on the path.

[0084] Determine the critical path:

[0085] CritPath = argmax(T_path(Path i ));

[0086] that is, the path with the longest total execution time, where CritPath denotes the critical path, T_path(Path i ) denotes the total execution time of the i-th path, and argmax is used to return the variable value that makes the function take the maximum value.

[0087] Identify the nodes on the critical path, which are the bottleneck points affecting the overall performance.

[0088] The specific implementation of the critical path analysis algorithm uses an improved depth-first search method, as follows:

[0089] Algorithm: Improved critical path analysis

[0090] Input: Algorithm dependency graph G(V, E), node execution time weight w(v)

[0091] Output: Critical path CritPath, critical node set CriticalNodes

[0092] Initialize the earliest start time ES(v) of all nodes in the graph to 0

[0093] Topologically sort all nodes in the graph:

[0094] For each node v:

[0095] ES(v) = max{ES(u) + w(u) | u ∈ predecessor(v)}

[0096] where ES(v) denotes the earliest start time of node v, ES(u) denotes the earliest start time of the predecessor node u, w(u) denotes the execution time of the predecessor node u, and max denotes the maximum value.

[0097] Initialize the latest start time LS(v) of all nodes in the graph to ES(terminal node)

[0098] Topologically sort all nodes in the graph in reverse order:

[0099] For each node v:

[0100] LS(v) = max{LS(u) - w(v) | u ∈ successors(v)};

[0101] where LS(v) denotes the latest start time of node v, LS(u) denotes the latest start time of successor node u, w(v) denotes the execution time of node v itself, and max denotes the maximum value.

[0102] Calculate the time slack of each node:

[0103] Slack(v) = LS(v) - ES(v);

[0104] where Slack(v) denotes the time slack of node v, LS(v) denotes the latest start time of node v, and ES(v) denotes the earliest start time of node v.

[0105] Critical node set:

[0106] CriticalNodes = {v | Slack(v) = 0};

[0107] where CriticalNodes denotes the critical node set, v denotes a node, and Slack(v) denotes the time slack of node v.

[0108] Critical path CritPath is the path formed by connecting the critical node set according to the dependency relationship.

[0109] Application example in steelmaking process algorithm pipeline: A steel production control system uses multiple algorithms to work together, including raw material proportioning optimization algorithm, temperature control algorithm, composition analysis algorithm, quality prediction algorithm, and energy consumption optimization algorithm. Through critical path analysis, it is found that the composition analysis algorithm is the bottleneck node on the critical path, and its execution time accounts for 42% of the total execution time. After optimizing the algorithm, the end-to-end processing delay of the entire pipeline is reduced by 31%, significantly improving production efficiency.

[0110] Sub-step 1.3, calculate node resource consumption indicators;

[0111] Perform resource consumption analysis on each algorithm node to determine its impact on system resources:

[0112] Collect resource indicators such as CPU usage CPU(v), memory occupancy MEM(v), and I / O operation amount IO(v) of each algorithm node v;

[0113] Calculate the comprehensive resource consumption indicator:

[0114] R(v) = a CPU(v) + b MEM(v) + g IO(v);

[0115] wherein R(v) represents the comprehensive resource consumption index of algorithm node v, a represents the weight coefficient of CPU resource, b represents the weight coefficient of memory MEM resource, and g represents the weight coefficient of IO resource;

[0116] The performance influence factor of the node is calculated by combining the resource consumption index and the position of the node on the critical path:

[0117] PIF(v) = R(v) IsCP(v);

[0118] wherein PIF(v) represents the performance influence factor of algorithm node v, R(v) represents the comprehensive resource consumption index of algorithm node v, and IsCP(v) represents whether node v is on the critical path.

[0119] Through the above analysis, the performance bottleneck degree of each node in the algorithm pipeline is sorted, which provides an accurate target for subsequent optimization.

[0120] Step 2, for the identified bottleneck node, based on the data dependency characteristic analysis of the data exchange mode between adjacent algorithm operations, the algorithm pairs with close dependency relationship are fused to reduce the data transmission and copy overhead;

[0121] The following sub-steps are included:

[0122] Sub-step 2.1, analyze the data dependency characteristics between algorithm operations;

[0123] The data dependency relationship between algorithm operations is analyzed in detail to identify the data transmission mode that can be optimized:

[0124] The data exchange characteristics between each pair of adjacent algorithm nodes (v1, v2) are analyzed, including data size, data structure type, access frequency, etc.

[0125] A data dependency strength matrix DS is constructed, wherein element DS(i,j) represents the data dependency strength between algorithm i and algorithm j, and the calculation formula is:

[0126]

[0127] wherein DS(i,j) represents the data dependency strength between algorithm i and algorithm j, D size (i,j) is the data exchange size, F access (i,j) is the access frequency, and T interval (i,j) is the execution time interval between the two algorithms.

[0128] A dependency strength threshold θ is set, and when DS(i,j) > θ, the corresponding algorithm pair is marked as a "strong dependency pair" as a candidate for operation fusion.

[0129] Sub-step 2.2, performing algorithm operation fusion;

[0130] For the identified strong dependency algorithm pair, perform algorithm operation fusion to reduce data transmission overhead:

[0131] For each strong dependency algorithm pair (v1, v2), analyze its operation characteristics and compatibility of resource requirements;

[0132] Evaluate the performance improvement indicator FP(v1,v2) after fusion, the calculation formula is:

[0133] FP(v1,v2) = T e xec(v1) + T e xec(v2) + T comm (v1,v2) - T fusion (v1,v2);

[0134] Where, PF(v1,v2) represents the performance improvement after fusion of algorithms v1 and v2, T e xec(v1), T e xec(v2) respectively represent the time overhead of executing algorithms v1 and v2 alone, T comm (v1,v2) is the data transmission time, T fusion (v1,v2) is the execution time after fusion;

[0135] When FP(v1,v2) > 0, create a fusion operation unit F(v1,v2), merge the execution logic of the two algorithms, and eliminate the intermediate data transmission step.

[0136] Sub-step 2.3, data transmission path optimization;

[0137] To support more efficient data transmission between industrial algorithms, optimize the transmission path:

[0138] Construct a communication delay matrix L = [T c omm(i,j)], where T c omm(i,j) represents the data transmission time from computing node i to node j;

[0139] For transmission tasks with data volume exceeding threshold θ, the following optimization measures are applied:

[0140] When T c omm(i,j) > T threshold , create a dedicated zero-copy data channel DCij between nodes i and j;

[0141] When T c omm(i,j)≤T threshold , use standard data transfer mechanism;

[0142] For frequently interacting node pairs, establish persistent communication connections to reduce connection setup overhead.

[0143] Application example in industrial image processing algorithm pipeline: A product quality inspection system of a factory contains multiple algorithms such as image acquisition, preprocessing, feature extraction, defect recognition, and result analysis. A large amount of image data (about 500MB per second) needs to be transmitted between the image preprocessing and feature extraction algorithms. After implementing the zero-copy data channel, the data transmission time between the two algorithms is reduced from 75ms to 8ms, the CPU usage is reduced by 28%, the overall processing delay is reduced by 36%, and the system can process higher resolution images without increasing hardware costs.

[0144] Step 3: Determine the optimal data partitioning strategy for different algorithms based on their data access patterns and parallel speedup ratios, and implement a flexible execution pipeline to dynamically adjust the parallelism during data stream processing;

[0145] Including the following sub-steps:

[0146] Sub-step 3.1: Analyze the data access pattern of the algorithm;

[0147] Analyze the data access pattern of each algorithm to identify its data parallel characteristics:

[0148] Collect the access pattern information of each algorithm operation v to data D, including sequential access, random access, block access, etc.

[0149] Analyze the data dependency type of the algorithm, including element-level dependency, block-level dependency, global dependency, etc.

[0150] Based on the access pattern and dependency type, evaluate the data parallel potential DP(v, D) of the algorithm, with higher values indicating greater parallel potential.

[0151] Sub-step 3.2: Calculate the parallel speedup ratio and determine the optimal partitioning strategy;

[0152] Based on the algorithm characteristics and data characteristics, calculate the parallel speedup ratio under different partitioning strategies:

[0153] For algorithm v and data D, consider n possible partitioning strategies {PS1, PS2,..., PS n}, where PS1, PS2, PS n represent the 1st, 2nd, and nth partitioning strategies, respectively, and n represents the total number of partitioning strategies.

[0154] For each partition strategy PS i , estimate its parallel execution time T parallel (V, D, PS i , n), where T parallel (V, D, PS i , n) represents the estimated time of executing algorithm v in parallel using partition strategy PS i to process data D with parallelism degree n;

[0155] Calculate the speedup ratio:

[0156]

[0157] where S(v, D, PS i ) represents the speedup ratio of algorithm v using partition strategy PS i to process data set D, T serial (v, D) is the serial execution time of algorithm v to process data D, and T parallel (V, D, PS i , n) is the execution time of using partition strategy PS i with parallelism degree n.

[0158] Select the partition strategy with the highest speedup ratio:

[0159]

[0160] where PS opt represents the optimal partition strategy that can achieve the maximum speedup ratio among all candidate partition strategies, represents the independent variable when taking the maximum value, represents the optimal strategy PS opt selected from all candidate partition strategies PS i that can make the speedup ratio S(v, D, PS i ) reach the maximum value.

[0161] Sub-step 3.3, construct a flexible execution pipeline;

[0162] Implement a flexible execution pipeline that can dynamically adjust the parallelism degree:

[0163] Create a pipeline execution engine PE to support dynamic parallelism configuration;

[0164] Implement a partition data manager PDM responsible for partitioning input data D according to the selected partition strategy PS opt ;

[0165] Establish a parallel task scheduler PTS responsible for allocating partitioned data to computing resources;

[0166] A runtime monitoring module (RTM) is implemented to collect execution performance metrics and system load information.

[0167] Based on the data collected by the RTM, the parallelism degree n and the partitioning strategy PS are dynamically adjusted opt , and the formula is:

[0168] n t+1 = n t + Δn;

[0169] Δn = f(L sys , P cur , P target );

[0170] where n t+1 is the adjusted parallelism degree at the next time point t+1, Δn is the change in parallelism degree, n t is the current parallelism degree, f is the adjustment function, L sys is the system load, P cur is the current performance, and P target is the target performance.

[0171] The specific composition and workflow of the elastic execution pipeline system are as follows:

[0172] System composition:

[0173] Pipeline execution engine (PE): core execution unit, including task queue and work thread pool;

[0174] Partition data manager (PDM): implements various data partitioning strategies, such as hash partitioning, range partitioning, block partitioning, etc.

[0175] Parallel task scheduler (PTS): task allocation system based on work-stealing algorithm;

[0176] Runtime monitoring module (RTM): collects CPU usage, memory usage, I / O throughput, etc.

[0177] Load balancer (LB): adjusts parallelism degree and partitioning strategy according to monitoring data;

[0178] Workflow:

[0179] Initialization phase: set initial parallelism degree n0 and partitioning strategy PS0 according to algorithm characteristics and data size;

[0180] Execution phase:

[0181] PDM partitions input data according to the current partitioning strategy PS_t;

[0182] PTS assigns partitioned tasks to work threads in PE;

[0183] RTM monitors the execution performance and system load in real-time;

[0184] Adjustment phase:

[0185] Trigger adjustment evaluation every fixed interval T;

[0186] Calculate performance deviation: δ = (P_target - P_cur) / P_target; where δ represents the degree of deviation between current performance and target performance, P_target represents the target performance level expected to be achieved, and P_cur represents the actual performance of the current system;

[0187] If |δ| > threshold ε, adjust parallelism and partitioning strategy;

[0188] If δ > 0 (performance deficiency): increase parallelism or select a more efficient partitioning strategy;

[0189] If δ < 0 (resource waste): reduce parallelism or adjust partition granularity.

[0190] Application example in industrial IoT data processing: A large chemical company's device monitoring system needs to process data from tens of thousands of sensors every minute for fault prediction and efficiency optimization. The original fixed parallelism system has increased response delay during peak load and serious resource waste during low load. After applying the elastic execution pipeline, the system can automatically adjust the parallel processing capacity according to the data flow, reducing the processing delay by 52% during peak period, and reducing the resource occupation by 41% during non-peak period. At the same time, it can cope with sudden changes in data flow, and still maintain stable processing delay under the condition of peak-to-valley ratio of 8:1.

[0191] Step 4, based on algorithm calculation feature vector and calculation resource characteristic vector, assign the most matching heterogeneous computing resources to the algorithm operation on the critical path, and adjust the execution priority of the algorithm operation in real time according to the system load and running scene;

[0192] Including the following sub-steps:

[0193] Sub-step 4.1, analyze the algorithm calculation characteristics and resource adaptability;

[0194] Analyze the calculation characteristics of industrial algorithms and evaluate their execution efficiency on different computing resources:

[0195] Extract the calculation feature vector CF(v) of each algorithm v, including calculation density, memory access mode, branch prediction difficulty, parallelism, etc.

[0196] Collect the available heterogeneous computing resource set R = {R1, R2,..., R mcharacteristic vector RF(r) of the i-th feature dimension, where R represents the set of available heterogeneous computing resources, R1, R2, R m represent the 1st, 2nd, m-th available heterogeneous computing resources, respectively, and m represents the total number of resources;

[0197] An algorithm-resource matching degree matrix ARM is constructed, where the element ARM(v, r) represents the execution efficiency of the algorithm v on the resource r, and the calculation formula is:

[0198]

[0199] where ARM(v, r) represents the matching degree score between the algorithm node v and the computing resource r, k represents the number of dimensions of the characteristic vector, w i is the weight of the i-th feature dimension, CF i (v) represents the i-th computing feature component of the algorithm node v, RF i (r) represents the i-th resource feature component of the computing resource r, and match(CF i (v), RF i (r)) is the matching degree function of the algorithm characteristic vector and the resource characteristic vector.

[0200] Sub-step 4.2, calculate the resource allocation score of the critical path operation;

[0201] Assign the most suitable computing resource to the operation on the critical path:

[0202] For each operation node v on the critical path, calculate its execution performance improvement indicator on each resource r:

[0203]

[0204] where S(v, r) represents the performance improvement score of assigning the operation v to the computing resource r, T serial (v) is the serial execution time of the operation v, T parallel (v, r) is the parallel execution time on the resource r, and Priority(v) is the priority weight of the operation v;

[0205] Select the resource-operation pair with the highest score: (v * , r * ) = argmax v，r S(v, r), and assign the resource r to the operation v, where (v * , r * ) represents the pair with the highest S(v, r) score among all possible resource-operation pairs (v, r), and argmax v，rIt means that among all possible combinations of algorithm operations v and computing resources r, find the pair of combinations that can maximize the objective function S(v, r);

[0206] Update the remaining resource set and the unassigned operation set, and repeat step 2 until all critical path operations have resources assigned.

[0207] Sub-step 4.3, implement the dynamic priority adjustment mechanism;

[0208] Establish a dynamic priority adjustment system to adjust algorithm priorities in real time based on system load and execution status:

[0209] Define the priority adjustment factor set AF = {af1, af2, ..., af k}, including waiting time, urgency, resource utilization, etc., where AF represents the priority adjustment factor set, af1, af2, af k They represent the 1st, 2nd, and kth priority adjustment factors respectively, and k represents the total number of priority adjustment factors;

[0210] For each algorithm operation v, initialize its base priority BP(v);

[0211] Implement the priority dynamic calculation function:

[0212]

[0213] Among them, Priority(v, t) represents the dynamic priority of the algorithm operation v at time point t, BP(v) represents the basic priority of operation v, represents the multiplication operation from i=1 to k, α i To adjust the factor weights, af i (v, t) is the value of the i-th adjustment factor of algorithm v at time t;

[0214] During the resource scheduling cycle, algorithm operations are performed according to dynamic priority sorting.

[0215] Through heterogeneous resource matching and dynamic priority allocation, the system can fully utilize diverse computing resources and respond to workload changes, thereby improving overall execution efficiency.

[0216] Step 5: Implement a fine-grained synchronization system to decompose inter-algorithm data dependencies into finer-grained partial dependencies, enabling downstream algorithms to begin processing as soon as upstream algorithms produce partial results, reducing waiting time and improving concurrency.

[0217] The following sub-steps are included:

[0218] Sub-step 5.1, build an asynchronous data preprocessing pipeline;

[0219] During the execution of upstream algorithms, asynchronously pre-process the data required by downstream algorithms:

[0220] Analyze the data dependency graph G(V, E) in the execution pipeline of algorithms, and identify data pre-processing opportunities;

[0221] For each pair of data-dependent algorithms (v1, v2), create an asynchronous pre-processing task P(v1, v2) during the execution of v1;

[0222] Implement a pre-processing task manager PTM responsible for scheduling and executing pre-processing tasks;

[0223] Design a pre-processing strategy, including data format conversion, dimension rearrangement, cache warm-up, etc., to prepare the optimal input format for downstream algorithm v2.

[0224] Sub-step 5.2, implement a fine-grained synchronization system;

[0225] Build a fine-grained synchronization system to minimize waiting time in parallel execution:

[0226] Decompose data dependencies into finer-grained partial dependencies;

[0227] Implement a data readiness notification system to notify downstream algorithm v2 immediately when upstream algorithm v1 produces partial results;

[0228] Build a data version management system DVM to maintain multiple version states {D1, D2,..., Dn} of data D, supporting concurrent read and write among them, where D1, D2, Dn represent the 1st, 2nd, n-th historical versions of data object D, and n represents the total number of historical versions. n n

[0229] Implement a conditional execution model based on data dependencies:

[0230] Ready(v)=∧ u∈pred(v) Complete(u, output(u, v));

[0231] Where Ready(v) represents the readiness condition of algorithm node v, pred(v) is the set of predecessor nodes of v, output(u, v) is the output data of node u passed to v, Complete(u, d) represents that node u has completed the generation of data v, and ∧ represents logical AND operation.

[0232] The detailed architecture and implementation of the fine-grained synchronization system are as follows:

[0233] System composition:

[0234] ​​Dependency Decomposition Unit (DDU): responsible for splitting coarse-grained data dependencies into fine-grained partial dependencies;

[0235] Data Readiness Notifier (DRN): a data readiness event notification system based on publish-subscribe model;

[0236] Version Manager (VM): maintains version history and status of each data object;

[0237] Conditional Execution Controller (CEC): manages execution conditions and trigger logic of algorithms;

[0238] Main Data Structures:

[0239] Data Block Descriptor (DBD):

[0240] {block_id: unique identifier of the data block,

[0241] data_ptr: memory pointer of the data block,

[0242] size: size of the data block,

[0243] version: version number,

[0244] status: readiness status,

[0245] producer: producer algorithm identification,

[0246] consumers: list of consumer algorithms

[0247] }

[0248] Dependency Relation Table (DRT): dependency graph stored using adjacency list

[0249] Key Algorithms:

[0250] Partial Dependency Identification Algorithm: based on data access pattern analysis, identifies data blocks that can be processed in parallel;

[0251] Incremental Notification Algorithm: triggers notifications only when data status changes, reducing communication overhead;

[0252] Version Conflict Resolution Algorithm: multi-version concurrency control based on timestamps;

[0253] Application example in intelligent manufacturing production line scheduling: A certain automobile parts production line uses multiple optimization algorithms to cooperate for production planning and scheduling. The traditional method requires that the pre-algorithm (such as order classification, material demand calculation) be completely executed before starting the subsequent algorithm (such as resource allocation, process scheduling). After applying the fine-grained synchronization system, the resource allocation algorithm can start processing the first 20% of the order data without waiting for all the orders to be processed. In practical applications, the planning generation time is shortened from 15 minutes to 4 minutes, the system responsiveness is significantly improved, and it can adapt to production plan changes faster and reduce the waiting time of the production line.

[0254] Sub-step 5.3, ensure data consistency and concurrency control;

[0255] Implement a data consistency guarantee system to ensure the correctness of high concurrency execution:

[0256] Define the data access mode type DAT = {read shared, write exclusive, read-write mixed};

[0257] For each algorithm operation v and data object d, mark its access mode DAT(v, d);

[0258] Implement a lock allocation algorithm based on access mode:

[0259] For read shared mode, assign a shared lock;

[0260] For write exclusive mode, assign an exclusive lock;

[0261] For read-write mixed mode, implement multi-version concurrency control;

[0262] Build a conflict detection and resolution system. When a data access conflict is detected, use priority or timestamp strategy to resolve the conflict.

[0263] Through asynchronous data preprocessing and fine-grained synchronization mechanism, the system maximizes the parallel execution opportunity while ensuring data consistency, and reduces the waiting time between algorithms.

[0264] Technical effects of the embodiment

[0265] Performance improvement: By reducing data transmission overhead between algorithms, optimizing execution path and improving parallel efficiency, the data transmission overhead between algorithms is reduced by 65%, the end-to-end processing delay is reduced by 48%, the overall throughput of the system is improved by 52%, and the ability to process large-scale industrial data is significantly enhanced.

[0266] Resource utilization optimization: Through global performance bottleneck identification and precise resource allocation, the situation of some resources being overloaded while others being idle is avoided, and the utilization rate of computing resources is improved by 60%, making the system run more balanced and efficient.

[0267] Adaptive Enhancement: By adopting adaptive data partitioning strategies and dynamic priority adjustment mechanisms, the system can automatically select the best execution strategy based on algorithm characteristics and runtime environment changes, adapting to various industrial big data processing scenarios without human intervention to achieve optimal execution plans.

[0268] Improved Scalability: Based on modular design and standardized interfaces, the system supports dynamically adding new algorithm operations to existing pipelines and automatically performs global re-optimization, enabling new algorithms to work efficiently with existing algorithms, facilitating continuous evolution and functional expansion of industrial algorithm models.

[0269] Stability Guarantee: Through fine-grained synchronization mechanisms and data consistency management, even under high concurrency execution conditions, the accuracy and reliability of the calculation results can be ensured, meeting the strict requirements of industrial control and decision-making systems.

[0270] Real Application Examples of the Embodiment

[0271] The embodiment has been applied in the intelligent manufacturing platform of a large steel enterprise, solving the performance bottleneck problem of multi-algorithm collaborative operation in the steel production process. The platform needs to run multiple industrial algorithms in real time, including raw material proportioning optimization, furnace temperature control, quality prediction, energy consumption optimization, and defect detection, etc. core algorithms. There are complex data dependency relationships between these algorithms, and the demand for computing resources varies significantly.

[0272] Project Background: In the production intelligent upgrading project of the steel enterprise, various industrial algorithms are needed to improve production efficiency and product quality. Especially on high-end steel production lines, the real-time, precision, and reliability of algorithms are extremely high. Before upgrading, each algorithm model runs independently, and data is exchanged through intermediate files or databases, causing a significant performance bottleneck:

[0273] High data transmission delay between algorithms affects overall response time;

[0274] Lack of global resource scheduling, leading to overloading of some hardware resources and idling of others;

[0275] Unable to dynamically adjust execution strategies for different workloads, peak processing capacity is limited;

[0276] Performance bottlenecks in complex algorithm pipelines are difficult to identify and optimize;

[0277] Data consistency is difficult to guarantee under high concurrency scenarios;

[0278] System scale: The platform needs to process about 20 TB of industrial data per day, support the cooperative operation of more than 30 different types of industrial algorithm models, and provide real-time control and decision support for 5 production lines in the factory. The system hardware environment includes 32 servers equipped with CPU, GPU and FPGA heterogeneous computing resources.

[0279] Implementation process instance:

[0280] The project implements the industrial algorithm model scheduling method of the application, and the main implementation process is as follows:

[0281] Algorithm dependency graph construction and bottleneck identification:

[0282] First, we constructed a complete algorithm dependency graph for the core production line of a steel enterprise. Table 1 shows some key algorithm node information:

[0283] Table 1: Key algorithm node characteristics and bottleneck analysis

[0284]

[0285] Through key path analysis, A3 (component analysis), A5 (defect detection) and A4 (quality prediction) are identified as the main performance bottlenecks, and optimization is prioritized.

[0286] Zero-copy data channel implementation:

[0287] According to the bottleneck analysis results, we prioritize the implementation of zero-copy data channels for algorithm pairs on the critical path. Table 2 shows the data dependency strength between some algorithm pairs and the implementation of zero-copy channels:

[0288] Table 2: Data dependency strength and zero-copy channel implementation

[0289]

[0290] The implementation results show that the zero-copy data channel reduces the average data transmission time by 97.1%, significantly improving the data flow efficiency between algorithms.

[0291] Adaptive data partitioning and parallel scheduling implementation:

[0292] Adaptive data partitioning strategy and elastic execution pipeline are implemented for key algorithms, and Table 3 shows the specific configuration and effect:

[0293] Table 3: Adaptive data partitioning and parallel scheduling configuration

[0294]

[0295] The elastic execution pipeline can automatically adjust the parallelism according to the system load, avoid resource waste while maintaining high performance.

[0296] Heterogeneous resource matching and priority allocation implementation:

[0297] The industrial intelligent manufacturing platform has multiple heterogeneous computing resources, and the algorithm characteristics are matched and dynamically adjusted in priority:

[0298] Table 4: Heterogeneous resource matching and priority configuration

[0299]

[0300] Through the matching of heterogeneous resources and dynamic priority adjustment, the system can automatically adjust the execution priority and resource allocation strategy of the algorithm according to the needs of different production scenarios.

[0301] Fine-grained synchronization system implementation:

[0302] A fine-grained synchronization system is implemented for data dependencies between key algorithms, reducing the waiting time between algorithms:

[0303] Table 5: Fine-grained synchronization configuration and effect

[0304]

[0305] After implementing the fine-grained synchronization system, the waiting time between key algorithms is reduced by an average of 69.1%, significantly improving the parallel processing capability of the system.

[0306] Technical effect verification:

[0307] The industrial algorithm model scheduling method of the present application has achieved significant technical effects in the application of the intelligent manufacturing platform of the steel enterprise, mainly in the following two aspects:

[0308] System performance improvement effect:

[0309] Through the implementation of the scheduling method of the present application, the overall performance of the system is greatly improved, as shown in Table 6:

[0310] Table 6: System performance improvement effect comparison

[0311]

[0312] Actual production benefit:

[0313] The implementation of the present application brings significant production benefits to the steel enterprise, and Table 7 shows the improvement effect of the main production indicators:

[0314] Table 7: Production benefit improvement effect

[0315]

[0316] The implementation effect shows that the application not only improves the technical performance of the system, but also brings real economic benefits, fully verifying the practical value and innovation of the application in the industrial big data environment.

[0317] The above describes the embodiments of the application, but the embodiments are not limited to the above specific implementation, and the above specific implementation is only illustrative but not restrictive, and the ordinary skilled in the art can make more forms of equivalent embodiments under the inspiration of the embodiments, which all belong to the protection of the embodiments.

Claims

1. The industrial algorithm model scheduling method for industrial big data is characterized by: The following steps are involved: Build an algorithm dependency graph for the execution pipeline composed of multiple industrial algorithms, where the graph nodes represent algorithm operations and the edges represent data dependencies. Use the critical path analysis algorithm to identify bottleneck nodes that affect overall performance. For the identified bottleneck nodes, we analyze the data exchange patterns between adjacent algorithm operations based on data dependency characteristics, and perform operation fusion on closely dependent algorithm pairs to reduce data transmission and copying overhead. Determine the optimal data partitioning strategy for different algorithms based on their data access patterns and parallel speedup ratios, and implement elastic execution pipelines to dynamically adjust the degree of parallelism during data stream processing. Based on the algorithm calculation feature vector and computing resource feature vector, the most suitable heterogeneous computing resources are allocated to the algorithm operations on the critical path, and the execution priority of the algorithm operations is adjusted in real time according to the system load and operation scenario; Implement a fine-grained synchronization system to decompose data dependencies between algorithms into finer-grained partial dependencies, so that downstream algorithms can start processing when upstream algorithms produce partial results, reducing waiting time and improving concurrency.

2. The industrial algorithm model scheduling method for industrial big data according to claim 1 is characterized in that: The critical path analysis algorithm includes the following steps: Assign execution time weights to each algorithm node, which are estimated based on historical execution data or performance models. Compute all possible paths from the start node to the end node: ； in, represents all possible paths from the starting node to the ending node, 、 、 Respectively represent 、 、 A complete execution path, Indicates the total number of paths; For each path , calculate its total execution time: ； in, Indicates the path The total execution time, Indicates the path The algorithm node on Indicates the execution time weight of the node, represents the sum of the weights of all nodes on the path; Determine the critical path: ))； That is, the path with the longest total execution time, where represents the critical path, Indicates the The total execution time of the paths, Used to return the variable value that makes the function reach the maximum value; Identify nodes on the critical path that are bottlenecks affecting overall performance.

3. The industrial algorithm model scheduling method for industrial big data according to claim 1 is characterized in that: The data dependency characteristics are characterized by a data dependency intensity matrix To quantify, the calculation formula is: ； in, Representation Algorithm and algorithm The data dependency strength between is the data exchange size, is the access frequency, is the execution time interval between the two algorithms.

4. The industrial algorithm model scheduling method for industrial big data according to claim 1, characterized in that: The data partitioning strategy is based on an algorithm to calculate the parallel speedup ratio of different data characteristics. The calculation formula of the parallel speedup ratio is: ； in, Representation Algorithm When processing the dataset Partitioning strategy The parallel speedup ratio, For the algorithm Processing Data The serial execution time, To adopt a partitioning strategy The degree of parallelism is The execution time of .

5. The industrial algorithm model scheduling method for industrial big data according to claim 1, characterized in that: The allocation of the most matching heterogeneous computing resources is done through the algorithm resource matching matrix Determine, the calculation formula is: ； in, Represents an algorithm node and computing resources The matching score between represents the number of dimensions of the feature vector, For the The weight of the feature dimension, Represents an algorithm node No. Calculate the characteristic components, Represents computing resources No. resource characteristic components, It is a function for calculating the matching degree between characteristic components and resource characteristic components.

6. The industrial algorithm model scheduling method for industrial big data according to claim 1, characterized in that: The reduction of data transmission and copying overhead includes the following steps: Create a shared memory area pool for data exchange between algorithms; For each pair of algorithms that need to exchange data, allocate a shared memory area; Modify the output interface of the algorithm so that its data is written directly to the shared memory area instead of the local memory; Modify the input interface of the receiving algorithm so that it reads data directly from the shared memory area to avoid data copying.

7. The industrial algorithm model scheduling method for industrial big data according to claim 1, characterized in that: The elastic execution pipeline is implemented to dynamically adjust the parallelism during data stream processing according to the following formula: ； ； in, For the next time point The adjusted parallelism, is the change in parallelism, is the current parallelism, To adjust the function, is the system load, For current performance, For target performance.

8. The industrial algorithm model scheduling method for industrial big data according to claim 1, characterized in that: The execution priority of the real-time adjustment algorithm operation is calculated by the following formula: ； in, Represents an algorithm node At the time point The real-time priority value of For the algorithm The basic priority, Indicates the number of adjustment factors that affect the priority, Indicates from arrive The multiplication operation, To adjust the factor weights, For in time algorithm No. An adjustment factor value.

9. The industrial algorithm model scheduling method for industrial big data according to claim 1, characterized in that: The fine-grained synchronization system determines the algorithm execution timing based on the data readiness condition, and its conditional expression is: ； in, Represents an algorithm node Ready state, Represents a logical "AND" operation, for The set of predecessor nodes, For nodes Pass to The output data, Representation node Completed data Generation.

10. An industrial algorithm model scheduling system for industrial big data, configured to implement the industrial algorithm model scheduling method for industrial big data according to any one of claims 1 to 9, comprising: Algorithm dependency graph building module, used to build the dependency graph of the algorithm execution pipeline and identify bottleneck nodes; Zero-copy data channel module, used to achieve efficient data transmission between algorithms; Adaptive partitioning and scheduling module, used to determine the optimal data partitioning strategy based on algorithm characteristics and dynamically adjust the degree of parallelism; Heterogeneous resource matching and priority allocation module, used to allocate the most suitable computing resources to the algorithm and adjust the execution priority; Fine-grained synchronization module, used to implement partial data dependencies and asynchronous execution mechanisms between algorithms.

Images