Dynamic optimization method and system based on calculation engine model driving operator chain

Abstract

The invention provides a dynamic optimization method and system for a driving operator chain based on a calculation engine model, and relates to the technical field of driving engines, and the method comprises the steps: constructing a multi-dimensional feature sensing network, obtaining operator chain operation data, and training and generating an operator chain performance prediction model. A hierarchical optimization decision mechanism is constructed based on the model, and operators are triggered to be automatically merged in a strategy layer according to an affinity coefficient; the execution layer optimizes the parallelism degree of the merged operators; and the resource layer generates a fine-grained resource allocation scheme to form an end-to-end optimization strategy. After deployment, the adaptive performance monitoring module continuously optimizes the weight of the monitoring index, activates a strategy tuning engine when the performance gain gradient is attenuated, dynamically adjusts optimization parameters and synchronizes optimization records to the knowledge graph, thereby realizing adaptive evolution of an optimization strategy. According to the method, through multi-dimensional feature perception, hierarchical optimization decision and adaptive performance monitoring, the operator chain execution efficiency and the resource utilization rate are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a driving engine technology, and in particular to a method and system for dynamically optimizing an operator chain based on a computing engine model. Background Art

[0002] The computing engine is the core component of modern data processing systems, responsible for performing various complex data processing tasks, such as data cleaning, conversion, aggregation, and analysis. With the continuous growth of data scale and the increasing complexity of application scenarios, higher requirements are placed on the performance and efficiency of computing engines. Traditional computing engines usually adopt static optimization strategies to optimize operator chains during the compilation or deployment phase, which cannot adapt to changes in the runtime environment.

[0003] Unable to dynamically adapt to changes in data flow: Traditional optimization methods are usually based on static data features and resource status, and are difficult to cope with dynamic scenarios such as data flow fluctuations and changes in load distribution. As a result, in actual operation, the performance of the computing engine may not meet expectations, or even result in performance bottlenecks.

[0004] Lack of collaborative optimization between operators: Most existing optimization methods focus on the optimization of a single operator, lacking consideration of the overall performance of the operator chain. The mutual influence and collaborative optimization opportunities between operators are ignored, resulting in limited overall performance improvement.

[0005] Resource allocation strategies are not flexible enough: Traditional resource allocation strategies are usually based on pre-set rules or empirical values, lacking the ability to perceive resource status in real time and adjust dynamically. This may lead to low resource utilization and inability to fully tap the performance potential of the computing engine. Summary of the invention

[0006] The embodiments of the present invention provide a method and system for dynamically optimizing an operator chain driven by a computing engine model, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention, Provides a dynamic optimization method based on the computing engine model to drive the operator chain, including: Construct a multi-dimensional feature perception network, which obtains operator chain operation data through a distributed collection layer, and the distributed collection layer is provided with a time series data collection unit, a load feature collection unit and a resource status collection unit; the time series data collection unit dynamically tracks the data flow characteristics between operators based on a sliding window mechanism and constructs a data flow fluctuation curve; the load feature collection unit obtains the operator calculation load characteristics using an adaptive sampling algorithm and generates a load distribution heat map; the resource status collection unit records the system resource occupancy status based on resource profiling technology to form a multi-dimensional resource utilization matrix; the data flow fluctuation curve, the load distribution heat map and the multi-dimensional resource utilization matrix are input into a deep learning model to train and generate an operator chain performance prediction model; A hierarchical optimization decision mechanism is constructed based on the operator chain performance prediction model. The optimization proxy model is trained by the reinforcement learning algorithm at the strategy layer. The optimization proxy model calculates the affinity coefficients of adjacent operators according to the prediction results. When the affinity coefficient is higher than the dynamic threshold, the automatic merging operation of operators is triggered. The heuristic algorithm is used at the execution layer to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score. An elastic scaling unit is constructed at the resource layer. The elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change in parallelism. The automatic merging operation of operators, the parallelism optimization and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy. Deploy the end-to-end optimization strategy, and start the adaptive performance monitoring module during the execution process. The adaptive performance monitoring module continuously optimizes the monitoring indicator weights based on online learning. The performance gain gradient is calculated by the back-propagation algorithm. When the performance gain gradient shows a decay trend, the strategy tuning engine is activated. The strategy tuning engine dynamically adjusts the optimization parameters using a genetic algorithm, and synchronizes the optimization records to the knowledge graph in real time. The knowledge graph continuously improves the optimization rule base through association analysis, provides knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of the optimization strategy.

[0008] A hierarchical optimization decision-making mechanism is constructed based on the operator chain performance prediction model. The optimization agent model is trained using a reinforcement learning algorithm at the strategy layer. The optimization agent model calculates the affinity coefficients of adjacent operators according to the prediction results, including: A hierarchical optimization decision mechanism is constructed based on the operator chain performance prediction model, and the hierarchical optimization decision mechanism includes a strategy layer, an execution layer, and a resource layer; the strategy layer receives the performance prediction result output by the operator chain performance prediction model, performs principal component analysis and dimensionality reduction processing on the performance prediction result, selects the principal component whose feature contribution rate is higher than the preset contribution threshold, maps the reduced-dimensional features to the interval of zero to one through the Min-Max normalization method, and generates a standardized feature matrix; A reinforcement learning algorithm is used at the strategy layer to train an optimization agent model, wherein the optimization agent model is based on a dual Q network architecture; the standardized feature matrix is ​​input into the optimization agent model, a state space including operator computing load, data flow characteristics, and resource utilization is constructed, an action space of operator merging, splitting, and migration is defined, and a reward function based on performance gain is designed; The optimization proxy model calculates affinity coefficients of adjacent operators based on training results, and the affinity coefficients are comprehensively calculated through three dimensions: data dependency intensity, resource complementarity, and performance improvement expectations.

[0009] At the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score; at the resource layer, an elastic scaling unit is constructed, and the elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change in parallelism; the automatic merging operation of the operators, the parallelism optimization, and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy, including: At the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and a computational complexity scoring model is constructed to analyze the computational density, memory access pattern, and data dependency of the operators to generate static feature evaluation results. The runtime processor usage, memory occupancy, and input and output throughput are monitored to generate dynamic load evaluation results. The static feature evaluation results and the dynamic load evaluation results are weighted to generate an operator complexity scoring matrix. The operator complexity scoring matrix is ​​input into a simulated annealing algorithm, the search temperature parameter is dynamically adjusted based on the degree of performance improvement, the optimal parallelism configuration is found in the search space, and local optimization is performed on the optimal parallelism configuration to obtain an optimized parallelism configuration scheme; the optimized parallelism configuration scheme is input into a time series analysis model to generate a short-term load change trend of the system, and the instance expansion and contraction threshold is set according to the short-term load change trend of the system to achieve dynamic adjustment of the number of instances; An elastic scaling unit is constructed at the resource layer. The elastic scaling unit establishes a mapping relationship model between parallelism and resource requirements according to the optimized parallelism configuration scheme and the instance scaling threshold. The mapping relationship model analyzes the operator complexity scoring matrix to identify an intensive load feature set. The intensive load feature set includes computing intensive load features, memory intensive load features, and input / output intensive load features. The intensive load feature set is input into a deep learning model to predict resource usage trends. The elastic scaling unit automatically generates a fine-grained resource allocation plan based on the resource usage trend, implements resource isolation for resource quotas, and executes resource policies. The merged operator, the optimized parallelism configuration scheme, and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy.

[0010] Deploy the end-to-end optimization strategy, start the adaptive performance monitoring module during the execution process, the adaptive performance monitoring module continuously optimizes the monitoring indicator weights based on online learning; calculate the performance gain gradient through the back propagation algorithm, and when the performance gain gradient shows a decay trend, activate the strategy tuning engine, including: Deploy the end-to-end optimization strategy, start the adaptive performance monitoring module, build a multi-layer perceptron performance prediction model based on the adaptive performance monitoring module, the input layer of the multi-layer perceptron performance prediction model receives basic monitoring indicators such as processor utilization, memory occupancy, and network throughput, the hidden layer of the multi-layer perceptron performance prediction model uses a ReLU activation function to perform nonlinear feature extraction, and the output layer of the multi-layer perceptron performance prediction model generates a system performance prediction result; The adaptive performance monitoring module uses a sliding time window mechanism to continuously collect performance data, calculates the Pearson correlation coefficient between the basic monitoring indicator and the system performance prediction result, and uses the Pearson correlation coefficient as the monitoring indicator weight to weightedly aggregate the basic monitoring indicator to obtain a weighted performance indicator; the weighted performance indicator is divided into training batches according to the time series, and the monitoring indicator weight is updated based on the stochastic gradient descent algorithm using an online learning method. The stochastic gradient descent algorithm uses L2 regularization to constrain the weight parameters, and uses the exponential moving average method to smooth the parameter update process to obtain the optimized monitoring indicator weight; Apply the optimized monitoring indicator weights to real-time performance monitoring, calculate the prediction deviation between the current system performance and the system performance prediction result, and calculate the performance difference between adjacent time windows for historical performance data by time window segmentation to obtain the performance change trend; The prediction deviation and the performance change trend are input into a back propagation algorithm, the back propagation algorithm calculates an end-to-end delay gradient and a throughput gradient, and weighted synthesizes a performance gain gradient based on the end-to-end delay gradient and the throughput gradient; the change trend of the performance gain gradient is detected, and when the performance gain gradient decays in three consecutive time windows, a policy tuning engine is activated.

[0011] The weighted performance index is divided into training batches according to the time series, and the monitoring index weight is updated based on the stochastic gradient descent algorithm using an online learning method. The stochastic gradient descent algorithm uses L2 regularization to constrain the weight parameters and uses the exponential moving average method to smooth the parameter update process. The optimized monitoring index weights include: A double-layer sliding window structure is used to divide the weighted performance index into time series, wherein the double-layer sliding window structure includes an inner time window, and each inner time window collects three hundred performance data sampling points to form a training batch; Based on the attention mechanism, the cosine distance between each sampling point in the training batch and the current system state is calculated to obtain a similarity value, and the similarity value is input into the softmax function to obtain a sample weight vector, and the sample weight vector is used to perform weighted processing on the training batch to obtain weighted training data; Inputting the weighted training data into a stochastic gradient descent algorithm, the stochastic gradient descent algorithm calculates a gradient vector of a monitoring indicator weight, and when a norm of the gradient vector exceeds a preset vector threshold, scaling the gradient vector proportionally to keep the gradient direction unchanged to obtain an adjusted gradient vector; An L2 regularization term is introduced into the loss function to constrain the update of the monitoring indicator weight, and the regularization coefficient of the L2 regularization term decays linearly with the training rounds; the adjusted gradient vector is smoothed by an exponential moving average method, and the adjusted gradient vector and the historical gradient vector are weighted averaged according to an attenuation coefficient of 0.9 to obtain a smoothed gradient vector; Based on the smooth gradient vector, the monitoring indicator weight is updated to obtain an updated monitoring indicator weight, the prediction mean square error in a plurality of consecutive inner time windows is calculated to obtain a prediction accuracy, and the change standard deviation of the updated monitoring indicator weight is calculated to obtain a weight stability index; Monitor the loss change trend of multiple consecutive training batches, increase the learning rate to accelerate convergence when the loss change trend continues to decline, and reduce the learning rate to improve stability when the loss change trend fluctuates; use the updated monitoring indicator weights and the adjusted learning rate for the next round of training iterations until the prediction accuracy and the weight stability index simultaneously meet the preset conditions, and output the optimized monitoring indicator weights.

[0012] The strategy tuning engine uses genetic algorithms to dynamically adjust optimization parameters and synchronizes optimization records to the knowledge graph in real time. The knowledge graph continuously improves the optimization rule base through association analysis to provide knowledge support for subsequent optimization decisions and realize the adaptive evolution of optimization strategies, including: Construct a multi-layer knowledge graph, wherein the multi-layer knowledge graph includes an entity layer, a relationship layer, and an attribute layer; the strategy tuning engine constructs chromosome encoding using a genetic algorithm; extracts the historical optimal configuration from the multi-layer knowledge graph, and performs parameter perturbations on the historical optimal configuration to generate an initial population; constructs a fitness function using system delay, throughput, and resource utilization, calculates the fitness distribution of the initial population, and obtains population convergence and diversity indicators; Dynamically adjusting the selection pressure coefficient based on the population convergence degree, and adopting the roulette strategy to select high-quality individuals; adaptively calculating the crossover probability based on the diversity index, and performing arithmetic crossover on the selected high-quality individuals; adaptively adjusting the mutation probability based on the convergence speed of consecutive generations, and performing Gaussian mutation on the individuals after the crossover; The individuals with the best fitness in each generation and the mutant individuals form a new population; the parameter configuration, performance data, and fitness evaluation results of the new population are used as optimization records and synchronized in real time to the entity layer, the relationship layer, and the attribute layer in the multi-layer knowledge graph; Performing association analysis on the optimization records in the multi-layer knowledge graph, mining parameter configuration rules using a priori algorithms, identifying performance change laws using time series analysis, and constructing parameter adjustment paths using causal reasoning; storing the parameter configuration rules, the performance change laws, and the parameter adjustment paths as optimization strategies in a rule library; Calculate the confidence of the optimization strategy in the rule base, and mark the strategy with a confidence lower than a preset confidence threshold as a strategy to be verified; perform A / B testing in a verification environment to verify the effectiveness of the strategy to be verified; update the optimization strategy that has passed the verification to the rule base to realize dynamic optimization of the rule base; A graph neural network is used to calculate the similarity between the current state and the historical scenario, and the optimization strategy corresponding to the similar scenario is extracted from the rule base; a parameter adjustment scheme is obtained by weighted combination based on the strategy confidence; and the parameter adjustment scheme is used as prior knowledge to guide the population evolution direction of the strategy tuning engine; Monitor the performance indicators during the optimization process, and trigger parameter rollback when performance degradation is detected; record performance fluctuation scenarios to the multi-layer knowledge graph, continuously improve the rule base through continuous two-way feedback, and realize the adaptive evolution of the optimization strategy.

[0013] Dynamically adjusting the selection pressure coefficient based on the population convergence degree, adopting the roulette strategy to select high-quality individuals; adaptively calculating the crossover probability based on the diversity index, and performing arithmetic crossover on the selected high-quality individuals; adaptively adjusting the mutation probability based on the convergence speed of consecutive generations, and performing Gaussian mutation on the individuals after the crossover, including: Dynamically adjusting the selection pressure coefficient based on the population convergence, increasing the selection pressure coefficient according to a first linear function when the population convergence is lower than a preset interval threshold, and reducing the selection pressure coefficient according to a second linear function when the population convergence is higher than the preset interval threshold; calculating the individual selection probability by dividing the power of the selection pressure coefficient of the individual fitness value by the sum of the group fitness, and selecting high-quality individuals according to the individual selection probability using a roulette strategy; The product of the compensation value of the diversity index and the adjustment coefficient is used as a dynamic adjustment amount, and the dynamic adjustment amount is added to the basic crossover probability to obtain an adaptive crossover probability; Calculating a crossover weight coefficient based on the fitness ratio of the high-quality individuals, and performing an arithmetic crossover operation on the high-quality individuals according to the crossover weight coefficient and the adaptive crossover probability to generate offspring individuals; The fitness change rate of the best individuals of two adjacent generations is calculated to obtain the population convergence speed; when the population convergence speed is less than a preset convergence threshold, the basic mutation probability is added to the mutation compensation amount to obtain the adaptive mutation probability; when the population convergence speed is greater than the preset convergence threshold, the basic mutation probability is multiplied by the attenuation function to obtain the adaptive mutation probability; The Gaussian variable step length is calculated based on the population convergence speed, the product of the Gaussian variable step length and a normally distributed random number is used as a mutation amount, and the mutation amount is applied to the offspring individual according to the adaptive mutation probability to perform Gaussian mutation.

[0014] According to a second aspect of the embodiments of the present invention, Provides a dynamic optimization system based on the computing engine model to drive the operator chain, including: The first unit is used to build a multi-dimensional feature perception network, which obtains operator chain operation data through a distributed collection layer, and the distributed collection layer is provided with a time series data collection unit, a load feature collection unit and a resource status collection unit; the time series data collection unit dynamically tracks the data flow characteristics between operators based on a sliding window mechanism and constructs a data flow fluctuation curve; the load feature collection unit uses an adaptive sampling algorithm to obtain the operator calculation load characteristics and generates a load distribution heat map; the resource status collection unit records the system resource occupancy status based on resource profiling technology to form a multi-dimensional resource utilization matrix; the data flow fluctuation curve, the load distribution heat map and the multi-dimensional resource utilization matrix are input into a deep learning model to train and generate an operator chain performance prediction model; The second unit is used to build a hierarchical optimization decision-making mechanism based on the operator chain performance prediction model, use the reinforcement learning algorithm to train the optimization agent model at the strategy layer, and the optimization agent model calculates the affinity coefficients of adjacent operators according to the prediction results; when the affinity coefficient is higher than the dynamic threshold, the operator automatic merging operation is triggered; at the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score; an elastic scaling unit is built at the resource layer, and the elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change in parallelism; the operator automatic merging operation, the parallelism optimization and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy; The third unit is used to deploy the end-to-end optimization strategy, and start the adaptive performance monitoring module during the execution process. The adaptive performance monitoring module continuously optimizes the monitoring indicator weights based on online learning; the performance gain gradient is calculated by the back propagation algorithm, and when the performance gain gradient shows a decay trend, the strategy tuning engine is activated; the strategy tuning engine dynamically adjusts the optimization parameters using a genetic algorithm, and synchronizes the optimization records to the knowledge graph in real time; the knowledge graph continuously improves the optimization rule base through association analysis, provides knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of the optimization strategy.

[0015] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0016] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0017] The beneficial effects of this application are as follows: 1. Improve the accuracy of operator chain performance prediction: This invention builds a multi-dimensional feature perception network, comprehensively considers multi-dimensional features such as data flow, computing load and resource status, and uses a deep learning model for training. It can more accurately predict the performance of the operator chain and provide a reliable basis for subsequent optimization decisions.

[0018] 2. Realize end-to-end dynamic optimization of the operator chain: The present invention constructs a hierarchical optimization decision-making mechanism, covering the strategy layer, execution layer and resource layer. Through operations such as automatic operator merging, parallelism optimization and resource elastic scaling, end-to-end dynamic optimization of the operator chain is realized, effectively improving computing efficiency and resource utilization.

[0019] 3. Ensure the adaptive evolution of optimization strategies: The present invention deploys an adaptive performance monitoring module, and combines technologies such as online learning, strategy tuning engine and knowledge graph to realize the adaptive evolution of optimization strategies. It can dynamically adjust optimization parameters and rules according to actual operating conditions, continuously improve optimization effects, and has the ability of long-term learning and improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a flow chart of a method for dynamically optimizing an operator chain driven by a computing engine model according to an embodiment of the present invention; Figure 2It is a schematic diagram of the structure of a dynamic optimization system for driving an operator chain based on a computing engine model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0023] Figure 1 FIG. 1 is a flow chart of a method for dynamically optimizing an operator chain based on a computing engine model driven by an embodiment of the present invention. Figure 1 As shown, the method includes: S11. Construct a multi-dimensional feature perception network, which obtains operator chain operation data through a distributed collection layer, and the distributed collection layer is provided with a time series data collection unit, a load feature collection unit and a resource status collection unit; the time series data collection unit dynamically tracks the data flow characteristics between operators based on a sliding window mechanism and constructs a data flow fluctuation curve; the load feature collection unit uses an adaptive sampling algorithm to obtain the operator calculation load characteristics and generates a load distribution heat map; the resource status collection unit records the system resource occupancy status based on resource profiling technology to form a multi-dimensional resource utilization matrix; the data flow fluctuation curve, the load distribution heat map and the multi-dimensional resource utilization matrix are input into a deep learning model to train and generate an operator chain performance prediction model; S12. A hierarchical optimization decision mechanism is constructed based on the operator chain performance prediction model. The optimization proxy model is trained by the reinforcement learning algorithm at the strategy layer. The optimization proxy model calculates the affinity coefficients of adjacent operators according to the prediction results. When the affinity coefficient is higher than the dynamic threshold, the automatic merge operation of the operators is triggered. The heuristic algorithm is used at the execution layer to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score. An elastic scaling unit is constructed at the resource layer. The elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change in parallelism. The automatic merge operation of the operators, the parallelism optimization and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy. S13. Deploy the end-to-end optimization strategy, and start the adaptive performance monitoring module during the execution process. The adaptive performance monitoring module continuously optimizes the monitoring indicator weights based on online learning. The performance gain gradient is calculated through the back-propagation algorithm. When the performance gain gradient shows a decay trend, the strategy tuning engine is activated. The strategy tuning engine uses a genetic algorithm to dynamically adjust the optimization parameters, and synchronizes the optimization records to the knowledge graph in real time. The knowledge graph continuously improves the optimization rule base through association analysis, provides knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of the optimization strategy.

[0024] In an optional implementation, a hierarchical optimization decision mechanism is constructed based on the operator chain performance prediction model, and a reinforcement learning algorithm is used to train an optimization agent model at the strategy layer. The optimization agent model calculates the affinity coefficient of adjacent operators according to the prediction results, including: A hierarchical optimization decision mechanism is constructed based on the operator chain performance prediction model, and the hierarchical optimization decision mechanism includes a strategy layer, an execution layer, and a resource layer; the strategy layer receives the performance prediction result output by the operator chain performance prediction model, performs principal component analysis and dimensionality reduction processing on the performance prediction result, selects the principal component whose feature contribution rate is higher than the preset contribution threshold, maps the reduced-dimensional features to the interval of zero to one through the Min-Max normalization method, and generates a standardized feature matrix; A reinforcement learning algorithm is used at the strategy layer to train an optimization agent model, wherein the optimization agent model is based on a dual Q network architecture; the standardized feature matrix is ​​input into the optimization agent model, a state space including operator computing load, data flow characteristics, and resource utilization is constructed, an action space of operator merging, splitting, and migration is defined, and a reward function based on performance gain is designed; The optimization agent model calculates affinity coefficients of adjacent operators based on training results, and the affinity coefficients are comprehensively calculated through three dimensions: data dependency intensity, resource complementarity, and performance improvement expectations.

[0025] First, build an operator chain performance prediction model. This model can predict the operator chain's execution time, resource consumption and other performance indicators based on historical performance data, operator characteristics and other information. For example, collect data such as the execution time, input data volume, and output data volume of an operator chain in the past week, and use machine learning algorithms (such as regression models, neural networks, etc.) to train a performance prediction model.

[0026] Next, we build a hierarchical optimization decision-making mechanism. This mechanism includes the strategy layer, execution layer, and resource layer. The strategy layer is responsible for formulating optimization strategies based on performance prediction results; the execution layer is responsible for executing optimization strategies; and the resource layer is responsible for managing and allocating computing resources.

[0027] Then, in the policy layer, the performance prediction results output by the performance prediction model are received. For example, the prediction model outputs that the estimated execution time of a certain operator chain is 10 minutes and the resource consumption is 100 CPU cores. The performance prediction results are subjected to principal component analysis and dimensionality reduction processing. For example, multiple performance indicators (execution time, resource consumption, etc.) are reduced to a few principal components, and the main feature information is retained. The principal components whose feature contribution rate is higher than the preset contribution threshold are selected. For example, the contribution threshold is set to 80%, and the principal components whose contribution rate exceeds 80% are selected. The reduced features are mapped to the range of zero to one through the maximum and minimum normalization method to generate a standardized feature matrix. For example, the values ​​of the principal components are scaled to between 0 and 1 to form a matrix.

[0028] In the policy layer, a reinforcement learning algorithm is used to train the optimization agent model. The model is based on a dual Q network architecture. The standardized feature matrix is ​​input into the optimization agent model. A state space containing operator computational load, data flow characteristics, and resource utilization is constructed. For example, the state space can contain information such as CPU usage, memory usage, input data volume, and output data volume of each operator. The action space for operator merging, splitting, and migration is defined. For example, two adjacent operators can be merged into one operator, an operator can be split into multiple operators, or an operator can be migrated to another machine for execution. A reward function based on performance gain is designed. For example, if an action can reduce the execution time of the operator chain, a positive reward is given; otherwise, a negative reward is given.

[0029] The optimization proxy model calculates the affinity coefficients of adjacent operators based on the training results. The affinity coefficient is calculated comprehensively based on three dimensions: data dependency strength, resource complementarity, and performance improvement expectations. For example, if there is a strong data dependency between two operators, and their resource requirements are complementary (for example, one operator requires a lot of CPU resources and the other operator requires a lot of memory resources), and merging the two operators can significantly improve performance, then their affinity coefficients are high.

[0030] Finally, according to the affinity coefficient and the decision results of the optimization proxy model, the execution layer performs corresponding optimization operations, such as merging operators with high affinity, splitting operators with low affinity, and migrating operators to appropriate resource nodes. The resource layer allocates and schedules resources according to the instructions of the execution layer.

[0031] The solution of this application can: Improve the performance of the operator chain: By predicting performance bottlenecks and intelligently optimizing the execution strategy of operators, the execution time and resource consumption of the operator chain can be effectively reduced, and the overall performance can be improved. For example, by merging operators with high affinity, the data transmission time and computing overhead can be reduced, thereby improving performance. Enhance resource utilization: Through intelligent resource scheduling and allocation, computing resources can be fully utilized, resource waste can be avoided, and resource utilization can be improved. For example, by migrating operators to appropriate resource nodes, resource loads can be balanced to avoid situations where some nodes are overloaded while other nodes are idle. Automated optimization decisions: By training the optimization agent model through reinforcement learning algorithms, automated optimization decisions can be made without manual intervention, reducing operation and maintenance costs. For example, the optimization agent model can automatically select the optimal optimization strategy based on the performance prediction results without manual parameter adjustment.

[0032] In an optional implementation, a heuristic algorithm is used at the execution layer to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score; an elastic scaling unit is constructed at the resource layer, and the elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change in parallelism; the automatic merging operation of the operators, the parallelism optimization and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy, including: At the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and a computational complexity scoring model is constructed to analyze the computational density, memory access pattern, and data dependency of the operators to generate static feature evaluation results. The runtime processor usage, memory occupancy, and input and output throughput are monitored to generate dynamic load evaluation results. The static feature evaluation results and the dynamic load evaluation results are weighted to generate an operator complexity scoring matrix. The operator complexity scoring matrix is ​​input into a simulated annealing algorithm, the search temperature parameter is dynamically adjusted based on the degree of performance improvement, the optimal parallelism configuration is found in the search space, and local optimization is performed on the optimal parallelism configuration to obtain an optimized parallelism configuration scheme; the optimized parallelism configuration scheme is input into a time series analysis model to generate a short-term load change trend of the system, and the instance expansion and contraction threshold is set according to the short-term load change trend of the system to achieve dynamic adjustment of the number of instances; An elastic scaling unit is constructed at the resource layer. The elastic scaling unit establishes a mapping relationship model between parallelism and resource requirements according to the optimized parallelism configuration scheme and the instance scaling threshold. The mapping relationship model analyzes the operator complexity scoring matrix to identify an intensive load feature set. The intensive load feature set includes computing intensive load features, memory intensive load features, and input / output intensive load features. The intensive load feature set is input into a deep learning model to predict resource usage trends. The elastic scaling unit automatically generates a fine-grained resource allocation plan based on the resource usage trend, implements resource isolation for resource quotas, and executes resource policies. The merged operator, the optimized parallelism configuration scheme, and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy.

[0033] First, perform operator merging. Merge operators that can be merged in the data flow graph, such as merging consecutive matrix multiplication operations into a larger matrix multiplication operation to reduce data transmission and scheduling overhead. For example, merge the three original independent matrix multiplication operators A*B, B*C, and C*D into one (A*B*C)*D operator.

[0034] Then, parallelism optimization is performed. The merged operators are optimized for parallelism to make full use of computing resources. This includes building a computational complexity scoring model. The model takes into account the static characteristics of the operator and the dynamic load evaluation results. Static characteristics include computational density (such as the ratio of the number of floating-point operations to the number of data reads), memory access patterns (such as sequential access or random access), and data dependencies. Dynamic load evaluation results include runtime processor usage, memory usage, and input and output throughput. For example, if an operator requires a large number of floating-point operations but a small amount of data reads, its computational density is high. The operator complexity scoring matrix is ​​generated by weighted calculation of static characteristics and dynamic load evaluation results. Assume that there are two operators. The static characteristic evaluation result of operator 1 is 0.8 and the dynamic load evaluation result is 0.7, with weights of 0.6 and 0.4 respectively; the static characteristic evaluation result of operator 2 is 0.6 and the dynamic load evaluation result is 0.9, with the same weights. Then the complexity score of operator 1 is 0.8 * 0.6 + 0.7 * 0.4 = 0.76, and the complexity score of operator 2 is 0.6* 0.6 + 0.9 * 0.4 = 0.72.

[0035] Next, use the simulated annealing algorithm. The operator complexity score matrix is ​​input into the simulated annealing algorithm to find the optimal parallelism configuration. The simulated annealing algorithm dynamically adjusts the search temperature parameter according to the degree of performance improvement to find the optimal solution in the search space. For example, the initial temperature is set to 100, and the temperature is reduced in each iteration until the temperature is below the preset threshold. At each temperature, the algorithm tries different parallelism configurations and evaluates their performance. If the new configuration performs better, it is accepted; if the performance is worse, it is accepted with a certain probability to avoid falling into a local optimal solution. Perform local optimization on the optimal parallelism configuration found, such as fine-tuning the parallelism, to further improve performance. Assuming that the optimal parallelism configuration is 4, after local optimization, the final parallelism configuration scheme is 3.

[0036] After that, the number of instances is adjusted dynamically. The optimized parallelism configuration scheme is input into the time series analysis model to predict the short-term load change trend of the system. According to the prediction results, the instance expansion and contraction thresholds are set to achieve dynamic adjustment of the number of instances. For example, if the load is predicted to increase in the next hour, the number of instances is increased in advance to avoid performance degradation caused by excessive load. Assuming that the load is predicted to increase by 20% in the next 5 minutes, exceeding the preset threshold of 15%, the instance expansion operation is triggered to increase the number of instances from 2 to 3.

[0037] Then, an elastic scaling unit is built. An elastic scaling unit is built at the resource layer. The unit establishes a mapping relationship model between parallelism and resource requirements based on the optimized parallelism configuration scheme and instance scaling threshold. The model analyzes the operator complexity scoring matrix and identifies intensive load feature sets, including compute-intensive, memory-intensive, and input-output intensive load features. For example, if an operator has a high computational density, it is considered a compute-intensive load. These intensive load feature sets are input into the deep learning model to predict resource usage trends. The elastic scaling unit automatically generates a fine-grained resource allocation scheme based on the prediction results, implements resource isolation for resource quotas, and executes resource policies. For example, if an operator is predicted to require a large amount of memory resources, more memory quotas are allocated to it. Assuming that it is predicted that operator 1 requires 2GB of memory and operator 2 requires 1GB of memory, the elastic scaling unit will allocate 2GB of memory to operator 1 and 1GB of memory to operator 2. In order to improve resource utilization, resource policies can be implemented on resource quotas, such as allowing total memory allocation to exceed the actual available memory.

[0038] Finally, the end-to-end optimization strategy is integrated. The merged operators, the optimized parallelism configuration scheme, and the fine-grained resource allocation scheme are integrated into an end-to-end optimization strategy to achieve global optimal performance.

[0039] The solution of this application can: Improve resource utilization: Through fine-grained resource allocation and resource strategies, resource utilization can be maximized and resource waste can be avoided. Reduce operating costs: By dynamically adjusting the number of instances and optimizing resource allocation, operating costs can be reduced, such as reducing the cost of cloud computing platforms. Improve system performance: Through parallel optimization and elastic scaling, system throughput can be increased and latency can be reduced, thereby improving overall system performance.

[0040] In an optional implementation, the end-to-end optimization strategy is deployed, and an adaptive performance monitoring module is started during execution, wherein the adaptive performance monitoring module continuously optimizes the monitoring indicator weights based on an online learning method; a performance gain gradient is calculated by a back propagation algorithm, and when the performance gain gradient shows a decay trend, the strategy tuning engine is activated, including: Deploy the end-to-end optimization strategy, start the adaptive performance monitoring module, build a multi-layer perceptron performance prediction model based on the adaptive performance monitoring module, the input layer of the multi-layer perceptron performance prediction model receives basic monitoring indicators such as processor utilization, memory occupancy, and network throughput, the hidden layer of the multi-layer perceptron performance prediction model uses a ReLU activation function to perform nonlinear feature extraction, and the output layer of the multi-layer perceptron performance prediction model generates a system performance prediction result; The adaptive performance monitoring module uses a sliding time window mechanism to continuously collect performance data, calculates the Pearson correlation coefficient between the basic monitoring indicator and the system performance prediction result, and uses the Pearson correlation coefficient as the monitoring indicator weight to weightedly aggregate the basic monitoring indicator to obtain a weighted performance indicator; the weighted performance indicator is divided into training batches according to the time series, and the monitoring indicator weight is updated based on the stochastic gradient descent algorithm using an online learning method. The stochastic gradient descent algorithm uses L2 regularization to constrain the weight parameters, and uses the exponential moving average method to smooth the parameter update process to obtain the optimized monitoring indicator weight; Apply the optimized monitoring indicator weights to real-time performance monitoring, calculate the prediction deviation between the current system performance and the system performance prediction result, and calculate the performance difference between adjacent time windows for historical performance data by time window segmentation to obtain the performance change trend; The prediction deviation and the performance change trend are input into a back propagation algorithm, the back propagation algorithm calculates an end-to-end delay gradient and a throughput gradient, and weighted synthesizes a performance gain gradient based on the end-to-end delay gradient and the throughput gradient; the change trend of the performance gain gradient is detected, and when the performance gain gradient decays in three consecutive time windows, a policy tuning engine is activated.

[0041] First, deploy an end-to-end optimization strategy. The purpose of this strategy is to optimize the overall performance of the system, such as reducing end-to-end latency or improving throughput. The specific content of the strategy depends on the actual system, for example, it can be the optimization of resource allocation, task scheduling, data transmission, etc. Suppose the strategy we deploy is to dynamically adjust the resource allocation ratio of each server in the server cluster to balance the load and maximize the overall throughput.

[0042] Next, start the adaptive performance monitoring module. The core function of this module is to continuously optimize the weights of monitoring indicators based on online learning to more accurately reflect system performance. This module includes submodules such as performance data collection, performance prediction model construction, monitoring indicator weight calculation and weight update.

[0043] During the performance data collection phase, the module uses a sliding time window mechanism to continuously collect basic monitoring indicators of the system, such as processor utilization, memory usage, and network throughput. Assume that the sliding time window size is set to 60 seconds and data is collected once per second.

[0044] Then, a multi-layer perceptron performance prediction model is constructed. The input layer of the model receives the collected basic monitoring indicators, such as processor utilization, memory occupancy, and network throughput. The hidden layer uses the ReLU activation function for nonlinear feature extraction. The output layer generates system performance prediction results, such as predicting the average end-to-end delay in the next minute. Assume that the model contains two hidden layers, each containing 10 neurons. The training data of the model comes from historical performance data, such as performance data from the past week.

[0045] After that, the weight of the monitoring indicator is calculated. The module calculates the Pearson correlation coefficient between the basic monitoring indicator and the system performance prediction result. The Pearson correlation coefficient is used to measure the linear correlation between two variables. The calculated Pearson correlation coefficient is used as the monitoring indicator weight, and the basic monitoring indicators are weighted and aggregated to obtain the weighted performance indicator. For example, suppose the Pearson correlation coefficients between the processor utilization, memory occupancy, and network throughput and the system performance prediction result are 0.8, 0.5, and 0.7, respectively. Then the weighted performance indicator is 0.8 * processor utilization + 0.5 * memory occupancy + 0.7 * network throughput.

[0046] Next, the weights of the monitoring indicators are updated using online learning. The weighted performance indicators are divided into training batches according to the time series. The weights of the monitoring indicators are updated using the online learning method based on the stochastic gradient descent algorithm. The stochastic gradient descent algorithm uses L2 regularization to constrain the weight parameters and uses the exponential moving average method to smooth the parameter update process to obtain the optimized monitoring indicator weights. Assume that each update uses a time window of data as a training batch, the learning rate is set to 0.01, the regularization coefficient is set to 0.1, and the exponential moving average decay rate is set to 0.9.

[0047] Then, the optimized monitoring indicator weights are applied to real-time performance monitoring. The prediction deviation between the current system performance and the system performance prediction result is calculated. For example, if the current system end-to-end delay is 100 milliseconds and the system performance prediction result is 90 milliseconds, the prediction deviation is 10 milliseconds. At the same time, the historical performance data is segmented by time window, and the performance difference between adjacent time windows is calculated to obtain the performance change trend. For example, if the average end-to-end delay of the past three time windows is 90 milliseconds, 95 milliseconds, and 100 milliseconds, respectively, the performance change trend is rising.

[0048] Next, the prediction deviation and performance change trend are input into the back propagation algorithm. The back propagation algorithm calculates the end-to-end delay gradient and the throughput gradient. The performance gain gradient is weighted and synthesized based on the end-to-end delay gradient and the throughput gradient. Assuming that the end-to-end delay weight is 0.6 and the throughput weight is 0.4, the performance gain gradient is 0.6 * end-to-end delay gradient + 0.4 * throughput gradient.

[0049] Finally, the performance gain gradient change trend is detected. When the performance gain gradient decays in three consecutive time windows, the policy tuning engine is activated. The policy tuning engine adjusts the deployed end-to-end optimization strategy based on the performance gain gradient change trend and other related information to further improve system performance.

[0050] The solution of this application can: Improve monitoring accuracy: By adaptively adjusting the weights of monitoring indicators, the monitoring system can pay more attention to indicators that are closely related to system performance, thereby more accurately reflecting the system performance status. Improve performance prediction accuracy: Using the multi-layer perceptron performance prediction model, it can more accurately predict the future performance of the system and provide a more reliable basis for policy adjustment. Realize automated policy optimization: Through the performance gain gradient to guide the policy tuning engine, realize automated policy adjustment, reduce manual intervention, and improve system operation and maintenance efficiency.

[0051] In an optional implementation, the weighted performance index is divided into training batches according to the time series, and the monitoring index weight is updated based on the stochastic gradient descent algorithm using an online learning method. The stochastic gradient descent algorithm uses L2 regularization to constrain the weight parameters, and uses the exponential moving average method to smooth the parameter update process. The optimized monitoring index weight includes: A double-layer sliding window structure is used to divide the weighted performance index into time series, wherein the double-layer sliding window structure includes an inner time window, and each inner time window collects three hundred performance data sampling points to form a training batch; Based on the attention mechanism, the cosine distance between each sampling point in the training batch and the current system state is calculated to obtain a similarity value, and the similarity value is input into the softmax function to obtain a sample weight vector, and the sample weight vector is used to perform weighted processing on the training batch to obtain weighted training data; Inputting the weighted training data into a stochastic gradient descent algorithm, the stochastic gradient descent algorithm calculates a gradient vector of a monitoring indicator weight, and when a norm of the gradient vector exceeds a preset vector threshold, scaling the gradient vector proportionally to keep the gradient direction unchanged to obtain an adjusted gradient vector; An L2 regularization term is introduced into the loss function to constrain the update of the monitoring indicator weight, and the regularization coefficient of the L2 regularization term decays linearly with the training rounds; the adjusted gradient vector is smoothed by an exponential moving average method, and the adjusted gradient vector and the historical gradient vector are weighted averaged according to an attenuation coefficient of 0.9 to obtain a smoothed gradient vector; Based on the smooth gradient vector, the monitoring indicator weight is updated to obtain an updated monitoring indicator weight, the prediction mean square error in a plurality of consecutive inner time windows is calculated to obtain a prediction accuracy, and the change standard deviation of the updated monitoring indicator weight is calculated to obtain a weight stability index; Monitor the loss change trend of multiple consecutive training batches, increase the learning rate to accelerate convergence when the loss change trend continues to decline, and reduce the learning rate to improve stability when the loss change trend fluctuates; use the updated monitoring indicator weights and the adjusted learning rate for the next round of training iterations until the prediction accuracy and the weight stability index simultaneously meet the preset conditions, and output the optimized monitoring indicator weights.

[0052] First, collect system performance data and build a performance data set. For example, collect CPU utilization, memory usage, disk IO rate and other performance indicators every 1 second, continuously collect data at 3,000 time points, and build a performance data set containing 3,000 sampling points.

[0053] Then, a double-layer sliding window structure is used to divide the performance data set into time series. The length of the inner time window is set to 300 sampling points, that is, each inner time window contains 300 continuously collected performance data. The length of the outer time window is set according to actual needs, for example, it is set to 10 inner time windows. First, starting from the starting position of the performance data set, the inner time windows with a length of 300 sampling points are intercepted in sequence to form the first training batch. Subsequently, the inner time window is slid back by one sampling point, and the next inner time window containing 300 sampling points is intercepted to form the second training batch, and so on until the entire performance data set is traversed.

[0054] Next, perform weighted processing on each training batch. Obtain the current system status, such as the current CPU utilization, memory usage, disk IO rate, etc. Calculate the cosine similarity between each sampling point in the training batch and the current system status, input the similarity value into the softmax function, and obtain the sample weight vector. For example, the similarity between the first sampling point and the current system status is 0.8, the second sampling point is 0.7, and so on. Input these similarity values ​​into the softmax function to obtain a normalized sample weight vector, such as [0.2, 0.18, …]. Multiply the sample weight vector with each sampling point in the training batch to obtain weighted training data.

[0055] Subsequently, the weighted training data is fed into the stochastic gradient descent algorithm to calculate the gradient vector of the monitoring indicator weight. A preset vector threshold is set, such as 10. If the norm of the calculated gradient vector exceeds 10, the gradient vector is scaled so that its norm is equal to 10, but the gradient direction remains unchanged. For example, if the calculated gradient vector is [20,15], its norm is 25, which exceeds the threshold of 10, it is scaled to [8, 6], and the norm becomes 10, but the direction remains unchanged.

[0056] The L2 regularization term is introduced into the loss function to constrain the update of the monitoring indicator weight. The regularization coefficient of the L2 regularization term decays linearly with the increase of training rounds. For example, the initial regularization coefficient is set to 0.1, and the regularization coefficient decreases by 0.001 after each round of training. The adjusted gradient vector is smoothed using the exponential moving average method, and the decay coefficient is set to 0.9. For example, the current gradient vector is [8, 6] and the historical gradient vector is [7, 5], then the smoothed gradient vector is 0.9 * [7, 5] + 0.1 * [8, 6] = [7.1, 5.1].

[0057] Update the monitoring indicator weight based on the smoothed gradient vector. For example, if the current monitoring indicator weight is [0.2, 0.3], the smoothed gradient vector is [7.1, 5.1], and the learning rate is 0.01, then the updated monitoring indicator weight is [0.2 + 0.01 *7.1, 0.3 + 0.01 * 5.1] = [0.271, 0.351]. Calculate the prediction mean square error in 10 consecutive inner time windows to get the prediction accuracy. Calculate the standard deviation of the change in the updated monitoring indicator weight to get the weight stability index.

[0058] Monitor the loss trend for multiple consecutive training batches. If the loss continues to decrease, increase the learning rate, for example, from 0.01 to 0.02. If the loss fluctuates, decrease the learning rate, for example, from 0.02 to 0.01. Use the updated monitoring indicator weights and adjusted learning rate for the next training iteration.

[0059] Repeat the above steps until the prediction accuracy and weight stability index meet the preset conditions at the same time. For example, the prediction accuracy reaches 95% and the weight stability index is less than 0.01, then output the optimized monitoring index weight.

[0060] The solution of this application can: Improve the accuracy of monitoring indicators: Through online learning and a double-layer sliding window mechanism, the weight of monitoring indicators can be dynamically adjusted according to changes in system status, thereby more accurately reflecting the actual performance of the system. Enhance the stability of the model: Using L2 regularization and exponential moving average methods can effectively suppress overfitting of the model and improve the generalization ability and stability of the model. Improve the adaptability of the model: By dynamically adjusting the learning rate, the model can converge faster and adapt to changes in performance data in different scenarios.

[0061] In an optional implementation, the strategy tuning engine uses a genetic algorithm to dynamically adjust the optimization parameters, and synchronizes the optimization records to the knowledge graph in real time; the knowledge graph continuously improves the optimization rule base through association analysis to provide knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of the optimization strategy, including: Construct a multi-layer knowledge graph, wherein the multi-layer knowledge graph includes an entity layer, a relationship layer, and an attribute layer; the strategy tuning engine constructs chromosome encoding using a genetic algorithm; extracts the historical optimal configuration from the multi-layer knowledge graph, and performs parameter perturbations on the historical optimal configuration to generate an initial population; constructs a fitness function using system delay, throughput, and resource utilization, calculates the fitness distribution of the initial population, and obtains population convergence and diversity indicators; Dynamically adjusting the selection pressure coefficient based on the population convergence degree, and adopting the roulette strategy to select high-quality individuals; adaptively calculating the crossover probability based on the diversity index, and performing arithmetic crossover on the selected high-quality individuals; adaptively adjusting the mutation probability based on the convergence speed of consecutive generations, and performing Gaussian mutation on the individuals after the crossover; The individuals with the best fitness in each generation and the mutant individuals form a new population; the parameter configuration, performance data, and fitness evaluation results of the new population are used as optimization records and synchronized in real time to the entity layer, the relationship layer, and the attribute layer in the multi-layer knowledge graph; Performing association analysis on the optimization records in the multi-layer knowledge graph, mining parameter configuration rules using a priori algorithms, identifying performance change laws using time series analysis, and constructing parameter adjustment paths using causal reasoning; storing the parameter configuration rules, the performance change laws, and the parameter adjustment paths as optimization strategies in a rule library; Calculate the confidence of the optimization strategy in the rule base, and mark the strategy with a confidence lower than a preset confidence threshold as a strategy to be verified; perform A / B testing in a verification environment to verify the effectiveness of the strategy to be verified; update the optimization strategy that has passed the verification to the rule base to realize dynamic optimization of the rule base; A graph neural network is used to calculate the similarity between the current state and the historical scenario, and the optimization strategy corresponding to the similar scenario is extracted from the rule base; a parameter adjustment scheme is obtained by weighted combination based on the strategy confidence; and the parameter adjustment scheme is used as prior knowledge to guide the population evolution direction of the strategy tuning engine; Monitor the performance indicators during the optimization process, and trigger parameter rollback when performance degradation is detected; record performance fluctuation scenarios to the multi-layer knowledge graph, continuously improve the rule base through continuous two-way feedback, and realize the adaptive evolution of the optimization strategy.

[0062] First, build a multi-layer knowledge graph. The knowledge graph contains three core layers: entity layer, relationship layer, and attribute layer. The entity layer is used to store key components in the system, such as servers, databases, caches, etc. The relationship layer describes the association between entities, such as the connection relationship between servers and databases. The attribute layer records the characteristic information of entities and relationships, such as the CPU utilization of servers, the response time of databases, etc. Taking an e-commerce platform as an example, the entity layer can contain entities such as "orders", "products", and "users"; the relationship layer can contain relationships such as "user orders" and "product-related orders"; the attribute layer can contain attributes such as "order amount", "product price", and "user activity".

[0063] Next, the strategy tuning engine uses a genetic algorithm to construct a chromosome encoding scheme. The parameters to be optimized are mapped to the gene bits of the chromosome. For example, the server's thread pool size, database connection pool size and other parameters are encoded into the chromosome. Taking the database connection pool size as an example, assuming that the connection pool size ranges from 10 to 100, a section of the gene bits on the chromosome can be used to represent the connection pool size, for example, using a 7-bit binary number, the corresponding decimal value range is 0-127, and then mapped to 10-100 through linear mapping.

[0064] Then, extract the historical optimal configuration from the multi-layer knowledge graph. By querying the knowledge graph, obtain the parameter configuration when the system performance was the best in the past period of time. For example, by querying the database connection pool size, server thread pool size and other parameters when the order processing speed of the e-commerce platform was the fastest in the past month, obtain the historical optimal configuration. Assume that the historical optimal configuration is a database connection pool size of 50 and a server thread pool size of 200.

[0065] Perform parameter perturbations on the historical optimal configuration to generate the initial population. Based on the historical optimal configuration, randomly adjust the parameters to generate multiple different parameter combinations to form the initial population. For example, randomly select the database connection pool size between 40 and 60, and randomly select the server thread pool size between 180 and 220, to generate 100 different parameter combinations.

[0066] After that, a fitness function is constructed. Key performance indicators such as system latency, throughput, and resource utilization are combined into a fitness function to evaluate the quality of parameter configuration. For example, the fitness function can be defined as the throughput weighted value minus the latency weighted value minus the opposite of the resource utilization weighted value.

[0067] Calculate the fitness distribution of the initial population and obtain the population convergence and diversity indicators. Perform fitness evaluation on each individual in the initial population, count the distribution of fitness values, and calculate the population convergence and diversity. For example, the variance of the fitness values ​​in the population can be calculated to measure diversity. The larger the variance, the better the diversity. Convergence can be measured by calculating the gap between the fitness value in the population and the optimal fitness value. The smaller the gap, the higher the convergence.

[0068] The selection pressure coefficient is dynamically adjusted based on the population convergence, and the roulette strategy is used to select high-quality individuals. According to the size of the population convergence, the selection pressure coefficient is adjusted to control the probability of high-quality individuals being selected during the selection process. The higher the convergence, the greater the selection pressure, and the higher the probability of high-quality individuals being selected. For example, if the population convergence is high, the selection pressure coefficient can be increased to make individuals with higher fitness values ​​more likely to be selected.

[0069] The crossover probability is adaptively calculated based on the diversity index, and arithmetic crossover is performed on the selected high-quality individuals. According to the size of the population diversity, the crossover probability is adaptively adjusted to control the frequency of the crossover operation. The higher the diversity, the smaller the crossover probability. For example, if the population diversity is high, the crossover probability can be reduced to reduce the number of crossover operations. The arithmetic crossover operation refers to the weighted average of the gene bits of two parent individuals to generate new offspring individuals.

[0070] Based on the convergence speed of successive generations, the mutation probability is adaptively adjusted, and Gaussian mutation is performed on the individuals after crossover. According to the change of the population convergence speed, the mutation probability is adaptively adjusted to control the intensity of the mutation operation. The faster the convergence speed, the smaller the mutation probability. For example, if the population converges quickly, the mutation probability can be reduced to reduce the amplitude of the mutation operation. The Gaussian mutation operation refers to adding a random number that follows a Gaussian distribution to the gene position to perturb the gene position.

[0071] The individuals with the best fitness in each generation and the mutated individuals form a new population. The individuals with the highest fitness in the current generation and the individuals that have undergone mutation operations are combined to form a new population and enter the next generation of evolution.

[0072] The parameter configuration, performance data, and fitness evaluation results of the new population are used as optimization records and synchronized to the multi-layer knowledge graph in real time. The parameter configuration, corresponding performance data, and fitness evaluation results of each individual in the new population are stored in the knowledge graph. For example, the database connection pool size is 55, the server thread pool size is 210, the corresponding throughput is 1000TPS, the delay is 50ms, the resource utilization rate is 80%, and the fitness value is 0.95. Such information is stored in the knowledge graph.

[0073] We conduct correlation analysis on the optimization records in the multi-layer knowledge graph, use a priori algorithms to mine parameter configuration rules, use time series analysis to identify performance change patterns, and use causal reasoning to build parameter adjustment paths. We analyze the historical optimization records stored in the knowledge graph and mine the correlation between parameter configuration and performance indicators. For example, we can find that when the database connection pool size increases, the throughput will first increase and then decrease, thus obtaining a parameter configuration rule.

[0074] The parameter configuration rules, performance change rules and parameter adjustment paths are stored in the rule base as optimization strategies. The mined parameter configuration rules, performance change rules and parameter adjustment paths are stored in the rule base as the basis for subsequent optimization decisions.

[0075] Calculate the confidence of the optimization strategies in the rule base, and mark the strategies with confidence lower than the preset confidence threshold as strategies to be verified. Evaluate the confidence of each optimization strategy in the rule base, and mark the strategies with confidence lower than the preset threshold as strategies to be verified. For example, if a rule is based on a small amount of data mining, its confidence may be low and further verification is required.

[0076] Perform A / B testing in the verification environment to verify the effectiveness of the strategy to be verified. Apply the strategy to be verified in the verification environment and compare it with the existing strategy to evaluate its effectiveness. For example, you can apply a new parameter configuration solution to a portion of users to observe whether its performance indicators are improved.

[0077] Update the verified optimization strategies to the rule base to achieve dynamic optimization of the rule base. Update the verified and effective strategies to the rule base and improve their confidence.

[0078] The graph neural network is used to calculate the similarity between the current state and historical scenarios, and the optimization strategies corresponding to similar scenarios are extracted from the rule base. According to the current system status, such as the current load situation, resource utilization, etc., the graph neural network is used to calculate the similarity between the current state and historical scenarios, and the optimization strategies corresponding to similar scenarios are extracted from the rule base.

[0079] The parameter adjustment scheme is obtained by weighted combination based on the confidence of the strategy. According to the confidence of the extracted optimization strategy, a weighted combination is performed to obtain the final parameter adjustment scheme. The higher the confidence of the strategy, the greater its weight.

[0080] The parameter adjustment scheme is used as prior knowledge to guide the population evolution direction of the strategy tuning engine. The obtained parameter adjustment scheme is used as prior knowledge to guide the population evolution direction of the genetic algorithm. For example, the parameter adjustment scheme can be used as part of the initial population.

[0081] Monitor performance indicators during the optimization process and trigger parameter rollback when performance degradation is detected. Continuously monitor system performance indicators and roll back parameters to their previous state if performance degradation is detected.

[0082] Record performance fluctuation scenarios in the multi-layer knowledge graph, continuously improve the rule base through continuous two-way feedback, and realize the adaptive evolution of optimization strategies. Record performance fluctuation scenarios in the knowledge graph and analyze the causes of performance fluctuations to improve optimization strategies and rule bases, and realize the adaptive evolution of optimization strategies.

[0083] The solution of this application can: Improve system performance: By dynamically adjusting parameters, the system is always kept in the best state, thereby improving system throughput, reducing latency, and improving resource utilization. Reduce manual intervention: Through the adaptive evolution mechanism, automatic learning and optimization of parameter configuration, reduce the need for manual intervention and reduce operation and maintenance costs. Enhance system stability: Through the parameter rollback mechanism and performance fluctuation scenario recording, performance problems can be discovered and resolved in a timely manner, enhancing system stability and reliability.

[0084] In an optional implementation, dynamically adjusting the selection pressure coefficient based on the population convergence degree, adopting a roulette strategy to select high-quality individuals; adaptively calculating the crossover probability based on the diversity index, and performing arithmetic crossover on the selected high-quality individuals; adaptively adjusting the mutation probability based on the convergence speed of consecutive generations, and performing Gaussian mutation on the individuals after the crossover include: Dynamically adjusting the selection pressure coefficient based on the population convergence, increasing the selection pressure coefficient according to a first linear function when the population convergence is lower than a preset interval threshold, and reducing the selection pressure coefficient according to a second linear function when the population convergence is higher than the preset interval threshold; calculating the individual selection probability by dividing the power of the selection pressure coefficient of the individual fitness value by the sum of the group fitness, and selecting high-quality individuals according to the individual selection probability using a roulette strategy; The product of the compensation value of the diversity index and the adjustment coefficient is used as a dynamic adjustment amount, and the dynamic adjustment amount is added to the basic crossover probability to obtain an adaptive crossover probability; Calculating a crossover weight coefficient based on the fitness ratio of the high-quality individuals, and performing an arithmetic crossover operation on the high-quality individuals according to the crossover weight coefficient and the adaptive crossover probability to generate offspring individuals; The fitness change rate of the best individuals of two adjacent generations is calculated to obtain the population convergence speed; when the population convergence speed is less than a preset convergence threshold, the basic mutation probability is added to the mutation compensation amount to obtain the adaptive mutation probability; when the population convergence speed is greater than the preset convergence threshold, the basic mutation probability is multiplied by the attenuation function to obtain the adaptive mutation probability; The Gaussian variable step length is calculated based on the population convergence speed, the product of the Gaussian variable step length and a normally distributed random number is used as a mutation amount, and the mutation amount is applied to the offspring individual according to the adaptive mutation probability to perform Gaussian mutation.

[0085] First, initialize the population. Randomly generate a certain number of individuals, each of which represents a potential solution to the problem. These individuals constitute the initial population. For example, generate 100 individuals, each of which contains 10 genes, and the value of the gene is randomly selected between 0 and 1.

[0086] Then, evaluate the fitness of the population. Evaluate each individual in the population and calculate its fitness value. The fitness value reflects the quality of the individual's solution to the problem. The higher the fitness value of the individual, the better the quality of its solution. For example, the value of the objective function is used as the fitness value. The larger the objective function value, the higher the fitness value.

[0087] Next, dynamically adjust the selection pressure coefficient based on the population convergence. Calculate the convergence of the current population, for example, the variance of the individual fitness in the population can be used to measure it. Set a convergence interval threshold in advance. If the current population convergence is lower than the lower limit of the threshold, increase the selection pressure coefficient in the first linear increasing manner; if the current population convergence is higher than the upper limit of the threshold, reduce the selection pressure coefficient in the second linear decreasing manner; if the current population convergence is within the threshold interval, keep the selection pressure coefficient unchanged. For example, if the convergence interval threshold is [0.01, 0.1], and the current convergence is 0.005, increase the selection pressure coefficient; if the current convergence is 0.2, reduce the selection pressure coefficient; if the current convergence is 0.05, keep the selection pressure coefficient unchanged.

[0088] Perform selection operations based on the adjusted selection pressure coefficient. Calculate the selection probability of each individual. Raise the individual fitness value to the power of the selection pressure coefficient and then divide it by the total fitness value of the population to obtain the selection probability of the individual. Use the roulette strategy to select high-quality individuals. For example, if the fitness value of an individual is 10, the selection pressure coefficient is 2, and the total fitness value of the population is 1000, then the selection probability of the individual is (10^2) / 1000 = 0.1.

[0089] Adaptively calculate the crossover probability based on the diversity index. Calculate the population diversity index, for example, the average difference between individual genes in the population can be used to measure it. Multiply the compensation value of the diversity index by the adjustment coefficient to get the dynamic adjustment amount. Add the dynamic adjustment amount to the basic crossover probability to get the adaptive crossover probability. For example, if the diversity index is 0.8, the compensation value is 0.2, the adjustment coefficient is 0.5, and the basic crossover probability is 0.7, then the adaptive crossover probability is 0.7 + 0.2 *0.5 = 0.8.

[0090] Perform a crossover operation. Calculate the crossover weight coefficient based on the fitness ratio of the selected high-quality individuals. Perform an arithmetic crossover operation on the selected high-quality individuals based on the crossover weight coefficient and the adaptive crossover probability to generate offspring individuals. For example, if the fitness of the two parent individuals is 10 and 20, their crossover weight coefficients are 10 / (10+20)=0.33 and 20 / (10+20)=0.67, respectively. Assuming the adaptive crossover probability is 0.8, perform a crossover operation with a probability of 0.8.

[0091] The mutation probability is adaptively adjusted based on the convergence speed of consecutive generations. The fitness change rate of the best individuals in two adjacent generations is calculated to obtain the population convergence speed. A convergence speed threshold is set in advance. If the population convergence speed is less than the convergence threshold, the basic mutation probability is added to the mutation compensation amount to obtain the adaptive mutation probability; if the population convergence speed is greater than the convergence threshold, the basic mutation probability is multiplied by the attenuation function to obtain the adaptive mutation probability.

[0092] Perform mutation operation. Calculate Gaussian variable asynchrony length based on population convergence speed, and multiply Gaussian variable asynchrony length by a random number that obeys standard normal distribution as mutation amount. Apply mutation amount to offspring individuals according to adaptive mutation probability to perform Gaussian mutation.

[0093] Repeat steps 3 to 8 until the termination condition is met, such as reaching the maximum number of iterations or finding a satisfactory solution.

[0094] The solution of this application can: Improved the algorithm's optimization ability: By dynamically adjusting the selection pressure, crossover probability, and mutation probability, the algorithm's global search and local search capabilities can be better balanced, thereby improving the algorithm's optimization ability. Enhanced the robustness of the algorithm: By adaptively adjusting parameters, the algorithm can better adapt to the characteristics of different problems, thereby enhancing the robustness of the algorithm. Accelerated the convergence speed of the algorithm: By dynamically adjusting the mutation probability according to the population convergence speed, the convergence speed of the algorithm can be accelerated.

[0095] Figure 2 FIG. 1 is a schematic diagram of a structure of a dynamic optimization system for driving an operator chain based on a computing engine model according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to build a multi-dimensional feature perception network, which obtains operator chain operation data through a distributed collection layer, and the distributed collection layer is provided with a time series data collection unit, a load feature collection unit and a resource status collection unit; the time series data collection unit dynamically tracks the data flow characteristics between operators based on a sliding window mechanism and constructs a data flow fluctuation curve; the load feature collection unit uses an adaptive sampling algorithm to obtain the operator calculation load characteristics and generates a load distribution heat map; the resource status collection unit records the system resource occupancy status based on resource profiling technology to form a multi-dimensional resource utilization matrix; the data flow fluctuation curve, the load distribution heat map and the multi-dimensional resource utilization matrix are input into a deep learning model to train and generate an operator chain performance prediction model; The second unit is used to build a hierarchical optimization decision-making mechanism based on the operator chain performance prediction model, use the reinforcement learning algorithm to train the optimization agent model at the strategy layer, and the optimization agent model calculates the affinity coefficients of adjacent operators according to the prediction results; when the affinity coefficient is higher than the dynamic threshold, the operator automatic merging operation is triggered; at the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score; an elastic scaling unit is built at the resource layer, and the elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change in parallelism; the operator automatic merging operation, the parallelism optimization and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy; The third unit is used to deploy the end-to-end optimization strategy, and start the adaptive performance monitoring module during the execution process. The adaptive performance monitoring module continuously optimizes the monitoring indicator weights based on online learning; the performance gain gradient is calculated by the back propagation algorithm, and when the performance gain gradient shows a decay trend, the strategy tuning engine is activated; the strategy tuning engine dynamically adjusts the optimization parameters using a genetic algorithm, and synchronizes the optimization records to the knowledge graph in real time; the knowledge graph continuously improves the optimization rule base through association analysis, provides knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of the optimization strategy.

[0096] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0097] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0098] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic optimization method based on a computing engine model-driven operator chain, characterized in that: include: Construct a multi-dimensional feature perception network, which obtains operator chain operation data through a distributed collection layer, and the distributed collection layer is provided with a time series data collection unit, a load feature collection unit and a resource status collection unit; the time series data collection unit dynamically tracks the data flow characteristics between operators based on a sliding window mechanism and constructs a data flow fluctuation curve; the load feature collection unit obtains the operator calculation load characteristics using an adaptive sampling algorithm and generates a load distribution heat map; the resource status collection unit records the system resource occupancy status based on resource profiling technology to form a multi-dimensional resource utilization matrix; the data flow fluctuation curve, the load distribution heat map and the multi-dimensional resource utilization matrix are input into a deep learning model to train and generate an operator chain performance prediction model; A hierarchical optimization decision-making mechanism is constructed based on the operator chain performance prediction model. The optimization agent model is trained using a reinforcement learning algorithm at the strategy layer. The optimization agent model calculates the affinity coefficients of adjacent operators based on the prediction results. When the affinity coefficient is higher than the dynamic threshold, the automatic merging operation of the operators is triggered. At the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score. Constructing an elastic scaling unit at the resource layer, the elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change of parallelism; integrating the operator automatic merging operation, the parallelism optimization and the fine-grained resource allocation plan into an end-to-end optimization strategy; Deploy the end-to-end optimization strategy, and start an adaptive performance monitoring module during execution, wherein the adaptive performance monitoring module continuously optimizes monitoring indicator weights based on an online learning method; Calculate the performance gain gradient by using a back-propagation algorithm, and activate the strategy tuning engine when the performance gain gradient shows a decay trend; The strategy tuning engine uses genetic algorithms to dynamically adjust optimization parameters and synchronizes optimization records to the knowledge graph in real time; the knowledge graph continuously improves the optimization rule base through association analysis, provides knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of optimization strategies.

2. The method according to claim 1, characterized in that A hierarchical optimization decision-making mechanism is constructed based on the operator chain performance prediction model. The optimization agent model is trained using a reinforcement learning algorithm at the strategy layer. The optimization agent model calculates the affinity coefficients of adjacent operators according to the prediction results, including: A hierarchical optimization decision mechanism is constructed based on the operator chain performance prediction model, and the hierarchical optimization decision mechanism includes a strategy layer, an execution layer, and a resource layer; the strategy layer receives the performance prediction result output by the operator chain performance prediction model, performs principal component analysis and dimensionality reduction processing on the performance prediction result, selects the principal component whose feature contribution rate is higher than the preset contribution threshold, maps the reduced-dimensional features to the interval of zero to one through the Min-Max normalization method, and generates a standardized feature matrix; A reinforcement learning algorithm is used at the strategy layer to train an optimization agent model, wherein the optimization agent model is based on a dual Q network architecture; the standardized feature matrix is ​​input into the optimization agent model, a state space including operator computing load, data flow characteristics, and resource utilization is constructed, an action space of operator merging, splitting, and migration is defined, and a reward function based on performance gain is designed; The optimization agent model calculates affinity coefficients of adjacent operators based on training results, and the affinity coefficients are comprehensively calculated through three dimensions: data dependency intensity, resource complementarity, and performance improvement expectations.

3. The method according to claim 1, characterized in that At the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score. At the resource layer, an elastic scaling unit is built, which automatically generates a fine-grained resource allocation plan based on the change in parallelism. Integrating the operator automatic merging operation, the parallelism optimization and the fine-grained resource allocation scheme into an end-to-end optimization strategy includes: At the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and a computational complexity scoring model is constructed to analyze the computational density, memory access pattern, and data dependency of the operators to generate static feature evaluation results. The runtime processor usage, memory occupancy, and input and output throughput are monitored to generate dynamic load evaluation results. The static feature evaluation results and the dynamic load evaluation results are weighted to generate an operator complexity scoring matrix. The operator complexity scoring matrix is ​​input into a simulated annealing algorithm, the search temperature parameter is dynamically adjusted based on the degree of performance improvement, the optimal parallelism configuration is found in the search space, and local optimization is performed on the optimal parallelism configuration to obtain an optimized parallelism configuration scheme; the optimized parallelism configuration scheme is input into a time series analysis model to generate a short-term load change trend of the system, and the instance expansion and contraction threshold is set according to the short-term load change trend of the system to achieve dynamic adjustment of the number of instances; An elastic scaling unit is constructed at the resource layer. The elastic scaling unit establishes a mapping relationship model between parallelism and resource requirements according to the optimized parallelism configuration scheme and the instance scaling threshold. The mapping relationship model analyzes the operator complexity scoring matrix to identify an intensive load feature set. The intensive load feature set includes computing intensive load features, memory intensive load features, and input / output intensive load features. The intensive load feature set is input into a deep learning model to predict resource usage trends. The elastic scaling unit automatically generates a fine-grained resource allocation plan based on the resource usage trend, implements resource isolation for resource quotas, and executes resource policies. The merged operator, the optimized parallelism configuration scheme, and the fine-grained resource allocation plan are integrated into an end-to-end optimization strategy.

4. The method according to claim 1, characterized in that: Deploy the end-to-end optimization strategy, and start an adaptive performance monitoring module during execution, wherein the adaptive performance monitoring module continuously optimizes monitoring indicator weights based on an online learning method; The performance gain gradient is calculated by the back propagation algorithm. When the performance gain gradient shows a decay trend, the activation strategy tuning engine includes: Deploy the end-to-end optimization strategy, start the adaptive performance monitoring module, build a multi-layer perceptron performance prediction model based on the adaptive performance monitoring module, the input layer of the multi-layer perceptron performance prediction model receives basic monitoring indicators such as processor utilization, memory occupancy, and network throughput, the hidden layer of the multi-layer perceptron performance prediction model uses a ReLU activation function to perform nonlinear feature extraction, and the output layer of the multi-layer perceptron performance prediction model generates a system performance prediction result; The adaptive performance monitoring module uses a sliding time window mechanism to continuously collect performance data, calculates the Pearson correlation coefficient between the basic monitoring indicator and the system performance prediction result, and uses the Pearson correlation coefficient as the monitoring indicator weight to weightedly aggregate the basic monitoring indicator to obtain a weighted performance indicator; the weighted performance indicator is divided into training batches according to the time series, and the monitoring indicator weight is updated based on the stochastic gradient descent algorithm using an online learning method. The stochastic gradient descent algorithm uses L2 regularization to constrain the weight parameters, and uses the exponential moving average method to smooth the parameter update process to obtain the optimized monitoring indicator weight; Apply the optimized monitoring indicator weights to real-time performance monitoring, calculate the prediction deviation between the current system performance and the system performance prediction result, and calculate the performance difference between adjacent time windows for historical performance data by time window segmentation to obtain the performance change trend; The prediction deviation and the performance change trend are input into a back propagation algorithm, the back propagation algorithm calculates an end-to-end delay gradient and a throughput gradient, and weighted synthesizes a performance gain gradient based on the end-to-end delay gradient and the throughput gradient; the change trend of the performance gain gradient is detected, and when the performance gain gradient decays in three consecutive time windows, a policy tuning engine is activated.

5. The method according to claim 4, characterized in that The weighted performance index is divided into training batches according to the time series, and the monitoring index weight is updated based on the stochastic gradient descent algorithm using an online learning method. The stochastic gradient descent algorithm uses L2 regularization to constrain the weight parameters and uses the exponential moving average method to smooth the parameter update process. The optimized monitoring index weights include: A double-layer sliding window structure is used to divide the weighted performance index into time series, wherein the double-layer sliding window structure includes an inner time window, and each inner time window collects three hundred performance data sampling points to form a training batch; Based on the attention mechanism, the cosine distance between each sampling point in the training batch and the current system state is calculated to obtain a similarity value, and the similarity value is input into the softmax function to obtain a sample weight vector, and the sample weight vector is used to perform weighted processing on the training batch to obtain weighted training data; Inputting the weighted training data into a stochastic gradient descent algorithm, the stochastic gradient descent algorithm calculates a gradient vector of a monitoring indicator weight, and when a norm of the gradient vector exceeds a preset vector threshold, scaling the gradient vector proportionally to keep the gradient direction unchanged to obtain an adjusted gradient vector; An L2 regularization term is introduced into the loss function to constrain the update of the monitoring indicator weight, and the regularization coefficient of the L2 regularization term decays linearly with the training rounds; the adjusted gradient vector is smoothed by an exponential moving average method, and the adjusted gradient vector and the historical gradient vector are weighted averaged according to an attenuation coefficient of 0.9 to obtain a smoothed gradient vector; Based on the smooth gradient vector, the monitoring indicator weight is updated to obtain an updated monitoring indicator weight, the prediction mean square error in a plurality of consecutive inner time windows is calculated to obtain a prediction accuracy, and the change standard deviation of the updated monitoring indicator weight is calculated to obtain a weight stability index; Monitor the loss change trend of multiple consecutive training batches, increase the learning rate to accelerate convergence when the loss change trend continues to decline, and reduce the learning rate to improve stability when the loss change trend fluctuates; use the updated monitoring indicator weights and the adjusted learning rate for the next round of training iterations until the prediction accuracy and the weight stability index simultaneously meet the preset conditions, and output the optimized monitoring indicator weights.

6. The method according to claim 1, characterized in that The strategy tuning engine uses genetic algorithms to dynamically adjust optimization parameters and synchronizes optimization records to the knowledge graph in real time; The knowledge graph continuously improves the optimization rule base through association analysis, provides knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of optimization strategies, including: Construct a multi-layer knowledge graph, wherein the multi-layer knowledge graph includes an entity layer, a relationship layer, and an attribute layer; the strategy tuning engine constructs chromosome encoding using a genetic algorithm; extracts the historical optimal configuration from the multi-layer knowledge graph, and performs parameter perturbations on the historical optimal configuration to generate an initial population; constructs a fitness function using system delay, throughput, and resource utilization, calculates the fitness distribution of the initial population, and obtains population convergence and diversity indicators; Dynamically adjusting the selection pressure coefficient based on the population convergence degree, and adopting the roulette strategy to select high-quality individuals; adaptively calculating the crossover probability based on the diversity index, and performing arithmetic crossover on the selected high-quality individuals; adaptively adjusting the mutation probability based on the convergence speed of consecutive generations, and performing Gaussian mutation on the individuals after the crossover; The individuals with the best fitness in each generation and the mutant individuals form a new population; the parameter configuration, performance data, and fitness evaluation results of the new population are used as optimization records and synchronized in real time to the entity layer, the relationship layer, and the attribute layer in the multi-layer knowledge graph; Performing association analysis on the optimization records in the multi-layer knowledge graph, mining parameter configuration rules using a priori algorithms, identifying performance change laws using time series analysis, and constructing parameter adjustment paths using causal reasoning; storing the parameter configuration rules, the performance change laws, and the parameter adjustment paths as optimization strategies in a rule library; Calculate the confidence of the optimization strategy in the rule base, and mark the strategy with a confidence lower than a preset confidence threshold as a strategy to be verified; perform A / B testing in a verification environment to verify the effectiveness of the strategy to be verified; update the optimization strategy that has passed the verification to the rule base to realize dynamic optimization of the rule base; A graph neural network is used to calculate the similarity between the current state and the historical scenario, and the optimization strategy corresponding to the similar scenario is extracted from the rule base; a parameter adjustment scheme is obtained by weighted combination based on the strategy confidence; and the parameter adjustment scheme is used as prior knowledge to guide the population evolution direction of the strategy tuning engine; Monitor the performance indicators during the optimization process, and trigger parameter rollback when performance degradation is detected; record performance fluctuation scenarios to the multi-layer knowledge graph, continuously improve the rule base through continuous two-way feedback, and realize the adaptive evolution of the optimization strategy.

7. The method according to claim 6, characterized in that Dynamically adjust the selection pressure coefficient based on the population convergence degree, and select high-quality individuals using a roulette strategy; adaptively calculate the crossover probability based on the diversity index, and perform arithmetic crossover on the selected high-quality individuals; Based on the convergence speed of successive generations, the mutation probability is adaptively adjusted, and Gaussian mutation is performed on the individuals after crossover, including: Dynamically adjusting the selection pressure coefficient based on the population convergence, increasing the selection pressure coefficient according to a first linear function when the population convergence is lower than a preset interval threshold, and reducing the selection pressure coefficient according to a second linear function when the population convergence is higher than the preset interval threshold; calculating the individual selection probability by dividing the power of the selection pressure coefficient of the individual fitness value by the sum of the group fitness, and selecting high-quality individuals according to the individual selection probability using a roulette strategy; The product of the compensation value of the diversity index and the adjustment coefficient is used as a dynamic adjustment amount, and the dynamic adjustment amount is added to the basic crossover probability to obtain an adaptive crossover probability; Calculating a crossover weight coefficient based on the fitness ratio of the high-quality individuals, and performing an arithmetic crossover operation on the high-quality individuals according to the crossover weight coefficient and the adaptive crossover probability to generate offspring individuals; The fitness change rate of the best individuals of two adjacent generations is calculated to obtain the population convergence speed; when the population convergence speed is less than a preset convergence threshold, the basic mutation probability is added to the mutation compensation amount to obtain the adaptive mutation probability; when the population convergence speed is greater than the preset convergence threshold, the basic mutation probability is multiplied by the attenuation function to obtain the adaptive mutation probability; The Gaussian variable step length is calculated based on the population convergence speed, the product of the Gaussian variable step length and a normally distributed random number is used as a mutation amount, and the mutation amount is applied to the offspring individual according to the adaptive mutation probability to perform Gaussian mutation.

8. A computing engine model-based operator chain dynamic optimization system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to build a multi-dimensional feature perception network, which obtains operator chain operation data through a distributed collection layer, and the distributed collection layer is provided with a time series data collection unit, a load feature collection unit and a resource status collection unit; the time series data collection unit dynamically tracks the data flow characteristics between operators based on a sliding window mechanism and constructs a data flow fluctuation curve; the load feature collection unit uses an adaptive sampling algorithm to obtain the operator calculation load characteristics and generates a load distribution heat map; the resource status collection unit records the system resource occupancy status based on resource profiling technology to form a multi-dimensional resource utilization matrix; the data flow fluctuation curve, the load distribution heat map and the multi-dimensional resource utilization matrix are input into a deep learning model to train and generate an operator chain performance prediction model; The second unit is used to build a hierarchical optimization decision-making mechanism based on the operator chain performance prediction model, and use the reinforcement learning algorithm to train the optimization agent model at the strategy layer. The optimization agent model calculates the affinity coefficients of adjacent operators according to the prediction results; when the affinity coefficient is higher than the dynamic threshold, the automatic merging operation of the operators is triggered; at the execution layer, a heuristic algorithm is used to optimize the parallelism of the merged operators, and the number of instances is dynamically adjusted based on the computational complexity score; Constructing an elastic scaling unit at the resource layer, the elastic scaling unit automatically generates a fine-grained resource allocation plan according to the change of parallelism; integrating the operator automatic merging operation, the parallelism optimization and the fine-grained resource allocation plan into an end-to-end optimization strategy; A third unit is used to deploy the end-to-end optimization strategy and start an adaptive performance monitoring module during execution, wherein the adaptive performance monitoring module continuously optimizes the monitoring indicator weights based on an online learning method; Calculate the performance gain gradient by using a back-propagation algorithm, and activate the strategy tuning engine when the performance gain gradient shows a decay trend; The strategy tuning engine uses genetic algorithms to dynamically adjust optimization parameters and synchronizes optimization records to the knowledge graph in real time; the knowledge graph continuously improves the optimization rule base through association analysis, provides knowledge support for subsequent optimization decisions, and realizes the adaptive evolution of optimization strategies.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images