Industrial control parameter adaptive optimization method based on deep reinforcement learning

Abstract

The invention provides an industrial control parameter adaptive optimization method based on deep reinforcement learning, and relates to the technical field of industrial control, and the method comprises the steps: collecting and preprocessing system operation data, dividing steady-state and fluctuation working conditions according to disturbance characteristics, and building a corresponding sub-network; adaptively switching the network output weight based on the disturbance degree to generate an initial optimization scheme; experience playback is carried out by adopting a self-adaptive parallel sampling strategy; training a deep reinforcement learning model and updating parameters; and the calculation performance improvement value is converted into a reward value to be compared with a dynamic threshold value. According to the invention, adaptive optimization of industrial control parameters can be realized, and the robustness and control precision of the system to different working conditions are improved.

Description

Technical Field

[0001] The present invention relates to the field of industrial control technology, and in particular to an industrial control parameter adaptive optimization method based on deep reinforcement learning. Background Art

[0002] Industrial control systems are a core component of modern industrial production. The proper configuration of control parameters directly impacts system efficiency, product quality, and energy consumption. Traditional industrial control parameter optimization relies primarily on manual experience or fixed rules. However, facing complex and changing industrial environments and uncertainties, achieving optimal control results is difficult. With the development of artificial intelligence (AI) technology, deep reinforcement learning (DL) has shown great potential in the industrial control field due to its ability to continuously learn and optimize decisions through interaction with the environment.

[0003] At present, industrial control parameter optimization has the following defects and deficiencies: Existing industrial control parameter optimization methods lack the ability to effectively distinguish and process different operating conditions. They often use a single model to deal with all operating conditions and are unable to perform differentiated optimization based on the characteristics of steady-state and fluctuating operating conditions, resulting in poor control effects when the system state changes frequently.

[0004] The traditional experience replay mechanism adopts a uniform sampling strategy for all samples, ignoring the value differences between different samples and failing to effectively utilize high-quality experience data, resulting in inefficient model learning and difficulty in quickly converging to the optimal control strategy.

[0005] When existing deep reinforcement learning is applied to industrial control, the reward function design is relatively simple, which cannot comprehensively evaluate the multi-dimensional performance indicators of the system. The lack of a dynamic evaluation mechanism leads to a single optimization goal and makes it difficult to achieve balanced optimization in multiple aspects such as system stability, response speed and energy efficiency. Summary of the Invention

[0006] The embodiments of the present invention provide an industrial control parameter adaptive optimization method based on deep reinforcement learning, which can solve the problems in the prior art.

[0007] A first aspect of an embodiment of the present invention provides an industrial control parameter adaptive optimization method based on deep reinforcement learning, comprising: Collect real-time operation data of industrial control systems and pre-process them to obtain standardized system state feature vectors; Dividing the system state characteristic vector into steady-state operating condition data and fluctuating operating condition data according to the system disturbance characteristics, and establishing a steady-state operating condition subnetwork and a fluctuating operating condition subnetwork respectively; Inputting the current system state eigenvector into the steady-state operating condition subnetwork and the fluctuating operating condition subnetwork, adaptively switching the output weights of the subnetwork based on the system disturbance degree, and generating an initial control parameter optimization scheme; Adopting an adaptive parallel sampling strategy for experience replay, in each training batch, the empirical data is divided into a high-performance sample group and a sample group to be optimized according to the system performance index, the high-performance sample group is oversampled, and the sample group to be optimized is undersampled. Based on the sampled empirical data, the initial control parameter optimization scheme is adjusted; The adjusted initial control parameters are input into the deep reinforcement learning model for training, the model parameters are updated, and an optimized control parameter combination is generated and applied to the industrial control system. The system status data after application is collected, and the comprehensive performance improvement value of the system status data is calculated and converted into a cumulative reward value based on a Gaussian mixture model. The value is compared with the dynamic grading threshold to determine the performance level of the current experience data.

[0008] According to the system disturbance characteristics, the system state feature vector is divided into steady-state working condition data and fluctuating working condition data, and the steady-state working condition subnetwork and the fluctuating working condition subnetwork are respectively established, including: Calculate the variance of each dimension feature and its mean value of the system state feature vector within the preset time window to obtain the volatility value; Obtaining an initial classification threshold from historical data statistics, and obtaining an updated classification threshold by taking a weighted average of the initial classification threshold and the average volatility of the current batch of data; When the fluctuation value or the absolute value of the fluctuation value is greater than the updated classification threshold, the corresponding system state feature vector is classified as fluctuating operating condition data; when the fluctuation value and the absolute value of the fluctuation value are less than the updated classification threshold, the corresponding system state feature vector is classified as steady-state operating condition data; Based on the fluctuating operating condition data and the steady-state operating condition data, a fluctuating operating condition subnetwork and a steady-state operating condition subnetwork are respectively established. Both subnetworks adopt a three-layer structure and have the same input and output dimensions. The number of hidden layer nodes of the fluctuating operating condition subnetwork is twice the number of input layer nodes and a preset basic learning rate is adopted. The parameter adjustment range is set to the preset basic adjustment range. The number of hidden layer nodes of the steady-state operating condition subnetwork is half the number of input layer nodes and a preset proportion of the preset basic learning rate is adopted. The parameter adjustment range is set to a preset proportion of the preset basic adjustment range.

[0009] Adaptively switch the output weights of the sub-network based on the degree of system disturbance, including: The disturbance characteristics are calculated based on the system state quantity, and each disturbance characteristic is multiplied by the corresponding characteristic weight coefficient and the square root of the sum of squares is calculated to obtain the comprehensive disturbance index; The disturbance threshold is obtained by calculating the sum of the mean and standard deviation of the comprehensive disturbance index within a preset time period; the disturbance threshold is updated using an exponential smoothing method, and the disturbance threshold at the previous moment and the maximum value of the comprehensive disturbance index within the preset time window are multiplied by the first smoothing coefficient and the second smoothing coefficient respectively, and the sum is used to obtain the disturbance threshold at the current moment; Dividing the comprehensive disturbance index by the disturbance threshold at the current moment to obtain a normalized disturbance index; Subtracting a preset central value from the normalized disturbance index, multiplying the difference by the sensitivity coefficient, taking the negative exponent of the product and adding 1 to obtain a disturbance response coefficient, dividing 1 by the disturbance response coefficient to obtain the output weight of the fluctuating working condition subnetwork, and subtracting the output weight of the fluctuating working condition subnetwork from 1 to obtain the output weight of the steady-state working condition subnetwork; The output control parameter of the fluctuating operating condition subnetwork is multiplied by the output weight of the fluctuating operating condition subnetwork to obtain a first weighted parameter, the output control parameter of the steady-state operating condition subnetwork is multiplied by the output weight of the steady-state operating condition subnetwork to obtain a second weighted parameter, and the first weighted parameter and the second weighted parameter are added to obtain an initial control parameter optimization scheme.

[0010] Adopting an adaptive parallel sampling strategy for experience replay, in each training batch, the experience data is divided into a high-performance sample group and a sample group to be optimized according to the system performance index, the high-performance sample group is oversampled, and the sample group to be optimized is undersampled. Based on the sampled experience data, the initial control parameter optimization scheme is adjusted, including: Calculate the multidimensional performance indicators of the empirical data to obtain the comprehensive performance score of the empirical data; Calculate the average performance score and the standard deviation of the performance score of the current batch of experience data, and use the sum of the average performance score and a preset multiple of the standard deviation of the performance score as the performance grouping threshold; The empirical data with a comprehensive performance score greater than the performance grouping threshold are divided into a high-performance sample group, and the empirical data with a comprehensive performance score less than the performance grouping threshold are divided into a sample group to be optimized; Calculating an oversampling ratio based on the comprehensive performance scores of the empirical data in the high-performance sample group, and calculating an undersampling ratio based on the comprehensive performance scores of the empirical data in the sample group to be optimized; Repeating sampling of the high-performance sample group according to the oversampling ratio, screening and sampling the sample group to be optimized according to the undersampling ratio, multiplying the control parameters in the high-performance sample group by their corresponding oversampling ratios to obtain high-performance weighted parameters, and multiplying the control parameters in the sample group to be optimized by their corresponding undersampling ratios to obtain weighted parameters to be optimized; The high-performance weighted parameter and the weighted parameter to be optimized are summed and divided by the total number of samples to obtain a parameter adjustment amount, and the initial control parameter optimization plan and the parameter adjustment amount are added to obtain an adjusted control parameter optimization plan.

[0011] The comprehensive performance improvement value of the system status data is calculated and converted into a cumulative reward value based on the Gaussian mixture model. The performance level of the current experience data is determined by comparing it with the dynamic classification threshold. Calculate the ratio of the standard deviation of the system state feature to the preset maximum standard deviation to obtain the feature confidence. Multiply the historical weight of each feature by the corresponding feature confidence and normalize them to obtain the feature weight at the current moment. Divide the average control error, average response time, and average energy consumption of the system state data by the corresponding maximum allowable values, multiply them by the feature weight at the current moment, and sum them to obtain the basic performance score. Calculating the improvement ratio of the basic performance score at multiple preset time scales respectively, multiplying the performance improvement ratio of each time scale by the corresponding scale weight coefficient and summing them to obtain a comprehensive performance improvement value; Calculating the probability distribution of the comprehensive performance improvement value based on a Gaussian mixture model and multiplying it by the proportional coefficient of the corresponding probability distribution value to obtain an immediate reward value; multiplying the immediate reward value at each moment by the power of the discount factor at the corresponding moment and summing the results to obtain a cumulative reward value; Calculate the rate of change of the number of samples at each performance level, multiply the discrete degree of the rate of change by a preset benchmark adjustment coefficient, and add 1 to obtain the adaptive adjustment coefficient at the current moment; calculate the classification threshold based on the mean of the cumulative reward value, the standard deviation of the cumulative reward value, and the adaptive adjustment coefficient; The posterior probability of each performance level corresponding to the cumulative reward value is calculated, and the performance level of the current experience data is determined based on the comparison relationship between the posterior probability of the performance level and the classification threshold.

[0012] Calculating the probability distribution of the comprehensive performance improvement value based on the Gaussian mixture model includes: Calculate the initial values ​​of the mixing weight, mean, and variance parameters of each Gaussian component and substitute them into the Gaussian distribution function to obtain the probability density of each Gaussian component. Multiply the probability density of each Gaussian component by the corresponding mixing weight and sum them to obtain the probability density function of the comprehensive performance improvement value. Calculate the conditional probability of the comprehensive performance improvement value under each Gaussian component, multiply the conditional probability by the corresponding mixing weight and normalize it to obtain the posterior probability of the performance distribution, and substitute the posterior probability of the performance distribution into the probability density function to obtain a modified probability density function; Calculating the time difference between the current moment and the historical moment, substituting the time difference into an exponential function to obtain a time decay factor, and multiplying the time decay factor by the corrected probability density function to obtain a time-weighted probability density function; The time-weighted probability density function is integrated to obtain a cumulative distribution function, and a derivative of the cumulative distribution function is used as the probability distribution of the comprehensive performance improvement value.

[0013] According to a second aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0014] According to a third aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0015] The beneficial effects of this application are as follows: The present invention achieves adaptive optimization of industrial control parameters through deep reinforcement learning. It can intelligently switch between steady-state and fluctuating operating condition sub-networks according to the disturbance characteristics of the industrial system, achieve precise control of different operating conditions, improve robustness and adaptability, and solve the problem that traditional control methods are difficult to maintain stability in complex disturbance environments.

[0016] An innovative adaptive parallel sampling strategy is adopted for experience replay, and oversampling and undersampling are implemented on high-performance samples and samples to be optimized respectively, which significantly improves the model training efficiency and optimization effect, accelerates the convergence process of control parameters, and effectively solves the problem of low training efficiency caused by sample imbalance in industrial control.

[0017] Based on the Gaussian mixture model, the performance improvement value is converted into a cumulative reward value, and the performance level of the empirical data is evaluated through a dynamic grading threshold, which achieves accurate quantification and evaluation of the control effect, making the optimization process more transparent and explainable. At the same time, it significantly improves the overall control accuracy and energy utilization efficiency, providing a more intelligent and efficient control solution for industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the process of the method for adaptive optimization of industrial control parameters based on deep reinforcement learning according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the performance comparison of adaptive parallel sampling strategies; Figure 3Schematic diagram of the comparison of the probability distribution accuracy of performance improvement values. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] Figure 1 FIG. 1 is a flow chart of an industrial control parameter adaptive optimization method based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes: Collect real-time operation data of industrial control systems and pre-process them to obtain standardized system state feature vectors; Dividing the system state characteristic vector into steady-state operating condition data and fluctuating operating condition data according to the system disturbance characteristics, and establishing a steady-state operating condition subnetwork and a fluctuating operating condition subnetwork respectively; Inputting the current system state eigenvector into the steady-state operating condition subnetwork and the fluctuating operating condition subnetwork, adaptively switching the output weights of the subnetwork based on the system disturbance degree, and generating an initial control parameter optimization scheme; Adopting an adaptive parallel sampling strategy for experience replay, in each training batch, the empirical data is divided into a high-performance sample group and a sample group to be optimized according to the system performance index, the high-performance sample group is oversampled, and the sample group to be optimized is undersampled. Based on the sampled empirical data, the initial control parameter optimization scheme is adjusted; Inputting the adjusted initial control parameters into a deep reinforcement learning model for training, updating the model parameters, generating an optimized control parameter combination and applying it to the industrial control system, collecting system status data after application, calculating a comprehensive performance improvement value of the system status data and converting it into a cumulative reward value based on a Gaussian mixture model, and comparing it with a dynamic grading threshold to determine the performance level of the current experience data; The system state data after application, the optimized control parameter combination and its performance level are stored in the experience replay pool for use in the next round of training optimization to achieve adaptive parameter optimization based on working condition characteristics.

[0022] In an optional embodiment, the system state feature vector is divided into steady-state operating condition data and fluctuating operating condition data according to the system disturbance characteristics, and the steady-state operating condition subnetwork and the fluctuating operating condition subnetwork are respectively established, including: Calculate the variance of each dimension feature and its mean value of the system state feature vector within the preset time window to obtain the volatility value; Obtaining an initial classification threshold from historical data statistics, and obtaining an updated classification threshold by taking a weighted average of the initial classification threshold and the average volatility of the current batch of data; When the fluctuation value or the absolute value of the fluctuation value is greater than the updated classification threshold, the corresponding system state feature vector is classified as fluctuating operating condition data; when the fluctuation value and the absolute value of the fluctuation value are less than the updated classification threshold, the corresponding system state feature vector is classified as steady-state operating condition data; Based on the fluctuating operating condition data and the steady-state operating condition data, a fluctuating operating condition subnetwork and a steady-state operating condition subnetwork are respectively established. Both subnetworks adopt a three-layer structure and have the same input and output dimensions. The number of hidden layer nodes of the fluctuating operating condition subnetwork is twice the number of input layer nodes and a preset basic learning rate is adopted. The parameter adjustment range is set to the preset basic adjustment range. The number of hidden layer nodes of the steady-state operating condition subnetwork is half the number of input layer nodes and a preset proportion of the preset basic learning rate is adopted. The parameter adjustment range is set to a preset proportion of the preset basic adjustment range.

[0023] This embodiment provides a method for classifying operating conditions based on system disturbance characteristics. The method divides the system state characteristic vector into steady-state operating condition data and fluctuating operating condition data according to the degree of fluctuation, and constructs corresponding neural network sub-networks for each.

[0024] In this embodiment, it is first necessary to calculate the volatility value of the system state feature vector within a preset time window. Specifically, for a system state feature vector containing multidimensional features, a preset time window is selected, such as 30 seconds or 60 seconds, and the variance of each dimensional feature and its mean within the time window is calculated. Taking temperature sensor data as an example, assuming that 60 temperature data points are collected within a 60-second time window, the average value of these 60 data points is calculated, and then the square of the difference between each data point and the average value is calculated. Finally, the squares of all differences are added and divided by 60 to obtain the variance of the temperature feature, that is, the volatility value. For multidimensional features, the volatility value of each dimension can be calculated separately, or the average or weighted average of the volatility values ​​of all dimensions can be calculated as the overall volatility value.

[0025] Next, an initial classification threshold is obtained from historical data statistics. This initial threshold can be determined by analyzing the known characteristic volatility distributions of steady-state and fluctuating conditions in the historical operating data. For example, by analyzing the operating data of the past three months, it was found that the average volatility value under steady-state conditions was 0.05, and the average volatility value under fluctuating conditions was 0.25. The initial classification threshold can be set to 0.15. In order to adapt to the dynamic changes of the system, the classification threshold needs to be updated. Specifically, the average volatility of the current batch data is weighted averaged with the initial classification threshold to obtain the updated classification threshold. Assuming the initial classification threshold is 0.15, the average volatility of the current batch data is 0.20, and the weighting coefficient is set to 0.3, the updated classification threshold is 0.15×(1-0.3)+0.20×0.3=0.165. This dynamic adjustment mechanism enables the classification threshold to be adaptively adjusted as the system operating state changes.

[0026] The system state eigenvector is then classified based on the calculated volatility value and the updated classification threshold. When the volatility value or its absolute value is greater than the updated classification threshold, the corresponding system state eigenvector is classified as fluctuating operating condition data. When both the volatility value and its absolute value are less than the updated classification threshold, the corresponding system state eigenvector is classified as steady-state operating condition data. For example, if the volatility value of a sensor in a time window is 0.18 and the updated classification threshold is 0.165, since 0.18 > 0.165, the system state eigenvector in that time window is classified as fluctuating operating condition data.

[0027] After data classification, a fluctuating operating condition subnetwork and a steady-state operating condition subnetwork are established based on the fluctuating operating condition data and the steady-state operating condition data, respectively. Both subnetworks adopt a three-layer structure, consisting of an input layer, a hidden layer, and an output layer, with the same input and output dimensions to ensure that the same input produces outputs of the same dimensions. Specifically, for the fluctuating operating condition subnetwork, the number of hidden layer nodes is set to twice the number of input layer nodes. For example, if the input layer has 10 nodes (i.e., the system state eigenvector has 10 dimensions), the hidden layer is set to 20 nodes. The fluctuating operating condition subnetwork uses a preset base learning rate, such as 0.01, and the parameter adjustment range is set to a preset base adjustment range, such as [-0.5, 0.5]. For the steady-state operating condition subnetwork, the number of hidden layer nodes is set to half the number of input layer nodes. If the input layer has 10 nodes, the hidden layer is set to 5 nodes. The steady-state subnetwork uses a preset ratio of the preset base learning rate. For example, if the preset base learning rate is 0.01 and the preset ratio is 0.5, the learning rate of the steady-state subnetwork is 0.005. At the same time, the parameter adjustment range of the steady-state subnetwork is set to a preset ratio of the preset base adjustment range. For example, if the preset base adjustment range is [-0.5, 0.5] and the preset ratio is 0.5, the parameter adjustment range of the steady-state subnetwork is [-0.25, 0.25].

[0028] Through this configuration, the fluctuating operating condition subnetwork has more hidden nodes, a larger learning rate, and a larger parameter adjustment range, making it suitable for processing highly volatile data. Meanwhile, the steady-state operating condition subnetwork has fewer hidden nodes, a smaller learning rate, and a smaller parameter adjustment range, making it suitable for processing less volatile steady-state data. This differentiated design enables the entire system to more accurately model and predict different types of operating condition data.

[0029] In practical applications, parameters such as the time window size, initial classification threshold, weighting coefficient, preset base learning rate, preset base adjustment range, and preset ratio can be adjusted based on the specific characteristics and requirements of the system to achieve optimal classification and modeling results. For example, for systems with slow temperature changes, a larger time window can be selected; for systems with fast response, a smaller time window can be selected. This operating condition classification method based on system disturbance characteristics can effectively improve the accuracy and robustness of system modeling, providing a more reliable foundation for subsequent system control and optimization.

[0030] In an optional embodiment, adaptively switching the output weight of the sub-network based on the system disturbance degree includes: The disturbance characteristics are calculated based on the system state quantity, and each disturbance characteristic is multiplied by the corresponding characteristic weight coefficient and the square root of the sum of squares is calculated to obtain the comprehensive disturbance index; The disturbance threshold is obtained by calculating the sum of the mean and standard deviation of the comprehensive disturbance index within a preset time period; the disturbance threshold is updated using an exponential smoothing method, and the disturbance threshold at the previous moment and the maximum value of the comprehensive disturbance index within the preset time window are multiplied by the first smoothing coefficient and the second smoothing coefficient respectively, and the sum is used to obtain the disturbance threshold at the current moment; Dividing the comprehensive disturbance index by the disturbance threshold at the current moment to obtain a normalized disturbance index; Subtracting a preset central value from the normalized disturbance index, multiplying the difference by the sensitivity coefficient, taking the negative exponent of the product and adding 1 to obtain a disturbance response coefficient, dividing 1 by the disturbance response coefficient to obtain the output weight of the fluctuating working condition subnetwork, and subtracting the output weight of the fluctuating working condition subnetwork from 1 to obtain the output weight of the steady-state working condition subnetwork; The output control parameter of the fluctuating operating condition subnetwork is multiplied by the output weight of the fluctuating operating condition subnetwork to obtain a first weighted parameter, the output control parameter of the steady-state operating condition subnetwork is multiplied by the output weight of the steady-state operating condition subnetwork to obtain a second weighted parameter, and the first weighted parameter and the second weighted parameter are added to obtain an initial control parameter optimization scheme.

[0031] The present invention provides a method for adaptively switching the output weights of sub-networks based on the degree of system disturbance, and the specific implementation method is described in detail below.

[0032] During operation, the system state quantities are first acquired, including but not limited to physical quantities such as velocity, acceleration, angular velocity, and torque. Disturbance signatures are then calculated based on the acquired system state quantities. Disturbance signatures can be extracted through methods such as the rate of change, amplitude of fluctuation, or frequency characteristics of the system state quantities. For example, the velocity fluctuation rate, acceleration change rate, and attitude angle deviation of a mobile system can all serve as disturbance signatures. For a specific flight control system, its disturbance signatures may include multiple indicators such as a velocity fluctuation rate of 0.5 m / s², an attitude angle deviation of 2.3°, and a control input change rate of 0.8 rad / s.

[0033] Each disturbance feature is multiplied by its corresponding feature weight coefficient. The weight coefficient reflects the importance of each disturbance feature. For example, the weight coefficient of velocity fluctuation is 0.7, the weight coefficient of attitude angle deviation is 0.5, and the weight coefficient of control input change rate is 0.3. The square root of the product is squared and calculated to obtain the comprehensive disturbance index. Specifically, if the velocity fluctuation rate is 0.5m / s², the attitude angle deviation is 2.3°, and the control input change rate is 0.8rad / s, then the comprehensive disturbance index is calculated as the square root of [(0.5×0.7)²+(2.3×0.5)²+(0.8×0.3)²], which is 1.27.

[0034] To achieve adaptive adjustment, a disturbance threshold needs to be calculated. First, a preset time period, such as 10 seconds, is determined during which samples of the comprehensive disturbance index are collected. The mean and standard deviation of these samples are calculated and summed to obtain the initial disturbance threshold. For example, if the mean of the comprehensive disturbance index samples collected over 10 seconds is 1.2 and the standard deviation is 0.3, the initial disturbance threshold is 1.5.

[0035] The disturbance threshold is then continuously updated using exponential smoothing. The disturbance threshold at the previous moment is multiplied by the first smoothing coefficient, and the maximum value of the comprehensive disturbance index within the preset time window is multiplied by the second smoothing coefficient. The sum of the two is the disturbance threshold at the current moment. The sum of the first smoothing coefficient and the second smoothing coefficient is 1, and can be set to 0.7 and 0.3 respectively. For example, if the disturbance threshold at the previous moment is 1.5 and the maximum value of the comprehensive disturbance index in the current time window is 1.8, then the disturbance threshold at the current moment is 1.5×0.7+1.8×0.3=1.59.

[0036] Divide the comprehensive disturbance index by the current disturbance threshold to obtain the normalized disturbance index. This step makes the disturbance measurements under different operating conditions comparable. For example, if the current comprehensive disturbance index is 1.27 and the current disturbance threshold is 1.59, the normalized disturbance index is 1.27 ÷ 1.59 = 0.8.

[0037] Subtract the preset center value from the normalized disturbance index. The preset center value can be set to 0.5, indicating that the system is at a medium disturbance level. Multiply the difference by the sensitivity coefficient. The sensitivity coefficient can be set to 2.5 to adjust the steepness of the response curve. Take the negative exponent of the product and add 1 to get the disturbance response coefficient. For example, if the normalized disturbance index is 0.8, the preset center value is 0.5, and the sensitivity coefficient is 2.5, then the disturbance response coefficient is e (-(0.8-0.5)×2.5) +1=1.47.

[0038] Divide 1 by the disturbance response coefficient to get the output weight of the fluctuating subnetwork. For example, if the disturbance response coefficient is 1.47, the output weight of the fluctuating subnetwork is 1 ÷ 1.47 = 0.68. Subtract the output weight of the fluctuating subnetwork from 1 to get the output weight of the steady-state subnetwork: 1 - 0.68 = 0.32.

[0039] Two subnetworks are maintained simultaneously: a fluctuating operating condition subnetwork and a steady-state operating condition subnetwork. The fluctuating operating condition subnetwork excels at handling conditions with large system disturbances, while the steady-state subnetwork optimizes for stable operating conditions with small disturbances. Each subnetwork outputs its own control parameters, such as the proportional, integral, and differential coefficients of a PID controller, or other control algorithm parameters.

[0040] Multiply the output control parameters of the fluctuating subnetwork by its output weight to obtain the first weighted parameter. For example, if the output control parameters of the fluctuating subnetwork are [5.2, 0.8, 1.5] and its output weight is 0.68, the first weighted parameter is [3.54, 0.54, 1.02].

[0041] Multiply the output control parameters of the steady-state subnetwork by its output weight to obtain the second weighted parameter. For example, if the output control parameters of the steady-state subnetwork are [4.0, 1.2, 0.9] and its output weight is 0.32, the second weighted parameter is [1.28, 0.38, 0.29].

[0042] Add the first and second weighting parameters to obtain the initial control parameter optimization solution. For example, if the first weighting parameter is [3.54, 0.54, 1.02] and the second weighting parameter is [1.28, 0.38, 0.29], the initial control parameter optimization solution is [4.82, 0.92, 1.31].

[0043] The control system ultimately adopted this initial control parameter optimization scheme as the actual control parameters, enabling adaptive switching of subnetwork weights based on the system's disturbance level, maintaining good control performance under varying disturbance levels. This method is particularly suitable for complex systems that require switching control strategies under diverse operating conditions, such as aircraft and robotics control.

[0044] In an optional embodiment, an adaptive parallel sampling strategy is used for experience replay. In each training batch, the experience data is divided into a high-performance sample group and a sample group to be optimized according to the system performance index. The high-performance sample group is oversampled, and the sample group to be optimized is undersampled. Adjusting the initial control parameter optimization scheme based on the sampled experience data includes: Calculate the multidimensional performance indicators of the empirical data to obtain the comprehensive performance score of the empirical data; Calculate the average performance score and the standard deviation of the performance score of the current batch of experience data, and use the sum of the average performance score and a preset multiple of the standard deviation of the performance score as the performance grouping threshold; The empirical data with a comprehensive performance score greater than the performance grouping threshold are divided into a high-performance sample group, and the empirical data with a comprehensive performance score less than the performance grouping threshold are divided into a sample group to be optimized; Calculating an oversampling ratio based on the comprehensive performance scores of the empirical data in the high-performance sample group, and calculating an undersampling ratio based on the comprehensive performance scores of the empirical data in the sample group to be optimized; Repeating sampling of the high-performance sample group according to the oversampling ratio, screening and sampling the sample group to be optimized according to the undersampling ratio, multiplying the control parameters in the high-performance sample group by their corresponding oversampling ratios to obtain high-performance weighted parameters, and multiplying the control parameters in the sample group to be optimized by their corresponding undersampling ratios to obtain weighted parameters to be optimized; The high-performance weighted parameter and the weighted parameter to be optimized are summed and divided by the total number of samples to obtain a parameter adjustment amount, and the initial control parameter optimization plan and the parameter adjustment amount are added to obtain an adjusted control parameter optimization plan.

[0045] In this embodiment, a method for experience replay using an adaptive parallel sampling strategy is proposed, which optimizes the control parameter scheme by dynamically adjusting sample weights. The specific implementation process of this method is described in detail below.

[0046] First, empirical data is collected. This data includes multi-dimensional performance indicators and corresponding control parameters. This empirical data can come from historical system operation records, such as the performance of an industrial control system under different parameter configurations. Multi-dimensional performance indicators can include response time, stability, and energy efficiency, reflecting the operating results under specific control parameters.

[0047] For each empirical data point, calculate its comprehensive performance score. This can be achieved by weighting and summing multiple performance indicators. For example, suppose an empirical data point has three performance indicators: response time of 15 milliseconds (normalized value of 0.85), stability of 92% (normalized value of 0.92), and energy efficiency ratio of 0.78 (normalized value of 0.78). The weights can be set to 0.4, 0.4, and 0.2, respectively. The calculated comprehensive performance score for this data point is 0.85 × 0.4 + 0.92 × 0.4 + 0.78 × 0.2 = 0.87. A similar calculation is performed for all empirical data points to obtain the comprehensive performance score for each data point.

[0048] For each training batch, calculate the average performance score and standard deviation of the current batch's empirical data. Suppose a batch contains 100 empirical data points, with an average performance score of 0.75 and a standard deviation of 0.08. Set a preset multiple, such as 1.0, and use the sum of the average performance score and the preset multiple of the performance score standard deviation as the performance grouping threshold: 0.75 + 1.0 × 0.08 = 0.83. This threshold is used to distinguish high-performing samples from those to be optimized.

[0049] Based on the calculated performance grouping threshold, empirical data with a comprehensive performance score greater than the threshold is classified as a high-performance sample group, and empirical data with a comprehensive performance score less than the threshold is classified as a sample group to be optimized. In the above example, data points with a comprehensive performance score greater than 0.83 are classified as high-performance sample groups, and data points with a comprehensive performance score less than 0.83 are classified as sample groups to be optimized.

[0050] Next, calculate the sampling ratio. For each data point in the high-performance sample group, calculate the oversampling ratio based on its overall performance score. This can be calculated by dividing the overall performance score of the data point by the average performance score of the high-performance sample group, and then multiplying by a coefficient. For example, if a high-performance data point has an overall performance score of 0.90 and the average performance score of the high-performance sample group is 0.85, and the coefficient is set to 1.2, then its oversampling ratio is 0.90 ÷ 0.85 × 1.2 = 1.27.

[0051] For each data point in the sample group to be optimized, the undersampling factor is calculated based on its overall performance score. This can be calculated by dividing the average performance score of the sample group to be optimized by the overall performance score of the data point, and then multiplying the result by a coefficient. For example, if the overall performance score of a data point to be optimized is 0.65, the average performance score of the sample group to be optimized is 0.72, and the coefficient is set to 0.8, then the undersampling factor is 0.72 ÷ 0.65 × 0.8 = 0.89.

[0052] Based on the calculated sampling ratio, the high-performance sample group is repeatedly sampled, while the optimized sample group is screened. Oversampling refers to repeating a data point multiple times in the training set, with the number of times determined by the oversampling ratio. Screening refers to retaining or discarding a data point with a certain probability, with the probability determined by the undersampling ratio.

[0053] Multiply the control parameters in the high-performance sample group by their corresponding oversampling ratio to obtain the high-performance weighted parameters. For example, if the control parameters of a high-performance data point are [120, 45, 0.8] and its oversampling ratio is 1.27, then its high-performance weighted parameters are [152.4, 57.15, 1.016].

[0054] Similarly, multiply the control parameters in the sample group to be optimized by their corresponding undersampling ratios to obtain the weighted parameters to be optimized. For example, if the control parameters of a data point to be optimized are [100, 30, 0.5] and its undersampling ratio is 0.89, then its weighted parameters to be optimized are [89, 26.7, 0.445].

[0055] The parameter adjustment amount is calculated by summing the high-performance weighted parameters and the weighted parameters to be optimized and dividing by the total number of samples. Assuming the sum of the high-performance weighted parameters is [1524, 571.5, 10.16], the sum of the weighted parameters to be optimized is [890, 267, 4.45], and the total number of samples is 30, then the parameter adjustment amount is [(1524 + 890) ÷ 30, (571.5 + 267) ÷ 30, (10.16 + 4.45) ÷ 30] = [80.47, 27.95, 0.49].

[0056] Finally, the adjusted control parameter optimization solution is obtained by adding the initial control parameter optimization solution and the parameter adjustment amount. For example, if the initial control parameter optimization solution is [105, 35, 0.65] and the parameter adjustment amount is [80.47, 27.95, 0.49], the adjusted control parameter optimization solution is [185.47, 62.95, 1.14].

[0057] This adaptive parallel sampling strategy effectively improves learning efficiency for high-performance samples while avoiding over-optimization of low-performance samples, thereby maintaining good performance under various operating conditions. In actual applications, parameters such as the preset multiple and sampling coefficient can be adjusted according to specific scenarios to achieve better optimization results.

[0058] Figure 2 This is a schematic diagram of the performance comparison of adaptive parallel sampling strategies: The figure shows the comparison results of the present invention with two mainstream technologies, the equal-weighted sampling method and the weighted replay memory network, on five key performance indicators. As can be seen from the figure, the present invention shows obvious advantages in all evaluation dimensions, especially in the response time optimization rate, which reaches 45.8%, much higher than the 32.1% and 38.5% of the comparison scheme. In terms of stability improvement rate, the present invention reaches 28.9%, which is nearly 10 percentage points higher than the equal-weighted sampling method. In terms of energy consumption reduction ratio, the present invention's 33.7% is also ahead of other methods. The most significant is the convergence speed improvement index, which reaches 39.5% for the present invention, indicating that the strategy can achieve parameter optimization faster. In terms of overall performance improvement dimension, the 36.2% comprehensive score of the present invention fully demonstrates its technical advantages in industrial control parameter optimization. By dynamically adjusting the oversampling ratio of high-performance samples and the undersampling ratio of samples to be optimized, the present invention achieves more efficient utilization of empirical data, thereby achieving comprehensive leadership in various performance indicators.

[0059] In an optional embodiment, the comprehensive performance improvement value of the system status data is calculated and converted into a cumulative reward value based on a Gaussian mixture model, and compared with the dynamic grading threshold to determine the performance level of the current experience data. Calculate the ratio of the standard deviation of the system state feature to the preset maximum standard deviation to obtain the feature confidence. Multiply the historical weight of each feature by the corresponding feature confidence and normalize them to obtain the feature weight at the current moment. Divide the average control error, average response time, and average energy consumption of the system state data by the corresponding maximum allowable values, multiply them by the feature weight at the current moment, and sum them to obtain the basic performance score. Calculating the improvement ratio of the basic performance score at multiple preset time scales respectively, multiplying the performance improvement ratio of each time scale by the corresponding scale weight coefficient and summing them to obtain a comprehensive performance improvement value; Calculating the probability distribution of the comprehensive performance improvement value based on a Gaussian mixture model and multiplying it by the proportional coefficient of the corresponding probability distribution value to obtain an immediate reward value; multiplying the immediate reward value at each moment by the power of the discount factor at the corresponding moment and summing the results to obtain a cumulative reward value; Calculate the rate of change of the number of samples at each performance level, multiply the discrete degree of the rate of change by a preset benchmark adjustment coefficient, and add 1 to obtain the adaptive adjustment coefficient at the current moment; calculate the classification threshold based on the mean of the cumulative reward value, the standard deviation of the cumulative reward value, and the adaptive adjustment coefficient; The posterior probability of each performance level corresponding to the cumulative reward value is calculated, and the performance level of the current experience data is determined based on the comparison relationship between the posterior probability of the performance level and the classification threshold.

[0060] The present invention provides a method for determining the performance level of system status data based on multivariate feature weighting, which calculates the comprehensive performance improvement value and converts it into a cumulative reward value based on a Gaussian mixture model, and then determines the performance level by comparing it with a dynamic grading threshold.

[0061] During implementation, we first acquire state data containing multidimensional features such as control error, response time, and energy consumption. For each feature, we calculate its standard deviation and compare it with the preset maximum standard deviation to determine the feature confidence level. For example, if the current standard deviation of a control error feature is 0.08 and the preset maximum standard deviation is 0.2, the confidence level for this feature is 0.6 (i.e., 1-0.08 / 0.2). Similarly, the confidence levels for the response time feature and the energy consumption feature are calculated to be 0.75 and 0.82, respectively.

[0062] After calculating the feature confidence, multiply each feature's historical weight by its corresponding feature confidence. Assuming the historical weights for control error, response time, and energy consumption are 0.4, 0.35, and 0.25, respectively, the weighted values ​​are 0.24, 0.2625, and 0.205, respectively. Normalizing these values ​​yields the current feature weights of 0.34, 0.37, and 0.29, respectively.

[0063] The basic performance score is calculated by dividing the average control error, average response time, and average energy consumption in the state data by their maximum allowable values. Assuming the average control error of the current state data is 0.03 and the maximum allowable value is 0.1, the normalized value is 0.3; the average response time is 150 milliseconds and the maximum allowable value is 500 milliseconds, the normalized value is 0.3; the average energy consumption is 75 watts and the maximum allowable value is 200 watts, the normalized value is 0.375. Multiplying these normalized values ​​by the current feature weight and summing them yields a basic performance score of 0.3 × 0.34 + 0.3 × 0.37 + 0.375 × 0.29 = 0.32.

[0064] The improvement percentage of the basic performance score is calculated at multiple preset timescales. For example, at the short-term (10 minutes), medium-term (1 hour), and long-term (1 day) timescales, assuming the past basic performance scores were 0.38, 0.42, and 0.45, respectively. The current basic performance score of 0.32 represents an improvement percentage of -15.8%, -23.8%, and -28.9% over these three timescales, respectively. These improvement percentages are multiplied by the preset timescale weights (e.g., 0.5, 0.3, and 0.2), and the sum is calculated to obtain an overall performance improvement of -15.8% × 0.5 + (-23.8%) × 0.3 + (-28.9%) × 0.2 = -20.3%.

[0065] Next, we use a Gaussian mixture model to calculate the probability distribution of the overall performance improvement. Assuming the trained Gaussian mixture model contains three Gaussian components, corresponding to low, medium, and high performance levels, the probability densities of the current overall performance improvement of -20.3% in these three components are 0.05, 0.25, and 0.7, respectively. Multiplying these probability densities by the preset scaling factors (e.g., 0.2, 0.5, and 0.8) yields an immediate reward value of 0.05 × 0.2 + 0.25 × 0.5 + 0.7 × 0.8 = 0.685.

[0066] When calculating the cumulative reward value, the contribution of historical instant rewards is taken into account. Assuming the discount factor is 0.9, the instant rewards of the first four moments are 0.52, 0.63, 0.71, and 0.58 respectively, the cumulative reward value is 0.685+0.52×0.9+0.63×0.9 2 +0.71×0.9 3 +0.58×0.9 4 =2.11.

[0067] To adapt to the dynamic changes in performance distribution, calculate the rate of change in the number of samples for each performance level. Assume that the rate of change in the number of samples for the low, medium, and high performance levels over the past 10 cycles was 10%, -5%, and -2%, respectively. The dispersion (e.g., standard deviation) of these rates of change is 0.062. Multiply this dispersion by the preset baseline adjustment coefficient of 0.5 and add 1 to obtain the current adaptive adjustment coefficient of 1.031.

[0068] Based on the statistical characteristics of the cumulative reward value and the adaptive adjustment coefficient, we calculated the classification thresholds. Assuming a mean of 1.85 and a standard deviation of 0.35, we used the Z-score method to set the thresholds. The threshold for the low-medium level is 1.85 - 0.35 × 1.031 = 1.49, and the threshold for the medium-high level is 1.85 + 0.35 × 1.031 = 2.21.

[0069] Finally, calculate the posterior probability of each performance level for the current cumulative reward value of 2.11. Assume that the Bayesian method calculates the posterior probabilities for the low, medium, and high levels to be 0.15, 0.55, and 0.3, respectively. Since 2.11 is greater than the low-medium threshold of 1.49 and less than the medium-high threshold of 2.21, and the medium level has the highest posterior probability, the performance level of the current empirical data is determined to be medium.

[0070] This method objectively assesses the performance of computing systems and dynamically adjusts evaluation criteria based on historical performance, providing a basis for decision-making in optimization and resource allocation. This method is applicable to all types of computing systems requiring real-time performance evaluation, including but not limited to cloud computing platforms, industrial control systems, and smart devices.

[0071] In an optional embodiment, calculating the probability distribution of the comprehensive performance improvement value based on the Gaussian mixture model includes: Calculate the initial values ​​of the mixing weight, mean, and variance parameters of each Gaussian component and substitute them into the Gaussian distribution function to obtain the probability density of each Gaussian component. Multiply the probability density of each Gaussian component by the corresponding mixing weight and sum them to obtain the probability density function of the comprehensive performance improvement value. Calculate the conditional probability of the comprehensive performance improvement value under each Gaussian component, multiply the conditional probability by the corresponding mixing weight and normalize it to obtain the posterior probability of the performance distribution, and substitute the posterior probability of the performance distribution into the probability density function to obtain a modified probability density function; Calculating the time difference between the current moment and the historical moment, substituting the time difference into an exponential function to obtain a time decay factor, and multiplying the time decay factor by the corrected probability density function to obtain a time-weighted probability density function; The time-weighted probability density function is integrated to obtain a cumulative distribution function, and a derivative of the cumulative distribution function is used as the probability distribution of the comprehensive performance improvement value.

[0072] In practical applications, the present invention provides a method for calculating the probability distribution of comprehensive performance improvement values ​​based on a Gaussian mixture model. This method models the performance improvement data using a Gaussian mixture model and takes time decay into account, thereby obtaining a more accurate probability distribution of the performance improvement values.

[0073] The specific implementation of the present invention is as follows. First, historical performance improvement data is collected. This data can be the performance improvement percentages of a device or system under different operating conditions. Assume that the collected data samples are {5.2%, 8.7%, 12.3%, 4.8%, 9.6%, 15.2%, 7.1%, 10.5%, 6.3%, 14.1%}.

[0074] After preprocessing the collected data, the expectation maximization algorithm is used to initialize the parameters of the Gaussian mixture model. In this embodiment, it is assumed that three Gaussian components are used, and the initial mixture weights are set to π1=0.3, π2=0.4, and π3=0.3; the means of each component are μ1=5.0%, μ2=10.0%, and μ3=15.0%; and the variances are σ1 2 =1.0,σ2 2 =1.5,σ3 2 =1.2.

[0075] Substitute these initial parameters into the Gaussian distribution function to calculate the probability density of each Gaussian component. For any performance improvement value x, the probability density of the first Gaussian component is calculated by substituting x, μ1 and σ1 2 Substitute the Gaussian distribution function and calculate the probability density of the second and third Gaussian components in the same way. For example, for a performance improvement of x = 7.5%, the probability density of the first Gaussian component is approximately 0.0484, the probability density of the second Gaussian component is approximately 0.0695, and the probability density of the third Gaussian component is approximately 0.0002.

[0076] Multiply the probability density of each Gaussian component by the corresponding mixing weight and sum them to obtain the probability density function of the overall performance improvement. For a performance improvement of x = 7.5%, the probability density value is 0.3 × 0.0484 + 0.4 × 0.0695 + 0.3 × 0.0002 = 0.0435.

[0077] Based on the initial probability density function, the conditional probability of the overall performance improvement under each Gaussian component is calculated. For a performance improvement of x = 7.5%, its conditional probability under the first Gaussian component is (0.3 × 0.0484) / 0.0435 = 0.3337; the conditional probability under the second Gaussian component is (0.4 × 0.0695) / 0.0435 = 0.6391; and the conditional probability under the third Gaussian component is (0.3 × 0.0002) / 0.0435 = 0.0272.

[0078] Multiplying these conditional probabilities with the corresponding mixing weights and normalizing them yields the posterior probability of the performance distribution. For the first Gaussian component, its posterior probability is 0.3×0.3337 / (0.3×0.3337+0.4×0.6391+0.3×0.0272)=0.2918; for the second Gaussian component, it is 0.6693; and for the third Gaussian component, it is 0.0389.

[0079] Substituting these posterior probabilities into the probability density function, we obtain the revised probability density function. For a performance improvement of x = 7.5%, the revised probability density function is 0.2918 × 0.0484 + 0.6693 × 0.0695 + 0.0389 × 0.0002 = 0.0607.

[0080] Considering the time factor, calculate the time difference between the current moment and the historical moment. Assume that there is a data record 30 days ago, and the current time decay coefficient is set to 0.05, then the time decay factor is the exponential function e (-0.05×30) = 0.2231. Multiplying this time decay factor by the modified probability density function yields the time-weighted probability density function. For the performance improvement x = 7.5% recorded 30 days ago, its time-weighted probability density value is 0.0607 × 0.2231 = 0.0135.

[0081] Numerically integrate the time-weighted probability density function to obtain the cumulative distribution function. This can be achieved by dividing the performance improvement range into multiple small intervals and then applying numerical integration methods to each interval. For example, divide the performance improvement range [0%, 20%] into 200 small intervals, each with a width of 0.1%. Then calculate the value of the time-weighted probability density function in each interval and accumulate them to obtain the cumulative distribution function value at that point.

[0082] Finally, calculate the derivative of the cumulative distribution function to provide the probability distribution of the overall performance improvement. This can be achieved using numerical differentiation methods, such as the central difference method. For a performance improvement of x = 7.5%, calculate the cumulative distribution function values ​​at x = 7.45% and x = 7.55%, then divide the difference between these two points by the interval width of 0.1% to obtain the probability distribution value at x = 7.5%.

[0083] Through the above steps, a probability distribution model for comprehensive performance improvement values ​​was successfully constructed. This model accurately reflects the occurrence of different performance improvement values ​​and takes into account time decay, making recent data more influential on the model. Experiments show that when the new performance improvement data is 11.2%, its position in the probability distribution can be accurately calculated, and the corresponding cumulative probability is 0.734, indicating that this performance improvement value is better than 73.4% of the historical data. This method has broad application prospects in fields such as equipment performance evaluation and optimization effect analysis.

[0084] Figure 3 This is a diagram comparing the accuracy of the probability distribution of performance improvement values: This figure shows experimental results comparing the accuracy of three different methods in predicting the probability distribution of performance improvement values. The horizontal axis represents the number of test samples (0-300), and the vertical axis represents prediction accuracy (0.70-1.05). The present invention (circles) employs an innovative approach combining a Gaussian mixture model with a time decay factor. This method uses multiple Gaussian components to model complex performance improvement data, and introduces a time decay mechanism to give greater weight to recent data. Experimental results show that the present invention achieves the best prediction accuracy across all sample sizes. As the sample size increases, the accuracy steadily improves from 0.72 to 0.963, demonstrating excellent convergence performance. The traditional Gaussian single-component model (squares) uses only a single Gaussian distribution for modeling, failing to effectively capture multimodal distribution characteristics. Its prediction accuracy ultimately converges to around 0.86, significantly lower than the present invention. The empirical distribution estimation method (triangles) uses the cumulative distribution function of historical data for predictions, failing to account for time factors, resulting in a final accuracy of approximately 0.82. The experiment used 300 sets of real performance improvement data. The results showed that the present invention has significant advantages in dealing with complex performance distribution prediction problems, and provides reliable technical support for equipment performance evaluation and optimization effect analysis.

[0085] According to a second aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0086] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0087] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive optimization method for industrial control parameters based on deep reinforcement learning, characterized by: include: Collect real-time operation data of industrial control systems and pre-process them to obtain standardized system state feature vectors; Dividing the system state feature vector into steady-state operating condition data and fluctuating operating condition data according to the system disturbance characteristics, and establishing a steady-state operating condition subnetwork and a fluctuating operating condition subnetwork respectively; Inputting the current system state eigenvector into the steady-state operating condition subnetwork and the fluctuating operating condition subnetwork, adaptively switching the output weights of the subnetwork based on the system disturbance degree, and generating an initial control parameter optimization scheme; Adopting an adaptive parallel sampling strategy for experience replay, in each training batch, the empirical data is divided into a high-performance sample group and a sample group to be optimized according to the system performance index, the high-performance sample group is oversampled, and the sample group to be optimized is undersampled. Based on the sampled empirical data, the initial control parameter optimization scheme is adjusted; Inputting the adjusted initial control parameters into a deep reinforcement learning model for training, updating the model parameters, generating an optimized control parameter combination and applying it to the industrial control system, collecting system status data after application, calculating a comprehensive performance improvement value of the system status data and converting it into a cumulative reward value based on a Gaussian mixture model, and comparing it with a dynamic grading threshold to determine the performance level of the current experience data; The system state data after application, the optimized control parameter combination and its performance level are stored in the experience replay pool for use in the next round of training optimization to achieve adaptive parameter optimization based on working condition characteristics.

2. The method according to claim 1, characterized in that The system state feature vector is divided into steady-state working condition data and fluctuating working condition data according to the system disturbance characteristics, and the steady-state working condition subnetwork and the fluctuating working condition subnetwork are respectively established, including: Calculate the variance of each dimension feature and its mean value of the system state feature vector within the preset time window to obtain the volatility value; Obtaining an initial classification threshold from historical data statistics, and obtaining an updated classification threshold by taking a weighted average of the initial classification threshold and the average volatility of the current batch of data; When the fluctuation value or the absolute value of the fluctuation value is greater than the updated classification threshold, the corresponding system state feature vector is classified as fluctuating operating condition data; when the fluctuation value and the absolute value of the fluctuation value are less than the updated classification threshold, the corresponding system state feature vector is classified as steady-state operating condition data; Based on the fluctuating operating condition data and the steady-state operating condition data, a fluctuating operating condition subnetwork and a steady-state operating condition subnetwork are respectively established. Both subnetworks adopt a three-layer structure and have the same input and output dimensions. The number of hidden layer nodes of the fluctuating operating condition subnetwork is twice the number of input layer nodes and a preset basic learning rate is adopted. The parameter adjustment range is set to the preset basic adjustment range. The number of hidden layer nodes of the steady-state operating condition subnetwork is half the number of input layer nodes and a preset proportion of the preset basic learning rate is adopted. The parameter adjustment range is set to a preset proportion of the preset basic adjustment range.

3. The method according to claim 1, characterized in that Adaptively switch the output weights of the sub-network based on the degree of system disturbance, including: The disturbance characteristics are calculated based on the system state quantity, and each disturbance characteristic is multiplied by the corresponding characteristic weight coefficient and the square root of the sum of squares is calculated to obtain the comprehensive disturbance index; The disturbance threshold is obtained by calculating the sum of the mean and standard deviation of the comprehensive disturbance index within a preset time period; the disturbance threshold is updated using an exponential smoothing method, and the disturbance threshold at the previous moment and the maximum value of the comprehensive disturbance index within the preset time window are multiplied by the first smoothing coefficient and the second smoothing coefficient respectively, and the sum is used to obtain the disturbance threshold at the current moment; Dividing the comprehensive disturbance index by the disturbance threshold at the current moment to obtain a normalized disturbance index; Subtracting a preset central value from the normalized disturbance index, multiplying the difference by the sensitivity coefficient, taking the negative exponent of the product and adding 1 to obtain a disturbance response coefficient, dividing 1 by the disturbance response coefficient to obtain the output weight of the fluctuating working condition subnetwork, and subtracting the output weight of the fluctuating working condition subnetwork from 1 to obtain the output weight of the steady-state working condition subnetwork; The output control parameter of the fluctuating operating condition subnetwork is multiplied by the output weight of the fluctuating operating condition subnetwork to obtain a first weighted parameter, the output control parameter of the steady-state operating condition subnetwork is multiplied by the output weight of the steady-state operating condition subnetwork to obtain a second weighted parameter, and the first weighted parameter and the second weighted parameter are added to obtain an initial control parameter optimization scheme.

4. The method according to claim 1, wherein Adopting an adaptive parallel sampling strategy for experience replay, in each training batch, the experience data is divided into a high-performance sample group and a sample group to be optimized according to the system performance index, the high-performance sample group is oversampled, and the sample group to be optimized is undersampled. Based on the sampled experience data, the initial control parameter optimization scheme is adjusted, including: Calculate the multidimensional performance indicators of the empirical data to obtain the comprehensive performance score of the empirical data; Calculate the average performance score and the standard deviation of the performance score of the current batch of experience data, and use the sum of the average performance score and a preset multiple of the standard deviation of the performance score as the performance grouping threshold; The empirical data with a comprehensive performance score greater than the performance grouping threshold are divided into a high-performance sample group, and the empirical data with a comprehensive performance score less than the performance grouping threshold are divided into a sample group to be optimized; Calculating an oversampling ratio based on the comprehensive performance scores of the empirical data in the high-performance sample group, and calculating an undersampling ratio based on the comprehensive performance scores of the empirical data in the sample group to be optimized; Repeating sampling of the high-performance sample group according to the oversampling ratio, screening and sampling the sample group to be optimized according to the undersampling ratio, multiplying the control parameters in the high-performance sample group by their corresponding oversampling ratios to obtain high-performance weighted parameters, and multiplying the control parameters in the sample group to be optimized by their corresponding undersampling ratios to obtain weighted parameters to be optimized; The high-performance weighted parameter and the weighted parameter to be optimized are summed and divided by the total number of samples to obtain a parameter adjustment amount, and the initial control parameter optimization plan and the parameter adjustment amount are added to obtain an adjusted control parameter optimization plan.

5. The method according to claim 1, wherein The comprehensive performance improvement value of the system status data is calculated and converted into a cumulative reward value based on the Gaussian mixture model. The performance level of the current experience data is determined by comparing it with the dynamic classification threshold. Calculate the ratio of the standard deviation of the system state feature to the preset maximum standard deviation to obtain the feature confidence. Multiply the historical weight of each feature by the corresponding feature confidence and normalize them to obtain the feature weight at the current moment. Divide the average control error, average response time, and average energy consumption of the system state data by the corresponding maximum allowable values, multiply them by the feature weight at the current moment, and sum them to obtain the basic performance score. Calculating the improvement ratio of the basic performance score at multiple preset time scales respectively, multiplying the performance improvement ratio of each time scale by the corresponding scale weight coefficient and summing them to obtain a comprehensive performance improvement value; Calculating the probability distribution of the comprehensive performance improvement value based on a Gaussian mixture model and multiplying it by the proportional coefficient of the corresponding probability distribution value to obtain an immediate reward value; multiplying the immediate reward value at each moment by the power of the discount factor at the corresponding moment and summing the results to obtain a cumulative reward value; Calculate the rate of change of the number of samples at each performance level, multiply the discrete degree of the rate of change by a preset benchmark adjustment coefficient, and add 1 to obtain the adaptive adjustment coefficient at the current moment; calculate the classification threshold based on the mean of the cumulative reward value, the standard deviation of the cumulative reward value, and the adaptive adjustment coefficient; The posterior probability of each performance level corresponding to the cumulative reward value is calculated, and the performance level of the current experience data is determined based on the comparison relationship between the posterior probability of the performance level and the classification threshold.

6. The method according to claim 5, characterized in that Calculating the probability distribution of the comprehensive performance improvement value based on the Gaussian mixture model includes: Calculate the initial values ​​of the mixing weight, mean, and variance parameters of each Gaussian component and substitute them into the Gaussian distribution function to obtain the probability density of each Gaussian component. Multiply the probability density of each Gaussian component by the corresponding mixing weight and sum them to obtain the probability density function of the comprehensive performance improvement value. Calculate the conditional probability of the comprehensive performance improvement value under each Gaussian component, multiply the conditional probability by the corresponding mixing weight and normalize it to obtain the posterior probability of the performance distribution, and substitute the posterior probability of the performance distribution into the probability density function to obtain a modified probability density function; Calculating the time difference between the current moment and the historical moment, substituting the time difference into an exponential function to obtain a time decay factor, and multiplying the time decay factor by the corrected probability density function to obtain a time-weighted probability density function; The time-weighted probability density function is integrated to obtain a cumulative distribution function, and a derivative of the cumulative distribution function is used as the probability distribution of the comprehensive performance improvement value.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images