Industrial equipment autonomous decision control method based on reinforcement learning

Abstract

The invention relates to the technical field of industrial equipment control, and discloses an industrial equipment autonomous decision-making control method based on reinforcement learning, and the method comprises the steps: obtaining the time sequence data of a vibration sensor, the continuous data of a temperature transmitter and the discrete data of a pressure instrument of industrial equipment through a multi-mode sensing module; multi-modal data alignment is achieved through a feature space mapping algorithm, the thermal drift compensation amount is calculated in combination with a thermodynamic state model, self-adaptive threshold segmentation processing is conducted on pressure data, the processed data are input into a reinforcement learning decision model to generate a fusion decision result, exploration-utilization balance parameters are adjusted through a strategy updating mechanism, and the fusion decision result is obtained. And outputting the autonomous decision control scheme. The method solves the problems of poor adaptability, low decision-making efficiency and the like of a traditional control method, can improve the autonomous decision-making capability, the control precision and the operation stability of the equipment, reduces manual intervention, reduces the cost, and is suitable for intelligent control of the industrial equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial equipment control, specifically to an industrial equipment autonomous decision control method based on reinforcement learning. BACKGROUND

[0002] Existing industrial equipment control methods mostly rely on pre-set control rules and models. These methods often show poor adaptability and low decision efficiency when facing complex and variable industrial environments.

[0003] Industrial equipment is affected by various factors during operation, such as mechanical component vibration, temperature changes during equipment operation, and pressure fluctuations within the system. These factors interweave, making the equipment's running state complex and difficult to predict. Traditional control methods have difficulty effectively processing these multi-modal running state data, and cannot accurately capture subtle changes during equipment operation, resulting in low control precision.

[0004] When industrial equipment encounters abnormal situations, traditional control methods often require manual intervention for fault diagnosis and disposal, which not only increases labor costs, but also may cause the expansion of equipment failure due to the delay of manual decision-making, causing huge losses to industrial production. Moreover, traditional methods lack the ability of autonomous learning and optimization, and cannot continuously improve control strategies based on equipment operation history and real-time data, making it difficult to adapt to changing working conditions in industrial production processes.

[0005] With the development of industrial internet and big data technology, the amount of data generated by industrial equipment has grown explosively. How to effectively utilize these data to achieve autonomous decision control of industrial equipment has become an important challenge in the field of industrial control. Existing control methods have problems such as insufficient data fusion and inaccurate feature extraction when dealing with large-scale, multi-modal data, making it difficult to build accurate equipment running state models, thereby affecting the autonomous decision-making ability of the equipment.

[0006] Reinforcement learning, as a machine learning method, has the ability of autonomous learning and optimization of decision-making, and can theoretically adapt to complex industrial control environments. However, applying reinforcement learning to the field of industrial equipment control still faces many technical challenges. For example, how to effectively align and fuse multi-modal equipment running state data to build a state space suitable for reinforcement learning; how to combine the physical characteristics and operation rules of industrial equipment to design a reasonable reward function and decision model; how to achieve exploration-exploitation balance of reinforcement learning while ensuring control safety. These problems need to be solved to promote the practical application of reinforcement learning in industrial equipment autonomous decision control. SUMMARY

[0007] The application aims to provide an industrial equipment autonomous decision control method based on reinforcement learning to solve the problems in the background art.

[0008] To achieve the above-mentioned purpose, the application provides the following technical solutions: an industrial equipment autonomous decision control method based on reinforcement learning, the method comprising: An operating state data set of the industrial equipment is obtained through a multi-modal perception module, the operating state data including vibration sensor time series data, temperature transmitter continuous data and pressure instrument discrete data; the vibration sensor time series data contains mechanical component frequency characteristics and amplitude distribution information, and the temperature transmitter continuous data contains heat conduction gradient and steady-state temperature field information; Based on the vibration sensor time series data, multi-modal data alignment is realized through a feature space mapping algorithm to generate a state space dimension mapping relationship after alignment; According to the temperature transmitter continuous data, a device thermal drift compensation amount is calculated in combination with a thermodynamic state model to generate a dynamic operation correction parameter; The pressure instrument discrete data is subjected to adaptive threshold segmentation processing to extract key operating state points and abnormal event boundaries; The state space dimension mapping relationship, the dynamic operation correction parameter and the key operating state points are input into a reinforcement learning decision model to generate a fusion decision result of equipment control actions and abnormal disposal; Based on the fusion decision result, an exploration-exploitation balance parameter of reinforcement learning is adjusted through a policy update mechanism to output an autonomous decision control scheme; the nodes of the policy update mechanism represent state space division, and the edges represent action selection priority and decision confidence weight.

[0009] Preferably, the multi-modal data alignment through the feature space mapping algorithm comprises: The vibration sensor time series data is subjected to time-frequency domain conversion processing to generate a state feature map of time-frequency joint distribution; Based on a predefined reference feature library, principal component feature vectors are extracted through kernel principal component analysis algorithm, and a space transformation matrix is calculated; The space transformation matrix is optimized using a local linear embedding algorithm to generate an accurate mapping relationship between multi-modal sensor coordinate systems; The accurate mapping relationship and the state space dimension mapping relationship are subjected to consistency verification to output an alignment error correction parameter.

[0010] Preferably, the calculation of the device thermal drift compensation amount in combination with the thermodynamic state model comprises: The temperature field of the temperature transmitter continuous data is reconstructed to extract real-time temperature variation curves of each component; According to the material thermal expansion coefficient database, the nonlinear drift amount calculation formula under the current temperature change is matched; The local drift compensation coefficient is generated by simulating the thermal deformation trend through a finite element thermal analysis model. The nonlinear drift amount calculation formula and the local drift compensation coefficient are combined to output the dynamic operation correction parameter.

[0011] Preferably, the adaptive threshold segmentation processing of the pressure instrument discrete data comprises: The time series decoding of the pressure instrument discrete data is performed to extract the pressure fluctuation distribution diagram and the threshold mapping relationship; Based on the variance change of the pressure fluctuation distribution diagram, the stable operation area and the abnormal fluctuation area are divided; According to the region stability weight setting, the segmentation threshold range is set, and the multi-scale quantile segmentation processing is performed; The threshold mapping relationship after segmentation is subjected to morphological opening operation with the abnormal event boundary to generate the smooth running state point.

[0012] Preferably, the reinforcement learning decision model comprises a state encoding module and an action evaluation module, and the state encoding module comprises: The feature dimension reduction calculation of the state space dimension mapping relationship is performed to generate a low-dimensional dense state representation vector; The dynamic operation correction parameter is converted into a vector dimension offset correction amount and is superimposed into the state representation vector; The time sequence connection of the key running state point is performed to generate the time constraint condition of the abnormal event; The state representation vector and the time constraint condition are fused through a constraint optimization algorithm to output the device control action and the abnormal disposal fusion decision result.

[0013] Preferably, the action evaluation module comprises: According to the abnormal event boundary, the abnormal type features are extracted, including the vibration over-limit amplitude, the temperature overshoot rate and the pressure sudden drop depth; Based on the association rule base of the abnormal type features and the control action, an action influence weight matrix is constructed; The comprehensive score of the action weight and the decision confidence is calculated through a deep Q network algorithm to generate a control strategy priority result; The control strategy priority result is associated with the action selection priority to update the edge weight matrix of the strategy update mechanism.

[0014] Preferably, the exploration-exploitation balance parameter of the reinforcement learning is adjusted through the strategy update mechanism, comprising: According to the state space division, the node attributes are initialized, and the initial edge weight matrix is generated based on the decision confidence weight; The device control action and the abnormal treatment fusion decision result are taken as a node state, and an edge weight matrix is composed of action execution time efficiency and decision confidence weight; The node path is traversed through a deep deterministic policy gradient algorithm, and a comprehensive benefit function of each path is calculated; An optimal policy update sequence is selected according to the comprehensive benefit function, and an autonomous decision control scheme is output.

[0015] Preferably, the parameter optimization method of the finite element thermal analysis model comprises: The initial mesh division density and material parameter sensitivity are calculated according to the historical thermal drift data distribution; The simulation parameter combination is traversed through a Latin hypercube sampling algorithm, and the parameters with the minimum error between the simulation result and the actual measured data are selected; The mesh division density and material parameter sensitivity are dynamically adjusted according to the error distribution, and the precision of the local drift compensation coefficient is optimized.

[0016] Preferably, the construction method of the comprehensive benefit function comprises: The path benefit is defined as a balance factor of time efficiency and decision confidence weight; The path benefit of each node is initialized to zero, and the path benefit of the starting point is a preset reference value; The maximum cumulative benefit of each node based on the previous nodes is calculated through a dynamic programming algorithm, and an optimal policy update sequence is recorded; A complete decision control scheme is generated by reverse deduction according to the optimal policy update sequence.

[0017] Preferably, the network optimization method of the deep Q network algorithm comprises: A deep neural network structure comprising a convolutional layer and a fully connected layer is constructed, and the convolutional layer is used to extract local features of a state representation vector; A history state-action-reward triple is stored through an experience replay mechanism, and network parameter updating is performed through batch sampling; The action selection priority is dynamically adjusted according to the Q value distribution output by the network, and the precision of the control strategy priority result is optimized.

[0018] Compared with the prior art, the beneficial effects of the present application are: The autonomous decision control method of the industrial equipment based on reinforcement learning provided by the present application can comprehensively and accurately reflect the running state of the equipment by acquiring the vibration sensor time series data, temperature transmitter continuous data and pressure instrument discrete data of the industrial equipment through a multi-modal perception module. Based on these data, multi-modal data alignment is realized through a feature space mapping algorithm, a dimensional mapping relationship of the aligned state space is generated, the problem of insufficient multi-modal data fusion is solved, and the subsequent decision model can more accurately process data.

[0019] The device thermal drift compensation amount is calculated in combination with a thermodynamic state model, and a dynamic operation correction parameter is generated, fully considering the influence of temperature change on device operation, thereby improving the operation precision and stability of the device under different temperature conditions. The adaptive threshold segmentation processing is performed on the discrete data of the pressure instrument, the key operation state points and abnormal event boundaries are extracted, and the abnormal conditions in the device operation process can be found in time, thereby providing accurate basis for abnormal treatment.

[0020] The processed state space dimension mapping relationship, the dynamic operation correction parameter and the key operation state points are input into a reinforcement learning decision model to generate a fusion decision result of the device control action and the abnormal treatment. The model can autonomously learn the operation law and the control strategy of the device, dynamically adjusts the control action according to the real-time data, and improves the autonomous decision-making ability and adaptability of the device. The exploration-exploitation balance parameter of the reinforcement learning is adjusted through a strategy updating mechanism, and an autonomous decision-making control scheme is output, so that the control strategy can be optimized and improved in the continuous operation process, thereby improving the reliability and effectiveness of the control strategy.

[0021] In the feature space mapping algorithm, the time-frequency domain conversion processing is performed on the time series data of the vibration sensor to generate a state feature map of the time-frequency joint distribution, and the kernel principal component analysis algorithm and the local linear embedding algorithm are combined to improve the precision and reliability of the multi-modal data alignment. In the calculation of the thermal drift compensation amount, the thermal deformation trend of the device can be more accurately simulated through the temperature field reconstruction, the material thermal expansion coefficient matching and the finite element thermal analysis model, and more accurate dynamic operation correction parameters can be generated.

[0022] The processing of the discrete data of the pressure instrument can more accurately divide the stable operation region and the abnormal fluctuation region, and extract the key operation state points through time series decoding, variance change analysis and multi-scale quantile segmentation processing. The state encoding module and the action evaluation module in the reinforcement learning decision model can effectively reduce, fuse and evaluate the state data to generate more reasonable control strategy priority results.

[0023] The strategy updating mechanism traverses the node path through the deep deterministic policy gradient algorithm, calculates the comprehensive revenue function of each path, selects the optimal strategy updating sequence, so that the control scheme can improve the decision efficiency and reliability while ensuring the control precision. The parameter optimization method of the finite element thermal analysis model and the network optimization method of the deep Q network algorithm further improve the precision and performance of the model, so that the entire control method can better adapt to complex industrial environments. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a working principle diagram of the industrial device autonomous decision-making control method based on reinforcement learning. Figure 2a design diagram for multi-modal data alignment processing; Figure 3 a design diagram for device thermal drift compensation calculation; Figure 4 a design diagram for pressure data adaptive threshold segmentation; Figure 5 a design diagram for a reinforcement learning action evaluation module. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0026] Please refer to Figures 1-5 The industrial equipment autonomous decision control method based on reinforcement learning comprises the following specific implementation steps: An operating state data set of the industrial equipment is obtained through a multi-modal perception module, wherein the operating state data comprises vibration sensor time series data, temperature transmitter continuous data and pressure instrument discrete data; wherein the vibration sensor time series data contains mechanical component frequency characteristics and amplitude distribution information, and the temperature transmitter continuous data contains heat conduction gradient and steady-state temperature field information.

[0027] Based on the vibration sensor time series data, multi-modal data alignment is realized through a feature space mapping algorithm to generate a dimension mapping relationship of the aligned state space. The specific process is as follows: the vibration sensor time series data is subjected to time-frequency domain conversion processing to generate a state feature map of time-frequency joint distribution; based on a predefined reference feature library, principal component feature vectors are extracted through kernel principal component analysis algorithm, and a space transformation matrix is calculated; the space transformation matrix is optimized through a local linear embedding algorithm to generate an accurate mapping relationship between multi-modal sensor coordinate systems; the accurate mapping relationship and the dimension mapping relationship of the state space are subjected to consistency checking, and an alignment error correction parameter is output.

[0028] According to the temperature transmitter continuous data, a device thermal drift compensation amount is calculated in combination with a thermodynamic state model to generate a dynamic operation correction parameter. Specifically, the temperature transmitter continuous data is subjected to temperature field reconstruction to extract real-time temperature variation curves of each component; a non-linear drift amount calculation formula under the current temperature variation is matched according to a material thermal expansion coefficient database; a local drift compensation coefficient is generated by simulating a thermal deformation trend through a finite element thermal analysis model; the non-linear drift amount calculation formula and the local drift compensation coefficient are combined to output the dynamic operation correction parameter.

[0029] The pressure instrument discrete data is adaptively threshold segmented to extract key operation state points and abnormal event boundaries. The processing steps are: time series decoding is performed on the pressure instrument discrete data to extract pressure fluctuation distribution and threshold mapping relationship; based on the variance change of the pressure fluctuation distribution, stable operation area and abnormal fluctuation area are divided; according to the area stability weight, the threshold range is set, and multi-scale quantile segmentation processing is performed; the segmented threshold mapping relationship is morphologically opened with the abnormal event boundary to generate smooth operation state points.

[0030] The state space dimension mapping relationship, dynamic operation correction parameter and key operation state point are input into a reinforcement learning decision model to generate a device control action and abnormal disposal fusion decision result. The model includes a state coding module and an action evaluation module: the state coding module performs feature dimension reduction calculation on the state space dimension mapping relationship to generate a low-dimensional dense state representation vector; the dynamic operation correction parameter is converted into a vector dimension offset correction, which is superimposed into the state representation vector; the key operation state point is time-connected to generate a time constraint condition of the abnormal event; the state representation vector and the time constraint condition are fused through a constraint optimization algorithm to output a fusion decision result. The action evaluation module extracts abnormal type features according to the abnormal event boundary, including vibration amplitude overrun, temperature overshoot rate and pressure drop depth; based on the association rule base of abnormal type features and control actions, an action influence weight matrix is constructed; the comprehensive score of action weight and decision confidence is calculated through a deep Q network algorithm to generate a control strategy priority result; the control strategy priority result is associated with the action selection priority to update the edge weight matrix of the strategy update mechanism.

[0031] Based on the fusion decision result, the exploration-exploitation balance parameter of reinforcement learning is adjusted through the strategy update mechanism to output an autonomous decision control scheme. The node of the strategy update mechanism represents the state space division, and the edge represents the action selection priority and the decision confidence weight. The specific process is: the node attributes are initialized according to the state space division, and the initial edge weight matrix is generated based on the decision confidence weight; the device control action and abnormal disposal fusion decision result are taken as the node state, and the edge weight matrix is composed of the action execution time efficiency and the decision confidence weight; the comprehensive revenue function of each path is calculated by traversing the node path through a deep deterministic policy gradient algorithm; the optimal strategy update sequence is selected according to the comprehensive revenue function, and the autonomous decision control scheme is output.

[0032] Embodiment 1: This embodiment describes in detail the specific process of multi-modal data alignment through feature space mapping algorithm based on vibration sensor time series data. Time-frequency domain conversion is performed on the vibration sensor time series data. The original data collected by the vibration sensor is a time series signal that changes over time, which contains the vibration information of mechanical components at different times. In order to analyze these data more comprehensively, it is necessary to convert them from the time domain to the frequency domain to obtain frequency characteristics and amplitude distribution information. Here, short-time Fourier transform (STFT) is used to realize time-frequency domain conversion. Short-time Fourier transform slides a window function on the time axis, Fourier transforms the data in each time window, and obtains the frequency components of the signal at different time points to generate a state feature map of time-frequency joint distribution. This feature map takes time as the horizontal axis and frequency as the vertical axis, and the amplitude of each point represents the energy intensity of the corresponding frequency at that time point, which can intuitively reflect the changes of mechanical component vibration characteristics over time.

[0033] Based on the pre-defined reference feature library, the principal component feature vector is extracted by kernel principal component analysis algorithm (KPCA), and the space transformation matrix is calculated. The reference feature library stores a large number of multi-modal data features under normal operating conditions as a reference standard for data alignment. Kernel principal component analysis is an extension of principal component analysis (PCA), which maps nonlinear data to high-dimensional feature space through kernel function, and then performs principal component analysis in high-dimensional space, so as to process nonlinear data relationship. Specifically, first, the state feature map of time-frequency joint distribution is input into the KPCA algorithm, which is mapped to high-dimensional space through kernel function (such as radial basis function RBF), and then the covariance matrix in high-dimensional space is calculated, and the eigenvalue and eigenvector of the covariance matrix are solved to extract the principal component feature vector. These principal component feature vectors represent the main change direction of data in high-dimensional space, which can preserve the key information of data to the greatest extent. According to the principal component feature vector, the space transformation matrix can be calculated, which is used to map data of different modalities to the same feature space to realize preliminary data alignment.

[0034] The local linear embedding algorithm (LLE) is used to optimize the spatial transformation matrix to generate an accurate mapping relationship between the coordinate systems of multi-modal sensors. The basic idea of the local linear embedding algorithm is to maintain the local linear structure of the data, that is, each data point can be linearly represented by the points in its neighborhood. In the data alignment process, different modal sensors may have different coordinate systems and measurement accuracies, resulting in some errors in the preliminary mapping results. The LLE algorithm optimizes the spatial transformation matrix through the following steps: first, find the k nearest neighbors of each data point in the high-dimensional space; then, calculate the linear representation coefficients of each data point by its nearest neighbors; finally, keep these linear representation coefficients unchanged in the low-dimensional space, and solve the optimal low-dimensional embedding coordinates. Through the optimization of the LLE algorithm, the mapping relationship between the coordinate systems of multi-modal sensors can be more accurate, and the local structure and global features of the data can be better preserved, so as to generate an accurate mapping relationship.

[0035] The accurate mapping relationship is consistent with the state space dimension mapping relationship, and the alignment error correction parameters are output. The state space dimension mapping relationship is defined before data alignment, which describes the dimension correspondence of different modal data in the state space. In order to ensure the consistency of the accurate mapping relationship and the state space dimension mapping relationship, consistency check is needed. Specifically, the multi-modal data after accurate mapping is matched in dimension according to the state space dimension mapping relationship, and then the mapping error in each dimension is calculated. For example, for vibration sensor data and temperature transmitter data, in a certain common physical dimension (such as time or spatial position), the difference between their mapped data values is compared, and the distribution of these differences is counted to obtain the alignment error. According to the size and distribution of the alignment error, alignment error correction parameters are output, which are used to adjust the subsequent data alignment process, such as correcting the parameters of the spatial transformation matrix or reselecting the reference features in the reference feature library, to improve the accuracy and reliability of data alignment. Through the sequential execution of the above four steps, the feature space mapping processing of the vibration sensor time series data is realized, and the aligned state space dimension mapping relationship is generated, providing accurate and consistent multi-modal data basis for subsequent autonomous decision control of industrial equipment.

[0036] Embodiment 2: The embodiment details the specific process of calculating the device thermal drift compensation amount and generating dynamic operation correction parameters combined with the thermodynamic state model. First, the temperature field reconstruction is performed on the continuous data of the temperature transmitter, and the real-time temperature change curve of each component is extracted. The temperature transmitter collects continuous temperature data of each part of the device, but these data usually exist in the form of discrete sampling points. In order to more intuitively reflect the temperature distribution and change trend of the whole device, temperature field reconstruction is needed. Specifically, the discrete temperature sampling points are processed by interpolation algorithm (such as cubic spline interpolation or radial basis function interpolation) to convert them into continuous temperature field distribution. In the reconstruction process, the geometric structure, material properties and heat source distribution of the device need to be considered to ensure that the reconstructed temperature field can accurately reflect the actual temperature state of the device. After completing the temperature field reconstruction, the real-time temperature change curve of each component is extracted from the temperature field, which clearly shows the temperature change of each component of the device at different times, providing basic data for subsequent thermal drift calculation.

[0037] According to the material thermal expansion coefficient database, the non-linear drift amount calculation formula under the current temperature change is matched. The material thermal expansion coefficient database stores the thermal expansion coefficient data of various industrial device materials in different temperature intervals, which are usually obtained by experimental measurement. After obtaining the real-time temperature change curve of each component of the device, the material type of each component is first determined, and then the corresponding thermal expansion coefficient is found in the database according to the current temperature change range. Since the thermal expansion characteristics of materials are often nonlinear, the thermal expansion coefficient will also change accordingly with the change of temperature, so it is necessary to select appropriate nonlinear model to describe the relationship between thermal expansion and temperature according to the specific situation of temperature change. For example, for some metal materials, the thermal expansion amount may be related to the square or cubic of temperature, so the corresponding nonlinear function needs to be used as the drift amount calculation formula. In this way, the thermal expansion behavior of materials at different temperatures can be described more accurately, and the calculation accuracy of thermal drift compensation amount can be improved.

[0038] The local drift compensation coefficients are generated by simulating the thermal deformation trend through a finite element thermal analysis model. The finite element thermal analysis model is a numerical simulation method that discretizes the device structure into multiple finite element units. By applying corresponding thermal boundary conditions and material properties to each unit, the thermal deformation of the device under different temperature fields can be simulated. In establishing the finite element thermal analysis model, the device needs to be first geometrically modeled and decomposed into appropriate finite element grids. The grid partition density needs to be determined according to the complexity of the device structure and the accuracy requirements of thermal analysis. This involves a parameter optimization method for the finite element thermal analysis model: first, calculate the initial grid partition density and material parameter sensitivity based on the historical thermal drift data distribution. The historical thermal drift data contains the deformation data of the device due to temperature changes during past operation. Through statistical analysis of these data, it can be determined which areas need denser grids and which material parameters have a more sensitive impact on thermal deformation.

[0039] The Latin hypercube sampling algorithm is used to traverse the simulation parameter combinations. This algorithm is a stratified sampling method that can obtain a more uniform parameter distribution with fewer sampling times, thereby improving the efficiency of parameter optimization. By traversing different grid partition densities and material parameter combinations, multiple finite element simulations are performed, and the parameter combination with the smallest error between the simulation results and the actual measurement data is selected. Finally, the grid partition density and material parameter sensitivity are dynamically adjusted according to the error distribution to further optimize the accuracy of the local drift compensation coefficients. Through the simulation of the finite element thermal analysis model, the thermal deformation and deformation trend of each part of the device can be obtained, and the local drift compensation coefficients can be generated. These coefficients represent the degree of compensation required by different parts during the thermal deformation process.

[0040] The nonlinear drift amount calculation formula is combined with the local drift compensation coefficient to output dynamic operation correction parameters. The nonlinear drift amount calculation formula describes the overall relationship between the material thermal expansion amount and the temperature change, while the local drift compensation coefficient reflects the thermal deformation difference of each part of the device due to structural and material differences. By combining the two, the nonlinear drift amount calculation formula needs to be corrected according to the material type of each component of the device, the temperature change, and the local drift compensation coefficient. Specifically, for each component, first calculate the theoretical thermal expansion amount according to its temperature change curve and the nonlinear drift amount calculation formula, then multiply it by the corresponding local drift compensation coefficient to get the actual thermal drift amount that needs to be compensated for that component. Convert these compensation amounts into corresponding operation parameter correction values, for example, for mechanical transmission components, thermal drift may cause changes in their size, affecting transmission accuracy, at which point the thermal drift compensation amount needs to be converted into a correction value for the transmission parameters; for electrical components, thermal drift may affect their electrical performance, and the compensation amount needs to be converted into a correction value for the electrical parameters. In this way, dynamic operation correction parameters are generated, which can reflect the operation deviation of the device due to temperature changes in real time and are used to correct the operation state of the device to ensure stable and reliable operation of the device under different temperature conditions.

[0041] In this embodiment, the specific process of adaptive threshold segmentation processing of discrete data of a pressure instrument to extract key operation state points and abnormal event boundaries is described in detail. Time series decoding is performed on the discrete data of the pressure instrument to extract the pressure fluctuation distribution diagram and the threshold mapping relationship. The discrete data collected by the pressure instrument is usually stored in a specific encoding format, which records the pressure values at different time points, but needs to be decoded before being used for analysis. The time series decoding process arranges the discrete pressure data in chronological order and restores it to a continuous time series form. During decoding, factors such as data sampling frequency, encoding rules, and possible noise interference need to be considered to ensure that the decoded data accurately reflects the real changes of pressure over time. After decoding, the pressure fluctuation distribution diagram is obtained by plotting the corresponding relationship between pressure value and time. This distribution diagram directly shows the fluctuation amplitude, frequency, and trend of pressure in different time periods. At the same time, based on historical data and process requirements of device operation, the mapping relationship between pressure value and threshold is established to preliminarily determine the possible threshold range, providing a basis for subsequent threshold segmentation.

[0042] Based on the variance change of the pressure fluctuation distribution, the stable operation area and the abnormal fluctuation area are divided. Variance is an important indicator to measure the degree of data dispersion. In the pressure fluctuation analysis, the pressure data under stable operation state usually has smaller variance, that is, the pressure value is relatively stable and the fluctuation amplitude is small. When the equipment is abnormal, the variance of pressure data will significantly increase, indicating that the pressure fluctuation is intensified. In specific operation, the pressure fluctuation distribution is first divided into several time periods in chronological order. The length of each time period can be determined according to the characteristics of the data and the analysis requirements. Then, the variance of the pressure data in each time period is calculated, and a reasonable variance threshold is set to distinguish the stable operation area and the abnormal fluctuation area. For example, when the variance of a certain time period is less than the set threshold, it is determined as a stable operation area; when the variance is greater than or equal to the set threshold, it is determined as an abnormal fluctuation area. When determining the variance threshold, factors such as the normal operation range of the equipment, the distribution of historical data and the process requirements should be considered to ensure the accuracy of the area division. Through this way, different operation state areas in the pressure fluctuation distribution can be clearly divided, providing a more targeted processing range for subsequent threshold segmentation.

[0043] The threshold segmentation range is set according to the regional stability weight, and the multi-scale quantile segmentation processing is performed. The regional stability weight is determined according to the historical operation stability of each region. The higher the stability, the closer the pressure data in the region to the normal operation state, and the larger the weight. The lower the stability, the smaller the weight. When setting the threshold segmentation range, for the stable operation area, the threshold range can be set relatively narrow because the pressure data is relatively stable, so as to more accurately capture the pressure change under normal operation state; for the abnormal fluctuation area, the threshold range needs to be appropriately widened to avoid misjudging some normal fluctuations as abnormal. Multi-scale quantile segmentation processing is to analyze pressure data from different scales. Common quantiles include quartiles, tenths, etc. Specifically, for each divided region, the pressure values corresponding to different quantiles are calculated, such as 25%, 50%, 75% quantiles, etc. These quantiles are used as candidate points for threshold segmentation. Through multi-scale quantile segmentation, the distribution characteristics of pressure data can be captured from different angles, avoiding the limitations that may be brought by single-scale segmentation. When performing segmentation, the threshold of different quantiles needs to be weighted according to the regional stability weight, so that the quantile threshold of the region with high stability occupies a larger weight in the final threshold range, thereby improving the accuracy and reliability of threshold segmentation.

[0044] The segmented threshold mapping relationship and the abnormal event boundary are subjected to morphological opening operation to generate smooth running state points. The morphological opening operation is a mathematical morphology-based image processing operation including two steps of erosion and dilation, and is commonly used for smoothing boundaries, removing noise and separating objects, etc. Here, the segmented threshold mapping relationship is regarded as a binary image, where the area above the threshold represents possible abnormalities and the area below the threshold represents normal running states. The abnormal event boundary is predefined according to historical data of device operation and process knowledge, and is used to define the range of abnormal event occurrence. By performing morphological opening operation on the threshold mapping relationship and the abnormal event boundary, first, the image is subjected to erosion operation to remove small protrusions and noise points on the boundary, and then subjected to dilation operation to restore the area reduced due to erosion, thereby smoothing the threshold boundary and the abnormal event boundary. After morphological opening operation processing, the burrs and discontinuous points that may occur in the threshold segmentation process can be effectively removed, so that the extracted key running state points and the abnormal event boundary are smoother and more continuous. The generated smooth running state points can accurately reflect the running conditions of the device under different pressure states, and provide clear and reliable input data for the subsequent reinforcement learning decision model. These state points not only contain the specific values of the pressure, but also ensure the continuity and stability of the time series through morphological processing, which helps to improve the accuracy of the fusion decision result of the device control action and abnormal disposal.

[0045] In this embodiment, the working process of the state encoding module in the reinforcement learning decision model is described in detail. The module generates fusion decision results of device control actions and abnormal disposal through processing and fusion of multi-source data. Feature dimension reduction calculation is performed on the state space dimension mapping relationship to generate a low-dimensional dense state representation vector. In the running process of industrial equipment, the vibration, temperature, pressure and other data obtained by the multi-modal perception module form a state space dimension mapping relationship after feature space mapping, which often has a high dimension. Direct processing will increase the computational complexity and may introduce the dimension disaster problem. Therefore, a feature dimension reduction algorithm needs to be used for processing. Common dimension reduction algorithms such as principal component analysis (PCA) or t-distributed stochastic neighbor embedding (t-SNE) can be selected according to the characteristics of the data. Taking PCA as an example, the core idea is to project high-dimensional data into a low-dimensional space through linear transformation while preserving the main variance information of the data. In specific operation, the state space dimension mapping relationship is first standardized to eliminate the influence of the dimension of different dimensions of data, and then the covariance matrix of the data is calculated, the eigenvalues and eigenvectors are solved, the principal components are selected according to the contribution rate of the eigenvalues, and then the high-dimensional data is mapped into a low-dimensional space spanned by the principal components to generate a low-dimensional dense state representation vector. This representation vector reduces the dimension while preserving the key features of the original data to the greatest extent, providing efficient data representation for subsequent processing.

[0046] The dynamic operation correction parameters are converted into offset correction amounts of vector dimensions and superimposed into the state representation vector. The dynamic operation correction parameters are calculated in combination with the thermodynamic state model and are used to compensate for the operation deviation of the equipment due to factors such as thermal drift. These parameters usually include thermal deformation compensation amounts of each component and transmission precision correction values. In order to integrate these parameters into the state representation, it is necessary to determine the corresponding relationship between the parameters and each dimension of the state representation vector. For example, the thermal deformation compensation amount of a certain component may correspond to the mechanical displacement dimension in the state representation vector. At this time, the compensation amount needs to be converted into the offset correction amount of this dimension, and the conversion process needs to consider the physical meaning and dimension of the parameter. After the conversion is completed, the offset correction amount is superimposed into the generated state representation vector according to the corresponding dimension. This superimposition operation is not a simple numerical addition, but different weights are given according to the influence degree of the parameters, for example, higher weights are given to key parameters with greater influence, so as to ensure that the state representation vector can more accurately reflect the actual operation state of the equipment. By integrating the dynamic operation correction parameters into the state representation vector, the model can directly consider the real-time deviation compensation information of the equipment when making decisions, and the accuracy of the decisions is improved.

[0047] The key operation state points are sequentially connected to generate the time constraint conditions of the abnormal events. The key operation state points are extracted by adaptive threshold segmentation processing of the discrete data of the pressure instrument, including key state points of normal operation and boundary points of abnormal events. The distribution and connection relationship of these points on the time sequence contains the change trend of the operation state of the equipment and the development process of the abnormal events. The sequential connection operation needs to connect these discrete key state points into a sequence with time sequence according to the time sequence. For example, a certain abnormal event may be composed of a pressure drop point, a temperature overshoot point and a vibration overrun point in a certain time sequence. At this time, these points need to be sequentially connected according to the occurrence time to form the time sequence of the abnormal event. In the connection process, the time interval and sequence relationship between the state points need to be recorded to generate the time constraint conditions of the abnormal events. These constraint conditions clearly define the time sequence and allowable time deviation range of each state point in the abnormal event, for example, it is stipulated that the temperature overshoot point must occur within a certain time after the pressure drop point, otherwise it may belong to a different type of abnormality. The generation of the time constraint conditions helps the model to consider the time correlation of the events when making decisions, avoids processing individual state points in isolation, and thus more accurately identifies the type and development stage of the abnormal event.

[0048] The state representation vector is fused with the time constraint condition by a constraint optimization algorithm, and the output device control action and abnormal treatment fusion decision result. The role of the constraint optimization algorithm is to process the state representation vector under the premise of meeting the time constraint condition, and generate the optimal control decision. First, the time constraint condition needs to be converted into a mathematical constraint form that can be processed by the algorithm, such as expressing the time sequence and interval requirements of each state point as inequality constraints. Then, the state representation vector is taken as the optimization variable, and a target function is defined, which is usually constructed based on the goal of device control (such as energy consumption minimization, efficiency maximization or abnormal treatment priority). For example, when the device detects an abnormality, the target function may tend to preferentially select control actions that can quickly suppress the development of the abnormality. In the solving process of the constraint optimization algorithm, it will find the solution that makes the target function optimal in the feasible solution space that meets all the time constraint conditions, and this solution corresponds to the optimal device control action and abnormal treatment fusion decision result. Common constraint optimization algorithms such as sequential quadratic programming (SQP) or interior point method can be selected according to the complexity and solution accuracy requirements of the problem. Through the processing of the algorithm, the real-time state information of the device contained in the state representation vector and the time constraint condition of the abnormal event are organically fused, and the final output decision result not only considers the current state of the device, but also takes into account the time development law of the abnormal event, thereby providing the device with more reasonable and time-effective control schemes and abnormal treatment strategies.

[0049] In this embodiment, the process of adjusting the exploration-exploitation balance parameter of reinforcement learning through the policy updating mechanism and outputting the autonomous decision control scheme is described in detail. The node attributes are initialized according to the state space division, and the initial edge weight matrix is generated based on the decision confidence weight. The division of the state space is based on different feature combinations of the industrial equipment running state, such as vibration frequency range, temperature interval, pressure value range, etc. as the division dimension, and the entire state space is divided into multiple non-overlapping nodes, each node representing a specific device running state. When initializing the node attributes, each node needs to be assigned with corresponding state feature parameters, such as the upper and lower limits of the vibration frequency range, the average value of the temperature interval, etc. These attributes are used to represent the running state characteristics represented by the node. The decision confidence weight is generated in the previous decision process, reflecting the confidence level of different decision results. Based on these weights, an initial edge weight matrix is constructed, the elements in the matrix represent the action selection priority and decision confidence weight from one node to another node, the rows and columns of the matrix correspond to different state nodes, and the size of the matrix value represents the possibility of action transition and the confidence level of the transition.

[0050] The device control action and abnormality treatment fusion decision result is taken as the node state, and the edge weight matrix is composed of action execution time efficiency and decision confidence weight. The device control action and abnormality treatment fusion decision result is generated by the state coding module and the action evaluation module of the reinforcement learning decision model, and contains the optimal control action and abnormality treatment strategy for the current device state. These decision results are mapped to the corresponding nodes in the state space as new node states. At this time, the composition elements of the edge weight matrix have changed, and are no longer based on the initial decision confidence weight, but have integrated the key factor of action execution time efficiency. The action execution time efficiency refers to the time required to execute a certain control action to transfer from the current state to the target state, which reflects the speed of action execution. When constructing the edge weight matrix, the action execution time efficiency and the decision confidence weight need to be considered comprehensively, and a comprehensive weight value is given to each edge. For example, the edge corresponding to the action with shorter execution time and higher decision confidence is given a higher weight value, and vice versa. Such an edge weight matrix can more comprehensively reflect the advantages and disadvantages of action transfer, and provide more accurate basis for subsequent path search and strategy update.

[0051] The depth deterministic policy gradient algorithm is used to traverse the node paths, and the comprehensive benefit function of each path is calculated. The depth deterministic policy gradient algorithm (DDPG) is a reinforcement learning algorithm that combines deep neural networks and deterministic policy gradients, suitable for policy optimization in continuous action space. In this step, the DDPG algorithm is used to traverse the node paths in the state space, and the core is to approximate the policy function and value function through neural networks. Specifically, first, the Actor-Critic neural network structure is constructed, where the Actor network is used to generate a deterministic policy, i.e., to output the optimal action given the current state; the Critic network is used to evaluate the value of the action, i.e., to calculate the Q value of the state-action pair. When traversing the node path, starting from the current node state, the action is selected according to the policy generated by the Actor network, and the next node state is transferred, and the corresponding reward value is obtained. Through continuous iteration of this process, all possible node paths are traversed. In the traversal process, the comprehensive benefit function of each path is calculated, and the construction method of the function is as follows: define the path benefit as a balance factor of time efficiency and decision confidence weight, where time efficiency reflects the length of path execution, and decision confidence weight reflects the reliability of path decision. Initialize the path benefit of each node to zero, and the starting point path benefit to a preset baseline value, which is usually set according to the basic requirements of device operation and historical experience. Then, the maximum cumulative benefit of each node based on the previous nodes is calculated by dynamic programming algorithm, and the optimal policy update sequence is recorded. The dynamic programming algorithm decomposes the complex path optimization problem into multiple sub-problems, and gradually solves the optimal solution of each sub-problem to finally obtain the optimal solution of the entire path. In the calculation process, the maximum cumulative benefit of each node is equal to the maximum cumulative benefit of its previous nodes plus the path benefit of the current edge, and all nodes' maximum cumulative benefits are calculated recursively in this way, and the optimal policy update sequence is determined.

[0052] According to the comprehensive benefit function, the optimal strategy update sequence is selected, and an autonomous decision control scheme is output. After calculating the comprehensive benefit function of each path, the cumulative benefit values of different paths are compared, and the path with the maximum cumulative benefit is selected as the optimal strategy update sequence. This sequence contains a series of optimal action transitions from the current state to the target state, reflecting the optimal strategy update method considering the time efficiency and decision confidence weight. According to the optimal strategy update sequence, a specific autonomous decision control scheme is generated, which contains detailed control action steps, time sequence of action execution, and specific parameter settings of each action, etc. For example, when the equipment detects a temperature overshoot anomaly, the autonomous decision control scheme may output control actions such as reducing heating power, starting cooling system, adjusting ventilation volume, etc. in turn, and specify the execution time and duration of each action. By outputting the control scheme, autonomous decision control of the industrial equipment is realized, enabling the equipment to take the optimal control strategy according to the current operating state and potential abnormal conditions, ensuring stable operation of the equipment and timely handling of abnormal conditions.

[0053] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or action from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or actions. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed or inherent to such process, method, article or device.

[0054] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for autonomous decision-making and control of industrial equipment based on reinforcement learning, characterized in that, include: A multimodal sensing module acquires a set of operational status data for industrial equipment. This operational status data includes time-series data from vibration sensors, continuous data from temperature transmitters, and discrete data from pressure gauges. The time-series data from vibration sensors contains frequency characteristics and amplitude distribution information of mechanical components, while the continuous data from temperature transmitters contains thermal conduction gradients and steady-state temperature field information. Based on the time-series data from vibration sensors, a feature space mapping algorithm is used to align the multimodal data, generating an aligned state space dimension mapping relationship. Based on the continuous data from temperature transmitters, combined with a thermodynamic state model, the thermal drift compensation amount for the equipment is calculated, generating dynamic operational correction parameters. The discrete data from the pressure gauges are subjected to adaptive threshold segmentation to extract key operating state points and abnormal event boundaries. The state space dimension mapping relationship, dynamic operation correction parameters, and key operating state points are input into a reinforcement learning decision model to generate a fusion decision result of equipment control actions and abnormal handling. Based on the fusion decision result, the exploration-utilization balance parameters of the reinforcement learning are adjusted through a policy update mechanism to output an autonomous decision control scheme. The nodes of the policy update mechanism represent the state space partitioning, and the edges represent the action selection priority and decision confidence weight.

2. The reinforcement learning-based autonomous decision-making and control method for industrial equipment according to claim 1, characterized in that, The multimodal data alignment achieved through the feature space mapping algorithm includes: The vibration sensor time-series data is subjected to time-frequency domain transformation processing to generate a state feature map of joint time-frequency distribution; Based on a predefined benchmark feature library, principal component feature vectors are extracted using the kernel principal component analysis algorithm, and the spatial transformation matrix is ​​calculated. The spatial transformation matrix is ​​optimized using a local linear embedding algorithm to generate a precise mapping relationship between the coordinate systems of the multimodal sensor. The consistency between the precise mapping relationship and the state space dimension mapping relationship is verified, and the alignment error correction parameters are output.

3. The reinforcement learning-based autonomous decision-making and control method for industrial equipment according to claim 1, characterized in that, The calculation of the thermal drift compensation amount of the equipment using the thermodynamic state model includes: The temperature field is reconstructed from the continuous data of the temperature transmitter, and the real-time temperature change curves of each component are extracted. Based on the material thermal expansion coefficient database, match the calculation formula for nonlinear drift under the current temperature change; The thermal deformation trend is simulated using a finite element thermal analysis model, and a local drift compensation coefficient is generated. By combining the nonlinear drift calculation formula with the local drift compensation coefficient, dynamic operation correction parameters are output.

4. The reinforcement learning-based autonomous decision-making and control method for industrial equipment according to claim 1, characterized in that, The adaptive threshold segmentation process for the discrete data of the pressure gauge includes: The discrete data from the pressure gauges are decoded over time to extract the pressure fluctuation distribution map and threshold mapping relationship; Based on the variance variation of the pressure fluctuation distribution map, the stable operating area and the abnormal fluctuation area are divided. The segmentation threshold range is set according to the regional stability weight, and multi-scale quantile segmentation is performed. The segmented threshold mapping relationship is subjected to morphological opening operation with the abnormal event boundary to generate smooth running state points.

5. The method for autonomous decision-making and control of industrial equipment based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning decision model includes a state encoding module and an action evaluation module, wherein the state encoding module includes: The state space dimension mapping relationship is subjected to feature dimensionality reduction calculation to generate a low-dimensional dense state representation vector; The dynamic operation correction parameters are converted into offset correction values ​​in the vector dimension and superimposed on the state representation vector; Perform time-series connections on the key operating state points to generate time constraints for abnormal events; By fusing the state representation vector with time constraints through a constraint optimization algorithm, the output device control actions and abnormal handling fusion decision results are obtained.

6. The method for autonomous decision-making and control of industrial equipment based on reinforcement learning according to claim 5, characterized in that, The motion evaluation module includes: Extract anomaly type features based on the boundary of the abnormal event, including vibration over-limit amplitude, temperature overshoot rate, and pressure drop depth; Based on the association rule base between anomaly type features and control actions, an action influence weight matrix is ​​constructed; The combined score of action weights and decision confidence is calculated using a deep Q-network algorithm to generate control strategy priority results. The control policy priority result is correlated with the action selection priority, and the edge weight matrix of the policy update mechanism is updated.

7. The method for autonomous decision-making and control of industrial equipment based on reinforcement learning according to claim 1, characterized in that, The adjustment of reinforcement learning exploration through a policy update mechanism—utilizing balancing parameters—includes: Initialize node attributes based on state space partitioning, and generate initial edge weight matrix based on decision confidence weights; The fusion decision result of the equipment control action and the abnormal handling is used as the node state, and the edge weight matrix is ​​composed of the action execution time efficiency and the decision confidence weight. The node paths are traversed using a deep deterministic strategy gradient algorithm to calculate the comprehensive benefit function for each path. The optimal strategy is selected based on the comprehensive benefit function to update the sequence, and an autonomous decision-making control scheme is output.

8. The reinforcement learning-based autonomous decision-making and control method for industrial equipment according to claim 3, characterized in that, The parameter optimization methods for the finite element thermal analysis model include: The initial mesh density and material parameter sensitivity are calculated based on the distribution of historical thermal drift data. By traversing the simulation parameter combinations using the Latin hypercube sampling algorithm, the parameter with the smallest error between the simulation result and the actual measurement data is selected. The mesh density and material parameter sensitivity are dynamically adjusted based on the error distribution to optimize the accuracy of the local drift compensation coefficient.

9. The reinforcement learning-based autonomous decision-making and control method for industrial equipment according to claim 7, characterized in that, The method for constructing the comprehensive return function includes: The path benefit is defined as a balance factor between time efficiency and decision confidence weight; Initialize the path revenue of each node to zero, and set the path revenue of the starting point to a preset baseline value; The maximum cumulative return of each node based on its predecessor nodes is calculated using a dynamic programming algorithm, and the optimal strategy update sequence is recorded. The complete decision control scheme is generated by reverse derivation of the optimal strategy update sequence.

10. The reinforcement learning-based autonomous decision-making and control method for industrial equipment according to claim 6, characterized in that, The network optimization method of the deep Q-network algorithm includes: Construct a deep neural network structure containing convolutional layers and fully connected layers. The convolutional layers are used to extract local features of the state representation vector. The network parameters are updated by storing historical state-action-reward triplets through an experience replay mechanism and batch sampling. The priority of action selection is dynamically adjusted based on the Q-value distribution of the network output to optimize the accuracy of the control strategy priority results.

Images