Factory equipment predictive maintenance scheme optimization method based on reinforcement learning

Abstract

The invention discloses a factory equipment predictive maintenance scheme optimization method based on reinforcement learning, and relates to the technical field of equipment maintenance scheme prediction, and the method comprises the steps: carrying out the fusion through employing a space-time attention mechanism based on an equipment health index, equipment operation duration and a current load level, generating an equipment state vector, constructing a reinforcement learning framework, and carrying out the optimization of a factory equipment predictive maintenance scheme. A multi-target reward function is embedded in the reinforcement learning framework, a PPO algorithm is adopted for optimization, a reinforcement learning model is generated, the reinforcement learning model is trained by using the equipment state vector, the trained reinforcement learning model is obtained, the trained reinforcement learning model is deployed to an actual industrial control unit, the latest equipment state vector is received in real time, and the final equipment state vector is obtained. And obtaining an optimal maintenance strategy. And converting the optimal maintenance strategy into a specific maintenance instruction and executing the specific maintenance instruction. Through the factory equipment predictive maintenance method, rapid response to emergencies is facilitated, greater loss is avoided, resource allocation is optimized, and enterprise competitiveness is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of equipment maintenance scheme prediction, in particular to a predictive maintenance scheme optimization method for factory equipment based on reinforcement learning. BACKGROUND

[0002] With the advent of the concept of industry and the development of Internet of Things technology, manufacturing is undergoing an unprecedented transformation, among which equipment maintenance, as a key link to ensure the stable operation of the production line, is becoming increasingly important. Predictive maintenance, as an advanced maintenance mode, has gradually attracted attention. By monitoring the equipment state in real time and analyzing the data to predict potential failures, maintenance work can be arranged at the best time to minimize downtime and maintenance costs. In recent years, with the application of big data analysis and machine learning advanced technologies, the predictive maintenance scheme has been improved, which can more accurately identify the equipment health status and provide preventive maintenance recommendations.

[0003] However, the existing equipment predictive maintenance method still needs to be improved. In terms of feature extraction, due to the high complexity and nonlinearity of equipment operation data in industrial environments, it is difficult to capture all key factors that affect the equipment health status simply relying on human experience, which may lead to the omission of important information and affect the accuracy of subsequent analysis and decision-making. Secondly, the existing prediction models are mostly based on traditional statistical methods and shallow machine learning algorithms, which have poor adaptability when dealing with complex industrial data, making it difficult to effectively identify deep fault patterns, thus limiting the prediction of equipment maintenance measures. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a predictive maintenance scheme optimization method for factory equipment based on reinforcement learning to solve the problem of difficult identification of deep fault patterns of equipment.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides a predictive maintenance scheme optimization method for factory equipment based on reinforcement learning, which comprises, collecting the operation data of the factory equipment, the equipment operation time, the current load level and the maintenance decision; using an Autoencoder model to reduce the dimension and extract features of the operation data of the factory equipment, and obtaining an equipment health index; fusing the equipment health index with the equipment operation time and the current load level to generate an equipment state vector; A reinforcement learning framework is constructed, a multi-objective reward function is embedded in the reinforcement learning framework, and a PPO algorithm is used for optimization to generate a reinforcement learning model; The reinforcement learning model is trained using the device state vector to obtain a trained reinforcement learning model; The trained reinforcement learning model is deployed in an actual industrial control unit to receive the latest device state vector in real time and obtain the best maintenance strategy; The best maintenance strategy is converted into specific maintenance instructions and executed.

[0007] As a preferred scheme of the factory equipment predictive maintenance scheme optimization method based on reinforcement learning, wherein: the operation data of the factory equipment includes temperature, pressure and vibration.

[0008] As a preferred scheme of the factory equipment predictive maintenance scheme optimization method based on reinforcement learning, wherein: the Autoencoder model is used to reduce the dimensionality and extract features of the operation data of the factory equipment, and the device health index is obtained, specifically, Collect the operation data of the historical factory as the training set and the verification set; The training set is input into the Autoencoder model to calculate the reconstruction error by forward propagation, and the weight parameters of each layer network of the encoder and the decoder in the Autoencoder model are updated using the Adam optimization algorithm according to the reconstruction error, the performance of the Autoencoder model is evaluated on the verification set to prevent overfitting, and the trained Autoencoder model is generated; The device operation data is preprocessed and input into the trained Autoencoder model, and the encoder of the Autoencoder model is used to compress the device operation data into a low-dimensional latent feature vector; The latent feature vector is reconstructed by the decoder to obtain reconstructed data; The error between the reconstructed data and the device operation data is calculated, and an exponential weighted moving average process is performed to generate a device health index.

[0009] As a preferred scheme of the factory equipment predictive maintenance scheme optimization method based on reinforcement learning, wherein: the device health index, the device operation time and the current load level are fused using a spatio-temporal attention mechanism to generate a device state vector, specifically, The device health index, the device operation time and the current load level are organized in time series to generate a three-dimensional matrix; The spatial attention analysis and the time attention analysis are performed on the three-dimensional matrix to generate a spatio-temporal enhanced matrix; Adopt the gating residual mechanism, the three-dimensional matrix and the space-time enhancement matrix are dynamically mixed, and the equipment state vector is generated.

[0010] As a preferred scheme of the factory equipment predictive maintenance scheme optimization method based on reinforcement learning, wherein: the reinforcement learning framework is constructed, the multi-objective reward function is embedded in the reinforcement learning framework, and the PPO algorithm is used for optimization to generate a reinforcement learning model, specifically, The equipment state vector is set as the state space, the maintenance decision is set as the action space, the reinforcement learning framework is generated, the policy network and the value network are constructed in the reinforcement learning framework, and the multi-objective reward function is embedded to generate a basic reinforcement learning model. The PPO algorithm is used in combination with the generalized advantage estimation method to guide the parameter optimization of the policy network and the value network of the basic reinforcement learning model to generate a final reinforcement learning model.

[0011] As a preferred scheme of the factory equipment predictive maintenance scheme optimization method based on reinforcement learning, wherein: the equipment state vector is used to train the reinforcement learning model, and the trained reinforcement learning model is obtained by inputting the equipment state vector as training data into the reinforcement learning model and training in combination with the deep Q network.

[0012] As a preferred scheme of the factory equipment predictive maintenance scheme optimization method based on reinforcement learning, wherein: the trained reinforcement learning model is deployed to an actual industrial control unit, real-time latest equipment state vectors are received, and the best maintenance strategy is obtained, specifically, The trained reinforcement learning model is accessed to the computing platform of the industrial control unit through the industrial communication protocol; The industrial control unit collects the latest equipment state vectors of the factory at a fixed sampling period and inputs the reinforcement learning model, and the reinforcement learning model generates the best maintenance strategy based on the policy network.

[0013] As a preferred scheme of the factory equipment predictive maintenance scheme optimization method based on reinforcement learning, wherein: the best maintenance strategy is converted into specific maintenance instructions and executed, specifically, The best maintenance strategy is converted into a set of maintenance instructions recognizable by the factory equipment by using an industrial control protocol parser, and is sent to the industrial control unit; After receiving the set of maintenance instructions, the industrial control unit verifies the validity of the set of maintenance instructions, and distributes the set of maintenance instructions to the corresponding execution terminal through an instruction scheduler to execute the maintenance In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the method for optimizing a predictive maintenance scheme of a factory equipment based on reinforcement learning according to the first aspect of the present application.

[0014] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of the method for optimizing a predictive maintenance scheme of a factory equipment based on reinforcement learning according to the first aspect of the present application.

[0015] The present application has the following beneficial effects: by constructing a reinforcement learning framework comprising a multi-objective reward function and using the PPO algorithm for optimization, not only the predictability and efficiency of factory equipment maintenance are greatly improved, but also the unplanned downtime is reduced, the service life of the equipment is effectively prolonged, the maintenance cost and production loss are reduced, and the goals of reducing operating costs and improving production efficiency are achieved. Finally, the trained reinforcement learning model is deployed to the industrial control unit to realize seamless connection between the reinforcement learning model and the actual operating environment, so that the industrial control unit can monitor the equipment state in real time and quickly adjust the maintenance strategy according to the latest data. This real-time greatly shortens the time interval from problem discovery to problem solving, helps to quickly respond to unexpected situations, avoids greater losses, and achieves the ultimate goal of optimizing resource allocation and enhancing enterprise competitiveness. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0017] Fig. 1 Flowchart for the method for optimizing a predictive maintenance scheme of a factory equipment based on reinforcement learning.

[0018] Fig. 2 Flowchart for reinforcement learning model construction and training.

[0019] Fig. 3 Flowchart for the equipment health index.

[0020] Fig. 4 Flowchart for executing maintenance instructions. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0022] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0023] Second, the "one embodiment" or "an embodiment" appearing in the specification herein indicates that a specific feature, structure, or characteristic can be included in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification herein does not all refer to the same embodiment, nor is it an embodiment that is mutually exclusive with other embodiments.

[0024] Reference is made to Figs. 1-4 For one embodiment of the present application, the embodiment provides a predictive maintenance scheme optimization method for factory equipment based on reinforcement learning, comprising the following steps: S1, collecting the running data of the factory equipment, the equipment running time, the current load level and the maintenance decision; The running data of the factory equipment includes temperature, pressure and vibration; It should be noted that the temperature sensor, pressure sensor and vibration sensor are used to collect the running data of the factory equipment in real time at a sampling frequency of 10 times per second, and the running data is transmitted to the data acquisition server through the industrial bus. The factory equipment running time is recorded by the PLC cumulative timer, with a precision of milliseconds, the current load level is calculated by measuring the three-phase current through the electric power transmitter, and the maintenance decision is extracted from the latest maintenance record in the MES work order database. The temperature data is measured by PT100 platinum resistance, with a range of 0-200℃, the pressure data is collected by a piezoresistive sensor, with a range of 0-10MPa, and the vibration data is measured by an IEPE acceleration sensor, with a bandwidth of 5-10kHz. After time stamp alignment and outlier elimination in the edge gateway, the running data, equipment running time, current load level and maintenance decision are packaged into JSON format and transmitted to the cloud storage.

[0025] S2, using an Autoencoder model to reduce the dimension and extract features of the running data of the factory equipment, and obtaining the equipment health index; Collecting the running data of the historical factory equipment as the training set and the verification set; It should be noted that the operation data of the historical plant equipment is extracted from the time series database to obtain temperature, pressure and vibration fields, and the time range is the last 36 months. The equipment operation data is first cleaned to remove null records caused by communication interruption, repair abnormal values caused by sensor drift, and the processed equipment operation data is randomly divided into a training set and a validation set in a ratio of 8:2. The training set contains about 12 million sampling records, covering all operating stages of the equipment throughout its life cycle, and the validation set contains 3 million records to ensure that it has 200 hours of continuous data before the equipment fails.

[0026] The training set is input into the Autoencoder model to calculate the reconstruction error by forward propagation, and the weights of each layer of the encoder and decoder in the Autoencoder model are updated using the Adam optimization algorithm based on the reconstruction error. The performance of the Autoencoder model is evaluated on the validation set to prevent overfitting, and a trained Autoencoder model is generated. It should be noted that the training set is input into the Autoencoder model for forward propagation, the encoder compresses the training set into a spatial representation, and based on the spatial representation, the decoder reconstructs the training set using a weighted hybrid loss function to calculate the reconstruction error. The weights of the fully connected layers of the encoder and decoder in the Autoencoder model are updated layer by layer using the Adam optimization algorithm based on the gradient of the reconstruction error. After each training round, the reconstruction error is calculated on the validation set, and when the validation set error does not decrease for 5 consecutive epochs, the training is terminated in advance. The weight parameters at the time of the minimum validation set error are saved as the final Autoencoder model.

[0027] The equipment operation data is preprocessed and input into the trained Autoencoder model, and the encoder of the Autoencoder model is used to compress the equipment operation data into a low-dimensional latent feature vector. It should be noted that after the equipment operation data is processed by missing value filling, Z-score standardization and sliding window normalization, it is input into the trained Autoencoder model. The encoder part of the Autoencoder model performs linear transformation and ReLU activation through three fully connected neural networks in sequence, gradually compressing the 128-dimensional operation data to a 16-dimensional latent feature vector. The first fully connected layer reduces the input dimension to 64, the second fully connected layer reduces it to 32, and the third fully connected layer outputs a 16-dimensional latent feature vector. Batch normalization layers are applied after each linear transformation to speed up convergence, and the final output latent feature vector retains more than 95% of the variance information of the equipment operation data.

[0028] The decoder is used to reconstruct the latent feature vector to obtain reconstructed data. It should be noted that the feature vector is input into the decoder part of the Autoencoder model, and the decoder is transformed through a three-layer fully connected neural network. The first layer of full connection expands the feature vector to 32 dimensions and applies LeakyReLU activation. The second layer of full connection expands to 64 dimensions and performs batch normalization processing. The third layer of full connection restores to the original 128-dimensional device operation data dimension and uses the Sigmoid activation function to constrain the output range. A Dropout layer with a proportion of 0.2 is added after each layer of transformation to prevent overfitting, and the final output is the reconstructed data.

[0029] The error between the reconstructed data and the operation data is calculated, and an exponentially weighted moving average process is performed to generate a device health index.

[0030] Further, the root mean square error between the reconstructed data and the operation data is calculated point by point, and after obtaining the 128-dimensional instantaneous error vector, the L2 norm is calculated as the original error value at the current time. An exponentially weighted moving average algorithm with a decay factor of 0.9 is used to smooth the original error value of the last 30 sampling periods, eliminating transient fluctuations. The smoothed original error value is mapped to the 0-1 interval through an S-shaped function, where 1 represents a completely healthy state and 0 represents a serious fault state. The final output value is used as the device health index, which is updated every 5 seconds and retains 3 decimal places of accuracy.

[0031] S3, based on the device health index, the device running time and the current load level, using a spatio-temporal attention mechanism for fusion, generating a device state vector; The device health index, the device running time and the current load level are organized in time series to generate a three-dimensional matrix. It should be noted that the device health index, the device running time and the current load level are synchronously collected at a fixed sampling interval (such as 1 second) to form a time-aligned data sequence. With a sliding window length of 60 seconds, the device health index, the device running time (converted from hours to seconds) and the current load level (percentage value) are arranged in time order. The device health index is used as the first dimension data column, the device running time is used as the second dimension data column, and the current load level is used as the third dimension data column, together forming a two-dimensional data slice of 60 rows x 3 columns. The continuously collected N time window data is stacked along the depth direction, and finally a three-dimensional matrix structure with a height of 60 (time points), a width of 3 (feature parameters) and a depth of N (number of time windows) is formed. The device health index, the device running time and the current load level maintain a fixed dimensional correspondence in the three-dimensional matrix, ensuring the consistency of the spatio-temporal features. Each slice of the three-dimensional matrix contains the evolution trajectory of the device health index, the device running time and the current load level within the complete time window, and the adjacent slices maintain time continuity through a 50% overlap rate.

[0032] The spatial attention analysis and the time attention analysis are performed on the three-dimensional matrix to generate a space-time enhanced matrix. It should be noted that a three-dimensional matrix composed of historical equipment health index, equipment running time and current load level is collected and input into a Transformer model for training. During the training process, the Transformer model first converts the historical three-dimensional matrix into a serialized feature vector through a linear projection layer, and adds learnable position encoding to preserve the time sequence information. In the encoder part, the multi-head self-attention mechanism calculates the mutual relationship of the historical equipment health index, the equipment running time and the current load level in the spatial dimension in parallel, and at the same time, the masking mechanism models the time dimension dependence to generate a feature representation with space-time correlation. The feedforward neural network performs nonlinear transformation on the attention output, and the residual connection and layer normalization ensure the training stability. After multiple rounds of iterative training, the Transformer model learns the space-time feature pattern of the historical three-dimensional matrix, and finally generates a trained Transformer model.

[0033] The three-dimensional matrix is input into the trained Transformer model, and the Transformer model performs cross-correlation calculation on the three feature dimensions of the equipment health index, the equipment running time and the current load level. A 3x3 spatial attention weight matrix is generated through feature vector dot product operation, wherein the main diagonal elements represent the self-correlation of each parameter (the self-correlation weight of the equipment health index is 0.85), and the non-diagonal elements reflect the interaction strength between parameters (such as the synergy coefficient of the equipment running time and the current load level is 0.62). The spatial attention weight matrix and the three-dimensional matrix are subjected to Hadamard product operation to output a space-enhanced matrix.

[0034] The three-dimensional matrix is expanded along the time dimension, and the equipment health index, the equipment running time and the current load level of 60 consecutive sampling points are analyzed respectively. At the same time, the Transformer model is used to extract time-dependent features to generate a 60-dimensional time attention vector, which highlights the equipment health index mutation point (such as the weight of the 37th time point is 0.91) and the current load level steady state interval (such as the average weight of the 15th-30th time point is 0.45). The time attention vector and the three-dimensional matrix are subjected to outer product operation to generate a time-enhanced matrix. The space-enhanced matrix and the time-enhanced matrix are multiplied by matrix to generate a space-time enhanced matrix, and are subjected to layer normalization processing. Finally, the dimension of the space-time enhanced matrix remains consistent with that of the input three-dimensional matrix, which is 60x3xN structure.

[0035] The three-dimensional matrix and the space-time enhanced matrix are dynamically mixed by using the gated residual mechanism to generate a device state vector.

[0036] Further, the three-dimensional matrix and the spatio-temporal enhancement matrix are input into a gated residual mechanism. The gated residual mechanism first performs linear transformation on the three-dimensional matrix and the spatio-temporal enhancement matrix respectively, sums the linear transformation results, and generates a dynamic gating weight through a sigmoid function. The three-dimensional matrix and the dynamic gating weight are multiplied element by element to obtain a weighted three-dimensional matrix, and the spatio-temporal enhancement matrix and the dynamic gating weight are multiplied element by element to obtain a weighted spatio-temporal enhancement matrix. The weighted three-dimensional matrix and the weighted spatio-temporal enhancement matrix are summed to form a preliminary mixed feature, and the preliminary mixed feature is nonlinearly transformed through a 1x1 convolution and then connected in residual with the three-dimensional matrix. Finally, the device state vector with spatio-temporal perception characteristics is output.

[0037] S4, a reinforcement learning framework is constructed, a multi-objective reward function is embedded in the reinforcement learning framework, and a PPO algorithm is used for optimization to generate a reinforcement learning model; The device state vector is set as the state space, the maintenance decision is set as the action space, a reinforcement learning framework is generated, a policy network and a value network are constructed in the reinforcement learning framework, and a multi-objective reward function is embedded to generate a basic reinforcement learning model; It should be noted that the device state vector constitutes the state space, the maintenance decision constitutes the action space, and a reinforcement learning framework based on Markov decision process is established. The policy network of the reinforcement learning framework adopts a deep neural network structure, inputs the device state vector and outputs the probability distribution of the maintenance decision; the value network adopts a deep neural network structure, inputs the device state vector and the maintenance decision and outputs the state-action value estimate. A multi-objective reward function including the device reliability index, the maintenance cost coefficient and the production efficiency is constructed, and a reward signal is generated by linear weighting. A basic reinforcement learning model is generated. The multi-objective reward function design method is to define the device reliability index as the negative logarithmic relationship of the device health index and the failure probability, to calculate the maintenance cost coefficient based on the working hours and spare parts fees consumed by the maintenance action, and to determine the production efficiency weight jointly by the device availability and the production plan completion degree; then a standardization process is established to normalize the device reliability index, the maintenance cost coefficient and the production efficiency to the same dimension, and the dynamic weight coefficients of the device reliability index, the maintenance cost coefficient and the production efficiency are calculated by using the entropy weight method, and finally the multi-objective reward function is obtained, and the specific expression is, ; Wherein, represents the comprehensive reward value, represents the device reliability index weight, represents the maintenance cost coefficient weight, represents the production efficiency weight; The PPO algorithm is used in combination with the generalized advantage estimation method to guide the parameter optimization of the policy network and the value network of the basic model, and a final reinforcement learning model is generated; Further, the PPO algorithm constrains the parameter update range of the base model policy network through a policy loss function, maintains training stability using an importance sampling ratio clipping mechanism, and the policy network outputs a probability distribution based on the device state vector to maintain the decision and collect trajectory data. The generalized advantage estimation method processes the cumulative return time series data generated by the multi-objective reward function to calculate the advantage function value to guide the gradient update direction of the policy network, and optimizes the evaluation accuracy of the value network for state-action value. The value network receives the device state vector and the maintenance decision, outputs the value estimate, and calculates the mean square error loss with the generalized advantage estimation result to update the parameters. The policy network and the value network are alternately optimized, the policy network explores the efficient maintenance decision, the value network calibrates the state value evaluation, the multi-objective reward function continuously provides the optimization signal, and the final reinforcement learning model is generated.

[0038] S5, training the reinforcement learning model using the device state vector to obtain a trained reinforcement learning model; The device state vector is input into the reinforcement learning model as training data, and the reinforcement learning model is trained in combination with the deep Q network. During the training process, the reinforcement learning model continuously adjusts the network parameters, so that the mapping relationship from the device state vector to the action decision is gradually optimized. After N iterations, the trained reinforcement learning model is obtained. It should be noted that during the training process, the device state vector is input into the reinforcement learning model, and the reinforcement learning model updates the parameters in combination with the deep Q network. The deep Q network receives the device state vector and predicts the Q value of each maintenance decision. Based on the predicted Q value of each maintenance decision and the reward signal, the target Q value is calculated through the Bellman equation, and the target Q value is compared with the predicted Q value to generate a mean square error loss function. The gradient descent algorithm is used to calculate the gradient of the loss function with respect to the parameters of the deep Q network and update the parameters through backpropagation, so that the mapping relationship from the device state vector to the maintenance decision Q value is gradually optimized. After N iterations of training, the deep Q network can accurately predict the Q value distribution of each maintenance action based on the device state vector, and the optimal strategy is formed by selecting the maintenance action corresponding to the maximum Q value. Finally, the trained reinforcement learning model is obtained.

[0039] S6, deploying the trained reinforcement learning model to an actual industrial control unit to receive the latest device state vector in real time and obtain the best maintenance strategy. The trained reinforcement learning model is accessed to the computing platform of the industrial control unit through the industrial communication protocol. It should be noted that the trained reinforcement learning model is physically connected and logically communicated with the computing platform of the industrial control unit through the industrial communication protocol. The computing platform of the industrial control unit is configured with a special interface adapter for parsing the data frame format transmitted by the industrial communication protocol. The trained reinforcement learning model is deployed in the designated memory area of the computing platform of the industrial control unit, and data exchange is realized through memory mapping with the industrial communication protocol driver layer.

[0040] The industrial control unit collects the latest device state vector of the factory at a fixed sampling period and inputs it into the reinforcement learning model, and the reinforcement learning model generates the optimal maintenance strategy based on the policy network. Further, the industrial control unit collects the latest device state vector from the factory device sensor network at a fixed sampling period, and transmits the device state vector to the reinforcement learning model input interface through the industrial communication protocol. After receiving the device state vector, the policy network of the reinforcement learning model performs feature extraction and state coding on the device state vector, and performs forward propagation calculation on each maintenance decision to obtain the probability selection. The maintenance decision with the highest probability is selected as the optimal maintenance strategy.

[0041] S7, converting the optimal maintenance strategy into specific maintenance instructions and executing them; The optimal maintenance strategy is converted into a set of maintenance instructions recognizable by the factory device using an industrial control protocol parser, and is sent to the industrial control unit. It should be noted that the optimal maintenance strategy output by the reinforcement learning model is transmitted to the industrial control protocol parser in a structured data format. The industrial control protocol parser decomposes the optimal maintenance strategy into a set of maintenance instructions, and assembles the set of maintenance instructions into standard industrial control protocol messages with address code, function code and data field according to the communication specification of the target factory device. The industrial control protocol parser sends the protocol message to the industrial control unit through the physical interface.

[0042] After receiving the set of maintenance instructions, the industrial control unit verifies the validity of the set of maintenance instructions, and distributes them to the corresponding execution terminal through the instruction scheduler for execution of maintenance. Further, after receiving the maintenance instruction set, the industrial control unit first verifies the format integrity and semantic compliance of the maintenance instruction set, checks whether the instruction operation code, device address and parameter range conform to the predefined protocol specification, and after verification, submits the maintenance instruction set to the instruction scheduler, which analyzes the physical location and communication path of the instruction target terminal according to the factory device topology table; the instruction scheduler distributes the maintenance instruction set to the corresponding execution terminal through the industrial control network according to the maintenance instruction priority and the real-time requirement of the device state; after receiving the maintenance instruction set, the execution terminal drives the servo mechanism and relay to complete the specific maintenance action, and at the same time, the industrial control unit collects the feedback signal of each execution terminal through the state monitoring channel to ensure that the maintenance instruction set is correctly executed.

[0043] The embodiment also provides a computer device suitable for the case of the predictive maintenance scheme optimization method for factory equipment based on reinforcement learning, which comprises a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the predictive maintenance scheme optimization method for factory equipment based on reinforcement learning proposed in the above embodiment.

[0044] The computer device can be a terminal, which comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0045] The embodiment also provides a storage medium on which a computer program is stored, the program being executed by a processor to implement the method for optimizing a predictive maintenance scheme of a factory equipment based on reinforcement learning proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0046] To sum up, the present application, by constructing a reinforcement learning framework containing a multi-objective reward function and using a PPO algorithm for optimization, not only greatly improves the predictability and efficiency of factory equipment maintenance, reduces unplanned downtime, but also effectively prolongs the service life of the equipment, reduces maintenance costs and production losses, and achieves the goal of reducing operating costs and improving production efficiency. Finally, the trained reinforcement learning model is deployed to an industrial control unit to realize seamless connection of the reinforcement learning model and the actual operating environment, so that the industrial control unit can monitor the equipment state in real time and quickly adjust the maintenance strategy according to the latest data. This real-time nature greatly shortens the time interval from problem discovery to problem solving, helps to quickly respond to unexpected situations, avoids greater losses, and achieves the ultimate goal of optimizing resource allocation and enhancing enterprise competitiveness.

[0047] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and all modifications or replacements should be covered in the scope of the claims of the present application.

Claims

1. A method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning, characterized in that: include, Collect operational data of factory equipment, equipment runtime, current load level, and maintenance decisions; The Autoencoder model is used to reduce the dimensionality and extract features from the operating data of factory equipment to obtain the equipment health index. A spatiotemporal attention mechanism is used to fuse device health index, device runtime and current load level to generate device state vector; A reinforcement learning framework is constructed, and a multi-objective reward function is embedded in the framework. The PPO algorithm is then used for optimization to generate a reinforcement learning model. Train a reinforcement learning model using device state vectors to obtain the trained reinforcement learning model; The trained reinforcement learning model is deployed to the actual industrial control unit to receive the latest equipment state vector in real time and obtain the best maintenance strategy. Translate the best maintenance strategy into specific maintenance instructions and execute them.

2. The method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning as described in claim 1, characterized in that: The operating data of the factory equipment includes temperature, pressure, and vibration.

3. The method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning as described in claim 2, characterized in that: The method involves using an Autoencoder model to perform dimensionality reduction and feature extraction on the factory equipment's operational data to obtain an equipment health index. Specifically... Collect historical factory equipment operation data as training and validation sets; The training set is input into the Autoencoder model for forward propagation to calculate the reconstruction error. Based on the reconstruction error, the Adam optimization algorithm is used to update the weight parameters of each layer of the encoder and decoder network in the Autoencoder model. The performance of the Autoencoder model is evaluated on the validation set to prevent overfitting and generate the trained Autoencoder model. The device operation data is preprocessed and input into the trained Autoencoder model. The encoder of the Autoencoder model is used to compress the device operation data into a low-dimensional latent feature vector. The latent feature vectors are reconstructed using a decoder to obtain the reconstructed data; The error between the reconstructed data and the equipment operation data is calculated, and an exponentially weighted moving average is applied to generate the equipment health index.

4. The method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning as described in claim 3, characterized in that: The device state vector is generated by fusing data based on device health index, device runtime, and current load level using a spatiotemporal attention mechanism. The equipment health index, equipment runtime, and current load level are organized according to time series to generate a three-dimensional matrix; Perform spatial and temporal attention analysis on the 3D matrix to generate a spatiotemporal enhancement matrix; A gated residual mechanism is used to dynamically mix the three-dimensional matrix and the spatiotemporal enhancement matrix to generate the device state vector.

5. The method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning as described in claim 4, characterized in that: The process involves constructing a reinforcement learning framework, embedding a multi-objective reward function within the framework, and optimizing it using the PPO algorithm to generate a reinforcement learning model. Specifically... The device state vector is set as the state space, and the maintenance decision is set as the action space. A reinforcement learning framework is generated, and a policy network and a value network are constructed in the reinforcement learning framework. A multi-objective reward function is embedded to generate a basic reinforcement learning model. The PPO algorithm, combined with the generalized advantage estimation method, is used to guide the parameter optimization of the policy network and value network of the basic reinforcement learning model, generating the final reinforcement learning model.

6. The method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning as described in claim 5, characterized in that: The process of training a reinforcement learning model using device state vectors and obtaining the trained reinforcement learning model involves inputting the device state vectors as training data into the reinforcement learning model and training it using a deep Q-network. During the training process, the reinforcement learning model continuously adjusts the network parameters to gradually optimize the mapping relationship between the device state vectors and action decisions. After N iterations, the trained reinforcement learning model is obtained.

7. The method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning as described in claim 6, characterized in that: The process involves deploying the trained reinforcement learning model to the actual industrial control unit, receiving the latest equipment state vector in real time, and obtaining the optimal maintenance strategy. The trained reinforcement learning model is connected to the computing platform of the industrial control unit via an industrial communication protocol; The industrial control unit collects the latest equipment state vectors in the factory at a fixed sampling period and inputs them into the reinforcement learning model. The reinforcement learning model generates the best maintenance strategy based on the policy network.

8. The method for optimizing predictive maintenance schemes for factory equipment based on reinforcement learning as described in claim 7, characterized in that: The process of translating the optimal maintenance strategy into specific maintenance instructions and executing them involves, in particular, The optimal maintenance strategy is converted into a set of maintenance instructions that the factory equipment can recognize using an industrial control protocol parser and then sent to the industrial control unit. After receiving the maintenance instruction set, the industrial control unit verifies the validity of the maintenance instruction set and distributes it to the corresponding execution terminal for maintenance through the instruction scheduler.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the reinforcement learning-based predictive maintenance scheme optimization method for factory equipment as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the reinforcement learning-based predictive maintenance scheme optimization method for factory equipment as described in any one of claims 1 to 8.

Images