Method and device for predicting residual life of equipment and electronic equipment

Abstract

The invention provides an equipment residual life prediction method and device and electronic equipment, and the method comprises the steps: carrying out the feature extraction of multi-modal data of target equipment, and obtaining a plurality of modal features; performing weight calculation on the plurality of modal features by using a time attention mechanism model to obtain weights of the plurality of modal features; performing feature fusion on the plurality of modal features according to the weights of the plurality of modal features to obtain fusion features; and predicting the residual life of the target equipment according to the fusion features. In the implementation process of the scheme, due to the fact that multi-modal data overcomes the defect that information of a single data source is insufficient, and the time attention mechanism model can perform weighting processing on important features of different time points, feature weights are dynamically distributed, key features of the important time points are highlighted, and the accuracy of the scheme is improved. Therefore, the sensitivity to the key information in the fusion features is improved, and the prediction accuracy of the residual life of the equipment is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of reliability engineering and deep learning, and specifically, to a method, device and electronic device for predicting the remaining life of equipment. Background Art

[0002] At present, the prediction of the remaining life of equipment is mostly based on a single data source. For example, multiple sensors of the same type are sampled to collect data from the target equipment, multiple sensor data are obtained, and the remaining life of the target equipment is predicted based on the multiple sensor data. In the specific practice process, it is found that the single data source has the problem of insufficient information, resulting in a low accuracy rate in the prediction of the remaining life of the equipment. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device and electronic device for predicting the remaining life of equipment, so as to improve the problem of low prediction accuracy of the remaining life of equipment.

[0004] The embodiment of the present application provides a method for predicting the remaining life of a device, including: extracting features from the multimodal data of the target device to obtain multiple modal features; using a time attention mechanism model to calculate the weights of the multiple modal features to obtain the weights of the multiple modal features; fusing the multiple modal features according to the weights of the multiple modal features to obtain the fused features; and predicting the remaining life of the target device according to the fused features. In the implementation process of the above scheme, the modal features extracted from the multimodal data of the target device are weighted by using the time attention mechanism model to obtain the weights of the multiple modal features, and after fusing the multiple modal features according to the weights of the multiple modal features, the remaining life of the target device is predicted according to the fused features. Since the multimodal data overcomes the defect of insufficient information of a single data source, and the time attention mechanism model can perform weighted processing on the important features at different time points, dynamically allocate feature weights and highlight the key features of important time points, thereby improving the sensitivity to the key information in the fused features, and effectively improving the prediction accuracy of the remaining life of the device.

[0005] Optionally, in an embodiment of the present application, multimodal data includes: visible light image time series data, vibration signal time series data and / or temperature time series data; feature extraction is performed on the multimodal data of the target device, including: using a convolutional neural network to extract features from visible light image time series data; and / or, using a time convolutional network to extract features from vibration signal time series data; and / or, using a long short-term memory network to extract features from temperature time series data. In the implementation process of the above scheme, each network model is optimized for the characteristics of the corresponding data to ensure the accuracy and efficiency of feature extraction. This highly targeted feature extraction method greatly improves the accuracy of feature extraction, enables the model to fully utilize the feature information of different data sources, and improves the accuracy and reliability of remaining life prediction.

[0006] Optionally, in an embodiment of the present application, before extracting features from the multimodal data of the target device, it also includes: image denoising, image enhancement, image cropping and / or image scaling of the visible light image time series data; and / or, removing trend items, noise reduction, time domain signal extraction and / or frequency domain signal extraction of the vibration signal time series data; and / or, denoising, missing data processing and / or outlier processing of the temperature time series data; and / or, normalizing or standardizing the vibration signal time series data and / or the temperature time series data. In the implementation process of the above scheme, by combining the visible light image time series data, the vibration signal time series data and / or the temperature time series data, not only can the operating status of the device be more comprehensively reflected, but also the accuracy and reliability of the prediction can be improved. Data of multiple different modalities are often complementary. Through fusion, the advantages of different modal data can be utilized to make up for the shortcomings of single modal data. Multimodal data provides multiple information perspectives so that the model can extract more comprehensive and multi-dimensional information, thereby improving the prediction performance and robustness of the model. Furthermore, the use of the above-mentioned multimodal data can improve the robustness of the system. Even if the data quality of a certain modality is low or missing, the data of other modalities can provide supplementary information, thereby maintaining the stability and reliability of the prediction.

[0007] Optionally, in an embodiment of the present application, multiple modal features are feature fused according to their weights, including: weighting multiple modal features according to their weights to obtain multiple weighted feature representations; using a temporal attention mechanism model to perform feature fusion on multiple weighted feature representations to obtain fused features. In the implementation of the above scheme, by introducing a temporal attention mechanism, features at different time points are dynamically weighted. This mechanism can adaptively identify and enhance the most important time point features for the task, effectively improving the quality of feature fusion. The application of this temporal attention mechanism enables the model to perform better in capturing time-dependent features, so that it can automatically focus on key time points, enhance sensitivity to changes in equipment status, and further improve the accuracy of equipment remaining life prediction.

[0008] Optionally, in an embodiment of the present application, a time attention mechanism model is used to perform feature fusion on multiple weighted feature representations, including: splicing multiple weighted feature representations to obtain spliced ​​feature representations; using the time attention mechanism model to perform feature fusion on the spliced ​​feature representations to obtain fused features. In the implementation process of the above scheme, the time attention mechanism can highlight which time points or time periods are most important for the prediction results, thereby providing explainability for the model's decision. Weighting and splicing allow the model to distinguish the importance of different features, further enhancing the model's ability to focus on specific features. Since different features may contribute to different degrees to the prediction at different time points, the time attention mechanism can automatically adjust the degree of attention to these features, thereby capturing more complex patterns. Because the spliced ​​feature representation contains more dimensional information, the time attention mechanism model can understand the data from a more comprehensive perspective, thereby improving the accuracy of the remaining life prediction of the equipment.

[0009] Optionally, in an embodiment of the present application, the remaining life of the target device is predicted based on the fused features, including: using a fully connected network model to predict the fused features to obtain the remaining life of the target device. In the implementation process of the above scheme, the complex nonlinear relationship between the input features is captured by the activation function in the fully connected layer, which is very critical for nonlinear problems such as predicting the remaining life, so that high-level abstract features can be automatically learned. Even if the input is a low-level fused feature, a more meaningful pattern for prediction can be extracted from it, effectively improving the prediction accuracy of the remaining life of the device.

[0010] Optionally, in an embodiment of the present application, after predicting the remaining life of the target device based on the fusion features, it also includes: reducing the dimensions of multiple modal features to obtain modal features after dimensionality reduction; and visually analyzing the modal features after dimensionality reduction, the fusion features, and the remaining life of the target device. In the implementation process of the above scheme, by visually analyzing the modal features after dimensionality reduction, the fusion features, and the remaining life of the target device, the distribution of the real RUL and the predicted RUL and their differences can be intuitively observed, which not only helps to debug and improve the model, but also enhances the interpretability of the model, allowing users to trust the prediction results of the model more.

[0011] The embodiment of the present application also provides a device for predicting the remaining life of an equipment, including: a modal feature extraction module, used to extract features from the multimodal data of the target equipment to obtain multiple modal features; a feature weight calculation module, used to use a temporal attention mechanism model to perform weight calculation on multiple modal features to obtain weights of multiple modal features; a modal feature fusion module, used to perform feature fusion on multiple modal features according to the weights of multiple modal features to obtain fused features; and a remaining life prediction module, used to predict the remaining life of the target equipment based on the fused features.

[0012] Optionally, in an embodiment of the present application, multimodal data includes: visible light image time series data, vibration signal time series data and / or temperature time series data; the modal feature extraction module includes: an image data feature extraction submodule for performing feature extraction on visible light image time series data using a convolutional neural network; and / or a vibration signal feature extraction submodule for performing feature extraction on vibration signal time series data using a time convolutional network; and / or a temperature data feature extraction submodule for performing feature extraction on temperature time series data using a long short-term memory network.

[0013] Optionally, in an embodiment of the present application, the modal feature extraction module also includes: an image data preprocessing submodule, which is used to perform image denoising, image enhancement, image cropping and / or image scaling on visible light image time series data; and / or, a vibration signal preprocessing submodule, which is used to remove trend items, perform noise reduction processing, time domain signal extraction and / or frequency domain signal extraction on vibration signal time series data; and / or, a temperature data preprocessing submodule, which is used to perform denoising, missing data processing and / or outlier processing on temperature time series data; and / or, a normalization and standardization processing submodule, which is used to perform normalization or standardization processing on vibration signal time series data and / or temperature time series data.

[0014] Optionally, in an embodiment of the present application, the modal feature fusion module includes: a modal feature weighted processing sub-module, used to perform weighted processing on multiple modal features according to the weights of the multiple modal features to obtain multiple weighted feature representations; a weighted feature representation fusion sub-module, used to use a temporal attention mechanism model to perform feature fusion on multiple weighted feature representations to obtain fused features.

[0015] Optionally, in an embodiment of the present application, the weighted feature representation fusion submodule includes: a feature representation splicing unit, used to splice multiple weighted feature representations to obtain a spliced ​​feature representation; a splicing feature fusion unit, used to use a temporal attention mechanism model to perform feature fusion on the spliced ​​feature representations to obtain a fused feature.

[0016] Optionally, in an embodiment of the present application, the remaining life prediction module includes: a fusion feature prediction submodule, which is used to predict the fusion features using a fully connected network model to obtain the remaining life of the target device.

[0017] Optionally, in an embodiment of the present application, the equipment remaining life prediction device also includes: a model feature dimensionality reduction module, which is used to reduce the dimensions of multiple modal features to obtain modal features after dimensionality reduction; a feature visualization analysis module, which is used to perform visualization analysis on the modal features after dimensionality reduction, fusion features and the remaining life of the target equipment.

[0018] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0019] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0020] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1A schematic diagram of a process flow of a method for predicting the remaining life of equipment provided in an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of a multimodal data processing process provided by an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of the structure of a temporal convolutional network provided in an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of the structure of a long short-term memory network provided by an embodiment of the present application is shown;

[0026] Figure 5 A schematic diagram of the structure of a device for predicting the remaining life of equipment provided in an embodiment of the present application is shown;

[0027] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0029] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed, but merely represents the selected embodiments of the embodiments of the present application.

[0030] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0031] Before introducing the equipment remaining life prediction method provided in the embodiment of the present application, some concepts involved in the embodiment of the present application are first introduced:

[0032] The remaining useful life (RUL) of a device refers to the length of time from the current point in time until the device or its components reach a predetermined failure state or require maintenance, repair or replacement. RUL is a forward-looking metric that attempts to quantify how long the device can operate normally without any additional maintenance or intervention.

[0033] It should be noted that the device remaining life prediction method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant, or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and a non-x86 server includes a mainframe, a minicomputer, and a UNIX server.

[0034] The following is an example of an application scenario in which the equipment remaining life prediction method is applicable: the equipment remaining life prediction method is used to predict the remaining life of the equipment in the target factory, that is, to predict the remaining time of the equipment failure. The target factory includes but is not limited to: oil refineries, machinery manufacturing plants, power plants, product processing plants, shoe factories and / or plastic processing plants, etc.

[0035] See also Figure 1The flowchart of the device remaining life prediction method provided by the embodiment of the present application is shown; the main idea of ​​the device remaining life prediction method is to calculate the weights of multiple modal features by using the temporal attention mechanism model to extract the modal features of the target device from the multimodal data, obtain the weights of multiple modal features, and after fusing the multiple modal features according to the weights of the multiple modal features, predict the remaining life of the target device according to the fused features. The implementation of the above-mentioned device remaining life prediction method may include:

[0036] Step S110: extracting features from the multimodal data of the target device to obtain multiple modal features.

[0037] The target device mentioned above can be any device in the factory, or any component of the device, such as a bearing or a motor. The multimodal data mentioned above refers to a collection of data from different sources and different types, which together describe the operating status, performance and health of the device, and can include: sensor data, visual data, text data, audio data, radio frequency identification data and other data types. For ease of understanding and explanation, the following uses visible light image time series data, vibration signal time series data and / or temperature time series data as examples for explanation.

[0038] Step S120: Use the temporal attention mechanism model to calculate the weights of multiple modal features to obtain the weights of the multiple modal features.

[0039] It is understandable that since each modal data in the multimodal data is time series data, the extracted modal features are also time-series related features. Therefore, the temporal attention mechanism model used above can learn the weights of the key modal features of the equipment as it degrades over time until its remaining life is exhausted. Among them, the above-mentioned degradation refers to the phenomenon that the performance of the equipment gradually decreases over time, which is usually caused by wear, fatigue, corrosion, aging or other forms of physical or chemical changes. The degradation process can be regarded as the change process of the equipment from the initial state (usually the best or new state) to the final failure state.

[0040] Step S130: performing feature fusion on the multiple modal features according to the weights of the multiple modal features to obtain fused features.

[0041] Step S140: predicting the remaining life of the target device based on the fused features.

[0042] In the implementation process of the above scheme, the modal features extracted from the multimodal data of the target device are weighted by using the temporal attention mechanism model to obtain the weights of multiple modal features, and after the multiple modal features are fused according to the weights of the multiple modal features, the remaining life of the target device is predicted based on the fused features. Since multimodal data overcomes the defect of insufficient information of a single data source, and the temporal attention mechanism model can weight the important features at different time points, dynamically allocate feature weights and highlight the key features of important time points, the sensitivity to the key information in the fused features is improved, and the prediction accuracy of the remaining life of the equipment is effectively improved.

[0043] See also Figure 2 The schematic diagram of the processing process of multimodal data provided by the embodiment of the present application is shown; as an optional implementation of the above step S110, the above multimodal data may include: visible light image time series data, vibration signal time series data and / or temperature time series data; the above implementation of feature extraction of multimodal data of the target device may include:

[0044] Step S111: Use a convolutional neural network to extract features from visible light image time series data.

[0045] It is understandable that before extracting features from the multimodal data of the target device, multimodal data can also be collected. The above-mentioned visible light image time series data can be collected by a high-definition camera in the horizontal direction or by a high-definition camera in the vertical direction. The high-definition camera is a camera with high resolution, preset sampling frequency and frame rate. The vibration signal time series data and / or temperature time series data can be collected by a variety of sensors. The above-mentioned camera can be fixedly installed at a position where the surface of the target device (such as a bearing) can be clearly photographed. A stable light source can also be set in the shooting environment to avoid the influence of light changes on the image quality. Specifically, a ring light or other uniform light source can be used to illuminate the bearing.

[0046] The implementation method of the above step S111 is, for example: inputting the visible light image time series data into the convolutional neural network to form a feature map with a matrix size of w, a convolution kernel size of k, a stride of s, and a number of padding layers of p, that is, the feature map after convolution can be expressed as Among them, M represents the feature map after convolution, w represents the matrix size of the feature map, p represents the number of padding layers of the feature map, k represents the convolution kernel size, and s represents the stride of the convolution. The convolution layer in the above convolutional neural network contains several convolution units, which can adaptively extract data features at the same time. The above convolution operation can choose continuous convolution or discrete convolution.

[0047] Optionally, in the specific practice, multiple pooling layers in the above-mentioned convolutional neural network can also be used to filter information on the feature map. The pooling process of this pooling layer can be expressed as in, represents the result of the lth pooling layer pooling the jth feature map, f is the pooling operation of the pooling layer, is the weight value of the connection between the jth feature map and the lth pooling layer, down is the sampling function, It represents the result of pooling the jth feature map by the l-1th pooling layer. is the bias term of the lth pooling layer relative to the jth feature map. There are two common methods in the pooling process: maximum pooling and average pooling. Maximum pooling refers to selecting the maximum value of the feature area as the pooled value of the area; average pooling refers to taking the average value of the preferred area as the pooled value.

[0048] And / or, the above-mentioned implementation method of extracting features from multimodal data of the target device may include:

[0049] Step S112: Use a time convolutional network to extract features from the vibration signal time series data.

[0050] The above vibration signal time series data can be collected by using sensors such as high-frequency accelerometers. During the collection of vibration signal time series data, it can also be ensured that its frequency response range and sensitivity are suitable for monitoring bearing vibration characteristics. The high-frequency accelerometer can be installed in the horizontal and vertical directions of the bearing to capture vibration signals in different directions. The above high-frequency accelerometer is in close contact with the target device being measured and is firmly fixed to avoid measurement errors.

[0051] See also Figure 3 The structural schematic diagram of the time convolution network provided by the embodiment of the present application is shown; the time convolutional network (TCN) is a neural network that uses void causal convolution as the core operation for time series data, wherein the void causal convolution is shown in the right branch in the figure, and the void causal convolution may include an input layer, a hidden layer, and an output layer, and the input layer and the hidden layer may be filled. In addition, the time convolution network also includes a 1×1 convolution (i.e., the point-by-point convolution layer on the left side of the figure), a void causal convolution layer, a weight normalization layer, an activation layer, and a random inactivation layer (dropout). Because the time series causal relationship and receptive field of the void causal convolution are superior to ordinary convolution, it can capture long-term dependencies and avoid introducing future information. Compared with ordinary convolution, the void causal convolution operation expands the convolution range without increasing the number of parameters; compared with causal convolution, the output of the void causal convolution can trace back a longer time interval.

[0052] The implementation method of the above step S112 is, for example, to use the hole causal convolution operation function in the temporal convolutional network: Feature extraction is performed on the vibration signal time series data; wherein F(c) represents the operation result of the void causal convolution operation function, u represents the time series vector represented by the input vibration signal time series data, * is the convolution operator, and u c ∈{u 1 ,u 2 ,u 3 ,…,u t-1} represents the cth element in the vibration signal time series data, f represents the convolution kernel, f(i) represents the i-th element value of the convolution kernel f, k is the filter size, d is the void rate, u c-d·i Represents the value of the element at cd·i in the time series vector u. The above-mentioned temporal convolutional network also includes a residual block. The residual connection sums the information before and after feature extraction to solve the problem of gradient dissipation in the back propagation process after the network is deepened.

[0053] And / or, the above-mentioned implementation method of extracting features from multimodal data of the target device may include:

[0054] Step S113: Use the long short-term memory network to extract features from the temperature time series data.

[0055] See also Figure 4 The schematic diagram of the structure of the long short-term memory network provided by the embodiment of the present application is shown; the implementation method of the above step S113 is, for example: using a long short-term memory (LSTM) network to extract features from temperature time series data; wherein LSTM is a variant of a recurrent neural network (RNN), which can effectively solve the gradient explosion or gradient disappearance problem of a simple recurrent neural network. LSTM controls the speed of information accumulation by introducing a gating mechanism based on the RNN network, including selectively adding new information and selectively forgetting previously accumulated information. LSTM introduces concepts such as long-term memory units, input gates, forget gates, output gates, and short-term memory units to shorten the running time of the entire network model and reduce errors. C t , S t are the long-term memory unit and short-term memory unit at time t respectively; C t is the candidate long-term memory unit at time t, f t is the forget gate unit at time t; u t is the input gate unit at time t; o t is the output gate unit at time t; a t is the input at time t; b tis the output at time t; Sigmoid(·) and Tanh(·) are commonly used activation functions

[0056] The above temperature time series data can be collected by a high-precision temperature sensor (such as a thermocouple) to ensure that its measurement range and accuracy are suitable for the operating temperature range of the target device (such as a bearing). Specifically, the temperature sensor can be installed at a key position near the bearing to ensure that the operating temperature of the bearing can be accurately reflected. In the specific practice, the sampling frequency of the data acquisition device can also be set, and the sampling interval can be determined according to the characteristics of the temperature change (such as sampling once per second or once every ten seconds).

[0057] During the operation of the equipment, the visible light image time series data, vibration signal time series data and temperature time series data are collected simultaneously. This kind of simultaneous acquisition of multi-modal data is currently relatively rare. The comprehensive application of this multi-modal data improves the accuracy of feature extraction and the accuracy of equipment remaining life prediction, enhances the robustness of the model in different environments, and effectively solves the problem of data scarcity.

[0058] As an optional implementation of the above step S111, before extracting features from the multimodal data of the target device, the multimodal data may be preprocessed. Data preprocessing is a crucial step in the data analysis and modeling process, and its purpose is to process the original data by cleaning and normalizing. The implementation of the above preprocessing may include:

[0059] Step S111a: performing image denoising, image enhancement, image cropping and / or image scaling on the visible light image time series data.

[0060] The implementation method of the above step S111a is, for example: using Gaussian filtering and median filtering techniques to perform image denoising on the visible light image time series data to retain important image details. Using histogram equalization, contrast enhancement, random rotation, flipping, scaling and translation and other techniques to enhance the image visibility and highlight the key features of the target device (such as bearings). Perform image cropping and / or image scaling on the visible light image time series data to scale the image to a uniform size so that it can be input into a convolutional neural network for feature extraction. Optionally, the visible light image time series data can also be converted into a grayscale image. The bearing images usually collected are in color, also known as RGB, containing 3 channels R (red), G (green) and B (blue), and the value of each channel is between 0 and 255. Specifically, the grayscale image can be obtained by performing weight calculation on the red, green and blue components, that is, each color component is multiplied by a weight value. The most representative formula is: I=0.299R+0.687G+0.114B; wherein I represents the grayscale image, R represents the red component, G represents the green component, and B represents the blue component.

[0061] And / or, step S111b: removing trend items, performing noise reduction processing, time domain signal extraction and / or frequency domain signal extraction on the vibration signal time series data.

[0062] The implementation method of the above step S111b is, for example: the vibration signal data can be processed by removing trend items using moving average, difference, etc., to remove possible linear or nonlinear trends, so as to more accurately extract periodic features. Use filters (such as wavelet transform, Fourier transform, low-pass filtering, high-pass filtering, band-pass filtering) to denoise the vibration signal time series data. Select the vibration signal time series data under different operating conditions, extract the time domain signal from the vibration signal time series data, and extract the time domain features from the vibration signal time domain data. Finally, use fast Fourier transform to convert the vibration signal time series data to obtain a frequency domain signal, and extract the frequency domain features from the frequency domain signal.

[0063] And / or, step S111c: performing denoising, missing data processing and / or outlier processing on the temperature time series data.

[0064] The implementation method of the above step S111c is, for example: the temperature time series data can be denoised and smoothed using a sliding average filter or a Kalman filter to remove high-frequency noise. If there are missing values ​​in the temperature data, the temperature time series data can be filled using interpolation methods such as linear interpolation and spline interpolation to complete the missing data processing, or directly discard the temperature time series data fragments with more missing values. When collecting temperature data information, it may be affected by unexpected factors, causing some data to deviate from the true value and abnormal valley-peak fluctuations to occur. Such data points are called outliers. Therefore, in the specific practice process, the 3σ criterion can be used to judge the outliers of the data set. If it is confirmed to be an outlier, the mean of the two adjacent data of the outlier can be used as the filling value of the outlier.

[0065] And / or, step S111d: normalizing or standardizing the vibration signal time series data.

[0066] And / or, step S111e: normalizing or standardizing the temperature time series data.

[0067] The implementation method of the above steps S111d to S111e is, for example, to use the formula Normalize the vibration signal time series data or temperature time series data; where X norm is the normalized time series data, X represents the original data before normalization, and X max and X minare the maximum and minimum values ​​in the time series data, respectively. Use the formula Standardize the vibration signal time series data or temperature time series data, where X ′ is the standardized time series data, X represents the original data before standardization, μ represents the mean of the time series data, and σ represents the standard deviation of the time series data.

[0068] As an optional implementation of the above-mentioned step S120, before using the temporal attention mechanism model to calculate the weights of multiple modal features, the temporal attention mechanism model can also be trained first. For example: first, the above-mentioned visible light image time series data, vibration signal time series data and / or temperature time series data are time-aligned to ensure that the time series data of each modality are at the same time step, and the modal time series data after time alignment is obtained, and the modal time series data after time alignment is converted into an input format suitable for the temporal attention mechanism model. Specifically, the sliding window technology can be used to divide the above time series data into multiple subsequences, each subsequence represents a data sample, and the size and step size of the sliding window can be set according to data characteristics, specific task characteristics and human experience. Specifically, the time series data in the sliding window of the previous step length is used as a training sample, and the time series data in the sliding window of the next step length is used as a training label, and the temporal attention mechanism model is trained to obtain a trained temporal attention mechanism model, so that the temporal attention mechanism model learns the ratio between the time series data of the previous step length and the time series data of the next step length as the weight of various modal features, so that the temporal attention mechanism model can multiply the time series data of the previous step length by the weight to obtain the time series data of the next step length. After obtaining the trained temporal attention mechanism model, the trained temporal attention mechanism model can be used to calculate the weights of multiple modal features to obtain the weights of multiple modal features.

[0069] As an optional implementation of the above-mentioned step S120, the above-mentioned temporal attention mechanism model can also select a multi-layer perceptron (Multi-Layer Perceptron, MLP). Specifically, after extracting the modal features from the multimodal time series data, in order to fuse the modal features from different sources, the above-mentioned trained temporal attention mechanism model can be used to adaptively assign weights to features of different time points and different modalities, thereby improving the accuracy and reliability of the prediction. The above-mentioned temporal attention mechanism is a variant of the attention mechanism, which is designed to process time series data. The temporal attention mechanism can dynamically assign weights according to the importance of the features at different time points, thereby capturing the key features of the key time points and improving the performance of the model. Use a multi-layer perceptron (MLP) to calculate the weights of multiple modal features to obtain the attention weights of multiple modal features. The attention weights here can be expressed using the formula: And the e in the formula t =v T tanh(W h h t +W s s); where a t represents the above attention weight, t represents the time point, e t represents the score at time point t, which is calculated by a multilayer perceptron. t represents the feature representation at time point t, s is the context vector, and W h and W s Represents different weight matrices, v T is the weight vector.

[0070] Optionally, in the process of training the above-mentioned temporal attention mechanism model, the mean square error loss function can be used to calculate the loss value, or the ReLU function can be introduced in the regularization term function as the activation function of the temporal attention mechanism model, so that the total loss function of the temporal attention mechanism model is obtained as follows: Among them, Loss represents the total loss value calculated by the total loss function, ReLU represents the ReLU function, is the predicted value of the i-th sample, y i is the true value of the i-th sample, λ 1 and λ 2 is the weight coefficient of the regularization term. In the implementation of the above solution, a ReLU-based regularization term is introduced into the loss function, so that the prediction result will neither exceed the maximum life value nor be negative. This design ensures the physical rationality of the prediction result. The improved total loss function increases the reasonable constraints on the prediction result while ensuring the prediction accuracy. This not only makes the prediction result more reliable, but also improves the stability of the model and the feasibility of practical application.

[0071] As an optional implementation of the above step S130, the implementation of fusing multiple modal features according to the weights of multiple modal features may include:

[0072] Step S131: performing weighted processing on the multiple modal features according to the weights of the multiple modal features to obtain multiple weighted feature representations.

[0073] The implementation method of the above step S131 is, for example, to use the formula The attention weights of multiple modal features and multiple modal features are weighted to obtain multiple weighted feature representations; where F att is a weighted feature representation, a t represents the attention weight of the t-th modality feature among the attention weights of multiple modality features, h tRepresents the tth modal feature among multiple modal features.

[0074] Step S132: Use the temporal attention mechanism model to perform feature fusion on multiple weighted feature representations to obtain fused features.

[0075] As an optional implementation of the above step S132, the above implementation of using the temporal attention mechanism model to perform feature fusion on multiple weighted feature representations may include: using the formula Concatenate([X 1 ,X 2 ,……,X n ].axis=k)=[X 1 |X 2 |……|X n ] Concatenate multiple weighted feature representations to obtain a concatenated feature representation where X i It represents the i-th weighted feature representation among multiple weighted feature representations, and its shape is (d 1 ,d 2 ,…,d k ), "axis" = k specifies splicing in the kth dimension. The function of the above-mentioned feature fusion layer is to horizontally splice the features of each modal data to form a fusion feature of a comprehensive feature vector. Use the convolutional neural network in the temporal attention mechanism model (not shown) to perform feature fusion on the spliced ​​feature representation to obtain fusion features. Specifically, the spliced ​​feature representation is input into the convolutional neural network of the temporal attention mechanism model. The convolutional neural network may include multiple convolutional layers. Multi-layer convolution operations of multiple convolutional layers are used to perform feature fusion on the spliced ​​feature representation to obtain fusion features. Then, spatial feature extraction is performed on the fusion features. Each convolutional layer includes a certain number of convolution kernels (filters), and each convolution kernel extracts different local features. ReLU is used as the activation function to introduce nonlinearity so that the model can learn more complex features.

[0076] As another optional implementation of the above step S132, in combination with the above implementation, after each layer of convolution operation, a pooling layer can be used to reduce the dimension of the feature while retaining important information. The pooling methods that can be used include maximum pooling and average pooling, which effectively reduce the computational complexity of the model while preventing overfitting. After the convolution layer and the pooling layer, the feature map is also flattened into a one-dimensional feature vector, and the one-dimensional feature vector here is used as the fusion feature of the final output. The fusion feature finally output here is a high-level feature vector obtained after processing by the convolution and pooling layers. The feature vector contains multimodal fusion features extracted from vibration signals, temperature data, and visible light image data, which can better represent the overall characteristics of the data. Effective use of multi-source data improves the comprehensiveness and accuracy of feature extraction and solves the problem of insufficient information from a single data source. Multimodal data fusion enhances the overall understanding of the health status of the equipment, thereby improving the reliability and accuracy of the remaining life RUL prediction of the equipment.

[0077] As an optional implementation of the above step S140, the above implementation of predicting the remaining life of the target device according to the fusion feature may include:

[0078] Step S141: Use the trained fully connected network model to predict the fusion features to obtain the remaining life of the target device.

[0079] After training the temporal attention mechanism model, the fully connected network model can also be trained. Before training the fully connected network model, it is necessary to prepare the training data in the training data set. The training data manufacturing process includes: time-aligning the above-mentioned visible light image time series data, vibration signal time series data and / or temperature time series data to ensure that the time series data of each modality are at the same time step, obtaining the time-aligned modal time series data, and then extracting multiple aligned modal features from the time-aligned modal time series data, using the trained temporal attention mechanism model to calculate the weights of the multiple aligned modal features, obtaining the weights of the multiple aligned modal features, fusing the multiple aligned modal features according to the weights of the multiple aligned modal features, obtaining the aligned fused features, and using the above-obtained aligned fused features as the training data for training the fully connected network model.

[0080] Before training the fully connected network model, it is necessary to prepare the training labels (i.e. RUL labels) in the training data set. The process of making RUL labels includes: determining the failure time point T of the device f , then, using the formula and According to the alignment fusion feature F (i:i+w-1)Calculate the RUL label; where t represents the current time step of the sliding window in the time dimension, w represents the size of the sliding window, i represents the current position of the sliding window in the aligned fusion feature, and F (i:i+w-1) Represents the features of the above aligned fusion features in a sliding window, RUL t represents the RUL label calculated at time step t, RUL i+w-1 represents the RUL label calculated at the i+w-1 time step, and the value range of both is 0 to 1. f Indicates the time point of device failure (which can be understood as the time when the previous device failed).

[0081] The implementation method of the above step S141 is, for example: after obtaining the above alignment fusion features and training labels (i.e., RUL labels), the fully connected neural network model can be trained using the alignment fusion features as training data and the RUL labels as training labels, thereby obtaining a trained fully connected neural network model. The trained fully connected network model is used to predict the fusion features to obtain the remaining life of the target device.

[0082] As an optional implementation of the above-mentioned method for predicting the remaining life of the equipment, after predicting the remaining life of the target equipment according to the fusion features, visualization can also be performed. This implementation may include:

[0083] Step S150: performing dimensionality reduction on multiple modal features to obtain modal features after dimensionality reduction.

[0084] The implementation method of the above step S150 is, for example: using T-SNE technology to reduce the dimension of multiple modal features to obtain the modal features after dimensionality reduction; wherein, T-SNE is a nonlinear dimensionality reduction technology, the core idea of ​​which is to map high-dimensional data to low-dimensional space while trying to maintain the similarity relationship between data points. It is very effective for visualizing the intrinsic structure of complex data and can help people better understand the data. The above similarity relationship can be that T-SNE uses a similarity measure between data points in high-dimensional data, usually a probability density function of a Gaussian distribution, and more similar data points will have higher probability values, while dissimilar data points will have lower probability values. The above mapping to low-dimensional space means that T-SNE creates a low-dimensional embedding for mapping high-dimensional data to low-dimensional space. The goal of the mapping is to minimize the difference between the similarity probability of data point pairs in high-dimensional space and the similarity probability of data point pairs in low-dimensional space. The similarity relationship can be T-SNE uses t-distribution to represent the similarity probability. The t-distribution has a fat-tailed property, which makes the distance between similar data points in the low-dimensional space larger, thus helping to maintain clustering.

[0085] Step S160: Perform visual analysis on the modal features, fusion features and remaining life of the target device after dimensionality reduction.

[0086] The implementation method of the above step S160 is, for example: reducing the multimodal features to 2D or 3D space through T-SNE, collecting the modal features, fusion features and remaining life of the target device after dimensionality reduction, and then using the T-SNE algorithm to project the visualization data from the 2D or 3D space to a low-dimensional space (e.g., two-dimensional space), and then selecting a 2D or 3D method to visualize the modal features, fusion features and remaining life of the target device after dimensionality reduction. Optionally, a scatter plot can also be drawn based on the modal data after dimensionality reduction, and the real RUL and predicted RUL are marked in the figure for easy comparison and analysis. Through the visualization results, the distribution of the real RUL and the predicted RUL is compared, the clustering and distribution of the data points are analyzed, and the areas with large model prediction errors are identified. According to the visualization results, samples and features with large model prediction errors are identified and analyzed, and the shortcomings of the model in certain features or time periods are found out by comparing the real values ​​and the predicted values.

[0087] See also Figure 5 The structure diagram of the device remaining life prediction device provided in the embodiment of the present application is shown; the embodiment of the present application provides a device remaining life prediction device 200, including:

[0088] The modal feature extraction module 210 is used to extract features from the multimodal data of the target device to obtain multiple modal features.

[0089] The feature weight calculation module 220 is used to use the temporal attention mechanism model to perform weight calculation on multiple modal features to obtain weights of multiple modal features.

[0090] The modal feature fusion module 230 is used to fuse multiple modal features according to their weights to obtain fused features.

[0091] The remaining life prediction module 240 is used to predict the remaining life of the target device according to the fusion features.

[0092] As an optional implementation of the above device, the multimodal data includes: visible light image time series data, vibration signal time series data and / or temperature time series data; the modal feature extraction module includes:

[0093] The image data feature extraction submodule is used to extract features from visible light image time series data using a convolutional neural network.

[0094] And / or, a vibration signal feature extraction submodule, used to extract features from vibration signal time series data using a time convolutional network.

[0095] And / or, a temperature data feature extraction submodule, for extracting features from temperature time series data using a long short-term memory network.

[0096] As an optional implementation of the above device, the modal feature extraction module further includes:

[0097] The image data preprocessing submodule is used to perform image denoising, image enhancement, image cropping and / or image scaling on visible light image time series data.

[0098] And / or, a vibration signal preprocessing submodule, used to remove trend items, perform noise reduction processing, extract time domain signals and / or extract frequency domain signals from vibration signal time series data.

[0099] And / or, a temperature data preprocessing submodule, used to perform denoising, missing data processing and / or outlier processing on the temperature time series data.

[0100] And / or, a normalization and standardization processing submodule is used to perform normalization or standardization processing on the vibration signal time series data and / or the temperature time series data.

[0101] As an optional implementation of the above device, the modal feature fusion module includes:

[0102] The modal feature weighting processing submodule is used to perform weighted processing on multiple modal features according to the weights of the multiple modal features to obtain multiple weighted feature representations.

[0103] The weighted feature representation fusion submodule is used to use the temporal attention mechanism model to fuse multiple weighted feature representations to obtain fused features.

[0104] As an optional implementation of the above device, the weighted feature representation fusion submodule includes:

[0105] The feature representation concatenation unit is used to concatenate multiple weighted feature representations to obtain a concatenated feature representation.

[0106] The splicing feature fusion unit is used to use the temporal attention mechanism model to perform feature fusion on the spliced ​​feature representations to obtain fused features.

[0107] As an optional implementation of the above device, the remaining life prediction module includes:

[0108] The fusion feature prediction submodule is used to predict the fusion features using a fully connected network model to obtain the remaining life of the target device.

[0109] As an optional implementation of the above device, the device for predicting the remaining life of equipment further includes:

[0110] The model feature dimensionality reduction module is used to reduce the dimensions of multiple modal features to obtain the modal features after dimensionality reduction.

[0111] The feature visualization analysis module is used to visualize the modal features, fusion features and remaining life of the target equipment after dimensionality reduction.

[0112] It should be understood that the device corresponds to the above-mentioned equipment remaining life prediction method embodiment, and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0113] See also Figure 6 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0114] The embodiment of the present application also provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program is executed by the processor 310 to execute the above method. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0115] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0116] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0117] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0118] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0119] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A method for predicting the remaining life of equipment, characterized in that: include: Perform feature extraction on the multimodal data of the target device to obtain multiple modal features; Using a temporal attention mechanism model to perform weight calculation on the multiple modal features to obtain weights of the multiple modal features; Performing feature fusion on the multiple modal features according to the weights of the multiple modal features to obtain a fused feature; The remaining life of the target device is predicted according to the fused features.

2. The method according to claim 1, characterized in that The multimodal data includes: visible light image time series data, vibration signal time series data and / or temperature time series data; the feature extraction of the multimodal data of the target device includes: Using a convolutional neural network to extract features from the visible light image time series data; And / or, using a time convolutional network to perform feature extraction on the vibration signal time series data; And / or, use a long short-term memory network to extract features from the temperature time series data.

3. The method according to claim 2, characterized in that Before extracting features from the multimodal data of the target device, the method further includes: Performing image denoising, image enhancement, image cropping and / or image scaling on the visible light image time series data; and / or, removing trend items, performing noise reduction processing, time domain signal extraction and / or frequency domain signal extraction on the vibration signal time series data; and / or, performing denoising, missing data processing and / or outlier processing on the temperature time series data; And / or, normalizing or standardizing the vibration signal time series data and / or the temperature time series data.

4. The method according to claim 1, characterized in that: The performing feature fusion on the multiple modal features according to the weights of the multiple modal features includes: Performing weighted processing on the multiple modal features according to the weights of the multiple modal features to obtain multiple weighted feature representations; The temporal attention mechanism model is used to perform feature fusion on the multiple weighted feature representations to obtain the fused features.

5. The method according to claim 4, characterized in that The using the temporal attention mechanism model to perform feature fusion on the multiple weighted feature representations includes: splicing the multiple weighted feature representations to obtain a spliced ​​feature representation; The temporal attention mechanism model is used to perform feature fusion on the concatenated feature representation to obtain the fused feature.

6. The method according to claim 1, characterized in that The predicting the remaining life of the target device according to the fusion feature includes: The fused features are predicted using a fully connected network model to obtain the remaining life of the target device.

7. The method according to any one of claims 1 to 6, characterized in that: After predicting the remaining life of the target device according to the fusion feature, the method further includes: Performing dimensionality reduction on the multiple modal features to obtain modal features after dimensionality reduction; A visual analysis is performed on the modal features after dimensionality reduction, the fusion features, and the remaining life of the target device.

8. A device for predicting the remaining life of equipment, characterized in that: include: The modal feature extraction module is used to extract features from the multimodal data of the target device to obtain multiple modal features; A feature weight calculation module, used to use a temporal attention mechanism model to perform weight calculation on the multiple modal features to obtain weights of the multiple modal features; A modal feature fusion module, used for fusing the multiple modal features according to their weights to obtain a fused feature; The remaining life prediction module is used to predict the remaining life of the target device according to the fusion feature.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

11. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images