Intelligent disk monitoring system based on multi-modal data fusion and fault early warning method

Abstract

The invention discloses an intelligent disk monitoring system based on multi-modal data fusion and a fault early warning method, and relates to the technical field of power plant management, and the method comprises the steps: collecting the real-time operation data of equipment; performing dimension reduction processing on the operation data by using a principal component analysis algorithm; inputting the data subjected to dimension reduction into a long short-term memory network model, modeling the time sequence dependence of the data by using the long short-term memory network model, and predicting the future operation state of the equipment; based on the prediction result of the operation state, performing feature extraction on the data through a convolutional neural network; according to an output result of the convolutional neural network, alarm signals of different levels are generated according to set threshold levels; and according to the grade of the alarm signal, adjusting operation parameters of the equipment through a feedback control mechanism. According to the intelligent disk monitoring system based on multi-modal data fusion, the accuracy, the real-time performance and the intelligent level of equipment monitoring are improved, and the fault early warning precision and the system response speed are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power plant management, and particularly to an intelligent monitoring system based on multi-modal data fusion and a fault warning method. Background Art

[0002] With the progress of industrialization and intelligence, the fault monitoring and warning system of equipment occupies an increasingly important position in various industrial productions and equipment management. Traditional equipment monitoring methods often rely on single monitoring parameters or empirical data, and these methods have problems such as limited monitoring scope, incomplete data processing, and slow response speed. In some complex industrial environments, the operating state of equipment is affected by various factors, including mechanical wear of equipment, environmental changes, load fluctuations, etc., which makes it more difficult to give early warnings of faults.

[0003] Currently, intelligent fault warning systems increasingly adopt data-driven methods, such as real-time data collection based on sensors, machine learning algorithms, artificial intelligence technologies, etc. These methods have made significant progress in fault prediction, but still face some challenges. First, a single data source often fails to comprehensively reflect the operating state of equipment. Especially in the application of multi-modal data fusion, how to effectively process data from different sensors and extract discriminative features is one of the current research difficulties. Second, existing time series data processing models, such as long short-term memory networks (LSTM), although able to handle the dependence of time series data, the prediction of equipment operating state often relies on a single model output, ignoring various potential non-linear relationships and local features in the equipment state, which limits the accuracy and robustness of the prediction results. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing equipment fault warning methods have the problem of information redundancy when processing high-dimensional data, and the traditional methods cannot effectively model the dependence of equipment time series data, resulting in insufficient prediction accuracy.

[0006] To solve the above technical problems, the present invention provides the following technical solution: A fault warning method for an intelligent monitoring system based on multi-modal data fusion, including:

[0007] Collect the real-time operation data of the equipment;

[0008] Use the principal component analysis algorithm to perform dimensionality reduction processing on the operation data;

[0009] Input the data after dimensionality reduction into the long short-term memory network model, and use the long short-term memory network model to model the temporal dependence of the data to predict the future operating state of the device;

[0010] Based on the prediction result of the operating state, extract features from the data through a convolutional neural network, mine the local non-linear relationships and potential patterns in the data, and judge the device state;

[0011] According to the output result of the convolutional neural network, generate alarm signals of different levels according to the set threshold levels;

[0012] According to the level of the alarm signal, adjust the operating parameters of the device through a feedback control mechanism.

[0013] As a preferred solution of the fault warning method of the intelligent monitoring system based on multi-modal data fusion according to the present invention, wherein: the real-time operating data includes device operating data, environmental data and historical fault data.

[0014] As a preferred solution of the fault warning method of the intelligent monitoring system based on multi-modal data fusion according to the present invention, wherein: the dimensionality reduction process includes collecting the real-time operating data of the device to form a data matrix;

[0015] Perform standardization processing on the data;

[0016] Perform local dimensionality reduction on the data in combination with a time window, and calculate the covariance matrix of each time window;

[0017] Perform eigenvalue decomposition on the covariance matrix to obtain eigenvectors;

[0018] Select the first k principal components and project them onto the principal components.

[0019] As a preferred solution of the fault warning method of the intelligent monitoring system based on multi-modal data fusion according to the present invention, wherein: the data after dimensionality reduction is expressed as,

[0020] D pca = D·V k

[0021] where D is the original device operating data matrix, with dimensions m×n, m being the number of samples and n being the number of features, and V k is the eigenvector matrix containing the first k principal components, with dimensions n×k, and D pca is the data matrix after dimensionality reduction, with dimensions m×k.

[0022] As a preferred solution of the fault warning method of the intelligent monitoring system based on multi-modal data fusion according to the present invention, wherein: predicting the future operating state of the device includes dynamically adjusting the time window length T according to the device operating state change rate γ w , expressed as,

[0023]

[0024] wherein, Δx represents the device state change amount, Δt represents the time step, and T short and T long are the short-term and long-term time window lengths respectively;

[0025] The dimension-reduced data D pca is segmented into multiple input sequences according to the adaptive time window T w Expressed as, Expressed as,

[0026]

[0027] wherein, i is the time step index, and x t is the dimension-reduced feature vector at time step t, coming from D pca ;

[0028] Output multiple time window sequences The length of each sequence is T w , and input it into the long short-term memory network model.

[0029] As a preferred solution of the fault warning method of the intelligent monitoring system based on multi-modal data fusion according to the present invention, wherein: constructing multiple LSTM sub-networks to capture the multi-level time series dependence relationship of the device operation data, and each sub-network corresponds to a different time scale T w , expressed as,

[0030]

[0031] wherein, j represents different time scale sub-networks, is the hidden state of the j-th sub-network at time t, is the input sequence of the j-th time scale;

[0032] The hidden states of each time scale sub-network are weighted and fused to obtain the comprehensive time series feature h t , expressed as,

[0033]

[0034] wherein, α j is the weight coefficient of the j-th sub-network, satisfying N is the number of time scale sub-networks;

[0035] Based on the comprehensive time series feature h t , calculate the attention weight β at each time step through the attention mechanism t , expressed as,

[0036]

[0037] where, W a and b a are learnable parameters, and T is the number of time steps;

[0038] Apply the attention weight to the comprehensive time series feature to obtain the weighted feature expressed as,

[0039]

[0040] According to the model prediction value and the true value y t calculate the prediction error e t , expressed as,

[0041]

[0042] Introduce a time series error feedback mechanism to assign a higher weight ω to the time steps with errors greater than the preset threshold t , define the weighted loss function, expressed as,

[0043]

[0044] ω t = exp(-|e t |)

[0045] Through the backpropagation algorithm, update the model parameter θ according to the weighted loss function , expressed as,

[0046]

[0047] where, η is the learning rate;

[0048] Use the optimized LSTM model to predict the future operating state of the device based on the weighted time series feature expressed as,

[0049]

[0050] where, Dense represents the fully connected layer, which is used to output the final prediction value.

[0051] As a preferred solution of the fault warning method of the intelligent monitoring system based on multi-modal data fusion according to the present invention, wherein: the determining the device state includes receiving the predicted result of the future operating state of the device, formatting the predicted result, and constructing the predicted result into a multi-dimensional time series feature matrix. When the predicted result is a continuous value, feature expansion is performed on the continuous value to form a multi-dimensional data matrix including multiple relevant continuous features; when the predicted result is a probability distribution, the probability values are pooled to form a multi-dimensional data matrix including probabilities of various categories.

[0052] Perform normalization processing on the multi-dimensional time series feature matrix to obtain the standardized feature data, and input the standardized feature data in a two-dimensional form with the dimension of time steps and feature dimensions into a convolutional neural network, where the feature dimension includes a continuous feature dimension or a category feature dimension.

[0053] Input the standardized feature data into a multi-channel convolutional neural network. Each channel corresponds to a convolutional extraction path for different types of features. Perform convolutional operations on the feature data through convolutional layers with adjustable filter numbers and convolutional kernel sizes in each channel, and use deformable convolutional layers to adapt to irregular patterns and local deformation features.

[0054] Perform pooling operations on the convolutional outputs of each channel to reduce the dimension and extract the main features, and fuse the pooling outputs of all channels to obtain a comprehensive feature representation.

[0055] Introduce an adaptive feature selection mechanism based on the comprehensive feature representation, adjust the weights of each feature through a gating signal, highlight the key features dynamically and suppress the irrelevant features, and obtain the feature representation after feature selection.

[0056] Use a multi-layer fully connected network and a non-linear activation function to perform pattern recognition processing on the feature representation after feature selection, mine complex patterns in the data, and output the final feature representation.

[0057] Based on the final feature representation, perform device state classification judgment through a fully connected layer, use the Softmax activation function to perform probability distribution processing on the classification result, and determine the device operating state category according to the maximum probability principle, and output the device state judgment result.

[0058] Another object of the present invention is to provide a fault warning method for an intelligent monitoring system based on multi-modal data fusion, which solves the problems of insufficient processing of high-dimensional data and inaccurate modeling of time series data in the existing fault warning methods.

[0059] To solve the above technical problems, the present invention provides the following technical solutions: A fault warning method for a smart monitoring system based on multi-modal data fusion, including: a data acquisition module, configured to collect real-time operation data of a device and preprocess the data; a data dimensionality reduction module, configured to perform dimensionality reduction processing on the preprocessed data using a principal component analysis algorithm; a time series prediction module, configured to input the dimensionality-reduced data into a long short-term memory network model, and use the long short-term memory network model to model the time series dependence of the data to predict the future operation state of the device; an anomaly detection module, configured to, based on the prediction result of the operation state, extract features from the data through a convolutional neural network, mine local non-linear relationships and potential patterns in the data, and judge the device state;

[0060] a fault warning module, configured to generate a warning signal for the operation state of the device according to the output result of the convolutional neural network, and generate alarm signals of different levels according to the set threshold levels; a feedback adjustment module, configured to adjust the operation parameters of the device through a feedback control mechanism according to the level of the alarm signal.

[0061] A computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the fault warning method for the smart monitoring system based on multi-modal data fusion as described above are implemented.

[0062] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the fault warning method for the smart monitoring system based on multi-modal data fusion as described above are implemented.

[0063] The beneficial effects of the present invention: The fault warning method for the smart monitoring system based on multi-modal data fusion provided by the present invention uses principal component analysis to perform dimensionality reduction processing on multi-source data, removes redundant information, and improves data processing efficiency. By modeling the dependence of device time series data through a long short-term memory network, it effectively predicts the future operation state of the device and solves the deficiencies of traditional methods in time series data processing. At the same time, by extracting local non-linear features from the data through a convolutional neural network, the discrimination ability of the device state is further enhanced. Based on the output prediction result, the present invention designs a multi-level alarm mechanism, generates corresponding alarm signals according to different alarm levels, and automatically adjusts the operation parameters of the device through a feedback control mechanism to ensure that the device is adjusted in time before a fault occurs, thereby avoiding damage or production interruption. The technical solution of the present invention combines dimensionality reduction, time series modeling, and deep learning technologies, improves the accuracy, real-time performance, and intelligence level of device monitoring, significantly improves the accuracy of fault warning and the system response speed, and is applicable to device health management in complex industrial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0065] Figure 1 It is the overall flowchart of a fault warning method for a smart monitoring system based on multi-modal data fusion provided by an embodiment of the present invention. Detailed implementation manners

[0066] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the detailed implementation manners of the present invention with reference to the accompanying drawings of the specification. Obviously, the described embodiments are some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0067] Many specific details are set forth in the following description to facilitate a thorough understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0068] Embodiment 1

[0069] Refer to Figure 1 , which is an embodiment of the present invention, and provides a fault warning method for a smart monitoring system based on multi-modal data fusion, including:

[0070] Step S1: Collect the real-time operation data of the device;

[0071] Step S2: Use the principal component analysis algorithm to perform dimensionality reduction processing on the operation data;

[0072] Step S3: Input the data after dimensionality reduction into the long short-term memory network model, and use the long short-term memory network model to model the time series dependence of the data to predict the future operation state of the device;

[0073] Step S4: Based on the prediction result of the operation state, extract features from the data through a convolutional neural network, mine the local non-linear relationships and potential patterns in the data, and judge the device state;

[0074] Step S5: According to the output result of the convolutional neural network, generate alarm signals of different levels according to the set threshold levels;

[0075] Step S6: Adjust the operating parameters of the device through a feedback control mechanism according to the level of the alarm signal.

[0076] The real-time operating data includes device operating data, environmental data, and historical fault data.

[0077] Device operating data includes, but is not limited to, current, voltage, power, temperature, vibration, pressure, etc.

[0078] Environmental data such as external temperature, humidity, wind speed, etc. are environmental factors affecting device operation.

[0079] Historical fault data includes records of past faults, alarm times, and their solutions, etc.

[0080] Device health status data such as device loss rate, maintenance records, life prediction, etc.

[0081] The dimensionality reduction process includes collecting the real-time operating data of the device to form a data matrix;

[0082] Normalize the data;

[0083] Perform local dimensionality reduction on the data in combination with a time series window, and calculate the covariance matrix of each time window;

[0084] Perform eigenvalue decomposition on the covariance matrix to obtain eigenvectors;

[0085] Select the first k principal components and project them onto the principal components.

[0086] The data after dimensionality reduction is represented as

[0087] D pca = D·V k

[0088] where D is the original device operating data matrix, with dimensions m×n, m being the number of samples and n being the number of features, and V k is the eigenvector matrix containing the first k principal components, with dimensions n×k, and D pca is the data matrix after dimensionality reduction, with dimensions m×k.

[0089] Specifically, step 1: Collect the real-time operating data of the device

[0090] Collect the real-time operating data of the device D = {x 1 , x 2 , …, x n}, and the data set contains the real-time outputs of multiple device sensors, such as temperature, pressure, current, voltage, etc. Assume that each sampling point has n features, and a total of m data points are collected. The data is represented as a matrix of m×n:

[0091]

[0092] where x i,j represents the j-th eigenvalue of the i-th sampling point.

[0093] Step 2: Data preprocessing

[0094] To ensure equal weights for each feature, first perform standardization on the data:

[0095]

[0096] where μ j is the mean of the j-th feature, and σ j is the standard deviation of the j-th feature. The standardized data is expressed as:

[0097]

[0098] where is the value of the standardized data.

[0099] Step 3: Optimize PCA dimensionality reduction

[0100] Introduce a time window and perform independent PCA dimensionality reduction on the data for each time window. Within each time window, calculate the covariance matrix C window: :

[0101]

[0102] where D window is the standardized data within window m window , and m window is the window size.

[0103] Step 4: Eigenvalue decomposition of the covariance matrix

[0104] Perform eigenvalue decomposition on the covariance matrix C window to obtain eigenvalues and eigenvectors:

[0105] C window v i = λ i v i

[0106] where λ i is the i-th eigenvalue, and v i is the corresponding eigenvector.

[0107] Step 5: Select principal components

[0108] According to the magnitudes of the eigenvalues, select the eigenvectors v 1 , v 2,…,v k , these eigenvectors form the principal component matrix V after dimensionality reduction k :

[0109] V k = [v 1 v 2 … v k

[0110] Step 6: Dimensionality reduction

[0111] By projecting the standardized data D std onto the principal component matrix V k , the dimensionality-reduced data D PCA is obtained:

[0112] D PCA = D std V k

[0113] where D PCA is the dimensionality-reduced data with k principal components.

[0114] The predicted future operating state of the device includes dynamically adjusting the time window length T according to the device operating state change rate γ w , expressed as

[0115]

[0116] where Δx represents the device state change amount, Δt represents the time step, and T short and T long are the short-term and long-term time window lengths respectively;

[0117] The dimensionality-reduced data D pca is segmented into multiple input sequences according to the adaptive time window T w expressed as

[0118]

[0119] where i is the time step index, and x t is the dimensionality-reduced eigenvector at time step t, coming from D pca ;

[0120] Output multiple time window sequences Each sequence has a length of T w , and is input into the long short-term memory network model.

[0121] Construct multiple LSTM sub-networks to capture the multi-level temporal dependencies of the device operation data, and each sub-network corresponds to a different time scale T w ​​, denoted as,

[0122]

[0123] where \(j\) represents different time-scale sub-networks (such as short-term, medium-term, long-term), is the hidden state of the \(j\)-th sub-network at time \(t\), is the input sequence of the \(j\)-th time scale;

[0124] The hidden states of each time-scale sub-network are weighted and fused to obtain the comprehensive time-series feature \(h\) t , denoted as,

[0125]

[0126] where \(\alpha\) j is the weight coefficient of the \(j\)-th sub-network, satisfying and \(N\) is the number of time-scale sub-networks;

[0127] Based on the comprehensive time-series feature \(h\) t , the attention weight \(\beta\) at each time step is calculated through the attention mechanism t , denoted as,

[0128]

[0129] where \(W\) a and \(b\) a are learnable parameters, and \(T\) is the number of time steps;

[0130] The attention weight is applied to the comprehensive time-series feature to obtain the weighted feature denoted as,

[0131]

[0132] According to the model prediction value and the true value \(y\) t the prediction error \(e\) is calculated t , denoted as,

[0133]

[0134] Introduce a time-series error feedback mechanism to assign a higher weight \(\omega\) to the time steps with errors greater than the preset threshold t , define the weighted loss function, denoted as,

[0135]

[0136] \(\omega\) t = exp(-|e t |)

[0137] Using the backpropagation algorithm, based on the weighted loss function Update the model parameters θ, expressed as

[0138]

[0139] where η is the learning rate;

[0140] Using the optimized LSTM model, based on the weighted time series features Predict the future operating state of the device Expressed as

[0141]

[0142] where Dense represents the fully connected layer, used to output the final predicted value.

[0143] In this embodiment, by introducing an adaptive time window mechanism, the time window length of the input data is dynamically adjusted to adapt to the change rate of the device operating state. The traditional long short-term memory (LSTM) model usually uses a fixed-size time window, which cannot effectively capture the key features at different time scales when the device operating state changes rapidly or tends to be stable. The adaptive time window mechanism automatically selects an appropriate time window length according to the current change rate of the device operating state, so as to ensure that the model can efficiently extract time series features in different operating environments. This mechanism not only improves the sensitivity of the model to sudden faults, but also enhances the ability to capture long-term trends, significantly improving the accuracy and robustness of the prediction.

[0144] In order to comprehensively capture the multi-level time series dependence relationships in the device operation data, this embodiment designs a multi-scale LSTM network structure. This structure constructs multiple parallel LSTM sub-networks, and each sub-network is responsible for data modeling at different time scales, such as short-term (minute-level) and long-term (hour-level) trend analysis. The outputs of each sub-network are weighted and fused to form comprehensive time series features and input them into the subsequent model. This multi-scale network structure can simultaneously process the short-term fluctuations and long-term changes of the device operation, avoiding the problem of information loss that may occur in a single time scale model when dealing with complex time series data. Through the multi-scale LSTM network, the model has stronger expressive ability in the extraction and modeling of multi-level time series features, significantly improving the accuracy of device state prediction.

[0145] To further enhance the model's attention to key temporal features, this embodiment introduces an attention mechanism into the LSTM network. This mechanism dynamically calculates the attention weights for each time step and weights the comprehensive temporal features, enabling the model to automatically identify and focus on the features that are most important for fault prediction. The introduction of the attention mechanism not only improves the model's ability to identify key features, reduces the interference of unimportant data on the prediction results, but also significantly enhances the accuracy and reliability of fault prediction. This technical feature enables the model to more efficiently extract and utilize key information in complex device operating environments, optimizing the prediction performance.

[0146] To further optimize the prediction accuracy of the model, this embodiment designs a temporal error feedback mechanism. This mechanism monitors the prediction error of the model in real time and dynamically adjusts the training weights of the model according to the magnitude of the error, enabling the model to obtain more learning resources at time steps with larger errors and promptly correct the prediction deviation. This mechanism effectively avoids the problem of prediction error accumulation that may occur in traditional LSTM models when facing complex temporal data, significantly enhancing the model's fitting ability and prediction accuracy. The introduction of the temporal error feedback mechanism enables the model to maintain stable performance in long-term sequence prediction, enhancing the overall reliability of the system.

[0147] Based on the prediction results of the optimized LSTM model, this embodiment designs a multi-level warning signal generation and feedback control mechanism. According to the comparison between the predicted device operating state and the preset threshold, the system automatically generates warning signals at different levels, corresponding to different severity levels of fault risks. The multi-level warning signals can not only provide more detailed fault risk information but also trigger corresponding feedback control measures to automatically adjust the device operating parameters and prevent the further deterioration of faults. This mechanism realizes the intelligent monitoring and automatic adjustment of device operation, improves the system's response speed and processing efficiency, and ensures the stable operation and safety of the device.

[0148] The determination of the device state includes receiving the prediction results of the future operating state of the device, formatting the prediction results, and constructing the prediction results into a multi-dimensional time series feature matrix. Among them, when the prediction result is a continuous value, the continuous value is subjected to feature expansion to form a multi-dimensional data matrix containing multiple relevant continuous features; when the prediction result is a probability distribution, the probability values are aggregated to form a multi-dimensional data matrix containing probabilities of various categories.

[0149] The multi-dimensional time series feature matrix is normalized to obtain the standardized feature data, and the standardized feature data is input into a convolutional neural network in a two-dimensional form with dimensions of time steps and feature dimensions, where the feature dimension includes a continuous feature dimension or a category feature dimension.

[0150] Input the standardized feature data into a multi-channel convolutional neural network. Each channel corresponds to a convolutional extraction path for different types of features. Perform convolutional operations on the feature data through convolutional layers with adjustable filter numbers and convolutional kernel sizes in each channel, and use deformable convolutional layers to adapt to irregular patterns and local deformation features;

[0151] Perform pooling operations on the convolutional outputs of each channel to reduce the dimension and extract the main features. Fuse the pooling outputs of all channels to obtain a comprehensive feature representation;

[0152] Introduce an adaptive feature selection mechanism based on the comprehensive feature representation. Adjust the weights of each feature through gating signals to dynamically highlight key features and suppress irrelevant features, and obtain the feature representation after feature selection;

[0153] Use a multi-layer fully connected network and a non-linear activation function to perform pattern recognition processing on the feature representation after feature selection, mine complex patterns in the data, and output the final feature representation;

[0154] Based on the final feature representation, perform device status classification and judgment through a fully connected layer. Use the Softmax activation function to perform probability distribution processing on the classification results, and determine the device operating status category through the maximum probability principle, and output the device status judgment result.

[0155] In step S4, input the predicted results of the future operating status of the device predicted by the long short-term memory (LSTM) model in step S3 Among them, represents the set of prediction results, represents the prediction result at time step t, and T represents the total number of time steps.

[0156] If is a continuous value (regression task), construct a multi-dimensional time series data matrix containing multiple relevant continuous features:

[0157]

[0158] Among them, Y reg is the multi-dimensional time series data matrix of the continuous value prediction results, k is the number of continuous features, is the predicted value of the kth continuous feature at time step t.

[0159] If is a probability value (multi-classification task), construct a multi-dimensional time series data matrix containing the probabilities of each category:

[0160]

[0161] Among them, Y probis a multi-dimensional time series data matrix of probability value prediction results, c is the number of classification categories, is the predicted probability of the category c corresponding to the time step t.

[0162] Perform normalization processing on the constructed multi-dimensional time series data matrix Y to obtain the normalized data matrix Y':

[0163]

[0164] Among them, Y' is the normalized data matrix, μ is the mean of the data, and σ is the standard deviation of the data.

[0165] Design a multi-channel convolutional neural network architecture to extract features

[0166] The input layer accepts the normalized multi-dimensional time series data matrix Y', whose shape is [T, k, 1] or [T, c, 1], where T is the time step length, k or c is the feature dimension, and 1 represents a single channel.

[0167] For different types of features, design multiple independent convolutional channels, and each channel focuses on extracting the local non-linear relationship and potential pattern of specific features.

[0168] For example,

[0169] Conv i = Conv1D(filters = f i , kernel_size = k i , activation = ReLU)(Y' i )

[0170] Among them, Conv i is the i-th convolutional channel, Conv1D is a one-dimensional convolutional operation, f i is the number of filters of the i-th convolutional channel, k i is the convolutional kernel size of the i-th convolutional channel, ReLU is a rectified linear activation function, and Y' i is the input data of the i-th feature dimension.

[0171] Perform pooling operation on the output Conv i of each convolutional channel to reduce the feature dimension and extract the main features:

[0172] Pool i = MaxPooling1D(pool_size = p i )(Conv i )

[0173] Among them, Pool i is the output of the i-th pooling layer, pi is the pooling window size of the i-th pooling layer.

[0174] Fuse all the pooled features to form a comprehensive feature representation F combined :

[0175] F combined = Concatenate(Pool 1 , Pool 2 , …, Pool n )

[0176] where F combined is the high-level feature representation after fusion, and n is the number of convolutional channels.

[0177] Dynamically adjust the size and number of convolutional kernels according to different stages of the device operation state to meet the requirements of different time scales and feature complexities.

[0178] For example, when the device state changes drastically, increase the number of convolutional kernels to capture more features; when the change is slow, reduce the number of convolutional kernels to avoid overfitting.

[0179] Introduce a deformable convolutional layer in the convolutional layer to enhance the model's adaptability to irregular patterns and local deformations in the device operation data, and improve the flexibility and accuracy of feature extraction.

[0180] Deformable convolution makes the convolutional kernel adapt to features of different shapes and sizes by learning the offsets of spatial sampling positions, improving the accuracy of feature extraction.

[0181] Based on the fused features, introduce a learnable feature selection mechanism to dynamically adjust the importance weights of each feature through a gating mechanism:

[0182]

[0183] where G i is the gating signal of the i-th feature, σ is the Sigmoid activation function used to limit the gating signal between 0 and 1, W g is the weight matrix of the gating layer, b g is the bias vector of the gating layer, and F ( c i o ) mbined is the i-th element of the comprehensive feature representation F combined of.

[0184] Apply the gating signal to adjust the feature weights:

[0185] F selected= G⊙F combined

[0186] where F selected is the feature representation after gated adjustment, G is the set of all gating signals, and ⊙ represents the element-wise multiplication operation.

[0187] By introducing non-linear activation functions and multi-layer fully connected networks, complex patterns in the data are further explored:

[0188] F nonlinear = ReLU(Dense(F selected , units = u 1 , activation = ReLU))

[0189] where F nonlinear is the feature representation after being processed by the first fully connected layer and the ReLU activation function, Dense is the fully connected layer for linearly combining features, and u 1 is the number of neurons in the first fully connected network;

[0190] ReLU is the rectified linear activation function.

[0191] F final = Dense(F nonlinear , units = u 2 , activation = ReLU)

[0192] where F final is the final feature representation after being processed by the second fully connected layer and the ReLU activation function, u 2 is the number of neurons in the second fully connected network, Dense is the fully connected layer, and ReLU is the rectified linear activation function.

[0193] Based on the non-linear feature representation F final , multi-class classification of the device state is performed through the fully connected layer:

[0194]

[0195] where is the state probability distribution vector of the device at time step t, c is the number of classification categories of the device state, and the Softmax function is used to convert the output into a probability distribution to ensure that the sum of probabilities of all categories is 1.

[0196] By the maximum probability principle, the final state category of the device at time step t is determined:

[0197]

[0198] where Statet is the predicted state category of the device at time step t.

[0199] It should be noted that when traditional Convolutional Neural Networks (CNNs) process multi-modal time series data, it is difficult to simultaneously capture the local non-linear relationships and potential patterns of different types of features, resulting in insufficient accuracy in device state judgment. The method of this embodiment uses a multi-channel CNN structure, where each channel independently extracts different types of features, and introduces a deformable convolutional layer to enhance the adaptability to irregular patterns. At the same time, combined with an adaptive feature selection mechanism, the feature weights are dynamically adjusted through gating signals to ensure the highlighting of key features. This significantly improves the model's feature extraction ability for multi-modal data, comprehensively captures the complex patterns and non-linear relationships in the device operation data, and enhances the accuracy and robustness of device state judgment.

[0200] Traditional methods have insufficient adaptability of fixed convolutional kernel parameters under different device operating states, resulting in low feature extraction efficiency and inability to effectively cope with the dynamic changes of device operating states.

[0201] The method of this embodiment uses a dynamic convolutional kernel adjustment and adaptive feature selection mechanism, and dynamically adjusts the feature weights through gating signals to achieve adaptation to different time scales and feature complexities. This improves the model's adaptability to different device operating states, ensures that key features can be effectively extracted under various dynamic changes, and improves the accuracy of fault judgment and the overall model performance.

[0202] The ability to identify complex fault patterns is limited, and traditional models are difficult to accurately judge the device state, resulting in untimely or false fault warnings.

[0203] The method of this embodiment further explores the complex patterns in the data by introducing a multi-layer fully connected network and non-linear activation functions, improving the model's expressive ability and depth. This enhances the model's ability to identify complex fault patterns, ensures the depth and accuracy of device state judgment, and reduces false alarms and missed alarms.

[0204] Traditional method models have insufficient robustness and generalization ability when facing diverse and dynamically changing data, affecting the stability of the fault warning system.

[0205] The method of this embodiment introduces data augmentation techniques such as time warping and noise injection in the data preprocessing stage, combined with multi-channel feature extraction and dynamic convolutional kernel adjustment, to improve the model's adaptability to diverse inputs. This significantly improves the model's robustness and generalization ability, ensures high-efficiency and stable performance under different device operating environments and states, and enhances the overall reliability of the system.

[0206] In traditional methods, the modeling of temporal dependence and the feature extraction process are independent of each other, making it difficult to achieve collaborative optimization and affecting the overall model performance.

[0207] The method of this embodiment uses end-to-end joint training of LSTM and CNN to coordinate optimization of the temporal dependency modeling and feature extraction process, thereby improving the performance and prediction effect of the overall model. This improves the performance and prediction effect of the overall model, ensures the efficiency and real-time performance of the fault warning system, adapts to the dynamic changes in the operating status of the equipment, and meets the needs of real-time monitoring and warning.

[0208] The existing system lacks intelligent monitoring and early warning mechanisms and is unable to respond to changes in equipment operating status in real time, resulting in untimely fault warnings.

[0209] The method of this embodiment realizes intelligent monitoring and real-time warning of equipment operation status through multi-channel CNN and adaptive feature selection mechanism, combined with nonlinear pattern recognition. It reduces human intervention and maintenance costs, improves the timeliness and accuracy of fault warning, ensures stable operation of equipment, and improves the automation and intelligence level of equipment management.

[0210] Receive the output result from the convolutional neural network;

[0211] Classifying the output results into level one alarm, level two alarm and level three alarm according to a preset threshold;

[0212] Generate corresponding first-level alarm signal, second-level alarm signal and third-level alarm signal.

[0213] When a level 1 alarm signal is detected, the feedback control mechanism first slightly reduces the operating load of the equipment, such as by fine-tuning the motor speed or slightly reducing the processing intensity, to relieve pressure and maintain the equipment within a safe operating range. At the same time, the temperature set point will be optimized to ensure that the equipment temperature is stable, and the operating speed will be adjusted appropriately to reduce mechanical vibration and wear. In addition, the auxiliary cooling system will be activated or its operation will be enhanced to assist the main cooling system in maintaining the equipment temperature. When a level 2 alarm signal is detected, the feedback control mechanism will make moderate adjustments to the equipment, including significantly reducing the equipment load, starting the backup power supply or backup equipment to ensure continuous operation, adjusting the operating mode to energy-saving or low-power consumption, and enhancing the efficiency of the cooling system to quickly restore the equipment to a safe state. If a level 3 alarm signal is detected, indicating a serious fault, the feedback control mechanism will immediately execute the emergency shutdown procedure, comprehensively reduce the equipment operating rate, and activate multiple safety protection measures such as automatic power off, explosion-proof devices or pressure relief systems to ensure the safety of equipment and operators. At the same time, a detailed fault report will be automatically generated and notified to maintenance personnel to ensure timely response and handling of fault problems. This series of adjustment steps ensures that the equipment can take corresponding control measures under different fault levels to prevent the fault from further deteriorating, thereby ensuring the stable operation of the equipment and the safety of the operators.

[0214] Example 2

[0215] For an embodiment of the present invention, there is provided an intelligent monitoring system based on multimodal data fusion, including:

[0216] A data acquisition module, configured to acquire real-time operation data of a device and preprocess the data;

[0217] A data dimensionality reduction module, configured to perform dimensionality reduction processing on the preprocessed data using a principal component analysis algorithm;

[0218] A time series prediction module, configured to input the dimensionality-reduced data into a long short-term memory network model, and use the long short-term memory network model to model the time series dependence of the data to predict the future operation state of the device;

[0219] An anomaly detection module, configured to, based on the prediction result of the operation state, extract features from the data through a convolutional neural network, mine local non-linear relationships and potential patterns in the data, and judge the device state;

[0220] A fault warning module, configured to generate a warning signal for the operation state of the device according to the output result of the convolutional neural network, and generate alarm signals of different levels according to the set threshold levels;

[0221] A feedback regulation module, configured to adjust the operation parameters of the device through a feedback control mechanism according to the level of the alarm signal.

[0222] Embodiment 3

[0223] An embodiment of the present invention, which is different from the previous two embodiments, is that:

[0224] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes a contribution to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0225] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definitional sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0226] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0227] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and the like.

[0228] Embodiment 4

[0229] This is an embodiment of the present invention, which provides a fault warning method for a smart monitoring system based on multi-modal data fusion. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0230] This experiment was carried out in the actual operating environment of a power plant, and the experimental equipment used was the operating boiler and generator set. The equipment selected for the experiment included 4 boiler units, equipped with sensors for real-time collection of data such as current, voltage, and temperature. The experimental duration was 90 minutes, and each unit was adjusted for load fluctuations within the load range. The operating data collected by the equipment was uploaded to the data analysis platform through the PLC system.

[0231] Using the traditional prediction method, first, the real-time operating data of the equipment was obtained through the data acquisition system, including temperature, pressure, current, load, etc. Direct statistical analysis was performed on the data, and an algorithm based on the linear regression model was used to predict the future state of the equipment. After data analysis, the prediction results were generated, and whether there were signs of equipment failure was judged through the set threshold. If the threshold was exceeded, an alarm signal was sent. This method did not fully consider the temporal dependence of the equipment state and failed to perform effective data dimensionality reduction or feature extraction, resulting in relatively rough prediction results and insufficient sensitivity to changes in the equipment state.

[0232] When using the method of the present invention, first, the real-time operating data of the equipment was collected to obtain various parameters including temperature, pressure, current, load, etc. Then, the principal component analysis (PCA) algorithm was used to reduce the dimensionality of the collected high-dimensional data, remove the noise in the data, and retain the main features. Next, the dimensionality-reduced data was input into the long short-term memory network (LSTM) model to model the temporal dependence of the equipment and predict the future state of the equipment. Then, based on the prediction results of the LSTM model, the convolutional neural network (CNN) was used to further extract features from the data, mine the local non-linear relationships and potential patterns in the data, and finally generate an early warning signal for the operating state of the equipment. Different levels of alarm signals were generated according to the set threshold levels, and according to the alarm levels, the operating parameters of the equipment were adjusted through the feedback control mechanism to achieve real-time optimal control. The experimental results are shown in Table 1.

[0233] Table 1 Comparison table of experimental results

[0234]

[0235] It can be seen from the table that the method of the present invention is significantly superior to the traditional method in terms of prediction error and number of alarms. Specifically, the present invention reduces the dimensionality of the data through PCA, effectively reduces the data noise, and improves the accuracy of the LSTM model in temporal prediction. In contrast, due to the lack of consideration of temporal dependence, the traditional method has a larger prediction error and more alarm times. In addition, under the application of the feedback control mechanism, the method of the present invention can adjust the equipment parameters more timely, thereby optimizing the operating state of the equipment.

[0236] The method of the present invention solves the efficiency and accuracy problems of traditional methods in dealing with high-dimensional data through multi-modal data fusion and PCA dimensionality reduction. The LSTM model can effectively capture the temporal features of the device state, while the CNN further mines the non-linear relationships in the data. Compared with traditional methods, the present invention improves the prediction accuracy by performing deep learning modeling on the data and realizes the optimal adjustment of the device operation parameters through a feedback control mechanism. Therefore, the present invention can significantly improve the operation stability of the device, reduce the probability of failures, and effectively reduce the device maintenance cost.

[0237] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A fault warning method for a smart monitoring system based on multimodal data fusion, characterized in that: include: Collect real-time operation data of equipment; Use principal component analysis algorithm to reduce the dimension of operation data; The reduced-dimensional data is input into the long short-term memory network model, and the long short-term memory network model is used to model the temporal dependency of the data and predict the future operating status of the equipment; Based on the prediction results of the operating status, the convolutional neural network is used to extract features from the data, mine the local nonlinear relationships and potential patterns in the data, and judge the equipment status; Based on the output results of the convolutional neural network, different levels of alarm signals are generated according to the set threshold levels; According to the level of the alarm signal, the operating parameters of the equipment are adjusted through the feedback control mechanism.

2. The fault warning method of the intelligent monitoring system based on multimodal data fusion according to claim 1 is characterized by: The real-time operation data includes equipment operation data, environmental data and historical fault data.

3. The fault warning method of the intelligent monitoring system based on multimodal data fusion as claimed in claim 2 is characterized by: The dimensionality reduction process includes collecting real-time operation data of the equipment to form a data matrix; Standardize the data; Combine the time series window to perform local dimensionality reduction on the data and calculate the covariance matrix of each time window; Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvector; Select the first k principal components to project onto the principal component.

4. The fault warning method of the intelligent monitoring system based on multimodal data fusion as claimed in claim 3 is characterized by: The data after dimension reduction is expressed as: D pca =D·V k Where D is the original device operation data matrix with a dimension of m×n, m is the number of samples, n is the number of features, and V k is the eigenvector matrix containing the first k principal components, with dimension n×k, D pca is the data matrix after dimensionality reduction, with dimension m×k.

5. The fault warning method of the intelligent monitoring system based on multi-modal data fusion according to claim 4 is characterized by: The predicting of the future operation state of the device includes dynamically adjusting the time window length T according to the change rate γ of the device operation state. w , expressed as, Among them, Δx represents the change of device state, Δt represents the time step, T short and T long are the short-term and long-term time window lengths, respectively; The reduced data D pca According to the adaptive time window T w Split into multiple input sequences It is expressed as, Where i is the time step index, x t is the reduced dimension feature vector at time step t, from D pca ; Output multiple time window sequences The length of each sequence is T w , input long short-term memory network model.

6. The fault warning method of the intelligent monitoring system based on multi-modal data fusion according to claim 5 is characterized by: Construct multiple LSTM sub-networks to capture the multi-level temporal dependencies of device operation data, each sub-network corresponds to a different time scale T w , expressed as, Among them, j represents different time scale sub-networks, is the hidden state of the j-th sub-network at time t, is the input sequence of the jth time scale; The hidden state of each time scale sub-network The comprehensive time series feature h is obtained by weighted fusion t , expressed as, Among them, α j is the weight coefficient of the jth sub-network, satisfying N is the number of time-scale subnetworks; Based on the comprehensive time series feature h t , the attention weight β of each time step is calculated through the attention mechanism t , expressed as, Among them, W a and b a is a learnable parameter, T is the number of time steps; Apply the attention weights to the comprehensive temporal features to obtain the weighted features It is expressed as, Predicted value based on the model and the true value y t Calculate the prediction error e t , expressed as, Introduce a timing error feedback mechanism to give higher weights to time steps with errors greater than a preset threshold t , define the weighted loss function, expressed as, ω t =exp(-|e t |) Through the back propagation algorithm, according to the weighted loss function Update the model parameters θ, expressed as, Where η is the learning rate; Using the optimized LSTM model, based on weighted time series features Predict the future operating status of the equipment It is expressed as, Among them, Dense represents the fully connected layer, which is used to output the final prediction value.

7. The fault warning method of the intelligent monitoring system based on multi-modal data fusion according to claim 6 is characterized by: The determining device state comprises receiving a prediction result of a future running state of the device, formatting the prediction result, and constructing the prediction result into a multidimensional time series feature matrix, wherein, when the prediction result is a continuous value, the continuous value is feature expanded to form a multidimensional data matrix containing a plurality of related continuous features; when the prediction result is a probability distribution, the probability value is aggregated to form a multidimensional data matrix containing probabilities of each category; Normalizing the multidimensional time series feature matrix to obtain standardized feature data, and inputting the standardized feature data into a convolutional neural network in a two-dimensional form with dimensions of time step and feature dimension, wherein the feature dimension includes a continuous feature dimension or a categorical feature dimension; The standardized feature data is input into a multi-channel convolutional neural network. Each channel corresponds to a convolution extraction path for different types of features. The feature data is convolved by setting a convolution layer with adjustable filter number and convolution kernel size in each channel, and a deformable convolution layer is used to adapt to irregular patterns and local deformation features. Perform a pooling operation on the convolution output of each channel to reduce the dimension and extract the main features, and fuse the pooled outputs of all channels to obtain a comprehensive feature representation; An adaptive feature selection mechanism is introduced based on comprehensive feature representation. The weight of each feature is adjusted through the gating signal, the key features are dynamically highlighted and irrelevant features are suppressed, and the feature representation after feature selection is obtained. Use a multi-layer fully connected network and nonlinear activation function to perform pattern recognition on the feature representation after feature selection, mine the complex patterns in the data and output the final feature representation; Based on the final feature representation, the equipment status is classified and judged through the fully connected layer, and the Softmax activation function is used to probabilistically distribute the classification results. The equipment operation status category is determined by the maximum probability principle, and the equipment status judgment result is output.

8. An intelligent monitoring system based on multimodal data fusion, characterized in that: include: Data acquisition module, used to collect real-time operation data of equipment and pre-process the data; Data dimension reduction module, used to reduce the dimension of preprocessed data using principal component analysis algorithm; The time series prediction module is used to input the reduced-dimensional data into the long short-term memory network model, use the long short-term memory network model to model the time series dependency of the data, and predict the future operating status of the equipment; The anomaly detection module is used to extract features from data through convolutional neural networks based on the prediction results of the operating status, mine local nonlinear relationships and potential patterns in the data, and determine the status of the equipment; The fault warning module is used to generate a warning signal of the equipment operation status according to the output results of the convolutional neural network, and generate alarm signals of different levels according to the set threshold level; The feedback adjustment module is used to adjust the operating parameters of the equipment through a feedback control mechanism according to the level of the alarm signal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the fault warning method of the intelligent monitoring system based on multimodal data fusion described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the fault warning method of the intelligent monitoring system based on multimodal data fusion described in any one of claims 1 to 7 are implemented.

Images