Intelligent operation and maintenance management method and system based on multi-mode AI

Abstract

The invention discloses an intelligent operation and maintenance management method and system based on multi-modal AI, and belongs to the technical field of data artificial intelligence and industrial intelligent operation and maintenance crossing. The method comprises the steps that multi-modal data is collected and preprocessed, an improved VMD is used for conducting modal component decomposition on vibration signals, a first-order energy moment and a second-order energy moment are calculated to generate an initial vibration feature vector, and the initial vibration feature vector is calculated; a de-noising auto-encoder SDAE is used to obtain a final vibration feature vector, and the multi-modal data comprises vibration signals, energy consumption, images and text data; an improved VMD method is introduced to be combined with a Lagrange multiplier and a self-adaptive step length mechanism, a frequency domain iteration optimization solving process is constructed, meanwhile, nonlinear and deep feature enhancement is achieved through a denoising self-encoder, and therefore the extraction capacity and stability of a system for key modal components in complex vibration signals are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection between data artificial intelligence and industrial intelligent operation and maintenance, and more specifically, to an intelligent operation and maintenance management method and system based on multimodal AI. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of deep learning, natural language processing, and image recognition, intelligent operation and maintenance has gradually evolved from the traditional rule-driven model to data-driven, multimodal fusion intelligent analysis.

[0003] Especially in the field of industrial equipment operation and maintenance, it can more comprehensively reflect the equipment operation status by collecting and analyzing multi-source data;

[0004] However, most existing intelligent operation and maintenance systems have some shortcomings. In particular, in vibration signal processing, traditional modal decomposition methods are susceptible to noise interference, which leads to the problem of insufficient stability of decomposition results. This makes it difficult to accurately extract key features, affecting the accuracy of subsequent fault identification.

[0005] However, there is still a lack of an efficient and unified feature fusion strategy for image and text modalities. Existing technical methods often process each modality independently and then simply splice them together, which fails to explore the deep correlation between modalities.

[0006] Existing systems, however, lack the integration of expert knowledge and a decision-making recommendation generation mechanism. This results in operations personnel still having to manually analyze large amounts of information, failing to effectively reduce cognitive load.

[0007] Based on this, we designed an intelligent operation and maintenance management method and system based on multimodal AI. Summary of the Invention

[0008] In order to solve the problems raised in the background technology, the present invention provides the following technical solutions:

[0009] Intelligent operation and maintenance management methods based on multimodal AI include:

[0010] Collect and preprocess multimodal data, use improved VMD to decompose the vibration signal into modal components, calculate the first-order energy moment and second-order energy moment to generate the initial vibration feature vector, and use denoising autoencoder (SDAE) to obtain the final vibration feature vector. The multimodal data includes vibration signals, energy consumption, images, and text data.

[0011] Use the pre-trained ResNet-50 model and the Tissue P system to extract image feature vectors, use the pre-trained BERT model to extract text feature vectors, and calculate the energy consumption feature vector;

[0012] Build an MRO model to fuse multimodal data, build a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base;

[0013] Build a visual interface to display the analyzed data and store the data collected and analyzed

[0014] Furthermore, the use of the improved VMD to decompose the vibration signal into modal components and the use of the denoising autoencoder SDAE to obtain the final vibration feature vector includes:

[0015] The preprocessing includes denoising, normalizing and aligning the multimodal data by timestamp;

[0016] Use discrete Fourier transform (DFT) to calculate the spectrum of the preprocessed vibration signal data, calculate the spectrum amplitude, set the peak threshold based on historical regression estimation, and use the fixed threshold screening method to screen out the number of frequency points with spectrum amplitudes greater than the peak threshold as the number of modal components;

[0017] Define the bandwidth penalty term and fidelity term of VMD, as well as the VMD optimization objective function, use the alternating direction multiplier method (ADMM) to iteratively solve, update each modal component in the frequency domain, update the center frequency, calculate the total energy of the modal component at the current iteration number, and use the ratio method to calculate the modal change ratio;

[0018] Update the regularization parameters, calculate the adaptive step size, update the Lagrange multiplier, set the maximum number of iterations, stop the iteration when the maximum number of iterations is reached, and output the final modal component set;

[0019] The final modal component set is converted into time domain representation through inverse FFT, and the first-order energy moment and second-order energy moment are calculated respectively, and then normalized and combined into the initial vibration eigenvector;

[0020] The initial feature vector is divided into a training set and a test set. The signal-to-noise ratio (SNR) of the preprocessed vibration signal is calculated based on the Euclidean norm. Gaussian noise is dynamically added to the training set according to the SNR to generate noise-added features.

[0021] Construct a three-layer denoising autoencoder (SDAE). The training set is divided into batches and input into the denoising autoencoder (SDAE). With the goal of minimizing the reconstruction error, the Adam optimizer is used for update training. During the training process, the noise standard deviation is dynamically updated according to the change of SNR, and the trained denoising autoencoder (SDAE) is output.

[0022] The test set is input into the trained denoising autoencoder SDAE, and the output of the intermediate layer is extracted as the final vibration feature vector.

[0023] Furthermore, the extracting of energy consumption, image and text data includes:

[0024] Obtain alarm logs and device operation videos, use the inter-frame difference method to detect significantly changed frames in the video, convert them to RGB format, merge them with the image data into a unified image set, calculate the clarity of the images in the unified image set based on Laplace variance, set a clarity threshold, retain images in the unified image set with clarity greater than the clarity threshold, and match and annotate the images with alarms through time synchronization association.

[0025] Submit the annotated image-label pairs to the operation and maintenance personnel for inspection and correction, output the final image-label pairs, and extract the mini-batch training data from them as fine-tuning of the ResNet-50 model;

[0026] The pre-trained ResNet-50 model is fine-tuned to obtain the convolutional feature map of the final image-label pair. The convolutional feature map is globally average pooled to compress the spatial dimension into a single value to generate an image feature vector. The image feature vector is then subjected to dimensionality reduction using principal component analysis (PCA) and partitioned into 2×2 sub-blocks. The sub-blocks are then assigned to the four cells of the Tissue P system. The gradient of each sub-block is calculated using the Sobel operator in each cell and average pooled to generate an edge feature vector. Local binary pattern (LBP) is then used to extract the texture features of the sub-blocks.

[0027] The rule weight is calculated according to the grayscale mean of the sub-block, and the edge feature vector and texture feature of each sub-block are weighted summed as the sub-block feature vector;

[0028] Each cell receives the features of other cells, updates its own features based on cosine similarity, takes the average of each updated sub-block feature as the global feature vector, and uses principal component analysis (PCA) to reduce the dimensionality of the global feature vector;

[0029] The image feature vector processed by dimensionality reduction and the global feature vector are concatenated to obtain the final image feature vector;

[0030] Use the pre-trained BERT model to extract the feature vectors of all sentences and combine them into a text feature vector;

[0031] The preprocessed energy consumption data is divided into sub-matrices, and the covariance matrix of the sub-matrix of each time window is calculated. Principal component analysis (PCA) is used for dimensionality reduction. The statistical features of the covariance matrix of the dimensionality reduction process are calculated, and all statistical features are combined as the energy consumption feature vector.

[0032] Furthermore, the construction of the MRO model for multimodal data fusion includes:

[0033] Normalize the feature vectors of each modality and concatenate them into the initial fused high-dimensional feature vector;

[0034] Build the MRO model and use the Xavier initialization method to initialize the weights and biases of the MRO model;

[0035] Collect historical multimodal data with labels, extract features from each, and input them into the MRO model for training. Perform MRO forward propagation on the training data to calculate unimodal, bimodal interaction features, and trimodal interaction features. With the goal of minimizing the cross-entropy loss function, use the Adam optimizer to update weights and biases. Stop training when the maximum number of training steps is reached. Output the trained MRO model, input the initial fused high-dimensional feature vector, and output the final unimodal, bimodal, and trimodal features.

[0036] The final unimodal, bimodal, and trimodal features are fused through residual connections to obtain the final fused feature vector.

[0037] Furthermore, the CNN model is constructed to perform state prediction and fault category prediction on the fused multimodal data, and generate optimization suggestions based on the expert rule base, including:

[0038] Normalize the final fusion feature vector;

[0039] Extract the historical features of the most recent time point from the database of equipment operation data to form time series features and perform normalization processing;

[0040] Build a CNN model, use historical multimodal data for training, and output the trained CNN model;

[0041] Input the final fused feature vector and normalized time series features into the trained CNN model for state prediction;

[0042] If the failure risk probability is greater than half, perform fault diagnosis and prediction, otherwise continue monitoring;

[0043] Historical fault data and expert maintenance experience are collected to construct an expert rule base. Each rule is defined as a condition-result pair. Rule weights are assigned using KDE density estimation. The CNN model prediction results are matched with the rule base, and a confidence threshold is set. If the confidence exceeds the confidence threshold, an alarm is issued and the final fault classification result and maintenance recommendations are output to the maintenance personnel, who then confirm and review them.

[0044] Extract convolutional energy consumption features from the CNN model, fuse the state prediction and fault classification prediction with the convolutional energy consumption features, input them into the trained CNN model, and output the probability value of energy consumption anomaly;

[0045] Use the average value method to set the energy consumption anomaly threshold for different states, and compare the probability value of energy consumption anomaly with the energy consumption anomaly threshold predicted by the current state. If the energy consumption feature is greater than the energy consumption anomaly threshold, it is marked as a high energy consumption anomaly and optimization suggestions are generated. Otherwise, it is marked as normal.

[0046] Furthermore, the constructing of a visual interface to display the analyzed multimodal data refers to constructing a visual interface to display fault diagnosis results, status prediction trends and fault areas;

[0047] Use React to draw fault diagnosis bar charts, energy consumption trend line charts, and video players;

[0048] Users can click on the fault diagnosis bar chart to view a detailed diagnostic report, drag the timeline to view historical energy consumption and predicted trends, and support video playback and fault area highlighting.

[0049] Furthermore, the storage of multimodal data generated by collection and analysis refers to storing the multimodal data generated by collection and analysis in a database, and setting security access measures. The database will back up the stored data in the cloud, and regularly perform integrity checks on the stored data and backup data. After the test is completed, an integrity test record will be generated and stored synchronously in the database.

[0050] An intelligent operation and maintenance management system based on multimodal AI, including:

[0051] A collection and preprocessing module is used to collect and preprocess multimodal data, wherein the multimodal data includes vibration signals, energy consumption, images and text data;

[0052] The feature extraction module is used to decompose the vibration signal into modal components using the improved VMD, calculate the first-order energy moment and the second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector;

[0053] A model building module for extracting image feature vectors using a pre-trained ResNet-50 model and the Tissue P system, extracting text feature vectors using a pre-trained BERT model, and calculating energy consumption feature vectors for energy consumption;

[0054] The fusion generation module is used to build an MRO model for multimodal data fusion, construct a CNN model for status prediction and fault category prediction of the fused multimodal data, and generate optimization suggestions based on the expert rule base;

[0055] The visualization storage module is used to build a visualization interface to display the analyzed data and store the data collected and analyzed.

[0056] In summary, the present invention has the following beneficial effects:

[0057] By introducing an improved VMD method combined with Lagrange multipliers and an adaptive step-size mechanism, a frequency-domain iterative optimization solution process is constructed. At the same time, a denoising autoencoder is used to achieve nonlinear and deep feature enhancement, thereby effectively improving the system's ability to extract key modal components from complex vibration signals and its stability.

[0058] By designing a weighting mechanism for the Tissue P system and image texture features, combined with the BERT pre-trained model to extract deep text semantics, and through an interaction mechanism driven by inter-modal similarity, the system enhances the complementarity and discriminative capabilities of multimodal information, thus avoiding the limitations of simple splicing fusion in existing technologies.

[0059] By fusing modal information at different levels through the residual structure, the dimensional completeness and information retention rate of the fused expression are effectively improved, and the accuracy of the system in subsequent fault identification and state prediction is significantly enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 This is a flow chart of the intelligent operation and maintenance management method based on multimodal AI of the present invention;

[0062] Figure 2 Schematic diagram of the multimodal AI-based intelligent operation and maintenance management system of the present invention;

[0063] Figure 3 Improved collaborative feature extraction flow chart of VMD and SDAE in this invention;

[0064] Figure 4 This is the multimodal fusion flow chart of the present invention. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] Example:

[0067] The following is combined with Figure 1-4 The present invention is described in further detail.

[0068] See also Figure 1-4 , the present invention provides a technical solution: an intelligent operation and maintenance management method based on multimodal AI, such as Figure 1-4 Shown, including:

[0069] Collect and preprocess multimodal data, use improved VMD to decompose the vibration signal into modal components, calculate the first-order energy moment and second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector. Multimodal data includes vibration signals, energy consumption, images and text data;

[0070] Use the pre-trained ResNet-50 model and the Tissue P system to extract image feature vectors, use the pre-trained BERT model to extract text feature vectors, and calculate the energy consumption feature vector;

[0071] Build an MRO model to fuse multimodal data, build a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base;

[0072] Build a visual interface to display the analyzed data and store the data collected and analyzed.

[0073] S1. Collect and preprocess multimodal data, use improved VMD to decompose the vibration signal into modal components, calculate the first-order energy moment and second-order energy moment to generate the initial vibration feature vector, and use denoising autoencoder SDAE to obtain the final vibration feature vector. Multimodal data includes vibration signals, energy consumption, images and text data;

[0074] Among them, multimodal data includes vibration signals, energy consumption, images and text data;

[0075] Preprocessing, including denoising, normalization, and timestamp alignment of multimodal data;

[0076] Denoising of multimodal data, including using wavelet transform to denoise vibration signal data, using average filtering to denoise energy consumption data, and using median filtering to denoise image data;

[0077] Use the discrete Fourier transform (DFT) to calculate the spectrum of the preprocessed vibration signal data, perform smoothing, calculate the spectrum amplitude, set the peak threshold based on historical regression estimation, and use the fixed threshold screening method to filter out the number of frequency points with spectrum amplitudes greater than the peak threshold as the number of modal components K (set the number to at least 3 to ensure that the main modes are captured);

[0078] Define the bandwidth penalty term B and fidelity term F of VMD as follows:

[0079]

[0080] Where k is the index of the modal component, is the discrete time derivative, δ[n] is the discrete Dirac function, when n=0, δ[n]=1, otherwise it is 0, n is the discrete time index, is the kernel function of Hilbert transform, j is the imaginary unit, * is the discrete convolution operation, e -jωkn is a complex exponential function, α is a regularization parameter used to balance the smoothness of the modal component and the fidelity of signal reconstruction, the initial value is set to a universal value, x norm [n] is the pre-processed vibration signal data, u k [n] is the kth modal component IMF, ω k is the center frequency of the kth modal component, the initial value f k,peak is the peak frequency of the kth modal component, f s is the sampling frequency;

[0081] Based on the bandwidth penalty term B and the fidelity term F, the VMD optimization objective function is defined as follows:

[0082]

[0083] The alternating direction multiplier method ADMM is used to iteratively solve the VMD optimization problem;

[0084] Update each modal component in the frequency domain, formula:

[0085]

[0086] in, is the frequency domain representation of the kth modal component at the m+1th iteration, ω is the integral variable, N is the length of the vibration signal data, and represents the total number of sampling points of the discrete vibration signal. To preprocess the spectrum of the vibration signal, is the frequency domain sum of the modes except the kth mode at the mth iteration, is the frequency domain representation of the Lagrange multiplier, is the inertia parameter of the mth iteration;

[0087] Use the updated modal components to update the center frequency, formula:

[0088]

[0089] The updated center frequency is used to update the modal components in the next iteration;

[0090] Calculate the L2 norm of the modal component change at the current iteration number, calculate the total energy of the modal component at the current iteration number, and use the ratio method to divide the L2 norm of the modal component change at the current iteration number by the total energy of the modal component at the current iteration number as the modal change ratio;

[0091] Calculate the reconstruction residual of the current number of iterations, calculate the frequency interval factor of the current number of iterations, and update the regularization parameter. Formula:

[0092]

[0093] Among them, β is the adjustment factor, which is set using the fixed value method. is the frequency interval factor, which reflects the dispersion of the modal frequency. is the modal change ratio, making the regularization parameter more accurately adapt to the iterative dynamics

[0094] Calculate the adaptive step size, formula:

[0095]

[0096] Among them, τ m is the adaptive step size of the mth iteration, τ max is the maximum step size set using the empirical method, μ is the adjustment factor, set using the fixed value method;

[0097] Use adaptive step size to update the Lagrange multiplier, formula:

[0098]

[0099] in, is the Lagrange multiplier of the m+1th iteration;

[0100] Set the maximum number of iterations. When the maximum number of iterations is reached, stop the iteration and output the final modal component set.

[0101] The final modal component set is converted to time domain representation through inverse FFT. For each final modal component represented in the time domain, the first-order energy moment and the second-order energy moment are calculated respectively. The formula is:

[0102]

[0103] in, is the first-order energy moment of the kth final modal component, is a time series used for energy moment calculation, f s is the sampling frequency, u k [n] is the final modal component represented in the time domain, is the second-order energy moment of the kth final modal component;

[0104] Calculate the mean frequency of each final modal component in the final modal component, normalize the first-order energy moment, the second-order energy moment and the mean frequency, and combine them into an initial vibration eigenvector;

[0105] Divide the initial feature vector into training set and test set;

[0106] The signal-to-noise ratio (SNR) of the preprocessed vibration signal is calculated based on the Euclidean norm. Gaussian noise is dynamically added to the training set according to the SNR to generate the noise-added feature. The formula is:

[0107]

[0108] in, is the noise-added training set, V norm is the training set, is the noise mean 0, variance σ 2 Gaussian noise, σ = γ·SNR -1 is the noise standard deviation, γ is the adjustment factor that controls the noise intensity and is determined based on the experimental method, and SNR is the signal-to-noise ratio of the preprocessed vibration signal;

[0109] Construct a three-layer denoising autoencoder (SDAE). Each layer uses the ReLU activation function. The training set is divided into batches and input into the denoising autoencoder (SDAE). With the goal of minimizing the reconstruction error, the Adam optimizer is used for update training. During the training process, the noise standard deviation is dynamically updated according to the change in SNR, and the trained denoising autoencoder (SDAE) is output.

[0110] The test set is input into the trained denoising autoencoder SDAE, and the output of the intermediate layer, such as the lowest dimension of the encoder, is extracted as the final vibration feature vector.

[0111] By defining a penalty term (to control modal smoothness) and a fidelity term (to ensure signal reconstruction consistency) and constructing an objective function, the signal is effectively decoupled and the independent dynamic features in the vibration signal are captured, achieving the beneficial effects of improving the resolution of modal decomposition and controlling cross-interference. The ADMM method is used to iteratively update each modal component and center frequency in the frequency domain. Combining Lagrange multipliers with an adaptive step-size mechanism, a stable solution to the non-convex VMD optimization problem is achieved. The modal frequency domain results are converted to time domain signals via an inverse FFT, and the first-order and second-order energy moments and mean frequencies are calculated to form the initial feature vectors, extracting representative statistical features from a time-frequency domain perspective. The energy moment reflects the signal concentration and distribution trend, while the mean frequency measures the modal dominant frequency. By constructing a three-layer denoising autoencoder (SDAE), dynamically adjusting the noise parameters based on the SNR, and minimizing the reconstruction error using the Adam optimizer, a deep and robust nonlinear feature extraction mechanism is implemented. The extracted intermediate layer features are: the bottleneck output of the encoder, which represents the core expression of the original signal;

[0112] S2, use the pre-trained ResNet-50 model and Tissue P system to extract image feature vectors, use the pre-trained BERT model to extract text feature vectors, and calculate the energy consumption feature vector;

[0113] Among them, the "image feature vector" refers to the preliminary vector obtained by the ResNet network and after dimensionality reduction; the "sub-block feature vector" is generated by fusing the edge and texture information of each sub-block after image division; the "global feature vector" is obtained by summarizing the mean of all sub-block features; the final image feature vector is spliced ​​by the "image feature vector" and the "global feature vector" for subsequent multimodal fusion processing

[0114] Among them, alarm logs are obtained from the equipment management system, and equipment operation videos are obtained from the data center monitoring system, and aligned with the timestamp and image data;

[0115] Use the inter-frame difference method to detect significantly changed frames in the video, convert them into RGB format, merge them with the image data into a unified image set, calculate the clarity of the images in the unified image set based on the Laplace variance, set a clarity threshold, and retain images in the unified image set whose clarity is greater than the clarity threshold;

[0116] The retained images are associated with the data in the alarm log through time synchronization, and the images are matched with the alarms and annotated. Images that do not match the alarms are marked as "normal", and preliminary annotated image-label pairs are output, specifically the annotated image data;

[0117] Submit the preliminarily annotated image-label pairs to the operation and maintenance personnel for review and correction, output the final image-label pairs, and extract one-tenth of the data from them as small-batch training data for fine-tuning the ResNet-50 model;

[0118] Train the ResNet-50 model on ImageNet, remove the last fully connected layer, keep the convolutional part, freeze the first 47 layers, and train the last 3 layers using mini-batch training data;

[0119] Use the pre-trained ResNet-50 model to obtain the convolutional feature map of the final image-label pair, perform global average pooling on the convolutional feature map, compress the spatial dimension into a single value, generate an image feature vector, and use principal component analysis (PCA) to reduce the dimensionality of the image feature vector;

[0120] The final image-label pair is split into 2×2 sub-blocks and assigned to the four cells of the Tissue P system. Each cell processes the assigned sub-blocks in parallel based on edge detection rules and texture analysis rules. It should be noted that PCA maintains the spatial structure of the image after dimensionality reduction, and divides the feature map into 2×2 regions through two-dimensional slicing.

[0121] The edge detection rule refers to using the Sobel operator to calculate the gradient of the sub-block and average pooling the gradient of the sub-block to generate the edge feature vector;

[0122] Texture analysis rules refer to extracting sub-block texture features using local binary patterns (LBP);

[0123] The rule weight is calculated based on the grayscale mean of the sub-block. The formula is:

[0124]

[0125] in, and are the edge detection weight and texture analysis weight of the i-th sub-block, σ i and μ i are the grayscale mean (based on RGB channel average) and grayscale standard deviation of the i-th sub-block, reflecting the edge strength;

[0126] Based on the edge detection weight and texture analysis weight, the edge feature vector and texture feature of each sub-block are weighted summed as the sub-block feature vector;

[0127] Each cell receives the features of other cells and updates its own features based on cosine similarity. The formula is:

[0128]

[0129] in, is the updated sub-block feature of the i-th cell, is the sub-block feature vector, is the sub-block feature vector of the o-th cell, α iois the feature similarity weight, and The cosine similarity of

[0130] The average of each updated sub-block feature is taken as the global feature vector, and the global feature vector is subjected to dimensionality reduction processing using principal component analysis (PCA);

[0131] The image feature vector processed by dimensionality reduction and the global feature vector are concatenated to obtain the final image feature vector;

[0132] Use BERT's Chinese word segmenter to segment the preprocessed text data, add "[CLS]" at the beginning of each sentence and "[SEP]" at the end of each sentence to generate a token sequence;

[0133] Convert the token sequence into the input format of the BERT model, use the pre-trained BERT model to extract the feature vectors of all sentences, and combine them into a text feature vector;

[0134] The preprocessed energy consumption data is divided into sub-matrices according to a fixed time window. The covariance matrix of the sub-matrix of each time window is calculated. Principal component analysis (PCA) is used for dimensionality reduction. The statistical characteristics of the covariance matrix after dimensionality reduction are calculated, including mean, standard deviation and trend slope. All statistical characteristics are combined as the energy consumption feature vector through linear regression fitting.

[0135] By using the inter-frame difference method to extract frames with significant changes, preliminary filtering of frames with potential alarm information is achieved. By dividing the image region and processing each sub-block in parallel in the Tissue P system, local modeling of the image spatial distribution characteristics is achieved. Through the dual feature extraction strategy of edge and texture, combined with the grayscale mean to adjust the analysis weight, multi-dimensional modeling of image features is achieved. Through the inter-cell feature interaction mechanism, the complementarity and enhancement of feature vectors are achieved, avoiding the isolated distortion of local analysis. By introducing the pre-trained BERT language model, deep semantic modeling of operation and maintenance text is achieved, achieving the beneficial effect of providing high-quality text semantic support for joint analysis of images and text. Through covariance analysis and principal component dimensionality reduction, energy consumption fluctuation patterns and trend characteristics are extracted, achieving the beneficial effect of introducing energy consumption data as an auxiliary judgment basis in multimodal information fusion, enhancing the system's intelligent analysis capability.

[0136] S3. Build an MRO model to fuse multimodal data, build a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base;

[0137] Among them, the feature vectors of each modality are normalized and concatenated into the initial fused high-dimensional feature vector;

[0138] Build the MRO model and use the Xavier initialization method to initialize the weights and biases of the MRO model;

[0139] Collect historical multimodal data with labels, including fault diagnosis labels and energy-saving labels, extract features from each, and input them into the MRO model for training;

[0140] Perform MRO forward propagation on the training data and calculate the unimodal formula:

[0141]

[0142] in, is the unimodal interaction feature of modality l, and is the weight and bias of the fully connected layer of modality l, Z l is the normalized eigenvector of mode l;

[0143] For each pair of modes, such as vibration and image, calculate the bimodal interaction features, formula:

[0144]

[0145] Among them, F bi (i, j) is the bimodal interaction feature of modal l and l', and is the weight and bias of the fully connected layer of bimodal interaction, ⊙ is the element-wise multiplication used to capture the interaction between modalities, Z l' is the normalized eigenvector of mode l';

[0146] For each set of three modalities (such as vibration-image-energy consumption), the three-modal interaction feature is calculated using the formula:

[0147]

[0148] Among them, F tri (l,l',l") is the trimodal interaction feature, and are the weights and biases of the fully connected layer for trimodal interaction, Z l” is the normalized eigenvector of mode l”;

[0149] With the goal of minimizing the cross entropy loss function, the Adam optimizer is used to update the weights and biases. When the maximum number of training times is reached, the training is stopped and the trained MRO model is output. The initial fused high-dimensional feature vector is input and the final single modality is output. Dual Mode F bi (l,l')', trimodal features F tri (l,l',l')';

[0150] The final unimodal, bimodal, and trimodal features are fused through residual connections, formula:

[0151]

[0152] Among them, F MRO is the final fusion feature vector;

[0153] By calculating the unimodal interaction feature vector corresponding to each modality, deep semantic extraction is achieved within the modality, which is equivalent to nonlinear mapping on a specific modal dimension. By performing element-by-element multiplication and linear transformation on any two modalities, such as vibration and image, collaborative semantic extraction between modalities on a specific dimension is achieved. Through high-order combination operations and nonlinear mapping of the three modalities, high-order semantic interactions with the joint participation of the three types of modalities are captured, integrating information sources of more dimensions, which helps to identify deep potential patterns. By fusing the final unimodal, bimodal, and trimodal features through residual connections and directly adding together the features of different-order modalities, fusion is achieved while maintaining the original information of each order modality, avoiding the problem of information attenuation in deep network training.

[0154] Furthermore, the final fused feature vector is normalized;

[0155] Extract the historical features of the most recent time point from the database of equipment operation data to form time series features and perform normalization processing;

[0156] Build a CNN model, use historical multimodal data for training, and output the trained CNN model;

[0157] Input the final fused feature vector and normalized time series features into the trained CNN model for state prediction;

[0158] State prediction involves inputting normalized time series features into the convolution layer to generate time series convolution features. The pooling layer then processes the time series convolution features and inputs them into the prediction fully connected layer to output the state prediction probability (healthy, sub-healthy, or failure risk).

[0159] If the failure risk probability is greater than half, perform fault diagnosis and prediction, otherwise continue monitoring;

[0160] Fault diagnosis and prediction involves inputting the final fused feature vector into a trained CNN model, extracting spatial features from the convolutional layer, reducing the dimensionality of the extracted results in the pooling layer, and then inputting them into the fully connected layer. The fully connected layer uses Softmax to map the features into fault categories.

[0161] Collect historical failure data and expert maintenance experience to construct an expert rule base. Each rule is defined as a condition-result pair, such as "If the vibration frequency is greater than 100 Hz and the temperature is greater than 50°C, the server is overheating."

[0162] Use KDE density estimation to assign rule weights, formula:

[0163]

[0164] Among them, qr p is the weight of the pth rule, Acc p is the historical accuracy of the rule (based on historical data verification), and P is the total number of rules;

[0165] Match the CNN model prediction results with the rule base, formula:

[0166]

[0167] Among them, C fault is the credibility of the CNN model prediction results, Match is if Y fault With WR p If the fault labels of the input features are consistent and the condition combination is met, it means the match is successful, otherwise the match fails, success is 1, failure is 0, and Y fault is the prediction result of CNN model, WR p is the pth rule;

[0168] Set a credibility threshold and compare it with the credibility. If the credibility is greater than the credibility threshold, an alarm is issued and the final fault classification result and operation and maintenance suggestions (such as "replace the server radiator") are output to the operation and maintenance personnel, who then confirm and check.

[0169] Extract convolutional energy consumption features F from the CNN model energy , the state prediction S trend and fault classification prediction Y fault And the convolution energy consumption features are fused, the formula is:

[0170] F weighted =Concat(F energy ,β·C fault ·Y fault ,β'·S trend );

[0171] Among them, F weighted is the weighted fusion feature, β and β' are the weights of fault classification and state prediction, respectively, and β is set based on the probability that the state prediction is a fault;

[0172] Input the weighted fusion features into the trained CNN model and output the probability value of energy consumption anomaly;

[0173] Based on state prediction and fault classification prediction, as well as the expert rule base, the average value method is used to set energy consumption abnormality thresholds for different states;

[0174] The probability value of energy consumption anomaly is compared with the energy consumption anomaly threshold predicted by the current state. If the energy consumption characteristic is greater than the energy consumption anomaly threshold, it is marked as high energy consumption anomaly and an optimization suggestion is generated (such as "reduce cooling power by 10%"). Otherwise, it is marked as normal.

[0175] By making decision branches based on the probability of fault risk, the system response efficiency is improved, resource waste is reduced, and a risk-aware integrated prediction-diagnosis process is established to improve the timeliness and effectiveness of fault response. The fault diagnosis and prediction module extracts spatial features for classification, improves the diagnostic granularity and interpretability, provides a basis for subsequent operation and maintenance decisions, and shortens the troubleshooting time. By integrating state prediction, fault classification and convolution energy consumption features, and by building an expert rule base and assigning weights, the system's fault tolerance and credibility interpretation capabilities are improved. It is particularly suitable for auxiliary decision-making in scenarios with insufficient data samples or critical judgments, and can achieve comprehensive analysis of equipment operating efficiency and energy consumption, provide a basis for energy-saving optimization strategies, and improve the system's intelligent management level.

[0176] S4. Build a visual interface to display the analyzed data and store the data collected and analyzed;

[0177] Among them, a visual interface is built to display fault diagnosis results, status prediction trends and fault areas (key frames are extracted from video data);

[0178] Use React to draw fault diagnosis bar charts, energy consumption trend line charts, and video players;

[0179] Users can click on the fault diagnosis bar chart to view a detailed diagnostic report, drag the timeline to view historical energy consumption and predicted trends, and support video playback and fault area highlighting.

[0180] By using React to draw fault diagnosis bar charts, energy consumption trend line charts and video players, multi-dimensional linkage and intuitive display of fault diagnosis, status prediction and video playback are achieved. By allowing users to click on the fault diagnosis bar chart to view detailed diagnostic reports, drag the timeline to view historical energy consumption and predicted trends, and supporting the linkage between video playback and fault area highlighting, the interactivity and efficiency of the user experience are improved, and it can also provide rich fault diagnosis, energy consumption analysis and prediction functions at different levels.

[0181] Furthermore, the multimodal data collected and analyzed will be stored in the database, and security access measures will be set up. The database will back up the stored data to the cloud, and regularly perform integrity checks on the stored data and backup data. After the test is completed, the integrity test record will be generated and stored synchronously in the database.

[0182] By uniformly storing the multimodal data (such as images, text, etc.) generated by collection and analysis in the database, centralized management and long-term preservation of heterogeneous data are achieved. By setting up security access measures (such as access control, identity authentication and data encryption), data permission management and security protection are achieved. The database supports cloud backup of stored data to ensure that in the event of local device or system failure, attack or natural disaster, data can still be quickly restored through cloud backup, thereby enhancing the system's disaster recovery capability and high availability, and significantly improving the system's reliable operation level.

[0183] This embodiment also provides an intelligent operation and maintenance management system based on multimodal AI, including:

[0184] Collection and preprocessing module, used to collect and preprocess multimodal data, including vibration signals, energy consumption, images and text data;

[0185] The feature extraction module is used to decompose the vibration signal into modal components using the improved VMD, calculate the first-order energy moment and the second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector;

[0186] A model building module for extracting image feature vectors using a pre-trained ResNet-50 model and the Tissue P system, extracting text feature vectors using a pre-trained BERT model, and calculating energy consumption feature vectors for energy consumption;

[0187] The fusion generation module is used to build an MRO model for multimodal data fusion, construct a CNN model for status prediction and fault category prediction of the fused multimodal data, and generate optimization suggestions based on the expert rule base;

[0188] The visualization storage module is used to build a visualization interface to display the analyzed data and store the data collected and analyzed.

[0189] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0190] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

Claims

1. An intelligent operation and maintenance management method based on multimodal AI, characterized by: include: Collect and preprocess multimodal data, use improved VMD to decompose the vibration signal into modal components, calculate the first-order energy moment and second-order energy moment to generate the initial vibration feature vector, and use denoising autoencoder (SDAE) to obtain the final vibration feature vector. The multimodal data includes vibration signals, energy consumption, images, and text data. Use the pre-trained ResNet-50 model and the Tissue P system to extract image feature vectors, use the pre-trained BERT model to extract text feature vectors, and calculate the energy consumption feature vector; Build an MRO model to fuse multimodal data, build a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base; Build a visual interface to display the analyzed data and store the data collected and analyzed.

2. The intelligent operation and maintenance management method based on multimodal AI according to claim 1 is characterized in that: The improved VMD is used to decompose the vibration signal into modal components, and the denoising autoencoder SDAE is used to obtain the final vibration feature vector, including: The preprocessing includes denoising, normalizing and aligning the multimodal data by timestamp; Use discrete Fourier transform (DFT) to calculate the spectrum of the preprocessed vibration signal data, calculate the spectrum amplitude, set the peak threshold based on historical regression estimation, and use the fixed threshold screening method to screen out the number of frequency points with spectrum amplitudes greater than the peak threshold as the number of modal components; Define the bandwidth penalty term and fidelity term of VMD, as well as the VMD optimization objective function, use the alternating direction multiplier method (ADMM) to iteratively solve, update each modal component in the frequency domain, update the center frequency, calculate the total energy of the modal component at the current iteration number, and use the ratio method to calculate the modal change ratio; Update the regularization parameters, calculate the adaptive step size, update the Lagrange multiplier, set the maximum number of iterations, stop the iteration when the maximum number of iterations is reached, and output the final modal component set; The final modal component set is converted into time domain representation through inverse FFT, and the first-order energy moment and second-order energy moment are calculated respectively, and then normalized and combined into the initial vibration eigenvector; The initial feature vector is divided into a training set and a test set. The signal-to-noise ratio (SNR) of the preprocessed vibration signal is calculated based on the Euclidean norm. Gaussian noise is dynamically added to the training set according to the SNR to generate noise-added features. Construct a three-layer denoising autoencoder (SDAE). The training set is divided into batches and input into the denoising autoencoder (SDAE). With the goal of minimizing the reconstruction error, the Adam optimizer is used for update training. During the training process, the noise standard deviation is dynamically updated according to the change of SNR, and the trained denoising autoencoder (SDAE) is output. The test set is input into the trained denoising autoencoder SDAE, and the output of the intermediate layer is extracted as the final vibration feature vector.

3. The intelligent operation and maintenance management method based on multimodal AI according to claim 2 is characterized in that: The extraction of energy consumption, image and text data includes: Obtain alarm logs and device operation videos, use the inter-frame difference method to detect significantly changed frames in the video, convert them to RGB format, merge them with the image data into a unified image set, calculate the clarity of the images in the unified image set based on Laplace variance, set a clarity threshold, retain images in the unified image set with clarity greater than the clarity threshold, and match and annotate the images with alarms through time synchronization association. Submit the annotated image-label pairs to the operation and maintenance personnel for inspection and correction, output the final image-label pairs, and extract the mini-batch training data from them as fine-tuning of the ResNet-50 model; The pre-trained ResNet-50 model is fine-tuned to obtain the convolutional feature map of the final image-label pair. The convolutional feature map is globally average pooled to compress the spatial dimension into a single value to generate an image feature vector. The image feature vector is then subjected to dimensionality reduction using principal component analysis (PCA) and partitioned into 2×2 sub-blocks. The sub-blocks are then assigned to the four cells of the Tissue P system. The gradient of each sub-block is calculated using the Sobel operator in each cell and average pooled to generate an edge feature vector. Local binary pattern (LBP) is then used to extract the texture features of the sub-blocks. The rule weight is calculated according to the grayscale mean of the sub-block, and the edge feature vector and texture feature of each sub-block are weighted summed as the sub-block feature vector; Each cell receives the features of other cells, updates its own features based on cosine similarity, takes the average of each updated sub-block feature as the global feature vector, and uses principal component analysis (PCA) to reduce the dimensionality of the global feature vector; The image feature vector processed by dimensionality reduction and the global feature vector are concatenated to obtain the final image feature vector; Use the pre-trained BERT model to extract the feature vectors of all sentences and combine them into a text feature vector; The preprocessed energy consumption data is divided into sub-matrices, and the covariance matrix of the sub-matrix of each time window is calculated. Principal component analysis (PCA) is used for dimensionality reduction. The statistical features of the covariance matrix of the dimensionality reduction process are calculated, and all statistical features are combined as the energy consumption feature vector.

4. The intelligent operation and maintenance management method based on multimodal AI according to claim 3 is characterized in that: The construction of the MRO model for multimodal data fusion includes: Normalize the feature vectors of each modality and concatenate them into the initial fused high-dimensional feature vector; Build the MRO model and use the Xavier initialization method to initialize the weights and biases of the MRO model; Collect historical multimodal data with labels, extract features from each, and input them into the MRO model for training. Perform MRO forward propagation on the training data to calculate unimodal, bimodal interaction features, and trimodal interaction features. With the goal of minimizing the cross-entropy loss function, use the Adam optimizer to update weights and biases. Stop training when the maximum number of training steps is reached. Output the trained MRO model, input the initial fused high-dimensional feature vector, and output the final unimodal, bimodal, and trimodal features. The final unimodal, bimodal, and trimodal features are fused through residual connections to obtain the final fused feature vector.

5. The multimodal AI-based intelligent operation and maintenance management method according to claim 5, characterized in that: The CNN model is constructed to perform status prediction and fault category prediction on the fused multimodal data, and generate optimization suggestions based on the expert rule base, including: Normalize the final fusion feature vector; Extract the historical features of the most recent time point from the database of equipment operation data to form time series features and perform normalization processing; Build a CNN model, use historical multimodal data for training, and output the trained CNN model; Input the final fused feature vector and normalized time series features into the trained CNN model for state prediction; If the failure risk probability is greater than half, perform fault diagnosis and prediction, otherwise continue monitoring; Historical fault data and expert maintenance experience are collected to construct an expert rule base. Each rule is defined as a condition-result pair. Rule weights are assigned using KDE density estimation. The CNN model prediction results are matched with the rule base, and a confidence threshold is set. If the confidence exceeds the confidence threshold, an alarm is issued and the final fault classification result and maintenance recommendations are output to the maintenance personnel, who then confirm and review them. Extract convolutional energy consumption features from the CNN model, fuse the state prediction and fault classification prediction with the convolutional energy consumption features, input them into the trained CNN model, and output the probability value of energy consumption anomaly; Use the average value method to set the energy consumption anomaly threshold for different states, and compare the probability value of energy consumption anomaly with the energy consumption anomaly threshold predicted by the current state. If the energy consumption feature is greater than the energy consumption anomaly threshold, it is marked as a high energy consumption anomaly and optimization suggestions are generated. Otherwise, it is marked as normal.

6. The intelligent operation and maintenance management method based on multimodal AI according to claim 5 is characterized in that: The said constructing a visual interface to display the analyzed multimodal data refers to constructing a visual interface to display the fault diagnosis results, status prediction trends and fault areas; Use React to draw fault diagnosis bar charts, energy consumption trend line charts, and video players; Users can click on the fault diagnosis bar chart to view a detailed diagnostic report, drag the timeline to view historical energy consumption and predicted trends, and support video playback and fault area highlighting.

7. The intelligent operation and maintenance management method based on multimodal AI according to claim 5 is characterized in that: The storing of the multimodal data generated by the collection and analysis refers to storing the multimodal data generated by the collection and analysis in a database, and setting security access measures. The database backs up the stored data in the cloud, and regularly performs integrity checks on the stored data and the backup data. After the test is completed, an integrity test record is generated and synchronously stored in the database.

8. An intelligent operation and maintenance management system based on multimodal AI, based on the intelligent operation and maintenance management method based on multimodal AI according to any one of claims 1 to 7, characterized in that: include: A collection and preprocessing module is used to collect and preprocess multimodal data, wherein the multimodal data includes vibration signals, energy consumption, images and text data; The feature extraction module is used to decompose the vibration signal into modal components using the improved VMD, calculate the first-order energy moment and the second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector; A model building module for extracting image feature vectors using a pre-trained ResNet-50 model and the Tissue P system, extracting text feature vectors using a pre-trained BERT model, and calculating energy consumption feature vectors for energy consumption; The fusion generation module is used to build an MRO model for multimodal data fusion, construct a CNN model for status prediction and fault category prediction of the fused multimodal data, and generate optimization suggestions based on the expert rule base; The visualization storage module is used to build a visualization interface to display the analyzed data and store the data collected and analyzed.

Images