Intelligent Operation and Maintenance Management Methods and Systems Based on Multimodal AI

Abstract

This invention discloses an intelligent operation and maintenance management method and system based on multimodal AI, belonging to the interdisciplinary field of data artificial intelligence and industrial intelligent operation and maintenance. The method includes collecting and preprocessing multimodal data, using an improved VMD method to perform modal component decomposition on vibration signals, calculating first-order and second-order energy moments to generate initial vibration feature vectors, and using a denoising autoencoder (SDAE) to obtain the final vibration feature vector. The multimodal data includes vibration signals, energy consumption, images, and text data. By introducing an improved VMD method combined with Lagrange multipliers and an adaptive step-size mechanism, a frequency domain iterative optimization solution process is constructed. Simultaneously, a denoising autoencoder is used to achieve nonlinear, deep-level feature enhancement, thereby effectively improving the system's ability and stability in extracting key modal components from complex vibration signals.

Description

Technical Field

[0001] This invention relates to the field of data artificial intelligence and industrial intelligent operation and maintenance technology, and more specifically, it relates to an intelligent operation and maintenance management method and system based on multimodal AI. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially with the widespread application of branches such as deep learning, natural language processing and image recognition, intelligent operation and maintenance is gradually evolving from the traditional rule-driven model to data-driven, multimodal fusion intelligent analysis.

[0003] Especially in the field of industrial equipment operation and maintenance, it can collect and analyze multi-source data to more comprehensively reflect the operating status of the equipment;

[0004] However, most existing intelligent operation and maintenance systems have some shortcomings, especially in vibration signal processing. Traditional mode decomposition methods are easily affected by noise interference, which leads to insufficient stability of the decomposition results, making it difficult to accurately extract key features and affecting the accuracy of subsequent fault identification.

[0005] For image and text modalities, there is still a lack of an efficient and unified feature fusion strategy. Existing technical methods often process each modality independently and then simply splice them together, which fails to uncover the deep correlation between modalities.

[0006] In existing systems, there is a lack of consideration for the integration of expert knowledge, resulting in a lack of decision-oriented suggestion generation mechanisms. This means that maintenance personnel still need to manually analyze a large amount of information, which cannot effectively reduce cognitive load.

[0007] Based on this, we designed an intelligent operation and maintenance management method and system based on multimodal AI. Summary of the Invention

[0008] To address the problems mentioned in the background section, the present invention provides the following technical solution:

[0009] Intelligent operation and maintenance management methods based on multimodal AI include:

[0010] Multimodal data is collected and preprocessed. Modal component decomposition of vibration signals is performed using improved VMD. First-order energy moments and second-order energy moments are calculated to generate initial vibration feature vectors. The final vibration feature vector is obtained using a denoising autoencoder SDAE. The multimodal data includes vibration signals, energy consumption, image data, and text data.

[0011] Image feature vectors are extracted using a pre-trained ResNet-50 model and the Tissue P system, and text feature vectors are extracted using a pre-trained BERT model. Energy consumption feature vectors are calculated.

[0012] A multimodal data fusion model is constructed to perform MRO model fusion, and a CNN model is constructed to predict the state and fault category of the fused multimodal data. Optimization suggestions are generated based on the expert rule base.

[0013] Build a visual interface to display and analyze the data, and store the collected and analyzed data.

[0014] Furthermore, the step of using improved VMD to perform modal component decomposition on the vibration signal and using a denoising autoencoder SDAE to obtain the final vibration feature vector includes:

[0015] The preprocessing includes denoising, normalization, and timestamp alignment of the multimodal data;

[0016] The spectrum of the preprocessed vibration signal data is calculated using Discrete Fourier Transform (DFT). The spectral amplitude is calculated, and a peak threshold is set based on historical regression estimation. The number of frequency points with spectral amplitude greater than the peak threshold is selected using a fixed threshold screening method as the number of modal components.

[0017] Define the bandwidth penalty term and fidelity term of VMD, as well as the VMD optimization objective function. Solve iteratively using the Alternating Direction Multiplier Method (ADMM). Update each modal component in the frequency domain, update the center frequency, calculate the total energy of the modal components in the current iteration number, and calculate the modal change ratio using the ratio method.

[0018] Update the regularization parameters, calculate the adaptive step size, update the Lagrange multipliers, set the maximum number of iterations, stop the iteration when the maximum number of iterations is reached, and output the final set of modal components.

[0019] For the final set of modal components, the inverse FFT is used to convert it into a time domain representation. The first-order energy moment and the second-order energy moment are calculated separately, normalized, and combined into an initial vibration feature vector.

[0020] The initial feature vector is divided into a training set and a test set. The signal-to-noise ratio (SNR) of the preprocessed vibration signal is calculated based on the Euclidean norm. Gaussian noise is dynamically added to the training set according to the SNR to generate noisy features.

[0021] A three-layer denoising autoencoder SDAE is constructed. The training set is divided into batches and input into the denoising autoencoder SDAE. With the goal of minimizing the reconstruction error, the Adam optimizer is used for updating and training. During the training process, the noise standard deviation is dynamically updated according to the SNR change, and the trained denoising autoencoder SDAE is output.

[0022] The test set is input into the trained denoising autoencoder SDAE, and the output of the intermediate layer is extracted as the final vibration feature vector.

[0023] Furthermore, the extraction of energy consumption, image, and text data includes:

[0024] Obtain alarm logs and device operation videos, use the inter-frame difference method to detect significantly changing frames in the video, convert them to RGB format, merge them with image data into a unified image set, calculate the image sharpness in the unified image set based on Laplacian variance, set a sharpness threshold, retain images in the unified image set with a sharpness greater than the sharpness threshold, and match and label the images with alarms through time synchronization association.

[0025] The labeled image-label pairs are submitted to the operations and maintenance personnel for inspection and correction. The final image-label pairs are output, and mini-batch training data is extracted from them for fine-tuning the ResNet-50 model.

[0026] The pre-trained ResNet-50 model is fine-tuned to obtain the convolutional feature maps of the final image-label pairs. Global average pooling is performed on the convolutional feature maps to compress the spatial dimension into a single value, generating image feature vectors. Principal component analysis (PCA) is used to reduce the dimensionality of the image feature vectors, dividing them into 2×2 sub-blocks, which are then assigned to the four cells of the Tissue P system. The gradient of the sub-block is calculated using the Sobel operator in each cell, and average pooling is performed on the gradient of the sub-block to generate edge feature vectors. Local binary mode (LBP) is used to extract the texture features of the sub-blocks.

[0027] The rule weights are calculated based on the gray-scale mean of the sub-blocks. The edge feature vectors and texture features of each sub-block are weighted and summed to form the sub-block feature vector.

[0028] Each cell receives features from other cells, updates its own features based on cosine similarity, calculates the average of the updated sub-block features as the global feature vector, and performs dimensionality reduction processing on the global feature vector using principal component analysis (PCA).

[0029] The image feature vector and the global feature vector after dimensionality reduction are concatenated to obtain the final image feature vector.

[0030] The pre-trained BERT model is used to extract the feature vectors of all sentences and combine them into a text feature vector;

[0031] The preprocessed energy consumption data is divided into sub-matrices. The covariance matrix of each time window sub-matrix is ​​calculated. Principal component analysis (PCA) is used for dimensionality reduction. The statistical characteristics of the dimensionality-reduced covariance matrix are calculated. All statistical characteristics are combined as the energy consumption feature vector.

[0032] Furthermore, the construction of the MRO model for multimodal data fusion includes:

[0033] The feature vectors of each modality are normalized and concatenated to form an initial fused high-dimensional feature vector.

[0034] Construct the MRO model and initialize its weights and biases using the Xavier initialization method;

[0035] Collect labeled historical multimodal data, extract features from each, and input them into the MRO model for training. Perform MRO forward propagation on the training data to calculate unimodal, bimodal, and trimodal interaction features. With the goal of minimizing the cross-entropy loss function, use the Adam optimizer to update the weights and biases. Stop training when the maximum number of training iterations is reached, and output the trained MRO model. Input the initially fused high-dimensional feature vector to output the final unimodal, bimodal, and trimodal models.

[0036] The final unimodal, bimodal, and trimodal features are fused through residual connections to obtain the final fused feature vector.

[0037] Furthermore, the constructed CNN model performs state prediction and fault category prediction on the fused multimodal data, and generates optimization suggestions based on an expert rule base, including:

[0038] The final fused feature vector is then normalized.

[0039] Historical features from the most recent time point are extracted from the database of equipment operation data to form time series features, which are then normalized.

[0040] Build a CNN model, train it using historical multimodal data, and output the trained CNN model;

[0041] The final fused feature vector and normalized time series features are input into the trained CNN model for state prediction;

[0042] If the probability of failure risk is greater than 50%, perform failure diagnosis and prediction; otherwise, continue monitoring.

[0043] Collect historical fault data and expert operation and maintenance experience to construct an expert rule base. Each rule is defined as a condition-outcome pair. KDE density estimation is used to assign rule weights. The prediction results of the CNN model are matched with the rule base. A confidence threshold is set. If the confidence is greater than the confidence threshold, an alarm is triggered and the final fault classification result and operation and maintenance suggestions are output to the operation and maintenance personnel for confirmation and inspection.

[0044] Extract convolutional energy consumption features from the CNN model, fuse state prediction, fault classification prediction, and convolutional energy consumption features, input them into the trained CNN model, and output the probability value of abnormal energy consumption.

[0045] The average value method is used to set energy consumption anomaly thresholds for different states. The probability value of energy consumption anomaly is compared with the energy consumption anomaly threshold predicted for the current state. If the energy consumption characteristic is greater than the energy consumption anomaly threshold, it is marked as high energy consumption anomaly and optimization suggestions are generated; otherwise, it is marked as normal.

[0046] Furthermore, the construction of a visual interface to display and analyze multimodal data refers to the construction of a visual interface to display fault diagnosis results, status prediction trends, and fault areas;

[0047] Use React to create fault diagnosis bar charts, energy consumption trend line charts, and video players;

[0048] Users can click on the fault diagnosis bar chart to view detailed diagnostic reports, drag the timeline to view historical energy consumption and predicted trends, and support video playback and fault area highlighting linkage.

[0049] Furthermore, the storage of the multimodal data generated from collection and analysis refers to storing the collected and analyzed multimodal data in a database, setting up secure access measures, backing up the stored data to the cloud, and periodically performing integrity checks on the stored data and backup data. After the checks are completed, integrity check records are generated and synchronously stored in the database.

[0050] An intelligent operation and maintenance management system based on multimodal AI includes:

[0051] A collection and preprocessing module is used to collect and preprocess multimodal data, including vibration signals, energy consumption, image data, and text data.

[0052] The feature extraction module is used to perform modal component decomposition on the vibration signal using improved VMD, calculate the first-order energy moment and the second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector.

[0053] The model building module is used to extract image feature vectors using a pre-trained ResNet-50 model and the Tissue P system, extract text feature vectors using a pre-trained BERT model, and calculate the energy consumption feature vector.

[0054] The fusion generation module is used to build an MRO model for multimodal data fusion, build a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base.

[0055] The visualization and storage module is used to build a visual interface to display the analyzed data and to store the collected and analyzed data.

[0056] In summary, the present invention has the following beneficial effects:

[0057] By introducing an improved VMD method combined with Lagrange multipliers and an adaptive step size mechanism, a frequency domain iterative optimization solution process is constructed. At the same time, a denoising autoencoder is used to achieve nonlinear and deep feature enhancement, thereby effectively improving the system's ability to extract key modal components from complex vibration signals and its stability.

[0058] By designing a weighted mechanism for the Tissue P system and image texture features, and combining it with the BERT pre-trained model to extract deep text semantics, the system enhances the complementarity and discriminative ability of multimodal information through an interaction mechanism driven by intermodal similarity, thereby avoiding the limitations of simple splicing fusion in the original technology.

[0059] By fusing modal information from different levels through residual structures, the dimensional completeness and information retention rate of the fused representation are effectively improved, and the accuracy of the system in subsequent fault identification and state prediction is significantly enhanced. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of the intelligent operation and maintenance management method based on multimodal AI of the present invention;

[0062] Figure 2 This is a schematic diagram of the intelligent operation and maintenance management system based on multimodal AI of the present invention;

[0063] Figure 3 This is a flowchart illustrating the improved collaborative feature extraction process of VMD and SDAE in this invention.

[0064] Figure 4 This is a flowchart of the multimodal fusion process of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example:

[0067] The following is in conjunction with the appendix Figure 1-4 The present invention will be described in further detail below.

[0068] Please see Figure 1-4 This invention provides a technical solution: an intelligent operation and maintenance management method based on multimodal AI, such as... Figure 1-4 As shown, it includes:

[0069] Multimodal data is collected and preprocessed. Modal component decomposition of vibration signals is performed using improved VMD. First-order and second-order energy moments are calculated to generate initial vibration feature vectors. The final vibration feature vector is obtained using a denoising autoencoder SDAE. Multimodal data includes vibration signals, energy consumption, image and text data.

[0070] Image feature vectors are extracted using a pre-trained ResNet-50 model and the Tissue P system, and text feature vectors are extracted using a pre-trained BERT model. Energy consumption feature vectors are calculated.

[0071] A multimodal data fusion model is constructed to perform MRO model fusion, and a CNN model is constructed to predict the state and fault category of the fused multimodal data. Optimization suggestions are generated based on the expert rule base.

[0072] Build a visual interface to display the analyzed data and store the collected and analyzed data.

[0073] S1. Collect and preprocess multimodal data, use improved VMD to decompose the vibration signal into modal components, calculate the first-order energy moment and the second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector. Multimodal data includes vibration signal, energy consumption, image and text data.

[0074] Multimodal data includes vibration signals, energy consumption, image and text data;

[0075] Preprocessing includes denoising, normalization, and timestamp alignment of multimodal data;

[0076] Denoising of multimodal data includes using wavelet transform to denoise vibration signal data, using average filtering to denoise energy consumption data, and using median filtering to denoise image data;

[0077] The spectrum of the preprocessed vibration signal data is calculated using Discrete Fourier Transform (DFT) and smoothed. The spectral amplitude is calculated, and a peak threshold is set based on historical regression estimation. The number of frequency points with spectral amplitude greater than the peak threshold is selected using a fixed threshold screening method, which is used as the number of modal components K (the minimum number is set to 3 to ensure that the main modes are captured).

[0078] Define the bandwidth penalty term B and fidelity term F for VMD, using the following formula:

[0079]

[0080] Where k is the index of the modal component, Let δ[n] be the discrete-time derivative, δ[n] be the discrete Dirac function, δ[n] = 1 when n = 0, otherwise 0, and n be the discrete-time index. Here, is the kernel function of the Hilbert transform, j is the imaginary unit, * represents the discrete convolution operation, and e is the kernel function of the Hilbert transform. -jωkn It is a complex exponential function, where α is a regularization parameter used to balance the smoothness of modal components and the fidelity of signal reconstruction. The initial value is set to a universal value, x. norm [n] represents the preprocessed vibration signal data, u k [n] represents the IMF of the k-th modal component, ω k Let be the center frequency of the k-th modal component, with initial value . f k,peak f is the peak frequency of the k-th modal component. s The sampling frequency;

[0081] Based on the bandwidth penalty term B and the fidelity term F, the VMD optimization objective function is defined as follows:

[0082]

[0083] The VMD optimization problem is solved using the Alternating Direction Multiplier Method (ADMM) iterative method.

[0084] Update each modal component in the frequency domain, using the formula:

[0085]

[0086] in, Let ω be the frequency domain representation of the k-th modal component in the (m+1)th iteration, where ω is the integral variable, N is the length of the vibration signal data, and represents the total number of sampling points of the discrete vibration signal. To preprocess the spectrum of the vibration signal, Let f(x) be the frequency domain sum of all modes except the k-th mode in the m-th iteration. For the frequency domain representation of Lagrange multipliers, Let be the inertial parameter for the m-th iteration;

[0087] Update the center frequency using the updated modal components, formula:

[0088]

[0089] The updated center frequency is used for the modal component update in the next iteration;

[0090] Calculate the L2 norm of the modal component change at the current iteration number, calculate the total energy of the modal components at the current iteration number, and use the ratio method to take the value of the L2 norm of the modal component change at the current iteration number divided by the total energy of the modal components at the current iteration number as the modal change ratio;

[0091] Calculate the reconstruction residual for the current iteration, calculate the frequency interval factor for the current iteration, and update the regularization parameter. Formula:

[0092]

[0093] Where β is an adjustment factor, set using a fixed value method. This is the frequency spacing factor, reflecting the degree of dispersion of modal frequencies. To adjust the modal variation ratio, the regularization parameter is made to more accurately adapt to the iterative dynamics.

[0094] Calculate the adaptive step size using the following formula:

[0095]

[0096] Where, τ m Let τ be the adaptive step size for the m-th iteration. max The maximum step size is set using the empirical method, and μ is an adjustment factor set using the fixed value method.

[0097] Using an adaptive step size, update the Lagrange multipliers, as shown in the formula:

[0098]

[0099] in, For the (m+1)th iteration, the Lagrange multiplier is used.

[0100] Set the maximum number of iterations. When the maximum number of iterations is reached, stop iterating and output the final set of modal components.

[0101] For the final modal component set, it is converted to a time-domain representation using inverse FFT. For each final modal component in the time-domain representation, the first-order energy moment and the second-order energy moment are calculated separately, using the following formula:

[0102]

[0103] in, Let the first-order energy moment of the k-th final modal component be . This is a time series, used for energy moment calculation, f s U is the sampling frequency. k [n] represents the final modal components in the time domain representation. Let be the second-order energy moment of the k-th final modal component;

[0104] Calculate the mean frequency of each final modal component in the final modal components, normalize the first-order energy moment, second-order energy moment and mean frequency, and combine them into an initial vibration characteristic vector;

[0105] The initial feature vectors are divided into a training set and a test set;

[0106] The signal-to-noise ratio (SNR) of the preprocessed vibration signal is calculated based on the Euclidean norm. Gaussian noise is then dynamically added to the training set according to the SNR to generate noisy features. The formula is:

[0107]

[0108] in, For a noisy training set, V norm For the training set, The noise mean is 0 and the variance is σ. 2 Gaussian noise, σ = γ·SNR -1 γ is the noise standard deviation, γ is the adjustment factor to control the noise intensity and is determined based on experimental methods, and SNR is the signal-to-noise ratio of the preprocessed vibration signal.

[0109] A three-layer denoising autoencoder SDAE is constructed, with each layer using the ReLU activation function. The training set is divided into batches and input into the denoising autoencoder SDAE. With the goal of minimizing the reconstruction error, the Adam optimizer is used for updating and training. During the training process, the noise standard deviation is dynamically updated according to the SNR change, and the trained denoising autoencoder SDAE is output.

[0110] The test set is input into the trained denoising autoencoder SDAE, and the output of the intermediate layers, such as the lowest dimension of the encoder, is extracted as the final vibration feature vector.

[0111] By defining a penalty term (to control modal smoothness) and a fidelity term (to ensure signal reconstruction consistency) and constructing an objective function, the signal is effectively decoupled, capturing independent dynamic features in the vibration signal. This achieves the beneficial effects of improving the resolution of modal decomposition and controlling cross-interference. The ADMM method iteratively updates each modal component and center frequency in the frequency domain. Combined with Lagrange multipliers and an adaptive step-size mechanism, a stable solution to the non-convex VMD optimization problem is achieved. The modal frequency domain results are converted into time domain signals using inverse FFT, and the first / second order energy moments and mean frequency are calculated to form an initial feature vector, realizing the extraction of representative statistical features from a time-frequency domain perspective. The energy moment reflects the signal concentration and distribution trend, and the mean frequency measures the modal dominant frequency. By constructing a three-layer denoising autoencoder SDAE, dynamically adjusting noise parameters using SNR, and minimizing reconstruction error using the Adam optimizer, a deep and robust nonlinear feature extraction mechanism is achieved. The extracted intermediate layer feature is the encoder bottleneck output, representing the core expression of the original signal.

[0112] S2. Use the pre-trained ResNet-50 model and Tissue P system to extract image feature vectors, use the pre-trained BERT model to extract text feature vectors, and calculate the energy consumption feature vector.

[0113] Here, "image feature vector" refers to the initial vector obtained and dimensionality-reduced by the ResNet network; "sub-block feature vector" is generated by fusing the edge and texture information of each sub-block after image division; "global feature vector" is obtained by summing the average features of all sub-blocks; the final image feature vector is formed by concatenating the "image feature vector" and the "global feature vector" for subsequent multimodal fusion processing.

[0114] This includes obtaining alarm logs from the device management system and device operation videos from the data center monitoring system, aligning them by timestamp and image data;

[0115] The inter-frame difference method is used to detect significantly changing frames in the video, which are then converted to RGB format and merged with image data into a unified image set. The sharpness of the images in the unified image set is calculated based on the Laplacian variance, a sharpness threshold is set, and images with a sharpness greater than the sharpness threshold in the unified image set are retained.

[0116] The retained images are associated with the data in the alarm log through time synchronization. The images and alarms are matched and labeled. Images that do not match alarms are marked as "normal". The preliminary labeled image-label pairs are output, which are the labeled image data.

[0117] The initially labeled image-label pairs are submitted to the operations and maintenance personnel for inspection and correction. The final image-label pairs are output, and one-tenth of the data is extracted from them as mini-batch training data for fine-tuning the ResNet-50 model.

[0118] Train a ResNet-50 model on ImageNet, remove the last fully connected layer, keep the convolutional part, freeze the first 47 layers, and train the last 3 layers using mini-batch training data.

[0119] The pre-trained ResNet-50 model is used to obtain the convolutional feature maps of the final image-label pairs. Global average pooling is performed on the convolutional feature maps to compress the spatial dimension into a single value, generating image feature vectors. Principal component analysis (PCA) is used to reduce the dimensionality of the image feature vectors.

[0120] The final image-label pair is divided into 2×2 sub-blocks and assigned to the four cells of the Tissue P system. Each cell processes the assigned sub-blocks in parallel based on edge detection rules and texture analysis rules. It should be noted that the image spatial structure is still preserved after PCA dimensionality reduction. The feature map is divided into 2×2 regions by two-dimensional slicing.

[0121] Edge detection rules refer to using the Sobel operator to calculate the gradient of sub-blocks and then performing average pooling on the gradient of the sub-blocks to generate edge feature vectors;

[0122] Texture analysis rules refer to using Local Binary Pattern (LBP) to extract sub-block texture features;

[0123] The rule weight is calculated based on the average grayscale value of the sub-blocks, using the following formula:

[0124]

[0125]

[0126] in, and These are the edge detection weights and texture analysis weights for the i-th sub-block, respectively, σ i and μ i These are the mean gray level (based on the RGB channel average) and standard deviation of the gray level of the i-th sub-block, respectively, reflecting the edge strength;

[0127] Based on edge detection weights and texture analysis weights, the edge feature vector and texture feature of each sub-block are weighted and summed to obtain the sub-block feature vector.

[0128] Each cell receives features from other cells and updates its own features based on cosine similarity, using the formula:

[0129]

[0130] in, Let i be the sub-block features after the i-th cell is updated. For the feature vector of the sub-block, Let α be the feature vector of the sub-block from the o-th cell. io For feature similarity weights, and Cosine similarity;

[0131] The average value of each updated sub-block feature is used as the global feature vector, and the global feature vector is then reduced in dimensionality using principal component analysis (PCA).

[0132] The image feature vector and the global feature vector after dimensionality reduction are concatenated to obtain the final image feature vector.

[0133] The preprocessed text data is segmented using BERT's Chinese word segmenter. "[CLS]" is added to the beginning of each sentence and "[SEP]" is added to the end of each sentence to generate a token sequence.

[0134] The labeled sequence is converted into the input format of the BERT model, and the feature vectors of all sentences are extracted using the pre-trained BERT model and combined into a text feature vector.

[0135] The preprocessed energy consumption data is divided into sub-matrices according to fixed time windows. The covariance matrix of each sub-matrix is ​​calculated, and principal component analysis (PCA) is used for dimensionality reduction. The statistical characteristics of the dimensionality-reduced covariance matrix are calculated, including mean, standard deviation, and trend slope. The covariance matrix is ​​fitted by linear regression, and all statistical characteristics are combined as the energy consumption feature vector.

[0136] By using inter-frame differencing to extract frames with significant changes, preliminary filtering of frames with potential alarm information was achieved. Through image region segmentation and parallel processing of each sub-block in the Tissue P system, local modeling of image spatial distribution features was realized. By employing a dual feature extraction strategy of edge and texture, combined with gray-scale mean adjustment of analysis weights, multi-dimensional modeling of image features was achieved. Through inter-cell feature interaction mechanisms, feature vector complementarity and enhancement were realized, avoiding isolated distortion in local analysis. By introducing a pre-trained BERT language model for deep semantic modeling of operation and maintenance text, the beneficial effect of providing high-quality text semantic support for joint image and text analysis was achieved. Through covariance analysis and principal component dimensionality reduction, energy consumption fluctuation patterns and trend features were extracted, achieving the beneficial effect of introducing energy consumption data as an auxiliary judgment basis in multimodal information fusion, enhancing the system's intelligent analysis capabilities.

[0137] S3. Construct an MRO model to perform multimodal data fusion, construct a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base;

[0138] In this process, the feature vectors of each modality are normalized and concatenated to form an initial fused high-dimensional feature vector.

[0139] Construct the MRO model and initialize its weights and biases using the Xavier initialization method;

[0140] Collect historical multimodal data with labels, including fault diagnosis labels and energy saving labels, extract features from each, and input them into the MRO model for training.

[0141] Perform MRO forward propagation on the training data and calculate the single-modal formula:

[0142]

[0143] in, For the single-modal interaction characteristics of mode l, and Z represents the weights and biases of the fully connected layer for mode l. l is the normalized eigenvector of mode l;

[0144] For each pair of modes, such as vibration and image, calculate the bimodal interaction features using the formula:

[0145]

[0146] Among them, F bi (i,j) represents the bimodal interaction features of modes l and l'. and Z represents the weights and biases of the fully connected layer for bimodal interaction, ⊙ represents element-wise multiplication used to capture intermodal interactions. l' Let be the normalized eigenvector of mode l';

[0147] For each group of three modes (e.g., vibration-image-energy consumption), calculate the three-mode interaction characteristics using the following formula:

[0148]

[0149] Among them, F tri (l,l',l”) represents the trimodal interaction feature. and These represent the weights and biases of the fully connected layer for trimodal interaction, Z. l” The normalized eigenvectors of mode l;

[0150] To minimize the cross-entropy loss function, the Adam optimizer is used to update the weights and biases. Training stops after reaching the maximum number of iterations, and the trained MRO model is output. The initial fused high-dimensional feature vector is then input to output the final single-modal model. Dual-modal Fbi (l,l')'、Three-modal feature F tri (l,l',l”)';

[0151] The final unimodal, bimodal, and trimodal features are fused using residual connections, as shown in the formula:

[0152]

[0153] Among them, F MRO This is the final fused feature vector;

[0154] By calculating the single-modal interaction feature vector corresponding to each modality, deep semantic extraction is achieved within the modality, which is equivalent to performing nonlinear mapping on a specific modal dimension. By performing element-wise multiplication and linear transformation on any two modalities, such as vibration and image, collaborative semantic extraction between modalities on a specific dimension is achieved. Through high-order combination operations and nonlinear mapping of the three modalities, high-order semantic interactions under the joint participation of the three modalities are captured, which integrates more dimensional information sources and helps to identify deep latent patterns. By fusing the final single-modal, bimodal, and trimodal features through residual connections, the features of different orders of modalities are directly added, which achieves fusion while maintaining the original information of each order of modality and avoids the information decay problem in deep network training.

[0155] Furthermore, the final fused feature vector is normalized.

[0156] Historical features from the most recent time point are extracted from the database of equipment operation data to form time series features, which are then normalized.

[0157] Build a CNN model, train it using historical multimodal data, and output the trained CNN model;

[0158] The final fused feature vector and normalized time series features are input into the trained CNN model for state prediction;

[0159] State prediction involves inputting normalized time-series features into a convolutional layer to generate time-series convolutional features, and then inputting the time-series convolutional features into a prediction fully connected layer after pooling. The output state prediction probability (healthy, sub-healthy, fault risk) is then generated.

[0160] If the probability of failure risk is greater than 50%, perform failure diagnosis and prediction; otherwise, continue monitoring.

[0161] Fault diagnosis and prediction includes inputting the final fused feature vector into the trained CNN model, extracting spatial features from the convolutional layer, reducing the dimensionality of the results extracted by the convolutional layer by the pooling layer, and inputting them into the fully connected layer. The fully connected layer uses Softmax to map the features to fault categories.

[0162] Collect historical fault data and expert operation and maintenance experience to construct an expert rule base. Each rule is defined as a condition-result pair, such as "If the vibration frequency is >100Hz and the temperature is >50℃, then the server is overheating".

[0163] The weights of the assignment rules are estimated using the KDE density estimation formula:

[0164]

[0165] Among them, qr p Let Acc be the weight of the p-th rule. p P represents the historical accuracy rate of the rules (based on historical data verification), and P represents the total number of rules.

[0166] The CNN model predictions are matched against a rule base using the following formula:

[0167]

[0168] Among them, C fault To assess the confidence level of the CNN model's prediction results, Match is defined as follows: If Y... fault With WR p If the fault labels match and the input features satisfy the condition combination, the match is successful; otherwise, the match fails. Success is represented by 1, failure by 0, and Y... fault WR is the prediction result of the CNN model. p This is the p-th rule;

[0169] Set a confidence threshold and compare it with the confidence level. If the confidence level is greater than the confidence threshold, an alarm is triggered and the final fault classification result and operation and maintenance suggestions (such as "replace the server heat sink") are output to the operation and maintenance personnel for confirmation and inspection.

[0170] Extracting convolutional energy consumption features F from CNN models energy Predict the state S trend And fault classification prediction Y fault And the energy consumption features of convolution are fused together, as shown in the formula:

[0171] F weighted =Concat(F energy ,β·C fault ·Y fault ,β'·S trend );

[0172] Among them, F weighted For weighted fusion features, β and β' are the weights for fault classification and state prediction, respectively, with β set based on the probability of fault based on state prediction;

[0173] The weighted fused features are input into the trained CNN model, and the probability value of energy consumption anomalies is output.

[0174] Based on state prediction and fault classification prediction, as well as an expert rule base, the average value method is used to set the energy consumption anomaly threshold for different states.

[0175] The probability value of energy consumption anomaly is compared with the energy consumption anomaly threshold predicted by the current state. If the energy consumption characteristic is greater than the energy consumption anomaly threshold, it is marked as high energy consumption anomaly, and optimization suggestions (such as "reduce cooling power by 10%) are generated. Otherwise, it is marked as normal.

[0176] By using fault risk probability-based decision-making branches, the system's response efficiency is improved, resource waste is reduced, and a risk-aware prediction-diagnosis integrated process is constructed to enhance the timeliness and effectiveness of fault response. Spatial features are extracted and classified through the fault diagnosis prediction module to improve diagnostic granularity and interpretability, providing a basis for subsequent operation and maintenance decisions and shortening fault investigation time. By integrating state prediction, fault classification, and convolutional energy consumption features, and by constructing an expert rule base and assigning weights, the system's fault tolerance and credibility interpretation capabilities are improved. This is particularly suitable for auxiliary decision-making in scenarios with insufficient data samples or critical judgments, enabling comprehensive analysis of equipment operating efficiency and energy consumption, providing a basis for energy-saving optimization strategies, and improving the system's intelligent management level.

[0177] S4. Build a visual interface to display the analyzed data and store the collected and analyzed data;

[0178] Among them, a visual interface is built to display fault diagnosis results, status prediction trends and fault areas (by extracting key frames from video data);

[0179] Use React to create fault diagnosis bar charts, energy consumption trend line charts, and video players;

[0180] Users can click on the fault diagnosis bar chart to view detailed diagnostic reports, drag the timeline to view historical energy consumption and predicted trends, and support video playback and fault area highlighting linkage.

[0181] By using React to create fault diagnosis bar charts, energy consumption trend line charts, and video players, multi-dimensional linkage and intuitive display of fault diagnosis, status prediction, and video playback are achieved. Users can click on the fault diagnosis bar chart to view detailed diagnostic reports, drag the timeline to view historical energy consumption and predicted trends, and support linkage between video playback and fault area highlighting, which improves the interactivity and efficiency of the user experience. It can also provide rich fault diagnosis, energy consumption analysis, and prediction functions at different levels.

[0182] Furthermore, the collected and analyzed multimodal data is stored in a database, and secure access measures are set up. The database backs up the stored data to the cloud and performs integrity checks on the stored data and backup data regularly. After the checks are completed, integrity check records are generated and stored synchronously in the database.

[0183] By uniformly storing the collected and analyzed multimodal data (such as images and text) in the database, centralized management and long-term preservation of heterogeneous data are achieved. By setting up secure access measures (such as access control, identity authentication, and data encryption), data access management and security are achieved. The database supports cloud backup of the stored data, ensuring that data can still be quickly restored through cloud backup in the event of local device or system failure, attack, or natural disaster, thereby enhancing the system's disaster recovery capability and high availability, and significantly improving the system's reliable operation level.

[0184] This embodiment also provides an intelligent operation and maintenance management system based on multimodal AI, including:

[0185] The data collection and preprocessing module is used to collect and preprocess multimodal data, including vibration signals, energy consumption, image and text data.

[0186] The feature extraction module is used to perform modal component decomposition on the vibration signal using improved VMD, calculate the first-order energy moment and the second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector.

[0187] The model building module is used to extract image feature vectors using a pre-trained ResNet-50 model and the Tissue P system, extract text feature vectors using a pre-trained BERT model, and calculate the energy consumption feature vector.

[0188] The fusion generation module is used to build an MRO model for multimodal data fusion, build a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base.

[0189] The visualization and storage module is used to build a visual interface to display the analyzed data and to store the collected and analyzed data.

[0190] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0191] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. An intelligent operation and maintenance management method based on multimodal AI, characterized in that, include: Multimodal data is collected and preprocessed. Modal component decomposition of vibration signals is performed using improved VMD. First-order energy moments and second-order energy moments are calculated to generate initial vibration feature vectors. The final vibration feature vector is obtained using a denoising autoencoder SDAE. The multimodal data includes vibration signals, energy consumption, image data, and text data. Image feature vectors are extracted using a pre-trained ResNet-50 model and the Tissue P system, and text feature vectors are extracted using a pre-trained BERT model. Energy consumption feature vectors are calculated. The process involves using a pre-trained ResNet-50 model and the Tissue P system to extract image feature vectors. This includes acquiring alarm logs and device operation videos, using inter-frame difference to detect significantly changing frames in the video, converting them to RGB format, merging them with image data into a unified image set, calculating the sharpness of images in the unified image set based on Laplacian variance, setting a sharpness threshold, retaining images in the unified image set with a sharpness greater than the sharpness threshold, and matching and labeling images with alarms through time synchronization. The labeled image-label pairs are submitted to the operations and maintenance personnel for inspection and correction. The final image-label pairs are output, and mini-batch training data is extracted from them for fine-tuning the ResNet-50 model. The pre-trained ResNet-50 model is fine-tuned to obtain the convolutional feature maps of the final image-label pairs. Global average pooling is performed on the convolutional feature maps to compress the spatial dimension into a single value, generating image feature vectors. Principal component analysis (PCA) is used to reduce the dimensionality of the image feature vectors, dividing them into 2×2 sub-blocks, which are then assigned to the four cells of the Tissue P system. The gradient of the sub-block is calculated using the Sobel operator in each cell, and average pooling is performed on the gradient of the sub-block to generate edge feature vectors. Local binary mode (LBP) is used to extract the texture features of the sub-blocks. The rule weights are calculated based on the gray-scale mean of the sub-blocks. The edge feature vectors and texture features of each sub-block are weighted and summed to form the sub-block feature vector. Each cell receives features from other cells, updates its own features based on cosine similarity, calculates the average of the updated sub-block features as the global feature vector, and performs dimensionality reduction on the global feature vector using principal component analysis (PCA). The image feature vector and the global feature vector after dimensionality reduction are concatenated to obtain the final image feature vector. A multimodal data fusion model is constructed to perform MRO model fusion, and a CNN model is constructed to predict the state and fault category of the fused multimodal data. Optimization suggestions are generated based on the expert rule base. Build a visual interface to display the analyzed data and store the collected and analyzed data.

2. The intelligent operation and maintenance management method based on multimodal AI according to claim 1, characterized in that, The process of using improved VMD to perform modal component decomposition on the vibration signal and using a denoising autoencoder SDAE to obtain the final vibration feature vector includes: The preprocessing includes denoising, normalization, and timestamp alignment of the multimodal data; The spectrum of the preprocessed vibration signal data is calculated using Discrete Fourier Transform (DFT). The spectral amplitude is calculated, and a peak threshold is set based on historical regression estimation. The number of frequency points with spectral amplitude greater than the peak threshold is selected using a fixed threshold screening method as the number of modal components. Define the bandwidth penalty term and fidelity term of VMD, as well as the VMD optimization objective function. Solve iteratively using the Alternating Direction Multiplier Method (ADMM). Update each modal component in the frequency domain, update the center frequency, calculate the total energy of the modal components in the current iteration number, and calculate the modal change ratio using the ratio method. Update the regularization parameters, calculate the adaptive step size, update the Lagrange multipliers, set the maximum number of iterations, stop the iteration when the maximum number of iterations is reached, and output the final set of modal components. The final set of modal components is converted to a time-domain representation by inverse FFT, and the first-order energy moment and second-order energy moment are calculated and normalized to form an initial vibration feature vector. The initial feature vector is divided into a training set and a test set. The signal-to-noise ratio (SNR) of the preprocessed vibration signal is calculated based on the Euclidean norm. Gaussian noise is dynamically added to the training set according to the SNR to generate noisy features. A three-layer denoising autoencoder SDAE is constructed. The training set is divided into batches and input into the denoising autoencoder SDAE. With the goal of minimizing the reconstruction error, the Adam optimizer is used for updating and training. During the training process, the noise standard deviation is dynamically updated according to the SNR change, and the trained denoising autoencoder SDAE is output. The test set is input into the trained denoising autoencoder SDAE, and the output of the intermediate layer is extracted as the final vibration feature vector.

3. The intelligent operation and maintenance management method based on multimodal AI according to claim 2, characterized in that, The process of extracting text feature vectors using a pre-trained BERT model and calculating energy consumption feature vectors includes: The pre-trained BERT model is used to extract the feature vectors of all sentences and combine them into a text feature vector; The preprocessed energy consumption data is divided into sub-matrices. The covariance matrix of each time window sub-matrix is ​​calculated. Principal component analysis (PCA) is used for dimensionality reduction. The statistical characteristics of the dimensionality-reduced covariance matrix are calculated, and all statistical characteristics are combined as the energy consumption feature vector.

4. The intelligent operation and maintenance management method based on multimodal AI according to claim 3, characterized in that, The construction of the MRO model for multimodal data fusion includes: The feature vectors of each modality are normalized and concatenated to form an initial fused high-dimensional feature vector. Construct the MRO model and initialize its weights and biases using the Xavier initialization method; Collect labeled historical multimodal data, extract features from each, and input them into the MRO model for training. Perform MRO forward propagation on the training data to calculate unimodal, bimodal, and trimodal interaction features. With the goal of minimizing the cross-entropy loss function, use the Adam optimizer to update the weights and biases. Stop training when the maximum number of training iterations is reached, and output the trained MRO model. Input the initially fused high-dimensional feature vector to output the final unimodal, bimodal, and trimodal features. The final unimodal, bimodal, and trimodal features are fused through residual connections to obtain the final fused feature vector.

5. The intelligent operation and maintenance management method based on multimodal AI according to claim 4, characterized in that, The constructed CNN model performs state prediction and fault category prediction on the fused multimodal data, and generates optimization suggestions based on an expert rule base, including: The final fused feature vector is then normalized. Historical features from the most recent time point are extracted from the database of equipment operation data to form time series features, which are then normalized. Build a CNN model, train it using historical multimodal data, and output the trained CNN model; The final fused feature vector and normalized time series features are input into the trained CNN model for state prediction; If the probability of failure risk is greater than 50%, perform failure diagnosis and prediction; otherwise, continue monitoring. Collect historical fault data and expert operation and maintenance experience to construct an expert rule base. Each rule is defined as a condition-outcome pair. KDE density estimation is used to assign rule weights. The prediction results of the CNN model are matched with the rule base. A confidence threshold is set. If the confidence is greater than the confidence threshold, an alarm is triggered and the final fault classification result and operation and maintenance suggestions are output to the operation and maintenance personnel for confirmation and inspection. Extract convolutional energy consumption features from the CNN model, fuse state prediction, fault classification prediction, and convolutional energy consumption features, input them into the trained CNN model, and output the probability value of abnormal energy consumption. The average value method is used to set energy consumption anomaly thresholds for different states. The probability value of energy consumption anomaly is compared with the energy consumption anomaly threshold predicted for the current state. If the energy consumption characteristic is greater than the energy consumption anomaly threshold, it is marked as high energy consumption anomaly and optimization suggestions are generated; otherwise, it is marked as normal.

6. The intelligent operation and maintenance management method based on multimodal AI according to claim 5, characterized in that, The data analysis and visualization interface refers to the construction of a visualization interface to display fault diagnosis results, status prediction trends, and fault areas. Use React to create fault diagnosis bar charts, energy consumption trend line charts, and video players; Users can click on the fault diagnosis bar chart to view detailed diagnostic reports, drag the timeline to view historical energy consumption and predicted trends, and support video playback and fault area highlighting linkage.

7. The intelligent operation and maintenance management method based on multimodal AI according to claim 5, characterized in that, The storage of the collected and analyzed data refers to storing the collected and analyzed multimodal data in a database, setting up security access measures, backing up the stored data to the cloud, and periodically performing integrity checks on the stored data and backup data. After the checks are completed, integrity check records are generated and synchronously stored in the database.

8. A multimodal AI-based intelligent operation and maintenance management system, based on the multimodal AI-based intelligent operation and maintenance management method according to any one of claims 1 to 7, characterized in that, include: A collection and preprocessing module is used to collect and preprocess multimodal data, including vibration signals, energy consumption, image data, and text data. The feature extraction module is used to perform modal component decomposition on the vibration signal using improved VMD, calculate the first-order energy moment and the second-order energy moment to generate the initial vibration feature vector, and use the denoising autoencoder SDAE to obtain the final vibration feature vector. The model building module is used to extract image feature vectors using a pre-trained ResNet-50 model and the Tissue P system, extract text feature vectors using a pre-trained BERT model, and calculate the energy consumption feature vector. The fusion generation module is used to build an MRO model for multimodal data fusion, build a CNN model to predict the state and fault category of the fused multimodal data, and generate optimization suggestions based on the expert rule base. The visualization and storage module is used to build a visual interface to display the analyzed data and to store the collected and analyzed data.

Images