Thermal power plant equipment defect intelligent monitoring and early warning method and system based on multi-modal large model

Abstract

The invention relates to a thermal power plant equipment defect intelligent monitoring and early warning method based on a multi-modal large model. The method comprises the following steps: collecting operation data, namely multi-modal data, of equipment in real time; the method comprises the following steps of: constructing a multi-mode Transform model; carrying out cross-modal data fusion and analysis; calculating the abnormal degree of the equipment; future equipment state prediction is carried out, and the probability of potential fault occurrence is predicted; and carrying out optimization on the multi-mode Transform model. The invention further discloses a thermal power plant equipment defect intelligent monitoring and early warning system based on the multi-mode large model. According to the method, the multi-modal Transform model is adopted, cross-modal feature fusion is carried out in combination with the image, sound, sensor and text data, the accuracy is higher, and the false alarm rate and the missing report rate are lower; trend analysis is carried out on the equipment state, the fault occurrence time can be predicted, the prediction advance is greatly improved, and the probability of sudden faults is reduced through combination of anomaly detection and prediction. In different thermal power plants and different devices, the migration adaptability is high.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent monitoring and early warning of power equipment, and in particular to a method and system for intelligent monitoring and early warning of defects in thermal power plant equipment based on a multimodal large model. Background Art

[0002] Thermal power plant equipment monitoring involves collecting and analyzing real-time operating data from key equipment, such as transformers, energy meters, and sensors, to assess equipment status, detect anomalies, and predict potential failures. Its goal is to improve equipment reliability, reduce downtime, and optimize operational efficiency. Current thermal power plant equipment monitoring methods have the following main issues:

[0003] First, traditional thermal power plant equipment monitoring methods primarily rely on single data modalities, such as sensor data from temperature, vibration, and current, and lack the ability to integrate multimodal information such as images, sound, and text. This results in incomplete monitoring information and affects the accuracy of fault identification.

[0004] Second, traditional shallow machine learning or rule-based expert systems are unable to effectively cope with the complex operating conditions of thermal power plants, especially hidden faults, nonlinear failure modes, and equipment aging.

[0005] Third, existing monitoring methods can only detect problems after they occur and are unable to provide sufficient advance warning.

[0006] Fourth, there are significant differences in equipment types, environments, and working conditions across thermal power plants. Traditional fault monitoring methods require a lot of manual adjustments and annotations when migrating to different equipment or new environments. They also require a large number of professional engineers to conduct data analysis and equipment inspections, which is time-consuming and labor-intensive. Furthermore, the methods are subject to strong subjective factors, making them difficult to generalize and perform adaptive optimization. Summary of the Invention

[0007] In order to solve the problems of single-modal monitoring limitations, poor adaptability and low monitoring accuracy in existing thermal power plant equipment monitoring methods, the primary purpose of the present invention is to provide a thermal power plant equipment defect intelligent monitoring and early warning method based on a multi-modal large model with strong migration adaptability, which improves monitoring accuracy, reduces false alarm and missed alarm rates, and reduces the probability of sudden failures by performing cross-modal feature fusion.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model, the method comprising the following steps in sequence:

[0009] (1) Collecting the operating data of the equipment, i.e., multimodal data, in real time. The multimodal data includes image data, sound data, sensor data, and text data, and transmitting the multimodal data to the multimodal Transformer model;

[0010] (2) Constructing a multimodal Transformer model: The BEiT-3 model, Whisper model, Informer model, T5 model, Flamingo model, and autoencoder together form a multimodal Transformer model to encode different types of multimodal data and generate a unified multimodal representation;

[0011] (3) Perform cross-modal data fusion and analysis on the unified multimodal representation to obtain unified fusion features;

[0012] (4) Based on the unified fusion features, contrastive learning and autoencoder are used to calculate the abnormality of the device to determine whether the device is abnormal;

[0013] (5) Based on the unified fusion features, predict the future status of equipment, predict the probability of potential failures, and provide operation and maintenance suggestions and early warnings;

[0014] (6) Optimize the multimodal Transformer model through incremental learning and knowledge distillation.

[0015] Step (1) specifically includes the following steps:

[0016] (1a) Image acquisition: A high-definition industrial camera installed in the visible area of the electrical equipment is used to capture the appearance image of the equipment and detect physical defects; the obtained image is subjected to Gaussian filtering to remove noise and avoid the influence of noise. i Apply Gaussian filtering to denoise the pixel at position (x,y) in the image collected at that moment:

[0017]

[0018] Where σ is the standard deviation of the Gaussian kernel, I filtered (x,y,t i ) is t i The result of denoising the pixel at the (x, y) position at the moment after Gaussian filtering; I(u, v, t i ) represents t i The pixel value of the image collected at the moment (u, v);

[0019] (1b) Sound collection: High-sensitivity microphones placed in noise-sensitive areas of electrical equipment are used to collect operating sounds, identify abnormal noises, and j The sound signal S(f,tj ) Apply spectral subtraction to reduce noise:

[0020] S clean (f,t j )=max(S(f,t j )-α f N(f),0)

[0021] Where N(f) is the noise estimate; α f is the noise reduction coefficient, which is 1.5; S clean (f,t j ) is t j The sound signal after spectral subtraction noise reduction at all times;

[0022] The noise estimate N(f) is obtained using the minimum statistics method:

[0023]

[0024] Among them, W t is the time window; β is the correction factor, which is 1.2;

[0025] (1c) Sensor data acquisition: Electrical operating parameters are collected through current sensors, voltage sensors, temperature sensors, and vibration sensors to monitor the equipment status. Low-pass filtering is applied to the sensor data X(t) collected at time t:

[0026]

[0027] Where X(τ) represents the raw sensor data collected at time τ; h(t) represents the impulse response of the filter. The calculation of h(t) is as follows:

[0028]

[0029] Among them, ω c =2πf c , f c is the cutoff frequency; u(t) is the unit step function, when t≥0, u(t) is 1, otherwise, u(t) is 0;

[0030] (1d) Text data collection: Through operation logs, electricity meter readings, and maintenance records, historical data and context information are provided, and the text is segmented and formatted in a standardized manner;

[0031] (1e) Data synchronization and transmission: Synchronous real-time transmission, data alignment by time stamp, to obtain synchronized multimodal data. The synchronization formula is:

[0032] D sync (t)={I filtered (ti ),S clean (t j ),X filtered (t k ),D text (t m )}

[0033] Among them, D sync (t) represents the synchronized multimodal data at time t; I filtered (t i ) represents t i The filtered image data at the moment, S clean (t j ) represents t j The sound data after noise reduction at all times, X filtered (t k ) represents t k Apply low-pass filtering to the sensor data at all times, D text (t m )=Tokenize(T(t m )) means t m The text data after word segmentation at time T(t m ) represents t m The original text data at the moment; Tokenize represents the word segmentation operation;

[0034] |t i -t j |<∈,|t i -t k |<∈,|t i -t m |<∈, ∈ is the time deviation.

[0035] Step (2) specifically includes the following steps:

[0036] (2a) Using the BEiT-3 model for image coding:

[0037] v I =BEiT(x I )

[0038] in, is the output image feature vector, representing the visual features of the device appearance and thermal imaging, d I is 1024; x I is the input image, Where H and W are both 224;

[0039] (2b) Encode the sound signal using the Whisper model encoder:

[0040] vA =Whisper(x A )

[0041] in, is the output sound feature vector, reflecting the abnormal noise pattern during equipment operation, d A is 512; x A is the input sound spectrum;

[0042] (2c) Encode the sensor data using the encoder of the Informer model:

[0043] v S =Informer(x S )

[0044] in, The sensor feature vector representing the output captures the dynamic changes of the device state, d S is 512; x S is the input sensor time series;

[0045] (2d) Use T5 model for text encoding:

[0046] v T =T5(x T )

[0047] in, Represents the output text feature vector, which contains the semantic information of operation and maintenance logs and historical faults; d T is 1024; x T Represents the input text sequence;

[0048] (2e) The linear projection layer is used to map the feature vectors of each modality to a unified dimension D of 1024 for the representation vectors of each modality for subsequent fusion:

[0049] m i =W i ·v i +b i

[0050] Where m i represents the unified representation of the i-th modality, i∈{I,A,S,T}; represents the projection matrix, d i represents the dimension of the i-th modal eigenvector; represents the bias vector, v i represents the eigenvector of the i-th mode;

[0051] The final output is a unified multimodal representation vector set V:

[0052] V=[mI ,m A , m S ,m T ]

[0053] Where m I The representation vector representing the image modality; m A The representation vector representing the sound mode; m S The representation vector representing the sensor modality; m T A representation vector representing the text modality.

[0054] Step (3) specifically refers to: using Flamingo to achieve extended modal fusion and obtain a unified fusion feature v multimodal :

[0055] v multimodal =Flamingo(m I ,m A ,m S ,m T )

[0056] Where, v multimodal is a unified fusion feature, m I The representation vector representing the image modality; m A The representation vector representing the sound mode; m S The representation vector representing the sensor modality; m T A representation vector representing the text modality.

[0057] Step (4) specifically includes the following steps:

[0058] (4a) Unified fusion feature v of the input multimodal Normalize and get the normalized feature v norm :

[0059]

[0060] Among them, μ v Represents all v in the training set multimodal The mean of v Represents all v in the training set multimodal The standard deviation of represents the normalized features, and D is the unified dimension of each modality representation vector;

[0061] (4b) Contrastive learning detection: A large number of feature vector sets {v normal,1 ,v normal,2 ,…,v normal,M}, M is the number of normal samples; the negative sample library collects a set of known abnormal samples {v abnormal，1 ,…,vabnormal,N}, N is the number of abnormal samples;

[0062] Use cosine similarity to measure the normalized feature v norm The degree of closeness to the normal sample sin(v norm ,v normal,i ):

[0063]

[0064] Among them, v normal,i Represents the characteristics of the i-th normal sample;

[0065] Defining the loss function for contrastive learning Encourage norm Closer to normal samples and further away from abnormal samples:

[0066]

[0067] Among them, v j represents the characteristics of the jth sample in the total set of normal samples and abnormal samples; τ represents the temperature coefficient, which is 0.1;

[0068] Finally calculate v norm The average similarity S with normal samples contrastive :

[0069]

[0070] Among them, S contrastive A value close to 1 indicates that the current device status is normal, and a value close to 0 indicates an abnormality.

[0071] (4c) Autoencoder detection: The autoencoder consists of an encoder and a decoder. The encoder is used to convert v norm Compress to a low-dimensional latent space:

[0072] z=f enc (v norm )=W enc ReLU(W in v norm +b in )+b enc

[0073] in, d z <D,d z Take 256, D represents the unified dimension of each modality representation vector, which is 1024; W enc 、W in Both represent weight matrices; b in 、b encAll represent bias; ReLU represents activation function; z represents the low-dimensional feature vector after being encoded by the encoder;

[0074] The decoder reconstructs z back to the original dimension:

[0075]

[0076] Among them, W dec 、W hidden Both represent weight matrices; b hidden 、b dec Both represent bias; Represents the reconstructed features; f dec (z) represents decoding the low-dimensional feature vector z;

[0077] Use normal data to train the autoencoder and define a reconstruction loss Minimize the reconstruction error of normal data:

[0078]

[0079] After training is completed, the reconstruction error S is calculated for the test data recon :

[0080]

[0081] S recon When it is larger, it means the current device status is abnormal;

[0082] S recon Normalize by the statistics of the training set:

[0083]

[0084] Among them, μ recon Represents all S in the training set recon The average value of σ recon Represents all S in the training set recon The standard deviation of S' recon represents the normalized reconstruction error;

[0085] (4d) Fusion of the results of contrastive learning and autoencoder to generate the final anomaly score S anomaly :

[0086] S anomaly =α(1-S contrastive )+(1-α)S' recon

[0087] Among them, α∈[0,1] represents the weight parameter, which is 0.5; 1-S contrastive Indicates converting similarity into abnormality;

[0088] Set a threshold T ano , when S anomaly >T ano When , it is judged as abnormal;

[0089] If an anomaly is detected, the fault type needs to be further classified, which is achieved through a fully connected neural network:

[0090] P(fault)=softmax(W c ·v norm +b c )

[0091] in, represent weights and biases respectively, K is the number of fault categories; P(fault) represents the probability distribution of each type of fault; softmax represents the activation function;

[0092] During training, labeled fault data is needed to optimize the cross entropy loss:

[0093]

[0094] Among them, y k represents the true label, P k represents the kth item in the probability distribution of the model output, Represents the cross entropy loss, which is used to train the model.

[0095] Step (5) specifically includes the following steps:

[0096] (5a) For the unified fusion feature v multimodal Perform time series construction and normalization processing, and extract the feature vector V of T consecutive time steps from historical data t =[v multimodal,t-T+1 ,v multimodal,t-T+2 ,…,v multimodal,t ],in D m is the feature dimension, which is the same as the dimension D of the unified representation vector of each modality, that is, 1024; then each dimension is normalized:

[0097]

[0098] Among them, μ t , σ t Represents the time series mean and standard deviation of the training set; V' t represents the normalized time series;

[0099] (5b) The informer model is used to predict the future state characteristics of the device. The placeholder for the future time step is input, first passed through the Transformer encoding layer of the informer model decoder, and then input into the multi-head self-attention layer together with the output of the informer model encoder to output the predicted value:

[0100]

[0101] in, is the predicted value for the next step;

[0102] During training, it is necessary to minimize the mean square error between the predicted value and the true future state:

[0103]

[0104] Where, v multimodal,t+1 Indicates the real state of the next step, that is, the real future state, Indicates the predicted value for the next step; Represents the mean square error between the predicted value and the true future state;

[0105] (5c) Based on the prediction results Deviation analysis and failure probability assessment can provide early warning:

[0106] (5c1) Deviation analysis: Define a normal state benchmark and use the average eigenvector v calculated using normal operating data normal_benchmark As a normal state benchmark:

[0107]

[0108] Where M represents the number of normal samples; Represents the feature vector of the i-th normal sample;

[0109] Then calculate the predicted state Compared with the normal state benchmark v normal_benchmark The Euclidean distance of deviation :

[0110]

[0111] in, Indicates the predicted state; D deviation Used to measure the degree of deviation between the predicted state and the normal state benchmark;

[0112] Finally, set a deviation threshold T dev When D deviation >T dev When it exists, trigger an early warning;

[0113] (5c2) Failure probability assessment:

[0114] First build the failure mode library {v fault,1 ,…,v fault,K}, K is the number of fault categories, and then the cosine similarity is used to evaluate the predicted state The degree of proximity to each failure mode S k :

[0115]

[0116] Among them, v fault,k Denotes the kth failure mode and normalizes the result to probability:

[0117]

[0118] Among them, τ p Indicates the temperature coefficient, which is 0.1; P(fault k ) represents the probability of occurrence of the kth failure mode;

[0119] when When T prob Represents a threshold, T prob Take 0.7; Indicates the failure mode with the highest probability of occurrence; It means that as long as the probability of the failure mode occurring exceeds the threshold T prob , then the fault with the highest probability of occurrence is judged to be a high-risk fault type;

[0120] use Indicates the number of the failure mode with the highest probability of occurrence; The corresponding highest probability category provides corresponding suggestions; Indicates that from P(fault k ) to obtain the column number with the highest value, P(fault k ) corresponds to the probability of a fault category occurring, that is, the corresponding highest probability category.

[0121] Step (6) specifically includes the following steps:

[0122] (6a) New data X new After multimodal encoding and multimodal fusion, the fused feature vector v is obtained multimodal,new :

[0123] v multimodal,new

[0124] =Flamingo(BEiT(x I,new ),Whisper(x A,new),Informer(x S,new ),T5(x T,new ))

[0125] Where x I，new Represents the image data in the new data; x A,new Represents the sound signal in the new data; x S,new Represents the sensor data in the new data; x T,new Represents the text data in the new data; v multimodal,new Represents the fused feature vector of the new data;

[0126] Afterwards, v multimodal,new Normalize:

[0127]

[0128] Among them, μ and σ are the mean and standard deviation calculated based on historical data respectively; v norm,new represents the normalized fusion feature vector;

[0129] In trend forecasting, new data is organized into a time series V with a window length of T. new,t =[v multimodal,new,t-T+1 ,…,v multimodal,new,t ];

[0130] (6b) Adopting incremental learning mechanism to gradually learn new data, the main steps are as follows:

[0131] (6b1) Data buffering and selection:

[0132] First, you need to maintain a fixed-size buffer pool Capacity 1000, storage of historical data samples {(v norm,i ,y i )} and new data {(v norm,new ,y new )},y i with y new Both represent status labels;

[0133] Then use random replacement to update the buffer pool to ensure that the new and old data are balanced:

[0134]

[0135] in, Indicates the number of samples in the buffer pool; size(v new ) represents the number of samples of newly added new data; P(keep) represents the probability of each old sample in the buffer pool being retained;

[0136] (6b2) Model update:

[0137] Fine-tune the model on the new data while retaining the performance on the old data; fine-tune the model using the data in the buffer pool. For the device anomaly detection module, the loss function is:

[0138]

[0139] Where, τ c represents the temperature coefficient, which is 0.1; v normal represents a normal sample; v j represents the jth sample in the buffer pool;

[0140] For the equipment status trend prediction module, the loss function is:

[0141]

[0142] Where, Indicates the predicted value of the next step corresponding to the new data, that is, the predicted state; v multimodal,new，t+1 Indicates the actual state of the next step corresponding to the new data, that is, the actual future state;

[0143] At the same time, the model parameters are updated through mini-batch gradient descent;

[0144] (6c) Knowledge Distillation Optimization

[0145] First, get the output of the teacher model for the new data:

[0146]

[0147] Where, represents the teacher model; v norm,new,i Normalized features of the i-th new data; represents the output of the teacher model;

[0148] Then get the output of the student model for the new data:

[0149]

[0150] Where, represents the student model; represents the output of the student model;

[0151] Finally calculate the KL divergence loss

[0152]

[0153] Among them, D KL represents KL divergence;

[0154] Calculate total loss

[0155]

[0156] Among them, θ represents the balance factor, which is 0.7; represents the task-specific loss, which is formulated as:

[0157]

[0158] Where, To detect loss, To predict losses;

[0159] By optimizing That is, the student model is updated; then the updated student model replaces the old parameters of each module in the system; an update is triggered every time a certain amount of new data is received; if the performance after the update degrades by more than a certain proportion, the old model is rolled back.

[0160] Another object of the present invention is to provide an intelligent monitoring and early warning system for thermal power plant equipment defects based on a multimodal large model, comprising:

[0161] The multimodal data acquisition module collects the equipment's operating data, i.e., multimodal data, in real time and transmits the multimodal data to the multimodal Transformer model;

[0162] The multimodal data encoding module uses the BEiT-3 model, Whisper model, Informer model, and T5 model to encode different types of multimodal data and generate a unified multimodal representation;

[0163] The multimodal fusion analysis module uses the Flamingo model to perform cross-modal data fusion and analysis on the unified multimodal representation to obtain unified fusion features;

[0164] The device anomaly detection and fault diagnosis module uses an autoencoder to calculate the device's anomaly degree based on unified fusion features, using contrastive learning and autoencoders to determine whether the device is abnormal.

[0165] The equipment status trend prediction and early warning module uses the Informer model to predict future equipment status based on unified fusion features, predict the probability of potential failures, and provide operation and maintenance recommendations and early warnings;

[0166] The online learning and module optimization module optimizes the multimodal Transformer model through incremental learning and knowledge distillation.

[0167] It can be seen from the above technical solutions that the beneficial effects of the present invention are: First, the present invention adopts a multimodal Transformer model, combines image, sound, sensor and text data, and uses Flamingo to perform cross-modal feature fusion. Compared with traditional methods, it has higher accuracy and lower false alarm and omission rates; Second, the present invention adopts an Informer time series prediction model to perform trend analysis on equipment status, which can predict the time of failure, greatly improve the advance prediction, and provide operation and maintenance personnel with more sufficient maintenance time. By combining anomaly detection and prediction, the probability of sudden failure is reduced; Third, the present invention automatically learns the operating characteristics of different thermal power plants and different equipment through an online learning mechanism. When migrating to new equipment, only a small amount of new data is needed for fine-tuning to achieve accurate monitoring; at the same time, knowledge distillation is used to achieve incremental updates of the model. New, so that it can continuously adapt to new equipment operation modes without affecting the original performance; Fourth, the present invention adopts the T5 text analysis model, combined with historical operation and maintenance logs to automatically generate fault cause descriptions and provide maintenance suggestions; through BEiT image analysis + Whisper sound diagnosis, it can generate visual and audio playback fault diagnosis reports to help engineers quickly locate the source of the fault; Fifth, the present invention uses intelligent automated monitoring to reduce manual inspection time, provide automatic alarms and remote diagnosis, improve fault handling efficiency, reduce unplanned downtime, increase equipment availability, and reduce the average annual maintenance cost of thermal power plants; Sixth, the present invention adopts the pre-training-fine-tuning mechanism of the multimodal Transformer model, which has strong migration adaptability between different thermal power plants and different equipment, and reduces training costs. It can be quickly deployed to different factories, reducing deployment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0168] Figure 1 is a flow chart of the method of the present invention;

[0169] Figure 2 It is a system framework structure diagram of the present invention;

[0170] Figure 3 This is a processing flow chart of the multimodal data encoding module in the present invention;

[0171] Figure 4 This is the architecture diagram of the BEiT model in the present invention;

[0172] Figure 5 This is a structural diagram of the Whisper model encoder in the present invention;

[0173] Figure 6 This is a structural diagram of the encoder of the Informer model in the present invention;

[0174] Figure 7 This is a structural diagram of the decoder of the Informer model in the present invention. DETAILED DESCRIPTION

[0175] like Figure 1 As shown, a method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model includes the following steps in sequence:

[0176] (1) Collecting the operating data of the equipment, i.e., multimodal data, in real time. The multimodal data includes image data, sound data, sensor data, and text data, and transmitting the multimodal data to the multimodal Transformer model;

[0177] (2) Constructing a multimodal Transformer model: The BEiT-3 model, Whisper model, Informer model, T5 model, Flamingo model, and autoencoder together form a multimodal Transformer model to encode different types of multimodal data and generate a unified multimodal representation;

[0178] (3) Perform cross-modal data fusion and analysis on the unified multimodal representation to obtain unified fusion features;

[0179] (4) Based on the unified fusion features, contrastive learning and autoencoder are used to calculate the abnormality of the device to determine whether the device is abnormal;

[0180] (5) Based on the unified fusion features, predict the future status of equipment, predict the probability of potential failures, and provide operation and maintenance suggestions and early warnings;

[0181] (6) Optimize the multimodal Transformer model through incremental learning and knowledge distillation.

[0182] Step (1) specifically includes the following steps:

[0183] (1a) Image acquisition: A high-definition industrial camera installed in the visible area of the electrical equipment is used to capture the appearance image of the equipment and detect physical defects; the obtained image is subjected to Gaussian filtering to remove noise and avoid the influence of noise. i Apply Gaussian filtering to denoise the pixel at position (x,y) in the image collected at that moment:

[0184]

[0185] Where σ is the standard deviation of the Gaussian kernel, I filtered (x,y,t i ) is t i The result of denoising the pixel at the (x, y) position at the moment after Gaussian filtering; I(u, v, t i ) represents t iThe pixel value of the image collected at the moment (u, v);

[0186] (1b) Sound collection: High-sensitivity microphones placed in noise-sensitive areas of electrical equipment are used to collect operating sounds, identify abnormal noises, and j The sound signal S(f,t j ) Apply spectral subtraction to reduce noise:

[0187] S clean (f,t j )=max(S(f,t j )-α f N(f),0)

[0188] Where N(f) is the noise estimate; α f is the noise reduction coefficient, which is 1.5; S clean (f,t j ) is t j The sound signal after spectral subtraction noise reduction at all times;

[0189] In thermal power plant electrical equipment monitoring, considering that noise (such as discharge sound or environmental interference) may change dynamically over time, the noise estimation is obtained using the minimum statistical method. The principle is based on the statistical characteristics of the signal power spectrum. It assumes that the noise power is locally minimum in a short period of time. The method tracks the minimum value of the spectrum within a time window and applies bias correction to estimate the noise:

[0190]

[0191] Among them, W t is the time window; β is the correction factor, which is 1.2;

[0192] (1c) Sensor data acquisition: Electrical operating parameters are collected through current sensors, voltage sensors, temperature sensors, and vibration sensors to monitor the equipment status. Low-pass filtering is applied to the sensor data X(t) collected at time t:

[0193]

[0194] Where X(τ) represents the raw sensor data collected at time τ; h(t) represents the impulse response of the filter. The calculation of h(t) is as follows:

[0195]

[0196] Among them, ω c =2πf c , f cis the cutoff frequency; u(t) is the unit step function, when t≥0, u(t) is 1, otherwise, u(t) is 0;

[0197] (1d) Text data collection: Operation logs, electricity meter readings, and maintenance records provide historical data and contextual information, and the text is segmented and standardized. Segmentation is the process of breaking continuous text into meaningful words or phrases to facilitate subsequent analysis. Standardization mainly involves unifying the format to ensure data consistency, such as unit conversion and time format unification.

[0198] (1e) Data synchronization and transmission: Synchronous real-time transmission, data alignment by time stamp, to obtain synchronized multimodal data. The synchronization formula is:

[0199] D sync (t)={I filtered (t i ),S clean (t j ),X filtered (t k ),D text (t m )}

[0200] Among them, D sync (t) represents the synchronized multimodal data at time t; I filtered (t i ) represents t i The filtered image data at the moment, S clean (t j ) represents t j The sound data after noise reduction at all times, X filtered (t k ) represents t k Apply low-pass filtering to the sensor data at all times, D text (t m )=Tokenize(T(t m )) means t m The text data after word segmentation at time T(t m ) represents t m The original text data at the moment; Tokenize represents the word segmentation operation;

[0201] |t i -t j |<∈,|t i -t k |<∈,|t i -t m |<∈, ∈ is the time deviation.

[0202] like Figure 3As shown, step (2) specifically includes the following steps:

[0203] (2a) BEiT-3 model (Masked Image Modeling Transformer) is used for image encoding, such as Figure 4 As shown:

[0204] v I =BEiT(x I )

[0205] in, is the output image feature vector, representing the visual features of the device appearance and thermal imaging, d I is 1024; x I is the input image, Where H and W are both 224;

[0206] (2b) The encoder of the Whisper model (OpenAI speech model) is used to encode the sound signal, such as Figure 5 As shown:

[0207] v A =Whisper(x A )

[0208] in, is the output sound feature vector, reflecting the abnormal noise pattern during equipment operation, d A is 512; x A is the input sound spectrum;

[0209] (2c) Encode the sensor data using the encoder of the Informer model (self-attention temporal prediction model), such as Figure 6 As shown:

[0210] v S =Informer(x S )

[0211] in, The sensor feature vector representing the output captures the dynamic changes of the device state, d S is 512; x S is the input sensor time series;

[0212] (2d) Use the T5 model (Text-to-Text Transfer Transformer) for text encoding:

[0213] v T =T5(x T )

[0214] in, Represents the output text feature vector, which contains the semantic information of operation and maintenance logs and historical faults; d T is 1024; x T Represents the input text sequence;

[0215] (2e) The linear projection layer is used to map the feature vectors of each modality to a unified dimension D of 1024 for the representation vectors of each modality for subsequent fusion:

[0216] m i =W i ·v i +b i

[0217] Where m i represents the unified representation of the i-th modality, i∈{I,A,S,T}; represents the projection matrix, d i represents the dimension of the i-th modal eigenvector; represents the bias vector, v i represents the eigenvector of the i-th mode;

[0218] The final output is a unified multimodal representation vector set V:

[0219] V=[m I ,m A , m S ,m T ]

[0220] Where m I The representation vector representing the image modality; m A The representation vector representing the sound mode; m S The representation vector representing the sensor modality; m T A representation vector representing the text modality.

[0221] Step (3) specifically refers to: using Flamingo (DeepMind multimodal Transformer) to achieve extended modal fusion and obtain a unified fusion feature v multimodal :

[0222] v multimodal =Flamingo(m I ,m A ,m S ,m T )

[0223] Where, v multimodal is a unified fusion feature, m I The representation vector representing the image modality; m A The representation vector representing the sound mode; mS The representation vector representing the sensor modality; m T The representation vector representing the text modality. multimodal The Flamingo model integrates features from images, sounds, sensors, and text to capture comprehensive information about device status. It is a large visual language model that interweaves images and text.

[0224] Step (4) specifically includes the following steps:

[0225] (4a) Unified fusion feature v of the input multimodal Normalize and get the normalized feature v norm :

[0226]

[0227] Among them, μ v Represents all v in the training set multimodal The mean of v Represents all v in the training set multimodal The standard deviation of represents the normalized features, and D is the unified dimension of the representation vectors of each modality. Model training requires a data set, which requires collecting a series of {image, sound, sensor, text} pairs to form a data set. The data set is divided into a training set and a test set according to a certain ratio. The training set is used to train the model, and the test set is used to test the generalization of the model.

[0228] (4b) Contrastive learning detection: A large number of feature vector sets {v normal,1 ,v normal,2 ,…,v normal,M}, M is the number of normal samples; the negative sample library collects a set of known abnormal samples {v abnormal，1 ,…,v abnormal,N}, N is the number of abnormal samples;

[0229] Use cosine similarity to measure the normalized feature v norm The degree of closeness to the normal sample sin(v norm ,v normal,i ):

[0230]

[0231] Among them, v normal,i Represents the characteristics of the i-th normal sample;

[0232] Defining the loss function for contrastive learning Encourage norm Closer to normal samples and further away from abnormal samples:

[0233]

[0234] Among them, v j represents the characteristics of the jth sample in the total set of normal samples and abnormal samples; τ represents the temperature coefficient, which is 0.1;

[0235] Finally calculate v norm The average similarity S with normal samples contrastive :

[0236]

[0237] Among them, S contrastive A value close to 1 indicates that the current device status is normal, and a value close to 0 indicates an abnormality.

[0238] (4c) Autoencoder detection: The autoencoder detects anomalies by reconstruction error and assumes that the reconstruction error of normal data is small. The autoencoder consists of an encoder and a decoder. The encoder is used to convert v norm Compress to a low-dimensional latent space:

[0239] z=f enc (v norm )=W enc ReLU(W in v norm +b in )+b enc

[0240] in, d z <D,d z Take 256, D represents the unified dimension of each modality representation vector, which is 1024; W enc 、W in Both represent weight matrices; b in 、b enc All represent bias; ReLU represents activation function; z represents the low-dimensional feature vector after being encoded by the encoder;

[0241] The decoder reconstructs z back to the original dimension:

[0242]

[0243] Among them, W dec 、W hidden Both represent weight matrices; b hidden 、b dec Both represent bias; Represents the reconstructed features; f dec (z) represents decoding the low-dimensional feature vector z;

[0244] Use normal data to train the autoencoder. Normal data is {video, sound, sensor, text} data. The device state corresponding to normal data is normal. Define a reconstruction loss Minimize the reconstruction error of normal data:

[0245]

[0246] After training is completed, the reconstruction error S is calculated for the test data recon :

[0247]

[0248] S recon When it is larger, it means the current device status is abnormal;

[0249] S recon Normalize by the statistics of the training set:

[0250]

[0251] Among them, μ recon Represents all S in the training set recon The average value of σ recon Represents all S in the training set recon The standard deviation of S' recon represents the normalized reconstruction error;

[0252] (4d) Fusion of the results of contrastive learning and autoencoder to generate the final anomaly score S anomaly :

[0253] S anomaly =α(1-S contrastive )+(1-α)S' recon

[0254] Among them, α∈[0,1] represents the weight parameter, which is 0.5; 1-S contrastive Indicates converting similarity into abnormality;

[0255] Set a threshold T ano , when S anomaly >T ano When , it is judged as abnormal;

[0256] If an anomaly is detected, the fault type needs to be further classified, which is achieved through a fully connected neural network:

[0257] P(fault)=softmax(W c ·v norm +b c )

[0258] in, represent weights and biases respectively, K is the number of fault categories, such as "normal", "overheating", "abnormal vibration", etc.; P(fault) represents the probability distribution of each type of fault; softmax represents the activation function;

[0259] During training, labeled fault data is needed to optimize the cross entropy loss:

[0260]

[0261] Among them, y k represents the true label, P k represents the kth item in the probability distribution of the model output, Represents the cross entropy loss, which is used to train the model.

[0262] Step (5) specifically includes the following steps:

[0263] (5a) Predict the future operating status of the equipment and provide early warning based on the prediction results. The Informer (self-attention time series model) model is used to process the time series data of the equipment status. Combined with historical failure modes and normal status benchmarks, potential failures are predicted and operation and maintenance suggestions are provided.

[0264] By leveraging time series fusion features, we can predict future equipment status trends and identify potential failure risks in advance by analyzing deviations from normal conditions. The final output includes future state predictions and early warning signals, which are used to guide the intelligent operation and maintenance of thermal power plants.

[0265] The unified fusion feature v multimodal Perform time series construction and normalization processing, and extract the feature vector V of T consecutive time steps from historical data t =[v multimodal,t-T+1 ,v multimodal,t-T+2 ,…,v multimodal,t ],in D m is the feature dimension, which is the same as the dimension D of the unified representation vector of each modality, that is, 1024; then each dimension is normalized:

[0266]

[0267] Among them, μ t , σ t Represents the time series mean and standard deviation of the training set; V' t represents the normalized time series;

[0268] (5b) The Informer model is used to predict the future state characteristics of the device. The placeholder of the future time step is input, first passes through the Transformer encoding layer of the Informer model decoder, and then input into the multi-head self-attention layer together with the output of the Informer model encoder to output the predicted value, such as Figure 7 As shown:

[0269]

[0270] in, is the predicted value for the next step;

[0271] During training, it is necessary to minimize the mean square error between the predicted value and the true future state:

[0272]

[0273] Where, v multimodal,t+1 Indicates the real state of the next step, that is, the real future state, Indicates the predicted value for the next step; Represents the mean square error between the predicted value and the true future state;

[0274] (5c) Based on the prediction results Deviation analysis and failure probability assessment can provide early warning:

[0275] (5c1) Deviation analysis: Define a normal state benchmark and use the average eigenvector v calculated using normal operating data normal_benchmark As a normal state benchmark:

[0276]

[0277] Where M represents the number of normal samples; Represents the feature vector of the i-th normal sample;

[0278] Then calculate the predicted state Compared with the normal state benchmark v normal_benchmark The Euclidean distance of deviation :

[0279]

[0280] in, Indicates the predicted state; D deviation Used to measure the degree of deviation between the predicted state and the normal state benchmark;

[0281] Finally, set a deviation threshold T dev When D deviation >T dev When it exists, trigger an early warning;

[0282] (5c2) Failure probability assessment:

[0283] First build the failure mode library {v fault,1 ,…,v fault,K}, K is the number of fault categories, and then the cosine similarity is used to evaluate the predicted state The degree of proximity to each failure mode S k :

[0284]

[0285] Among them, v fault,k Denotes the kth failure mode and normalizes the result to probability:

[0286]

[0287] Among them, τ p Indicates the temperature coefficient, which is 0.1; P(fault k ) represents the probability of occurrence of the kth failure mode;

[0288] when When T prob Represents a threshold, T prob Take 0.7; Indicates the failure mode with the highest probability of occurrence; It means that as long as the probability of the failure mode occurring exceeds the threshold T prob , then the fault with the highest probability of occurrence is judged to be a high-risk fault type;

[0289] use Indicates the number of the failure mode with the highest probability of occurrence; The corresponding highest probability category provides corresponding suggestions. For example, if the prediction result is "overheating", the suggestion is "check the cooling system". Indicates that from P(fault k ) to obtain the column number with the highest value, P(fault k ) corresponds to the probability of a fault category occurring, that is, the corresponding highest probability category.

[0290] Step (6) specifically includes the following steps:

[0291] (6a) New data X new After multimodal encoding and multimodal fusion, the fused feature vector v is obtained multimodal,new :

[0292] v multimodal,new

[0293] =Flamingo(BEiT(x I,new ),Whisper(x A,new ),Informer(x S,new ),T5(x T,new ))

[0294] Where x I，new Represents the image data in the new data; x A,new Represents the sound signal in the new data; x S,new Represents the sensor data in the new data; x T,new Represents the text data in the new data; v multimodal,new The fused feature vector representing the new data;

[0295] Afterwards, v multimodal,new Normalize:

[0296]

[0297] Among them, μ and σ are the mean and standard deviation calculated based on historical data respectively; v norm,new represents the normalized fusion feature vector;

[0298] In trend forecasting, new data is organized into a time series V with a window length of T. new,t =[v multimodal,new,t-T+1 ,…,v multimodal,new,t ];

[0299] (6b) Adopting incremental learning mechanism to gradually learn new data, the main steps are as follows:

[0300] (6b1) Data buffering and selection:

[0301] First, you need to maintain a fixed-size buffer pool Capacity 1000, storage of historical data samples {(v norm,i ,y i )} and new data {(v norm,new ,y new )},y i with y new Both represent status labels;

[0302] Then use random replacement to update the buffer pool to ensure that the new and old data are balanced:

[0303]

[0304] in, Indicates the number of samples in the buffer pool; size(v new) represents the number of samples of newly added new data; P(keep) represents the probability of each old sample in the buffer pool being retained;

[0305] (6b2) Model update:

[0306] Fine-tune the model on the new data while retaining the performance on the old data; fine-tune the model using the data in the buffer pool. For the device anomaly detection module, the loss function is:

[0307]

[0308] Where, τ c represents the temperature coefficient, which is 0.1; v normal represents a normal sample; v j represents the jth sample in the buffer pool;

[0309] For the equipment status trend prediction module, the loss function is:

[0310]

[0311] Where, Indicates the predicted value of the next step corresponding to the new data, that is, the predicted state; v multimodal,new，t+1 Indicates the actual state of the next step corresponding to the new data, that is, the actual future state;

[0312] At the same time, the model parameters are updated through mini-batch gradient descent;

[0313] (6c) Knowledge Distillation Optimization

[0314] First, get the output of the teacher model for the new data:

[0315]

[0316] Where, represents the teacher model; v norm,new,i Normalized features of the i-th new data; represents the output of the teacher model;

[0317] Then get the output of the student model for the new data:

[0318]

[0319] Where, represents the student model; represents the output of the student model;

[0320] Finally calculate the KL divergence loss

[0321]

[0322] Among them, D KL represents KL divergence;

[0323] Calculate total loss

[0324]

[0325] Among them, θ represents the balance factor, which is 0.7; represents the task-specific loss, which is formulated as:

[0326]

[0327] Where, To detect loss, To predict losses;

[0328] By optimizing This involves updating the student model, then replacing the old parameters of each module in the system with the updated student model. This update is triggered every time a certain amount of new data is received. If the performance of the updated model degrades by more than a certain percentage, the system is rolled back to the old model. The certain amount of new data refers to 70 to 100 pieces of data, and the degraded performance refers to a drop of more than 3%.

[0329] like Figure 2 As shown, this system includes:

[0330] The multimodal data acquisition module collects the equipment's operating data, i.e., multimodal data, in real time and transmits the multimodal data to the multimodal Transformer model;

[0331] The multimodal data encoding module uses the BEiT-3 model, Whisper model, Informer model, and T5 model to encode different types of multimodal data and generate a unified multimodal representation;

[0332] The multimodal fusion analysis module uses the Flamingo model to perform cross-modal data fusion and analysis on the unified multimodal representation to obtain unified fusion features;

[0333] The device anomaly detection and fault diagnosis module uses an autoencoder to calculate the device's anomaly degree based on unified fusion features, using contrastive learning and autoencoders to determine whether the device is abnormal.

[0334] The equipment status trend prediction and early warning module uses the Informer model to predict future equipment status based on unified fusion features, predict the probability of potential failures, and provide operation and maintenance recommendations and early warnings;

[0335] The online learning and module optimization module optimizes the multimodal Transformer model through incremental learning and knowledge distillation.

[0336] In summary, the present invention proposes to use multimodal fusion to improve fault detection accuracy: traditional thermal power plant equipment monitoring systems mainly rely on single sensor data, and the fault detection accuracy is low; while the present invention adopts a multimodal large model, combines image, sound, sensor and text data, and uses Flamingo for cross-modal feature fusion. Compared with traditional methods, it has higher accuracy and lower false alarm and omission rates.

[0337] The present invention significantly improves the lead time of early warning: the equipment early warning of traditional monitoring systems is usually based on fixed thresholds or rule methods, and can only issue an alarm after a fault occurs or when it is about to occur; the present invention uses the Informer time series prediction model to perform trend analysis on the equipment status, which can predict the time of fault occurrence. The prediction lead time is greatly improved, providing operation and maintenance personnel with more sufficient maintenance time. By combining anomaly detection and prediction, the probability of sudden failures is reduced.

[0338] The present invention improves the intelligence and adaptability of equipment fault diagnosis: Traditional systems require experts to manually set thresholds or maintain them based on rule bases, and their ability to adapt to different equipment types and operating environments is limited. However, the present invention uses an online incremental learning mechanism to automatically learn the operating characteristics of different thermal power plants and different equipment. When migrating to new equipment, only a small amount of new data is needed for fine-tuning to achieve accurate monitoring. Knowledge distillation is also used to achieve incremental updates of the model, allowing it to continuously adapt to new equipment operating modes without affecting its original performance.

[0339] This invention enhances the explainability of fault causes: Traditional AI systems often lack explainability in fault detection results, hindering rapid decision-making by maintenance personnel. However, this invention utilizes the T5 model, combined with historical maintenance logs, to automatically generate fault cause descriptions and provide repair recommendations. BEiT image analysis combined with Whisper sound diagnosis generates visual and audio playback fault diagnosis reports, helping engineers quickly locate the source of the fault.

[0340] The present invention reduces manual operation and maintenance costs and improves operation and maintenance efficiency: Traditional thermal power plant equipment monitoring requires a large number of operation and maintenance personnel to conduct inspections, manual data analysis, and manual early warning, resulting in high operation and maintenance costs and slow response speed; the present invention uses intelligent automated monitoring to reduce manual inspection time, provide automatic alarms and remote diagnosis, improve fault handling efficiency, reduce unplanned downtime, increase equipment availability, and reduce the average annual maintenance cost of thermal power plants.

[0341] The present invention is applicable to different thermal power plant equipment and has strong versatility. Traditional systems often require separate detection models to be built for different equipment, which makes reuse difficult. The present invention adopts a pre-training and fine-tuning mechanism for a multimodal large model, which has strong migration adaptability between different thermal power plants and different equipment, reduces training costs, and can be quickly deployed to different plants, reducing deployment costs.

[0342] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model, characterized by: The method comprises the following steps in sequence: (1) Collecting the operating data of the equipment, i.e., multimodal data, in real time. The multimodal data includes image data, sound data, sensor data, and text data, and transmitting the multimodal data to the multimodal Transformer model; (2) Constructing a multimodal Transformer model: The BEiT-3 model, Whisper model, Informer model, T5 model, Flamingo model, and autoencoder together form a multimodal Transformer model to encode different types of multimodal data and generate a unified multimodal representation; (3) Perform cross-modal data fusion and analysis on the unified multimodal representation to obtain unified fusion features; (4) Based on the unified fusion features, contrastive learning and autoencoder are used to calculate the abnormality of the device to determine whether the device is abnormal; (5) Based on the unified fusion features, predict the future status of equipment, predict the probability of potential failures, and provide operation and maintenance suggestions and early warnings; (6) Optimize the multimodal Transformer model through incremental learning and knowledge distillation.

2. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 1 is characterized by: Step (1) specifically includes the following steps: (1a) Image acquisition: A high-definition industrial camera installed in the visible area of the electrical equipment is used to capture the appearance image of the equipment and detect physical defects; the obtained image is subjected to Gaussian filtering to remove noise and avoid the influence of noise. i Apply Gaussian filtering to denoise the pixel at position (x,y) in the image collected at that moment: Where σ is the standard deviation of the Gaussian kernel, I filtered (x,y,t i ) is t i The result of denoising the pixel at the (x, y) position at the moment after Gaussian filtering; I(u, v, t i ) represents t i The pixel value of the image collected at the moment (u, v); (1b) Sound collection: High-sensitivity microphones placed in noise-sensitive areas of electrical equipment are used to collect operating sounds, identify abnormal noises, and j The sound signal S(f,t j ) Apply spectral subtraction to reduce noise: S clean (f,t j )=max(S(f,t j )-α f N(f),0) Where N(f) is the noise estimate; α f is the noise reduction coefficient, which is 1.5; S clean (f,t j ) is t j The sound signal after spectral subtraction noise reduction at all times; The noise estimate N(f) is obtained using the minimum statistics method: Among them, W t is the time window; β is the correction factor, which is 1.2; (1c) Sensor data acquisition: Electrical operating parameters are collected through current sensors, voltage sensors, temperature sensors, and vibration sensors to monitor the equipment status. Low-pass filtering is applied to the sensor data X(t) collected at time t: Where X(τ) represents the raw sensor data collected at time τ; h(t) represents the impulse response of the filter. The calculation of h(t) is as follows: Among them, ω c =2πf c , f c is the cutoff frequency; u(t) is the unit step function, when t≥0, u(t) is 1, otherwise, u(t) is 0; (1d) Text data collection: Through operation logs, electricity meter readings, and maintenance records, historical data and context information are provided, and the text is segmented and formatted in a standardized manner; (1e) Data synchronization and transmission: Synchronous real-time transmission, data alignment by time stamp, to obtain synchronized multimodal data. The synchronization formula is: D sync (t)={I filtered (t i ),S clean (t j ),X filtered (t k ),D text (t m )} Among them, D sync (t) represents the synchronized multimodal data at time t; I filtered (t i ) represents t i The filtered image data at the moment, S clean (t j ) represents t j The sound data after noise reduction at all times, X filtered (t k ) represents t k Apply low-pass filtering to the sensor data at all times, D text (t m )=Tokenize(T(t m )) means t m The text data after word segmentation at time T(t m ) represents t m The original text data at the moment; Tokenize represents the word segmentation operation; |t i -t j |<∈,|t i -t k |<∈,|t i -t m |<∈, ∈ is the time deviation.

3. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 1 is characterized by: Step (2) specifically includes the following steps: (2a) Using the BEiT-3 model for image coding: v I =BEiT(x I ) in, is the output image feature vector, representing the visual features of the device appearance and thermal imaging, d I is 1024; x I is the input image, Where H and W are both 224; (2b) Encode the sound signal using the Whisper model encoder: v A =Whisper(x A ) in, is the output sound feature vector, reflecting the abnormal noise pattern during equipment operation, d A is 512; x A is the input sound spectrum; (2c) Encode the sensor data using the encoder of the Informer model: v S =Informer(x S ) in, The sensor feature vector representing the output captures the dynamic changes of the device state, d S is 512; x S is the input sensor time series; (2d) Use T5 model for text encoding: v T =T5(x T ) in, Represents the output text feature vector, which contains the semantic information of operation and maintenance logs and historical faults; d T is 1024; x T Represents the input text sequence; (2e) The linear projection layer is used to map the feature vectors of each modality to a unified dimension D of 1024 for the representation vectors of each modality for subsequent fusion: m i =W i ·v i +b i Where m i represents the unified representation of the i-th modality, i∈{I,A,S,T}; represents the projection matrix, d i represents the dimension of the i-th modal eigenvector; represents the bias vector, v i represents the eigenvector of the i-th mode; The final output is a unified multimodal representation vector set V: V=[m I ,m A ,m S ,m T ] Where m I The representation vector representing the image modality; m A The representation vector representing the sound mode; m S The representation vector representing the sensor modality; m T A representation vector representing the text modality.

4. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 1 is characterized by: Step (3) specifically refers to: using Flamingo to achieve extended modal fusion and obtain a unified fusion feature v multimodal : v multimodal =Flamingo(m I ,m A ,m S ,m T ) Where, v multimodal is a unified fusion feature, m I The representation vector representing the image modality; m A The representation vector representing the sound mode; m S The representation vector representing the sensor modality; m T A representation vector representing the text modality.

5. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 1 is characterized by: Step (4) specifically includes the following steps: (4a) Unified fusion feature v of the input multimodal Normalize and get the normalized feature v norm : Among them, μ v Represents all v in the training set multimodal The mean of v Represents all v in the training set multimodal The standard deviation of represents the normalized features, and D is the unified dimension of each modality representation vector; (4b) Contrastive learning detection: A large number of feature vector sets {v normal,1 ,v normal,2 ,…,v normal,M }, M is the number of normal samples; the negative sample library collects a set of known abnormal samples {v abnormal，1 ,…,v abnormal,N }, N is the number of abnormal samples; Use cosine similarity to measure the normalized feature v norm The degree of closeness to the normal sample sin(v norm ,v normal,i ): Among them, v normal,i Represents the characteristics of the i-th normal sample; Defining the loss function for contrastive learning Encourage norm Closer to normal samples and further away from abnormal samples: Among them, v j represents the characteristics of the jth sample in the total set of normal samples and abnormal samples; τ represents the temperature coefficient, which is 0.1; Finally calculate v norm The average similarity S with normal samples contrastive : Among them, S contrastive A value close to 1 indicates that the current device status is normal, and a value close to 0 indicates an abnormality. (4c) Autoencoder detection: The autoencoder consists of an encoder and a decoder. The encoder is used to convert v norm Compress to a low-dimensional latent space: z=f enc (v norm )=In enc ·ReLU(W in v norm +b in )+b enc in, d z <D,d z Take 256, D represents the unified dimension of each modality representation vector, which is 1024; W enc 、W in Both represent weight matrices; b in 、b enc All represent bias; ReLU represents activation function; z represents the low-dimensional feature vector after being encoded by the encoder; The decoder reconstructs z back to the original dimension: Among them, W dec 、W hidden Both represent weight matrices; b hidden 、b dec Both represent bias; Represents the reconstructed features; f dec (z) represents decoding the low-dimensional feature vector z; Use normal data to train the autoencoder and define a reconstruction loss Minimize the reconstruction error of normal data: After training is completed, the reconstruction error S is calculated for the test data recon : S recon When it is larger, it means the current device status is abnormal; S recon Normalize by the statistics of the training set: Among them, μ recon Represents all S in the training set recon The average value of σ recon Represents all S in the training set recon The standard deviation of recon represents the normalized reconstruction error; (4d) Fusion of the results of contrastive learning and autoencoder to generate the final anomaly score S anomaly : S anomaly =α(1-S contrastive )+(1-α)S' recon Among them, α∈[0,1] represents the weight parameter, which is 0.5; 1-S contrastive Indicates converting similarity into abnormality; Set a threshold T ano , when S anomaly >T ano When , it is judged as abnormal; If an anomaly is detected, the fault type needs to be further classified, which is achieved through a fully connected neural network: P(fault)=softmax(W c ·v norm +b c ) in, represent weights and biases respectively, K is the number of fault categories; P(fault) represents the probability distribution of each type of fault; softmax represents the activation function; During training, labeled fault data is needed to optimize the cross entropy loss: Among them, y k represents the true label, P k represents the kth item in the probability distribution of the model output, Represents the cross entropy loss, which is used to train the model.

6. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 1 is characterized by: Step (5) specifically includes the following steps: (5a) For the unified fusion feature v multimodal Perform time series construction and normalization processing, and extract the feature vector V of T consecutive time steps from historical data t =[v multimodal,t-T+1 ,v multimodal,t-T+2 ,…,v multimodal,t ],in D m is the feature dimension, which is the same as the dimension D of the unified representation vector of each modality, that is, 1024; then each dimension is normalized: Among them, μ t ,σ t Represents the time series mean and standard deviation of the training set; V' t represents the normalized time series; (5b) The informer model is used to predict the future state characteristics of the device. The placeholder for the future time step is input, first passed through the Transformer encoding layer of the informer model decoder, and then input into the multi-head self-attention layer together with the output of the informer model encoder to output the predicted value: in, is the predicted value for the next step; During training, it is necessary to minimize the mean square error between the predicted value and the true future state: Where, v multimodal,t+1 Indicates the real state of the next step, that is, the real future state, Indicates the predicted value for the next step; Represents the mean square error between the predicted value and the true future state; (5c) Based on the prediction results Early warning can be achieved through deviation analysis and failure probability assessment.

7. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 1 is characterized by: Step (6) specifically includes the following steps: (6a) New data X new After multimodal encoding and multimodal fusion, the fused feature vector v is obtained multimodal,new : v multimodal,new =Flamingo(BEiT(x I,new ),Whisper(x A,new ),Informer(x S,new ),T5(x T,new )) Where x I，new Represents the image data in the new data; x A,new Represents the sound signal in the new data; x S,new Represents the sensor data in the new data; x T,new Represents the text data in the new data; v multimodal,new The fused feature vector representing the new data; Afterwards, v multimodal,new Normalize: Among them, μ and σ are the mean and standard deviation calculated based on historical data respectively; v norm,new represents the normalized fusion feature vector; In trend forecasting, new data is organized into a time series V with a window length of T. new,t =[v multimodal,new,t -T+1,…,v multimodal,new,t ]; (6b) Using incremental learning mechanism to gradually learn new data; (6c) Knowledge Distillation Optimization First, get the output of the teacher model for the new data: Where, represents the teacher model; v norm,new,i Normalized features of the i-th new data; represents the output of the teacher model; Then get the output of the student model for the new data: Where, represents the student model; represents the output of the student model; Finally calculate the KL divergence loss Among them, D KL represents KL divergence; Calculate total loss Among them, θ represents the balance factor, which is 0.7; represents the task-specific loss, which is formulated as: Where, To detect loss, To predict losses; By optimizing That is, the student model is updated; then the updated student model replaces the old parameters of each module in the system; an update is triggered every time a certain amount of new data is received; if the performance after the update degrades by more than a certain proportion, the old model is rolled back.

8. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 6 is characterized by: Step (5c) specifically comprises the following steps: (5c1) Deviation analysis: Define a normal state benchmark and use the average eigenvector v calculated using normal operating data normal_benchmark As a normal state benchmark: Where M represents the number of normal samples; Represents the feature vector of the i-th normal sample; Then calculate the predicted state Compared with the normal state benchmark v normal_benchmark The Euclidean distance of deviation : in, Indicates the predicted state; D deviation Used to measure the degree of deviation between the predicted state and the normal state benchmark; Finally, set a deviation threshold T dev When D deviation >T dev When it exists, an early warning is triggered; (5c2) Failure probability assessment: First build the failure mode library {v fault,1 ,…,v fault,K }, K is the number of fault categories, and then the cosine similarity is used to evaluate the predicted state The degree of proximity to each failure mode S k : Among them, v fault,k Denotes the kth failure mode and normalizes the result to probability: Among them, τ p Indicates the temperature coefficient, which is 0.1; P(fault k ) represents the probability of occurrence of the kth failure mode; when When T prob Represents a threshold, T prob Take 0.7; Indicates the failure mode with the highest probability of occurrence; It means that as long as the probability of the failure mode occurring exceeds the threshold T prob , then the fault with the highest probability of occurrence is judged to be a high-risk fault type; use Indicates the number of the failure mode with the highest probability of occurrence; The corresponding highest probability category provides corresponding suggestions; Indicates that from P(fault k ) to obtain the column number with the highest value, P(fault k ) corresponds to the probability of a fault category occurring, that is, the corresponding highest probability category.

9. The method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to claim 7 is characterized by: Step (6b) specifically comprises the following steps: (6b1) Data buffering and selection: First, you need to maintain a fixed-size buffer pool Capacity 1000, storage of historical data samples {(v norm,i ,y i )} and new data {(v norm,new ,y new )},y i with y new Both represent status labels; Then use random replacement to update the buffer pool to ensure that the new and old data are balanced: in, Indicates the number of samples in the buffer pool; size(v new ) represents the number of samples of newly added new data; P(keep) represents the probability of each old sample in the buffer pool being retained; (6b2) Model update: Fine-tune the model on the new data while retaining the performance on the old data; fine-tune the model using the data in the buffer pool. For the device anomaly detection module, the loss function is: Where, τ c represents the temperature coefficient, which is 0.1; v normal represents a normal sample; v j represents the jth sample in the buffer pool; For the equipment status trend prediction module, the loss function is: Where, Indicates the predicted value of the next step corresponding to the new data, that is, the predicted state; v multimodal,new,t+1 Indicates the actual state of the next step corresponding to the new data, that is, the actual future state; At the same time, the model parameters are updated through mini-batch gradient descent.

10. A system for implementing the method for intelligent monitoring and early warning of thermal power plant equipment defects based on a multimodal large model according to any one of claims 1 to 9, characterized in that: include: The multimodal data acquisition module collects the equipment's operating data, i.e., multimodal data, in real time and transmits the multimodal data to the multimodal Transformer model; The multimodal data encoding module uses the BEiT-3 model, Whisper model, Informer model, and T5 model to encode different types of multimodal data and generate a unified multimodal representation; The multimodal fusion analysis module uses the Flamingo model to perform cross-modal data fusion and analysis on the unified multimodal representation to obtain unified fusion features; The device anomaly detection and fault diagnosis module uses an autoencoder to calculate the device's anomaly degree based on unified fusion features, using contrastive learning and autoencoders to determine whether the device is abnormal. The equipment status trend prediction and early warning module uses the Informer model to predict future equipment status based on unified fusion features, predict the probability of potential failures, and provide operation and maintenance recommendations and early warnings; The online learning and module optimization module optimizes the multimodal Transformer model through incremental learning and knowledge distillation.

Images