Power plant early warning method based on fine-grained modal information injection and multi-scene task adaptation

Abstract

The invention discloses a power plant early warning method based on fine-grained modal information injection and multi-scene task adaptation, and the method comprises the steps: training a multi-modal deep large model VTCL based on the historical accumulated data of a power plant, injecting the fine-grained text modal information into the model, and achieving the efficient recognition and detection of all devices and scenes in the power plant; and meanwhile, a LoRA fine tuning method is adopted to enable the model VTCL to adapt to various application scenes of the power plant, the parameter size of the model is reduced, the detection capability of the specific scene of the model is improved, and the application capability of the model is improved, so that real-time early warning of equipment and environment in the power plant is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent power information processing, and in particular to a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation. Background Art

[0002] As an important base for power production, the stable operation of power generation enterprises is directly related to the safety and reliable power consumption of the power system. Once a safety accident occurs in a power plant, it may lead to power supply interruption, causing huge losses to social production and people's lives. Power plants involve many complex processes and equipment, such as boilers, steam turbines, generators, etc. These equipment may have various potential dangers during operation, such as fire, explosion, leakage, etc. With the continuous growth of electricity demand, the scale of power plants continues to expand, the number of equipment increases, and the difficulty of safety management increases. At the same time, the processes and equipment involved in power plants are diverse and technically complex, and the requirements for safety management are also getting higher and higher. Regular inspections are an important means to ensure the safe operation of power plants. Traditional manual inspections have problems such as high labor intensity, poor working environment, and unstable inspection results. Therefore, inspection robots are introduced to replace manual inspections to help digital construction.

[0003] With the continuous development of artificial intelligence and deep learning, many intelligent solutions have been applied in the power field to solve the problems encountered in power production. In order to realize the automatic identification of the integrity and environmental status of important equipment in power plants, it is necessary to use multi-source video signals such as fixed cameras, mobile cameras, and inspection robots, and use deep vision big models to realize intelligent monitoring of abnormal conditions such as lost objects, missing parts, gas leakage, liquid leakage, and personnel violations in the scene. When the deep vision big model is deployed and applied in actual power plant scenarios, it needs to be able to adapt to various complex application scenarios, and the hardware resources required for model deployment must also be considered. Therefore, a lightweight algorithm that can adapt to multiple scenarios is of great research value, which can help the construction of digital power plants, improve the safety and reliability of daily operations of power plants, and promote the development of new quality productivity. Summary of the invention

[0004] The present invention mainly provides a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation that can adapt to various complex application scenarios.

[0005] The present invention aims to solve the above technical problems mainly through the following technical solutions: a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation, based on a multi-modal deep large model VTCL (Vision-Text-Concate-LoRA), the multi-modal deep large model includes a basic model and a specific detection scenario incremental module, the power plant early warning method includes a model training process and an inference process, the model training process is specifically:

[0006] S1. Summarize the historical operation data of the power plant to form a training data set and a test data set. Both the training data set and the test data set contain general data and specific data. Specific data refers to data of specific complex scenarios.

[0007] The historical operation data of the power plant includes videos and video image files obtained by front-end sensing devices such as inspection robots and surveillance cameras. These data are labeled and classified, stored by topic, and the content is described in text to obtain training data sets and test data sets.

[0008] S2. Use the regular data in the training data set to train the basic model obtained in the previous round to obtain the basic model parameters of this round; the basic model used in the first round is the original model or the pre-trained model;

[0009] S3. Use the specific data in the training data set to fine-tune the incremental module of the specific detection scenario obtained in the previous round using the LoRA method to obtain the incremental module parameters of the specific detection scenario in this round;

[0010] There are specific complex detection scenarios in the power plant operating environment. Due to occlusion, light and insufficient training data, the detection accuracy of the multimodal deep large model will be affected. Therefore, the LoRA fine-tuning method is introduced to build an incremental detection module for specific complex scene detection.

[0011] S4. Superimpose the basic model parameters of this round and the incremental module parameters of this round of specific detection scenarios to obtain the multimodal deep large model of this round; when facing specific complex scenarios, the model switches to the incremental recognition module for specific complex scenarios to improve the accuracy of the model in various complex detection scenarios in power plant early warning tasks. At the same time, because the LoRA method uses low-rank decomposition technology, it only increases a small number of model parameters while improving the model accuracy, realizing a lightweight target detection method;

[0012] S5, input the test set into the current round of multimodal deep large model, calculate the loss value through the composite loss function, if the loss value is lower than the set threshold, fix the model parameters and end the training; if the loss value is not lower than the set threshold, jump to step S2 to start a new round of training;

[0013] The reasoning process is: input the real-time operation data of the power plant into the trained multi-modal deep big model, and the output is the warning value. The final output includes: detection box (target location) + confidence (warning credibility) + warning category (warning of different problems).

[0014] This solution injects fine-grained text modal information into the model to achieve efficient recognition and detection of various equipment and scenarios in the power plant; the LoRA fine-tuning method is used to make the model VTCL adapt to various application scenarios of the power plant, reduce the parameter size of the model and improve the detection capability of the model in specific scenarios, improve the application capability of the model, and realize real-time early warning of various equipment and environments in the power plant.

[0015] Preferably, the step S2 is specifically as follows:

[0016] S201, inputting the text content in the regular data in the training data set into BERT (Bidirectional Encoder Representation from Transformers) for vector encoding and feature extraction to obtain a text feature vector;

[0017] S202, inputting the visual content in the regular data in the training data set into VIT (Vision Transformer) for vector encoding and feature extraction to obtain a visual feature vector;

[0018] S203, performing a fine-grained modal feature injection operation: injecting a text feature vector into a visual feature vector, and injecting a visual feature vector into a text feature vector, so that the visual modality of the model can focus on the corresponding text modality, and the text modality can also focus on the corresponding visual modality, thereby improving the multi-modal understanding ability of the model in the complex working environment of the power plant;

[0019] Compared with the traditional VTCL model, this solution improves the structure of the VTCL model by introducing a fine-grained modal feature injection operation to enhance the multimodal understanding ability of the model; specifically, this solution adds a Cross-Attention mechanism between the two modalities;

[0020] S204, fusing the text feature vector and the visual feature vector into which the fine-grained modal features have been injected, to obtain a multimodal global feature vector with the fine-grained modal features;

[0021] S205. Use the multimodal global feature vector to train the multimodal deep large model obtained in the previous round. The loss function is a composite loss function to obtain the basic model parameters of this round.

[0022] Preferably, the step S201 is specifically as follows:

[0023] The text content first passes through the word embedding layer of BERT and is represented as three parts: Token Embedding, Segment Embedding, and Position Embedding. The three parts are added bit by bit to output the word embedding vector, as shown in formula (1):

[0024] V T =BERT TE (X T )+BERT SE (X T )+BERT PE (X T ) (1)

[0025] Among them, X T Represents the input text content, BERT TE , BERT SE and BERT PE Respectively represent the Token Embedding, Segment Embedding and Position Embedding operations on the input text content;

[0026] Embed the word into vector V T Enter the BERT network architecture: First, the global features of the text are extracted through the multi-head attention layer, GeLU is used as the activation function of the network, and then it goes through the normalization & residual link layer, and finally the text feature vector is output through the feedforward neural network layer. The whole process is shown in equations (2)-(5):

[0027] V MA =MultiAttention(V T ) (2)

[0028]

[0029] Where MultiAttention in formula (2) represents the calculation of the multi-head attention mechanism, ⊕ in formula (3) represents the residual link, W1 and W2 in formula (4) represent weight parameters, b1 and b2 represent bias parameters, V TO A text feature vector representing the output of BERT.

[0030] Preferably, the step S202 is specifically as follows:

[0031] The VIT model first processes the input representation of the visual content, and after Patch Embedding and Position Embedding operations, the two parts of the representation are fused as the input of the VIT model. The process is shown in formula (6):

[0032] V I =VIT PAE (X I )+VIT PE (X I ) (6)

[0033] Where XI Represents visual content input, VIT PAE Indicates Patch Embedding operation, VIT PE Represents PositionEmbedding operation;

[0034] The visual vector V I Input VIT network, extract features through VIT network, and output visual feature vector V IO .

[0035] Preferably, in step S203, fine-grained modal feature injection is used to inject the text feature vector into the visual feature vector as follows:

[0036] Initialize several groups W Q , W K and W V The matrices represent the query matrix, key matrix and value matrix respectively. The query vector Q, key vector K and value vector V of the visual modality are generated through linear transformation. A set of linear transformation processes is shown in formula (7):

[0037]

[0038] In the formula and is a set of initialized modal parameter matrices. The i in the subscript refers to the head in the multi-head attention mechanism in the Transformer architecture, and I indicates that it is related to the visual feature vector;

[0039] The fine-grained modal feature injection operation of the visual modality is shown in equations (8) to (10):

[0040]

[0041] △Z I =Concat(△Z 1I ,△Z 2I ,…△Z kI )W 0I (9)

[0042] Z VI =V IO +△Z I (10)

[0043] In formula (8), CA I→T (V TO ,V IO ) is the cross-modal interaction process, which includes the linear transformation process in formula (7). is the scaling factor of numerical calculation, d is the dimension of query vector Q and key vector K, and Equation (9) is the process of fusing each group of cross-modal interaction vectors. k represents the kth group of cross-modal interaction process, W 0I is the visual parameter matrix, which is used to map the dimension of the cross-modal interaction fusion vector, and finally inject the text modality features into the visual modality to form the vector Z VI . The superscript T in represents transposition, and the subscript T in represents correlation with the text feature vector.

[0044] Preferably, in step S203, fine-grained modal feature injection is used to inject the feature vector of the visual modality into the feature vector of the text modality as follows:

[0045] First, the query vector Q, key vector K and value vector V of the text modality are generated through linear transformation. A set of linear transformation processes is shown in formula (11):

[0046]

[0047] In the formula is a set of initialized modal parameter matrices;

[0048] The fine-grained modal feature injection operation of the text modality is shown in equations (12) to (14):

[0049]

[0050] △Z T =Concat(△Z 1T ,△Z 2T ,…△Z kT )W 0T (13)

[0051] Z VT =V TO +△Z T (14)

[0052] Where W 0T is the text parameter matrix, which is used to map the dimension of the cross-modal interaction fusion vector, and finally inject the visual modality features into the text modality to form the vector Z VT .

[0053] Preferably, step S204 is specifically as follows:

[0054] The text modality feature vector injected with fine-grained modality features is fused with the feature vector Z of the visual modality by vector concatenation, as shown in formula (15):

[0055] Z=Concate[Z VI ; ZVT ] (15)

[0056] After a linear layer mapping, a multimodal global feature vector Z with fine-grained modal features is obtained. + , as shown in formula (16):

[0057] Z + =ZW Z +b Z (16)

[0058] Where W Z , b Z are weight parameters and bias parameters respectively.

[0059] Preferably, step S3 is specifically as follows:

[0060] S301, freezing the parameters of the basic model in the multimodal deep large model obtained in the previous round;

[0061] S302, decomposing the parameter matrix in the frozen layer of the incremental detection module into two low-rank matrices. The low-rank decomposition process of the weight parameter matrix during LoRA fine-tuning is shown in formula (17):

[0062] W=W0X+ΔWX=W0X+UVX (17)

[0063] Where W0 represents the weight parameter in the frozen layer, ΔW represents the weight parameter of the incremental detection module for specific complex scene detection, ΔW = UV, W0 + ΔW = W0 + UV is used to represent the parameter update, U and V are low-rank decomposition matrices;

[0064] S303, initializing parameters in the low-rank decomposition matrix;

[0065] S304, inputting specific data into an incremental detection module of a specific detection scenario, and outputting a multimodal incremental vector for the specific detection scenario;

[0066] S305. Use the multimodal incremental vector for the specific detection scenario to train the multimodal deep large model obtained in the previous round. The loss function is a composite loss function to obtain the incremental module parameters of the specific detection scenario in this round.

[0067] This solution uses LoRA (Low-Rank Adaptation) technology to build an incremental detection module and uses low-rank decomposition technology to reduce the number of parameters. The LoRA module uses low-rank decomposition technology to decompose the weight matrix in the original model into the product of two low-rank matrices. This method can significantly reduce the number of parameters while maintaining model performance. Specifically, assuming that the original weight matrix is ​​W, it is decomposed into two low-rank matrices A and B, so that W≈AB, where the ranks of A and B are much smaller than the rank of W.

[0068] For different special scene data, LoRA (Low-Rank Adaptation) technology is used to fine-tune the basic model to obtain the corresponding incremental parameters. The incremental detection module is composed of multiple linear layers and can be inserted into different positions of the basic model to enhance the model's ability to cope with the scenario.

[0069] Preferably, the composite loss function is a superposition of three losses: target detection loss, positioning loss and classification loss.

[0070] This solution uses multi-source video equipment such as fixed cameras, mobile cameras, and inspection robots in power plants to collect data from past years to form a multimodal data set, and trains a multimodal deep large model VTCL to achieve intelligent monitoring of abnormal conditions such as missing objects, missing parts, gas leaks, liquid leaks, and personnel violations in the scene. In order to improve the model's multimodal understanding ability in the complex working environment of the power plant, fine-grained modal feature injection operations are added to the training of the multimodal deep large model, so that the model's visual modality can focus on the corresponding text modality, and the text modality can also focus on the corresponding visual modality, which enhances the model's target detection effect;

[0071] In order to improve the target detection accuracy of the multimodal deep large model VTCL in specific complex scenarios in the power plant operating environment, this scheme adopts the LoRA fine-tuning method to face the incremental detection information in different complex scenarios, and uses the incremental modules obtained after LoRA fine-tuning as supplementary modules of the multimodal deep large model, thereby improving the detection performance of the model in the face of specific complex scenarios; the LoRA fine-tuning method is adopted to increase the flexibility and generalization ability of the model by introducing random variables and random operations on the basis of the original model, learn the incremental detection information for specific complex scenarios, and use the various modules obtained after LoRA fine-tuning as supplementary modules of the multimodal deep large model. In this way, the detection accuracy of the model for specific complex scenarios can be improved, and only a small number of parameters are added while still maintaining the response speed of the model, which is of great significance for early warning in the actual application of power plant operating environment.

[0072] The substantial effect brought about by the present invention is that, by using the multimodal deep large model VTCL (Vision-Text-Concate-LoRA) containing fine-grained multimodal information, task adaptation and fine-tuning are performed in multiple scenarios, the model's recognition ability for each specific task is enhanced, and the number of model parameters is compressed. Detection is performed based on text semantic labels, and detection tasks for multiple targets such as the integrity and environmental status of important equipment in power plants are completed, greatly improving the model's early warning accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1It is a schematic diagram of a model training process of the present invention;

[0074] Figure 2 It is a schematic diagram of a basic model training process of the present invention;

[0075] Figure 3 It is a schematic diagram of the training process of an incremental module for a specific detection scenario of the present invention. DETAILED DESCRIPTION

[0076] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0077] Embodiment: A power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation, based on a multi-modal deep large model, the multi-modal deep large model includes a basic model and a specific detection scenario incremental module, the power plant early warning method includes a model training process and an inference process, such as Figure 1 As shown in the figure, the model training process includes:

[0078] S1. According to the training requirements of the multimodal deep large model, the data generated by the power plant over the years are summarized and sorted out to form a training data set for the multimodal deep large model of the power plant;

[0079] The data comes from the video and video image files acquired by the inspection robots, surveillance cameras and other front-end sensing devices in the power plant. The files are labeled and classified, stored by topic, and the content is described in text. The data set required for model training is collected. The data content includes the wearing of safety helmets, the normal wearing of work clothes, the safe operation process of equipment, and the normal operation of equipment. The aggregated data set is divided into a training set and a test set in a ratio of 6:4;

[0080] S2. Input the multimodal data of the power plant into the multimodal deep large model VTCL for training. Use the text deep model and the visual deep model to vectorize and extract features from the text content and visual content of the input data, and output text feature vectors and visual feature vectors. Add fine-grained modal feature injection operations to the large model VTCL to improve the multimodal understanding ability of the model in the complex working environment of the power plant. Finally, fuse the text modal feature vector injected with fine-grained modal features and the feature vector of the visual modality to output a multimodal global feature vector with fine-grained modal features. Figure 2 As shown, the specific process is as follows:

[0081] S201, input the text content of the power plant multimodal data set into BERT (Bidirectional Encoder Representation from Transformers) for vector encoding and feature extraction, and output a text feature vector;

[0082] The text content of the power plant multimodal data first passes through the word embedding layer of BERT and is represented as three parts: TokenEmbedding, Segment Embedding, and Position Embedding. The three parts are added bit by bit to output the word embedding vector, as shown in formula (1):

[0083] V T =BERT TE (X T )+BERT SE (X T )+BERT PE (X T ) (1)

[0084] Among them, X T Text content input representing multimodal data of a power plant, BERT TE , BERT SE , BERT PE Respectively represent Token Embedding, Segment Embedding, and Position Embedding operations on the input text;

[0085] Embed the word into vector V T Input BERT network architecture, which can have multiple layers during training. First, it passes through the multi-head attention layer to extract the global features of the text. Here, GeLU is used instead of Softmax as the activation function of the network. The GeLU activation function has sparse characteristics. Unlike the Softmax function, this function can output any negative value and is more flexible. Then, it passes through the normalization & residual link layer to stabilize the parameters in the network structure, smooth the gradient, and optimize the gradient disappearance or gradient explosion problems that may occur during the training process. Finally, the text feature vector is output through the feedforward neural network layer. The process is shown in equations (2)-(5);

[0086] V MA =MultiAttention(V T ) (2)

[0087]

[0088] The MultiAttention in formula (2) represents the calculation of the multi-head attention mechanism, which will not be repeated here. represents the residual link, W1 and W2 represent weight parameters, b1 and b2 represent bias parameters, V TO Represents the text feature vector output by BERT;

[0089] S202, inputting the visual content of the power plant multimodal data set into VIT (Vision Transformer) for vector encoding and feature extraction, and outputting a visual feature vector;

[0090] The VIT model first processes the input representation of the visual content, and after Patch Embedding and Position Embedding operations, the two parts of the representation are fused as the input of the VIT model. Unlike the Position Embedding in BERT, which uses sine and cosine functions, VIT uses natural numbers such as 0, 1, 2, etc. for position encoding. The process is shown in formula (6):

[0091] V I =VIT PAE (X I )+VIT PE (X I ) (6)

[0092] Where X I Represents visual content input, VIT PAE Indicates Patch Embedding operation, VIT PE Represents PositionEmbedding operation;

[0093] The visual vector V I Input VIT network, which uses the same Transformer-Encoder architecture as BERT. The network extracts features and outputs visual feature vector V IO ;

[0094] S203. Add a fine-grained modal feature injection operation to the large model VTCL, inject the feature vector of the text modality into the feature vector of the visual modality, and inject the feature vector of the visual modality into the feature vector of the text modality, so that the visual modality of the model can focus on the corresponding text modality, and the text modality can also focus on the corresponding visual modality, thereby improving the multi-modal understanding ability of the model in the complex working environment of the power plant;

[0095] Different modalities have stronger heterogeneity, and the features of other modalities cannot be directly integrated into the features of another modality. In order to bridge the gap between different modalities, fine-grained modal feature injection can achieve feature fusion across modalities, thereby improving the multi-modal understanding ability of the model under the complex working environment of the power plant and improving the early warning effect of the power plant;

[0096] Fine-grained modality feature injection is used to inject the feature vector of the text modality into the feature vector of the visual modality. Initialize multiple groups W Q , W K , W VThe matrices represent the query matrix, key matrix and value matrix respectively. The query vector Q, key vector K and value vector V of the visual modality are generated through linear transformation. A set of linear transformation processes is shown in formula (7):

[0097]

[0098] In the formula is a set of initialized modal parameter matrices; the i in the subscript refers to the different heads in the multi-head attention mechanism in the Transformer architecture. In the multi-head attention mechanism, the query, key, and value matrices of different heads will have different indexes to distinguish;

[0099] The fine-grained modal feature injection operation of the visual modality is shown in equations (8) to (10):

[0100]

[0101] △Z I =Concat(△Z 1I ,△Z 2I ,…△Z kI )W 0I (9)

[0102] Z VI =V IO +△Z I (10)

[0103] In formula (8), CA I→T (V TO ,V IO ) is the cross-modal interaction process, which includes the linear transformation process in formula (7), divided by This is to prevent the molecular internal volume from being too large. is a scaling factor for numerical calculations, which prevents the dot product value from being too large, causing the Softmax function to output too large, thus causing the gradient vanishing problem. Keeping the dot product result within an appropriate range makes the gradient more stable, which helps the convergence of model training. The value of d here is usually the dimension value of the query vector Q and the key vector K. The square root of d is because the variance of the numerator dot product is proportional to d, and the square root can keep the growth of the numerator and denominator consistent, thereby effectively controlling the range of the dot product value in the numerator. Formula (9) is the process of fusing each group of cross-modal interaction vectors, △Z k represents the kth group of cross-modal interaction process, W 0I is a parameter matrix used to map the dimensions of the cross-modal interaction fusion vector, and finally inject the text modality features into the visual modality to form a vector Z VI ;

[0104] Fine-grained modality feature injection is used to inject the feature vector of the visual modality into the feature vector of the text modality. The process is the same as above. First, the query vector Q, key vector K and value vector V of the text modality are generated through linear transformation. A set of linear transformation processes is shown in formula (11):

[0105]

[0106] In the formula is a set of initialized modal parameter matrices;

[0107] The fine-grained modal feature injection operation of the text modality is shown in equations (12) to (14):

[0108]

[0109] △Z T =Concat(△Z 1T ,△Z 2T ,…△Z kT )W 0T (13)

[0110] Z VT =V TO +△Z T (14)

[0111] Where W 0T is a parameter matrix used to map the dimension of the cross-modal interaction fusion vector, and finally inject the visual modality features into the text modality to form a vector Z VT ;

[0112] S204, fusing the text modality feature vector injected with the fine-grained modality feature and the feature vector of the visual modality, and outputting a multimodal global feature vector with the fine-grained modality feature;

[0113] The text modality feature vector injected with fine-grained modality features is fused with the feature vector of the visual modality by vector concatenation, as shown in formula (15):

[0114] Z=Concate[Z VI ; Z VT ] (15)

[0115] After a linear layer mapping, a multimodal global feature vector with fine-grained modal features is obtained, as shown in formula (16):

[0116] Z + =ZW Z +b Z (16)

[0117] Where W Z , b Zare weight parameter and bias parameter respectively;

[0118] S205, training basic model parameters according to the composite loss function: optimizing parameters according to the loss function formed by the linear combination of the three loss functions, and training basic model parameters of the multimodal deep large model;

[0119] S3. There are specific complex detection scenarios in the power plant operating environment. Due to occlusion, light and insufficient training data, the detection accuracy of the multimodal deep large model will be affected. For this reason, the LoRA fine-tuning method is introduced to construct an incremental detection module for specific complex scene detection. The data of specific complex scenes is input, and LoRA fine-tunes the incremental parameters for each specific detection scenario; Figure 3 As shown, the specific process is:

[0120] S301, freeze the weight parameters of the multimodal deep large model VTC trained by the multimodal data of the power plant; the initial multimodal deep large model VTCL has pre-trained weight parameters, and then lora fine-tuning is performed on the multimodal deep large model VTC that has been trained by the multimodal data of the power plant;

[0121] During model fine-tuning, LoRA freezes the weight parameters of the multimodal deep large model, that is, these weight parameters are not updated during model training, thereby reducing the number of weight parameters for specific complex scene detection tasks without adding additional inference delays.

[0122] S302, using LoRA to build an incremental detection module for specific complex scene detection, and using low-rank decomposition technology to reduce the number of parameters;

[0123] The incremental detection module for specific complex scene detection adopts low-rank decomposition technology, that is, decomposing the parameter matrix in the frozen layer into two low-rank matrices, so as to reduce the number of model parameters and improve the speed of fine-tuning. The low-rank decomposition process of the weight parameter matrix during LoRA fine-tuning is shown in formula (17):

[0124] W=W0X+ΔWX=W0X+UVX (17)

[0125] Where W0 represents the weight parameter in the frozen layer, ΔW represents the weight parameter of the incremental detection module for specific complex scene detection, ΔW = UV, W0 + ΔW = W0 + UV is used to represent the parameter update, U and V are low-rank decomposition matrices;

[0126] S303, initializing parameters in the low-rank decomposition matrix;

[0127] During the fine-tuning process, the parameter matrix W0 in the frozen layer does not participate in the gradient update, the parameters remain unchanged, and the parameters in the matrix U are initialized using a random Gaussian distribution, that is, the parameter distribution in the matrix U satisfies a Gaussian distribution with a mean of zero and a variance of a fixed value (artificially set value); the matrix V uses a zero matrix to initialize the parameters, that is, the initial values ​​of the parameters in the matrix V are all set to 0;

[0128] S304, inputting the data of the specific scene into an incremental detection module of the specific detection scene, and outputting a multimodal incremental vector for the specific detection scene;

[0129] S305, training the incremental detection module parameters according to the loss function in step E;

[0130] During fine-tuning, the parameters are optimized according to the composite loss function formed by the linear combination of the three loss functions to train the parameters of the incremental detection module;

[0131] S4. Superimpose the trained basic model parameters with the parameters of the incremental modules of each specific detection scenario fine-tuned by LoRA. When the model faces a specific complex scenario, it switches to the incremental recognition module for the specific complex scenario to improve the accuracy of the model in various complex detection scenarios in the power plant early warning task. At the same time, because the LoRA method uses the technology of low-rank decomposition, it only increases a small amount of model parameters while improving the model accuracy, realizing a target detection method adapted to multi-task scenarios.

[0132] Through the fine-tuning process, the trainable low-rank decomposition matrix is ​​injected into each layer of the Transformer architecture. At this time, the fine-tuned parameters of the incremental modules of each specific detection scenario are superimposed on the parameters of the deep large model. The multimodal deep large model obtains the incremental recognition information of specific detection scenarios in the power plant operating environment. These fine-tuning parameters enable the model to learn the subtle features and patterns at a specific time, so that the model can better adapt to the power plant early warning tasks in various detection scenarios.

[0133] S5. Input the test set into this round of multimodal deep large model, calculate the loss value through the composite loss function, if the loss value is lower than the set threshold, fix the model parameters and end the training; if the loss value is not lower than the set threshold, jump to step S2 to start a new round of training.

[0134] The composite loss function is the superposition of three losses: target detection loss, positioning loss, and classification loss.

[0135] The combined application of three loss functions enables the model to effectively identify and locate objects during the detection process. The target detection loss function is used to measure the network's ability to distinguish between the target and the background. This part uses a binary cross entropy loss function to evaluate whether the network can accurately predict whether the target exists or not; the positioning loss function is used to evaluate the network's positioning accuracy of the target position. It is calculated using a smooth L1 loss to measure the error between the network's prediction of the target bounding box coordinates and the actual target position. The classification loss function focuses on the network's classification accuracy of the target category; cross entropy loss is used to measure the consistency between the network's prediction of the target category and the actual target category. The total loss function is a linear combination of these three parts. The relative importance of each loss function is balanced by adjusting the weight parameters to better train the model and improve model performance;

[0136] During inference, the real-time operating data of the power plant is input into the trained multi-modal deep large model VTCL. Through the fine-grained modal extraction of the model and the detection of incremental modules for specific scenarios within the model, efficient early warning in the power plant operating environment can be achieved.

[0137] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0138] Although this article uses more terms such as model parameters, specific data, and composite loss functions, it does not exclude the possibility of using other terms. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation, based on a multi-modal deep large model, characterized in that: The multimodal deep large model includes a basic model and a specific detection scenario incremental module. The power plant early warning method includes a model training process and a reasoning process. The model training process is specifically as follows: S1. Summarize the historical operation data of the power plant to form a training data set and a test data set; both the training data set and the test data set contain general data and specific data; S2. Use the regular data in the training data set to train the basic model obtained in the previous round to obtain the basic model parameters of this round; S3. Use the specific data in the training data set to fine-tune the incremental module of the specific detection scenario obtained in the previous round using the LoRA method to obtain the incremental module parameters of the specific detection scenario in this round; S4, superimposing the basic model parameters of this round and the incremental module parameters of this round of specific detection scenarios to obtain a multimodal deep large model of this round; S5, input the test set into the current round of multimodal deep large model, calculate the loss value through the composite loss function, if the loss value is lower than the set threshold, fix the model parameters and end the training; if the loss value is not lower than the set threshold, jump to step S2 to start a new round of training; The reasoning process is: input the real-time operation data of the power plant into the trained multi-modal deep large model, and the output is the warning value.

2. According to claim 1, a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation is characterized in that: The step S2 is specifically as follows: S201, inputting the text content in the regular data in the training data set into BERT for vector encoding and feature extraction to obtain a text feature vector; S202, inputting the visual content in the regular data in the training data set into VIT for vector encoding and feature extraction to obtain a visual feature vector; S203, performing a fine-grained modal feature injection operation: injecting a text feature vector into a visual feature vector, and injecting a visual feature vector into a text feature vector; S204, fusing the text feature vector and the visual feature vector into which the fine-grained modal features have been injected, to obtain a multimodal global feature vector with the fine-grained modal features; S205. Use the multimodal global feature vector to train the multimodal deep large model obtained in the previous round. The loss function is a composite loss function to obtain the basic model parameters of this round.

3. According to claim 2, a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation is characterized in that: The step S201 is specifically as follows: The text content first passes through the word embedding layer of BERT and is represented as three parts: Token Embedding, Segment Embedding, and Position Embedding. The three parts are added bit by bit to output the word embedding vector, as shown in formula (1): V T =BERT TE (X T )+BERT SE (X T )+BERT PE (X T ) (1) Among them, X T Represents the input text content, BERT TE , BERT SE and BERT PE Respectively represent the Token Embedding, Segment Embedding and Position Embedding operations on the input text content; Embed the word into vector V T Enter the BERT network architecture: First, the global features of the text are extracted through the multi-head attention layer, GeLU is used as the activation function of the network, and then it goes through the normalization & residual link layer, and finally the text feature vector is output through the feedforward neural network layer. The whole process is shown in equations (2)-(5): V MA =MultiAttention(V T ) (2) Where MultiAttention in formula (2) represents the calculation of the multi-head attention mechanism, ⊕ in formula (3) represents the residual link, W1 and W2 in formula (4) represent weight parameters, b1 and b2 represent bias parameters, V TO A text feature vector representing the output of BERT.

4. According to claim 3, a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation is characterized in that: The step S202 is specifically as follows: The VIT model first processes the input representation of the visual content, and after Patch Embedding and Position Embedding operations, the two parts of the representation are fused as the input of the VIT model. The process is shown in formula (6): V I =VIT PAE (X I )+VIT PE (X I ) (6) Where X I Represents visual content input, VIT PAE Indicates Patch Embedding operation, VIT PE Represents PositionEmbedding operation; The visual vector V I Input VIT network, extract features through VIT network, and output visual feature vector V IO .

5. According to claim 4, a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation is characterized in that: In step S203, fine-grained modal feature injection is used to inject the text feature vector into the visual feature vector: Initialize several groups W Q , W K and W V The matrices represent the query matrix, key matrix and value matrix respectively. The query vector Q, key vector K and value vector V of the visual modality are generated through linear transformation. A set of linear transformation processes is shown in formula (7): In the formula and is a set of initialized modal parameter matrices. The i in the subscript refers to the head in the multi-head attention mechanism in the Transformer architecture, and I indicates that it is related to the visual feature vector; The fine-grained modal feature injection operation of the visual modality is shown in equations (8) to (10): △Z I =Concat(△Z 1I ,△Z 2I ,…△Z kI )W 0I (9) With VI =V IO +△Z I (10) In formula (8), CA I→T (V TO ,V IO ) is the cross-modal interaction process, which includes the linear transformation process in formula (7). is the scaling factor of numerical calculation, d is the dimension of query vector Q and key vector K, and Equation (9) is the process of fusing each group of cross-modal interaction vectors. k represents the kth group of cross-modal interaction process, W 0I is the visual parameter matrix, which is used to map the dimension of the cross-modal interaction fusion vector, and finally inject the text modality features into the visual modality to form the vector Z VI .

6. According to claim 5, a power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation is characterized in that: In step S203, fine-grained modal feature injection is used to inject the feature vector of the visual modality into the feature vector of the text modality: First, the query vector Q, key vector K and value vector V of the text modality are generated through linear transformation. A set of linear transformation processes is shown in formula (11): In the formula is a set of initialized modal parameter matrices; The fine-grained modal feature injection operation of the text modality is shown in equations (12) to (14): △Z T =Concat(△Z 1T ,△Z 2T ,…△Z kT )W 0T (13) With VT =V TO +△Z T (14) Where W 0T is the text parameter matrix, which is used to map the dimension of the cross-modal interaction fusion vector, and finally inject the visual modality features into the text modality to form the vector Z VT .

7. A power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation according to claim 6, characterized in that: Step S204 is specifically as follows: The text modality feature vector injected with fine-grained modality features is fused with the feature vector Z of the visual modality by vector concatenation, as shown in formula (15): Z=Concate[Z VI ;WITH VT ] (15) After a linear layer mapping, a multimodal global feature vector Z with fine-grained modal features is obtained. + , as shown in formula (16): Z + =ZW Z +b Z (16) Where W Z , b Z are weight parameters and bias parameters respectively.

8. A power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation according to claim 1 or 7, characterized in that: Step S3 is specifically as follows: S301, freezing the parameters of the basic model in the multimodal deep large model obtained in the previous round; S302, decomposing the parameter matrix in the frozen layer of the incremental detection module into two low-rank matrices. The low-rank decomposition process of the weight parameter matrix during LoRA fine-tuning is shown in formula (17): W=W0X+ΔWX=W0X+UVX (17) Where W0 represents the weight parameter in the frozen layer, ΔW represents the weight parameter of the incremental detection module for specific complex scene detection, ΔW = UV, W0 + ΔW = W0 + UV is used to represent the parameter update, U and V are low-rank decomposition matrices; S303, initializing parameters in the low-rank decomposition matrix; S304, inputting specific data into an incremental detection module of a specific detection scenario, and outputting a multimodal incremental vector for the specific detection scenario; S305. Use the multimodal incremental vector for the specific detection scenario to train the multimodal deep large model obtained in the previous round. The loss function is a composite loss function to obtain the incremental module parameters of the specific detection scenario in this round.

9. A power plant early warning method based on fine-grained modal information injection and multi-scenario task adaptation according to claim 8, characterized in that: The composite loss function is the superposition of three losses: target detection loss, positioning loss and classification loss.

Images