Multi-modal large model pre-training method and system based on query converter and power equipment fault diagnosis method

Abstract

The invention discloses a multi-modal large model pre-training method based on a query converter. The multi-modal large model pre-training method comprises the steps that pictures of a set scene are acquired, and a picture-text pair data set is generated; taking the learnable query as position information and replacing the learnable query with four-dimensional coordinates; correcting the structure of the query converter; carrying out high-dimensional space conversion on learnable queries and inputting the learnable queries into a query converter to carry out first-stage training on the multi-modal large model; the output of the query converter is combined with the text information and input into the frozen large language model to complete second-stage training of the multi-modal large model; and finishing the final pre-training of the multi-modal large model based on the query converter after finishing the first stage training and the second stage training. The invention also discloses a system for realizing the multi-modal large model pre-training method based on the query converter, and a power equipment fault diagnosis method comprising the multi-modal large model pre-training method based on the query converter. The method is higher in reliability, better in accuracy and less in resource occupation.

Description

Technical Field

[0001] The present invention belongs to the field of electrical automation, and in particular relates to a multi-modal large model pre-training method and system based on a query transformer, and a power equipment fault diagnosis method. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, electricity has become an indispensable secondary energy source in people's production and life, bringing endless convenience to people's production and life. Therefore, ensuring the stable and reliable supply of electricity has become one of the most important tasks of the power system.

[0003] At present, with the rapid development of intelligent technology, multimodal big models have also been widely used in power systems, such as power intelligent customer service. Power intelligent customer service based on multimodal models can understand and process customers' text, voice and even picture information, and provide more intelligent and personalized power customer service. In addition, multimodal big models can analyze equipment images and power grid operation data in real time, and combine historical fault information to provide more accurate power grid fault diagnosis and prediction. In terms of load forecasting, multimodal big models can process a large amount of historical load data, weather data, economic activity data, etc., predict future power demand through history and current situation, and help power grid operators better dispatch power resources.

[0004] At present, common multimodal large models use the trained visual encoder and text encoder as twin towers to build a bridge to make up for the modality gap by maximizing the similarity of image and text features. However, as the scale of visual models and language models increases, the end-to-end training cost of multimodal large models becomes increasingly high. Therefore, some researchers have proposed a multimodal large model pre-training strategy BLIP-2: BLIP-2 freezes the visual encoder and language model, constructs a query transformer (Q-Former), and completes the alignment of visual and language representations and the generation and learning of visually guided language through two-stage training.

[0005] The query transformer consists of two Transformer encoders. The first encoder takes a series of learnable queries as input. These learnable queries will interact with the visual features output by the image encoder after the self-attention layer, and finally pass through the feedforward neural network to obtain the final visual features. The other encoder is used to encode text. Depending on the input samples, the model will determine whether the learnable query and text embedding need to interact in the self-attention layer according to the specific situation: specifically, in the image-text comparison learning task that forces the alignment of visual information and semantic information, the two cannot interact; in the image-based text generation task that requires the learnable query to obtain visual information related to the text, only the text embedding can interact with the learnable query; and in the image-text matching task that further aligns the visual information and semantic information, the two can interact with each other. In the second stage of training, the two training tasks of using the frozen large language model for text generation based on the features output by the query transformer and completing the features output by the query transformer with some text features by the large language model finally achieve visually guided text generation.

[0006] However, the query transformer solution currently used still has millions of parameters and still requires a large amount of data sets for training to converge, which puts a great deal of resource pressure on the power system. Summary of the invention

[0007] One of the purposes of the present invention is to provide a multimodal large model pre-training method based on a query transformer with high reliability, good accuracy and less resource consumption.

[0008] A second objective of the present invention is to provide a system for implementing the multimodal large model pre-training method based on query transformer.

[0009] A third object of the present invention is to provide a method for diagnosing faults in electric power equipment including the multimodal large model pre-training method based on a query transformer.

[0010] The multimodal large model pre-training method based on query transformer provided by the present invention comprises the following steps:

[0011] S1. Obtain pictures under the set scene and generate a picture-text pair dataset;

[0012] S2. A query transformer based on BLIP-2 that takes the learnable query as location information and replaces the learnable query with four-dimensional coordinates;

[0013] S3. According to the learnable query represented by the four-dimensional coordinates obtained in step S2, the structure of the query transformer is modified to reduce redundant calculations in the query transformer;

[0014] S4. The learnable query represented by the four-dimensional coordinates obtained in step S2 is converted into a high-dimensional space and input into the query transformer, and the multimodal large model is trained in the first stage according to the task type;

[0015] S5. Based on the idea of ​​alignment first and then fusion, the output of the query transformer is combined with the text information and input into the frozen large language model to complete the second stage training of the multimodal large model;

[0016] S6. After completing the first stage training and the second stage training, the pre-training of the final query transformer-based multimodal large model is completed.

[0017] The step S1 of obtaining a picture under a set scene and generating a picture-text pair dataset specifically includes the following steps:

[0018] Collect image data under set scenes;

[0019] Perform data cleaning on the collected image data;

[0020] For the images after data cleaning, adjust the images to the set size and convert them to RGB format;

[0021] The NumPy library is used to calculate the pixel mean and variance of each image in the three channels of R, G, and B. For all images, the mean and variance of all images in the three channels of R, G, and B are calculated.

[0022] Use a pre-trained caption generation model to generate description text for each image;

[0023] Design a unique identifier for each image, name the image file with the corresponding unique identifier, and unify the image format;

[0024] Record the data information of each image and the corresponding text, and generate a JSON structure to complete the construction and reading of the image-text pair data set.

[0025] The BLIP-2-based query transformer described in step S2 takes the learnable query as position information and replaces the learnable query with four-dimensional coordinates, specifically including the following steps:

[0026] Perform size statistics on the pictures obtained in step S1 to obtain the average size of the pictures;

[0027] According to the average scale of the pictures, set a unified scale and convert all pictures to the size of the unified scale;

[0028] According to the size of the set uniform scale, the number of learnable queries n is set;

[0029] Divide the image into grids, and record the center coordinates, width and height of each grid area;

[0030] Use the normalized coordinate system to define the coordinate system for each grid;

[0031] Represent the center coordinates, width, and height of each grid region as learnable queries;

[0032] Normalize the obtained learnable query;

[0033] In the training phase, the four-dimensional coordinates that can be learned and queried are used as parameters to be optimized, and algorithms such as gradient descent are used for optimization and update; at the same time, the optimized coordinates are guaranteed to be within the set range;

[0034] Finally, the optimized learnable query is applied to the original image to extract features of the corresponding grid regions.

[0035] Step S3, according to the learnable query represented by the four-dimensional coordinates obtained in step S2, modifies the structure of the query transformer, specifically comprising the following steps:

[0036] The first self-attention layer in the first Transformer encoder inside the query transformer removes the interaction between the learnable query and text embeddings to complete the modification of the query transformer's structure.

[0037] The step S4 converts the learnable query represented by the four-dimensional coordinates obtained in step S2 into a high-dimensional space and inputs it into the query transformer, and performs the first-stage training of the multimodal large model according to the task type, which specifically includes the following steps:

[0038] Set the number of channels for multimodal large model processing tensors;

[0039] A feedforward neural network is used to perform nonlinear transformation on the learnable query represented by four-dimensional coordinates to ensure that the hidden dimension of the learnable query is the same as the number of channels of the set processing tensor;

[0040] The first stage of training of the multimodal large model is carried out according to the task type:

[0041] The task type is a picture-text comparison learning task:

[0042] Get the image features output by the query transformer and text features encoded by the query transformer Where B is the batch size, N is the number of learnable queries, and D is the embedding dimension;

[0043] The similarity sim from image to text is calculated using the following formula: i2t [i,j,k]:

[0044] sim i2t [i,j,k]=I i [k]·T j

[0045] Where I i [k] is the feature obtained by querying similar features on the features of the i-th image and aggregating them; T j is the jth text feature;

[0046] For each pair of image-text, take the maximum similarity S from image to text i2t [i,j] is

[0047] The similarity between text and image is calculated using the following formula: t2i [j,i,k]:

[0048] sim t2i [j,i,k]=T j I i [k]

[0049] For each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is

[0050] Calculate the image to text loss function loss i2t loss i2t =CrossEntropy(S i2t [i,j],y); where CrossEntropy() is the intermediate loss function and y is the target label;

[0051] Calculate the text to image loss function loss t2i Loss t2i =CrossEntropy(S t2i [j,i],y);

[0052] Calculate the final loss function loss ITC for

[0053] The task type is a picture-text matching task:

[0054] Image features encoded by the frozen visual encoder and text embedding Where B is the batch size, L is the sequence length, and D is the embedding dimension;

[0055] Calculate the similarity between image and text i2t[i,j,k] is sim i2t [i,j,k]=I i [k]·T j , for each pair of image-text, take the maximum similarity S from image to text i2t [i,j] is S i2t [i,j]=m k ax(sim i2t [i,j,k]);

[0056] Calculate the similarity between text and image t2i [j,i,k] is sim t2i [j,i,k]=T j I i [k], for each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is

[0057] The softmax function is used to convert the similarity matrix into a probability distribution, and the diagonal elements of the similarity matrix are modified to set extremely low values ​​to avoid sampling; it is expressed as

[0058] S i2t [i,i]=AA

[0059] S t2i [j,j]=AA

[0060]

[0061] Where AA is the lowest value set; weights i2t [i,j] is the probability that the feature of the i-th image is similar to the feature of the j-th text; weights t2i [j,i] is the probability of matching the j-th text feature and the i-th image feature;

[0062] Take the index corresponding to the maximum value from the probability distribution and get the negative sample image to text index matrix neg_idx i2t and negative sample text to image index neg_idx t2i ;

[0063] Freeze the image features I and text embedding T encoded by the visual encoder; after the text embedding and the learnable query are concatenated, they are input into the query transformer. The image features will be used as key-value pairs in the cross-attention module of the Transformer encoder in the query transformer, expressed as

[0064] output=QFromer(I,Concat(LearnedQueries,T))

[0065] Where output is the output of the learnable query after the query transformer; QFromer() is the transformation of the learnable query by the query transformer, and the query transformer consists of two Transformer encoders; Concat() is the concatenation operation; LearnedQueries is the randomly initialized learnable query;

[0066] By taking out the first L brother tensor in the second dimension, we get the final output output_feature;

[0067] The image-text matching score obtained through the classification head is score=Linear(output_feature), where Linear() is a linear change function;

[0068] Take the average of the image-text matching score dimension 1 to get avearge_score;

[0069] Calculate the final loss function loss ITM loss ITM =CrossEntropy(averge_score,y);

[0070] The task type is an image-based text generation task:

[0071] Given a query transformer output V is the vocabulary size; predicted_score[b,t,v] represents the probability of the vth word in the vocabulary at the tth time step of the bth sample; the text in the image-text pair is converted to a label Where labels[b,t] represents the target word index in the tth time step of the bth sample;

[0072] Calculate the final loss function loss IGT for

[0073] Finally, according to the calculated final loss function, the first stage of training of the multimodal large model is carried out.

[0074] The feedforward neural network specifically comprises the following steps:

[0075] The constructed feedforward neural network includes 2 or 3 basic layers;

[0076] The basic layer includes a linear transformation layer and a nonlinear activation function layer;

[0077] ReLU or GELU is used as the activation function of the nonlinear activation function layer.

[0078] Based on the idea of ​​alignment before fusion described in step S5, the output of the query transformer is combined with the text information and input into the frozen large language model to complete the second stage training of the multimodal large model, which specifically includes the following steps:

[0079] The output I of the query transformer is processed by a fully connected layer to obtain the image features in the text embedding space, thereby converting it to the text embedding space;

[0080] For text generation tasks:

[0081] Input the feature into the frozen large language model decoder and get the predicted output predicted_score as predicted_score=LLM decoder (feature); among them, LLM decoder () is the processing process of the frozen large language model decoder;

[0082] Convert the text in the image-text pair to labels;

[0083] Calculate the loss function loss as

[0084] For text completion tasks:

[0085] Embed the prefix text to get the vector representation of the text information text_embedding;

[0086] After connecting feature and text_embedding, we input them into the frozen large language model and get the predicted output predicted_score:

[0087] predicted_score=LLM(Concat(feature,text_embedding));

[0088] Convert the suffix text in the image-text pair to the label suffix_labels, where suffix_labels[b,t] represents the index of the target word in the t-th time step of the b-th sample;

[0089] Calculate the loss function loss as

[0090]

[0091] Finally, according to the calculated final loss function, the multimodal large model is trained in the second stage.

[0092] The present invention also provides a system for implementing the multimodal large model pre-training method based on a query transformer, comprising a data set construction module, a coordinate conversion module, a structure correction module, a first training module, a second training module and a pre-training module; the data set construction module, the coordinate conversion module, the structure correction module, the first training module, the second training module and the pre-training module are connected in series in sequence; the data set construction module is used to obtain pictures under a set scene, and generate a picture-text pair data set, and upload the data information to the coordinate conversion module; the coordinate conversion module is used to use the learnable query as position information based on the query transformer of BLIP-2 according to the received data information, and replace the learnable query with a four-dimensional coordinate, and upload the data information to the structure correction module; the structure correction module is used to obtain the learnable query represented by the obtained four-dimensional coordinate according to the received data information The query transformer is used to modify the structure of the query transformer to reduce redundant calculations in the query transformer, and upload the data information to the first training module; the first training module is used to perform high-dimensional space conversion on the learnable query represented by the obtained four-dimensional coordinates according to the received data information and input it into the query transformer, perform the first stage training of the multimodal large model according to the task type, and upload the data information to the second training module; the second training module is used to combine the output of the query transformer with the text information and input it into the frozen large language model according to the received data information based on the idea of ​​alignment and then fusion, complete the second stage training of the multimodal large model, and upload the data information to the pre-training module; the pre-training module is used to complete the final pre-training of the multimodal large model based on the query transformer after completing the first stage training and the second stage training according to the received data information.

[0093] The present invention also provides a method for diagnosing faults of electric power equipment including the multi-modal large model pre-training method based on query transformer, which specifically includes the following steps:

[0094] S1. Obtain multimodal data information of power equipment;

[0095] S2. Based on the multimodal data obtained in step S1, the multimodal large model pre-training method based on the query transformer is used to pre-train the multimodal large model to obtain a multimodal large model for fault diagnosis of power equipment;

[0096] S3. Use the large multi-modal model for fault diagnosis of power equipment obtained in step S2 to perform real-time fault diagnosis of power equipment:

[0097] S4. Provide decision support based on diagnostic results;

[0098] S5. Optimize and update the large multi-modal model of power equipment fault diagnosis.

[0099] The query transformer-based multimodal large model pre-training method, system and power equipment fault diagnosis method provided by the present invention accelerate the convergence of learnable queries to sparse areas on the image by taking the learnable query as position information and using four-dimensional coordinates for identification; improve the computational efficiency of the multimodal large model by modifying the structure of the query transformer; finally, the alignment of multimodal information and the text generation task guided by visual information are completed through a two-stage training method; therefore, the present invention can not only complete the pre-training of the multimodal large model and the corresponding power equipment fault diagnosis, but also has higher reliability, better accuracy and less resource occupation. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 Schematic diagram of the method flow of the pre-training method of the present invention.

[0101] Figure 2 Schematic diagram of the functional modules of the system of the present invention.

[0102] Figure 3 The figure is a flow chart of the method for diagnosing faults of electric power equipment according to the present invention. DETAILED DESCRIPTION

[0103] like Figure 1 The figure shows a method flow chart of the pre-training method of the present invention: The multi-modal large model pre-training method based on the query transformer disclosed in the present invention comprises the following steps:

[0104] S1. Obtain images in a set scenario and generate an image-text pair dataset; specifically, the steps are as follows:

[0105] Collect image data in a set scenario; for example, if it is used in a power transmission inspection scenario, use images in the power transmission inspection scenario; if it is used in an equipment fault diagnosis scenario, collect equipment fault images and normal operation images

[0106] Clean the collected image data; review the images and discard those with unclear scenes, unobtrusive features, or very large feature repetitions; for unclear images, they can be discarded or image denoising technology based on deep learning can be used to improve image clarity;

[0107] For the images after data cleaning, adjust the images to the set size and convert them to RGB format;

[0108] The NumPy library is used to calculate the pixel mean and variance of each image in the three channels of R, G, and B. For all images, the mean and variance of all images in the three channels of R, G, and B are calculated.

[0109] Use pre-trained caption generation models, such as BLIP-2, to generate description text for each image; for specific domains, such as power grid failures, fine-tune the caption generation model to improve the accuracy and professionalism of the description; review the generated description text to verify the grammatical correctness and domain relevance of the generated text;

[0110] Design a unique identifier for each image, name the image file with the corresponding unique identifier, and unify the image format;

[0111] Record the data information of each image and the corresponding text, and generate a JSON structure to complete the construction and reading of the image-text pair data set;

[0112] S2. A query transformer based on BLIP-2 takes the learnable query as the location information and replaces the learnable query with a four-dimensional coordinate; specifically, the steps include:

[0113] Perform size statistics on the pictures obtained in step S1 to obtain the average size of the pictures;

[0114] Count the image sizes in the dataset to get the average size of the images; set a uniform scale based on the average scale of the images, and convert all images to a uniform size; for example, if the average image size is between 224x224 and 448x448, then convert all images to a size of 224x224; to get the image size, you can use Python's image processing library (such as PIL or OpenCV) to get the image size information;

[0115] According to the size of the set uniform scale, the number of learnable queries n is set; in specific implementation, the following strategy can be adopted: for small images (such as 224x224 pixels), 32 learnable queries are set; for medium-sized images (such as 448x448 pixels), 64 learnable queries are set; for large images (such as 896x896 pixels or larger), 128 learnable queries are set; this strategy ensures that the number of learnable queries matches the complexity of the image;

[0116] Considering that learnable queries converge to sparse specific image regions very slowly, we can directly divide the image into regions according to the number of learnable queries and assign each region to a learnable query. grids, and record the center coordinates, width and height of each grid area; in specific implementation, the image areas are numbered from 0 to n; then a dictionary can be used to record the center coordinates, width and height of each image area;

[0117] Use a normalized coordinate system to define the coordinate system of each grid; for example, the upper left corner of the grid can be defined as (0,0) and the lower right corner can be defined as (1,1), which can adapt to images of different sizes;

[0118] Represent the center coordinates, width, and height of each grid region as learnable queries;

[0119] Normalize the learned query. Specifically, normalize the center coordinates and the region width and height by dividing them by the width and height of the image.

[0120] In the training phase, the four-dimensional coordinates of the learnable query are used as the parameters to be optimized, and the gradient descent algorithm is used to optimize and update them so that they can better capture the key information in the image; at the same time, the optimized coordinates are guaranteed to be within the set range, and the width and height are not 0. This can be achieved by using the sigmoid function or cropping operation;

[0121] Finally, the optimized learnable query is applied to the original image to extract the features of the corresponding grid area. This process can be seen as a compression of the original image information, converting the high-dimensional image data into a low-dimensional query representation.

[0122] S3. According to the learnable query represented by the four-dimensional coordinates obtained in step S2, the structure of the query transformer is modified to reduce redundant calculations in the query transformer; specifically comprising the following steps:

[0123] The query transformer’s structure is modified by removing the learnable query and text embedding interactions in the first self-attention layer of the first transformer encoder inside the query transformer.

[0124] The main reason for this correction is that the learnable query at this layer only contains position information, and the interaction with the text embedding has no obvious meaning at this time; by the second self-attention layer, the learnable query has extracted image features with the same position information. At this time, interacting with the text embedding in the self-attention layer can introduce text information related to the image features in the learnable query, so as to further extract image features containing similar position information, image features and text embedding as the current learnable query in the cross-attention module; therefore, the interaction between the learnable query and the text embedding in the first self-attention layer of the first Transformer encoder is redundant;

[0125] S4. The learnable query represented by the four-dimensional coordinates obtained in step S2 is converted into a high-dimensional space and input into the query transformer, and the multimodal large model is trained in the first stage according to the task type; specifically, the steps include:

[0126] Set the number of channels for processing tensors in a multimodal large model; common choices include 256, 512, or 1024; considering the need for lightweight, a relatively small number of channels, such as 256, can be selected;

[0127] A feedforward neural network is used to perform nonlinear transformation on the learnable query represented by four-dimensional coordinates to ensure that the hidden dimension of the learnable query is the same as the number of channels of the set processing tensor;

[0128] In specific implementation, the feedforward neural network comprises the following steps:

[0129] The constructed feedforward neural network includes 2 or 3 basic layers;

[0130] The basic layer includes a linear transformation layer and a nonlinear activation function layer;

[0131] Use ReLU or GELU as the activation function of the nonlinear activation function layer to enhance the network's expressive power;

[0132] The first stage of training of the multimodal large model is carried out according to the task type:

[0133] The task type is a text-image comparison learning task: a unimodal mask matrix is ​​used to control the interaction between the learnable query and the text embedding. At this time, the learnable query and the text embedding are invisible to each other. When encoding the text embedding in the query transformer, a special [CLS] embedding is added at the beginning of the sequence. Finally, the output of the learnable query is similar to the output of this [CLS] embedding, and the maximum value is selected as the similarity of the image-text pair.

[0134] Get the image features output by the query transformer and text features encoded by the query transformer Where B is the batch size, N is the number of learnable queries, and D is the embedding dimension;

[0135] The similarity sim from image to text is calculated using the following formula: i2t [i,j,k]:

[0136] sim i2t [i,j,k]=I i [k]·T j

[0137] Where I i [k] is the feature obtained by querying similar features on the features of the i-th image and aggregating them; T j is the jth text feature;

[0138] For each pair of image-text, take the maximum similarity S from image to text i2t[i,j] is

[0139] The similarity between text and image is calculated using the following formula: t2i [j,i,k]:

[0140] sim t2i [j,i,k]=T j I i [k]

[0141] For each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is

[0142] Calculate the image to text loss function loss i2t Loss i2t =CrossEntropy(S i2t [i,j],y); where CrossEntropy() is the intermediate loss function and y is the target label;

[0143] Calculate the text to image loss function loss t2i Loss t2i =CrossEntropy(S t2i [j,i],y);

[0144] Calculate the final loss function loss ITC for

[0145] The task type is a graph-text matching task: a bidirectional self-attention mask matrix is ​​used to control the interaction between the learnable query and the text embedding. At this time, the learnable query and the text embedding are mutually visible, so the learnable query can capture multimodal information; the final output of the learnable query will be passed through a two-class linear classifier to obtain a graph-text matching score; considering the number of learnable queries, the average of the graph-text matching scores of all learning queries is used as the final graph-text matching score; in addition, a sample expansion strategy is used in the graph-text matching task to construct positive and negative samples;

[0146] Image features encoded by the frozen visual encoder and text embedding Where B is the batch size, L is the sequence length, and D is the embedding dimension;

[0147] Calculate the similarity between image and text i2t [i,j,k] is sim i2t [i,j,k]=I i [k]·T j, for each pair of image-text, take the maximum similarity S from image to text i2t [i,j] is S i2t [i,j]=m k ax(sim i2t [i,j,k]);

[0148] Calculate the similarity between text and image t2i [j,i,k] is sim t2i [j,i,k]=T j I i [k], for each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is S t2i [j,i]=m k ax(sim t2i [j,i,k]);

[0149] The softmax function is used to convert the similarity matrix into a probability distribution, and the diagonal elements of the similarity matrix are modified to set extremely low values ​​to avoid sampling; it is expressed as

[0150] S i2t [i,i]=AA

[0151] S t2i [j,j]=AA

[0152]

[0153] Where AA is the lowest value set; weights i2t [i,j] is the probability that the feature of the i-th image is similar to the feature of the j-th text; weights t2i [j,i] is the probability of matching the j-th text feature and the i-th image feature;

[0154] Take the index corresponding to the maximum value from the probability distribution and get the negative sample image to text index matrix neg_idx i2t and negative sample text to image index neg_idx t2i ;

[0155] Freeze the image features I and text embedding T encoded by the visual encoder; after the text embedding and the learnable query are concatenated, they are input into the query transformer. The image features will be used as key-value pairs in the cross-attention module of the Transformer encoder in the query transformer, expressed as

[0156] output=QFromer(I,Concat(LearnedQueries,T))

[0157] Where output is the output of the learnable query after the query transformer; QFromer() is the transformation of the learnable query by the query transformer, and the query transformer consists of two Transformer encoders; Concat() is the concatenation operation; LearnedQueries is the randomly initialized learnable query;

[0158] By taking out the first L brother tensor in the second dimension, we get the final output output_feature;

[0159] The image-text matching score obtained through the classification head is score=Linear(output_feature), where Linear() is a linear change function;

[0160] Take the average of the image-text matching score dimension 1 to get avearge_score;

[0161] Calculate the final loss function loss ITM loss ITM =CrossEntropy(averge_score,y);

[0162] The task type is an image-based text generation task: a multimodal causal mask matrix is ​​used to control the interaction between the learnable query and the text embedding; in the self-attention layer, only the text embedding can interact with the learnable query, while the learnable query cannot interact with the text embedding; the purpose of this is that the information required for text generation can be extracted through the learnable query and then passed to the text embedding through self-attention interaction; therefore, the image-based text generation task forces the model to extract visual features related to the text; in addition, for this task, the [CLS] embedding in the query transformer is replaced with a special [DEC] embedding to mark the decoding task; the multimodal causal mask matrix also controls the interaction between the text embeddings in the sequence, specifically, the current text embedding can interact with the previous text embedding but not with the subsequent text embedding; at the end of the text encoder, a linear layer "nn.linear(hidden_size,vocab_size)" is added to act as an encoder to complete the text generation;

[0163] Given a query transformer output V is the vocabulary size; predicted_score[b,t,v] represents the probability of the vth word in the vocabulary at the tth time step of the bth sample; the text in the image-text pair is converted to a label Where labels[b,t] represents the target word index in the tth time step of the bth sample;

[0164] Calculate the final loss function loss IGT for

[0165] Finally, according to the calculated final loss function, the first stage of training of the multimodal large model is carried out;

[0166] S5. Based on the idea of ​​alignment first and then fusion, the output of the query transformer is combined with the text information and input into the frozen large language model to complete the second stage training of the multimodal large model; specifically, the steps are as follows:

[0167] The output I of the query transformer is processed by a fully connected layer to obtain the image features in the text embedding space, and then converted to the text embedding space; specifically, a linear layer is created through "nn.Linear(hidden_dim,embed_dim)" to achieve the dimension conversion of the query transformer output;

[0168] For text generation tasks:

[0169] Input the feature into the frozen large language model decoder and get the predicted output predicted_score as predicted_score=LLM decoder (feature); among them, LLM decoder () is the processing process of the frozen large language model decoder;

[0170] Convert the text in the image-text pair to labels;

[0171] Calculate the loss function loss as

[0172] For text completion tasks: the model will divide the text in each image-text pair into two parts: prefix and suffix, and train the model's ability to generate suffix text based on visual information and prefix text;

[0173] Embed the prefix text to get the vector representation of the text information text_embedding;

[0174] After connecting feature and text_embedding, we input them into the frozen large language model and get the predicted output predicted_score:

[0175] predicted_score=LLM(Concat(feature,text_embedding));

[0176] Convert the suffix text in the image-text pair to the label suffix_labels, where suffix_labels[b,t] represents the index of the target word in the t-th time step of the b-th sample;

[0177] Calculate the loss function loss as

[0178]

[0179] Finally, according to the calculated final loss function, the multimodal large model is trained in the second stage;

[0180] S6. After completing the first stage training and the second stage training, the pre-training of the final query transformer-based multimodal large model is completed.

[0181] like Figure 2 The functional module schematic diagram of the system of the present invention is shown as follows: the system disclosed in the present invention for implementing the multimodal large model pre-training method based on query transformer comprises a data set construction module, a coordinate conversion module, a structure correction module, a first training module, a second training module and a pre-training module; the data set construction module, the coordinate conversion module, the structure correction module, the first training module, the second training module and the pre-training module are connected in series in sequence; the data set construction module is used to obtain pictures under a set scene, generate a picture-text pair data set, and upload the data information to the coordinate conversion module; the coordinate conversion module is used to use the learnable query as position information based on the query converter of BLIP-2 according to the received data information, and replace the learnable query with a four-dimensional coordinate, and upload the data information to the structure correction module; the structure correction module is used to obtain the picture under a set scene according to the obtained four-dimensional coordinate, and upload the data information to the structure correction module according to the received data information. The learnable query represented by the label is corrected to reduce redundant calculations in the query transformer, and the data information is uploaded to the first training module; the first training module is used to perform high-dimensional space conversion on the learnable query represented by the obtained four-dimensional coordinates according to the received data information and input it into the query transformer, perform the first stage training of the multimodal large model according to the task type, and upload the data information to the second training module; the second training module is used to combine the output of the query transformer with the text information and input it into the frozen large language model according to the received data information based on the idea of ​​alignment and then fusion, complete the second stage training of the multimodal large model, and upload the data information to the pre-training module; the pre-training module is used to complete the final pre-training of the multimodal large model based on the query transformer after completing the first stage training and the second stage training according to the received data information.

[0182] like Figure 3The present invention also provides a method for diagnosing a fault in an electric power device including the multi-modal large model pre-training method based on a query transformer, which specifically includes the following steps:

[0183] S1. Obtain multimodal data information of power equipment;

[0184] The multimodal data information includes: collecting image data of power equipment, including equipment appearance images, infrared thermal imaging images, etc.; collating text data such as equipment historical maintenance records, fault reports, etc.;

[0185] S2. Based on the multimodal data obtained in step S1, the multimodal large model pre-training method based on the query transformer is used to pre-train the multimodal large model to obtain a multimodal large model for fault diagnosis of power equipment;

[0186] In specific implementation, we use image and text data as input to train the model to understand the relationship between different modalities; through the query transformer mechanism, we enhance the model's ability to understand and integrate multimodal data;

[0187] S3. Use the large multi-modal model for fault diagnosis of power equipment obtained in step S2 to perform real-time fault diagnosis of power equipment:

[0188] In specific implementation, multimodal data of power equipment is collected in real time; the collected data is input into a trained multimodal large model; the model analyzes the multimodal data and identifies potential failure modes; and the fault diagnosis results are output, including the fault type, fault severity and recommended maintenance measures;

[0189] S4. Provide decision support based on diagnostic results;

[0190] In specific implementation, equipment maintenance suggestions are automatically generated based on fault diagnosis results; for emergency faults, the alarm system is triggered and relevant personnel are notified; the diagnosis results are stored in the database for subsequent model updates and optimizations;

[0191] S5. Continuous optimization and update:

[0192] During the specific implementation, actual maintenance feedback is collected regularly to verify the accuracy of model diagnosis; new fault case data is used to continuously fine-tune and update the model; and the model's adaptability to new equipment and newly emerging fault types is optimized.

Claims

1. A multimodal large model pre-training method based on query transformer, comprising the following steps: S1. Obtain pictures under the set scene and generate a picture-text pair dataset; S2. A query transformer based on BLIP-2 that takes the learnable query as location information and replaces the learnable query with four-dimensional coordinates; S3. According to the learnable query represented by the four-dimensional coordinates obtained in step S2, the structure of the query transformer is modified to reduce redundant calculations in the query transformer; S4. The learnable query represented by the four-dimensional coordinates obtained in step S2 is converted into a high-dimensional space and input into the query transformer, and the multimodal large model is trained in the first stage according to the task type; S5. Based on the idea of ​​alignment first and then fusion, the output of the query transformer is combined with the text information and input into the frozen large language model to complete the second stage training of the multimodal large model; S6. After completing the first stage training and the second stage training, the pre-training of the final query transformer-based multimodal large model is completed.

2. The multimodal large model pre-training method based on query transformer according to claim 1 is characterized in that The step S1 of obtaining a picture under a set scene and generating a picture-text pair dataset specifically includes the following steps: Collect image data under set scenes; Perform data cleaning on the collected image data; For the images after data cleaning, adjust the images to the set size and convert them to RGB format; The NumPy library is used to calculate the pixel mean and variance of each image in the three channels of R, G, and B. For all images, the mean and variance of all images in the three channels of R, G, and B are calculated. Use a pre-trained caption generation model to generate description text for each image; Design a unique identifier for each image, name the image file with the corresponding unique identifier, and unify the image format; Record the data information of each image and the corresponding text, and generate a JSON structure to complete the construction and reading of the image-text pair data set.

3. The multimodal large model pre-training method based on query transformer according to claim 2 is characterized in that The BLIP-2-based query transformer described in step S2 takes the learnable query as position information and replaces the learnable query with four-dimensional coordinates, specifically including the following steps: Perform size statistics on the pictures obtained in step S1 to obtain the average size of the pictures; According to the average scale of the pictures, set a unified scale and convert all pictures to the size of the unified scale; According to the size of the set uniform scale, the number of learnable queries n is set; Divide the image into grids, and record the center coordinates, width and height of each grid area; Use the normalized coordinate system to define the coordinate system for each grid; Represent the center coordinates, width, and height of each grid region as learnable queries; Normalize the obtained learnable query; In the training phase, the four-dimensional coordinates that can be learned and queried are used as parameters to be optimized, and algorithms such as gradient descent are used for optimization and update; at the same time, the optimized coordinates are guaranteed to be within the set range; Finally, the optimized learnable query is applied to the original image to extract features of the corresponding grid regions.

4. The multimodal large model pre-training method based on query transformer according to claim 3 is characterized in that Step S3, according to the learnable query represented by the four-dimensional coordinates obtained in step S2, modifies the structure of the query transformer, specifically comprising the following steps: The first self-attention layer in the first Transformer encoder inside the query transformer removes the interaction between the learnable query and text embeddings to complete the modification of the query transformer's structure.

5. The multimodal large model pre-training method based on query transformer according to claim 4 is characterized in that The step S4 converts the learnable query represented by the four-dimensional coordinates obtained in step S2 into a high-dimensional space and inputs it into the query transformer, and performs the first-stage training of the multimodal large model according to the task type, which specifically includes the following steps: Set the number of channels for multimodal large model processing tensors; A feedforward neural network is used to perform nonlinear transformation on the learnable query represented by four-dimensional coordinates to ensure that the hidden dimension of the learnable query is the same as the number of channels of the set processing tensor; The first stage of training of the multimodal large model is carried out according to the task type: The task type is a picture-text comparison learning task: Get the image features output by the query transformer and text features encoded by the query transformer Where B is the batch size, N is the number of learnable queries, and D is the embedding dimension; The similarity sim from image to text is calculated using the following formula: i2t [i,j,k]: sim i2t [i,j,k]=I i [k]·T j Where I i [k] is the feature obtained by querying similar features and aggregating the features of the i-th image by the k-th learnable query; T j is the jth text feature; For each pair of image-text, take the maximum similarity S from image to text i2t [i,j] is The similarity between text and image is calculated using the following formula: t2i [j,i,k]: sim t2i [j,i,k]=T j ·I i [k] For each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is Calculate the image to text loss function loss i2t Loss i2t =CrossEntropy(S i2t [i,j],y); where CrossEntropy() is the intermediate loss function and y is the target label; Calculate the text to image loss function loss t2i Loss t2i =CrossEntropy(S t2i [j,i],y); Calculate the final loss function loss ITC for The task type is a picture-text matching task: Image features encoded by the frozen visual encoder and text embedding Where B is the batch size, L is the sequence length, and D is the embedding dimension; Calculate the similarity between image and text i2t [i,j,k] is sim i2t [i,j,k]=I i [k]·T j , for each pair of image-text, take the maximum similarity S from image to text i2t [i,j] is Calculate the similarity between text and image t2i [j,i,k] is sim t2i [j,i,k]=T j I i [k], for each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is The softmax function is used to convert the similarity matrix into a probability distribution, and the diagonal elements of the similarity matrix are modified to set extremely low values ​​to avoid sampling; it is expressed as S i2t [i,i]=AA S t2i [j,j]=AA Where AA is the lowest value set; weights i2t [i,j] is the probability that the feature of the i-th image is similar to the feature of the j-th text; weights t2i [j,i] is the probability of matching the j-th text feature and the i-th image feature; Take the index corresponding to the maximum value from the probability distribution and get the negative sample image to text index matrix neg_idx i2t and negative sample text to image index neg_idx t2i ; Freeze the image features I and text embedding T encoded by the visual encoder; after the text embedding and the learnable query are concatenated, they are input into the query transformer. The image features will be used as key-value pairs in the cross-attention module of the Transformer encoder in the query transformer, expressed as output=QFromer(I,Concat(LearnedQueries,T)) Where output is the output of the learnable query after the query transformer; QFromer() is the transformation of the learnable query by the query transformer, and the query transformer consists of two Transformer encoders; Concat() is the concatenation operation; LearnedQueries is the randomly initialized learnable query; By taking out the first L brother tensor in the second dimension, we get the final output output_feature; The image-text matching score obtained through the classification head is score=Linear(output_feature), where Linear() is a linear change function; Take the average of the image-text matching score dimension 1 to get avearge_score; Calculate the final loss function loss ITM Loss ITM =CrossEntropy(averge_score,y); The task type is an image-based text generation task: Given a query transformer output V is the vocabulary size; predicted_score[b,t,v] represents the probability of the vth word in the vocabulary at the tth time step of the bth sample; the text in the image-text pair is converted to a label Where labels[b,t] represents the target word index in the tth time step of the bth sample; Calculate the final loss function loss IGT for Finally, according to the calculated final loss function, the first stage of training of the multimodal large model is carried out.

6. The multimodal large model pre-training method based on query transformer according to claim 5 is characterized in that The feedforward neural network specifically comprises the following steps: The constructed feedforward neural network includes 2 or 3 basic layers; The basic layer includes a linear transformation layer and a nonlinear activation function layer; ReLU or GELU is used as the activation function of the nonlinear activation function layer.

7. The multimodal large model pre-training method based on query transformer according to claim 6 is characterized in that Based on the idea of ​​alignment before fusion described in step S5, the output of the query transformer is combined with the text information and input into the frozen large language model to complete the second stage training of the multimodal large model, which specifically includes the following steps: The output I of the query transformer is processed by a fully connected layer to obtain the image features in the text embedding space, thereby converting it to the text embedding space; For text generation tasks: Input the feature into the frozen large language model decoder and get the predicted output predicted_score as predicted_score=LLM decoder (feature); among them, LLM decoder () is the processing process of the frozen large language model decoder; Convert the text in the image-text pair to labels; Calculate the loss function loss as For text completion tasks: Embed the prefix text to get the vector representation of the text information text_embedding; After connecting feature and text_embedding, we input them into the frozen large language model and get the predicted output predicted_score: predicted_score=LLM(Concat(feature,text_embedding)); Convert the suffix text in the image-text pair to the label suffix_labels, where suffix_labels[b,t] represents the index of the target word in the t-th time step of the b-th sample; Calculate the loss function loss as Finally, according to the calculated final loss function, the multimodal large model is trained in the second stage.

8. A system for implementing the multimodal large model pre-training method based on query transformer according to any one of claims 1 to 7, characterized in that It includes a data set construction module, a coordinate conversion module, a structure correction module, a first training module, a second training module and a pre-training module; the data set construction module, the coordinate conversion module, the structure correction module, the first training module, the second training module and the pre-training module are connected in series in sequence; the data set construction module is used to obtain pictures under the set scene, generate a picture-text pair data set, and upload the data information to the coordinate conversion module; The coordinate conversion module is used to replace the learnable query with four-dimensional coordinates based on the query transformer of BLIP-2 according to the received data information, and upload the data information to the structure correction module; The structure correction module is used to correct the structure of the query transformer according to the received data information and the learnable query represented by the obtained four-dimensional coordinates, so as to reduce redundant calculations in the query transformer, and upload the data information to the first training module; The first training module is used to perform high-dimensional space conversion on the learnable query represented by the obtained four-dimensional coordinates according to the received data information and input it into the query transformer, perform the first stage training on the multimodal large model according to the task type, and upload the data information to the second training module; The second training module is used to combine the output of the query transformer with the text information and input it into the frozen large language model based on the idea of ​​aligning first and then fusing according to the received data information, so as to complete the second stage training of the multimodal large model and upload the data information to the pre-training module; The pre-training module is used to complete the pre-training of the final multimodal large model based on the query transformer after completing the first stage training and the second stage training according to the received data information.

9. A method for diagnosing faults in electric power equipment comprising the multi-modal large model pre-training method based on query transformer according to any one of claims 1 to 8, specifically comprising the following steps: S1. Obtain multimodal data information of power equipment; S2. Based on the multimodal data obtained in step S1, the multimodal large model pre-training method based on the query transformer is used to pre-train the multimodal large model to obtain a multimodal large model for fault diagnosis of power equipment; S3. Using the large multi-modal model for fault diagnosis of power equipment obtained in step S2, real-time fault diagnosis of power equipment is performed; S4. Provide decision support based on diagnostic results; S5. Optimize and update the large multi-modal model of power equipment fault diagnosis.

Images