Query transformer-based multimodal large model pre-training method and system, and power equipment fault diagnosis method

Abstract

The application discloses a multimodal large model pre-training method based on a query transformer, comprising the following steps: acquiring pictures of a set scene and generating a picture-text pair dataset; taking a learnable query as position information and replacing it with a four-dimensional coordinate; correcting the query transformer structure; performing high-dimensional space conversion on the learnable query and inputting the learnable query into the query transformer to perform first-stage training on a multimodal large model; combining the query transformer output with text information and inputting the query transformer output into a frozen large language model to complete second-stage training on the multimodal large model; and completing the final pre-training of the multimodal large model based on the query transformer after completing the first-stage and second-stage training. The application further discloses a system for implementing the multimodal large model pre-training method based on the query transformer, and a power equipment fault diagnosis method comprising the multimodal large model pre-training method based on the query transformer. The application has higher reliability, better accuracy and less resource occupation.

Description

Technical Field

[0001] The present invention belongs to the field of electrical automation, and in particular relates to a multimodal large model pre-training method and system based on a query converter, and a power equipment fault diagnosis method. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, electricity has become an indispensable secondary energy source in people's production and life, bringing endless convenience to people's production and life. Therefore, ensuring a stable and reliable supply of electricity has become one of the most important tasks of the power system.

[0003] Currently, with the rapid development of intelligent technology, multimodal large models have found widespread application in power systems. For example, intelligent power customer service, based on multimodal models, can understand and process text, voice, and even image information from customers, providing more intelligent and personalized power customer service. Furthermore, multimodal large models can analyze device images and grid operation data in real time, while combining historical fault information to provide more accurate grid fault diagnosis and prediction. In terms of load forecasting, multimodal large models can process large amounts of historical load data, weather data, economic activity data, and more. Using historical and current data, they can predict future power demand, helping grid operators better dispatch power resources.

[0004] Currently, common large multimodal models use pre-trained visual encoders and text encoders as twin towers, bridging the modality gap by maximizing the similarity between image and text features. However, as the scale of visual and language models increases, the cost of training large end-to-end multimodal models becomes increasingly high. Therefore, researchers have proposed BLIP-2, a pre-training strategy for large multimodal models: BLIP-2 freezes the visual encoder and language model, constructs a query transformer (Q-Former), and completes the alignment of visual and language representations and the generation of visually guided language through two-stage training.

[0005] The query transformer consists of two Transformer encoders. The first encoder inputs a series of learnable queries, which, after passing through a self-attention layer, interact with the visual features output by the image encoder. Finally, the final visual features are obtained through a feed-forward neural network. The second encoder encodes the text. Depending on the input sample, the model determines whether the learnable query and text embedding need to interact in the self-attention layer. Specifically, in image-text comparison learning tasks that force alignment of visual and semantic information, the two cannot interact. In image-based text generation tasks that require the learnable query to obtain visual information related to the text, only the text embedding can interact with the learnable query. However, in image-text matching tasks that further align visual and semantic information, the two can interact. In the second stage of training, two training tasks, using a frozen large language model for text generation based on the query transformer output features and completing the query transformer output features with partial text features using the large language model, ultimately achieve visually guided text generation.

[0006] However, the query transformer solutions currently used still have millions of parameters and require large datasets for training to converge, which puts a significant resource strain on the power system. Summary of the Invention

[0007] One of the objectives of the present invention is to provide a multimodal large model pre-training method based on a query transformer with high reliability, good accuracy and low resource consumption.

[0008] A second object of the present invention is to provide a system for implementing the multimodal large model pre-training method based on query transformer.

[0009] A third object of the present invention is to provide a method for diagnosing faults in electric power equipment that includes the multimodal large model pre-training method based on a query transformer.

[0010] The multimodal large model pre-training method based on the query transformer provided by the present invention comprises the following steps:

[0011] S1. Obtain images in a given scenario and generate an image-text pair dataset;

[0012] S2. A query transformer based on BLIP-2 that treats learnable queries as location information and replaces the learnable queries with 4D coordinates;

[0013] S3. According to the learnable query represented by the four-dimensional coordinate obtained in step S2, the structure of the query transformer is modified to reduce redundant calculations in the query transformer;

[0014] S4. Performing a high-dimensional space transformation on the learnable query represented by the four-dimensional coordinates obtained in step S2 and inputting the result into the query transformer, and performing the first phase of training on the multimodal large model according to the task type;

[0015] S5. Based on the idea of ​​alignment followed by fusion, the query transformer output is combined with the text information and input into the frozen large language model, completing the second stage of training for the multimodal large model.

[0016] S6. After completing the first stage training and the second stage training, the final pre-training of the query transformer-based multimodal large model is completed.

[0017] The step S1 of obtaining an image in a set scene and generating an image-text pair dataset specifically includes the following steps:

[0018] Collect image data under the set scene;

[0019] Perform data cleaning on the collected image data;

[0020] For the cleaned images, resize them to the set size and convert them to RGB format;

[0021] The NumPy library is used to calculate the pixel mean and variance of each image in the R, G, and B channels; for all images, the mean and variance of all images in the R, G, and B channels are calculated;

[0022] Use a pre-trained caption generation model to generate description text for each image;

[0023] Design a unique identifier for each image, name the image file according to the corresponding unique identifier, and unify the image format;

[0024] Record the data information of each image and the corresponding text, and generate a JSON structure to complete the construction and reading of the image-text pair dataset.

[0025] The BLIP-2-based query transformer described in step S2 takes the learnable query as position information and replaces the learnable query with four-dimensional coordinates, specifically including the following steps:

[0026] Perform size statistics on the pictures obtained in step S1 to obtain the average size of the pictures;

[0027] According to the average scale of the pictures, set a unified scale and convert all pictures to the size of the unified scale;

[0028] According to the size of the set uniform scale, set the number of learnable queries n;

[0029] Divide the image into grids, and record the center coordinates, width and height of each grid area;

[0030] Use the normalized coordinate system to define the coordinate system of each grid;

[0031] Represent the center coordinates, width, and height of each grid region as a learnable query;

[0032] Normalize the obtained learnable query;

[0033] During the training phase, the four-dimensional coordinates of the learnable query are used as the parameters to be optimized, and algorithms such as gradient descent are used for optimization and update. At the same time, the optimized coordinates are guaranteed to be within the set range.

[0034] Finally, the optimized learnable query is applied to the original image to extract features corresponding to the grid regions.

[0035] Step S3, based on the learnable query represented by the four-dimensional coordinates obtained in step S2, modifies the structure of the query transformer, specifically comprising the following steps:

[0036] The first self-attention layer in the first Transformer encoder inside the query transformer removes the interaction between the learnable query and text embedding to complete the structural revision of the query transformer.

[0037] Step S4 converts the learnable query represented by the four-dimensional coordinates obtained in step S2 into a high-dimensional space and inputs it into the query transformer. The first stage of training the multimodal large model is performed according to the task type, which specifically includes the following steps:

[0038] Set the number of channels for processing tensors in a multimodal large model;

[0039] A feedforward neural network is used to perform a nonlinear transformation on the learnable query represented by the four-dimensional coordinates to ensure that the hidden dimension of the learnable query is the same as the number of channels of the set processing tensor;

[0040] The first stage of training of the multimodal large model is carried out according to the task type:

[0041] The task type is a picture-text comparison learning task:

[0042] Get the image features output by the query transformer and text features encoded by the query transformer Where B is the batch size, N is the number of learnable queries, and D is the embedding dimension;

[0043] The following formula is used to calculate the similarity sim between image and text i2t [i,j,k]:

[0044] sim i2t [i,j,k]=I i [k]·T j

[0045] Where I i [k] is the feature obtained by querying similar features on the features of the i-th image by the k-th learnable query; T j is the jth text feature;

[0046] For each image-text pair, take the maximum similarity S from the image to the text i2t [i,j] is

[0047] The following formula is used to calculate the similarity sim between text and image t2i [j,i,k]:

[0048] sim t2i [j,i,k]=T j I i [k]

[0049] For each image-text pair, take the maximum similarity S from the text to the image t2i [j,i] is

[0050] Calculate the image to text loss function loss i2t loss i2t =CrossEntropy(S i2t [i,j],y); where CrossEntropy() is the intermediate loss function and y is the target label;

[0051] Calculate the text-to-image loss function loss t2i loss t2i =CrossEntropy(S t2i [j,i],y);

[0052] Calculate the final loss function loss ITC for

[0053] The task type is a picture-text matching task:

[0054] Image features encoded by the frozen visual encoder and text embedding Where B is the batch size, L is the sequence length, and D is the embedding dimension;

[0055] Calculate the similarity between image and text i2t[i,j,k] is sim i2t [i,j,k]=I i [k]·T j , for each pair of image-text, take the maximum similarity S from image to text i2t [i,j] is S i2t [i,j]=m k ax(sim i2t [i,j,k]);

[0056] Calculate the similarity between text and image t2i [j,i,k] is sim t2i [j,i,k]=T j I i [k], for each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is

[0057] The softmax function is used to convert the similarity matrix into a probability distribution, and the diagonal elements of the similarity matrix are modified to set extremely low values ​​to avoid sampling; it is expressed as

[0058] S i2t [i,i]=AA

[0059] S t2i [j,j]=AA

[0060]

[0061] Where AA is the set minimum value; weights i2t [i,j] is the probability that the feature of the i-th picture is similar to the feature of the j-th text; weights t2i [j,i] is the probability of matching the j-th text feature and the i-th image feature;

[0062] Take the index corresponding to the maximum value from the probability distribution and get the negative sample image to text index matrix neg_idx i2t and negative sample text to image index neg_idx t2i ;

[0063] Freeze the image features I and text embedding T encoded by the visual encoder; after the text embedding and the learnable query are concatenated, they are input to the query transformer. The image features will be used as key-value pairs in the cross-attention module of the Transformer encoder in the query transformer, expressed as

[0064] output=QFromer(I,Concat(LearnedQueries,T))

[0065] Where output is the output of the learnable query after the query transformer; QFromer() is the transformation of the learnable query by the query transformer, which consists of two Transformer encoders; Concat() is the concatenation operation; LearnedQueries is a randomly initialized learnable query;

[0066] By taking out the first L brother tensor in the second dimension, we get the final output output_feature;

[0067] The image-text matching score obtained by the classification head is score=Linear(output_feature), where Linear() is a linear change function;

[0068] Take the average of the image-text matching score dimension 1 to get avearge_score;

[0069] Calculate the final loss function loss ITM loss ITM =CrossEntropy(averge_score,y);

[0070] The task type is image-based text generation task:

[0071] Output of a given query transformer V is the vocabulary size; predicted_score[b,t,v] represents the probability of the vth word in the vocabulary at the tth time step of the bth sample; the text in the image-text pair is converted to a label Where labels[b,t] represents the target word index in the t-th time step of the b-th sample;

[0072] Calculate the final loss function loss IGT for

[0073] Finally, the first stage of training of the multimodal large model is performed based on the calculated final loss function.

[0074] The feedforward neural network specifically includes the following steps:

[0075] The constructed feedforward neural network includes 2 or 3 basic layers;

[0076] The basic layer includes a linear transformation layer and a nonlinear activation function layer;

[0077] ReLU or GELU is used as the activation function of the nonlinear activation function layer.

[0078] Based on the idea of ​​alignment followed by fusion, step S5 combines the query transformer output with the text information and inputs it into the frozen large language model, completing the second phase of training for the multimodal large model. Specifically, the steps are as follows:

[0079] The output I of the query transformer is processed by a fully connected layer to obtain the image features in the text embedding space, thereby converting it to the text embedding space;

[0080] For text generation tasks:

[0081] Input the feature into the frozen large language model decoder and get the predicted output predicted_score as predicted_score=LLM decoder (feature); among them, LLM decoder () is the processing of the frozen large language model decoder;

[0082] Convert the text in the image-text pair to labels;

[0083] Calculate the loss function loss as

[0084] For text completion tasks:

[0085] After embedding the prefix text, the vector representation of the text information is obtained text_embedding;

[0086] After connecting feature and text_embedding, input it into the frozen large language model and get the predicted output predicted_score:

[0087] predicted_score=LLM(Concat(feature,text_embedding));

[0088] Convert the suffix text in the image-text pair to the label suffix_labels, where suffix_labels[b,t] represents the index of the target word in the t-th time step of the b-th sample;

[0089] Calculate the loss function loss as

[0090]

[0091] Finally, the multimodal large model is trained in the second stage according to the calculated final loss function.

[0092] The present invention also provides a system for implementing the multimodal large model pre-training method based on the query transformer, comprising a data set construction module, a coordinate conversion module, a structure correction module, a first training module, a second training module and a pre-training module; the data set construction module, the coordinate conversion module, the structure correction module, the first training module, the second training module and the pre-training module are connected in series in sequence; the data set construction module is used to obtain pictures under a set scene, and generate a picture-text pair data set, and upload the data information to the coordinate conversion module; the coordinate conversion module is used to use the learnable query as position information based on the query transformer of BLIP-2 according to the received data information, and replace the learnable query with four-dimensional coordinates, and upload the data information to the structure correction module; the structure correction module is used to obtain the learnable query represented by the obtained four-dimensional coordinates according to the received data information. The query transformer is modified to reduce redundant calculations in the query transformer, and the data information is uploaded to the first training module; the first training module is used to perform high-dimensional space conversion on the learnable query represented by the obtained four-dimensional coordinates according to the received data information and input it into the query transformer, perform the first stage training of the multimodal large model according to the task type, and upload the data information to the second training module; the second training module is used to combine the output of the query transformer with the text information and input it into the frozen large language model based on the received data information, based on the idea of ​​alignment and then fusion, to complete the second stage training of the multimodal large model, and upload the data information to the pre-training module; the pre-training module is used to complete the first stage training and the second stage training according to the received data information, and then complete the final pre-training of the multimodal large model based on the query transformer.

[0093] The present invention also provides a method for diagnosing faults in electric power equipment including the multimodal large model pre-training method based on the query transformer, which specifically includes the following steps:

[0094] S1. Obtain multimodal data information of power equipment;

[0095] S2. Based on the multimodal data obtained in step S1, the multimodal large model pre-training method based on the query transformer is used to pre-train the multimodal large model to obtain a multimodal large model for power equipment fault diagnosis;

[0096] S3. Using the large multimodal model for power equipment fault diagnosis obtained in step S2, perform real-time fault diagnosis of power equipment:

[0097] S4. Provide decision support based on diagnostic results;

[0098] S5. Optimize and update the multimodal large model for power equipment fault diagnosis.

[0099] The query transformer-based multimodal large model pre-training method, system, and power equipment fault diagnosis method provided by the present invention accelerate the convergence of learnable queries to sparse areas on the image by using learnable queries as position information and identifying them with four-dimensional coordinates; improve the computational efficiency of the multimodal large model by modifying the structure of the query transformer; and finally, complete the alignment of multimodal information and the text generation task guided by visual information through a two-stage training method. Therefore, the present invention can not only complete the pre-training of the multimodal large model and the corresponding power equipment fault diagnosis, but also has higher reliability, better accuracy, and less resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 Schematic diagram of the flow chart of the pre-training method of the present invention.

[0101] Figure 2 Schematic diagram of the functional modules of the system of the present invention.

[0102] Figure 3 The figure is a flow chart of the method for diagnosing faults of electric power equipment according to the present invention. DETAILED DESCRIPTION

[0103] like Figure 1 The figure shows a flow chart of the pre-training method of the present invention: The multimodal large model pre-training method based on the query transformer disclosed in the present invention comprises the following steps:

[0104] S1. Obtain images from a given scenario and generate an image-text pair dataset. This includes the following steps:

[0105] Collect image data under the set scenario; for example, if it is to be applied to the power transmission inspection scenario, the images under the power transmission inspection scenario will be used; if it is to be used for equipment fault diagnosis scenario, it is necessary to collect equipment fault images and normal operation images

[0106] Perform data cleaning on the collected image data; review the images and discard those with unclear scenes, weak features, or very repetitive features; unclear images can be discarded or image denoising technology based on deep learning can be used to improve image clarity;

[0107] For the cleaned images, resize them to the set size and convert them to RGB format;

[0108] The NumPy library is used to calculate the pixel mean and variance of each image in the R, G, and B channels; for all images, the mean and variance of all images in the R, G, and B channels are calculated;

[0109] Use pre-trained caption generation models, such as BLIP-2, to generate caption text for each image. For specific domains, such as power grid failures, fine-tune the caption generation model to improve the accuracy and professionalism of the captions. Review the generated caption text to verify its grammatical correctness and domain relevance.

[0110] Design a unique identifier for each image, name the image file according to the corresponding unique identifier, and unify the image format;

[0111] Record the data information of each image and the corresponding text, and generate a JSON structure to complete the construction and reading of the image-text pair dataset;

[0112] S2. A query transformer based on BLIP-2 takes the learnable query as location information and replaces it with four-dimensional coordinates. This includes the following steps:

[0113] Perform size statistics on the pictures obtained in step S1 to obtain the average size of the pictures;

[0114] Count the image sizes in the dataset to get the average size of the images. Set a uniform scale based on the average size of the images and convert all images to the uniform size. For example, if the average size of the images is between 224x224 and 448x448, convert all images to 224x224. To get the image size, use a Python image processing library (such as PIL or OpenCV) to get the image size information.

[0115] The number of learnable queries n is set according to the size of the set unified scale. In specific implementation, the following strategy can be adopted: for small images (such as 224x224 pixels), 32 learnable queries are set; for medium-sized images (such as 448x448 pixels), 64 learnable queries are set; for large images (such as 896x896 pixels or larger), 128 learnable queries are set. This strategy ensures that the number of learnable queries matches the complexity of the image.

[0116] Considering that the learnable query converges to the sparse specific image region very slowly, we can directly divide the image into regions according to the number of learnable queries and assign each region to a learnable query; divide the image into A grid is created, and the center coordinates, width, and height of each grid area are recorded. In specific implementation, the image areas are numbered from 0 to n. Then a dictionary can be used to record the center coordinates, width, and height of each image area.

[0117] Use a normalized coordinate system to define the coordinate system of each grid; for example, the upper left corner of the grid can be defined as (0,0) and the lower right corner as (1,1), which can adapt to images of different sizes;

[0118] Represent the center coordinates, width, and height of each grid region as a learnable query;

[0119] Normalize the learned query representation; specifically, divide the center coordinates and the region width and height by the width and height of the image to complete the normalization;

[0120] During the training phase, the four-dimensional coordinates of the learnable query are used as the parameters to be optimized, and algorithms such as gradient descent are used to optimize and update them to better capture the key information in the image. At the same time, the optimized coordinates are guaranteed to be within the set range, and the width and height are not 0. This can be achieved by using the sigmoid function or cropping operations.

[0121] Finally, the optimized learnable query is applied to the original image to extract features of the corresponding grid area. This process can be seen as compressing the original image information, converting the high-dimensional image data into a low-dimensional query representation.

[0122] S3. Based on the learnable query represented by the four-dimensional coordinates obtained in step S2, the structure of the query transformer is modified to reduce redundant calculations in the query transformer; specifically, the steps include:

[0123] The query transformer’s structure is modified by removing the learnable query and text embedding interaction from the first self-attention layer in the first transformer encoder.

[0124] The main reason for this correction is that the learnable query at this layer only contains position information, and the interaction with the text embedding is meaningless at this time. By the second self-attention layer, the learnable query has already extracted image features with the same position information. At this time, interacting with the text embedding in the self-attention layer can introduce text information related to the image features into the learnable query, thereby further extracting image features that contain similar position information, image features, and text embeddings as the current learnable query in the cross-attention module. Therefore, the interaction between the learnable query and the text embedding in the first self-attention layer of the first Transformer encoder is redundant.

[0125] S4. The learnable query represented by the four-dimensional coordinates obtained in step S2 is converted into a high-dimensional space and input into the query transformer, and the first stage of training of the multimodal large model is performed according to the task type. Specifically, the steps include:

[0126] Set the number of channels for processing tensors in a multimodal large model. Common choices include 256, 512, or 1024. Considering the need for lightweighting, a relatively small number of channels, such as 256, can be selected.

[0127] A feedforward neural network is used to perform a nonlinear transformation on the learnable query represented by the four-dimensional coordinates to ensure that the hidden dimension of the learnable query is the same as the number of channels of the set processing tensor;

[0128] In specific implementation, the feedforward neural network specifically includes the following steps:

[0129] The constructed feedforward neural network includes 2 or 3 basic layers;

[0130] The basic layer includes a linear transformation layer and a nonlinear activation function layer;

[0131] Use ReLU or GELU as the activation function of the nonlinear activation function layer to enhance the expressive power of the network;

[0132] The first stage of training of the multimodal large model is carried out according to the task type:

[0133] The task type is an image-text comparison learning task: a unimodal mask matrix is ​​used to control the interaction between the learnable query and the text embedding. In this case, the learnable query and the text embedding are mutually invisible. When encoding the text embedding in the query transformer, a special [CLS] embedding is added at the beginning of the sequence. Finally, the output of the learnable query is similar to the output of this [CLS] embedding, and the maximum value is selected as the similarity of the image-text pair.

[0134] Get the image features output by the query transformer and text features encoded by the query transformer Where B is the batch size, N is the number of learnable queries, and D is the embedding dimension;

[0135] The following formula is used to calculate the similarity sim between image and text i2t [i,j,k]:

[0136] sim i2t [i,j,k]=I i [k]·T j

[0137] Where I i [k] is the feature obtained by querying similar features on the features of the i-th image by the k-th learnable query; T j is the jth text feature;

[0138] For each image-text pair, take the maximum similarity S from the image to the text i2t[i,j] is

[0139] The following formula is used to calculate the similarity sim between text and image t2i [j,i,k]:

[0140] sim t2i [j,i,k]=T j I i [k]

[0141] For each image-text pair, take the maximum similarity S from the text to the image t2i [j,i] is

[0142] Calculate the image to text loss function loss i2t loss i2t =CrossEntropy(S i2t [i,j],y); where CrossEntropy() is the intermediate loss function and y is the target label;

[0143] Calculate the text-to-image loss function loss t2i loss t2i =CrossEntropy(S t2i [j,i],y);

[0144] Calculate the final loss function loss ITC for

[0145] The task type is a graph-text matching task. A bidirectional self-attention mask matrix is ​​used to control the interaction between the learnable query and the text embedding. In this case, the learnable query and the text embedding are mutually visible, allowing the learnable query to capture multimodal information. The final output of the learnable query is passed through a two-class linear classifier to obtain a graph-text matching score. Taking into account the number of learnable queries, the average graph-text matching score of all learned queries is used as the final graph-text matching score. In addition, a sample augmentation strategy is used in the graph-text matching task to construct positive and negative samples.

[0146] Image features encoded by the frozen visual encoder and text embedding Where B is the batch size, L is the sequence length, and D is the embedding dimension;

[0147] Calculate the similarity between image and text i2t [i,j,k] is sim i2t [i,j,k]=I i [k]·T j, for each pair of image-text, take the maximum similarity S from image to text i2t [i,j] is S i2t [i,j]=m k ax(sim i2t [i,j,k]);

[0148] Calculate the similarity between text and image t2i [j,i,k] is sim t2i [j,i,k]=T j I i [k], for each pair of image-text, take the maximum similarity S from text to image t2i [j,i] is S t2i [j,i]=m k ax(sim t2i [j,i,k]);

[0149] The softmax function is used to convert the similarity matrix into a probability distribution, and the diagonal elements of the similarity matrix are modified to set extremely low values ​​to avoid sampling; it is expressed as

[0150] S i2t [i,i]=AA

[0151] S t2i [j,j]=AA

[0152]

[0153] Where AA is the set minimum value; weights i2t [i,j] is the probability that the feature of the i-th picture is similar to the feature of the j-th text; weights t2i [j,i] is the probability of matching the j-th text feature and the i-th image feature;

[0154] Take the index corresponding to the maximum value from the probability distribution and get the negative sample image to text index matrix neg_idx i2t and negative sample text to image index neg_idx t2i ;

[0155] Freeze the image features I and text embedding T encoded by the visual encoder; after the text embedding and the learnable query are concatenated, they are input to the query transformer. The image features will be used as key-value pairs in the cross-attention module of the Transformer encoder in the query transformer, expressed as

[0156] output=QFromer(I,Concat(LearnedQueries,T))

[0157] Where output is the output of the learnable query after the query transformer; QFromer() is the transformation of the learnable query by the query transformer, which consists of two Transformer encoders; Concat() is the concatenation operation; LearnedQueries is a randomly initialized learnable query;

[0158] By taking out the first L brother tensor in the second dimension, we get the final output output_feature;

[0159] The image-text matching score obtained by the classification head is score=Linear(output_feature), where Linear() is a linear change function;

[0160] Take the average of the image-text matching score dimension 1 to get avearge_score;

[0161] Calculate the final loss function loss ITM loss ITM =CrossEntropy(averge_score,y);

[0162] The task type is an image-based text generation task: a multimodal causal mask matrix is ​​used to control the interaction between the learnable query and the text embedding; in the self-attention layer, only the text embedding can interact with the learnable query, while the learnable query cannot interact with the text embedding; the purpose of doing this is because the information required for text generation can be extracted through the learnable query and then passed to the text embedding through self-attention interaction; therefore, the image-based text generation task forces the model to extract visual features related to the text; in addition, for this task, the [CLS] embedding in the query transformer is replaced with a special [DEC] embedding to mark the decoding task; the multimodal causal mask matrix also controls the interaction between each text embedding in the sequence, specifically, the current text embedding can interact with the previous text embedding but not with the subsequent text embedding; at the end of the text encoder, a linear layer "nn.linear(hidden_size,vocab_size)" is added to act as an encoder to complete the text generation;

[0163] Output of a given query transformer V is the vocabulary size; predicted_score[b,t,v] represents the probability of the vth word in the vocabulary at the tth time step of the bth sample; the text in the image-text pair is converted to a label Where labels[b,t] represents the target word index in the t-th time step of the b-th sample;

[0164] Calculate the final loss function loss IGT for

[0165] Finally, according to the calculated final loss function, the first stage of training of the multimodal large model is carried out;

[0166] S5. Based on the idea of ​​alignment followed by fusion, the query transformer output is combined with the text information and input into the frozen large language model to complete the second stage of training the multimodal large model. This includes the following steps:

[0167] The query transformer output I is processed through a fully connected layer to obtain the image features in the text embedding space, thereby converting it to the text embedding space. Specifically, a linear layer is created through "nn.Linear(hidden_dim, embed_dim)" to achieve dimensionality conversion of the query transformer output.

[0168] For text generation tasks:

[0169] Input the feature into the frozen large language model decoder and get the predicted output predicted_score as predicted_score=LLM decoder (feature); among them, LLM decoder () is the processing of the frozen large language model decoder;

[0170] Convert the text in the image-text pair to labels;

[0171] Calculate the loss function loss as

[0172] For text completion tasks: The model divides the text in each image-text pair into a prefix and a suffix, and trains the model to generate the suffix text based on visual information and the prefix text.

[0173] After embedding the prefix text, the vector representation of the text information is obtained text_embedding;

[0174] After connecting feature and text_embedding, input it into the frozen large language model and get the predicted output predicted_score:

[0175] predicted_score=LLM(Concat(feature,text_embedding));

[0176] Convert the suffix text in the image-text pair to the label suffix_labels, where suffix_labels[b,t] represents the index of the target word in the t-th time step of the b-th sample;

[0177] Calculate the loss function loss as

[0178]

[0179] Finally, according to the calculated final loss function, the multimodal large model is trained in the second stage;

[0180] S6. After completing the first stage training and the second stage training, the final pre-training of the query transformer-based multimodal large model is completed.

[0181] like Figure 2 The diagram shows the functional modules of the system of the present invention: the system disclosed in the present invention for implementing the multimodal large model pre-training method based on query transformer comprises a data set construction module, a coordinate conversion module, a structure correction module, a first training module, a second training module and a pre-training module; the data set construction module, the coordinate conversion module, the structure correction module, the first training module, the second training module and the pre-training module are connected in series in sequence; the data set construction module is used to obtain pictures under the set scene, and generate picture-text pair data sets, and upload the data information to the coordinate conversion module; the coordinate conversion module is used to use the learnable query as position information based on the query converter of BLIP-2 according to the received data information, and replace the learnable query with four-dimensional coordinates, and upload the data information to the structure correction module; the structure correction module is used to obtain the picture under the set scene, and generate the picture-text pair data sets, and upload the data information to the structure correction module according to the received data information; The learnable query represented by the label is corrected, the structure of the query transformer is modified to reduce redundant calculations in the query transformer, and the data information is uploaded to the first training module; the first training module is used to perform high-dimensional space conversion on the learnable query represented by the four-dimensional coordinate according to the received data information and input it into the query transformer, perform the first stage training of the multimodal large model according to the task type, and upload the data information to the second training module; the second training module is used to combine the output of the query transformer with the text information and input it into the frozen large language model based on the received data information, based on the idea of ​​first alignment and then fusion, to complete the second stage training of the multimodal large model, and upload the data information to the pre-training module; the pre-training module is used to complete the first stage training and the second stage training according to the received data information, and then complete the final pre-training of the multimodal large model based on the query transformer.

[0182] like Figure 3The present invention further provides a method for diagnosing a fault in an electric power device, including the multimodal large model pre-training method based on a query transformer, which specifically includes the following steps:

[0183] S1. Obtain multimodal data information of power equipment;

[0184] The multimodal data information includes: collecting image data of power equipment, including equipment appearance images, infrared thermal imaging images, etc.; organizing text data such as equipment historical maintenance records and fault reports;

[0185] S2. Based on the multimodal data obtained in step S1, the multimodal large model pre-training method based on the query transformer is used to pre-train the multimodal large model to obtain a multimodal large model for power equipment fault diagnosis;

[0186] In specific implementation, we use image and text data as input to train the model to understand the relationship between different modalities; through the query transformer mechanism, we enhance the model's ability to understand and integrate multimodal data;

[0187] S3. Using the large multimodal model for power equipment fault diagnosis obtained in step S2, perform real-time fault diagnosis of power equipment:

[0188] In specific implementation, multimodal data of power equipment is collected in real time; the collected data is input into a trained multimodal large model; the model analyzes the multimodal data to identify potential failure modes; and outputs fault diagnosis results, including fault type, fault severity, and recommended maintenance measures.

[0189] S4. Provide decision support based on diagnostic results;

[0190] In specific implementation, equipment maintenance suggestions are automatically generated based on the fault diagnosis results; in the case of emergency faults, the alarm system is triggered and relevant personnel are notified; the diagnosis results are stored in the database for subsequent model updates and optimization;

[0191] S5. Continuous optimization and update:

[0192] During specific implementation, actual maintenance feedback is collected regularly to verify the accuracy of model diagnosis; new fault case data is used to continuously fine-tune and update the model; and the model's adaptability to new equipment and newly emerging fault types is optimized.

Claims

1. A multimodal large model pre-training method based on a query transformer, comprising the following steps: S1. Obtain images from a given scenario and generate a dataset of image-text pairs. S2. A query transformer based on BLIP-2 that treats learnable queries as location information and replaces them with 4D coordinates. S3. According to the learnable query represented by the four-dimensional coordinate obtained in step S2, the structure of the query transformer is modified to reduce redundant calculations in the query transformer; S4. The learnable query represented by the four-dimensional coordinates obtained in step S2 is converted into a high-dimensional space and input into the query transformer, and the first stage of multimodal large model training is performed according to the task type; S5. Based on the idea of ​​alignment followed by fusion, the query transformer output is combined with the text information and input into the frozen large language model, completing the second stage of training for the multimodal large model. S6. After completing the first and second stage training, complete the pre-training of the final query transformer-based multimodal large model.

2. The multimodal large model pre-training method based on query transformer according to claim 1 is characterized in that The step S1 of obtaining an image in a set scene and generating an image-text pair dataset specifically includes the following steps: Collect image data under the set scene; Perform data cleaning on the collected image data; For the cleaned images, resize them to the set size and convert them to RGB format; The NumPy library is used to calculate the pixel mean and variance of each image in the R, G, and B channels; for all images, the mean and variance of all images in the R, G, and B channels are calculated; Use a pre-trained caption generation model to generate description text for each image; Design a unique identifier for each image, name the image file according to the corresponding unique identifier, and unify the image format; Record the data information of each image and the corresponding text, and generate a JSON structure to complete the construction and reading of the image-text pair dataset.

3. The multimodal large model pre-training method based on query transformer according to claim 2 is characterized in that The BLIP-2-based query transformer described in step S2 takes the learnable query as position information and replaces the learnable query with four-dimensional coordinates, specifically including the following steps: Perform size statistics on the pictures obtained in step S1 to obtain the average size of the pictures; According to the average scale of the pictures, set a unified scale and convert all pictures to the size of the unified scale; Set the number of learnable queries based on the size of the set unified scale ; Divide the image into grids, and record the center coordinates, width and height of each grid area; Use the normalized coordinate system to define the coordinate system of each grid; Represent the center coordinates, width, and height of each grid region as a learnable query; Normalize the obtained learnable query; During the training phase, the four-dimensional coordinates of the learnable query are used as the parameters to be optimized, and the gradient descent algorithm is used for optimization and update; at the same time, the optimized coordinates are guaranteed to be within the set range; Finally, the optimized learnable query is applied to the original image to extract features corresponding to the grid regions.

4. The multimodal large model pre-training method based on query transformer according to claim 3 is characterized in that Step S3, based on the learnable query represented by the four-dimensional coordinates obtained in step S2, modifies the structure of the query transformer, specifically comprising the following steps: The first self-attention layer in the first Transformer encoder inside the query transformer removes the interaction between the learnable query and text embedding to complete the structural revision of the query transformer.

5. The multimodal large model pre-training method based on query transformer according to claim 4 is characterized in that Step S4 converts the learnable query represented by the four-dimensional coordinates obtained in step S2 into a high-dimensional space and inputs it into the query transformer. The first stage of training the multimodal large model is performed according to the task type, which specifically includes the following steps: Set the number of channels for processing tensors in a multimodal large model; A feedforward neural network is used to perform a nonlinear transformation on the learnable query represented by the four-dimensional coordinates to ensure that the hidden dimension of the learnable query is the same as the number of channels of the set processing tensor; The first stage of training of the multimodal large model is carried out according to the task type: The task type is a picture-text comparison learning task: Get the image features output by the query transformer and text features encoded by the query transformer ;in is the batch size, is the number of learnable queries, is the embedding dimension; The following formula is used to calculate the similarity between image and text : In the formula For the kth learnable query, query similar features on the i-th image features and aggregate the obtained features; is the jth text feature; For each image-text pair, take the maximum similarity between image and text for ; The following formula is used to calculate the similarity between text and image : For each image-text pair, take the maximum similarity between text and image for ; Calculating the image-to-text loss function for ;in, is the intermediate loss function and , is the target label; Calculating text-to-image loss function for ; Calculate the final loss function for ; The task type is a picture-text matching task: Image features encoded by the frozen visual encoder and text embedding ,in is the batch size, is the sequence length, is the embedding dimension; Calculating image to text similarity for , for each pair of image-text, take the maximum similarity from image to text for ; Calculating text-to-image similarity for , for each pair of image-text, take the maximum similarity between text and image for ; The softmax function is used to convert the similarity matrix into a probability distribution, and the diagonal elements of the similarity matrix are modified to set extremely low values ​​to avoid sampling; it is expressed as In the formula is the set extremely low value; is the probability that the feature of the i-th picture is similar to the feature of the j-th text; is the probability of matching the jth text feature with the i-th image feature; Take the index corresponding to the maximum value from the probability distribution and get the negative sample image to text index matrix and negative text-to-image indexing ; Freeze the image features encoded by the visual encoder and text embedding After the text embedding and the learnable query are concatenated, they are fed into the query transformer. The image features are used as key-value pairs in the cross-attention module of the Transformer encoder in the query transformer, expressed as In the formula is the output of the learnable query after the query transformer; To transform the query transformer to the learnable query, the query transformer consists of two Transformer encoders; For splicing operation; is a randomly initialized learnable query; By taking out the front Brother tensor, get the final output ; Get the image-text matching score through the classification head for ,in is a linear change function; Matching pictures and text to scores Taking the average on dimension 1, we get ; Calculate the final loss function for ; The task type is image-based text generation task: Output of a given query transformer , is the vocabulary size; Represents the probability of the vth word in the vocabulary at the tth time step of the bth sample; the text in the image-text pair is converted to a label ,in represents the target word index in the t-th time step of the b-th sample; Calculate the final loss function for ; Finally, the first stage of training of the multimodal large model is performed based on the calculated final loss function.

6. The multimodal large model pre-training method based on query transformer according to claim 5 is characterized in that The feedforward neural network specifically includes the following steps: The constructed feedforward neural network includes 2 or 3 basic layers; The basic layer includes a linear transformation layer and a nonlinear activation function layer; ReLU or GELU is used as the activation function of the nonlinear activation function layer.

7. The multimodal large model pre-training method based on query transformer according to claim 6 is characterized in that Based on the idea of ​​alignment followed by fusion, step S5 combines the query transformer output with the text information and inputs it into the frozen large language model, completing the second phase of training for the multimodal large model. Specifically, the steps are as follows: The output of the query transformer After the fully connected layer is processed, the image in the text embedding space is obtained , thereby converting to the text embedding space; For text generation tasks: Will Input to the frozen large language model decoder to get the predicted output for ;in, The process of freezing the large language model decoder; Convert text in image-text pairs into labels ; Calculating the loss function for ; For text completion tasks: The prefix text is embedded to obtain the vector representation of the text information ; Will and After connection, it is input into the frozen large language model to obtain the predicted output for ； Convert the suffix text in the image-text pair to a label ,in represents the index of the target word in the t-th time step of the b-th sample; Calculating the loss function for Finally, the multimodal large model is trained in the second stage according to the calculated final loss function.

8. A system for implementing the multimodal large model pre-training method based on query transformer according to any one of claims 1 to 7, characterized in that It includes a data set construction module, a coordinate conversion module, a structure correction module, a first training module, a second training module, and a pre-training module; the data set construction module, the coordinate conversion module, the structure correction module, the first training module, the second training module, and the pre-training module are connected in series in sequence; the data set construction module is used to obtain images under a set scene, generate an image-text pair dataset, and upload the data information to the coordinate conversion module; The coordinate conversion module is used to convert the learnable query into four-dimensional coordinates based on the received data information and the query transformer of BLIP-2, and upload the data information to the structure correction module; The structure correction module is used to correct the structure of the query transformer according to the received data information and the learnable query represented by the obtained four-dimensional coordinates, so as to reduce redundant calculations in the query transformer, and upload the data information to the first training module; The first training module is used to perform a high-dimensional space conversion on the learnable query represented by the four-dimensional coordinates based on the received data information and input it into the query transformer, perform the first phase of training on the multimodal large model according to the task type, and upload the data information to the second training module; The second training module is used to combine the output of the query transformer with the text information based on the received data information, based on the idea of ​​alignment before fusion, and input it into the frozen large language model to complete the second stage of training of the multimodal large model, and upload the data information to the pre-training module; The pre-training module is used to complete the pre-training of the final multimodal large model based on the query transformer after completing the first stage training and the second stage training according to the received data information.

9. A method for diagnosing faults in electric power equipment comprising the multimodal large model pre-training method based on a query transformer according to any one of claims 1 to 7, comprising the following steps: S1. Obtain multimodal data information of power equipment; S2. Based on the multimodal data obtained in step S1, the multimodal large model pre-training method based on the query converter according to one of claims 1 to 7 is used to pre-train the multimodal large model to obtain a multimodal large model for power equipment fault diagnosis; S3 using step S2 obtained by the multi-modal large model of power equipment fault diagnosis, real-time fault diagnosis of power equipment; S4. Provide decision support based on diagnostic results; S5. Optimize and update the large multimodal model for power equipment fault diagnosis.

Images