A multi-modal data fusion method based on graphic-text interaction

Abstract

The application discloses a kind of multimodal data fusion methods based on picture-text interaction, comprising the following steps: S1, obtain multimodal data, the multimodal data includes inspection image data and equipment state data;S2, obtain the image feature map of inspection image by convolutional neural network;S3, by text extractor, equipment state data is preprocessed to obtain text feature map;S4, construct multi-head attention module to obtain text attention weight and image attention weight;S5, based on text attention weight and image attention weight, obtain image text hybrid feature;S6, by multi-head cross attention module, obtain the bidirectional interactive information of training data and target;S7, by feature map hybrid module, obtain hybrid feature map information and output prediction result.The scheme improves the recognition accuracy by fusion learning and analysis of multimodal features.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a multimodal data fusion method based on graphic and text interaction. Background Technology

[0002] Currently, artificial intelligence has broad application prospects in various fields. As information dissemination methods become increasingly diversified, the types of data mining results are also becoming more abundant. Broadly speaking, this data can be divided into two categories: structured data, such as text and numerical tables, and unstructured data, including images, audio, and video. Given the interactivity and redundancy between the two, using multimodal techniques to fuse different types of data can reduce data storage space and add multiple dimensions of feature information to the same descriptive object. The processed data, when applied to data analysis and prediction, can effectively improve the accuracy of the results.

[0003] In traditional power equipment fault analysis, given data in one modality (such as images or videos describing a problem), manual analysis is often required to obtain regular data in other modalities (objective textual solutions). This process is time-consuming, and the results vary in their relevance to the problem due to differing levels of understanding. Furthermore, most methods focus on extracting features from individual modalities without fusing and learning these features. Therefore, a multimodal data fusion algorithm is crucial for solving prediction problems with given input and output data types.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to address the lack of multimodal feature fusion learning and analysis in existing prediction methods. It proposes a multimodal data fusion method based on image-text interaction, which constructs a multimodal dataset based on images and corresponding text information by using multimodal information and a bidirectional cross-attention method. Two multi-head self-attention (MSA) modules are constructed to learn image and text features respectively. A bidirectional cross-attention method is proposed for cross-information learning. Two decoders are constructed, including an MSA block and a multi-head cross-attention (MCA) block, for cross-information learning. The obtained feature maps are input into a feature map fusion module for feature extraction and output prediction results, effectively improving the accuracy of the results.

[0006] In a first aspect, one technical solution provided in this embodiment of the invention is a multimodal data fusion method based on image-text interaction, comprising the following steps:

[0007] S1. Acquire multimodal data, which includes inspection image data and equipment status data;

[0008] S2. Obtain the image feature map of the inspection image through a convolutional neural network;

[0009] S3. The device status data is preprocessed using a text extractor to obtain a text feature map;

[0010] S4. Construct a multi-head attention module to obtain text attention weights and image attention weights;

[0011] S5. Obtain image-text hybrid features based on text attention weights and image attention weights;

[0012] S6. Obtain bidirectional interactive information between training data and the target through a multi-head cross-attention module;

[0013] S7. Obtain the mixed feature map information through the feature map mixing module and output the prediction result.

[0014] Preferably, in step S2, local features of the image are learned through a convolutional neural network to obtain the image feature map of the inspection image.

[0015] Preferably, the convolutional neural network includes: a convolutional layer, a batch normalization layer, an activation layer, and a max pooling layer.

[0016] Preferably, step S3 includes the following steps:

[0017] S31. Obtain text information representing device status data;

[0018] S32. Segment the text information by word, and divide the text information into phrases of length k;

[0019] S33. Record the sequence number of each phrase using a dictionary;

[0020] S34. The entire text information sequence is represented by the sequence number.

[0021] Preferably, before step S4, it is also necessary to build an encoder to learn the global features of the inspected images and text.

[0022] Preferably, the data processing steps of the encoder include:

[0023] Data of different lengths are normalized using a normalization layer;

[0024] Calculate multi-head attention for normalized or normalized data features;

[0025] Solving nonlinear problems of data features through feedforward neural networks;

[0026] Add the original data to the output data of the multi-head attention module.

[0027] Preferably, the feedforward neural network includes: a linear layer, a GELU activation function layer, and a DropPath layer.

[0028] Preferably, step S5 includes:

[0029] S51. The inspection images are sequentially passed through a convolutional neural network and a multi-head attention module to obtain the image modality attention weight feature map;

[0030] S52. Text data is sequentially processed by a text extractor and a multi-head attention module to obtain a text modality attention weight feature map;

[0031] S53. The image modality attention weight feature map and the text modality attention weight feature map are fused together.

[0032] Preferably, step S6 includes the following steps:

[0033] In the N-layer decoder, the training data continuously updates the feature sequence through the feature information from the multi-head cross-attention module of the target data;

[0034] Simultaneously, the target data is also continuously updating its feature sequence using feature information from the multi-head cross-attention module of the training data;

[0035] The feature information of the target data is transformed into different Keys / Values ​​that interact with the Queries of the training data, thereby achieving a two-way interactive effect.

[0036] Preferably, in step S7, the feature map mixing module includes a two-dimensional convolutional layer, a one-dimensional convolutional layer, a multi-layer perceptron layer, and a fully connected layer.

[0037] The beneficial effects of this invention are as follows: A multimodal data fusion method based on image-text interaction is proposed. This method constructs a multimodal dataset based on images and corresponding text information by using multimodal information and a bidirectional cross-attention approach. Two multi-head self-attention (MSA) modules are constructed to learn image and text features respectively. A bidirectional cross-attention method is proposed for cross-information learning. Two decoders are constructed, including an MSA block and a multi-head cross-attention (MCA) block, for cross-information learning. The obtained feature maps are input into a feature map fusion module for feature extraction, and the prediction results are output, effectively improving the accuracy of the results.

[0038] The above description of the invention is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0039] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0040] Figure 1 This is a flowchart of a multimodal data fusion method based on graphic and text interaction according to the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only one preferred embodiment of this invention and are only used to explain this invention. They do not limit the scope of protection of this invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0042] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations (or steps) can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures; the process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0043] Example: Figure 1 As shown, a multimodal data fusion algorithm based on image-text interaction is described. The steps of this method are as follows:

[0044] S1. Acquire multimodal data, which includes inspection image data and equipment status data.

[0045] S2. Obtain the image feature map of the inspection image through a convolutional neural network.

[0046] Specifically, it includes the following steps:

[0047] S21, Obtain the structural image through the image tool library;

[0048] S22, construct a convolutional neural network to learn local features of an image. A convolutional neural network mainly includes convolutional layers, batch normalization layers, activation layers, and max-pooling layers;

[0049] S23, the operation of the convolutional layer can be represented as P conv =f(F in *W)+b. Wherein The input features are represented by W and b, which are the parameter matrices and biases of the convolution kernel, and f represents the convolution operation.

[0050] S24, batch normalization is performed on the feature map after the convolution operation, which can be represented as P. BN =BN(P conv );

[0051] S25, the nonlinearity of the model is increased by using the GELU activation function;

[0052] S26. Finally, the data is compressed and the main features are extracted by using a max pooling layer.

[0053] S27, using the constructed convolutional neural network, the local feature map of the image can be obtained, represented as: F lcoal =Conv(F in );

[0054] Wherein Conv represents the convolutional neural network constructed in this invention.

[0055] S3. The device status data is preprocessed using a text extractor to obtain a text feature map.

[0056] Specifically, it includes:

[0057] S31, Segment the text sequence by word, dividing the text sequence into phrases of length k;

[0058] S32, Construct a dictionary to record the order in which phrases appear;

[0059] S33, replace the original words with the numerical indices corresponding to the words in the dictionary, and encode the entire text into a sequence of numbers.

[0060] S4. Construct a multi-head attention module to obtain text attention weights and image attention weights.

[0061] Specifically, it includes:

[0062] S41, construct a self-attention mechanism encoder to learn global features of images / text. The encoder mainly includes a normalization layer, multi-head attention computation, a feedforward neural network, and residual connections;

[0063] S42, the normalization layer processes data of different lengths through a linear normalization layer, and can be represented as P. LN =Norm(F in ),in This represents the feature input, and Norm is the LayerNorm normalization process;

[0064] S43, Since the attention mechanism does not consider positional information, positional embedding information is added to the initial data;

[0065] S44, performs multi-head attention calculation on the normalized features. First, for the input x, construct the matrix M of the three vector projections. Q M K and M V Multiplying them together yields three distinct matrices Q, K, and V. Therefore, the calculation of attention can be expressed as: Where d k This is represented by the dimension of matrix K. To prevent overfitting, multi-head attention is used; the operation can be represented as: MultiHead(Q′,K′,V′)=Concat(head1,…,head) h W O Among them, head i =Attention(Q) i ,K i V i )and It is a linear projection matrix;

[0066] S45 uses a feedforward neural network to solve nonlinear problems that a single-layer perceptron cannot handle. This network includes linear layers, GELU activation function layers, and DropPath layers. The DropPath layers are used to improve the network's generalization ability.

[0067] S46 uses residual connections to add the original data to the output of the multi-head attention module, which can reduce model complexity and prevent gradient vanishing.

[0068] S5. Obtain image-text hybrid features based on text attention weights and image attention weights.

[0069] Specifically, it includes:

[0070] S51, the image modality features are processed through the above convolutional neural network and multi-head attention mechanism to obtain the image modality attention weight feature map X. img ;

[0071] S52, the text modality features are processed by the text extractor and multi-head attention mechanism described above to obtain the text modality attention weight feature map X. txt ;

[0072] S53, the embedding representations from the two modal information are combined and represented as X. emb =αX img +(1-α)X txt .

[0073] S6. Obtain bidirectional interactive information between training data and the target through a multi-head cross-attention module.

[0074] The bidirectional multi-head cross-attention module has a similar structure to the multi-head attention mechanism, but differs in its computational method and the objects it considers; the joint embedding representation obtained after step S4... and the target embedding representation obtained through the same steps Where T and D represent the sequence length and feature map dimension, respectively; the Q, K, and V matrices of the cross-attention are obtained: Q α =X α W Qα K β =X β W Kβ V β =X β W Vβ The implicit dependency between training data and the target can be represented as follows: The above describes the single-head cross-attention mechanism; the changes in the multi-head cross-attention mechanism are the same as those in the multi-head attention mechanism; MCA(Z) β→α )=[CA1(Z β→α CA2(Z) β→α );…;CA k (Z β→α )]U mca Z β→α This represents a feature map with positional encoding, CA k Representing a single-head cross-attention mechanism, U mca This represents a linear mapping matrix.

[0075] In the N-layer decoder, the training data continuously updates the feature sequence through the feature information from the multi-head cross-attention module of the target data;

[0076] Simultaneously, the target data is also continuously updating its feature sequence using feature information from the multi-head cross-attention module of the training data;

[0077] The feature information of the target data is transformed into different Keys / Values ​​that interact with the Queries of the training data, thereby achieving a two-way interactive effect.

[0078] S7. Obtain the mixed feature map information through the feature map mixing module and output the prediction result.

[0079] Specifically, it includes:

[0080] S71, the hybrid module includes two-dimensional convolution, one-dimensional convolution, multi-layer perceptron layer and fully connected layer;

[0081] S72 connects the feature maps of the training data and the target, and extracts features through convolutional layers;

[0082] S73, finally, the feature map is input into the fully connected layer to obtain the final prediction result P, denoted as...

[0083] The specific embodiments described above are preferred embodiments of a multimodal data fusion method based on text and image interaction according to the present invention, and are not intended to limit the specific scope of the present invention. The scope of the present invention includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. A multimodal data fusion method based on image-text interaction, characterized in that, Includes the following steps: S1. Acquire multimodal data, which includes inspection image data and equipment status data; S2. Obtain image feature maps of the inspection image data through a convolutional neural network; S3. The device status data is preprocessed using a text extractor to obtain a text feature map; S4. Construct a multi-head attention module to obtain the text attention weights corresponding to the text feature maps and the image attention weights corresponding to the image feature maps; S5. Obtain image-text hybrid features based on text attention weights and image attention weights; S6. Obtain bidirectional interactive information between training data and target data through a multi-head cross-attention module; S7. The feature map fusion module processes the bidirectional interactive information to obtain fused feature map information and outputs the prediction results. Step S5 includes: S51. The inspection image data are sequentially passed through a convolutional neural network and a multi-head attention module to obtain the image modality attention weight feature map; S52. The text data corresponding to the device status data are sequentially processed by the text extractor and the multi-head attention module to obtain the text modality attention weight feature map. S53. The image modality attention weight feature map and the text modality attention weight feature map are fused to obtain the image-text hybrid feature; S6 includes the following steps: In the N-layer decoder, the training data continuously updates the feature sequence through the feature information from the multi-head cross-attention module of the target data; Simultaneously, the target data is also continuously updating its feature sequence using feature information from the multi-head cross-attention module of the training data; The feature information of the target data is transformed into different Keys / Values ​​that interact with the Queries of the training data to obtain bidirectional interactive information that represents the bidirectional interaction effect.

2. The multimodal data fusion method based on image-text interaction according to claim 1, characterized in that, In step S2, the local features of the image are learned through a convolutional neural network to obtain the image feature map of the inspection image.

3. The multimodal data fusion method based on image-text interaction according to claim 2, characterized in that, The convolutional neural network includes: convolutional layers, batch normalization layers, activation layers, and max pooling layers.

4. The multimodal data fusion method based on image-text interaction according to claim 1, characterized in that, Step S3 includes the following steps: S31. Obtain text information representing device status data; S32. Segment the text information by word, and divide the text information into phrases of length k; S33. Record the sequence number of each phrase using a dictionary; S34. The entire text information sequence is represented by the sequence number.

5. The multimodal data fusion method based on image-text interaction according to claim 1, characterized in that, Before step S4, it is also necessary to build an encoder to learn the global features of inspected images and text.

6. The multimodal data fusion method based on image-text interaction according to claim 5, characterized in that, The data processing steps of the encoder include: Data of different lengths are normalized using a normalization layer; Calculate multi-head attention for normalized or normalized data features; Solving nonlinear problems of data features through feedforward neural networks; Add the original data to the output data of the multi-head attention module.

7. The multimodal data fusion method based on image-text interaction according to claim 6, characterized in that, The feedforward neural network includes: a linear layer, a GELU activation function layer, and a DropPath layer.

8. The multimodal data fusion method based on image-text interaction according to claim 1, characterized in that, In step S7, the feature map mixing module includes a two-dimensional convolutional layer, a one-dimensional convolutional layer, a multi-layer perceptron layer, and a fully connected layer.

Images