Method for establishing multi-modal data fusion model based on multi-attention mechanism

Abstract

The application discloses a method for establishing a multi-modal data fusion model based on a multi-attention mechanism, and relates to the technical field of interactive information and fusion between modes. The method comprises the following steps: an acquisition step, an extraction step, a fusion step and a construction step. Through three attention modules, the method fully mines multi-modal information and interactive information between the multi-modal, fuses the interactive information between the multi-modal through a soft attention mechanism, filters redundant and noise data, improves the utilization rate of the interactive information between the multi-modal by the model, and thus improves the accuracy of the model in sentiment analysis. The effectiveness of the proposed model is verified through comparative experiments and ablation experiments.

Description

Technical Field

[0001] This invention relates to the field of intermodal interaction information and fusion technology, and in particular to a method for establishing a multimodal data fusion model based on a multi-attention mechanism. Background Technology

[0002] Multimodal intelligence is a problem that must be solved in the development of artificial intelligence. With the maturity of unimodal intelligence technology, multimodal intelligence has gradually attracted more attention. Traditional multimodal fusion is divided into feature-level fusion, decision-level fusion, and hybrid fusion according to the fusion stage. However, due to the powerful flexibility of deep learning, this classification is not very applicable to the current complex deep learning methods.

[0003] Existing research mostly focuses on the consistency of multimodal data, while neglecting the impact of intermodal interaction information on multimodal data fusion.

[0004] Therefore, proposing a method for establishing a multimodal data fusion model based on a multi-attention mechanism to address the difficulties in existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method for establishing a multimodal data fusion model based on a multi-attention mechanism, which can address the impact of intermodal interaction information on multimodal data fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The method for establishing a multimodal data fusion model based on a multi-attention mechanism includes the following steps:

[0008] S1. Acquisition Steps: Acquire multimodal data;

[0009] S2. Extraction steps: Input the acquired multimodal data into multiple independent LSTMs for feature transformation, and then use multiple attention modules to extract the interaction information between multimodalities to obtain the features of multimodal interaction information;

[0010] S3. Fusion Step: Perform feature fusion on the features of the interaction information between multiple modalities to obtain the post-feature matrix;

[0011] S4. Construction steps: Based on the post-feature matrix, construct a multimodal data fusion model.

[0012] Optionally, in the above method, the multimodal data obtained in S1 includes, but is not limited to, text data, audio data, and visual data.

[0013] Optionally, the above method may also include using LSTM to perform feature mapping on the multimodal data before performing the S2 extraction step:

[0014]

[0015] Where T represents text data, A represents audio data, and V represents visual data, using... Let R represent the three modalities of the input data, where R is the real number field, l {T,A,V} d represents the length of the feature sequence. {T,A,V} h represents the feature dimension. {T,A,V} Where d is the size of the LSTM hidden layer, and d is the dimension of the common features;

[0016] That is, LSTM maps the features of text modality, audio modality and visual modality into the same subspace.

[0017] Optionally, in the above method, S2 utilizes multiple attention modules to extract multimodal interaction information, specifically: unimodal interaction information extraction, bimodal interaction information extraction, and multimodal interaction information extraction.

[0018] Among them, the extraction of unimodal interaction information yields the unimodal attention feature matrix (Att). T Att A Att V );

[0019] Bimodal interaction information is extracted to obtain a bimodal attention feature matrix (TA, TV, AV);

[0020] Multimodal interaction information extraction yields a trimodal fusion feature matrix (F AV→T ,F TV→A ,F TA→V ).

[0021] Optionally, in the above method, the specific content of feature fusion for the features of multimodal interaction information in S3 is as follows:

[0022] First, matrix addition is used to fuse the single-modal attention feature matrix, the dual-modal attention feature matrix, and the trimodal fusion feature matrix respectively;

[0023] Secondly, a fully connected layer is used to transform the dimensions of the fused bimodal attention feature matrix and the trimodal fusion feature matrix, so that the feature matrices of different modalities can be fused.

[0024] The matrix addition method is used again to fuse the single-modal attention feature matrix, the dual-modal attention feature matrix, and the trimodal fusion feature matrix;

[0025] Finally, the post-feature matrix is ​​obtained using a soft attention mechanism.

[0026] Optionally, the above methods may also include using ablation experiments to validate the established multimodal data fusion model.

[0027] The above methods may optionally include modal ablation and model ablation experiments.

[0028] As can be seen from the above technical solution, compared with the prior art, the present invention provides a method for establishing a multimodal data fusion model based on a multi-attention mechanism. The beneficial effects are as follows: by using three attention modules, multimodal information and interaction information between multimodalities are fully explored; by using a soft attention mechanism to fuse interaction information between multimodalities, redundant and noisy data are filtered out, improving the model's utilization rate of interaction information between multimodalities, thereby improving the accuracy of the model's sentiment analysis; the effectiveness of the proposed model is also verified through comparative experiments and ablation experiments. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0030] Figure 1 A flowchart illustrating the method for establishing a multimodal data fusion model based on a multi-attention mechanism provided by this invention;

[0031] Figure 2 This is a block diagram of a multimodal data fusion model based on a multi-attention mechanism provided by the present invention;

[0032] Figure 3 A block diagram of a self-attention mechanism provided in an embodiment of the present invention;

[0033] Figure 4 The figure shows the ablation experiment results of the MOSI model provided in the embodiment of the present invention;

[0034] Figure 5 The figure shows the ablation experiment results of the MOSEI model provided in the embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Reference Figure 1 As shown, this invention discloses a method for establishing a multimodal data fusion model based on a multi-attention mechanism, comprising the following steps:

[0037] S1. Acquisition Steps: Acquire multimodal data;

[0038] S2. Extraction steps: Input the acquired multimodal data into multiple independent LSTMs for feature transformation, and then use multiple attention modules to extract the interaction information between multimodalities to obtain the features of multimodal interaction information;

[0039] S3. Fusion Step: Perform feature fusion on the features of the interaction information between multiple modalities to obtain the post-feature matrix;

[0040] S4. Construction steps: Based on the post-feature matrix, construct a multimodal data fusion model.

[0041] Furthermore, S1 acquires multimodal data including, but not limited to, text data, audio data, and visual data.

[0042] Furthermore, before performing the S2 extraction step, LSTM is used to perform feature mapping on the multimodal data:

[0043]

[0044] Where T represents text data, A represents audio data, and V represents visual data, using... Let R represent the three modalities of the input data, where R is the real number field, l {T,A,V} d represents the length of the feature sequence. {T,A,V} h represents the feature dimension. {T,A,V} Where d is the size of the LSTM hidden layer, and d is the dimension of the common features;

[0045] That is, LSTM maps the features of text modality, audio modality and visual modality into the same subspace.

[0046] Furthermore, S2 utilizes multiple attention modules to extract multimodal interaction information, specifically divided into unimodal interaction information extraction, bimodal interaction information extraction, and multimodal interaction information extraction.

[0047] Among them, the extraction of unimodal interaction information yields the unimodal attention feature matrix (Att). T Att A Att V );

[0048] Bimodal interaction information is extracted to obtain a bimodal attention feature matrix (TA, TV, AV);

[0049] Multimodal interaction information extraction yields a trimodal fusion feature matrix (F AV→T ,F TV→A ,F TA→V ).

[0050] Furthermore, the specific details of feature fusion for multimodal interaction information in S3 are as follows:

[0051] First, matrix addition is used to fuse the single-modal attention feature matrix, the dual-modal attention feature matrix, and the trimodal fusion feature matrix respectively;

[0052] Secondly, a fully connected layer is used to transform the dimensions of the fused bimodal attention feature matrix and the trimodal fusion feature matrix, so that the feature matrices of different modalities can be fused.

[0053] The matrix addition method is used again to fuse the single-modal attention feature matrix, the dual-modal attention feature matrix, and the trimodal fusion feature matrix;

[0054] Finally, the post-feature matrix is ​​obtained using a soft attention mechanism.

[0055] Furthermore, it also includes using ablation experiments to validate the established multimodal data fusion model.

[0056] Ablation experiments include modal ablation and model ablation.

[0057] In one specific embodiment, see Figure 2 The diagram shows the block diagram of the multimodal data fusion model based on a multi-attention mechanism provided by this invention. This model inputs text, audio, and visual data into three independent LSTMs for feature extraction, and then feeds these into three attention modules respectively. The purpose is to allow the model to focus on the interaction information between the transformed features and the modality. Specifically, the model learns unimodal features through a unimodal self-attention module, using an attention mechanism to focus on the unimodal features to be learned; it learns the interaction information between bimodal features through a bimodal attention module; and it performs modal transformation through a multimodal attention mechanism to learn the interaction information between multimodal features. Finally, the learned features from different modalities are fused using a soft attention mechanism.

[0058] (I) Feature Mapping Based on LSTM

[0059] Multimodal sentiment analysis typically involves three modalities: text (T), audio (A), and visual (V), using... Let R represent the three modalities of the input data, where R is the real number field, l {T,A,V} d represents the length of the feature sequence. {T,A,V} This represents the feature dimension. Data from the three modalities is fed into a Long Short-Term Memory (LSTM) network for feature mapping.

[0060]

[0061] Where h {T,A,V} Let be the size of the LSTM hidden layer, and d be the dimension of the common features. Based on the consistency characteristics of multimodal data, it is assumed that the three modalities represent the same sentiment bias, thus there exists a common subspace V, and this subspace is assumed to have a set of basis vectors ε1, ε2, ..., ε. n Data from all three modalities can be uniquely represented by this set of bases, that is, by using LSTM to map the features of text, audio and visual modalities to the same subspace.

[0062] (II) Interaction Information Extraction Based on Attention Mechanism

[0063] (1) Single-modal information extraction

[0064] Assume the text feature is F T ∈R l×d The audio feature is F A ∈R l×d Visual features are F V ∈R l×d For three modalities of data, three unimodal self-attention modules are required. The attention feature matrices for text, audio, and vision are calculated separately and then represented using Att. T Att A Att V The specific calculation method for the self-attention feature matrix of a single modality is as follows, taking audio data as an example:

[0065]

[0066]

[0067] O A =N A ·F A

[0068] Att A =O A ⊙F A

[0069] Among them, M A The information matrix N represents the audio data. A (i,j) represents the correlation between the i-th feature and the j-th feature of the audio modality, O A Att represents the attention representation matrix of the audio data. A represents the attention feature matrix of the audio data, and ⊙ represents the Hadamard product.

[0070] Similarly, the attention matrices Att for text and vision are calculated using a similar method. T and Att V .

[0071] (2) Extraction of bimodal interaction information

[0072] In the bimodal attention module, the features of the three modalities are combined pairwise to calculate the attention feature matrices between the two modalities: the attention feature matrix TA for text and audio, the attention feature matrix TV for text and vision, and the attention feature matrix AV for audio and vision. The specific calculation method is as follows, taking the text and vision attention matrix as an example:

[0073] First, calculate the interaction information matrices M1, M2 ∈ R for the bimodal interaction. l×l The attention distribution matrix N1, N2 ∈ R of the bimodal mode is calculated using the softmax function. l×l N A (i,j) represents the correlation between the i-th text sequence feature and the j-th visual sequence feature. Then, the attention distribution matrix is ​​multiplied by the input features to obtain the attention representation matrix O1,O2∈R. l×d Then, the attention information matrix A1, A2 ∈ R is obtained using the Hadamard product. l×d After concatenation, the attention feature matrices F of the text and vision are obtained. TV ∈R l×2d Finally, fully connected (FC) layers are used to fuse the features, resulting in the final attention feature matrix TV∈R. l×d

[0074]

[0075]

[0076]

[0077]

[0078] O1=N1·F T

[0079] O2=N2·F V

[0080] A1=O1⊙F V

[0081] A2=O2⊙F T

[0082] F TV =concat[A1,A2]

[0083] TV = FC(F) TV )

[0084] Similarly, the same method is used to obtain the attention feature matrix TA for text and audio and the attention feature matrix AV for audio and vision.

[0085] (3) Extraction of multimodal interaction information

[0086] In the multimodal attention module, a similar structure is used to convert audio and visual features into text features. These converted audio and visual features are then used to supplement the text features, and finally, the converted features are fused. Specifically, the features are converted into audio and video features. Finally, a three-modal fused feature matrix (F) is obtained. AV→T ,F TV→A ,F TA→V Modality transformation is performed by constraining different modalities. The following example illustrates the conversion of audio and visual features into text features, with the specific calculation process described below.

[0087] like Figure 3 As shown, the Query in the self-attention mechanism is defined as... Key is Value is Where F T ,F A ∈R l×d For text and audio features, The weight matrix is ​​used to represent the transformation from audio features to text features.

[0088]

[0089] Where Y T and Q T Having the same length l, d k Using the scaling factor, this self-attention mechanism transforms audio and visual features into text features. Similarly, visual features are then transformed into text features, and the transformed audio and visual features are fed into the Transformer for fusion. The overall calculation method of the multimodal attention module is as follows:

[0090] First, a Transformer is used for feature transformation, taking the conversion of audio and video features into text features as an example. Then, a fully connected (FC) layer is used to concatenate and fuse the transformed features.

[0091] X AV =Transformer(F A ,F V ,F T )

[0092] X VA =Transformer(F V ,F A ,F T )

[0093]

[0094] Where F T ,F A ∈R l×d X is a feature for both text and audio. A ,X V ∈R l×d To convert good audio and visual features, The feature concatenation operation is represented by FC, which stands for fully connected layer. AV→T ∈R l×d This is the feature matrix after transformation and fusion.

[0095] (III) Feature Fusion Based on Soft Attention Mechanism

[0096] The obtained single-modal attention feature matrix (Att) T Att A Att V ), bimodal attention feature matrix (TA, TV, AV) and trimodal fusion feature matrix (F AV→T ,F TV→A ,F TA→V The nine features are fused together, and then a soft attention mechanism is used for information filtering. The specific calculation process is as follows:

[0097] Input: Single-modal attention feature matrix Att T Att A Att V Two-modal attention feature matrices TA, TV, and AV; three-modal fusion feature matrix F. AV→T ,F TV→A ,F TA→V ;

[0098] Output: Characteristic matrix F scored ;

[0099]

[0100]

[0101]

[0102]

[0103] F = FC(F) sbt )

[0104] U = Tanh(FW1)·W2;

[0105] att = softmax(U);

[0106] F scored =F⊙att;

[0107] return F scored ;

[0108] First, matrix addition is used to fuse the feature matrices obtained from the three modules respectively; then, a fully connected layer is used to transform the dimensions of the fused bimodal and trimodal feature matrices so that the feature matrices of different modules can be fused; then, matrix addition is used again to fuse the feature matrices of the three modules, and finally, a soft attention mechanism is used to obtain the post-feature matrix.

[0109] Where F self ,F bi_att F tri_att ,F,F scored ∈R l×d F self ∈R l×d It is the single-modal attention feature matrix after fusion through fully connected layers, F bi_att ∈R l×d It is the dual-modal attention feature matrix after fusion through fully connected layers, F tri_att ∈R l×d It is the multimodal attention feature matrix after fusion through fully connected layers. F sbt ∈R l×3d Let F ∈ R be the feature matrix after fusing different modalities. l×d It is the feature matrix obtained after transformation by a fully connected layer, F scored ∈R l×d This is the feature matrix obtained after applying a soft attention mechanism. FC stands for fully connected layer, W1∈R l×l ,W2∈R l×1 This is the weight matrix. For feature concatenation, Tanh and softmax are activation functions.

[0110] (iv) Evaluation indicators and parameter settings

[0111] (1) Evaluation indicators

[0112] The evaluation metrics are binary classification accuracy (Acc2), F1 score (F1), 7-class classification accuracy (Acc7), mean absolute error (MAE), and the correlation coefficient between the model's predictions and the true labels (Corr). The specific calculation methods are as follows:

[0113] The Mean Absolute Error (MAE) is the average of the absolute differences between predicted and actual values, representing the average error of the model's predictions. The calculation formula is as follows:

[0114]

[0115] Where n is the number of samples, y i Let i be the true value of the i-th sample. Let be the predicted value for the i-th sample.

[0116] Pearson correlation coefficient: Used to measure the linear relationship between two variables. Its value ranges from -1 to 1, where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no linear relationship. The formula for calculating the Pearson correlation coefficient is as follows:

[0117]

[0118] Where cov(X,Y) represents the covariance of X and Y, and td(X) and std(Y) represent the standard deviations of X and Y, respectively.

[0119] (2) Parameter settings

[0120] This experiment was conducted on an Ubuntu system with the following configuration: Ubuntu 18.04.6, CPU Intel Xeon E5-2600 v4, GPU Tesla V100-16GB, CUDA 11.0, Python 3.8, and PyTorch 1.8. The proposed model was trained using the Adam optimizer with an initial learning rate of 0.001, a batch size of 32, and 40 epochs. The LSTM module in the model has 1 layer and 120 hidden layers. The model has 2 Transformer modules, 4 attention heads, and the dropout probability (attn_dropout) is set to 0.1 in the attention mechanism, 0.1 in the ReLU activation function, 0.1 in the residual connection, and 0.25 in the embedding vector dropout. Specific parameter settings are shown in Table 1.

[0121] Table 1 Experimental parameter settings

[0122]

[0123] (V) Ablation Test

[0124] (1) Modal ablation

[0125] To demonstrate that multimodal features are effective in improving the accuracy of sentiment analysis, ablation experiments were conducted on feature inputs for seven modality combinations: unimodal (T, A, V), bimodal (T+A, T+V, A+V), and trimodal (T+A+V). For unimodal inputs, BiLSTM was used for feature extraction, and the extracted features were then fed into a fully connected layer for sentiment classification. For bimodal inputs, a bimodal fusion attention module was used for feature extraction, and the extracted features were then fed into a fully connected layer for sentiment classification. For trimodal inputs, a multimodal fusion sentiment analysis model based on multi-attention was used for sentiment classification. Modality ablation experiments were conducted on the MOSI and MOSEI datasets, using Acc2 and F1 scores as evaluation metrics.

[0126] Table 2. Results of Modal Ablation Experiment

[0127]

[0128] The modal ablation experiments are shown in Table 2. On the MOSI dataset, the model with input T+A+V performed best, achieving Acc2 and F1 scores of 79.6% and 79.6%, respectively. In bimodal input, the model with input T+V performed best, achieving Acc2 and F1 scores of 76.4% and 76.0%, respectively. In unimodal input, the model with input T performed best, achieving Acc2 and F1 scores of 73.8% and 73.6%, respectively. On the MOSEI dataset, the model with input T+A+V performed best, achieving Acc2 and F1 scores of 85.3% and 85.5%, respectively. In bimodal input, the model with input T+V performed best, achieving Acc2 and F1 scores of 83.0% and 83.9%, respectively. In unimodal input, the model with input T performed best, achieving Acc2 and F1 scores of 78.8% and 78.8%, respectively. The experimental results above demonstrate that fusing multimodal information can improve the accuracy of the final prediction, validating the necessity of fusing multimodal information. In bimodal input, the model incorporating text modality performs better. In unimodal input, the model with text modality as input performs best, proving that text modality features have the greatest impact on sentiment classification results, containing the most significant sentiment features. However, on the MOSI dataset, the model with A+V input has lower accuracy than the model with V input. This is partly due to the smaller size of the MOSI dataset, leading to poor model generalization. On the other hand, in unimodal input, the model with A input performs the worst, reflecting the insignificant sentiment features of audio data, resulting in the A+V model performing worse than the V model on the MOSI dataset. However, on the MOSEI dataset, the model with A+V input outperforms models with either A or V input, proving the effectiveness of the bimodal fusion mechanism.

[0129] (2) Model Ablation

[0130] To further analyze the performance of the proposed model in mining multimodal interaction information, four sets of comparative experiments were designed, as follows:

[0131] Unimodal: Retains the model's unimodal attention module and attention mechanism module, but discards the bimodal attention module and multimodal attention module.

[0132] Bimodal: Retains the model's bimodal attention module and attention mechanism module, while discarding the unimodal attention module and multimodal attention module.

[0133] Trimodal: Retains the model's multimodal attention module and attention mechanism module, but discards the unimodal attention module and bimodal attention module.

[0134] FC: When fusing features, the attention mechanism is not used; a fully connected layer is used instead of the attention module.

[0135] in, Figure 4 The results of the MOSI model ablation experiment are as follows. Figure 5 The results of the MOSEI model ablation experiments show that for any dataset, removing any one of the unimodal, bimodal, or multimodal attention modules degrades model performance. For the multimodal attention module, since it focuses on the interaction information between the three modalities, compensating for the lack of bimodal interaction information, this module has the greatest impact on model performance. For the feature fusion module, if only a simple fully connected layer is used for feature fusion, abandoning the attention mechanism, although the model can fuse more features, these rich features also contain information redundancy and noise interference, affecting the final model's performance.

[0136] (vi) Comparative Experiment

[0137] This invention proposes a multi-attention-based multimodal sentiment analysis model and compares it with several existing multimodal sentiment analysis models. These benchmark methods include:

[0138] EF_LSTM: First, the initial inputs of the three modalities are concatenated and fused using an early fusion method. Then, the concatenated features are fed into a bidirectional LSTM network to capture long-range dependencies in the sequence.

[0139] TFN: A tensor-based multimodal data fusion method that applies the outer product of unimodal embeddings to jointly learn the interactions of unimodal, bimodal, and trimodal modes.

[0140] LMF: An efficient low-rank multimodal fusion method that uses mode-specific factors to model each mode and integrates them into a common low-rank matrix.

[0141] MFN: A method that uses memory fusion networks to introduce memory modules to capture interactions between different views and enable effective information exchange between them.

[0142] MULT: An end-to-end approach to handle long-distance dependencies between cross-modal elements by learning multimodal representations through the use of a cross-modal Transformer to transform source modalities into target modalities.

[0143] Table 3. Results of MOSI Comparison Experiment

[0144]

[0145]

[0146] Table 4. Results of MOSEI Comparison Experiment

[0147]

[0148] Comparative experiments were conducted, and the multimodal data fusion model proposed in this invention was compared with other baseline models on the MOSI and MOSEI datasets. The experimental results are shown in Tables 3 and 4. On the MOSI dataset, the proposed model showed slight improvements over the MULT model in all metrics. Specifically, under the same metrics, the proposed model improved the MULT model by 1.8%, 0.3%, and 1.3% in Acc7, Acc2, and F1 scores, respectively, while reducing the MAE score by 0.037 and improving the Corr score by 0.002. However, the EF_LSTM model achieved the best results in the Acc7 score. On the MOSEI dataset, the proposed model significantly outperformed the MULT model under the same metrics. Specifically, the proposed model improved the MULT model by 4.5%, 5.1%, and 5.0% in Acc7, Acc2, and F1 scores, respectively, while reducing the MAE score by 0.045 and improving the Corr score by 0.031. Based on the above experimental results, the model proposed in this invention achieved good results on the MOSEI dataset, but only showed a slight improvement on other MOSEI datasets. This is likely because the MOSEI dataset is relatively small, leading to overfitting during training and thus reducing the model's generalization ability.

[0149] The model proposed in this invention performs well on the MOSEI dataset, but shows only a slight improvement on the MOSI dataset. This is because the MOSI dataset is relatively small, leading to overfitting during training on this dataset and thus reducing the model's generalization ability.

[0150] To address the challenge of fully mining intermodal interaction information, this invention proposes a multimodal data fusion model based on multi-attention. This model utilizes three attention modules to extract intermodal interaction information and employs a soft attention mechanism to filter redundant information. The invention also compares this model with existing models, and experimental results show that this model performs better on sentiment analysis tasks. Furthermore, modality ablation experiments demonstrate that multimodal data provides more accurate sentiment classification results compared to single-modal data. Model ablation experiments further prove that the three attention modules proposed in this invention can effectively mine intermodal interaction information.

[0151] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for establishing a multimodal data fusion model based on a multi-attention mechanism, characterized in that, Includes the following steps: S1. Acquisition Steps: Acquire multimodal data; S2. Extraction steps: Input the acquired multimodal data into multiple independent LSTMs for feature transformation, and then use multiple attention modules to extract the interaction information between multimodalities to obtain the features of multimodal interaction information; S3. Fusion Step: Perform feature fusion on the features of the interaction information between multiple modalities to obtain the post-feature matrix; S4. Construction steps: Based on the post-feature matrix, construct a multimodal data fusion model; S1 acquires multimodal data, including but not limited to text data, audio data, and visual data; The specific content of feature fusion for multimodal interaction information in S3 is as follows: First, matrix addition is used to fuse the single-modal attention feature matrix, the dual-modal attention feature matrix, and the trimodal fusion feature matrix respectively; Secondly, a fully connected layer is used to transform the dimensions of the fused bimodal attention feature matrix and the trimodal fusion feature matrix, so that the feature matrices of different modalities can be fused. Again, matrix addition is used to process the single-modal attention feature matrix. , , Bimodal attention feature matrix , , Three-modal fusion feature matrix , , To merge; Finally, the post-feature matrix is ​​obtained using a soft attention mechanism; in It is a single-modal attention feature matrix after fusion through fully connected layers. It is a dual-modal attention feature matrix after fusion through fully connected layers. It is a multimodal attention feature matrix fused through fully connected layers. The feature matrix after fusing different modalities. It is the feature matrix obtained after transformation by a fully connected layer. It is the feature matrix obtained after applying the soft attention mechanism. It is a fully connected layer. This is the weight matrix. For feature splicing operations, This is the activation function.

2. The method for establishing a multimodal data fusion model based on a multi-attention mechanism according to claim 1, characterized in that, This also includes feature mapping of the multimodal data using LSTM before the S2 extraction step: in, For text data, For audio data, For visual data, use To represent the input data in three modalities, where For the real number field, Indicates the length of the feature sequence. Represents the feature dimension. for Hidden layer size, It is the dimension of common features; That is, through Features from text, audio, and visual modalities are mapped to the same subspace.

3. The method for establishing a multimodal data fusion model based on a multi-attention mechanism according to claim 1, characterized in that, S2 utilizes multiple attention modules to extract multimodal interaction information, specifically including: unimodal interaction information extraction, bimodal interaction information extraction, and multimodal interaction information extraction. Among them, the extraction of unimodal interaction information yields the unimodal attention feature matrix. ; Bimodal interaction information extraction yields a bimodal attention feature matrix. ; Multimodal interaction information extraction yields a trimodal fusion feature matrix. .

4. The method for establishing a multimodal data fusion model based on a multi-attention mechanism according to claim 1, characterized in that, It also includes using ablation experiments to validate the established multimodal data fusion model.

5. The method for establishing a multimodal data fusion model based on a multi-attention mechanism according to claim 4, characterized in that, Ablation experiments include modal ablation and model ablation.

Images