A robust multimodal sentiment analysis method
By translating the missing modal features through the MGT model and fusing multimodal information with the MUT model, the problems of modality missing and misalignment in multimodal sentiment analysis are solved, and the accuracy of sentiment analysis is improved.
Patent Information
- Application Number
- CN202310248746.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-09
- Publication Date
- 2025-09-05
- Estimated Expiration
- 2043-03-09
AI Technical Summary
Existing multimodal sentiment analysis methods are difficult to effectively solve the problems of missing modalities and non-aligned multimodal sequences, resulting in a decline in sentiment analysis performance.
The MGT model is used for modal translation. The missing modal features are generated through the trained modal translation module. The MUT model is used for multimodal information fusion. A special sequence MMSeq with all-zero initialization is introduced to improve robustness.
At the same time, it overcomes the problems of modality missing and multimodal sequence misalignment, and improves the accuracy of sentiment analysis.
Smart Images
Figure CN116522107B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of deep learning sentiment analysis, and in particular relates to a robust multimodal sentiment analysis method. Background Art
[0002] In the real world, people often communicate in multiple modalities. For example, during a conversation, the information we perceive includes not only the words spoken but also their facial expressions, body movements, and the pitch of their voice. Precisely because of this multimodal information, we can better understand the other person's intentions.
[0003] Multimodal sentiment analysis combines information from multiple modalities to determine the emotional nature of a task, which is crucial for human-computer interaction. However, enabling machines to accurately identify people's emotions remains a significant challenge. A significant body of work exists in the field of multimodal sentiment analysis, much of which focuses on multimodal fusion methods. However, it overlooks the critical issues of modality loss and misaligned multimodal sequences, two key challenges faced in practical applications. Therefore, effectively and simultaneously addressing these two currently challenging issues in multimodal sentiment analysis remains of great theoretical and practical value.
[0004] With the continuous maturity and development of various neural network models and computing resources, deep learning methods have achieved remarkable results in various fields. In recent years, multimodal sentiment analysis frameworks have emerged in large numbers. In addition to the frameworks dedicated to multimodal fusion methods mentioned above, there have also been some works that target the missing of one or more modalities. These mainly use methods similar to machine translation to translate the missing modalities, thereby improving the model's robustness to modality loss. Furthermore, for non-aligned multimodal sequences, a number of works have emerged that use attention mechanisms to align multimodal sequences, making end-to-end multimodal sentiment analysis possible and eliminating the previously complex task of modal sequence alignment. However, these works either address only the problem of missing modalities or only the problem of non-aligned multimodal sequences. When both are present, their frameworks are no longer competent for sentiment analysis. Summary of the Invention
[0005] The purpose of this application is to provide a robust multimodal sentiment analysis method to overcome the problem that previous work only addresses one aspect, so as to further improve the final sentiment analysis performance.
[0006] In order to achieve the above objectives, the technical solutions of this application are as follows:
[0007] A robust multimodal sentiment analysis method, comprising:
[0008] Obtain at least one of the three modal feature sequences of the object to be predicted: text, audio, and video. For the missing modal feature sequences, generate the missing modality through the trained modality translation module, thereby obtaining three modal feature sequences;
[0009] The three modal feature sequences are input into the trained emotion prediction module to obtain the emotion prediction results of the object to be predicted.
[0010] Furthermore, the robust multimodal sentiment analysis method further includes a training modality translation module, wherein the training modality translation module includes:
[0011] Map the three modal feature sequences of text, audio, and video in the training samples to the same feature space through a fully connected layer respectively;
[0012] For the target modal sequence to be translated, embed a zero-initialized special character [multi] at its head and delete the last character of the modal sequence;
[0013] Add position encoding and modality type encoding to multimodal features respectively;
[0014] The multimodal features with position coding and modality type coding are spliced in the temporal dimension and input into the multimodal generation transformer to generate multimodal splicing features;
[0015] The features belonging to the target modality in the multimodal splicing features are extracted, mapped back to their original dimensions after passing through the fully connected layer, and the modal sequence of the translated output is obtained. The loss function is then used for training to obtain a trained modal translation module.
[0016] Furthermore, the multi-modal generation converter is expressed as follows:
[0017]
[0018]
[0019]
[0020] Among them, MGT represents multimodal generative transformer, MGM represents multimodal generative mask, LN represents layer normalization, FFN represents feedforward neural network, represents the input of the first layer of MGT, X G Represents the multimodal features after splicing the time series dimension, Represents the output of the mth layer of MGT;
[0021] Furthermore, MGM can be calculated by the following formula:
[0022]
[0023]
[0024] in, represents the output of MGM, X represents the input of MGM, Represent the weight of the network respectively; d k =d v =d / h, d represents the feature dimension, h represents the number of heads in the multi-head attention mechanism, is the attention matrix, the subscript ij represents the matrix MASK G The i-th row and j-th column of , when its value is 0, it means that the character at i can pay attention to the character at j, when its value is -∞, it means that the character at i cannot pay attention to the character at j.
[0025] Furthermore, the three modal feature sequences are input into the trained emotion prediction module to obtain the emotion prediction results of the object to be predicted, including:
[0026] The three modal feature sequences are respectively subjected to one-dimensional convolution operations to map the three modal feature sequences into the same feature space;
[0027] The three modal feature sequences are passed through the gated recurrent unit respectively to introduce inductive bias;
[0028] A special sequence MMSeq initialized with all zeros is introduced, and the special sequence MMSeq and the three modal feature sequences with inductive bias are added to position encoding and modality type encoding respectively;
[0029] The special sequence MMSeq with position coding and modality type coding is concatenated with the three modal feature sequences in the temporal dimension and processed by the multimodal understanding transformer to obtain the transformed multimodal concatenation feature.
[0030] The special sequence MMSeq in the output of the multimodal understanding transformer is taken out as the final unified multimodal representation, and the sentiment prediction result is obtained after passing it through the fully connected layer.
[0031] Furthermore, the multimodal understanding transformer is expressed as follows:
[0032]
[0033]
[0034]
[0035] Among them, MUT represents multimodal understanding transformer, MUM represents multimodal understanding mask, LN represents layer normalization, FFN represents feedforward neural network, represents the input of the first layer of MUT, X U Represents the multimodal features after splicing the time series dimension, represents the output of the mth layer of MUT;
[0036] Furthermore, MUM can be calculated by the following formula:
[0037]
[0038]
[0039] in, represents the output of MUM, X represents the input of MUM, Represent the weight of the network, d k =d v =d / h, d represents the feature dimension, h represents the number of heads in the multi-head attention mechanism, is the attention matrix, the subscript ij represents the matrix MASK U The i-th row and j-th column of , when its value is 0, it means that the character at i can pay attention to the character at j, when its value is -∞, it means that the character at i cannot pay attention to the character at j.
[0040] This application proposes a robust multimodal sentiment analysis method that uses the MGT model to unify the encoding and decoding architecture used in modal translation and the MUT model to fuse multimodal information. Furthermore, to further achieve a unified multimodal representation, a special sequence, MMSeq, initialized with all zeros, is introduced. This technical solution not only overcomes the problems of modality loss and multimodal sequence misalignment, but also improves the accuracy of sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 Flowchart of the robust multimodal sentiment analysis method for this application.
[0042] Figure 2 Schematic diagram of the modal translation module and sentiment prediction module according to an embodiment of the present application;
[0043] Figure 3 This is the attention mechanism diagram for all translation directions of MGT used in this application;
[0044] Figure 4 Diagram of the attention mechanism used in MUT for this application. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0046] In one embodiment, Figure 1 As shown in the figure, a robust multimodal sentiment analysis method is proposed, which includes:
[0047] Step S1: obtaining at least one of the three modal feature sequences of the object to be predicted, namely, text, audio, and video. For the missing modal feature sequences, generating the missing modalities through a trained modal translation module, thereby obtaining three modal feature sequences;
[0048] Step S2: Input the three modal feature sequences into the trained emotion prediction module to obtain the emotion prediction result of the object to be predicted.
[0049] Specifically, the robust multimodal sentiment analysis method further includes a training modality translation module, and the training modality translation module includes:
[0050] Step S1.1: Map the three modal feature sequences of text, audio, and video in the training sample to the same feature space through a fully connected layer respectively.
[0051] Training samples are obtained from various text, audio, and video data posted by users on the platform. After feature extraction, corresponding multimodal features can be obtained. Text features are extracted by the Glove pre-trained model, audio features are extracted by COVAREP, and video features are extracted by Facet.
[0052] The multimodal feature data is represented as:
[0053]
[0054]
[0055]
[0056] Where T {L,A,V} and d {L,A,V} Represent the sequence length and feature dimension of each modality respectively, Represents the features of each time step within each modality (for text L, the features of each time step are the features of each word; for audio A, the features of each time step are the features of each audio sampling point; for video V, the features of each time step are the features of each video frame).
[0057] Then, the feature sequences of the three modalities of text, audio, and video are passed through the fully connected layer respectively so that the features of the three modalities are mapped to the same feature space, as shown in the following formula:
[0058]
[0059] ReLU(x)=max(0,x)
[0060] in, and Represent the learnable weight matrix and offset respectively. d represents the common feature dimension held by the three modalities after mapping through the fully connected layer. ReLU represents the activation function. Thus, we get
[0061] It should be noted that when training the modal translation module, each modal feature sequence needs to be input into the fully connected layer and then input into the subsequent MGT for processing. Figure 2 Shown are examples of input text and audio modalities (video modality is missing) and translation into video modality.
[0062] Step S1.2: For the target modal sequence to be translated, embed a zero-initialized special character [multi] at its head, and delete the last character of the modal sequence.
[0063] This embodiment Figure 2 As shown, language and audio are used as source modalities (existing modalities) and video is used as target modality (missing modality) for explanation. The same is applicable when there is only one existing modality or other modalities are used as target modalities, and will not be repeated below.
[0064] This embodiment requires the target modal sequence to be shifted one position to the right because during training, the correct answer of the translation output needs to be informed, so that in the subsequent reasoning stage, the modal translation module can correctly translate according to the previous text, one time step at a time, until the entire target modal sequence is translated.
[0065] For example, for the text "I LOVE YOU", shifting one position right and deleting the last character becomes " <bos>ILOVE", in which <bos>Indicates the beginning of a sequence. During training, enter " <bos>I LOVE", and the expected output is "I LOVE YOU", in order to train the network from" <bos>" to "I", from "I" to "LOVE", from "LOVE" to "YOU". Here, the function of the special character [multi] is the same as <bos>Similar, both represent a meaningless sequence start character.
[0066] Therefore, the video sequence of the target modality has the following operations:
[0067]
[0068] Step S1.3: Add position coding and modality type coding to the multimodal features respectively.
[0069] In this embodiment, position coding and modality type coding are added to the multimodal features respectively, and the formula is as follows:
[0070]
[0071]
[0072]
[0073] Among them, PE represents position encoding, which marks the index of each time step in each modal sequence, such as PE(L)=[1,…,T L ],PE(A)=[1,…,T A ],PE(V)=[1,…,T V ]. ME represents the modality type encoding, which marks the type of each modality, such as ME(L)=0, ME(A)=1, ME(V)=2.
[0074] Step S1.4: splice the multimodal features with position coding and modality type coding in the temporal dimension and input them into the multimodal generation transformer to generate multimodal splicing features.
[0075] This application adds position encoding and modality type encoding to multimodal features, so that the Multimodal Generation Transformer (MGT) can perceive sequence information and modality category information.
[0076] MGT mainly consists of N layers of multimodal generation masks (MGM). Specifically, for the input For example, the MGT layer m can be calculated by the following formula:
[0077]
[0078]
[0079]
[0080] Among them, MGT represents multimodal generative transformer, MGM represents multimodal generative mask, LN represents layer normalization, FFN represents feedforward neural network, represents the input of the first layer of MGT, X G Represents the multimodal features after splicing the time series dimension, Represents the output of the mth layer of MGT.
[0081] Furthermore, MGM can be calculated by the following formula:
[0082]
[0083]
[0084] in, represents the output of MGM, X represents the input of MGM, Represent the weights of the network respectively. If the number of heads in the multi-head attention mechanism is h, then here, d k =d v =d / h, d represents the feature dimension. Attention matrix The subscript ij determines whether each pair of characters can form attention. The subscript ij represents the i-th row and j-th column of the matrix MASKG. When its value is 0, it means that the character at i can pay attention to the character at j. When its value is -∞, it means that the character at i cannot pay attention to the character at j.
[0085] For all modal translation directions, this embodiment has 9 types of MASK G ,like Figure 3 shown.
[0086] For the case where one mode is missing, as shown in the first row, we choose to make each time step of the existing mode pay attention to itself and the [multi] character; let the [multi] character pay attention to all the information of the two existing modes and pay attention to itself; let each time step of the missing mode pay attention to itself and all the previous information (all the previous information refers to all the time steps before the [multi] character and itself, so MASK G The text is arranged in a staircase in the lower right corner. The reason why you can't see the information after it is because this is the missing mode to be translated. The text after it is translated based on the previous text. Seeing the information after it in advance is equivalent to information leakage).
[0087] Similarly, for the case where two modalities are missing, there is a similar process (the existing modality focuses on itself and [multi], [multi] focuses on the existing modality and itself, and the missing modality focuses on the previous text). However, translating two missing modalities at the same time is still a very big challenge. Therefore, for example, if there is a text modality L, and we want to translate the audio modality A and the video modality V, this embodiment will choose to use two MGTs to complete this task, namely MGT L→A and MGT L→V , rather than using a separate MGT L→(A,V) .
[0088] Step S1.5: Extract the features belonging to the target modality from the multimodal concatenation features, map them back to their original dimensions after passing through the fully connected layer, obtain the modal sequence of the translated output, and train them using the loss function to obtain a trained modal translation module.
[0089] For MCT (L,A)→V Output Z [N] , take out the video sequence belonging to the target modality and record it as In order to be able to have a feature dimension of d Remap back to feature dimension d V The modal sequence of is calculated as follows:
[0090]
[0091] in, and Represent the learnable weight matrix and bias respectively.
[0092] When training the modality translation network, this embodiment uses MSE (mean squared error) loss:
[0093]
[0094] The use of loss functions to train network models is a relatively mature technology in this field and will not be discussed here.
[0095] Therefore, in the inference phase, we can start from [multi], decode each character one by one, and finally translate and generate the complete translated video modality. As shown in the following formula:
[0096]
[0097]
[0098] in, It represents the maximum value of the optimized probability p(·) and returns X that maximizes the probability p(·) V The value of (ie: ).
[0099] In a specific embodiment, the three modal feature sequences are input into a trained emotion prediction module to obtain the emotion prediction result of the object to be predicted, including:
[0100] Step 2.1: Perform one-dimensional convolution operations on the three modal feature sequences respectively to map them into the same feature space.
[0101] In this embodiment, for the three modal feature sequences of text, audio and video of the object to be predicted, if the modal feature sequence of a certain modality is missing, the modal feature sequence is translated by the modal translation module. In other words, regardless of whether the modality is missing, the input to the emotion prediction module will be three complete modal sequences X. L 、X A and X V (If the modality is not missing, the original three modal sequences are input; if it is missing, the missing modalities are first supplemented through the previous modal translation network, and then the three modal sequences are input). The calculation is as follows:
[0102]
[0103] Among them, k {L,A,V} represents the convolution kernel size used when performing one-dimensional convolution operations on the three modalities, and d represents the common feature dimension held by the three modalities after mapping by one-dimensional convolution.
[0104] Step 2.2: Pass the three modal feature sequences through the gated recurrent unit respectively to introduce inductive bias.
[0105] The Gated Recurrent Unit (GRU) is a variant of the Recurrent Neural Network (RNN) used to process time series data. The purpose is to further extract high-level semantic features of each modality. The advantage is that the GRU has an inductive bias of sequentiality and time invariance, that is, it has assumed that there is a connection between the time steps in the sequence (so in this respect, the GRU can better extract features when facing smaller-scale data, while the transformer has no inductive bias and requires a large amount of data input to obtain better results. Therefore, the use of the GRU in this embodiment is on the one hand to better process small-scale data, and on the other hand to make the entire network appear very lightweight).
[0106] The gated recurrent unit is represented by the following formula:
[0107]
[0108] Step 2.3: Introduce the special sequence MMSeq initialized with all zeros, and add the special sequence MMSeq and the three modal feature sequences that introduce inductive bias into position encoding and modal type encoding respectively.
[0109] Introducing a special sequence of all-zero initialization Then, position encoding and modality type encoding are added to the four sequences respectively, so that the Multimodal Understanding Transformer (MUT) can perceive the sequence information and modality category information, as shown in the following formula:
[0110]
[0111]
[0112]
[0113]
[0114] Among them, PE represents position encoding, which marks the index of each time step in each modal sequence, such as PE(L)=[1,…,T L ],PE(A)=[1,…,T A ],PE(V)=[1,…,T V ],PE(M)=[1,…,T M ]. ME represents the modality type encoding, which marks the type of each modality, such as ME(L)=0, ME(A)=1, ME(V)=2, ME(M)=3.
[0115] Step 2.4: The special sequence MMSeq with position coding and modality type coding and the three modal feature sequences are spliced in the time series dimension, and processed by the multimodal understanding transformer to obtain the transformed multimodal splicing features.
[0116] The Multimodal Understanding Transformer (MUT) is mainly composed of the Multimodal Understanding Mask (MUM). Specifically, for the input For example, the MUT layer m can be calculated by the following formula:
[0117]
[0118]
[0119]
[0120] Among them, MUT represents multimodal understanding transformer, MUM represents multimodal understanding mask, LN represents layer normalization, FFN represents feedforward neural network, represents the input of the first layer of MUT, X U Represents the multimodal features after splicing the time series dimension, Represents the output of the mth layer of MUT.
[0121] Furthermore, MUM can be calculated by the following formula:
[0122]
[0123]
[0124]
[0125] in, represents the output of MUM, X represents the input of MUM, Represent the weights of the network respectively. If the number of heads in the multi-head attention mechanism is h, then here, d k =d v =d / h, d represents the feature dimension. Attention matrix Determines whether each pair of characters can form attention. The subscript ij represents the matrix MASK U The i-th row and j-th column of , when its value is 0, it means that the character at i can pay attention to the character at j, when its value is -∞, it means that the character at i cannot pay attention to the character at j.
[0126] For multimodal understanding tasks, MASK U like Figure 4 As shown, for MMSeq (i.e. Figure 4 The M in the text sequence L, audio sequence A, and video sequence V) focuses on all time steps of the text sequence L, audio sequence A, and video sequence V, enabling the special sequence MMSeq to aggregate information from the three modalities of text, audio, and video. For the text sequence L, it first focuses on itself, performing self-attention to further extract its own high-order features. Then, it also focuses on the special sequence MMSeq, collecting the multimodal information contained in MMSeq. For the audio sequence a and video sequence V, similar to the text sequence L, they focus on both themselves and MMSeq, performing self-attention to extract their own high-order semantics while also collecting the multimodal information contained in MMSeq.
[0127] Step 2.5: Take out the special sequence MMSeq from the output of the multimodal understanding transformer as the final unified multimodal representation, and pass it through the fully connected layer to obtain the sentiment prediction result.
[0128] In this embodiment, the special sequence MMSeq in the output of the multimodal understanding transformer is extracted and recorded as As the final unified multimodal representation, it is passed through a fully connected layer to obtain the emotion prediction result.
[0129] It should be noted that when training the sentiment prediction module, the L1 (mean absolute error) loss is used:
[0130]
[0131] The use of loss functions to train network models is a relatively mature technology in this field and will not be discussed here.
[0132] This application also experimentally validates the technical solution of this application. The experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets, comparing them with current mainstream multimodal sentiment analysis algorithms in the presence of missing modalities and non-aligned multimodal sequences. The performance was measured using binary classification accuracy (Acc), with higher accuracy being preferred.
[0133] The experimental results are shown in Table 1, Table 2 and Table 3:
[0134]
[0135] Table 1
[0136]
[0137] Table 2
[0138]
[0139] Table 3
[0140] Tables 1, 2, and 3 show the test results for the CMU-MOSI dataset in the case of missing modalities, the CMU-MOSI dataset in the case of non-aligned multimodal sequences, and the CMU-MOSEI dataset in the case of non-aligned multimodal sequences, respectively. Comparison of the three tables shows that the proposed method outperforms other multimodal sentiment analysis algorithms from 2017 to 2021 in terms of multimodal sentiment analysis performance. Therefore, the proposed method has certain advantages over other algorithms.
[0141] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.< / bos> < / bos> < / bos> < / bos> < / bos>
Claims
1. A robust multimodal sentiment analysis method, characterized in that: The robust multimodal sentiment analysis method comprises: Obtain at least one of the three modal feature sequences of the object to be predicted: text, audio, and video. For the missing modal feature sequences, generate the missing modality through the trained modality translation module, thereby obtaining three modal feature sequences; Input the three modal feature sequences into the trained emotion prediction module to obtain the emotion prediction results of the object to be predicted; The three modal feature sequences are input into the trained emotion prediction module to obtain the emotion prediction results of the object to be predicted, including: The three modal feature sequences are respectively subjected to one-dimensional convolution operations to map the three modal feature sequences into the same feature space; The three modal feature sequences are passed through the gated recurrent unit respectively to introduce inductive bias; A special sequence MMSeq initialized with all zeros is introduced, and the special sequence MMSeq and the three modal feature sequences with inductive bias are added to position encoding and modality type encoding respectively; The special sequence MMSeq with position coding and modality type coding is concatenated with the three modal feature sequences in the temporal dimension and processed by the multimodal understanding transformer to obtain the transformed multimodal concatenation feature. The special sequence MMSeq in the output of the multimodal understanding transformer is extracted as the final unified multimodal representation, and the sentiment prediction result is obtained after passing it through the fully connected layer; The multimodal understanding transformer is expressed as follows: Among them, MUM represents multimodal understanding mask, LN represents layer normalization, and FFN represents feedforward neural network. represents the input of the first layer of the multimodal understanding transformer, X U Represents the multimodal features after splicing the time series dimension, represents the output of the mth layer of the multimodal understanding transformer; Furthermore, MUM is calculated by the following formula: in, represents the output of MUM, X represents the input of MUM, Represent the weight of the network, d k =d v =d / h, d represents the feature dimension, h represents the number of heads in the multi-head attention mechanism, is the attention matrix, the subscript ij represents the matrix MASK U The i-th row and j-th column of , when its value is 0, it means that the character at i can pay attention to the character at j, when its value is -∞, it means that the character at i cannot pay attention to the character at j.
2. The robust multimodal sentiment analysis method according to claim 1, characterized in that The robust multimodal sentiment analysis method further includes a training modality translation module, wherein the training modality translation module includes: Map the three modal feature sequences of text, audio, and video in the training samples to the same feature space through a fully connected layer respectively; For the target modal sequence to be translated, embed a zero-initialized special character [multi] at its head and delete the last character of the modal sequence; Add position encoding and modality type encoding to multimodal features respectively; The multimodal features with position coding and modality type coding are spliced in the temporal dimension and input into the multimodal generation transformer to generate multimodal splicing features; The features belonging to the target modality in the multimodal splicing features are extracted, mapped back to their original dimensions after passing through the fully connected layer, and the modal sequence of the translated output is obtained. The loss function is then used for training to obtain a trained modal translation module.
3. The robust multimodal sentiment analysis method according to claim 2, characterized in that: The multi-modal generation converter is expressed as follows: Among them, MGM represents multimodal generation mask, LN represents layer normalization, and FFN represents feedforward neural network. represents the input of the first layer of the multimodal generative transformer, X G Represents the multimodal features after splicing the time series dimension, represents the output of the mth layer of the multimodal generative transformer; Furthermore, MGM is calculated by the following formula: in, represents the output of MGM, X represents the input of MGM, Represent the weight of the network respectively; d k =d v =d / h, d represents the feature dimension, h represents the number of heads in the multi-head attention mechanism, is the attention matrix, the subscript ij represents the matrix MASK G The i-th row and j-th column of , when its value is 0, it means that the character at i can pay attention to the character at j, when its value is -∞, it means that the character at i cannot pay attention to the character at j.