Label-based multi-modal data missing completion method and model

Abstract

The invention discloses a label-based multi-modal data missing complementing method. The method comprises the following steps: S1, a label coding module of a multi-modal data missing complementing model adopts a binary vector with a fixed length as a coding label to express the availability of each modal of data; s2, a universal space projection module of the multi-modal data missing complementation model firstly obtains self-correlation public space representation of each modal through linear transformation, and then vertically connects the public space representation and the coding label to form a complete joint representation; and S3, inputting the joint representation into a Transform encoder-decoder network of the multi-modal data missing completion model, and obtaining the multi-modal data after completion. According to the scheme, effective information fusion can be carried out among different data modalities (such as visual sense, auditory sense and text), and efficient data processing performance can be kept even under the condition that part of data is missing.

Description

Technical Field

[0001] The present invention relates to the field of multimodal data fusion, and in particular to a label-based multimodal data missing completion method and model. Background Art

[0002] With the popularization of the Internet and smart devices, the generation and exchange of multimodal content has become increasingly frequent. In this context, multimodal sentiment analysis technology has emerged. It analyzes and integrates data from different information sources (such as text, audio, and video) to accurately extract and analyze human emotions and opinions. Existing research shows that by integrating multiple modal information such as vision, hearing, and text, the accuracy and efficiency of sentiment analysis can be enhanced in a complementary manner. However, this process often faces a significant technical difficulty, namely, data incompleteness. In practical applications, due to various reasons such as technical limitations or privacy protection, data of certain modalities may be incomplete or missing. For example, video images may lose key visual information due to camera occlusion, or audio data may become unavailable due to environmental noise.

[0003] Traditional multimodal sentiment analysis methods usually assume that all modal data are available, which is often not true in practical applications. Therefore, how to effectively process and complete these missing modal data has become a research focus in this field. Previous methods, such as directly ignoring the missing modality or using simple matrix filling techniques, often fail to effectively restore the true nature of the data and may lead to inaccurate sentiment analysis results. More advanced technologies attempt to infer missing data by establishing potential associations between modalities, such as using cascaded residual autoencoders (CRA) and Transformer-based networks to try to solve this problem. Although these methods have improved the ability to handle missing modalities to a certain extent, most of them are still designed around the situation where a single modality is missing, without fully considering the complex scenario where multiple modalities are missing at the same time.

[0004] Therefore, it is particularly urgent to develop a multimodal sentiment analysis system that can effectively deal with multiple missing modal situations. The system needs to be able to handle not only the missing of any single modality, but also the missing of multiple modalities at the same time, ensuring that high sentiment recognition accuracy and stability can be maintained in various incomplete data scenarios. Summary of the invention

[0005] The present invention mainly solves the technical problem that the prior art cannot cope with the situation where multiple modalities are missing at the same time, and provides a label-based multimodal data missing completion method and model that can complete both single-modal and multi-modal missing.

[0006] The present invention aims to solve the above technical problems mainly through the following technical solutions: a label-based multimodal data missing completion method, comprising the following steps:

[0007] S1. The label encoding module of the multimodal data missing completion model uses a fixed-length binary vector as the encoding label to express the availability of each modality of the data;

[0008] S2, the universal space projection module of the multimodal data missing completion model first obtains the autocorrelated common space representation of each modality through linear transformation, and then vertically connects these common space representations and encoded labels to form a complete joint representation;

[0009] S3. Input the joint representation into the Transformer encoder-decoder network of the multimodal data missing completion model to obtain the completed multimodal data.

[0010] Preferably, the step S2 is specifically as follows:

[0011] Based on Transformer's multi-head attention mechanism:

[0012]

[0013] Where Q, K and V are query, key and value respectively, and d is the dimension of the input;

[0014] For a specific modality M (including visual video, auditory audio, and text), multi-head attention is applied to obtain more information from different semantic spaces, denoted as E M :

[0015] E M =MultiHead(Q,K,V)=Connect(head1,head2,...,head h )W O

[0016] Where W O ∈R d*d is the multi-head attention weight matrix, h is the number of heads, head i represents the i-th long position, 1≤i≤h; given input E M , the i-th head i The calculation is as follows:

[0017]

[0018] Therefore, the updated modality representation can be expressed as follows:

[0019] E M=MultiHead(X M , X M , X M )

[0020] X M Represents each modal data;

[0021] Among them, the modality M∈(v(video),a(audio),t(text));

[0022] For each mode M, it is mapped to a new space through a vertical linking operation:

[0023] C v =[W va E v ||W vt E v ]

[0024] C a =[W va E a ||W ta E a ]

[0025] C t =[W vt E t ||W ta E t ]

[0026] Among them, W va is the weight matrix of vision and hearing, which is used to project visual (auditory) information into the common space according to the auditory (visual) mode. vt is the weight matrix of visual and textual information, which is used to project visual (textual) information into the common space according to the textual (auditory) mode. ta is the correlation weight matrix between text and hearing, which is used to project text (auditory) information into the common space according to the auditory (text) mode, and || represents the vertical connection operation;

[0027] Finally, the transformed output of all modes and the encoded label E tag are stacked vertically together to form a comprehensive joint representation: E all =[C v ||C a ||C t ||E tag ].

[0028] Preferably, the training objectives of the multimodal data missing completion model include classification loss, forward difference loss, reverse reconstruction loss and label recovery loss; the calculation formula of the classification loss is:

[0029]

[0030] Where N is the number of samples, y i is the true label of the i-th sample, is the probability distribution predicted by the model;

[0031] The forward difference loss L forward Through pre-training output E pre and Transformer encoder output D out The difference between them is calculated using the Jensen-Shannon (JS) divergence loss formula:

[0032]

[0033] Where D KL is the KL divergence function:

[0034] D KL is the differential loss divergence function. Since there are three modes directly used for classification, there are three different losses.

[0035] The goal of the reverse reconstruction loss is to supervise the reconstruction of the joint common vector; therefore, similar to the forward difference loss, by computing the Transformer decoder output D out and the updated public union representation E all The JS divergence loss between them is obtained, and the reverse reconstruction loss L backward The calculation formula is:

[0036]

[0037] D OUT is the output of the Transformer decoder;

[0038] Among them, labels are attached to mark the missing modalities, hoping that the network can pay more attention to them. In order to better guide the reconstruction of the attached labels, this scheme designs a label recovery loss to guide the process. The reason for choosing the mean absolute error (MAE) loss is that MAE is not very sensitive to outliers with absolute functions. Therefore, MAE is used to calculate E tag and D out The loss between the last four digits of , i.e., the label recovery loss L tag The calculation formula is: In the formula, N is the number of labels, D tag Yes D OUT The last four digits indicate the missing state of the modality.

[0039] The training uses forward transfer and reverse parameter adjustment until the loss is stable, which is the gradient descent method.

[0040] Preferably, in step S3, the overall loss function during the training of the entire model network adopts a dynamic weight adjustment method based on uncertainty to calculate the total loss:. This method assigns a weight related to its uncertainty to each task, aiming to balance the contribution of each task and improve the effect of multi-task learning. First, a corresponding uncertainty parameter σ is defined for each loss function. These parameters reflect the uncertainty of the model for each task, and these uncertainties will be used to calculate the weight of each loss; the specific formula is as follows:

[0041]

[0042] σ cls is the uncertainty parameter of classification loss, σ forward is the uncertainty parameter of the forward difference loss, σ backward is the uncertainty parameter of the reverse reconstruction loss, σ tag The uncertainty parameter for label recovery loss. The uncertainty parameter can be initially set to 1, indicating complete uncertainty. The uncertainty parameter automatically adjusts its weight during training based on the learning of the task.

[0043] A label-based multimodal data missing completion model completes data through the above-mentioned method, including:

[0044] Label encoding module: A fixed-length binary vector is used as the encoding label to express the availability of each modality of the data: a 4-digit number "0" or "1" is used to mark the missing modality. If part of the input modality is missing, the first digit is set to "0", otherwise it is set to "1"; in addition, the last three digits are used to mark the corresponding visual, auditory and textual modalities in turn. If a modality is missing, the corresponding bit is marked as 1, otherwise it is marked as 0. Assume that the auditory modality is missing and the mark is set to "0010"; for the case where multiple modalities are missing, the mark is set to "0110" to mark the visual and auditory modalities. There are two advantages to this: 1) The label encoding module can cover single-modal and multi-modal missing conditions; 2) The encoded mark can complementarily help learn joint representations. To simplify the mathematical expression, use E tag Indicates all tags;

[0045] Universal space projection module: Each modality is first transformed linearly to obtain the autocorrelated common space representation, and then these common space representations and encoded labels are vertically connected to form a complete joint representation. Compared with the previous method of directly using the feedforward neural layer with the same projection parameters, this scheme adopts a pairwise projection mode to obtain a more universal space, which not only obtains richer information about the conversion between different modalities, but also avoids the problem of modality alignment failure in previous methods;

[0046] Transformer encoder-decoder network: The joint representation is input into the Transformer encoder-decoder network to obtain the completed multimodal data.

[0047] The label encoding module of this solution is used to identify which modal data is available and which is missing. This module generates specific label encodings to assist the data processing algorithm in focusing on the missing modalities and optimize the information completion process.

[0048] The multimodal fusion mechanism of this scheme can dynamically adjust the weight of information between different modalities to make up for the lack of a certain modality and ensure the continuity and accuracy of data processing.

[0049] The adaptive completion algorithm of this scheme is based on machine learning technology, which can automatically generate high-confidence predictions of missing data based on existing modal data, significantly improving data integrity and analysis accuracy.

[0050] The substantial effect brought about by the present invention is that it can not only handle the missing of any single modality, but also cope with the situation where multiple modalities are missing at the same time, ensuring that high emotion recognition accuracy and stability can be maintained in various incomplete data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of a label-based multimodal data missing completion method of the present invention. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0053] Embodiment: A label-based multimodal data missing completion method of this embodiment, such as Figure 1 As shown, the following steps are included:

[0054] S1. The label encoding module of the multimodal data missing completion model uses a fixed-length binary vector as the encoding label to express the availability of each modality of the data;

[0055] S2, the universal space projection module of the multimodal data missing completion model first obtains the autocorrelated common space representation of each modality through linear transformation, and then vertically connects these common space representations and encoded labels to form a complete joint representation;

[0056] S3. Input the joint representation into the Transformer encoder-decoder network of the multimodal data missing completion model to obtain the completed multimodal data.

[0057] Step S2 is specifically as follows:

[0058] Based on the multi-head attention mechanism of transformer:

[0059]

[0060] Where Q, K and V are query, key and value respectively, and d is the dimension of the input;

[0061] For a specific modality M, multi-head attention is applied to obtain more information from different semantic spaces:

[0062] E M =MultiHead(Q,K,V)=Connect(head1,head2,...,head h )W O

[0063] Where W O ∈R d*d is the multi-head attention weight matrix, h is the number of heads; given input E M , the i-th head i The calculation is as follows:

[0064]

[0065] Therefore, the updated modality representation can be expressed as follows:

[0066] E M =MultiHead(X′ M , X′ M , X′ M )

[0067] Among them, the modality M∈(v(video),a(audio),t(text));

[0068] For each mode M, it is mapped to a new space through a vertical linking operation:

[0069] C v =[W va E v ||Wvt E v ]

[0070] C a =[W va E a ||W ta E a ]

[0071] C t =[W vt E t ||W ta E t ]

[0072] Among them, W va , W vt and W ta is the weight matrix, || represents the vertical connection operation;

[0073] Finally, the transformed output of all modes and the encoded label E tag are stacked vertically together to form a comprehensive joint representation: E all =[C v ||C a ||C t ||E tag ].

[0074] The training objectives of the multimodal data missing completion model include classification loss, forward difference loss, backward reconstruction loss and label recovery loss; the calculation formula of the classification loss is:

[0075]

[0076] Where N is the number of samples, y i is the true label of the i-th sample, is the probability distribution predicted by the model;

[0077] The forward difference loss L forward Through pre-training output E pre and the output E of the Transformer encoder out The difference between them is calculated using the Jensen-Shannon (JS) divergence loss formula:

[0078]

[0079] Where D KL is the KL divergence function;

[0080] The goal of the reverse reconstruction loss is to supervise the reconstruction of the joint common vector; therefore, similar to the forward difference loss, by computing the Transformer decoder output Dout and the updated public union representation E all The JS divergence loss between them is obtained, and the reverse reconstruction loss L backward The calculation formula is:

[0081]

[0082] D OUT is the output of the Transformer decoder;

[0083] Among them, labels are attached to mark the missing modalities, hoping that the network can pay more attention to them. In order to better guide the reconstruction of the attached labels, this scheme designs a label recovery loss to guide the process. The reason for choosing the mean absolute error (MAE) loss is that MAE is not very sensitive to outliers with absolute functions. Therefore, MAE is used to calculate E tag and D out The loss between the last four digits of , i.e., the label recovery loss L tag The calculation formula is:

[0084] In step S3, the overall loss function during the training of the entire model network uses a dynamic weight adjustment method based on uncertainty to calculate the total loss:. This method assigns a weight related to its uncertainty to each task, aiming to balance the contribution of each task and improve the effect of multi-task learning. First, define a corresponding uncertainty parameter σi for each loss function. These parameters reflect the uncertainty of the model for each task, and these uncertainties will be used to calculate the weight of each loss; the specific formula is as follows:

[0085]

[0086] The label-based multimodal data missing completion model of this embodiment includes the following key modules:

[0087] 1. Label encoding module: A label encoding mechanism is designed to identify missing modalities. Each input data modality is attached with a label indicating the availability of the modality. The label encoding module can not only cover the case of missing single modality, but also handle the case of missing multiple modalities at the same time, guide the network to focus on these missing modalities, and assist in joint representation learning. Specifically, the module uses a fixed-length binary vector as an encoding label to express the availability of each modality of the data: a 4-digit number "0" or "1" is used to mark the missing modality. If part of the input modality is missing, the first digit is set to "0", otherwise it is set to "1", and the last three digits are used to mark the corresponding visual, auditory and textual modalities in turn. "1" indicates that the modality is available, while "0" indicates that the modality is missing. This label is not only used to indicate which modal data is missing, but also dynamically adjusts the network architecture by modifying the input layer of the neural network so that it only processes the existing modal data. This adaptive input mechanism enables the network to respond flexibly to different data missing situations.

[0088] 2. Universal Space Projection Module: In order to establish an effective joint representation between different modalities, this scheme adopts a new universal space projection method. Each modality first obtains the autocorrelated common space representation through linear transformation, and then vertically connects these representations and the encoded labels to form a complete joint representation. This method can ensure the alignment and consistency of data even when the multimodal data is incomplete. Specifically, this module designs a linear projection mechanism so that features from different modalities can be mapped into a shared feature space for effective information fusion. The transformed outputs of all modalities will be vertically superimposed (concatenate) together with the label vector to form a comprehensive joint representation. This joint representation not only contains the information of each modality, but also integrates the availability information of the modality, thereby optimizing the data quality and integrity of subsequent processing steps.

[0089] 3. Transformer encoder-decoder network: This scheme introduces a new spatial projection mode to align common vectors. The Transformer encoder-decoder network is used to learn the missing modality features. The encoding vector is supervised by a pre-trained network trained on the complete modality data. Finally, the output of the Transformer encoder is used for sentiment classification. This auto-completion algorithm uses the output from the Transformer encoder-decoder network and the output of the pre-trained network trained on the complete modality data to calculate the JS divergence, which provides feedback to help the current model adjust its parameters to simulate the behavior of the complete data.

[0090] The training objectives of the present invention include classification loss, forward difference loss, reverse reconstruction loss and label recovery loss. The forward difference loss guides the learning of the missing modality by comparing the JS divergence of the pre-trained output and the encoder output. This loss measures the difference between the output of the model in the case of missing modalities and the output of the pre-trained model (the model trained when all modalities are available). The reverse reconstruction loss supervises the reconstruction of the joint common vector. This loss is used to ensure that the original modality data can be reconstructed from the joint representation of the model, especially when some modality data is originally missing. The label recovery loss guides the reconstruction of the label by calculating the mean square error (MAE) between the label and the decoder output. This loss term is used to ensure that the model can accurately predict whether each modality is missing or not.

[0091] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0092] Although this article uses more terms such as label encoding, linear transformation, and joint representation, it does not exclude the possibility of using other terms. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A label-based multimodal data missing completion method, characterized in that: The following steps are involved: S1. The label encoding module of the multimodal data missing completion model uses a fixed-length binary vector as the encoding label to express the availability of each modality of the data; S2, the universal space projection module of the multimodal data missing completion model first obtains the autocorrelated common space representation of each modality through linear transformation, and then vertically connects these common space representations and encoded labels to form a complete joint representation; S3. Input the joint representation into the Transformer encoder-decoder network of the multimodal data missing completion model to obtain the completed multimodal data.

2. The label-based multimodal data missing completion method according to claim 1, characterized in that: The step S2 is specifically as follows: Based on Transformer's multi-head attention mechanism: Where Q, K and V are query, key and value respectively, and d is the dimension of the input; For a specific modality M, multi-head attention is applied to obtain more information from different semantic spaces, denoted as E M : E M =MultiHead(Q,K,V)=Connect(head1,head2,...,head h )W o Where W O ∈R d*d is the multi-head attention weight matrix, h is the number of heads, head i represents the i-th long position, 1≤i≤h; given input E M , the i-th head i The calculation is as follows: head i =Attention(E M W i Q ,E M W i K ,E M W i V ) Among them, the modality M∈(v(video),a(audio),t(text)); For each mode M, it is mapped to a new space through a vertical linking operation: C v =[W va HAVE BEEN v ||W vt HAVE BEEN v ] C a =[W va HAVE BEEN a ||W ta HAVE BEEN a ] C t =[W vt HAVE BEEN t ||W ta HAVE BEEN t ] Among them, W va is the weight matrix of vision for hearing, W vt is the weight matrix of vision for text, W ta is the weight matrix related to the hearing of the text, and || represents the vertical connection operation; Finally, the transformed output of all modes and the encoded label E tag are stacked vertically together to form a comprehensive joint representation: E all =[C v ||C a ||C t ||E tag ].

3. The label-based multimodal data missing completion method according to claim 2, characterized in that: The training objectives of the multimodal data missing completion model include classification loss, forward difference loss, backward reconstruction loss and label recovery loss; the calculation formula of the classification loss is: Where N is the number of samples, y i is the true label of the i-th sample, is the probability distribution predicted by the model; The forward difference loss L forward Through pre-training output E pre and Transformer encoder output E out The difference between them is calculated using the Jensen-Shannon divergence loss algorithm, and the formula is: Where D KL is the KL divergence function; Backward reconstruction loss L backward The calculation formula is: D OUT is the output of the Transformer decoder; Label recovery loss L tag The calculation formula is: In the formula, N is the number of labels, D tag Yes D OUT The last four digits indicate the missing state of the modality.

4. The label-based multimodal data missing completion method according to claim 3, characterized in that: In step S3, the overall loss function during the training of the entire model network uses a dynamic weight adjustment method based on uncertainty to calculate the total loss. The specific formula is as follows: σ cls is the uncertainty parameter of classification loss, σ forward is the uncertainty parameter of the forward difference loss, σ backward is the uncertainty parameter of the reverse reconstruction loss, σ tag Uncertainty parameter for label recovery loss.

5. A label-based multimodal data missing completion model, which completes data completion by the method according to claim 1, characterized in that: include: Label encoding module: A fixed-length binary vector is used as the encoding label to express the availability of each modality of the data: a 4-digit number "0" or "1" is used to mark the missing modality. If part of the input modality is missing, the first digit is set to "0", otherwise it is set to "1". In addition, the last three digits are used to mark the corresponding visual, auditory and textual modalities in turn. If a modality is missing, the corresponding bit is marked as 1, otherwise it is marked as 0. Common space projection module: Each modality is first transformed linearly to obtain the autocorrelated common space representation, and then these common space representations and encoded labels are vertically connected to form a complete joint representation; Transformer encoder-decoder network: The joint representation is input into the Transformer encoder-decoder network to obtain the completed multimodal data.

Images