A multi-modal data missing adaptive completion method based on private semantic feature guidance
Patent Information
- Application Number
- CN202610727235.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-25
- Publication Date
- 2026-09-22
AI Technical Summary
传统算法对缺失模态的补全不够充分，往往忽略了模态私有信息在补全过程中的引导作用，这不仅导致补全后的特征偏离原始模态分布，也限制了补全数据质量的提升
本发明通过解耦提取私有语义特征与共享语义特征，并以私有语义特征作为查询向量、共享语义特征作为键值向量输入Transformer解码器生成重构特征，能够在保留各模态独有细节的同时融合跨模态互补信息，避免补全特征向模态共性漂移，从而显著提升重建精度与原始分布保真度。私有语义编码器对文本、音频、视频分别采用TextCNN和LSTM，捕获模态特有分布；共享语义编码器采用两层参数共享的Transformer编码器，提取模态不变共性。以私有语义为导向、以共享语义为信息源的重构机制，使生成过程具有明确的分布约束，从根本上解决了现有方法因忽略私有信息导致的补全特征偏离问题。
Smart Images
Figure CN122796366A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal data missing adaptive completion method guided by private semantic features. Background Technology
[0002] With the rapid development of artificial intelligence technology, single-modal information is no longer sufficient to meet the perception and decision-making needs of complex tasks. Multimodal learning has become an important technology for improving model expressive power and decision-making performance. However, in practical applications, modality loss is common due to factors such as sensor failure and data transmission interruption. This requires models to maintain good robustness even with incomplete modalities and to effectively infer complete information. Traditional algorithms are insufficient in completing missing modalities, often neglecting the guiding role of modality-private information in the completion process. This not only causes the completed features to deviate from the original modality distribution but also limits the improvement of the quality of the completed data. Based on the above problems, this invention proposes a multimodal data loss adaptive completion algorithm guided by private semantic features. First, the algorithm extracts private semantic features and shared semantic features in the private semantic space and shared semantic space, respectively. Then, guided by the private semantic features, complementary information is mined from the shared semantic features to achieve efficient reconstruction of missing data. This algorithm makes full use of previously ignored private semantic information, enhancing the complementary effects between modalities while preserving the unique details of each modality. This effectively avoids the problem of features deviating from the original distribution and significantly improves the quality of reconstructed data. Summary of the Invention
[0003] The purpose of this invention is to provide a multimodal data missing adaptive completion method based on private semantic features, which uses a private semantic encoder and a shared semantic encoder to extract private semantic features and shared semantic features respectively.
[0004] Private semantic features refer to semantic information that is unique to each modality and not shared with other modalities, such as the syntactic structure of text, the action sequence of video, and the timbre features of audio; shared semantic features refer to semantic information shared by multiple modalities, such as sentiment polarity and event type.
[0005] Subsequently, these two types of features are input into a reconstructed feature generator to generate reconstructed features. Next, the reconstructed features are fused with the original missing modality features to obtain the completed features. To ensure that the completed features maintain semantic consistency with the original modality and to enhance the generator's ability to perceive the degree of modality loss, this invention designs two loss functions, one for constraining the completed features and the other for the generator. This effectively completes missing modality data and improves the model's performance in modality-missing scenarios.
[0006] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution: A multimodal data missing adaptive completion method based on private semantic features includes the following steps: Obtain missing modal features with missing data in the original multimodal data, and complete modal features corresponding to the missing modal features; The private semantic features and shared semantic features of the missing modality features are extracted respectively using a private semantic encoder and a shared semantic encoder; Using the private semantic features as a reconstruction guide, complementary information is mined from the shared semantic features to generate reconstruction features for the missing modalities; By fusing the reconstructed features with the missing modality features, the completion features are obtained; At least two types of loss functions are used for constraints: a completion loss function, used to reduce the difference between the completed feature and the complete modal feature; and an integrity loss function, used to perform integrity assessment constraints on the reconstructed feature based on the degree of missing modal feature.
[0007] Furthermore, the private semantic features The extraction process satisfies: in, For the missing modal features, Indicates modality type; the private semantic encoder The TextCNN network is used for text modalities, while the LSTM network is used for audio and video modalities.
[0008] Furthermore, the shared semantic features The extraction process satisfies: Wherein, the shared semantic encoder It consists of two Transformer encoders, and all modalities share the network parameters of the shared semantic encoder.
[0009] Furthermore, the reconstructed features are generated. The steps employ an attention-based generator. To achieve, specifically to satisfy: Wherein, the generator For the Transformer decoder, with the private semantic features As a query vector, with the shared semantic features The reconstructed features are calculated using a masked attention mechanism, serving as key and value vectors.
[0010] Furthermore, the fusion is an element-wise addition fusion, and the completion feature... satisfy: .
[0011] Furthermore, the completion loss function is the mean squared error loss function, which satisfies: in, This refers to the complete modal features.
[0012] Furthermore, the integrity loss function is constructed using an integrity evaluation network, which is a two-layer feedforward network (MLP), with the reconstructed features as input. The output is the modal integrity prediction score; the integrity loss function satisfies: in, The modality missing rate is given, and the target integrity score is given. The integrity loss function is defined as the mean absolute error between the predicted score and the target integrity score.
[0013] Furthermore, the multimodal data includes text modality, video modality, and audio modality, and the missing modality features... They are respectively represented as , , ,in , , These represent text, video, and audio, respectively.
[0014] On the other hand, this invention proposes a multimodal data missing adaptive completion system guided by private semantic features, comprising: The data acquisition module is used to acquire missing modal features and corresponding complete modal features in the original multimodal data where data is missing. A private semantic encoder is used to extract private semantic features of the missing modality features; A shared semantic encoder is used to extract shared semantic features from the missing modality features; A generator, based on an attention mechanism, is used to extract complementary information from the shared semantic features and generate reconstructed features by using the private semantic features as a reconstruction guide. The feature fusion module is used to fuse the reconstructed features and the missing modality features to obtain the completed features; The loss optimization module is used to perform joint optimization using a completion loss function and a completeness loss function, wherein the completeness loss function applies a completeness assessment constraint to the reconstructed features based on the degree of missing modality features.
[0015] The beneficial effects of this invention are: This invention decouples the extraction of private and shared semantic features, using the private semantic features as query vectors and the shared semantic features as key vectors to input into a Transformer decoder to generate reconstructed features. This approach preserves the unique details of each modality while fusing cross-modal complementary information, avoiding feature shift towards modal commonalities and significantly improving reconstruction accuracy and fidelity to the original distribution. The private semantic encoder employs TextCNN for text and LSTM for audio and video respectively, capturing modality-specific distributions; the shared semantic encoder uses a two-layer parameter-sharing Transformer encoder to extract modality-invariant commonalities. This reconstruction mechanism, guided by private semantics and using shared semantics as the information source, provides clear distribution constraints for the generation process, fundamentally solving the problem of feature deviation caused by neglecting private information in existing methods.
[0016] This invention employs a joint optimization mechanism consisting of completion loss and integrity loss. These mechanisms respectively constrain the distance between completed and complete features in the feature space, as well as the generator's ability to perceive modality missing rates. This ensures stable completion performance under different missing rates, significantly improving the algorithm's adaptability and robustness. The completion loss uses mean squared error to directly drive the completed features to approximate the true distribution. The integrity loss outputs an integrity prediction score through a two-layer feedforward network and constructs a mean absolute error with 1-r as the target, backpropagating the missing rate information to the generator, enabling it to learn an intrinsic encoding of the degree of missing data. The synergistic effect of these two losses overcomes the inherent limitation of single-loss models in adapting to changes in missing rates.
[0017] This invention achieves optimal or leading results on multiple metrics, including ACC-2, ACC-5, ACC-7, F1, MAE, and Corr, on English datasets CMU-MOSI and CMU-MOSEI, and Chinese dataset CH-SIMS, compared to existing advanced algorithms such as MISA, Self-MM, MMIM, CENET, TETFN, TFR-Net, ALMT, LNLN, and P-RMF. It also demonstrates stable cross-language generalization ability, proving the universality and industrial application value of the private semantic-guided completion paradigm. The performance improvement on Chinese datasets, in particular, further verifies that this invention is independent of specific languages or data distributions and can be widely deployed in practical multimodal learning systems.
[0018] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of the overall framework of the model of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Example 1
[0023] The multimodal data missing adaptive completion method based on private semantic features described in this embodiment includes the following steps: S1: Obtain input data. Obtain missing modal features in the original multimodal data where data is missing. Specifically, the missing modal features are denoted as... subscript This represents the modality type, including text, video, and audio. Simultaneously, the complete modality features corresponding to the missing modality features are obtained. and modality missing rate In a typical test scenario, the missing modality features are obtained by randomly discarding some modality features from the original complete multimodal data according to a preset missing rate. For example, when the missing rate... A value of 0.3 indicates that 30% of the modal data was randomly removed.
[0024] S2: Extract private and shared semantic features. The private and shared semantic features of the missing modality features are extracted using a private semantic encoder and a shared semantic encoder, respectively. Specifically, the missing modality features... Send to private semantic encoder To obtain private semantic features This process is represented by formula (1): In this embodiment, the proprietary semantic encoder employs different neural network structures for different modalities. For the text modality, a TextCNN network is used for encoding; for the audio and video modalities, a Long Short-Term Memory (LSTM) network is used. This differentiated encoding approach can more effectively capture the unique information of each modality.
[0025] The missing modality features Feed into the shared semantic encoder To obtain shared semantic features This process is represented by formula (2): In this embodiment, the shared semantic encoder is composed of two stacked Transformer encoder layers. It is particularly important to note that the network parameters of this shared semantic encoder are completely shared across the text, video, and audio modalities. This enables the encoder to learn shared semantic information across modalities.
[0026] S3: Generate reconstructed features for the missing modality. The private semantic features and the shared semantic features are input into an attention-based generator. Using the private semantic features as a reconstruction guide, complementary information is mined from the shared semantic features to generate reconstructed features for the missing modality. Let the generator be denoted as... Then reconstruct features The generation process is represented by formula (3): In a preferred implementation of this embodiment, the generator employs a Transformer decoder. During the reconstruction process, the generator incorporates the private semantic features. As a query vector, the shared semantic features are also included. As key and value vectors, they are computed using a masked attention mechanism. This design ensures that when generating reconstructed features, the model focuses only on valid modal information, avoiding information leakage and thus achieving efficient reconstruction.
[0027] S4: Feature fusion to obtain completed features. The reconstructed features are fused with the missing modality features to obtain completed features. Let the completed features be... In this embodiment, the fusion is performed by adding elements one by one, as expressed by formula (4): Through this fusion operation, the completed features retain the true information of the original parts that are not missing, and fill in the reconstructed information of the missing parts, thus obtaining a complete modal feature representation.
[0028] S5: Construct the loss function and train the model. This embodiment uses at least two types of loss functions to jointly constrain the generator and the completed features, namely the completion loss function and the integrity loss function.
[0029] The completion loss function is used to reduce the completion features. With the complete modal features The differences between them allow the completed features to semantically approximate the original modality distribution. In this embodiment, the completion loss function specifically adopts the mean squared error loss function, denoted as . Defined by formula (5): The integrity loss function is used to enhance the generator's response to the modality missing rate. The perception capability. Specifically, this embodiment introduces an integrity evaluation network. This network is a two-layer feedforward network, denoted as MLP. The reconstructed features... The input is fed into this MLP, and the output is the modal integrity prediction score. The integrity loss function is defined as the mean absolute error between the predicted score and the target integrity score. Here, the target integrity score is set as... That is, the proportion of the complete part. This loss function is denoted as... Defined by formula (6): During model training, the total loss function is a weighted sum of the two loss functions mentioned above. The backpropagation algorithm is used to optimize all trainable parameters in the private semantic encoder, shared semantic encoder, generator, and integrity evaluation network. After sufficient training, the model can perform high-quality imputation of missing modal features at arbitrary missing rates.
[0030] Example 2
[0031] This embodiment verifies the effectiveness and feasibility of the proposed multimodal data missing adaptive completion algorithm based on private semantic features through experiments on a public dataset. For ease of description, this algorithm is abbreviated as PSG-MACN.
[0032] Experimental Dataset This embodiment uses the following three publicly available multimodal sentiment analysis datasets: The CMU-MOSI dataset contains 2199 English video clips, divided into 1284 training samples, 229 validation samples, and 686 test samples. Sentiment ratings range from -3 to -3, representing extremely negative and extremely positive sentiments, respectively.
[0033] The CMU-MOSEI dataset contains 22,856 video clips from YouTube, divided into 16,326 training samples, 1,871 validation samples, and 4,659 test samples. The sentiment rating ranges from -3 to +3.
[0034] The CH-SIMS dataset contains 2281 Chinese film and television video clips, divided into 1368 training samples, 456 validation samples, and 457 test samples. The sentiment score ranges from -1 to +1.
[0035] Experimental setup In this embodiment, the modal missing rate r is set from 0.0 to 0.9 with a step size of 0.1 to construct test scenarios with different levels of missing data. For each missing rate, the average value of each evaluation metric is calculated and used as the comprehensive performance evaluation result of the model under that missing data level. All comparison models are trained and tested under the same data partitioning.
[0036] Evaluation indicators To uniformly measure the completion performance of this invention, the following evaluation indicators are adopted.
[0037] For the CMU-MOSI and CMU-MOSEI datasets, the following were selected: seven-class accuracy (ACC-7), five-class accuracy (ACC-5), two-class accuracy (ACC-2), F1 score, mean absolute error (MAE), and Pearson correlation coefficient (Corr).
[0038] For the CH-SIMS dataset, considering the differences in its annotation range, we selected the five-class accuracy (ACC-5), three-class accuracy (ACC-3), two-class accuracy (ACC-2), F1 score, MAE, and Corr.
[0039] Comparison Algorithm The performance of the present invention PSG-MACN is compared with the following existing advanced algorithms: MISA, Self-MM, MMIM, CENET, TETFN, TFR-Net, ALMT, LNLN, and P-RMF.
[0040] Experimental results Table 1. Performance comparison of PSG-MACN with existing state-of-the-art algorithms on the CMU-MOSI dataset. As shown in Table 1, PSG-MACN outperforms the comparison algorithms on all evaluation metrics on the CMU-MOSI dataset. Specifically, PSG-MACN achieves the best values in Acc-2, F1, Acc-5, Acc-7, and Corr metrics, with the MAE dropping to 1.018, the lowest among all algorithms. This indicates that the reconstructed features generated by this algorithm have higher fitting accuracy and linear correlation with the true features.
[0041] Table 2. Performance comparison of PSG-MACN with existing state-of-the-art algorithms on the CMU-MOSEI dataset. As shown in Table 2, PSG-MACN also achieved the best performance on the CMU-MOSEI dataset across the vast majority of metrics. In particular, it achieved 49.55% and 48.65% on the Acc-5 and Acc-7 metrics respectively, significantly higher than other compared algorithms; its Corr reached 0.601, the highest among all algorithms, while its MAE dropped to 0.642, the lowest value, further validating the effectiveness and robustness of this algorithm on large-scale multimodal datasets.
[0042] Table 3. Performance comparison of PSG-MACN with existing state-of-the-art algorithms on the CH-SIMS dataset. As shown in Table 3, on the Chinese dataset CH-SIMS, PSG-MACN achieves the best results compared to other comparative algorithms in terms of Acc-2, Acc-3, Acc-5, MAE, and Corr metrics. Specifically, it achieves 61.03% Acc-3, 36.15% Acc-5, a Corr of 0.456, and a MAE reduction to 0.489. The performance improvement ranges from 0.96% to 4.21%, validating the algorithm's ability in missing modality completion and demonstrating its good generalization performance in cross-linguistic multimodal learning scenarios.
[0043] In summary, this invention proposes a multimodal data missing adaptive completion method guided by private semantic features. After obtaining the missing modality features, private and shared semantic features are extracted using a private semantic encoder and a shared semantic encoder, respectively. The private semantic features are used as query vectors, and the shared semantic features as key vectors, which are then input into a Transformer decoder to generate reconstructed features. The reconstructed features are then fused with the missing modality features to obtain the completed features. Simultaneously, a dual constraint mechanism of completion loss and integrity loss is constructed: the completion loss uses mean squared error to reduce the distance between the completed and complete features, while the integrity loss evaluates the integrity of the reconstructed features through a feedforward network and is supervised by the modality missing rate. This invention fully utilizes private semantic information to guide the completion process, preserving the unique details of each modality and avoiding feature deviation from the original distribution. It achieves high-quality reconstruction under different missing rates and outperforms existing technologies on various language datasets.
[0044] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims
1. A multimodal data missing adaptive completion method based on private semantic features, characterized in that, Includes the following steps: Obtain missing modal features with missing data in the original multimodal data, and complete modal features corresponding to the missing modal features; The private semantic features and shared semantic features of the missing modality features are extracted respectively using a private semantic encoder and a shared semantic encoder; Using the private semantic features as a reconstruction guide, complementary information is mined from the shared semantic features to generate reconstruction features for the missing modalities; By fusing the reconstructed features with the missing modality features, the completion features are obtained; At least two types of loss functions are used for constraints: a completion loss function, used to reduce the difference between the completed feature and the complete modal feature; and an integrity loss function, used to perform integrity assessment constraints on the reconstructed feature based on the degree of missing modal feature.
2. The method as described in claim 1, characterized in that, The private semantic features The extraction process satisfies: in, For the missing modal features, Indicates modality type; the private semantic encoder The TextCNN network is used for text modalities, while the LSTM network is used for audio and video modalities.
3. The method as described in claim 1, characterized in that, The shared semantic features The extraction process satisfies: Wherein, the shared semantic encoder It consists of two Transformer encoders, and all modalities share the network parameters of the shared semantic encoder.
4. The method as described in claim 1, characterized in that, Generate the reconstructed features The steps employ an attention-based generator. To achieve, specifically to satisfy: Wherein, the generator For the Transformer decoder, with the private semantic features As a query vector, with the shared semantic features The reconstructed features are calculated using a masked attention mechanism, serving as key and value vectors.
5. The method as described in claim 1, characterized in that, The fusion is an element-wise addition fusion, and the completion feature satisfy: 。 6. The method as described in claim 1, characterized in that, The completion loss function is the mean squared error loss function, which satisfies: in, This refers to the complete modal features.
7. The method as described in claim 1, characterized in that, The integrity loss function is constructed using an integrity evaluation network, which is a two-layer feedforward network (MLP), with the reconstructed features as input. The output is the modal integrity prediction score; the integrity loss function satisfies: in, The modality missing rate is given, and the target integrity score is given. The integrity loss function is defined as the mean absolute error between the predicted score and the target integrity score.
8. The method according to any one of claims 1 to 7, characterized in that, The multimodal data includes text modality, video modality, and audio modality, and the missing modality features... They are respectively represented as , , ,in , , These represent text, video, and audio, respectively.
9. A multimodal data missing adaptive completion system guided by private semantic features, characterized in that, include: The data acquisition module is used to acquire missing modal features and corresponding complete modal features in the original multimodal data where data is missing. A private semantic encoder is used to extract private semantic features of the missing modality features; A shared semantic encoder is used to extract shared semantic features from the missing modality features; A generator, based on an attention mechanism, is used to extract complementary information from the shared semantic features and generate reconstructed features by using the private semantic features as a reconstruction guide. The feature fusion module is used to fuse the reconstructed features and the missing modality features to obtain the completed features; The loss optimization module is used to perform joint optimization using a completion loss function and a completeness loss function, wherein the completeness loss function applies a completeness assessment constraint to the reconstructed features based on the degree of missing modality features.