A multi-modal data enhancement method and system for random missing modalities
By constructing a multimodal data augmentation model that includes modules for missing modality generation, inverse generation, OT alignment, and attention, the problem of model performance degradation caused by missing modalities is solved, and the accuracy and robustness of multimodal data analysis are improved.
Patent Information
- Application Number
- CN202511717040.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-13
AI Technical Summary
Existing technologies for multimodal sentiment analysis suffer from insufficient reliability of modality completion strategies and inadequate information in modality adaptation strategies, leading to performance degradation when modalities are missing.
A multimodal data augmentation model is constructed, which includes a missing modality generation module, a reverse generation module, an OT alignment cue module, and an attention module. A bidirectional generation constraint mechanism is established through generation cue and reverse cue, and dynamic alignment is performed in combination with OT-guided cue to enhance multimodal representation.
It improves the accuracy and robustness of the model in different missing modal scenarios, promotes effective interaction of information within and between modalities, and enhances the accuracy and robustness of multimodal data analysis.
Smart Images
Figure CN121524948A_ABST
Abstract
Description
[0001] A multi-modal data enhancement method and system for random missing modalities TECHNICAL FIELD
[0002] The present application relates to the technical field of multi-modal data processing, and in particular to a multi-modal data enhancement method and system for random missing modalities. BACKGROUND
[0003] People perceive emotions in social activities, and emotional expression essentially has multi-modal characteristics, including text, auditory and visual signals. Different modalities work together to provide a series of rich emotional cues. By utilizing the complementary characteristics of multi-modal signals, people can more effectively identify the emotional state of users. However, in actual scenarios, missing modalities are inevitable. This is due to a variety of reasons such as sensor failure, network delay and privacy restrictions. This missing often leads to incomplete modal sequences, which in turn leads to a significant decline in the performance of multi-modal models.
[0004] Missing modalities are a common problem in multi-modal emotion analysis. Existing methods usually deal with the problem of missing modal data through two main strategies: modal completion strategies and modal adaptation methods. Modal completion strategies aim to recover missing modal features using available modalities or fill in complete modal sequences using prompts. However, the feature distribution of generated modalities is not balanced compared to real modalities, which can lead to model analysis errors. Similarly, filling in complete modal sequences using prompts can introduce noise, further interfering with the accuracy of the model. Therefore, modal completion strategy methods face challenges in terms of distribution reliability and robustness to noise interference. Modal adaptation methods focus on enabling modalities to perform effectively even in the absence of missing modalities without completely recovering missing data. Therefore, modal adaptation methods are suitable for situations where modalities are missing. However, limited information limits the ability of the model to make accurate predictions. SUMMARY
[0005] The existing technology has the problem of performance decline of the model in the case of missing modalities due to the insufficient reliability of the modal completion strategy and the insufficient information of the modal adaptation strategy, and the present application proposes a multi-modal data enhancement method and system for random missing modalities.
[0006] In a first aspect, the present application provides a multi-modal data enhancement method for random missing modalities, comprising: Step 1: obtaining multi-modal data with random missing modalities to construct a multi-modal missing data set, and taking the multi-modal data as real modal data; Step 2: constructing a multi-modal data enhancement model comprising a missing modality generation module, an inverse generation module, an OT alignment prompt module and an attention module, and training the multi-modal data enhancement model using the multi-modal missing data set; The missing modality generation module is configured to reconstruct missing modality features by means of a generation prompt and a projection network using real modality data; the reverse generation module is configured to reversely generate original modality features from the missing modality features through a reverse prompt and a reverse projection network; the OT alignment module is configured to align real modality features and the missing modality features through OT guided prompt adaptive selection of anchor points to obtain aligned modality features; and the attention module is configured to enhance each modality feature output by the OT alignment module and integrate to output a multi-modal representation. Step 3: input the multi-modal data to be enhanced into the trained multi-modal data enhancement model to obtain an enhanced multi-modal representation.
[0007] Further, the missing modality generation module is configured to reconstruct missing modality features by means of a generation prompt and a projection network using real modality data, and specifically includes: The real modality data is converted into real modality features through a convolution block; The generation prompt guides the concatenation of the real modality features in the sequence dimension to generate a first fusion feature: wherein, represents the generation prompt, represents the real modality features, represents a multi-modal set, represents a convolution layer, represents the first fusion feature; The projection network converts the first fusion feature into a missing modality space for feature reconstruction to obtain missing modality features: wherein, represents the missing modality features, represents an activation function, represents the conversion of the first fusion feature from modality to modality .
[0008] Further, the reverse generation module is configured to reversely generate original modality features from the missing modality features through a reverse prompt and a reverse projection network, and specifically includes: the missing modality features are concatenated with the reverse prompt to generate a second fusion feature: wherein, represents the missing modality features, represents the reverse prompt, represents a concatenation layer, represents the second fusion feature; The second fusion feature is subjected to an inverse projection convolution layer and is connected in residual with the second fusion feature to reconstruct the feature to obtain the original modality feature: wherein, represents the original modality feature, represents an activation function, represents layer normalization, represents an inverse projection convolution layer, represents a skip connection.
[0009] Further, the projection network adopts a multi-layer perceptron (MLP) model; the inverse projection network shares convolution kernel parameters and activation functions with the projection network.
[0010] Further, the OT alignment module is used to adaptively select an anchor point alignment for the real modality feature and the missing modality feature through OT guidance prompts, to obtain an aligned modality feature, and specifically includes: pooling the real modality feature and the missing modality feature, and connecting the pooled features with OT guidance prompts; converting the connected features into latent representation vectors by using an encoder; obtaining a kernel matrix of each modality feature based on a radial basis function; the radial basis function kernel is defined as follows: wherein, represents an adaptive kernel bandwidth parameter, represents a kernel similarity between an i-th sample and a j-th sample in the kernel matrix, represents a latent representation vector of the i-th sample, represents a latent representation vector of the j-th sample, represents a basic coefficient of the kernel bandwidth, represents a feature dimension; selecting one modality from the real modality as an anchor modality based on the OT guidance prompt, and taking the remaining real modality and missing modality as to-be-aligned modalities; calculating optimal transport of the anchor modality and latent representations of each to-be-aligned modality; wherein, represents a transfer matrix, a RBF kernel matrix composed of latent representations of anchor modalities, a RBF kernel matrix composed of latent representations of anchor modalities, a RBF kernel matrix composed of latent representations of anchor modalities, a RBF kernel matrix composed of latent representations of anchor modalities, a Gromov-Wasserstein loss matrix constructed, a probability distribution vector of samples of the modalities to be aligned, a probability distribution vector of samples of the anchor modalities, a batch size of the pooled features; mapping the features of the modalities to be aligned to the feature space of the anchor modalities by the optimal transport; sequence adaptation on the mapped features of the modalities to be aligned and the features of the anchor modalities to obtain aligned modalities, wherein the sequence adaptation formula is as follows: wherein, a time residual, a modality feature, a time series length, i.e., a time step number, a feature vector of the modality feature at the time step, an aligned modality, a broadcast addition operation, a mapped modality.
[0011] Further, the attention module is used to enhance and integrate the features corresponding to the real modality data and the aligned missing modality data to output a multi-modal representation, specifically comprising: enhancing and integrating the real modality features and the aligned missing modality features by a transformer encoder to output a multi-modal representation.
[0012] Further, the multi-modal data enhancement model is trained by using a total loss function, and the formula of the total loss function is as follows: wherein, a global weight, and a weight coefficient, a total loss function, a main task loss function, a reverse loss function, an alignment loss function; wherein, the formula of the main task loss function is as follows: wherein B represents the number of samples, b represents the index of the bth sample in B, represents the predicted sentiment score of the bth sample, represents the true sentiment label of the bth sample; The reverse loss function calculation formula is as follows: wherein, represents an adaptive parameter, represents a real modality feature, represents an original modality feature generated by the reverse generation module; The alignment loss function calculation formula is as follows: wherein, represents a transmission matrix, and the element represents the matching weight between the ith sample of the modality to be aligned and the jth sample of the anchor modality, and all transmission matrix sets satisfying the edge constraint, respectively represent the probability distribution vectors of the modality to be aligned and the anchor modality on the sample, and respectively represent the kernel matrices of the modality to be aligned and the anchor modality.
[0013] In a second aspect, the present application provides a multi-modal data enhancement system for random missing modalities, comprising: A data set construction unit is configured to obtain multi-modal data of random missing modalities to construct a multi-modal missing data set, and use the multi-modal data as real modality data. A multi-modal data enhancement model construction and training unit is configured to construct a multi-modal data enhancement model comprising a missing modality generation module, a reverse generation module, an OT alignment prompt module, and an attention module, and train the multi-modal data enhancement model using the multi-modal missing data set. The missing modality generation module is configured to reconstruct missing modality features by means of a generation prompt and a projection network using real modality data; the reverse generation module is configured to reversely generate original modality features from the missing modality features through a reverse prompt and an inverse projection network; the OT alignment module is configured to adaptively select anchor points for alignment by OT guided prompts for real modality features and the missing modality features, to obtain aligned modality features; and the attention module is configured to enhance and integrate each modality feature output by the OT alignment module to output multi-modal representation. A target multi-modal data enhancement unit is configured to input multi-modal data to be enhanced into the trained multi-modal data enhancement model to obtain enhanced multi-modal representation.
[0014] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method according to the first aspect when executing the program.
[0015] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method according to the first aspect.
[0016] The present application has the following beneficial effects: The multi-modal data enhancement method provided by the present application is for random missing modalities. A multi-modal data enhancement model comprising a missing modality generation module, a reverse generation module, an OT alignment prompt module, and an attention module is constructed to perform multi-modal data enhancement. Specifically, the missing modality generation module and the reverse generation module generate a prompt and a reverse prompt and establish a bidirectional generation constraint mechanism, which enforces consistency between the real modality and the reconstructed modality. The prompt in the missing modality generation module guides the reconstruction of the missing modality, while the reverse prompt in the reverse generation module maintains the consistency of the generated features and the real features through reverse reconstruction. The OT alignment prompt module also generates a dynamic alignment mechanism through an OT guide prompt, so that the anchor modality adapts to different missing scenarios. The bidirectional generation constraint mechanism and the dynamic alignment mechanism work together to promote effective interaction of intra-modality and inter-modality information, and can improve the accuracy and robustness of the model under different missing modality scenarios. BRIEF DESCRIPTION OF DRAWINGS
[0017] Figure 1 One of the flowchart diagrams of a multi-modal data enhancement method for random missing modalities provided by an embodiment of the present application; Figure 2 One of the flowchart diagrams of a multi-modal data enhancement method for random missing modalities provided by an embodiment of the present application; Figure 3 A framework diagram of a multi-modal data enhancement system for random missing modalities provided by an embodiment of the present application; Figure 4 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0018] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.
[0019] As Figure 1 indicated, the multi-modal data enhancement method for random missing modalities provided by the embodiment of the present application comprises: S1: acquiring multi-modal data with random missing modalities to construct a multi-modal missing data set, and taking the multi-modal data as real modal data; the multi-modal data comprises audio, video and text. For example, if the missing modalities in a group of multi-modal data are audio modalities, then the video and text modalities are taken as real modal data; it should be noted that the method provided by the embodiment of the present application is directed to the missing modalities of two modalities, for example, if the missing modalities in a group of multi-modal data are video and text modalities, then the audio modality is taken as real modal data.
[0020] S2: constructing a multi-modal data enhancement model comprising a missing modality generation module, an inverse generation module, an OT alignment prompt module and an attention module, and training the multi-modal data enhancement model by using the multi-modal missing data set; As Figure 2 indicated, wherein the missing modality generation module is used to reconstruct missing modality features by means of a generation prompt and a projection network with real modality data; the inverse generation module is used to reversely generate original modality features by means of an inverse prompt and an inverse projection network with the missing modality features; the OT alignment module is used to adaptively select anchor points for alignment by means of an OT guide prompt with the real modality features and the missing modality features, to obtain aligned modality features; and the attention module is used to enhance and integrate the modality features output by the OT alignment module to output multi-modal representations.
[0021] S3: inputting multi-modal data to be enhanced into the trained multi-modal data enhancement model to obtain enhanced multi-modal representations.
[0022] The multi-modal data enhancement method for random missing modalities provided by the embodiment of the present application uses three kinds of prompts: a generation prompt, an inverse prompt and an OT guide prompt. These prompts are used to generate missing modality features, reversely guide the reconstruction of original available features, and adaptively adjust anchor modalities in the alignment process.
[0023] The multi-modal data enhancement model provided by the present application will be described in detail below.
[0024] The missing modality generation module is used to reconstruct missing modality features by means of a generation prompt and a projection network with real modality data, and specifically comprises: generating real modality features by means of a convolution block with real modality data; generating a first fusion feature capable of effectively encoding local dependency relationships by means of a generation prompt to splice the real modality features in a sequence dimension: wherein, denotes a generation prompt, denotes a real modality feature, denotes a multi-modal set, , denotes a number of modalities of the multi-modal; denotes a convolutional layer, denotes a first fusion feature; a design guide prompt is used to dynamically adjust the generation process, wherein and denote the dimension and length of the prompt, respectively. For example, the multi-modal represents audio, video and text modalities, respectively, then (0, 1 and 2 represent text, audio and video modalities, respectively), the generation prompt wherein , , denote the generation prompts of the audio, video and text modalities, respectively.
[0025] The generation prompt is derived from the missing modality completion task, which functions to explicitly inform the model of “which modality needs to be reconstructed” through the prompt vector when the face video is missing or the audio is missing, thereby activating the corresponding cross-modal generation mapping in the latent space. For example, when only the speech is available but the video is missing, the generation prompt guides the model to generate the corresponding expression dynamics or visual emotional features according to the audio rhythm, tone, prosody and other semantic clues, so that the missing modality approaches the real distribution in a statistical sense.
[0026] The projection network converts the first fusion feature into the missing modality space for feature reconstruction, to obtain a missing modality feature: wherein, denotes a missing modality feature, denotes an activation function, denotes a conversion of the first fusion feature from a modality to a modality .
[0027] Further, the projection network in the missing modality generation module adopts a lightweight MLP model.
[0028] The reverse generation module is used to reversely generate the original modality feature from the missing modality feature through a reverse prompt and a reverse projection network, and specifically includes: splicing the missing modality feature and the reverse prompt to generate a second fusion feature: wherein, denotes a missing modality feature, denotes a reverse prompt, represents a splicing layer, represents a second fusion feature. The distribution offset in the generated missing modal feature will significantly reduce the effect of multi-modal fusion, so the embodiment of the application designs a reverse prompt to guide the inverse projection network to reconstruct the original feature from the generated feature. For example, , wherein respectively correspond to the audio, video and text modal reverse prompts, and the structure of the generated prompt is the same, , wherein and respectively represent the feature dimension and length of the prompt. The reverse generated prompt is used to guide the reconstruction of the original modal feature from the generated missing modal feature while maintaining semantic consistency.
[0029] The source of the reverse prompt is the need for consistency checking of the generated modal. After generating the video feature using the audio, the reverse prompt will further require the model to infer the audio feature from the generated video, so as to form an invertible semantic mapping. For example, if the model generates a "smiling expression" visual feature based on the voice, it should be able to infer the "positive tone" audio pattern from the visual feature after the reverse prompt. This reverse constraint can ensure that the generated modal and the existing modal maintain semantic consistency, and reduce the generation bias from the mechanism.
[0030] The second fusion feature is subjected to an inverse projection convolution layer and is connected in residual to the second fusion feature for feature reconstruction to obtain the original modal feature: , wherein represents the original modal feature, represents an activation function, represents layer normalization, represents an inverse projection convolution layer, represents a skip connection. The internal residual connection structure adds the input feature of the inverse projection network to the convolution output, retains the original information and reconstructs the residual, thereby enhancing the inverse reconstruction capability.
[0031] Further, the inverse projection network shares the convolution kernel parameters and the activation function with the projection network in the missing modal generation module.
[0032] It can be understood that the reverse generation module and the missing modal generation module together form a bidirectional generation constraint mechanism. This mechanism forces the distribution between the generated missing modal feature and the real modal feature to be consistent.
[0033] The OT alignment prompt module is used to adaptively select anchor point alignment for the real modal feature and the missing modal feature through OT guided prompts to obtain aligned modal features, and specifically comprises: pooling operations are performed on the real modality features and the missing modality features, and the pooled features are connected with the OT guide prompt; The OT guide prompt is derived from the requirement of the multi-modal fusion stage for "real modality as anchor point". The generated modality may have a distribution offset from the real modality, so the optimal transport (OT) alignment constraint is introduced in the prompt to make the model automatically take the real modality as a reference when fusion. For example, when real audio is fused with generated video, the OT guide prompt will prompt the model to align the "generated video emotion trajectory" to the "real audio emotion change curve", so that the two establish an optimal one-to-one matching relationship in the feature space. In this way, the fusion result is prevented from being dragged by the generated modality, and the cross-modal consistency is maintained.
[0034] Specifically, the pooling operation is performed on the features to realize sample-level alignment.
[0035] The connected features are converted into latent representation vectors by using an encoder. wherein, represents the latent representation vector, represents the encoder, represents the pooled features, represents the OT guide prompt.
[0036] A kernel matrix of each modality feature is obtained based on a radial basis function; the radial basis function kernel is defined as follows: wherein, represents an adaptive kernel bandwidth parameter, represents a kernel similarity between the i-th sample and the j-th sample in the kernel matrix, represents the latent representation vector of the i-th sample, represents the latent representation vector of the j-th sample, represents a base coefficient of the kernel bandwidth, represents a feature dimension; An anchor modality is selected from the real modalities based on the OT guide prompt, and the remaining real modalities and the missing modalities are taken as the modalities to be aligned; An optimal transport of the latent representations of the anchor modality and the modalities to be aligned is calculated. wherein, denotes the transfer matrix, denotes the RBF kernel matrix composed of latent representations of the modalities to be aligned, denotes the RBF kernel matrix composed of latent representations of the anchor modality, denotes the RBF kernel matrix composed of latent representations of the modalities to be aligned, denotes the RBF kernel matrix composed of latent representations of the modalities to be aligned, denotes the Gromov-Wasserstein loss matrix constructed, denotes the probability distribution vector of the samples of the modality to be aligned, denotes the probability distribution vector of the samples of the anchor modality, denotes the batch size of the pooled features; Specifically, this step solves the transfer matrix by minimizing the Gromov-Wasserstein (GW) distance between the RBF kernel matrices of the two modalities. kl denotes the alignment weight between the k-th sample of the i-th modality and the l-th sample of the anchor modality r.
[0037] Map the missing modality features to the anchor modality feature space through optimal transport: wherein, denotes the weighted aligned features of modality i, denotes the transpose of the transfer matrix from modality i to modality r, denotes the pooled feature matrix of modality i. This operation ensures the consistency of semantic information under different modalities in the anchor space while preserving the modality-specific structure.
[0038] Perform sequence adaptation on the mapped missing modality features and anchor modality features to obtain aligned missing modality features; wherein the sequence adaptation formula is as follows: wherein, denotes the temporal residual, denotes the modality feature, denotes the length of the time series, i.e. the number of time steps, denotes the feature vector of the modality feature at the time step, denotes the aligned modality feature, denotes the broadcast addition operation, denotes the mapped modality feature.
[0039] Specifically, the sequence adaptation operation can alleviate the potential impact of time sensitivity on model performance. After the operation of the above reverse generation module, a feature integrating time information and aligned semantics is generated.
[0040] The generation of the prompt, the reverse prompt and the OT guide prompt is from the structural requirement of the model in the multi-modal missing and alignment scenario, and is designed based on the "condition control" principle of the decision path of the hidden space of the large model in prompt learning, and is represented by a vector.
[0041] The attention module is used for enhancing and integrating the multi-modal representation of the real modal data corresponding features and the aligned missing modal features, taking the main modal features as the query, adaptively filtering and integrating the key information from the auxiliary modal according to the similarity, so as to realize the cross-modal alignment and effective supplement (for example, when the text modal is the main modal, the video modal and the audio modal are auxiliary modal). The multi-head mechanism further captures different types of cross-modal dependency, improving the integrity and robustness of the fused representation.
[0042] Specifically, the real modal features and the aligned missing modal features are enhanced and integrated by the transformer encoder to output the multi-modal representation, including the following steps: Modal mapping. In cross-modal fusion, this module first projects the features of different modalities into a unified attention representation space. Among them, the main modal features are mapped as query vectors , which are used to actively "ask" the information of another modal; the auxiliary modal features are mapped as key vectors and value vectors , which are used to calculate the relevance with the main modal and provide the extracted specific information respectively. As shown in the following formula. Through this role-based projection method, the model can realize cross-modal information retrieval based on semantic similarity in the subsequent steps.
[0043] Cross-modal attention calculation. The module uses the dot product of the main modal query vector and the auxiliary modal key vector to calculate the similarity matrix of the two. By scaling and applying Softmax, the model obtains the cross-modal attention weight, which represents "what semantic information in the auxiliary modal does the main modal most want to focus on at each position". Then, the weight and the value vector are weighted and summed to output the cross-modal supplement features for the main modal. This step enables the model to automatically filter the most relevant and matching part of the auxiliary modal when fusing, instead of performing simple concatenation or averaging.
[0044] Multi-head mechanism and output integration. To capture different types of cross-modal relationships, the module adopts a multi-head attention mechanism. Each attention head learns a unique query-key-value mapping in an independent low-dimensional subspace, and the outputs of multiple attention heads are spliced and then mapped through a linear mapping for integration, which can form a fusion representation that contains multiple types of cross-modal information at the same time.
[0045] Further, the multi-modal data augmentation model is trained using a total loss function. The model first calculates the main task loss, which measures the error between the model's prediction and the true label. The auxiliary loss includes the reverse loss and the alignment loss, which are linearly combined using weight coefficients and to constrain the effects of the two modules, and global weights are used to balance the main loss and the auxiliary loss to form the final total loss function. The total loss function formula is as follows: wherein, is the global weight, and are the weight coefficients, is the total loss function, is the main task loss function, is the reverse loss function, is the alignment loss function; wherein, The main task loss function calculation formula is as follows: wherein, B represents the number of samples, b represents the index of the bth sample in B, is the predicted sentiment score of the bth sample, i.e., the predicted sentiment score corresponding to the multi-modal feature of sample b obtained by the multi-modal data augmentation module; is the true sentiment label of the bth sample, i.e., the true sentiment label corresponding to the multi-modal feature of sample b; The reverse loss function calculation formula is as follows: wherein, is the adaptive parameter, which enables the reverse generation module to dynamically balance the two optimization objectives of feature space alignment and semantic orientation, rather than relying on a manually fixed trade-off. is the real modality feature, is the original modality feature generated by the reverse generation module; the reverse loss function can further jointly strengthen feature space alignment and semantic orientation.
[0046] The alignment loss function is calculated as follows: wherein, represents a transmission matrix, and the element represents the matching weight between the i-th sample of the modalities to be aligned and the j-th sample of the anchor modalities, and all the transmission matrix sets satisfying the edge constraint, respectively represent the probability distribution vectors of the modalities to be aligned and the anchor modalities on the samples, and respectively represent the kernel matrices of the modalities to be aligned and the anchor modalities.
[0047] As shown in Figure 3 , the embodiment of the present application also provides a multi-modal data enhancement system for random missing modalities, comprising: a data set construction unit, configured to obtain multi-modal data of random missing modalities to construct a multi-modal missing data set, and take the multi-modal data as real modal data; a multi-modal data enhancement model construction and training unit, configured to construct a multi-modal data enhancement model comprising a missing modality generation module, an inverse generation module, an OT alignment prompt module and an attention module, and train the multi-modal data enhancement model by using the multi-modal missing data set; wherein the missing modality generation module is configured to reconstruct missing modality features by means of a generation prompt and a projection network with the real modality data; the inverse generation module is configured to reversely generate original modality features from the missing modality features through an inverse prompt and an inverse projection network; the OT alignment module is configured to adaptively select anchor points for alignment by OT guided prompts on the real modality features and the missing modality features, to obtain aligned modality features; and the attention module is configured to enhance and integrate the modality features output by the OT alignment module to output multi-modal representations; a target multi-modal data enhancement unit, configured to input multi-modal data to be enhanced into the trained multi-modal data enhancement model to obtain enhanced multi-modal representations.
[0048] Figure 4 An example of an entity structure diagram of an electronic device is shown in Figure 4 , which can include a processor 401, a communications interface 402, a memory 403 and a communications bus 404, wherein the processor 401, the communications interface 402 and the memory 403 complete mutual communication through the communications bus 404. The processor 401 can invoke logical instructions in the memory 403 to execute a multi-modal data enhancement method, which comprises the above-mentioned embodiments S1~S3.
[0049] In addition, the logical instructions in the memory 403 described above are implemented in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.
[0050] The embodiments of the present application also provide a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions. When the program instructions are executed by a computer, the computer can execute the method provided by the method embodiments described above, for example, the embodiments S1-S3.
[0051] The embodiments of the present application also provide a non-transitory computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method provided by the method embodiments described above is implemented, for example, the embodiments S1-S3.
[0052] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A multimodal data augmentation method for randomly missing modalities, characterized in that, include: Step 1: Obtain multimodal data with random missing modalities to construct a multimodal missing dataset, and use the multimodal data as the real modal data; Step 2: Construct a multimodal data augmentation model that includes a missing modality generation module, an inverse generation module, an OT alignment cue module, and an attention module, and train the multimodal data augmentation model using the multimodal missing dataset; The missing modality generation module is used to reconstruct missing modality features using real modality data with the help of generation prompts and projection networks; The reverse generation module is used to reverse-generate the original modal features from the missing modal features through reverse prompting and reverse projection networks; the OT alignment module is used to adaptively select anchor points to align the real modal features and the missing modal features through OT guidance prompts to obtain aligned modal features; the attention module is used to enhance each modal feature output by the OT alignment module and integrate them to output a multimodal representation. Step 3: Input the multimodal data to be augmented into the trained multimodal data augmentation model to obtain the augmented multimodal representation.
2. The multimodal data augmentation method for randomly missing modalities according to claim 1, characterized in that, The missing modality generation module is used to reconstruct missing modality features using real modality data with the aid of generation prompts and projection networks, specifically including: The real modal data is used to generate real modal features through convolutional blocks; The generation prompt guides the concatenation of real modality features along the sequence dimension to generate the first fused feature: in, This indicates that a prompt has been generated. Represents true modal features, Represents a multimodal set. Indicates a convolutional layer. Indicates the first fusion feature; The projection network transforms the first fused features into the missing modality space for feature reconstruction, thus obtaining the missing modality features: in, Indicates missing modal features. This represents the activation function. Indicates the first fusion feature From modality Convert to mode .
3. A multimodal data augmentation method for randomly missing modalities according to claim 2, characterized in that, The reverse generation module is used to reverse-generate the original modality features from the missing modality features through reverse prompting and reverse projection networks. Specifically, it includes: concatenating the missing modality features with the reverse prompting to generate a second fused feature. in, Indicates missing modal features. This indicates a reverse suggestion. Indicates the splicing layer. Indicates the second fusion feature; The second fused feature is then subjected to an inverse projection convolutional layer and residually connected to it to reconstruct the original modality features. in, Represents the original modal features. This represents the activation function. Representation layer normalization, This indicates a backprojection convolutional layer. This indicates a skip connection.
4. A multimodal data augmentation method for randomly missing modalities according to claim 3, characterized in that, The projection network adopts a multilayer perceptron (MLP) model; the inverse projection network shares the convolution kernel parameters and activation function with the projection network.
5. A multimodal data augmentation method for randomly missing modalities according to claim 2, characterized in that, The OT alignment module is used to adaptively select anchor points to align the true modal features and the missing modal features through OT guidance prompts, thereby obtaining aligned modal features, specifically including: The real modal features and the missing modal features are pooled, and the pooled features are then concatenated with the OT guidance prompts. The encoder is used to convert the concatenated features into a latent representation vector. The kernel matrix of each modal feature is obtained based on the radial basis function; the radial basis function kernel is defined as follows: in, Indicates the adaptive kernel bandwidth parameter. Represents the kernel matrix of the first... The sample and the first Kernel similarity between samples Indicates the first The latent representation vector of each sample Indicates the first The latent representation vector of each sample The fundamental coefficient representing the kernel bandwidth. Indicates feature dimension; Based on the OT guidance prompt, select one mode from the real modes as the anchor mode, and use the remaining real modes and missing modes as modes to be aligned; Calculate the optimal transmission of the anchor mode and the potential representation of each of the modes to be aligned; in, Represents the transition matrix. The RBF kernel matrix represents the latent representation of the modes to be aligned. The RBF kernel matrix represents the latent representation of the anchor modes. Indicates based on and The constructed Gromov-Wasserstein loss matrix, This represents the probability distribution vector of the modal samples to be aligned. This represents the probability distribution vector of the anchor mode sample. Indicates the batch size of the features after pooling; The optimal transmission maps the modal features to be aligned to the anchor modal feature space. Sequence adaptation is performed on the mapped modal features to be aligned and the anchor modal features to obtain aligned modal features; wherein the sequence adaptation formula is as follows: in, Indicates the time residual. Representing modal features, This indicates the length of the time series, i.e., the number of time steps. The modal feature represents the first Feature vectors at each time step This represents the aligned modal features. This indicates a broadcast addition operation. This represents the modal features after mapping.
6. A multimodal data augmentation method for randomly missing modalities according to claim 2, characterized in that, The attention module is used to enhance and integrate the features corresponding to the real modality data and the aligned missing modality features, and output a multimodal representation, specifically including: The transformer encoder enhances and integrates the real modal features and aligned missing modal features to output a multimodal representation.
7. A multimodal data augmentation method for randomly missing modalities according to claim 1, characterized in that, The multimodal data augmentation model is trained using a total loss function, the formula of which is as follows: in, Represents the global weight. and Indicates the weighting coefficient. Represents the total loss function. This represents the loss function of the main task. Represents the inverse loss function. Denotes the alignment loss function; where, The formula for calculating the main task loss function is as follows: Where B represents the number of samples, and b represents the index of the b-th sample in B. This represents the predicted sentiment score for the b-th sample. This represents the true sentiment label of the b-th sample; The formula for calculating the inverse loss function is as follows: in, Indicates adaptive parameters, Represents true modal features, This represents the original modal features generated by the reverse generation module; The alignment loss function is calculated using the following formula: in, Represents the transfer matrix, elements This represents the matching weight between the i-th sample of the mode to be aligned and the j-th sample of the anchor mode, while The set of all transfer matrices that satisfy the edge constraints. Let represent the probability distribution vectors of the mode to be aligned and the anchor mode on the sample, respectively. These represent the kernel matrices of the mode to be aligned and the anchor mode, respectively.
8. A multimodal data augmentation system for randomly missing modalities, characterized in that, include: A dataset construction unit is used to acquire multimodal data with random missing modalities to construct a multimodal missing dataset, and to use the multimodal data as real modal data; A multimodal data augmentation model construction and training unit is used to construct a multimodal data augmentation model that includes a missing modality generation module, an inverse generation module, an OT alignment cue module, and an attention module, and to train the multimodal data augmentation model using the multimodal missing dataset; The missing modality generation module is used to reconstruct missing modality features using real modality data with the help of generation prompts and projection networks; The reverse generation module is used to reverse-generate the original modal features from the missing modal features through reverse prompting and reverse projection networks; the OT alignment module is used to adaptively select anchor points to align the real modal features and the missing modal features through OT guidance prompts to obtain aligned modal features; the attention module is used to enhance each modal feature output by the OT alignment module and integrate them to output a multimodal representation. The target multimodal data augmentation unit is used to input the multimodal data to be augmented into the trained multimodal data augmentation model to obtain the augmented multimodal representation.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.