Missing perception prompting method and system based on modal specific and general information

Abstract

The invention discloses a missing perception prompting method and system based on modal specific and general information, and belongs to the field of multi-modal learning. Performing modal missing processing on the initial features of the original video sample; unifying feature dimensions and sizes of modal missing features; wherein after unifying the feature dimensions, splicing to obtain a multi-modal joint feature; performing size unification and then performing fusion to obtain a modal universal prompt; using a gating fusion mechanism to control the fusion of the learnable matrix corresponding to the modal to obtain a modal specific prompt; fusing the modal general prompt and the modal specific prompt to obtain a specific-general prompt as a missing perception prompt; and the specific-general prompt and the multi-modal joint features are spliced to serve as input of a multi-modal model, and a full connection layer is introduced to complete a multi-modal downstream task. According to the method, the learnable specific-universal prompt of mode missing perception is introduced into the multi-mode model, so that cross-mode application in a data mode incomplete scene is realized, and the performance of a multi-mode downstream task is improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning, and in particular to a method and system for providing missing perception prompts based on modality-specific and universal information. Background Art

[0002] The field of multimodal learning aims to utilize the complementarity and synergy between different modalities to effectively integrate multimodal information and improve the performance of various machine learning tasks. With the advent of the digital age, the amount of video data has exploded and has become the main carrier of information dissemination on the Internet and social media. Videos usually contain information in multiple modalities, such as visual modality, audio modality, text modality, etc. These modalities complement each other and provide a multi-angle and multi-level understanding of the video content. At present, the surge in multimodal models and large-scale video data has jointly promoted the development of this field. However, the application of multimodal learning in two real-world scenarios, incomplete data modalities and limited resources, still faces challenges.

[0003] To address the above two challenges, existing technologies design learnable prompts based on the idea of ​​"treating different modal missing types as different model inputs" to replace fine-tuning of the overall model, thereby reducing the amount of computation and improving robustness to missing modal data.

[0004] However, existing technologies focus on designing specialized structures for a certain type of specific task and have achieved good performance. However, due to the limitations of the model structure, they can generally only be applied to a fixed number of modal inputs or specific tasks. More importantly, they all ignore the key to solving the challenge of missing modalities - how to learn shared features between modalities and specific features within the modality to improve model performance. They do not realize that the missing modality is only a part of the unique information within the modality, and therefore do not fully utilize the complementarity of multimodal information to guide the model to make predictions. Summary of the invention

[0005] In order to overcome the problem that existing technologies are limited to fixed numbers of modalities and tasks, and the complementarity of multimodal information is not fully utilized, the present invention provides a missing perception prompt method and system based on modality-specific and common information to achieve multimodal prompt learning in modality missing scenarios.

[0006] The specific technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention proposes a missing perception prompting method based on modality-specific and general information, comprising:

[0008] Obtaining original video samples and text annotations, performing multimodal feature extraction on the original video samples, and obtaining multimodal initial features; the text annotations refer to the original video sample labels, which are determined according to downstream tasks;

[0009] Perform partial modality missing processing on the multimodal initial features of the original video samples to obtain modality missing features;

[0010] Map the modal missing features to the same feature dimension space to obtain the modal missing features of the unified feature dimension; concatenate the modal missing features of the unified feature dimension to obtain multi-modal joint features;

[0011] Map the modal missing features of the unified feature dimension to the same sequence length space to obtain the modal missing features of the unified size, and generate the modal universal prompt after fusion;

[0012] A gated fusion mechanism is constructed using the Frobenius norm and uniform-sized modality missing features to determine modality missingness and control the fusion of modality-corresponding learnable matrices to generate modality-specific cues.

[0013] Fusion of modality-universal cues and modality-specific cues to generate specific-universal cues as missing perception cues;

[0014] The multimodal joint features are spliced ​​after the missing perception prompt, input into the pre-trained multimodal model, extract the last token in the output of the pre-trained multimodal model, input into the fully connected layer, and complete the training of the multimodal model, fully connected layer, feature mapping, and learnable matrix based on the downstream video task according to the output result of the fully connected layer.

[0015] Furthermore, the feature extraction of the original video sample to obtain the initial feature includes:

[0016] A pre-trained feature extraction model is used as a feature extractor of a modality in multi-modality, an original video is input, and corresponding modality features are extracted as initial features of the modality; feature extractors of different modalities are used to obtain multi-modal initial features of the original video; the multi-modality includes text modality, visual modality, and audio modality.

[0017] Furthermore, the multimodal initial features of the original video sample are partially subjected to modality loss processing to obtain modality loss features, including:

[0018] Some modal features of the original video samples are replaced with all-zero matrices of the same size to indicate modal loss; for all samples, 2 n -1 modality missing cases are set with the same probability, and n is the total number of modalities of the original video samples.

[0019] Furthermore, the calculation formula of the modal general prompt is:

[0020]

[0021] Among them, pc is the modality-general cue; n is the total number of modalities of the original video sample; represents the modal missing feature of the i-th mode, L i and D i Respectively represent x i The sequence length and feature dimension of f d (·) indicates that the feature dimension is adjusted to d; f l (·) indicates adjusting the sequence length to l.

[0022] Furthermore, the calculation formula of the modality-specific prompt is:

[0023]

[0024] Among them, p s is a modality-specific prompt; M i ∈R l×d is the learnable matrix; l×d represents the sequence length and feature dimension of the learnable matrix, which is consistent with the modal missing feature size of uniform size; ‖·‖ F represents the Frobenius norm, which is used to determine whether the calculated matrix is ​​an all-zero matrix, and then determine whether the corresponding modal feature is missing; b i is the coefficient of the learnable matrix corresponding to modality i, which is used to determine whether the corresponding learnable matrix participates in the generation of modality-specific prompts.

[0025] Furthermore, the specific-general prompt is obtained by adding the modal general prompt and the modal specific prompt.

[0026] Furthermore, the multimodal joint features are concatenated after the missing perception prompt to obtain the input of the multimodal model, which is expressed as:

[0027] x input =[p s-c ,f d (x1),f d (x2),…,f d (x n )]

[0028] Among them, x input represents the input of the multimodal model; [...] represents the concatenation operation, p s-c Indicates specific-general prompts, f d (x n ) represents the modal missing features of the unified feature dimension of modal n; the multimodal model adopts a single-stream Transformer model.

[0029] Furthermore, the downstream video tasks include a multimodal sentiment analysis task, a multimodal emotion classification task and a multimodal video retrieval task.

[0030] Furthermore, the multimodal sentiment analysis task adopts the L1 loss function, the multimodal emotion classification task adopts the cross entropy loss function, and the multimodal video retrieval task adopts the bidirectional maximum margin ranking loss function.

[0031] In a second aspect, the present invention proposes a missing perception prompting system based on modality-specific and common information, which is used to implement the above-mentioned missing perception prompting method based on modality-specific and common information.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention is a method and system for missing-aware prompting based on modality-specific and universal information. In implementation, the present invention uses modality-missing-aware specific-universal prompting and a single-stream Transformer architecture multimodal model.

[0034] (1) The specific-universal cues of modality-missing perception can be decomposed into modality-universal cues and modality-specific cues, so as to promote the model to capture cross-modal universal information and modality-specific information respectively, and make full use of the complementarity of multimodal information for model prediction.

[0035] (2) The modality-specific cues in the modality-missing-aware specific-universal cues can capture the modality-missing information to improve the robustness of the model in scenarios with incomplete data modalities.

[0036] (3) The present invention is not limited by the number of modalities by using a single-stream Transformer multimodal model; the model has strong understanding ability and generalization performance, and can extract rich semantic information from the input; the model is structurally universal and easy to integrate, thus realizing its application in a variety of multimodal tasks and effectively improving the performance of the model on various data sets.

[0037] In summary, through the specific-universal cues of modality missing awareness and the single-stream Transformer architecture multimodal model, the present invention can fully learn the cross-modal general information and the intra-modal specific information (including but not limited to the case of modality missing) in the multimodal information, and realize how to apply multimodal learning to different multimodal tasks when the data modality is incomplete. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of the present invention under general classification or regression tasks (such as multimodal sentiment analysis, multimodal emotion classification tasks);

[0039] Figure 2 It is a schematic diagram of the process of the present invention under the multimodal video retrieval task;

[0040] Figure 3 It is a schematic diagram of the process of generating a modal universal prompt of the present invention;

[0041] Figure 4 It is a schematic diagram of the modality-specific prompt generation process of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described and illustrated below in conjunction with the accompanying drawings and specific implementation methods.

[0043] The present invention proposes a missing perception prompting method based on modality-specific and general information, which mainly includes the following steps:

[0044] Step 1: Obtain original video samples and text annotations, extract features from the original video samples, and obtain initial features, which together with the text annotations constitute a data set; the text annotation refers to the original video sample label, which is determined according to the downstream task;

[0045] Step 2, performing partial modality missing processing on the multimodal initial features of the original video sample to obtain modality missing features;

[0046] Step 3, map the modal missing features to the same feature dimension space to obtain the modal missing features of unified feature dimensions;

[0047] Step 4: concatenate the missing features of the modalities with the same feature dimension to obtain multimodal joint features;

[0048] Step 5, mapping the modal missing features of uniform feature dimension to the same sequence length space to obtain the modal missing features of uniform size;

[0049] Step 6: Fusion of modal missing features of uniform size to generate modal universal prompts;

[0050] Step 7, using the Frobenius norm and the uniform-sized modality missing features to construct a gated fusion mechanism to determine the modality missing situation, control the fusion of the modality corresponding learnable matrix, and generate modality-specific prompts;

[0051] Step 8, fusing the modality-general prompts and the modality-specific prompts to generate specific-general prompts as missing perception prompts;

[0052] Step 9, concatenating the multimodal joint features after the missing perception prompt (i.e., specific-universal prompt) and inputting them into the pre-trained single-stream multimodal model;

[0053] Step 10: Extract the last token in the model output and input it into the fully connected layer to complete the training and reasoning of downstream tasks.

[0054] In the above step 1, an optional method for extracting features from the original video sample is as follows:

[0055] The VGGish model pre-trained on the YouTube-8m dataset is used as an audio feature extractor to extract audio features from the original video sample and obtain an audio modality feature matrix; the ResNet50 model pre-trained on the VGGFace2 dataset is used as a face feature extractor to extract face features from the original video sample as a visual modality feature matrix; the I3D model pre-trained on the Kinetics dataset is used as an action feature extractor to extract action features from the original video sample as an action modality feature matrix; but not limited to the above pre-trained models and modality features, commonly used modalities include text (such as optical character recognition) modality, visual (such as action, scene, appearance, face, etc.) modality, audio modality, etc. The definition and extraction process of each modality are common knowledge in the art and will not be repeated here.

[0056] In step 2 above, an optional method for obtaining the missing features of the modality is as follows:

[0057] Part of the modal feature matrix is ​​replaced with a full zero matrix of the same size to indicate the missing mode. In this embodiment, for all samples, 2 n -1 modality missing case sets the same probability, n is the total number of modalities of the original video sample; taking 3 modalities as an example, there are 2 3 modal conditions, of which 2 3 -1 missing modality situation, specifically 3 missing one modality situation, 3 missing two modalities situation and 1 missing three modalities situation, setting the missing probability of each situation to 5%.

[0058] In the above step 3, an optional method for obtaining the modal missing features of the unified feature dimension is as follows:

[0059] The modal missing features of each mode (L i and D i Represents the missing features x of mode i i The sequence length and feature dimension of D is input into the one-dimensional convolution layer, and its feature dimension is changed from D i Convert to a unified target dimension d. The size of the convolution kernel is set to 1 to ensure that only the feature dimension is transformed without changing the sequence length. The expression is f d (x i ). The final output feature matrix size is L i ×d, thereby completing the dimension alignment of each modality feature. The structure of unifying feature dimensions is not limited to the above-mentioned one-dimensional convolutional layer, but can also be a fully connected layer or other linear projection methods.

[0060] In the above step 5, an optional way of obtaining the sequence length space mapped to the same sequence length space is as follows:

[0061] The modal missing features of the unified feature dimension of each modality Input a one-dimensional convolutional layer and change its sequence length from L i Convert to a uniform target length l. The size of the convolution kernel is set to 1 to ensure that only the sequence length is transformed without changing the feature dimension. The expression is f l (f d (x i )). The final output feature matrix size is l×d, thereby completing the size alignment of each modality feature. The structure of unified sequence length is not limited to the above-mentioned one-dimensional convolutional layer, but can also be a fully connected layer or other linear projection methods.

[0062] In step 6 above, if Figure 3 As shown, an alternative way to obtain a modal general prompt is as follows:

[0063] The missing features of the modes with the same size of each mode are added together to obtain the general prompt of the mode. The calculation formula is:

[0064]

[0065] Among them, p c is the general prompt of the mode; n is the total number of modes; represents the modal missing feature of the i-th mode, L i and D i Respectively represent x i The sequence length and feature dimension of f d (·) indicates that the feature dimension is adjusted to d; f l (·) indicates adjusting the sequence length to l.

[0066] In step 7 above, if Figure 4 As shown, an alternative way to get a modal-specific prompt is as follows:

[0067] 7.1) Use the Frobenius norm and the modal missing features of each modality with a uniform size to construct a gated fusion mechanism to determine the modal missing situation. Specifically, calculate the Frobenius norm of the modal missing features of each modality with a uniform size. If the result is greater than 0, it means that the corresponding modal feature is not missing. Let the learnable matrix coefficient b i is 1; if the result is 0, it means that the corresponding modal feature is missing, so the learnable matrix coefficient b i is 0, and the calculation formula is:

[0068]

[0069] in,‖·‖ F represents the Frobenius norm, which is used to determine whether the calculated matrix is ​​an all-zero matrix, and then determine whether the corresponding modal feature is missing; b i are the coefficients of the learnable matrix.

[0070] 7.2) Using the learnable matrix coefficient b i The fusion of the learnable matrix corresponding to the control modality generates modality-specific prompts. Specifically, the learnable matrix coefficients are multiplied with the learnable matrix of the corresponding modality and then added to each other. The calculation formula is:

[0071]

[0072] Among them, p s is a modality-specific prompt; M i ∈R l×d is the learnable matrix corresponding to mode i; b i is the coefficient of the learnable matrix, which is used to determine whether the corresponding learnable matrix participates in the generation of modality-specific cues.

[0073] In step 8 above, an optional way to obtain the specific-general prompt is as follows:

[0074] Add the modal general prompt and the modal specific prompt to get the specific-general prompt, which is calculated as:

[0075] p s-c =p s +p c

[0076] Among them, p s-c Is a specific-general prompt.

[0077] In the above step 9, in order to enable the model to accept the complete multimodal model input without processing the input of each modality separately, a single-stream multimodal model is selected. Furthermore, the present invention does not rely on a specific model in design and is not limited to specific tasks. It selects a single-stream Transformer architecture multimodal model that is general, commonly used and easy to integrate into other models.

[0078] In step 10 above, an optional method for completing the processing and reasoning of downstream tasks is as follows:

[0079] The multimodal downstream tasks involved in the present invention are mainly divided into three types: multimodal sentiment analysis, multimodal sentiment classification and multimodal video retrieval. Among them, multimodal sentiment analysis requires the model to output continuous sentiment scores, and multimodal sentiment classification requires the model to assign each sample to a specific discrete emotion category (such as happiness, anger, sadness, etc.). The flowcharts of the present invention for these two tasks are as follows: Figure 1As shown; Multimodal video retrieval requires the model to retrieve the most relevant video from the video collection according to a given text query, or to retrieve the most relevant text description according to a given video content. The flowchart of the present invention for this task is as shown Figure 2 As shown; for each task, the calculation formula of the corresponding loss function is listed here:

[0080] The loss function used for the multimodal sentiment analysis task is calculated as:

[0081]

[0082] in, represents the L1 loss function; N represents the number of samples; y j and They represent the true sentiment label and predicted sentiment score of the j-th sample respectively. The predicted sentiment score refers to the last token output by the multimodal model after being processed by the fully connected layer.

[0083] The loss function for the multimodal emotion classification task is calculated as:

[0084]

[0085] in, represents the cross entropy loss function; N represents the number of samples; C represents the total number of classification categories; y j,c represents the true label of the jth sample under category c; Represents the model's predicted probability for category c. The predicted probability refers to the last token output by the multimodal model after being processed by the fully connected layer.

[0086] The loss function for the multimodal video retrieval task is calculated as:

[0087]

[0088] in, is the bidirectional maximum margin sorting loss function; N is the number of samples; sim(v j ,c k ) is the video representation v j With subtitles indicating c k (negative sample pair); sim(v j ,c j ) is the video representation v j The corresponding correct subtitle is c j The similarity score between (positive sample pairs); m represents the interval, which is a hyperparameter used to control the minimum gap between the similarity of positive sample pairs and the similarity of negative sample pairs. The above video shows v jIt refers to the last token output by the multimodal model after being processed by the fully connected layer. The subtitles represent c k It refers to the subtitle text generated by the subtitle representation generation module after being pre-trained BERT encoded. The subtitle representation generation process is common knowledge in the field and will not be repeated here.

[0089] The whole framework is pre-trained using a high-resource dataset, and then most of the framework parameters are frozen. The prompts and prompt generation-related architectures and fully connected layers are fine-tuned using a low-resource dataset. During fine-tuning training, for different tasks, the corresponding loss function is used and the gradient descent learning method is adopted to fine-tune the trainable parameters in the framework (including the dimension adjustment layer, the modality-specific prompt generation module, the modality-general prompt generation module, and the fully connected layer), so that the model can complete the corresponding multimodal tasks. When performing the corresponding multimodal tasks, the multimodal features of the video are obtained. If a certain modality is missing, it is directly replaced by an all-zero matrix of the same size. If it is not missing, all modal features are used to perform reasoning.

[0090] The above method is applied to the following embodiments to demonstrate the technical effects of the present invention, and the specific steps in the embodiments are not repeated here.

[0091] The present invention conducts experiments on five multimodal datasets CMU-MOSEI, CMU-MOSI, IEMOCAP, MSR-VTT and MSVD, among which CMU-MOSEI (a high-resource dataset, used for pre-training) and CMU-MOSI are used to evaluate multimodal sentiment analysis tasks, IEMOCAP is used to evaluate multimodal emotion classification tasks, MSR-VTT (a high-resource dataset, used for pre-training) and MSVD are used to evaluate multimodal video retrieval tasks.

[0092] In order to objectively evaluate the performance of the present invention, the present invention uses the binary classification accuracy (Acc2), F1 score, multi-classification accuracy (Acc7), Pearson correlation coefficient (Corr), and mean absolute error (MAE) evaluation criteria to evaluate the sentiment analysis effect in the selected data set, wherein, except for MAE, the higher the index value means the better the performance of the model; the unweighted average recall rate (UAR, Unweighted Average Recall), accuracy (Acc) and F1 score evaluation criteria are used to evaluate the emotion classification effect, and the higher the index value means the better the performance of the model; the Rank@K (R@K), Mean Rank (Mean@R) and Median Rank (Med@R) evaluation criteria are used to evaluate the video retrieval effect, wherein, except for R@K, the lower the index value means the better the performance of the model.

[0093] To verify the effectiveness of the present invention, in the multimodal sentiment analysis task, this embodiment selected Self-MM, MMIM, CENet and ALMT, four currently popular and high-performance models, as benchmark models for comparative experiments; in the multimodal emotion classification task, this embodiment selected MMGCN, MM-DFN, MultiEMO and SDT, four currently popular and high-performance models, as benchmark models for comparative experiments. The experiment used three modes, so there were 2 3 modal conditions, of which 2 3 -1 missing modality, specifically 3 missing one modality, 3 missing two modalities and 1 missing three modalities. In the experiment, the present invention sets the missing ratio of each case to be the same, 5%, so the completeness rate of the data is 100%-(2 3 -1)×5%=65%.

[0094] In order to verify that the present invention has a certain degree of versatility in terms of the number of modalities and tasks, this embodiment selects a multimodal video retrieval task that is quite different from the above-mentioned task, and conducts comparative experiments with the MMT model under 4 and 5 modalities. The tasks are specifically divided into two types: (1) given a text query, retrieve the most relevant video from a video collection (Text to Video); (2) based on a given video content, retrieve the most relevant text description (Video to Text). Since the data set selected for the experiment itself has modality missing, no additional modality missing processing is performed.

[0095] According to the steps described in the specific implementation manner, the experimental results obtained are shown in Tables 1 to 5, and the method of the present invention is expressed as Specific-Common Prompt.

[0096] Table 1: Evaluation results of sentiment analysis of modality-missing scenes obtained by the present invention for the CMU-MOSEI dataset

[0097] Model Acc2 F1 Acc7 Corr MAE Self-MM 72.5 72.3 37.1 0.614 0.865 MMIM 68.2 68.2 41.5 0.638 0.729 CENet 68.6 68.3 42.9 0.639 0.731 ALMT 74.4 74.5 41.0 0.637 0.775 Specific-Common Prompt (the present invention) 78.9 78.8 44.6 0.655 0.729

[0098] Table 2: Evaluation results of sentiment analysis of modality-missing scenes obtained by the present invention for the CMU-MOSI dataset

[0099] Model Acc2 F1 Acc7 Corr MAE Self-MM 70.4 70.3 30.4 0.551 1.092 MMIM 65.9 65.9 32.3 0.584 1.002 CENet 66.5 66.8 34.4 0.588 0.998 ALMT 72.3 72.1 31.1 0.589 1.049 Specific-Common Prompt (the present invention) 76.7 76.8 36.2 0.600 0.998

[0100] Table 3: Evaluation results of emotion classification in modality-missing scenes obtained by the present invention on the IEMOCAP dataset

[0101] Model UAR F1 Acc MMGCN 53.0 52.8 57.1 MM-DFN 57.5 57.1 57.4 MultiEMO 58.1 57.5 60.9 SDT 63.2 63.3 65.3 Specific-Common Prompt (the present invention) 68.8 67.2 67.2

[0102] Table 4: Evaluation results of modality-missing scene video retrieval obtained by the present invention for the MSR-VTT dataset

[0103]

[0104]

[0105] Table 5: Evaluation results of modality-missing scene video retrieval obtained by the present invention for the MSVD dataset

[0106]

[0107] It can be found from Tables 1, 2, and 3 that the method of the present invention has achieved the best results in all indicators of multimodal sentiment analysis tasks and multimodal emotion classification tasks, and the performance of multimodal sentiment analysis and multimodal emotion classification is significantly better than the original advanced methods. Benefiting from the introduction of a specific-general prompt method that can be decomposed into modality-specific prompts and modality-general prompts for modality-missing scenarios, the present invention promotes the capture of cross-modal general information in multimodal information and modality-specific information within the modality, including but not limited to the situation of modality missing, and realizes the full use of the complementarity of multimodal information.

[0108] It can be found from Tables 4 and 5 that when the number of modalities is expanded to 4 and 5, and the task becomes a multimodal video retrieval task, the method of the present invention also achieves the best results in all indicators. The method of the present invention has a certain degree of versatility in the number of modalities and tasks.

[0109] In this embodiment, a missing perception prompt system based on modality-specific and general information is also provided, which is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0110] A feature extraction module, which uses a feature extractor to extract multimodal initial features from the original video samples;

[0111] A modality missing processing module is used to perform partial modality missing processing on the multimodal initial features of the original video sample to obtain modality missing features;

[0112] The dimension adjustment layer is used to map the modal missing features to the same feature dimension space to obtain the modal missing features of the unified feature dimension;

[0113] The modality-specific cue generation module is used to map the modality missing features of uniform feature dimension to the same sequence length space to obtain the modality missing features of uniform size, and then use the Frobenius norm and the modality missing features of uniform size of each modality to construct a gated fusion mechanism to control the fusion of the modality corresponding learnable matrix to generate modality-specific cues;

[0114] The modality universal prompt generation module is used to fuse the modality missing features of the same size in each modality to generate a modality universal prompt;

[0115] A specific-universal cue generation module, which is used to fuse modality-specific cues and modality-universal cues to generate specific-universal cues as missing perception cues;

[0116] An input representation module is used to concatenate the missing features of the modalities of the unified feature dimension of each modality to form a multimodal joint feature; the multimodal joint feature is concatenated after the specific-general prompt as the input of the multimodal model;

[0117] A multimodal model that generates outputs for downstream tasks based on the concatenation of loss-aware cues and multimodal joint features;

[0118] The downstream video task module is used to process the last token output by the multimodal model according to different video tasks to obtain the downstream video task results.

[0119] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0120] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0121] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A missing perception prompting method based on modality-specific and general information, characterized in that: include: Obtain original video samples and text annotations, perform multimodal feature extraction on the original video samples, and obtain multimodal initial features; The text annotations refer to the original video sample labels, which are determined according to the downstream tasks; Perform partial modality missing processing on the multimodal initial features of the original video samples to obtain modality missing features; Map the modal missing features to the same feature dimension space to obtain the modal missing features of the unified feature dimension; concatenate the modal missing features of the unified feature dimension to obtain multi-modal joint features; Map the modal missing features of the unified feature dimension to the same sequence length space to obtain the modal missing features of the unified size, and generate the modal universal prompt after fusion; A gated fusion mechanism is constructed using the Frobenius norm and the uniform-sized modality missing features to determine the modality missing situation and control the fusion of the modality corresponding learnable matrix to generate modality-specific prompts. Fusion of modality-universal cues and modality-specific cues to generate specific-universal cues as missing perception cues; The multimodal joint features are spliced ​​after the missing perception prompt, input into the pre-trained multimodal model, extract the last token in the output of the pre-trained multimodal model, input into the fully connected layer, and complete the training of the multimodal model, fully connected layer, feature mapping, and learnable matrix based on the downstream video task according to the output result of the fully connected layer.

2. The missing perception prompting method based on modality-specific and general information according to claim 1, characterized in that: The feature extraction of the original video sample to obtain the initial feature includes: A pre-trained feature extraction model is used as a feature extractor of a modality in multi-modality, an original video is input, and corresponding modality features are extracted as initial features of the modality; feature extractors of different modalities are used to obtain multi-modal initial features of the original video; the multi-modality includes text modality, visual modality, and audio modality.

3. The missing perception prompting method based on modality-specific and general information according to claim 1, characterized in that: The method of performing partial modality missing processing on the multimodal initial features of the original video sample to obtain modality missing features includes: Some modal features of the original video samples are replaced with all-zero matrices of the same size to indicate modal loss; for all samples, 2 n -1 modality missing cases are set with the same probability, and n is the total number of modalities of the original video samples.

4. The missing perception prompting method based on modality-specific and general information according to claim 1, characterized in that: The calculation formula of the modal general prompt is: Among them, p c is the modality-general cue; n is the total number of modalities of the original video sample; represents the modal missing feature of the i-th mode, L i and D i Respectively represent x i The sequence length and feature dimension of f d (·) indicates that the feature dimension is adjusted to d; f l (·) indicates adjusting the sequence length to l.

5. The missing perception prompting method based on modality-specific and general information according to claim 1, characterized in that: The calculation formula for the modality-specific prompt is: Among them, p s is a modality-specific prompt; M i ∈R l×d is the learnable matrix; l×d represents the sequence length and feature dimension of the learnable matrix, which is consistent with the modal missing feature size of uniform size; ‖·‖ F represents the Frobenius norm, which is used to determine whether the calculated matrix is ​​an all-zero matrix, and then determine whether the corresponding modal feature is missing; b i is the coefficient of the learnable matrix corresponding to modality i, which is used to determine whether the corresponding learnable matrix participates in the generation of modality-specific prompts.

6. The missing perception prompting method based on modality-specific and general information according to claim 1, characterized in that: The specific-general prompt is obtained by adding the modal general prompt and the modal specific prompt.

7. The missing perception prompting method based on modality-specific and general information according to claim 1, characterized in that: The multimodal joint features are concatenated after the missing perception prompt to obtain the input of the multimodal model, which is expressed as: x input =[p s-c ,f d (x1),f d (x2),…,f d (x n )] Among them, x input represents the input of the multimodal model; [...] represents the concatenation operation, p s-c Indicates specific-general prompts, f d (x n ) represents the modal missing features of the unified feature dimension of modal n; the multimodal model adopts a single-stream Transformer model.

8. The missing perception prompting method based on modality-specific and general information according to claim 1, characterized in that: The downstream video tasks include multimodal sentiment analysis tasks, multimodal emotion classification tasks and multimodal video retrieval tasks.

9. The missing perception prompting method based on modality-specific and general information according to claim 8, characterized in that: The multimodal sentiment analysis task adopts the L1 loss function, the multimodal emotion classification task adopts the cross entropy loss function, and the multimodal video retrieval task adopts the bidirectional maximum margin ranking loss function.

10. A missing perception prompt system based on modality-specific and general information, used to implement the method of claim 1, characterized in that: include: A feature extraction module, which uses a feature extractor to extract multimodal initial features from the original video samples; A modality missing processing module is used to perform partial modality missing processing on the multimodal initial features of the original video sample to obtain modality missing features; The dimension adjustment layer is used to map the modal missing features to the same feature dimension space to obtain the modal missing features of the unified feature dimension; The modality-specific cue generation module is used to map the modality missing features of uniform feature dimension to the same sequence length space to obtain the modality missing features of uniform size, and then use the Frobenius norm and the modality missing features of uniform size of each modality to construct a gated fusion mechanism to control the fusion of the modality corresponding learnable matrix to generate modality-specific cues; The modality universal prompt generation module is used to fuse the modality missing features of the same size in each modality to generate a modality universal prompt; A specific-universal cue generation module, which is used to fuse modality-specific cues and modality-universal cues to generate specific-universal cues as missing perception cues; An input representation module is used to concatenate the missing features of the modalities of the unified feature dimension of each modality to form a multimodal joint feature; the multimodal joint feature is concatenated after the specific-general prompt as the input of the multimodal model; A multimodal model that generates outputs for downstream tasks based on the concatenation of loss-aware cues and multimodal joint features; The downstream video task module is used to process the last token output by the multimodal model according to different video tasks to obtain the downstream video task results.

Images