Emotion prediction method and system based on missing modal robust learning

Abstract

The present application provides a sentiment prediction method and system based on missing modal robust learning, relating to the technical field of multi-modal sentiment analysis. The present application extracts complete initial features in sentiment data samples, outputs corresponding missing input features, missing masks and missing form markers according to a course modal missing mechanism, constructs a consistency distillation framework including a teacher model and a student model and trains to obtain corresponding modal features output by the student model under the condition of the current missing input features, splices the corresponding modal features, the missing masks and the missing form markers to obtain a gating input vector, uses a gating network to output original gating weights of each mode and normalizes, applies sparsity constraints to the normalized weights to obtain redistributed weights, and after adaptive weighted fusion of robust global fusion features, outputs stable and accurate sentiment prediction results, and overall improves the robustness of multi-modal sentiment prediction.

Description

Technical Field

[0001] This invention relates to the technical field of multimodal sentiment analysis, and more specifically, to a sentiment prediction method and system based on robust learning of missing modalities. Background Technology

[0002] With the rapid development of artificial intelligence and affective computing, multimodal sentiment analysis (MSA) technology has been widely used in scenarios such as short video sentiment analysis, online education, human-computer interaction systems, and intelligent customer service. Existing mainstream multimodal sentiment analysis methods usually rely on the assumption of "modal completeness," that is, the system is assumed to be able to simultaneously and completely acquire multimodal data such as text, speech, and vision for feature extraction, cross-modal alignment, and fusion.

[0003] However, in real-world engineering applications, visual and acoustic signals are often incomplete due to external interference, equipment damage, network latency, or user privacy restrictions (such as actively turning off the microphone or camera). This "missing modality" typically includes sample-level whole-modal loss (e.g., no audio in an entire video), temporal segment / block loss (e.g., a section with silence or no video), and finer-grained frame or word segment loss. When faced with these modality loss scenarios, traditional multimodal sentiment analysis models built on the "completeness assumption" often experience drastic performance degradation or even inference failure.

[0004] To address this issue, existing technologies have proposed some solutions for incomplete modes, but they still have the following drawbacks: 1. Reconstruction- or Generative-Based Repair Methods: Some existing solutions attempt to utilize Generative Adversarial Networks (GANs) or cross-modal mapping techniques to "reconstruct" the missing visual or audio features before fusing them. This approach not only incurs high computational and memory costs, but also, in real-world complex missing scenarios, due to the significant semantic gap between cross-modal representations, forced reconstruction often introduces a large amount of noise, leading to unreliable prediction results.

[0005] 2. Lack of robust training methods with systematic scheduling: Some studies have introduced knowledge distillation or contrastive learning mechanisms to align the representations of complete and missing samples. However, most of these methods directly input randomly missing data into the model for training. When faced with extremely high missing rates or sudden large-area missing data, due to the lack of reasonable training difficulty intervention and dynamic modality fusion strategies, the model is prone to gradient explosion, optimization instability, or representation collapse. Summary of the Invention

[0006] To address the issue of inaccurate sentiment prediction results under modality missing conditions in existing technologies, this invention proposes a sentiment prediction method and system based on robust learning of missing modalities. It combines a curriculum-based modality missing mechanism with a consistency distillation architecture and a missing-aware fusion gate, effectively overcoming the robustness bottleneck under extreme missing distributions.

[0007] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A sentiment prediction method based on robust learning of missing modalities includes the following steps: S1: Obtain the original multimodal sentiment data sample, and extract the complete initial features of each modality in the original multimodal sentiment data sample respectively; S2: According to the preset course-based modal missing mechanism, calculate the missing rate of the complete initial features, perform dynamic masking processing on the complete initial features based on the missing rate, and output the corresponding missing input features, missing mask and missing morphology marker. S3: Construct and train a consistency distillation framework that includes a teacher model and a student model. Input the complete initial features into the teacher model and input the corresponding missing input features into the student model to obtain the corresponding modal features output by the student model under the current missing input feature conditions. S4: Based on the corresponding modal features, obtain the soft quality information vector, jointly encode the missing mask and missing morphology marker to form the hard missing information vector; concatenate the soft quality information vector and the hard missing information vector to obtain the gated input vector; S5: Input the gated input vector into the lightweight gated network in the preset missing perception fusion gate, output the original gated weights of each modality and normalize them, apply sparsity constraints to the normalized weights, and obtain the redistributed weights. S6: Using the redistributed weights, the feature representations of each modality are weighted and summed to obtain cross-modal sentiment features; S7: Based on the cross-modal sentiment features, obtain the final sentiment prediction result.

[0008] Preferably, the modalities in the original multimodal emotion data samples include text modality, speech modality, and visual modality. Using a pre-trained feature extraction network, complete initial features are extracted for each of the three modalities: text, speech, and vision. The expression for the complete initial features is as follows: ,in, Represents text modality, Represents speech modality, The visual modality is represented; the complete initial features are extracted offline using a frozen pre-trained unimodal encoder.

[0009] Preferably, the expression for calculating the missing rate is:

[0010] in, Indicates the missing rate. and These represent the preset minimum and maximum missing rates, respectively. Indicates the scaling factor. This represents the Sigmoid function. Indicates the inflection point of difficulty. Indicates the normalized training progress. , , Indicates the current training batch. This indicates the total number of training cycles.

[0011] Preferably, the process of performing dynamic masking on the complete initial features is as follows: Based on the current training progress s stage, a masking operation is performed on the complete initial features based on the missing rate. In the early stages of training, independent random missing features are the primary method. For the feature vector of each complete initial feature, the missing rate is used as the masking method. Randomly set some feature values ​​to zero; in the middle of the training process, introduce structural missing features, including both integer modal missing features and continuous block missing features. For integer modal missing features, use the missing rate... One modality is randomly selected with a probability of zeroing all its features; in cases of continuous block missing data, a starting point is randomly selected for consecutive text word segments, speech segments, or video segments, based on the missing rate. Features of a frame or segment are continuously discarded according to the length ratio; in the later stages of training, the proportion of structural missing features is maintained so that the missing rate is close to the preset maximum missing rate.

[0012] Preferably, based on the outputs of the teacher model and the student model, multi-level consistency constraints are applied. These multi-level consistency constraints include Logit consistency constraints, intermediate representation consistency constraints, and category prototype consistency constraints. The multi-level distillation loss is calculated and summed according to preset weights to obtain the total consistency distillation loss. Based on the total consistency distillation loss, the parameters of the student model are updated only through backpropagation and gradient descent algorithms. The parameters of the teacher model are then updated based on the parameters of the student model using the exponential moving average (EMA) strategy.

[0013] Preferably, the process of obtaining the soft mass information vector based on the corresponding modal features is as follows: Based on the modal features output by the student model under the current missing input conditions, the modality usable confidence index is calculated. ,in, , Represents text modality, Represents speech modality, Represents visual modality; For text modalities, the uncertainty is calculated based on the entropy of the output probability distribution of the feature extractor or the prediction confidence, thus obtaining the usable confidence level of the text modality. ; For speech modalities, the signal-to-noise ratio or signal profundity is estimated to obtain the usable confidence level of the speech modality. ; For visual modalities, the blurriness and contrast of the input image are calculated, or the input image is fed into a pre-trained quality assessment network to extract a quality score, thus obtaining the usable confidence score of the visual modality. ; Confidence metrics for all modalities , , By concatenating the data, we obtain the soft mass information vector.

[0014] Preferably, the gated input vector is input into a lightweight gated network in a preset missing-aware fusion gate, and the original gated weights for each modality are output and normalized to satisfy the expression:

[0015] in, This represents the initial fusion weights of the k-th modality after Softmax normalization. The original gating weights for each mode are represented. Sparse constraints are applied to the normalized weights, including an L1 regularization term and an entropy regularization term. The L1 regularization term satisfies the expression:

[0016] The entropy regularization term satisfies the expression:

[0017] The total loss function for sentiment prediction is:

[0018] in, This indicates the loss inherent in the sentiment prediction task itself; This represents the total loss during uniform distillation; Indicates the regularization weight coefficient; The gated input vector is input into a lightweight gated network within a pre-defined missing-aware fusion gate, and the redistributed fusion weights are obtained through forward propagation. .

[0019] Preferably, the feature representations of each modality are weighted and summed using the redistributed weights to obtain cross-modal sentiment features. ;

[0020] in, , , These are the final fusion weights assigned to the text, speech, and vision modalities after sparse constraint adjustment. , , These correspond to the modal features output by the student model under the current missing input conditions for the text, speech, and visual modalities, respectively.

[0021] This invention also proposes a sentiment prediction system based on missing modality robust learning to implement the aforementioned sentiment prediction method based on missing modality robust learning. The system includes: The feature extraction module is used to acquire the original multimodal sentiment data samples and extract the complete initial features of each modality in the original multimodal sentiment data samples. The course-based modal missing module is used to calculate the missing rate of the complete initial features according to the preset course-based modal missing mechanism, perform dynamic masking processing on the complete initial features based on the missing rate, and output the corresponding missing input features, missing mask and missing morphology marker. The Consistent Distillation module is used to construct and train a consistent distillation framework that includes a teacher model and a student model. Complete initial features are input into the teacher model, and its corresponding missing input features are input into the student model to obtain the corresponding modal features output by the student model under the current missing input features. The missing information fusion module is used to obtain a soft quality information vector based on the corresponding modal features, jointly encode the missing mask and missing morphology marker to form a hard missing information vector; concatenate the soft quality information vector and the hard missing information vector to obtain a gating input vector; input the gating input vector into a lightweight gating network in a preset missing information fusion gate, output the original gating weights of each modality, normalize them, apply sparsity constraints to the normalized weights to obtain the redistributed weights; and use the redistributed weights to perform a weighted summation of the feature representations of each modality to obtain cross-modal sentiment features. The prediction output module is used to obtain the final sentiment prediction result based on the cross-modal sentiment features.

[0022] The present invention also proposes a computer device including a memory and a processor, wherein the memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the above-described sentiment prediction method based on missing modality robust learning.

[0023] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention proposes a sentiment prediction method and system based on robust learning of missing modalities. It extracts complete initial features for each modality from the original multimodal sentiment data samples, ensuring that the extracted complete initial features fully retain the core sentiment semantic information of that modality. According to a pre-defined curriculum-based modality missing mechanism, the complete initial features are subjected to missing rate calculation and masking operations, outputting corresponding missing input features, missing masks, and missing morphological markers, thus improving robustness under complex missing scenarios. A consistency distillation framework including a teacher model and a student model is constructed and trained. The complete initial features are input into the teacher model, and its corresponding missing input features are input into the student model to obtain the student model's performance. The invention outputs corresponding modal features under the current missing input condition, ensuring that even with missing features, the student model's output modal features can still effectively approximate the complete initial feature representation. A gated input vector is obtained by concatenating the soft quality information vector with the hard missing information vector. This vector is then input into a pre-defined lightweight gating network. The gating input vector provides a more sufficient decision-making basis for allocating fusion weights. The original gating weights for each modality are output and normalized. Sparsity constraints are applied to the normalized weights to further improve the rationality of weight allocation, resulting in redistributed weights. Weighted summation is then performed on the feature representations of each modality to obtain robust cross-modal sentiment features, ultimately yielding the final sentiment prediction result. This invention can output stable and accurate sentiment prediction results even in real-world scenarios with arbitrary modal missing features and uneven input quality, effectively improving the overall robustness of multimodal sentiment prediction. Attached Figure Description

[0024] Figure 1 A flowchart illustrating the sentiment prediction method based on robust learning of missing modalities proposed in this embodiment of the invention; Figure 2 This diagram illustrates the structure of the sentiment prediction system based on robust learning of missing modalities proposed in this embodiment of the invention. Figure 3 A flowchart illustrating the uniform distillation module proposed in this embodiment of the invention; Figure 4 This is a flowchart illustrating the missing information perception fusion module proposed in this embodiment of the invention. Figure 5 This is a schematic diagram of the structure of the computer device proposed in the embodiments of the present invention. Detailed Implementation

[0025] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes a sentiment prediction method based on robust learning of missing modalities. See [link to relevant documentation]. Figure 1 This includes the following steps: S1: Obtain the original multimodal sentiment data samples and extract the complete initial features of each modality in the original multimodal sentiment data samples; S2: Based on the preset course-based modal missing mechanism, calculate the missing rate of the complete initial features, perform dynamic masking on the complete initial features based on the missing rate, and output the corresponding missing input features, missing mask and missing morphology marker. S3: Construct and train a consistency distillation framework that includes a teacher model and a student model. Input the complete initial features into the teacher model and input the corresponding missing input features into the student model to obtain the corresponding modal features output by the student model under the current missing input conditions. S4: Based on the corresponding modal features, obtain the soft quality information vector, jointly encode the missing mask and missing morphology marker to form the hard missing information vector; concatenate the soft quality information vector and the hard missing information vector to obtain the gated input vector; S5: Input the gated input vector into the lightweight gated network in the preset missing perception fusion gate, output the original gated weights of each modality and normalize them, apply sparsity constraints to the normalized weights, and obtain the redistributed weights. S6: Using the redistributed weights, the feature representations of each modality are weighted and summed to obtain cross-modal sentiment features; S7: Obtain the final sentiment prediction result based on cross-modal sentiment features.

[0028] In this embodiment, complete initial features are extracted to transform the original modality data samples into digitized features, providing basic feature inputs for subsequent dynamic missing feature construction and feature fusion, ensuring that the initial features fully retain the core emotional semantic information of the modality. This embodiment also employs a curriculum-based modality missing feature mechanism to dynamically control the modality missing rate and missing feature morphology during training, significantly improving the generalization and adaptation capability to complex missing feature scenarios. Based on the consistency distillation framework, a teacher model with complete initial features and more reliable knowledge supervises the learning of student models with missing inputs, improving the prediction accuracy of the student models. By concatenating soft-quality information and hard missing feature information, more sufficient decision-making basis is provided for the subsequent gating network to allocate fusion weights, enabling a more accurate characterization of the actual usability of the input modality. This embodiment utilizes sparse regularization constraints to drive weights towards usable modalities, suppressing invalid interference from missing or low-quality modalities, further improving the rationality of weight allocation, fully mining the effective emotional information of existing modalities, and obtaining robust global fusion features, which is conducive to obtaining more accurate sentiment prediction results.

[0029] Example 2 This embodiment further proposes a sentiment prediction method based on robust learning of missing modalities, building upon Embodiment 1. The original multimodal sentiment data samples include text, speech, and visual modalities. A pre-trained feature extraction network is used to extract complete initial features for each of the three modalities: text, speech, and vision. The expression for the complete initial features is as follows: ,in, Represents text modality, Represents speech modality, Representing visual modalities; extracting complete initial features offline using a frozen pre-trained unimodal encoder. In this embodiment, a frozen pre-trained single-modal encoder is preferred to extract complete initial features offline, and lightweight training is only performed on the subsequent fusion layer and loss network to control memory overhead.

[0030] In one alternative embodiment, the expression for calculating the missing rate is:

[0031] in, Indicates the missing rate. and These represent the preset minimum and maximum missing rates, respectively. Indicates the scaling factor. This represents the Sigmoid function. Indicates the inflection point of difficulty. Indicates the normalized training progress. , , Indicates the current training batch. This indicates the total number of training cycles.

[0032] The process of performing dynamic masking on complete initial features is as follows: Based on the current training progress s stage, a masking operation is performed on the complete initial features based on the missing rate. In the early stages of training, independent random missing features are the primary method. For the feature vector of each complete initial feature, the missing rate is used as the masking method. Randomly set some feature values ​​to zero; in the middle of the training process, introduce structural missing features, including both integer modal missing features and continuous block missing features. For integer modal missing features, use the missing rate... One modality is randomly selected with a probability of zeroing all its features; in cases of continuous block missing data, a starting point is randomly selected for consecutive text word segments, speech segments, or video segments, based on the missing rate. Features of a frame or segment are continuously discarded according to the length ratio; in the later stages of training, the proportion of structural missing features is maintained so that the missing rate is close to the preset maximum missing rate.

[0033] In this embodiment, the curriculum-based modality missing mechanism dynamically schedules the difficulty of missing modalities, gradually increasing the difficulty as training progresses. This ensures stable convergence in the early stages of training while ultimately adapting to complex modality missing conditions in real-world scenarios. The curriculum-based modality missing mechanism provides a smooth optimization path from easy to difficult for the consistency distillation framework, dynamically controlling the modality missing rate and missing modality morphology during training using a monotonic scheduling function. Without this mechanism, performing consistency distillation directly under high missing rates would easily lead to gradient explosion or representation collapse in the student model due to insufficient input information.

[0034] In the early stages of training, independent random missing values ​​are prioritized to quickly adapt to simple missing scenarios and achieve basic convergence. In the middle stages, more complex structural missing values, closer to real-world scenarios, are introduced. Single-only missing values ​​correspond to complete missing values ​​of the entire modality, reflecting situations where a modality is often completely uncollected in real-world scenarios. Continuous block missing values ​​correspond to the loss of continuous local regions within a single modality, such as multiple consecutive words in text, a continuous segment of signal in speech, or multiple consecutive frames in video. These are common missing values ​​due to signal interruption and occlusion in real-world scenarios, and are more challenging and realistic than random single-point missing values. In the later stages of training, a high missing value rate is maintained, and all types of missing values ​​are mixed to provide a rich variety of missing value distributions, fully learning robust fusion strategies for arbitrary missing value scenarios.

[0035] In one optional embodiment, multi-level consistency constraints are applied based on the outputs of the teacher model and the student model. These multi-level consistency constraints include Logit consistency constraints, intermediate representation consistency constraints, and category prototype consistency constraints. The multi-level distillation loss is calculated and summed according to preset weights to obtain the total consistency distillation loss. Based on the total consistency distillation loss, the parameters of the student model are updated only through backpropagation and gradient descent algorithms. The parameters of the teacher model are then updated based on the parameters of the student model using an exponential moving average (EMA) strategy.

[0036] In this embodiment, the consistent distillation framework including the teacher-student model ensures that the model can still effectively approximate the complete initial feature representation even when the feature is missing. Without the consistent distillation framework, the large number of incomplete samples generated by the curriculum-based modality missing mechanism can only play the role of data augmentation and cannot guarantee the consistency of the decision boundary of the model in the complete and missing states.

[0037] The specific design of the uniform distillation framework includes: (1) Forward propagation: The teacher model does not participate in gradient backpropagation, receives complete initial features, and ensures that the teacher model can always provide the "standard answer under the full modality"; the student model receives missing input features generated by the curriculum-based modality missing mechanism.

[0038] (2) The teacher parameters are updated using the exponential moving average (EMA) strategy, expressed as follows:

[0039] in, Indicates the momentum coefficient. This represents the current parameters of the teacher model. This represents the current parameters of the student model.

[0040] (3) Multi-level consistency constraints: The multi-level consistency constraint loss includes temperature-regulated Logit soft target distillation loss, intermediate layer representation alignment loss based on contrastive learning, and category prototype alignment loss. In particular, the category prototype alignment loss constructs a standard prototype by maintaining a global sliding category memory on the complete teacher side, thereby constraining the distribution of student category features in the missing state, bringing student features closer to the corresponding category prototype, and maintaining a preset interval with non-corresponding category prototypes. In the early stage of training, the loss weights are warmed up and smoothed, increasing the loss weights from the initial value to the target weight value.

[0041] In one optional embodiment, the process of obtaining the soft mass information vector based on the corresponding modal features is as follows: Based on the modal features output by the student model under the current missing input conditions, the modality usable confidence index is calculated. ,in, , Represents text modality, Represents speech modality, Represents visual modality; For text modalities, the uncertainty is calculated based on the entropy of the output probability distribution of the feature extractor or the prediction confidence, thus obtaining the usable confidence level of the text modality. ; For speech modalities, the signal-to-noise ratio or signal profundity is estimated to obtain the usable confidence level of the speech modality. ; For visual modalities, the blurriness and contrast of the input image are calculated, or the input image is fed into a pre-trained quality assessment network to extract a quality score, thus obtaining the usable confidence score of the visual modality. ; Confidence metrics for all modalities , , By concatenating the data, we obtain the soft mass information vector.

[0042] In this embodiment, different quality assessment methods are adapted to the physical / semantic characteristics of different modalities, avoiding errors caused by a uniform metric. The confidence indices for the three modalities are... , , The vectors are directly concatenated to obtain the final soft quality information vector. This soft quality information vector, along with the binary missing mask and missing shape label, is input into the subsequent gating network as the basis for fusion weight allocation.

[0043] In an optional embodiment, the gated input vector is input into a lightweight gating network in a preset missing-aware fusion gate, and the original gating weights for each modality are output and normalized to satisfy the expression:

[0044] in, This represents the initial fusion weights of the k-th modality after Softmax normalization. Let represent the original gating weights for each mode. Sparse constraints are applied to the normalized weights, including L1 regularization and entropy regularization. The L1 regularization term satisfies the expression:

[0045] The entropy regularization term satisfies the expression:

[0046] The total loss function for sentiment prediction is:

[0047] in, This indicates the loss inherent in the sentiment prediction task itself; This represents the total loss during uniform distillation; Indicates the regularization weight coefficient; The gated input vector is input into a lightweight gated network within a pre-defined missing-aware fusion gate, and the redistributed fusion weights are obtained through forward propagation. .

[0048] In this embodiment, the L1 regularization term and the entropy regularization term work together to promote weight sparsity, making the weights of unusable modes approach 0, and retaining only the high weights of usable modes. This is the loss value for the L1 regularization term, which penalizes non-zero weights and pushes the weights of more low-quality / missing modes closer to 0. The loss value is the entropy regularization term, which, together with L1 regularization, further promotes the sparsity of the weight distribution.

[0049] In an optional embodiment, the feature representations of each modality are weighted and summed using the redistributed weights to obtain cross-modal sentiment features. ;

[0050] in, , , These are the final fusion weights assigned to the text, speech, and vision modalities after sparse constraint adjustment. , , These correspond to the modal features output by the student model under the current missing input conditions for the text, speech, and visual modalities, respectively.

[0051] In this embodiment, the weights learned by the gating network are used to sum the single-modal features of different qualities, so that the contributions of high-availability / complete modalities are greater and the contributions of low-quality / missing modalities are suppressed, ultimately resulting in robust fusion features.

[0052] Example 3 This embodiment proposes a sentiment prediction system based on robust learning of missing modalities. See [link to documentation]. Figure 2 It includes a feature extraction module, a curriculum-based modality missing module, a consistency distillation module, a missing-aware fusion module, and a prediction output module.

[0053] The feature extraction module is used to obtain the original multimodal sentiment data samples and extract the complete initial features of each modality in the original multimodal sentiment data samples.

[0054] The curriculum-based modal missing module is used to calculate the missing rate of complete initial features according to the preset curriculum-based modal missing mechanism. Based on the missing rate, it performs dynamic masking processing on the complete initial features and outputs the missing input features, missing mask and missing morphology markers corresponding to the complete initial features.

[0055] The Consistent Distillation module is used to construct and train a consistent distillation framework that includes a teacher model and a student model. Complete initial features are input into the teacher model, and its corresponding missing input features are input into the student model to obtain the corresponding modal features output by the student model under the current missing input feature conditions. See also Figure 3 Based on the outputs of the teacher model and the student model, multi-level consistency constraints are applied, including Logit consistency constraints, intermediate representation consistency constraints, and category prototype consistency constraints, resulting in the total consistency distillation loss.

[0056] like Figure 4 As shown, the missing information fusion module is used to calculate the modality available confidence index based on the corresponding modality features, and concatenate the confidence indices of all modalities to obtain a soft quality information vector. The missing mask and missing morphology marker are jointly encoded to form a hard missing information vector. The soft quality information vector and the hard missing information vector are concatenated to obtain a gating input vector. The gating input vector is input into a lightweight gating network in a preset missing information fusion gate, outputting the original gating weights of each modality, which are then normalized. Sparsity constraints are applied to the normalized weights to obtain the redistributed weights. The redistributed weights are then used to perform a weighted summation of the feature representations of each modality to obtain cross-modal sentiment features.

[0057] The prediction output module is used to obtain the final sentiment prediction result based on cross-modal sentiment features.

[0058] This embodiment also provides a computer device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described embodiment of the sentiment prediction method based on missing modality robust learning and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0059] For details, see Figure 5 The computer device provided in this application embodiment includes a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405, and a memory 406.

[0060] Transceiver 402 is used to acquire raw multimodal sentiment data samples. Bus 401 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus interface 404 provides an interface between bus 401 and transceiver 402. Processor 405 is responsible for managing bus 401 and general processing, and memory 406 can be used to store data used by processor 405 during operation.

[0061] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A sentiment prediction method based on robust learning of missing modalities, characterized in that, Includes the following steps: S1: Obtain the original multimodal sentiment data sample, and extract the complete initial features of each modality in the original multimodal sentiment data sample respectively; S2: According to the preset course-based modal missing mechanism, calculate the missing rate of the complete initial features, perform dynamic masking processing on the complete initial features based on the missing rate, and output the corresponding missing input features, missing mask and missing morphology marker. S3: Construct and train a consistency distillation framework that includes a teacher model and a student model. Input the complete initial features into the teacher model and input the corresponding missing input features into the student model to obtain the corresponding modal features output by the student model under the current missing input feature conditions. S4: Based on the corresponding modal features, obtain the soft quality information vector, jointly encode the missing mask and missing morphology marker to form the hard missing information vector; concatenate the soft quality information vector and the hard missing information vector to obtain the gated input vector; S5: Input the gated input vector into the lightweight gated network in the preset missing perception fusion gate, output the original gated weights of each modality and normalize them, apply sparsity constraints to the normalized weights, and obtain the redistributed weights. S6: Using the redistributed weights, the feature representations of each modality are weighted and summed to obtain cross-modal sentiment features; S7: Based on the cross-modal sentiment features, obtain the final sentiment prediction result.

2. The sentiment prediction method based on robust learning of missing modalities according to claim 1, characterized in that, The original multimodal emotion data samples include text, speech, and visual modalities. Using a pre-trained feature extraction network, complete initial features are extracted for each of the three modalities: text, speech, and vision. The expression for these complete initial features is as follows: ,in, Represents text modality, Represents speech modality, The visual modality is represented; the complete initial features are extracted offline using a frozen pre-trained unimodal encoder.

3. The sentiment prediction method based on missing modality robust learning according to claim 1, characterized in that, The expression for calculating the missing rate is: in, Indicates the missing rate. and These represent the preset minimum and maximum missing rates, respectively. Indicates the scaling factor. This represents the Sigmoid function. Indicates the inflection point of difficulty. Indicates the normalized training progress. , , Indicates the current training batch. This indicates the total number of training cycles.

4. The sentiment prediction method based on missing modality robust learning according to claim 3, characterized in that, The process of performing dynamic masking on the complete initial features is as follows: Based on the current training progress s stage, a masking operation is performed on the complete initial features based on the missing rate. In the early stages of training, independent random missing features are the primary method. For the feature vector of each complete initial feature, the missing rate is used as the masking method. Randomly set some feature values ​​to zero; in the middle of the training process, introduce structural missing features, including both integer modal missing features and continuous block missing features. For integer modal missing features, use the missing rate... The probability is used to randomly select one of the modalities and set all its features to zero; In cases of consecutive block missing data, for consecutive text word segments, audio segments, or video segments, a starting point is randomly selected based on the missing rate. The feature of continuously discarding a frame or segment based on its length proportion; In the later stages of training, the proportion of structural missing data is maintained so that the missing rate is close to the preset maximum missing rate.

5. The sentiment prediction method based on robust learning of missing modalities according to claim 1, characterized in that, Based on the outputs of the teacher and student models, multi-level consistency constraints are applied, including Logit consistency constraints, intermediate representation consistency constraints, and category prototype consistency constraints. The multi-level distillation loss is calculated and summed according to preset weights to obtain the total consistency distillation loss. Based on the total consistency distillation loss, the parameters of the student model are updated only through backpropagation and gradient descent algorithms. The parameters of the teacher model are updated based on the parameters of the student model through the exponential moving average (EMA) strategy.

6. The sentiment prediction method based on missing modality robust learning according to claim 1, characterized in that, The process of obtaining the soft mass information vector based on the corresponding modal features is as follows: Based on the modal features output by the student model under the current missing input conditions, the modality usable confidence index is calculated. ,in, , Represents text modality, Represents speech modality, Represents visual modality; For text modalities, the uncertainty is calculated based on the entropy of the output probability distribution of the feature extractor or the prediction confidence, thus obtaining the usable confidence level of the text modality. ; For speech modalities, the signal-to-noise ratio or signal profundity is estimated to obtain the usable confidence level of the speech modality. ; For visual modalities, the blurriness and contrast of the input image are calculated, or the input image is fed into a pre-trained quality assessment network to extract a quality score, thus obtaining the usable confidence score of the visual modality. ; Confidence metrics for all modalities , , By concatenating the data, we obtain the soft mass information vector.

7. The sentiment prediction method based on missing modality robust learning according to claim 1, characterized in that, The gated input vector is input into a lightweight gated network in a predefined missing-aware fusion gate. The original gated weights for each modality are output and normalized to satisfy the expression: in, This represents the initial fusion weights of the k-th modality after Softmax normalization. The original gating weights for each mode are represented. Sparse constraints are applied to the normalized weights, including an L1 regularization term and an entropy regularization term. The L1 regularization term satisfies the expression: The entropy regularization term satisfies the expression: The total loss function for sentiment prediction is: in, This indicates the loss inherent in the sentiment prediction task itself; This represents the total loss during uniform distillation; Indicates the regularization weight coefficient; The gated input vector is input into a lightweight gating network within a pre-defined missing-aware fusion gate. After forward propagation through the lightweight gating network, the redistributed fusion weights are obtained. .

8. The sentiment prediction method based on missing modality robust learning according to claim 1, characterized in that, By using the redistributed weights, the feature representations of each modality are weighted and summed to obtain cross-modal sentiment features. ; in, , , These are the final fusion weights assigned to the text, speech, and vision modalities after sparse constraint adjustment. , , These correspond to the modal features output by the student model under the current missing input conditions for the text, speech, and visual modalities, respectively.

9. A sentiment prediction system based on missing modality robust learning, used to implement the sentiment prediction method based on missing modality robust learning as described in any one of claims 1 to 8, characterized in that, include: The feature extraction module is used to acquire the original multimodal sentiment data samples and extract the complete initial features of each modality in the original multimodal sentiment data samples. The course-based modal missing module is used to calculate the missing rate of the complete initial features according to the preset course-based modal missing mechanism, perform dynamic masking processing on the complete initial features based on the missing rate, and output the corresponding missing input features, missing mask and missing morphology marker. The Consistent Distillation module is used to construct and train a consistent distillation framework that includes a teacher model and a student model. Complete initial features are input into the teacher model, and its corresponding missing input features are input into the student model to obtain the corresponding modal features output by the student model under the current missing input features. The missing information fusion module is used to obtain a soft quality information vector based on the corresponding modal features, jointly encode the missing mask and missing morphology marker to form a hard missing information vector, concatenate the soft quality information vector and the hard missing information vector to obtain a gating input vector, input the gating input vector into a lightweight gating network in a preset missing information fusion gate, output the original gating weights of each modality, normalize them, apply sparsity constraints to the normalized weights, and obtain the redistributed weights. By using the redistributed weights, the feature representations of each modality are summed in a weighted manner to obtain cross-modal sentiment features; The prediction output module is used to obtain the final sentiment prediction result based on the cross-modal sentiment features.

10. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, the processor performs the steps of the sentiment prediction method based on missing modality robust learning as described in any one of claims 1 to 8.

Images