A multi-modal sentiment analysis method and system based on dynamic weight distribution

Abstract

The application provides a multi-modal sentiment analysis method and system based on dynamic weight distribution, and belongs to the technical field of artificial intelligence and multi-modal sentiment recognition. The application comprises the following steps: performing single-modal feature extraction on a video segment and an audio segment through a deep network; introducing a routing network as a gate, taking the visual high-level feature vector and the auditory high-level feature vector obtained through single-modal feature extraction and splicing as input vectors, and generating modal weights; constructing a parallel task framework comprising two independent single-modal classification tasks and a main multi-modal task to perform multi-modal sentiment prediction; constructing a composite loss function, adjusting model parameters in the parallel task framework through a back propagation algorithm, and performing multi-modal sentiment recognition after the adjustment is completed. Through sample-level dynamic weight distribution, the application can effectively retain information specific to each mode in mode fusion, thereby effectively improving the multi-modal sentiment analysis precision.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal emotion recognition technology, and particularly relates to a multimodal emotion analysis method and system based on dynamic weight allocation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of human-computer interaction technology, multimodal emotion recognition has become a key research direction in the field of artificial intelligence. Existing models typically determine human emotional states by fusing information from multiple modalities, such as facial expressions and voice tone.

[0004] However, current mainstream multimodal fusion methods generally have some limitations: (1) The training method adopts a static fusion mechanism, that is, most models use a fixed fusion strategy, such as feature splicing, simple weighting or fixed attention, which cannot be adaptively adjusted according to the dynamic changes in the contribution of each modality information in different emotion samples. For example, when judging "forced smile", acoustic features may be more important than visual features, and the fixed fusion strategy cannot capture this dynamic characteristic at the sample level.

[0005] (2) There is a problem of loss of modality-specific information. During the feature fusion process, the model tends to focus too much on the common information between modalities, while ignoring or diluting the unique clues that exist only in a specific modality and are crucial for emotion discrimination, thus limiting the final performance of the model. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a multimodal sentiment analysis method and system based on dynamic weight allocation. Through sample-level dynamic weight allocation, it can effectively retain the unique information of each modality in modality fusion, thereby effectively improving the accuracy of multimodal sentiment analysis.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a multimodal sentiment analysis method based on dynamic weight allocation.

[0008] A multimodal sentiment analysis method based on dynamic weight allocation includes: The video and audio segments to be analyzed are acquired, and single-modal feature extraction is performed through a deep network to obtain visual high-level feature vectors and auditory high-level feature vectors. A routing network is introduced as a gating mechanism for the dynamic expert network. By using the concatenated visual high-level feature vector and auditory high-level feature vector as input vectors, a set of modality weights is generated to quantify the contribution of video modality and audio modality to the emotion judgment of the current sample. A parallel task framework is constructed, comprising two independent unimodal classification tasks and one subject-specific multimodal task: The visual high-level feature vector and the auditory high-level feature vector are respectively input into the corresponding unimodal classification head for unimodal prediction; the visual high-level feature vector and the auditory high-level feature vector are weighted and fused based on the obtained modality weights to generate preliminary fused features, which are defined as teacher features; the preliminary fused features are further processed based on a deep fusion network module to generate final fused features, which are defined as student features; and multimodal sentiment prediction is performed based on the final fused features. A composite loss function is constructed, and the model parameters within the parallel task framework are adjusted using the backpropagation algorithm. Multimodal emotion recognition is then performed after the model parameters are adjusted.

[0009] Furthermore, single-modal feature extraction is performed on the video segment, including: processing the video frames that make up the video segment using a pre-trained DINOv3 model to extract high-quality visual semantic features; and then encoding them through a temporal modeling network to obtain a high-level visual feature vector.

[0010] Furthermore, single-modal feature extraction is performed on the audio segment, including: extracting the MFCC features of the audio frames that make up the audio segment; and then encoding them through a temporal coding network to obtain an auditory high-level feature vector.

[0011] Furthermore, the routing network is a multilayer perceptron, which employs the Softmax activation function to ensure that the sum of the output modal weights is 1.

[0012] Furthermore, the data processing flow of the deep fusion network module includes: sequentially performing linear transformation, ReLU nonlinear activation, Dropout operation, and a second linear transformation on the input features; subsequently, adding the transformation result to the original input to form a residual connection.

[0013] Furthermore, the composite loss function is expressed as: ; in, This represents the total value of the composite loss function, i.e., the total loss. Indicates loss of the main task. Indicates single-mode loss. This represents the knowledge distillation loss. This represents the loss of entropy balance; , and These represent the hyperparameters used to balance the single-modal loss, knowledge distillation loss, and entropy balance loss, respectively.

[0014] Furthermore, a multimodal sentiment analysis method based on dynamic weight allocation also includes a hybrid data augmentation step, namely: dynamically augmenting the audio and video features in the original batch to generate augmented samples containing multiple visibility scenarios including complete information, partially attenuated information, and complete loss of a single modality, and merging the original samples and the augmented samples together into an augmented training batch for subsequent model training.

[0015] A second aspect of this invention provides a multimodal sentiment analysis system based on dynamic weight allocation.

[0016] A multimodal sentiment analysis system based on dynamic weight allocation includes: The feature extraction module is configured to: acquire video and audio segments to be analyzed, and perform single-modal feature extraction through a deep network to obtain visual high-level feature vectors and auditory high-level feature vectors respectively; The modality weight dynamic generation module is configured to: introduce a routing network as a gating mechanism for the dynamic expert network, and generate a set of modality weights to quantify the contribution of video modality and audio modality to the current sample emotion discrimination by using the concatenated visual high-level feature vector and auditory high-level feature vector as input vectors. The sentiment prediction module is configured to: construct a parallel task framework comprising two independent unimodal classification tasks and one subject-specific multimodal task; specifically, input the visual high-level feature vector and the auditory high-level feature vector into the corresponding unimodal classification head for unimodal prediction; perform weighted fusion of the visual high-level feature vector and the auditory high-level feature vector based on the obtained modality weights to generate preliminary fused features and define them as teacher features; perform secondary processing of the preliminary fused features based on a deep fusion network module to generate final fused features and define them as student features; and perform multimodal sentiment prediction based on the final fused features. The emotion recognition module is configured to: construct a composite loss function, adjust the model parameters within the parallel task framework through the backpropagation algorithm, and perform multimodal emotion recognition after the model parameters are adjusted.

[0017] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of a multimodal sentiment analysis method based on dynamic weight allocation as described in the first aspect of the present invention.

[0018] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a multimodal sentiment analysis method based on dynamic weight allocation as described in the first aspect of the present invention.

[0019] The above one or more technical solutions have the following beneficial effects: (1) This invention introduces a routing network as a gating mechanism. By using the concatenated visual high-level feature vector and auditory high-level feature vector as input vectors, a set of modality weights is generated to quantify the contribution of video and audio modalities to the emotion judgment of the current sample. This endows the model with the ability to dynamically evaluate the modality importance for each independent sample. The model no longer treats all samples equally, but can intelligently determine whether to focus more on visual or auditory cues based on the features of the current sample, thereby achieving true adaptive fusion and significantly improving the recognition accuracy in complex and ever-changing scenarios.

[0020] (2) This invention employs a knowledge distillation mechanism to provide a clear supervisory signal for the complex nonlinear fusion process. While learning deep fusion, the "student" network is forced to approximate the "ideal teacher" defined by dynamic weights in terms of feature structure. This constraint effectively prevents the key single-modal information from being over-diluted during the fusion process, preserving the upper limit of the model's discriminative ability. The multi-task learning framework also ensures that the single-modal encoder can learn high-quality feature representations. At the same time, entropy balance loss, as an effective regularization method, encourages the routing network to explore more diverse weight allocations, effectively avoiding the model from falling into "modal collapse" during training, i.e., the predicament of over-reliance on a single modality, thereby enhancing the model's robustness to changes in the quality of input data.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of a multimodal sentiment analysis method based on dynamic weight allocation in Embodiment 1 of the present invention.

[0024] Figure 2 This is a flowchart of single-modal feature extraction from a video segment in Embodiment 1 of the present invention.

[0025] Figure 3This is a flowchart of single-modal feature extraction of an audio segment in Embodiment 1 of the present invention.

[0026] Figure 4 This is a flowchart of generating modal weights in Embodiment 1 of the present invention.

[0027] Figure 5 This is a flowchart of generating teacher features in Embodiment 1 of the present invention.

[0028] Figure 6 This is a flowchart of generating student features in Embodiment 1 of the present invention.

[0029] Figure 7 This is a flowchart illustrating the implementation process of the parallel task framework in Embodiment 1 of the present invention. Detailed Implementation

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0032] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0033] Example 1 This embodiment discloses a multimodal sentiment analysis method based on dynamic weight allocation.

[0034] like Figure 1 As shown, a multimodal sentiment analysis method based on dynamic weight allocation includes: Step S1: Obtain the video and audio segments to be analyzed, and extract single-modal features through a deep network to obtain visual high-level feature vectors and auditory high-level feature vectors. Step S2: Introduce a routing network as the gating system for the dynamic expert network. By using the spliced ​​visual high-level feature vector and auditory high-level feature vector as input vectors, generate a set of modality weights to quantify the contribution of video modality and audio modality to the current sample emotion judgment. Step S3: Construct a parallel task framework comprising two independent unimodal classification tasks and one subject-specific multimodal task: Input the visual high-level feature vector and the auditory high-level feature vector into the corresponding unimodal classification head for unimodal prediction; perform weighted fusion of the visual high-level feature vector and the auditory high-level feature vector based on the obtained modality weights to generate preliminary fused features and define them as teacher features; perform secondary processing of the preliminary fused features based on the deep fusion network module to generate final fused features and define them as student features; perform multimodal sentiment prediction based on the final fused features. Step S4: Construct a composite loss function, adjust the model parameters within the parallel task framework using the backpropagation algorithm, and perform multimodal emotion recognition after the model parameters are adjusted.

[0035] As described above, this invention proposes an innovative training paradigm. This method first extracts high-level semantic features of each modality through a deep network. The core of this method lies in the fact that instead of direct fusion, a routing network is introduced as the gating mechanism for a "dynamic expert network." This network takes the extracted high-level features as input and dynamically generates a set of modality importance weights for the current sample. Next, within a unified framework of multi-task learning and knowledge distillation, the model optimizes three objectives in parallel: two independent single-modal classification tasks and a main multi-modal classification task. The multi-modal fusion process is cleverly constructed as a "student-teacher" learning model: features directly weighted by dynamic weights are considered the "teacher," while the fused features, processed by a dedicated deep network, are considered the "student." Finally, the entire model is optimized end-to-end using a composite loss function that includes the main task loss, single-modal loss, knowledge distillation loss, and entropy balance loss. Based on this process, this invention, through sample-level dynamic weight allocation, can effectively preserve the unique information of each modality in modality fusion, thereby effectively improving the accuracy of multimodal sentiment analysis. To facilitate understanding of the technical solution of the present invention, the specific implementation methods of the technical solution of the present invention will be further explained and described below.

[0036] In step S1, the video and audio segments to be analyzed are acquired, and single-modal feature extraction is performed through a deep network to obtain visual high-level feature vectors and auditory high-level feature vectors.

[0037] 1) Extract single-modal features from video clips.

[0038] like Figure 2 As shown, a pre-trained DINOv3 model is used to process the video frames that make up the video segment in order to extract high-quality visual semantic features; then, the features are encoded through a temporal modeling network to obtain a high-level visual feature vector.

[0039] In the specific implementation process, firstly, a large-scale visual pre-trained model, DINOv3-ViT-S / 16 (whose parameters are frozen during training), is used to extract high-quality frame-level features. Considering the high dimensionality of the pre-trained model's output (e.g., 384 dimensions), it is adapted (e.g., reduced to 128 dimensions) to the dimensions required by subsequent networks through a linear projection layer. Finally, the dimensionality-reduced feature sequence is fed into a temporal encoder consisting of one-dimensional convolutional layers and a non-local attention module to capture long-range temporal dependencies. The network output sequence is then subjected to global average pooling to obtain the final high-level visual feature vector, video_pooled.

[0040] 2) Extract single-modal features from audio segments.

[0041] like Figure 3 As shown, firstly, Mel-frequency cepstral coefficients (MFCC) features are extracted using the librosa library, with a sampling rate of 22050Hz and 10 MFCC coefficients. MFCC features can effectively characterize the spectral characteristics of audio. Through Mel filter banks and discrete cosine transform, the audio signal is converted into a compact feature representation. The preprocessed audio MFCC features are also saved as NumPy arrays. Then, they are fed into a similarly structured temporal encoder composed of one-dimensional convolutional and non-local attention modules. Encoding is performed through an internal temporal coding network to obtain the auditory high-level feature vector `audio_pooled`. The temporal encoder consists of 1D convolutional blocks for extracting temporal features and 2D convolutional blocks for extracting spatial features. The temporal feature extraction network uses 1D convolutional layers combined with Batch Normalization, ReLU activation function, and Maxpooling layers to learn temporal features from the input. The spatial feature extraction network uses a nonlocal global feature interaction mechanism to achieve information transfer between arbitrary spatial locations, thereby effectively capturing global contextual information. The structure first extracts the temporal sequence features of audio and video through 1D convolutional blocks, and then extracts the spatial sequence features of audio and video through 2D convolutional blocks, thus extracting the input features.

[0042] In step S2, a routing network is introduced as the gating mechanism for the dynamic expert network. Using the concatenated visual and auditory high-level feature vectors as input vectors, a set of modality weights is generated to quantify the contribution of the video and audio modalities to the emotion judgment of the current sample. The dynamic expert network is the routing network composed of the multilayer perceptron. An expert model is introduced to learn the importance (i.e., weights) of each modality (i.e., speech and video). and In traditional multimodal learning, different modalities are assigned equal weights by default, and multimodal analysis is achieved by learning from a large amount of data using a model. This invention achieves dynamic weight updates at the sample level through routing network learning.

[0043] like Figure 4 As shown, the auditory high-level feature vector and the visual high-level feature vector are concatenated, and the concatenated result is input into the routing network, which generates a set of dynamic modal weights for the current sample. The routing network is a multilayer perceptron, whose input is the concatenation of the auditory feature vector and the visual feature vector, and whose output layer uses the Softmax activation function to ensure that the sum of the output dynamic modal weights is 1.

[0044] In the specific implementation process, the audio_pooled and video_pooled samples obtained in step S1 are concatenated along the feature dimension. The resulting high-dimensional vector has a dimension of 128 + 128 = 256, and this feature vector contains the condensed semantic information of the current sample after deep encoding. This vector is then input into a lightweight routing network (Router), which consists of two fully connected layers with an output layer dimension of 2, corresponding to the audio and video modalities. The softmax function is used for activation to generate a set of dynamic modal weights that sum to 1. In traditional multimodal learning, different modalities are assigned equal weights by default, and multimodal analysis is achieved by learning from a large amount of data. This invention achieves sample-level dynamic weight updates through routing network learning. The multilayer perceptron learns different modalities, and when a new sample is input, it assigns it what it considers the optimal weight. and This weight represents the importance of different modalities within this batch of samples. Subsequent knowledge distillation and multi-task learning feature fusion is based on this weight, where... surface, These represent the weight vectors respectively. This corresponds to the weight scalars for the speech and visual modalities. This refers to the model's dynamic assessment of the importance of the audio and video modalities of the current sample.

[0045] In step S3, a parallel task framework is constructed, comprising two independent unimodal classification tasks and one subject-specific multimodal task: the visual high-level feature vector and the auditory high-level feature vector are respectively input into independent unimodal classification heads to obtain unimodal prediction results; using the dynamic modal weights generated in step S2, the auditory high-level feature vector and the visual high-level feature vector are weighted and fused to obtain preliminary fused features, which are defined as "teacher features"; the preliminary fused features are input into a deep fusion network module composed of linear layers, nonlinear activation functions, and residual connections for secondary processing to obtain the final fused features, which are defined as "student features", and this feature is input into the multimodal classification head to obtain multimodal prediction results.

[0046] In practical implementation, the model's learning process is constructed as a parallel framework containing a "student-teacher" metaphor, including: First, parallel unimodal prediction is performed: independent classification heads, i.e., nn.Linear layers, are provided for audio_pooled and video_pooled respectively, directly outputting the unimodal sentiment prediction results logits_a and logits_v. These are two auxiliary tasks in multi-task learning.

[0047] Next, the characteristics of the "teacher" are constructed: such as Figure 5 As shown, the dynamic mode weights obtained in step S2 are used. The audio_pooled and video_pooled samples are linearly weighted and summed to obtain the "teacher" feature, teacher_proj. This feature represents an ideal and pure fusion direction determined by the importance of the current samples. The calculation formula is as follows: ; in, and These represent the high-level feature vectors of speech and vision after processing by a single-modal expert encoder, respectively.

[0048] Then, the "student" characteristics are generated: such as Figure 6 As shown, the result of the weighted summation, i.e., the preliminary fusion features, is fed into a dedicated deep fusion network module ("Student Network"). This module has a standard feedforward network block structure, and its data processing flow is as follows: the input features are sequentially subjected to linear transformation, ReLU nonlinear activation, Dropout operation, and a second linear transformation. Finally, the transformation result is added to the original input of this module to form a residual connection. This module performs complex nonlinear transformations and deep processing on the preliminary fusion features, ultimately outputting the "student" features. .

[0049] Finally, multimodal prediction is performed: "student" features are... The data is fed into a multimodal classification head to obtain the final sentiment prediction result. This is the main task.

[0050] In step S4, a composite loss function is constructed, the model parameters within the parallel task framework are adjusted using the backpropagation algorithm, and multimodal emotion recognition is performed after the model parameters are adjusted.

[0051] like Figure 7 As shown, to achieve the technical effects of this invention, a composite loss function consisting of four parts is designed, and the model is optimized end-to-end through this function. Each term of this loss function specifically addresses a core problem in the background technology and directly contributes to the beneficial effects of this invention. Its structure is as follows: ; in, This represents the total value of the composite loss function, i.e., the total loss. Indicates loss of the main task. Indicates single-mode loss. This represents the loss from knowledge distillation. This represents the loss of entropy balance; , and These represent the hyperparameters used to balance the single-modal loss, knowledge distillation loss, and entropy balance loss, respectively.

[0052] 1) Main task loss .

[0053] This loss term is the core driving force of model training, aiming to minimize the difference between the final multimodal sentiment prediction and the true label. It directly affects the final prediction output by the "student network". In this embodiment, the invention uses the standard cross-entropy loss function for calculation, as shown in the following formula: ; in, For the prediction output of the multimodal branch, For the corresponding real emotional tags; This represents the calculation of the cross-entropy loss function. It involves optimizing the loss of the main task. How does the model learn to incorporate the deeply integrated "student characteristics"? Map to the correct sentiment category.

[0054] 2) Single-mode loss .

[0055] This loss term is the core of the multi-task learning framework, and its role is to ensure that the single-modal encoder itself can learn discriminative features. If the single-modal branch itself cannot effectively distinguish sentiment, then any subsequent fusion will be futile. Single-modal loss. This is achieved by supervising the prediction performance of the two single-modal branches in parallel, as shown in the following equation: ; in, and These are the independent predicted outputs for the auditory and visual branches, respectively. This is achieved by minimizing the unimodal loss. This invention forces auditory encoders and visual encoders to extract core emotion-related information from their respective modalities. This not only provides high-quality feature "raw materials" for subsequent fusion and knowledge distillation, but also makes the definition of the "teacher" model reliable and meaningful.

[0056] 3) Knowledge distillation loss .

[0057] This loss term is a key innovation of this invention, preserving modality-specific information. It constructs the complex fusion process as a "student-teacher" learning paradigm, providing a clear, dynamically weighted "ideal direction" for the nonlinear transformation of the "student network." This loss aims to minimize "student characteristics." With "Teacher Characteristics" The structural differences between them. In this embodiment, the present invention uses mean squared error (MSE) to measure the L2 distance in this feature space, as shown in the following formula: ; in, These are "student characteristics" that have undergone secondary processing by a deep integration module. The "teacher characteristics" are obtained by directly weighting and summing the dynamic weights. The operation indicates that "teacher characteristics" serve only as a supervisory signal and do not participate in gradient calculation. This is achieved by optimizing the knowledge distillation loss. This invention guides the "student network" to learn complex modal interactions while its output features structurally approximate the "teacher" template dominated by the more important modality in the current sample. This imitation learning mechanism effectively prevents key, unique single-modal information from being over-smoothed or diluted during the fusion process, thereby effectively preserving the modality's unique information and improving the model's ability to recognize subtle and complex emotions.

[0058] 4) Entropy balance loss .

[0059] This loss term is a key regularization mechanism to ensure the stability and effectiveness of the dynamic weighting mechanism. Its role is to penalize "modal collapse," where the model over-relies on a single, fixed modality during training. This invention achieves balance by maximizing the Shannon entropy of the weights w output by the routing network. In optimization, maximizing entropy is equivalent to minimizing negative entropy. Therefore, entropy balance loss... It is defined as follows: ; in, It is dynamic modal weights The first in The weights of each modality This is the total number of modes (2 in this example). The formula calculates a value proportional to the negative Shannon entropy. When the weight distribution tends to extremes (e.g., [0.99, 0.01], where entropy is very low), the entropy balance loss... The absolute value will be very large (a large negative number); when the weight distribution tends to be balanced (e.g., [0.5, 0.5], the entropy is very high), the entropy balance loss... The absolute value will be very small (close to 0). This will balance the entropy loss. Minimizing the total loss function is equivalent to driving the weight distribution towards higher entropy and greater balance, thereby encouraging the routing network to actively perform true dynamic evaluation for each sample, enhancing the robustness and generalization ability of the entire model.

[0060] In each training iteration, after calculating the total value of the aforementioned composite loss function, the gradient of this total loss with respect to all trainable parameters of the model is calculated using the backpropagation algorithm, and the parameters are updated using the SGD optimizer. This process unifies all technical objectives within a single framework for collaborative optimization, ultimately training an advanced model capable of intelligent, dynamic, and robust multimodal emotion recognition.

[0061] Furthermore, the multimodal sentiment analysis method based on dynamic weight allocation provided by the present invention also includes a hybrid data augmentation step, namely: dynamically amplifying the audio features and video features in the original batch to generate enhanced samples containing multiple visibility scenes including complete information, partially attenuated information and complete loss of single modality, and merging the original samples and the enhanced samples together into an amplified training batch for subsequent model training.

[0062] To illustrate the significant advancements of this invention, the following experimental comparisons were conducted in this embodiment: Experimental Environment Setup: The experiment used the PyTorch framework for deep learning to build and run the model. An NVIDIA GeForce RTX 3090 GPU was used for model training, with CUDA Version 12.4. The model optimizer was SGD, and the cross-entropy loss function was used. The learning rate was set to 4e-2, and the batch size was 16. The experimental results are shown in Table 1. Table 1. Comparison of classification performance between the method of this invention and existing state-of-the-art (SOTA) methods.

[0063] As shown in Table 1, experiments were conducted on the public datasets CH-SIMS and CH-SIMSv2 in this embodiment. The dynamic weight allocation method of this invention improved the accuracy by 1.2% compared to the existing best-in-class (SOTA) model. It is evident that this invention provides an innovative training method for a multimodal sentiment classification model, achieving dynamic adaptive fusion at the sample level. This effectively preserves modality-specific discriminative information, thereby enhancing the model's robustness and generalization ability.

[0064] Example 2 This embodiment discloses a multimodal sentiment analysis system based on dynamic weight allocation.

[0065] A multimodal sentiment analysis system based on dynamic weight allocation includes: The feature extraction module is configured to: acquire video and audio segments to be analyzed, and perform single-modal feature extraction through a deep network to obtain visual high-level feature vectors and auditory high-level feature vectors respectively; The modality weight dynamic generation module is configured to: introduce a routing network as a gating mechanism for the dynamic expert network, and generate a set of modality weights to quantify the contribution of video modality and audio modality to the current sample emotion discrimination by using the concatenated visual high-level feature vector and auditory high-level feature vector as input vectors. The sentiment prediction module is configured to: construct a parallel task framework comprising two independent unimodal classification tasks and one subject-specific multimodal task; specifically, input the visual high-level feature vector and the auditory high-level feature vector into the corresponding unimodal classification head for unimodal prediction; perform weighted fusion of the visual high-level feature vector and the auditory high-level feature vector based on the obtained modality weights to generate preliminary fused features and define them as teacher features; perform secondary processing of the preliminary fused features based on a deep fusion network module to generate final fused features and define them as student features; and perform multimodal sentiment prediction based on the final fused features. The emotion recognition module is configured to: construct a composite loss function, adjust the model parameters within the parallel task framework through the backpropagation algorithm, and perform multimodal emotion recognition after the model parameters are adjusted.

[0066] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0067] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of a multimodal sentiment analysis method based on dynamic weight allocation as described in Embodiment 1 of this disclosure.

[0068] Example 4 The purpose of this embodiment is to provide an electronic device.

[0069] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a multimodal sentiment analysis method based on dynamic weight allocation as described in Embodiment 1 of this disclosure.

[0070] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0071] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0072] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multimodal sentiment analysis method based on dynamic weight allocation, characterized in that, include: The video and audio segments to be analyzed are acquired, and single-modal feature extraction is performed through a deep network to obtain visual high-level feature vectors and auditory high-level feature vectors. A routing network is introduced as a gating mechanism for the dynamic expert network. By using the concatenated visual high-level feature vector and auditory high-level feature vector as input vectors, a set of modality weights is generated to quantify the contribution of video modality and audio modality to the emotion judgment of the current sample. A parallel task framework is constructed, comprising two independent unimodal classification tasks and one subject multimodal task: the visual high-level feature vector and the auditory high-level feature vector are respectively input into the corresponding unimodal classification head for unimodal prediction; the visual high-level feature vector and the auditory high-level feature vector are weighted and fused based on the obtained modality weights to generate preliminary fused features and define them as teacher features; The preliminary fusion features are further processed based on the deep fusion network module to generate the final fusion features and define them as student features; Multimodal sentiment prediction is performed based on the final fused features; A composite loss function is constructed, and the model parameters within the parallel task framework are adjusted using the backpropagation algorithm. Multimodal emotion recognition is then performed after the model parameters are adjusted.

2. The multimodal sentiment analysis method based on dynamic weight allocation as described in claim 1, characterized in that, The process of extracting single-modal features from video segments includes: processing the video frames that make up the video segments using a pre-trained DINOv3 model to extract high-quality visual semantic features; and then encoding them through a temporal modeling network to obtain high-level visual feature vectors.

3. The multimodal sentiment analysis method based on dynamic weight allocation as described in claim 1, characterized in that, The single-modal feature extraction of the audio segment includes: extracting the MFCC features of the audio frames that make up the audio segment; and then encoding them through a temporal coding network to obtain an auditory high-level feature vector.

4. The multimodal sentiment analysis method based on dynamic weight allocation as described in claim 1, characterized in that, The routing network is a multilayer perceptron, which uses the Softmax activation function to ensure that the sum of the output modal weights is 1.

5. The multimodal sentiment analysis method based on dynamic weight allocation as described in claim 1, characterized in that, The data processing flow of the deep fusion network module includes: sequentially performing linear transformation, ReLU nonlinear activation, Dropout operation, and a second linear transformation on the input features; subsequently, adding the transformation result to the original input to form a residual connection.

6. The multimodal sentiment analysis method based on dynamic weight allocation as described in claim 1, characterized in that, The composite loss function is expressed as: ； in, This represents the total value of the composite loss function, i.e., the total loss. Indicates loss of the main task. Indicates single-mode loss. This represents the loss from knowledge distillation. This represents the loss of entropy balance; , and These represent the hyperparameters used to balance the single-modal loss, knowledge distillation loss, and entropy balance loss, respectively.

7. The multimodal sentiment analysis method based on dynamic weight allocation as described in claim 1, characterized in that, It also includes a hybrid data augmentation step, namely: dynamically augmenting the audio and video features in the original batch to generate augmented samples containing various visibility scenes including complete information, partially attenuated information, and complete loss of a single modality, and merging the original samples and the augmented samples together into an augmented training batch for subsequent model training.

8. A multimodal sentiment analysis system based on dynamic weight allocation, characterized in that, include: The feature extraction module is configured to: acquire video and audio segments to be analyzed, and perform single-modal feature extraction through a deep network to obtain visual high-level feature vectors and auditory high-level feature vectors respectively; The modality weight dynamic generation module is configured to: introduce a routing network as a gating mechanism for the dynamic expert network, and generate a set of modality weights to quantify the contribution of video modality and audio modality to the current sample emotion discrimination by using the concatenated visual high-level feature vector and auditory high-level feature vector as input vectors. The sentiment prediction module is configured to: construct a parallel task framework comprising two independent unimodal classification tasks and one subject multimodal task; specifically, input the visual high-level feature vector and the auditory high-level feature vector into the corresponding unimodal classification head for unimodal prediction; and perform weighted fusion of the visual high-level feature vector and the auditory high-level feature vector based on the obtained modality weights to generate preliminary fused features and define them as teacher features. The preliminary fusion features are further processed based on the deep fusion network module to generate the final fusion features and define them as student features; Multimodal sentiment prediction is performed based on the final fused features; The emotion recognition module is configured to: construct a composite loss function, adjust the model parameters within the parallel task framework through the backpropagation algorithm, and perform multimodal emotion recognition after the model parameters are adjusted.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps of a multimodal sentiment analysis method based on dynamic weight allocation as described in any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal sentiment analysis method based on dynamic weight allocation as described in any one of claims 1-7.

Images