A training method and system of a multi-modal emotion classification model

Abstract

The present application belongs to the field of artificial intelligence and multi-modal emotion recognition technology, and discloses a training method and system of a multi-modal emotion classification model, comprising obtaining video data and extracting audio-video features; performing soft and hard mixed data enhancement on the audio-video features to obtain enhanced audio-video features, weighting and fusing the original audio-video features and the enhanced audio-video features in the batch dimension to generate multi-modal missing training data; performing audio-video feature extraction on the multi-modal missing training data, cross-modal feature fusion on the extracted features to obtain fused features, and emotion classification and recognition based on the fused features; defining a loss function, training the model, and obtaining a trained model. The present application generates diversified training samples through a soft and hard mixed data enhancement strategy, ensures the semantic consistency of multi-modal data through a multi-modal collaborative enhancement strategy, and effectively improves the robustness, generalization ability and emotion classification precision of the model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal emotion recognition technology, and in particular to a training method and system for a multimodal emotion classification model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Multimodal emotion recognition is a technology that integrates multiple information modalities such as speech, vision, and text to automatically detect and classify human emotional states. It is of great value in the clinical diagnosis and treatment of adolescent emotional and behavioral disorders (such as bipolar disorder) for condition assessment, intervention formulation, and efficacy tracking. However, adolescent patients experience frequent and complex emotional fluctuations, and single-modal methods are easily affected by factors such as speech rate, volume, background noise, facial expression suppression, facial occlusion, and insufficient lighting, leading to decreased recognition accuracy. Therefore, multimodal fusion has become an inevitable trend to improve accuracy and robustness. However, existing methods still have some prominent problems in practical applications. They lack adaptability to modal loss, and recognition performance drops significantly when a modality is missing or of poor quality in the diagnosis or monitoring environment. They also lack data distribution differences and generalization ability, as public datasets and clinically collected data differ in their presentation, emotion distribution, and language habits, and direct application of these data can limit generalization performance. Furthermore, they lack feature fusion depth, as existing methods often use shallow fusion methods such as simple splicing or weighted averaging, which cannot fully model the global dependencies and complementary information between audio and video modalities. Finally, they lack targeted data augmentation, as most methods only augment a single modality and lack strategies to maintain consistency and synergy among multiple modalities, making it difficult to simulate modal loss or partial attenuation in real-world scenarios. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a training method and system for a multimodal sentiment classification model. By employing a hybrid soft and hard data augmentation strategy, diverse training samples are generated. Through a multimodal collaborative augmentation strategy, the semantic consistency of multimodal data during the augmentation process is ensured, effectively improving the model's robustness, generalization ability, and sentiment classification accuracy.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a training method for a multimodal sentiment classification model, comprising the following steps:

[0007] Acquire video data for sentiment classification, preprocess the video data, and obtain video and audio features;

[0008] The obtained audio and video features are subjected to both soft and hard data augmentation processing to obtain augmented audio and video features. The original audio and video features and the augmented audio and video features are then weighted and fused in the batch dimension to generate multimodal missing training data.

[0009] Video and audio features are extracted from the multimodal missing training data, and the extracted features are fused across modalities to obtain fused features. Sentiment classification and recognition are then performed based on the fused features.

[0010] Define a loss function and train the multimodal sentiment classification model to obtain a well-trained multimodal sentiment classification model.

[0011] As an alternative implementation, the hybrid soft and hard data augmentation includes soft augmentation and hard augmentation. Soft augmentation employs a mild, weak augmentation strategy to make slight changes to the original data while keeping its main semantic information unchanged. Hard augmentation, on the other hand, employs an aggressive, strong augmentation strategy to significantly change the representation of the original data and enhance the robustness of the model.

[0012] As an alternative implementation, the soft enhancement generates soft enhancement coefficients for audio and video features through a reward function, and iteratively updates the soft enhancement coefficients through an update function.

[0013] As an alternative implementation, the reward function includes an accuracy reward and a stability penalty, wherein the accuracy reward reflects the classification accuracy of the model in the current batch, and the stability penalty is used to suppress drastic fluctuations in the soft augmentation coefficients between adjacent batches.

[0014] As an alternative implementation method, the reward function is:

[0015] ;

[0016] in, This represents the reward score for the current batch. This represents the recognition accuracy of the current batch. This represents the current audio modality missing enhancement coefficient. λ represents the enhancement coefficient for missing audio and video modalities in the previous batch, and λ is the coefficient for stability penalty.

[0017] As an alternative implementation, the update function is:

[0018] ;

[0019] Where α is the weight learning rate, This represents the initial value of the enhancement coefficient. This represents the reward score for the current batch. This represents the reward score from the previous batch.

[0020] Secondly, the present invention provides a training system for a multimodal emotion classification model, comprising:

[0021] The data acquisition and feature extraction module is configured to: acquire video data for sentiment classification, preprocess the video data, and obtain video features and audio features;

[0022] The training data generation module is configured to: perform soft and hard hybrid data augmentation processing on the obtained audio and video features respectively to obtain augmented audio and video features; and perform weighted fusion of the original audio and video features and the augmented audio and video features in the batch dimension to generate multimodal missing training data.

[0023] The feature fusion module is configured to: extract video features and audio features from the multimodal missing training data respectively, fuse the extracted features across modalities to obtain fused features, and perform emotion classification and recognition based on the fused features;

[0024] The model training module is configured to: define a loss function, train the multimodal sentiment classification model, and obtain a trained multimodal sentiment classification model.

[0025] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0026] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0027] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] This invention proposes a training method and system for a multimodal emotion classification model, effectively addressing the problems of missing multimodal information, modality imbalance, and difficulty in feature fusion in traditional emotion recognition models. By introducing a hybrid hardware and software data augmentation strategy, diverse training samples can be generated, effectively improving the model's robustness and generalization ability, especially demonstrating strong adaptability when dealing with common modality-missing scenarios in the real world. The system's computational parameters are only 2.14M, far lower than the computational parameters of other large models, possessing potential for emotion detection deployment. Furthermore, this invention ensures semantic consistency between audio and video modalities during data augmentation through a multimodal collaborative augmentation strategy, thereby enhancing the collaborative learning ability of multimodal data and further improving the model's emotion classification accuracy. In the feature extraction stage, a hybrid spatiotemporal convolutional network is used for audio and video feature extraction, which not only captures temporal features but also effectively models global dependencies through a NonLocal module, improving the model's understanding of spatiotemporal features. In the cross-modal fusion stage, the interactive attention mechanism introduced in this invention enhances the deep information interaction between audio and video modalities, accurately identifying and fusing consistent features across modalities while suppressing irrelevant or conflicting features, thereby further improving the accuracy and robustness of sentiment classification. Finally, the model uses a fully connected classifier to perform sentiment recognition on the fused multimodal features, employing a cross-entropy loss function to optimize the training process, resulting in excellent classification performance in multimodal sentiment recognition tasks.

[0030] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0031] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0032] Figure 1 A flowchart illustrating a training method for a multimodal sentiment classification model provided in Embodiment 1 of the present invention;

[0033] Figure 2 This is a framework diagram of a training method for a multimodal emotion classification model provided in Embodiment 1 of the present invention;

[0034] Figure 3 A diagram illustrating the effect of data augmentation based on a reward mechanism;

[0035] Figure 4 This is a schematic diagram of the structure of a hybrid spatiotemporal convolutional network;

[0036] Figure 5 This is a schematic diagram of the interactive attention mechanism. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed description is exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0041] Example 1

[0042] like Figure 1-2 As shown in the figure, this embodiment provides a training method for a multimodal sentiment classification model, including the following steps:

[0043] Step 1: Acquire video data for sentiment classification, preprocess the video data to obtain video features and audio features;

[0044] Step 2: Perform hybrid hardware and software data augmentation on the obtained audio and video features to obtain augmented audio and video features. Then, perform weighted fusion of the original audio and video features and the augmented audio and video features in the batch dimension to generate multimodal missing training data.

[0045] Step 3: Extract video and audio features from the multimodal missing training data, fuse the extracted features across modalities to obtain fused features, and perform sentiment classification and recognition based on the fused features;

[0046] Step 4: Define the loss function and train the multimodal sentiment classification model to obtain the trained multimodal sentiment classification model.

[0047] The present invention will now be described in detail.

[0048] Step 1: Obtain video data for sentiment classification, preprocess the video data, and obtain video and audio features.

[0049] Specifically, the acquired video data includes both speech and visual modalities. The speech and visual modalities are preprocessed separately as follows:

[0050] 1) Visual data preprocessing.

[0051] First, the video is sampled frame by frame. The original video has a frame rate of 30fps. A uniform sampling strategy is used, selecting 15 frames evenly from a 3.6-second video segment to ensure the integrity and representativeness of temporal information. Then, face detection and cropping are performed. A multi-task cascaded convolutional network (MTCNN) is used for face detection. MTCNN employs a cascaded structure containing three sub-networks, enabling simultaneous face detection, keypoint localization, and face alignment. For each video frame, MTCNN outputs the face bounding box coordinates (x1, y1, x2, y2), based on which the face region is cropped. Finally, the cropped face images are uniformly adjusted to 224×224 pixels. Figure 1 The data preprocessing section is shown below. Simultaneously, the image undergoes RGB channel conversion, changing the BGR format to RGB format to meet the input requirements of subsequent network models. The preprocessed video data is saved as a NumPy array format, i.e. The dimensions are (15, 224, 224, 3), which correspond to the number of frames, height, width, and number of RGB channels, respectively.

[0052] 2) Voice data preprocessing.

[0053] First, the length of the original audio files was standardized, uniformly truncated or padded to 3.6 seconds to maintain synchronization with the video data. For audio files shorter than 3.6 seconds, zero padding was applied at the end; for excessively long audio files, equal portions were trimmed from both ends to preserve the central part of the audio content and ensure that key information expressing emotion was not lost. Mel-frequency cepstral coefficients (MFCC) features were extracted using the librosa library, with a sampling rate of 22050Hz and 10 MFCC coefficients. MFCC features effectively characterize the spectral properties of audio. Through Mel filter banks and discrete cosine transform, the audio signal was converted into a compact feature representation. The preprocessed audio MFCC features were also saved as NumPy arrays. The dimension is (10, T), where T is the number of time frames.

[0054] Step 2: Perform hybrid hardware and software data augmentation on the obtained audio and video features to obtain augmented audio and video features. Then, perform weighted fusion of the original audio and video features and the augmented audio and video features in the batch dimension to generate multimodal missing training data.

[0055] Specifically, referring to Figure 3 , the steps of using the reward mechanism combined with soft and hard mixed data augmentation to form multi-modal missing training data include:

[0056] 1) Soft and hard mixed data augmentation strategy using the reward mechanism

[0057] In view of the problems such as insufficient generalization ability under incomplete modal information of samples and limitations of samples in real scenarios in the multi-modal sentiment recognition task, this application proposes a soft and hard mixed data augmentation strategy using the reward mechanism. This method generates diverse training samples through the combination of "soft augmentation" and "hard augmentation". Soft augmentation mainly adopts a relatively mild weak augmentation strategy, slightly transforming the original data while keeping its main semantic information unchanged. Hard augmentation adopts a more radical strong augmentation strategy, significantly changing the manifestation form of the original data to enhance the robustness of the model.

[0058] Specifically, the soft augmentation strategy of soft and hard mixed data augmentation takes the preprocessed audio-visual feature information and as the object of the soft and hard mixed data augmentation strategy. First, for each sample in each batch, the reward function is used to generate the soft augmentation coefficient C (0 < C < 1) of the speech modality and the soft augmentation coefficient 1 - C of the visual modality. In the reward function, an accuracy reward is set, which directly reflects the classification accuracy of the model in the current batch. The higher the accuracy, the more beneficial the current augmentation coefficient allocation is for model recognition, and the greater the reward value. A stability penalty is set to suppress the drastic fluctuation of the augmentation coefficient between adjacent batches. If the difference between the current augmentation coefficient and the augmentation coefficient of the previous batch is large, the penalty value increases, encouraging a smoother adjustment process of the augmentation coefficient. The influence of this item is controlled by the hyperparameter λ.

[0059] ; (1)

[0060] Among them, represents the current reward score, represents the recognition accuracy of the current batch, represents the missing augmentation coefficient of the current audio modality, represents the missing augmentation coefficient of the audio-visual modality of the previous batch, is used to suppress the drastic fluctuation of the augmentation coefficient, and λ is the coefficient of the stability penalty.

[0061] Then, the enhancement coefficient is updated through an update function. The update of the enhancement coefficient is based on the change trend of the reward value to prevent the model from overly relying on the complete modality. Specifically, if the current reward value is higher than that of the previous batch, it indicates that the current allocation of the enhancement coefficient is better, and the enhancement coefficient should continue to be adjusted in the current direction; otherwise, the adjustment should be slowed down or reversed. The update formula for the enhancement coefficient is as follows:

[0062] ;(2)

[0063] where α is the weight learning rate, ensuring that the enhancement coefficient is within the set minimum / maximum range. The update range of the enhancement coefficient , represents the initial value of the enhancement coefficient, represents the current reward score, represents the reward score of the previous batch.

[0064] Apply upper and lower bounds to update the enhancement coefficient C and maintain the complementarity of the enhancement coefficients of the speech and visual modalities through . During the training process, losses are generated through forward propagation and network parameters are updated backward; at the same time, the reward is calculated and the enhancement coefficient C is updated according to the above formula; after the update, the speech enhancement coefficient C (0 < C < 1) and the enhancement coefficient 1 - C of the visual modality take effect immediately in the next batch.

[0065] Expand the constants C and 1 - C to and in the dimensional space respectively, that is, the soft enhancement matrices of the audio-visual modality and are obtained. Then, the feature information of audio and vision is weighted respectively to achieve soft enhancement. The hard enhancement strategy of the hard and soft mixed data enhancement is to directly set the enhancement matrix of the speech modality or the visual modality to zero, that is, set the hard enhancement matrix of the speech modality to a constant value of 0, while the hard enhancement matrix of the video modality is 1; similarly, set the hard enhancement matrix of the visual modality to a constant value of 0, while the hard enhancement matrix of the speech modality is 1.

[0066] 2) Collaborative enhancement of multi-modal data

[0067] Traditional data augmentation methods often target a single modality, making it difficult to guarantee semantic consistency across multimodal data. To address this, this application proposes a multimodal collaborative augmentation strategy that maintains temporal and semantic consistency of audio and visual modal augmentation. Specifically, the original feature information and the audio / video feature information augmented using a combination of hard and soft methods are weighted and represented in a batch dimension. For the audio modality, each sample is first weighted and fused across different modalities to generate a weighted representation:

[0068] (3)

[0069] (4)

[0070] in, The original audio features, For audio enhancement matrix, These are audio weighting coefficients. The missing audio modal representation generated by hard enhancement.

[0071] For video modalities, a similar weighted fusion process is used:

[0072] (5)

[0073] (6)

[0074] in, Features of the original video For video enhancement matrix, For video weighting coefficients, This represents the missing modal representation of the video generated by hard enhancement.

[0075] After the above operations, the batch size is expanded from B to 4×B, thus adding three incomplete modal feature information pairs to the training data, namely, the audio and video features input after collaborative hardware-software hybrid data augmentation. and It includes incomplete feature information in four scenarios: "complete information," "partial information attenuation," "missing audio," and "missing video," such as... Figure 3 As shown.

[0076] To ensure the model can learn different modality missing scenarios in a balanced way, the system performs a global random shuffle operation after feature expansion. Specifically, by generating a random permutation index of length 4×B along the BatchSize dimension, the audio and video feature information and corresponding labels are rearranged synchronously. The calculation formula is as follows:

[0077] (7)

[0078] (8)

[0079] in, and It is a randomized index of audio and video. and These represent the shapes of the audio and video feature information, respectively. dim=0 represents the batch dimension, and formula 8 represents the random index. and Arrange the audio and video features.

[0080] Step 3: Extract video and audio features from the multimodal missing training data, fuse the extracted features across modalities to obtain fused features, and perform emotion classification and recognition based on the fused features.

[0081] Specifically, refer to Figure 4 The steps for feature extraction using the audio and video branches of multimodal data include:

[0082] The hybrid spatiotemporal convolutional network consists of 1D convolutional blocks for extracting temporal features and 2D convolutional blocks for extracting spatial features. The temporal convolutional network uses 1D convolutional layers combined with Batch Normalization, ReLU activation, and Maxpooling layers to learn temporal features from the input. The spatial convolutional network utilizes a nonlocal global feature interaction mechanism to achieve information transfer between arbitrary spatial locations, thereby effectively capturing global contextual information. This structure first extracts temporal series features of audio and video through 1D convolutional blocks, and then extracts spatial series features of the audio and video features through 2D convolutional blocks, thus extracting the spatiotemporal features of the feature input.

[0083] The video branch consists of two parts: the first part extracts facial features from a single video frame, and the second part extracts spatiotemporal features from the entire video sequence. To achieve an end-to-end trainable model, this application uses the EfficientFace architecture as a 2D feature extractor to extract features independently for each frame.

[0084] In the first part of the video branch, when a k-frame video sequence after hybrid hard and soft data augmentation is input, each frame is processed independently by a 2D feature extractor to obtain facial feature information for each frame. The features of each frame are then stacked temporally to form a temporal representation of facial feature information. The hybrid spatiotemporal convolutional network in the second part is then used to further extract spatiotemporal features.

[0085] In the second part of the video branch, it is assumed that certain feature representations have been extracted from the input video data using EfficientFace. To further enhance the model's ability to model global dependencies, the temporal feature representation is first modeled locally using 1D convolutional blocks with Batch Normalization and ReLU activation functions. Then, the temporal features are mapped to spatial features, calculated using the following formula:

[0086] (9)

[0087] The Reshape operation represents the transformation of time-series features. Mapped to Spatial characteristics. For example, when T=15, it is mapped to H=15, W=1, keeping the total dimensional space unchanged.

[0088] In this feature space, a lightweight NonLocal module is introduced for spatial feature extraction. First, the input feature information is processed through three 1×1 convolutions to generate Q, K, and V vectors. Then, the correlation between all locations is calculated using dot products and softmax. Next, the global features are weighted and aggregated using correlation weights to improve global dependency modeling capabilities. Finally, the NonLocal output is concatenated with the original feature residuals to enhance feature representation, thereby efficiently completing the spatiotemporal feature learning of the video branch. The calculation formula is as follows:

[0089] (10)

[0090] (11)

[0091] (12)

[0092] in, The dimension representing each batch is The input audio or video features, Representative input Middle position The output characteristics, , and Represents 1×1 convolution weights. and The audio and video features representing the input are located at... and The vector, This represents the similarity function calculated using the dot product. This represents the feature mapping function obtained through a linear transformation. N represents the number of positions, where N represents the total number of spatial positions. .

[0093] The audio branch adopts a similar structural design to the video branch, using four hybrid spatiotemporal convolutional networks to extract audio features. This design ensures both the depth of feature extraction and computational efficiency.

[0094] Reference Figure 5 The steps for multimodal fusion using an interactive attention mechanism to extract feature information include:

[0095] To achieve deep information interaction between audio and visual modalities, the interactive attention mechanism in this application differs from traditional self-attention mechanisms. It only uses Q and K to calculate the correlation weights between modalities, without directly weighting and summing V. Specifically, the attention weight matrix is ​​calculated using the features of one modality as Q and the features of the other modality as K, according to the following formula:

[0096] (13)

[0097] Among them, using and These represent the query and key for audio and video modalities, respectively. The dimension representing the key. It is a normalization method.

[0098] Therefore, the fusion of audio and video can use interactive attention gating mechanisms to control which parts of the information are more important at each moment:

[0099] (14)

[0100] (15)

[0101] in, The function is a gating mechanism for adaptive adjustment of audio and video information based on interactive attention. It determines the most important information features of audio and video information at a certain moment.

[0102] In the process of cross-modal feature fusion, such as Figure 5 As shown, taking the video branch as an example, the linear transformation information of the audio modal features is used as Q, and the video modal feature information is used as K. Then, through... and After scaling and normalization, Softmax facilitates competition within the attention matrix, highlighting more important features for each modality and yielding an importance score for each key relative to each query. For the video modality, the relative importance of each attribute is calculated by querying the importance scores of each attribute in the audio modality relative to the video modality; this is the relevance weight. The same principle applies to the audio branch; therefore, we obtain the relevance weights. and Then, dropout is used to prevent overfitting. Finally, the output correlation weight matrix is ​​used to multiply the original feature information element by element to achieve mid-term fusion of multimodal features.

[0103] In the cross-modal feature fusion process, the focus is on identifying which video modalities are more similar to audio modalities and which audio modalities are more similar to video modalities. Interactive attention mechanisms in multimodal fusion emphasize information similar to one modality from another while suppressing dissimilar features. Therefore, features exhibiting high consistency between the two modalities receive greater weight in the final prediction, guiding the model to automatically focus on modality-independent or highly consistent features across modalities. This achieves multimodal fusion through relevance weighting.

[0104] Step 4: Define the loss function and train the multimodal sentiment classification model to obtain the trained multimodal sentiment classification model.

[0105] Specifically, in this embodiment, training using the fused features to complete the sentiment classification task includes the following steps:

[0106] The sentiment analysis model maps fused multimodal features to specific sentiment categories to achieve final sentiment recognition. The model mainly consists of two parts: global pooling and feature fusion, and a linear classifier. After multimodal feature extraction and fusion, the model performs global pooling on the high-dimensional features of the audio and visual modalities respectively. The output features of the audio and visual branches are globally averaged in the temporal dimension, compressing the sequence features of audio and video into a fixed-length global feature vector. Subsequently, the pooled audio and visual features are concatenated in the feature dimension to form a joint multimodal feature representation. This fused feature fully integrates the global information of the audio and visual modalities, providing a basis for subsequent sentiment classification. After obtaining the fused feature, the model uses a fully connected linear classifier (FC) to distinguish sentiment categories, i.e.:

[0107] (16)

[0108] Where y represents the predicted sentiment classification.

[0109] Specifically, the fused features are input into a linear layer, which outputs a score for each sentiment category. During the training phase, the model is optimized using a cross-entropy loss function. Finally, the category scores output by the model are normalized using softmax to obtain the probability distribution of each sentiment category, and the category with the highest probability is selected as the final recognition result.

[0110] Simulation results and analysis:

[0111] As shown in Table 1, the simulation results were tested using publicly available facial expression datasets, speech intonation databases, and collected patient data. This method improved the emotion classification accuracy by an average of 4% in the full modality (AV) case. Furthermore, the classification accuracy was tested when only unimodal data was input to the model. It was found that the model trained with modality-missing data simulating real-world scenarios showed the highest accuracy improvement of 30% and the highest average accuracy (M) improvement of 20% in the audio / video unimodal (A / V) input case.

[0112] Our proposed training method and system model maintain high emotion recognition performance even in multimodal missing scenarios (including situations with only a single modality input or partial modality information attenuation). In comparative experiments, regardless of whether a single modality is missing or modality information is partially lost, the classification accuracy of our method is significantly better than the baseline model that does not employ our strategy. These results demonstrate that, addressing the insufficient adaptability to modal missing features in the prior art, this invention effectively generates training samples covering multiple modal missing scenarios through hybrid hardware and software data augmentation, significantly improving the robustness and adaptability of the model in real-world diagnostic and treatment environments. Regarding the issues of data distribution differences and insufficient generalization ability, the multimodal collaborative augmentation mechanism effectively enhances the complementarity between different modalities while maintaining temporal and semantic consistency, enabling the model to better generalize to clinically collected data. For the problem of insufficient feature fusion depth, the hybrid spatiotemporal convolutional network combined with an interactive attention mechanism not only achieves multi-level modeling of local and global features but also accurately captures cross-modal dependencies, significantly improving the discriminative power of fused features. Furthermore, addressing the existing problem of weak targeting in data augmentation, the collaborative augmentation strategy of this invention ensures consistency among multimodal data while simulating real-world modal missing and partial attenuation scenarios, thereby effectively improving the stability and accuracy of the model in variable environments. In summary, the method of this invention demonstrates significant advantages in solving the four major problems mentioned in the prior art, providing solid technical support for the practical application of multimodal emotion recognition technology in the diagnosis and treatment of adolescent emotional and behavioral disorders.

[0113] Table 1. Simulation results;

[0114]

[0115] Example 2

[0116] This embodiment provides a training system for a multimodal sentiment classification model, including:

[0117] The data acquisition and feature extraction module is configured to: acquire video data for sentiment classification, preprocess the video data, and obtain video features and audio features;

[0118] The training data generation module is configured to: perform soft and hard hybrid data augmentation processing on the obtained audio and video features respectively to obtain augmented audio and video features; and perform weighted fusion of the original audio and video features and the augmented audio and video features in the batch dimension to generate multimodal missing training data.

[0119] The feature fusion module is configured to: extract video features and audio features from the multimodal missing training data respectively, fuse the extracted features across modalities to obtain fused features, and perform emotion classification and recognition based on the fused features;

[0120] The model training module is configured to: define a loss function, train the multimodal sentiment classification model, and obtain a trained multimodal sentiment classification model.

[0121] It should be noted that the above modules correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules can be executed in a computer system as part of the system.

[0122] In further embodiments, the following is also provided:

[0123] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0124] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0125] A computer-readable storage medium for storing computer instructions that, when executed by a processor, perform the method of Embodiment 1.

[0126] The method in Example 1 can be directly executed by a hardware processor, or it can be executed by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0127] A computer program product includes a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0128] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0129] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0130] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0131] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A training method for a multimodal sentiment classification model, characterized in that, Includes the following steps: Acquire video data for sentiment classification, preprocess the video data, and obtain video and audio features; The obtained audio and video features are subjected to both soft and hard data augmentation processing to obtain augmented audio and video features. The original audio and video features and the augmented audio and video features are then weighted and fused in the batch dimension to generate multimodal missing training data. The hybrid soft and hard data augmentation includes soft augmentation and hard augmentation. Soft augmentation employs a mild, weak augmentation strategy to slightly transform the original data while preserving its main semantic information. Hard augmentation, on the other hand, employs an aggressive, strong augmentation strategy to significantly alter the representation of the original data and enhance the robustness of the model. The soft augmentation generates soft augmentation coefficients for audio and video features through a reward function and iteratively updates these coefficients using an update function. The reward function includes an accuracy reward and a stability penalty. The accuracy reward reflects the classification accuracy of the model in the current batch, while the stability penalty is used to suppress drastic fluctuations in the soft augmentation coefficients between adjacent batches. The reward function is: ； in, This represents the reward score for the current batch. This represents the recognition accuracy of the current batch. This represents the current audio modality missing enhancement coefficient. λ represents the enhancement coefficient for missing audio and video modalities in the previous batch, and λ is the coefficient for stability penalty. The update function is: ； Where α is the weight learning rate, This represents the initial value of the enhancement coefficient. This represents the reward score for the current batch. This represents the reward score from the previous batch. Ensure the enhancement factor remains within the set minimum / maximum range, and specify the update range for the enhancement factor. , This represents the current missing enhancement coefficient for the audio modality; the hard enhancement strategy of hybrid hard and soft data augmentation is to directly set the enhancement matrix of the speech modality or visual modality to zero. C new To update the enhancement coefficients; Video and audio features are extracted from the multimodal missing training data, and the extracted features are fused across modalities to obtain fused features. Sentiment classification and recognition are then performed based on the fused features. Define a loss function and train the multimodal sentiment classification model to obtain a well-trained multimodal sentiment classification model.

2. A training system for a multimodal sentiment classification model, characterized in that, include: The data acquisition and feature extraction module is configured to: acquire video data for sentiment classification, preprocess the video data, and obtain video features and audio features; The training data generation module is configured to: perform soft and hard hybrid data augmentation processing on the obtained audio and video features respectively to obtain augmented audio and video features; and perform weighted fusion of the original audio and video features and the augmented audio and video features in the batch dimension to generate multimodal missing training data. The hybrid soft and hard data augmentation includes soft augmentation and hard augmentation. Soft augmentation employs a mild, weak augmentation strategy to slightly transform the original data while preserving its main semantic information. Hard augmentation, on the other hand, employs an aggressive, strong augmentation strategy to significantly alter the representation of the original data and enhance the robustness of the model. The soft augmentation generates soft augmentation coefficients for audio and video features through a reward function and iteratively updates these coefficients using an update function. The reward function includes an accuracy reward and a stability penalty. The accuracy reward reflects the classification accuracy of the model in the current batch, while the stability penalty is used to suppress drastic fluctuations in the soft augmentation coefficients between adjacent batches. The reward function is: ； in, This represents the reward score for the current batch. This represents the recognition accuracy of the current batch. This represents the current audio modality missing enhancement coefficient. λ represents the enhancement coefficient for missing audio and video modalities in the previous batch, and λ is the coefficient for stability penalty. The update function is: ； Where α is the weight learning rate, This represents the initial value of the enhancement coefficient. This represents the reward score for the current batch. This represents the reward score from the previous batch. Ensure the enhancement factor remains within the set minimum / maximum range, and specify the update range for the enhancement factor. , This represents the current missing enhancement coefficient for the audio modality; the hard enhancement strategy of hybrid hard and soft data augmentation is to directly set the enhancement matrix of the speech modality or visual modality to zero. C new To update the enhancement coefficients; The feature fusion module is configured to: extract video features and audio features from the multimodal missing training data respectively, fuse the extracted features across modalities to obtain fused features, and perform emotion classification and recognition based on the fused features; The model training module is configured to: define a loss function, train the multimodal sentiment classification model, and obtain a trained multimodal sentiment classification model.

3. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method of claim 1.

4. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in claim 1.

5. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method of claim 1.

Images