Multi-modal sentiment classification model training method and system

Abstract

The invention belongs to the technical field of artificial intelligence and multi-modal emotion recognition, and discloses a multi-modal emotion classification model training method and system, and the method comprises the steps: obtaining video data, and extracting audio and video features; performing soft and hard mixed data enhancement on the audio and video features to obtain enhanced audio and video features, and performing weighted fusion on the original audio and video features and the enhanced audio and video features in the batch dimension to generate multi-modal missing training data; performing audio and video feature extraction on the multi-modal missing training data, performing cross-modal feature fusion on the extracted features to obtain fused features, and performing sentiment classification recognition based on the fused features; and defining a loss function, and training the model to obtain a trained model. According to the method, diversified training samples are generated through a soft and hard mixed data enhancement strategy, the semantic consistency of multi-modal data is ensured through a multi-modal collaborative enhancement strategy, and the robustness, generalization ability and sentiment classification precision of the model are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and multimodal emotion recognition, and in particular to a training method and system for a multimodal emotion classification model. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Multimodal emotion recognition is a technology that automatically detects and classifies human emotional states by integrating multiple information modalities, including speech, vision, and text. It plays a crucial role in the clinical diagnosis and treatment of adolescent emotional and behavioral disorders (such as bipolar disorder), including assessment of symptoms, development of interventions, and tracking of treatment outcomes. However, adolescents experience frequent and complex emotional fluctuations, and single-modality approaches are susceptible to interference from factors such as speech rate, volume, background noise, facial suppression, facial occlusion, and insufficient lighting, leading to reduced recognition accuracy. Therefore, multimodal fusion has become an inevitable trend to improve accuracy and robustness. However, existing methods still have some outstanding problems in practical applications. The adaptability to modal loss is insufficient. When a certain modal information is missing or the quality is impaired in the diagnosis, treatment or monitoring environment, the recognition performance drops significantly. There are differences in data distribution and insufficient generalization ability. There are differences in expression, emotion distribution and language habits between public data sets and clinically collected data. Direct application will lead to limited generalization performance. The feature fusion depth is insufficient. Existing methods mostly use shallow fusion such as simple splicing or weighted averaging, which cannot fully model the global dependency and complementary information between audio and video modalities. Data augmentation is not targeted. Most methods only enhance a single modality and lack strategies to maintain multimodal consistency and coordination, making it difficult to simulate modal loss or partial attenuation in real scenarios. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a training method and system for a multimodal sentiment classification model. Through a soft and hard hybrid data enhancement strategy, diversified training samples are generated. Through a multimodal collaborative enhancement strategy, the semantic consistency of multimodal data during the enhancement process is ensured, effectively improving the robustness, generalization ability and sentiment classification accuracy of the model.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for training a multimodal sentiment classification model, comprising the following steps: Obtain video data for emotion classification, preprocess the video data, and obtain video features and audio features; Perform soft and hard mixed data enhancement processing on the obtained audio and video features to obtain enhanced audio and video features, and perform weighted fusion of the original audio and video features and the enhanced audio and video features in the batch dimension to generate multimodal missing training data; Extract video and audio features from multimodal missing training data, perform cross-modal feature fusion on the extracted features, obtain fused features, and perform sentiment classification and recognition based on the fused features; Define the loss function, train the multimodal sentiment classification model, and obtain the trained multimodal sentiment classification model.

[0006] As an optional implementation, the soft-hard hybrid data enhancement includes soft enhancement and hard enhancement. Soft enhancement adopts a mild weak enhancement strategy to slightly transform the original data, keeping its main semantic information unchanged, while hard enhancement adopts a radical strong enhancement strategy to significantly change the representation of the original data and enhance the robustness of the model.

[0007] As an optional implementation, the soft enhancement generates soft enhancement coefficients of audio and video features through a reward function, and iteratively updates the soft enhancement coefficients through an update function.

[0008] As an optional implementation, the reward function includes an accuracy reward and a stability penalty, wherein the accuracy reward reflects the classification accuracy of the current batch of the model, and the stability penalty is used to suppress the sharp fluctuation of the soft enhancement coefficient between adjacent batches.

[0009] As an optional implementation, the reward function is: ; in, Represents the reward score of the current batch, Represents the recognition accuracy of the current batch, Represents the current audio mode missing enhancement coefficient, represents the audio and video modality missing enhancement coefficient of the previous batch, and λ is the coefficient of stability penalty.

[0010] As an optional implementation, the update function is: ; Among them, α is the weight learning rate, represents the initial value of the enhancement coefficient, Represents the reward score of the current batch, Represents the reward score of the previous batch.

[0011] In a second aspect, the present invention provides a training system for a multimodal sentiment classification model, comprising: The data acquisition and feature extraction module is configured to: acquire video data for emotion classification, preprocess the video data, and obtain video features and audio features; The training data generation module is configured to: perform soft and hard mixed data enhancement processing on the obtained audio and video features to obtain enhanced audio and video features, and perform weighted fusion of the original audio and video features and the enhanced audio and video features in the batch dimension to generate multimodal missing training data; The feature fusion module is configured to extract video features and audio features from multimodal missing training data, perform cross-modal feature fusion on the extracted features to obtain fused features, and perform sentiment classification and recognition based on the fused features; The model training module is configured to: define a loss function, train the multimodal sentiment classification model, and obtain a trained multimodal sentiment classification model.

[0012] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0013] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0014] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the method described in the first aspect when executed by a processor.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a training method and system for a multimodal emotion classification model, which can effectively solve the problems of multimodal information loss, modal imbalance and feature fusion difficulties existing in traditional emotion recognition models. By introducing a soft and hard hybrid data enhancement strategy, a variety of training samples can be generated, effectively improving the robustness and generalization ability of the model, especially when dealing with common modal missing scenarios in the real world, it shows strong adaptability; the computing parameters of this system are only 2.14M, which is much lower than the computing parameters of other large models, and has the potential for emotion detection deployment. In addition, the present invention ensures the semantic consistency of audio and video modalities in the data enhancement process through a multimodal collaborative enhancement strategy, thereby improving the collaborative learning ability of multimodal data and further enhancing the emotion classification accuracy of the model. In the feature extraction stage, a hybrid spatiotemporal convolutional network is used for audio and video feature extraction, which can not only capture temporal features, but also effectively model global dependencies through the NonLocal module, thereby improving the model's understanding of spatiotemporal features. During the cross-modal fusion phase, the interactive attention mechanism introduced in this paper enhances deep information interaction between audio and video modalities, accurately identifying and fusing consistent features across modalities and suppressing irrelevant or conflicting features, thereby further improving the accuracy and robustness of sentiment classification. Finally, the model uses a fully connected classifier to perform sentiment recognition on the fused multimodal features. Using a cross-entropy loss function to optimize the training process, the model achieves excellent classification performance in multimodal sentiment recognition tasks.

[0016] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0018] Figure 1 A flowchart of a method for training a multimodal sentiment classification model provided in Example 1 of the present invention; Figure 2 A framework diagram of a training method for a multimodal sentiment classification model provided in Example 1 of the present invention; Figure 3 This is an image of the effect of data enhancement based on the reward mechanism; Figure 4 Schematic diagram of the structure of the hybrid spatiotemporal convolutional network; Figure 5 Schematic diagram of the interactive attention mechanism. DETAILED DESCRIPTION

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0021] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but includes other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0023] Example 1 like Figure 1-2 As shown, this embodiment provides a training method for a multimodal sentiment classification model, comprising the following steps: Step 1: Obtain video data for emotion classification, preprocess the video data, and obtain video features and audio features; Step 2: Perform soft and hard mixed data enhancement processing on the obtained audio and video features to obtain enhanced audio and video features, and perform weighted fusion of the original audio and video features and the enhanced audio and video features in the batch dimension to generate multimodal missing training data; Step 3: Extract video features and audio features from the multimodal missing training data, perform cross-modal feature fusion on the extracted features, obtain fused features, and perform sentiment classification and recognition based on the fused features; Step 4: Define the loss function, train the multimodal sentiment classification model, and obtain the trained multimodal sentiment classification model.

[0024] The solution of the present invention is described in detail below.

[0025] Step 1: Obtain video data for emotion classification, preprocess the video data, and obtain video features and audio features.

[0026] Specifically, the acquired video data includes speech and visual modalities, and the data preprocessing for speech and visual modalities is performed as follows: 1) Visual data preprocessing.

[0027] First, the video frames are sampled. The frame rate of the original video is 30fps. A uniformly distributed sampling strategy is used to evenly select 15 frames from the 3.6-second video clip to ensure the integrity and representativeness of the temporal information. Then, face detection and cropping are performed. The multi-task cascade convolutional network MTCNN is used for face detection. MTCNN adopts a cascade structure and contains three sub-networks. It can simultaneously complete face detection, key point location and face alignment tasks. For each video frame, MTCNN outputs the face bounding box coordinates (x1, y1, x2, y2), and the face area is cropped based on this. Finally, the cropped face image is uniformly resized to 224×224 pixels, as shown in the following example. Figure 1 As shown in the data preprocessing section. At the same time, the image is converted to RGB channel, and the BGR format is converted to RGB format to adapt to the input requirements of the subsequent network model. The preprocessed video data is saved in numpy array format, that is The dimensions are (15, 224, 224, 3), corresponding to the number of frames, height, width, and RGB channels, respectively.

[0028] 2) Voice data preprocessing.

[0029] First, unify the length of the original audio files and uniformly cut or fill them to 3.6 seconds to keep them synchronized with the video data. For audio files less than 3.6 seconds in length, zero padding is performed at the end; for very long audio files, they are equally cropped from both ends to keep the central part of the audio content and ensure that the key information of emotional expression is not lost. The librosa library is used to extract Mel-frequency cepstral coefficient (MFCC) features with a sampling rate of 22050Hz and a number of MFCC coefficients of 10. MFCC features can effectively characterize the spectral characteristics of audio, and convert audio signals into compact feature representations through Mel filter banks and discrete cosine transforms. The preprocessed audio MFCC features are also saved in numpy array format, i.e. The dimension of is (10, T), where T is the number of time frames.

[0030] Step 2: Perform soft and hard mixed data enhancement processing on the obtained audio and video features to obtain enhanced audio and video features, and perform weighted fusion of the original audio and video features and the enhanced audio and video features in the batch dimension to generate multimodal missing training data.

[0031] Specifically, refer to Figure 3 The steps for using the reward mechanism combined with soft and hard mixed data enhancement to form multimodal missing training data include: 1) Utilizing a soft and hard hybrid data augmentation strategy with a reward mechanism In view of the problems such as insufficient generalization ability under incomplete modal information of samples in the multi-modal emotion recognition task and the limitations of samples in the real scenario, this application proposes to use a reward mechanism combined with a soft-hard hybrid data augmentation strategy. This method generates diverse training samples through the combination of "soft augmentation" and "hard augmentation". Soft augmentation mainly adopts a relatively gentle weak augmentation strategy to slightly transform the original data while keeping its main semantic information unchanged. Hard augmentation adopts a more radical strong augmentation strategy to significantly change the manifestation form of the original data and enhance the robustness of the model.

[0032] Specifically, the soft augmentation strategy of the soft-hard hybrid data augmentation takes the preprocessed audio-visual feature information and as the object of the soft-hard hybrid data augmentation strategy. First, for each sample in a batch, the soft augmentation coefficient C (0 < C < 1) of the speech modality and the soft augmentation coefficient 1 - C of the visual modality are generated through a reward function. In the reward function, an accuracy reward is set, which directly reflects the classification accuracy of the model for the current batch. The higher the accuracy, the more beneficial the current augmentation coefficient allocation is for model recognition, and the greater the reward value. A stability penalty is set to suppress the drastic fluctuation of the augmentation coefficient between adjacent batches. If the current augmentation coefficient differs significantly from the previous batch's augmentation coefficient, the penalty value increases, encouraging a smoother adjustment process of the augmentation coefficient. The influence of this item is controlled by the hyperparameter λ.

[0033] ;(1) Among them, represents the current reward score, represents the recognition accuracy of the current batch, represents the missing augmentation coefficient of the current audio modality, represents the missing augmentation coefficient of the audio-visual modality of the previous batch, is used to suppress the drastic fluctuation of the augmentation coefficient, and λ is the coefficient of the stability penalty.

[0034] Then, the augmentation coefficient is updated through an update function. The update of the augmentation coefficient is based on the change trend of the reward value to avoid the model relying too much on the complete modality. Specifically, if the current reward value is higher than the previous batch, it means that the current augmentation coefficient allocation is better, and the augmentation coefficient should continue to be adjusted in the current direction; otherwise, the adjustment should be slowed down or reversed. The augmentation coefficient update formula is as follows: ;(2) Among them, α is the weight learning rate, ensures that the augmentation coefficient is within the set minimum / maximum range, and the update range of the augmentation coefficient , represents the initial value of the augmentation coefficient, Represents the current reward score, Represents the reward score of the previous batch.

[0035] Update the upper and lower bounds of the enhancement coefficient C And through Maintain the complementarity of the enhancement coefficients of the speech and visual modalities. During the training process, losses are generated through forward propagation and the network parameters are updated backward; at the same time, the reward is calculated And update the enhancement coefficient C according to the above formula; after the update, the speech enhancement coefficient C (0 < C < 1) and the enhancement coefficient 1 - C of the visual modality take effect immediately in the next batch.

[0036] Expand the constants C and 1 - C to And In the dimensional space of, that is, the soft enhancement matrices of the audio-visual modalities are obtained And , and then the feature information of audio and visual is weighted respectively to achieve soft enhancement. The hard enhancement strategy of the hard-soft mixed data enhancement is to directly set the enhancement matrix of the speech modality or the visual modality to zero, that is, set the hard enhancement matrix of the speech modality To a constant value of 0, while the hard enhancement matrix of the video modality Is 1; similarly, set the hard enhancement matrix of the visual modality To a constant value of 0, while the hard enhancement matrix of the speech modality Is 1.

[0037] 2) Co-enhancement of multi-modal data Traditional data enhancement methods mostly target single modalities and it is difficult to ensure the semantic consistency between multi-modal data. For this reason, this application proposes a multi-modal co-enhancement strategy to maintain the consistency of its time sequence and its label semantics during the enhancement process of audio and visual modalities. Specifically, the original feature information and the audio-visual feature information after hard-soft mixed data enhancement are weighted and represented in the batch dimension. For the audio modality, first, each sample is weighted and fused under different modalities, and a weighted representation is generated: ; (3) ; (4) Among them, Is the original audio feature, Is the enhancement matrix of the audio, Is the audio weighting coefficient, Is the missing representation of the audio modality generated by hard enhancement.

[0038] For the video modality, a similar weighted fusion process is adopted: ; (5) ;(6) in, is the original audio feature, is the audio enhancement matrix, is the audio weighting coefficient, Audio modality-missing representations generated for hard enhancement.

[0039] After the above operation, the batch size is expanded from B to 4×B, so the training data adds three incomplete modal feature information pairs, that is, the input of audio and video features after collaborative soft and hard mixed data enhancement and It includes incomplete feature information in four scenarios: "complete information", "partial information attenuation", "audio loss" and "video loss", such as Figure 3 shown.

[0040] To ensure that the model can evenly learn different modality loss scenarios, the system performs a global random shuffle after feature expansion. Specifically, a random permutation index of length 4×B is generated in the BatchSize dimension, and the audio and video feature information and corresponding labels are synchronously rearranged. The calculation formula is as follows: ;(7) ;(8) in, and is a random permutation index of audio and video, and Represent the shapes of audio and video feature information respectively, dim=0 represents the batch dimension, and formula 8 represents the random index and Arrange audio and video features.

[0041] Step 3: Extract video features and audio features from the multimodal missing training data respectively, fuse the extracted features across modalities to obtain fused features, and perform sentiment classification and recognition based on the fused features.

[0042] Specifically, refer to Figure 4 , the steps of extracting features from the audio and video branches of multimodal data include: The hybrid spatiotemporal convolutional network consists of 1D convolutional blocks for extracting temporal features and 2D convolutional blocks for extracting spatial features. The temporal convolutional network uses 1D convolutional layers combined with batch normalization, ReLU activation functions, and maxpool layers to learn temporal features from the input. The spatial convolutional network uses a nonlocal global feature interaction mechanism to transfer information between arbitrary spatial locations, effectively capturing global context. This structure first extracts temporal features of audio and video using 1D convolutional blocks, then extracts spatial features of the audio and video features using 2D convolutional blocks, ultimately extracting the spatiotemporal characteristics of the feature input.

[0043] The video branch consists of two parts: the first part extracts facial features from a single video frame, and the second part extracts spatiotemporal features from the entire video sequence. To achieve an end-to-end trainable model, this application uses the EfficientFace architecture as a 2D feature extractor, extracting features independently for each frame.

[0044] In the first part of the video branch, when a video sequence of k frames after soft and hard mixed data enhancement is input, each of the k frames is processed independently by the 2D feature extractor to obtain the facial feature information of each frame. The features of each frame are stacked in the temporal dimension to form a temporal facial feature information representation. ,The hybrid spatiotemporal convolutional network of the second part is input to further extract spatiotemporal features.

[0045] In the second part of the video branch, it is assumed that certain feature representations have been extracted from the input video data through EfficientFace In order to further improve the model's ability to model global dependencies, we first use a 1D convolutional layer with batch normalization and a ReLU activation function to locally model the temporal features. Then, we map the temporal dimension features to spatial features. The calculation formula is: ;(9) Among them, the Reshape operation represents the time series dimension feature Mapped to For example, when T=15, it is mapped to H=15, W=1, keeping the total dimensional space unchanged.

[0046] In this feature space, a lightweight NonLocal module is introduced to extract spatial features. First, the input feature information is subjected to three 1×1 convolutions to generate Q, K, and V vectors. Then, the correlation between all positions is calculated through dot product and softmax. The global features are then weighted and aggregated using the correlation weights to improve the global dependency modeling capability. Finally, the Nonlocal output is connected to the original feature residual to enhance the feature expression, thereby efficiently completing the spatiotemporal feature learning of the video branch. Its calculation formula is as follows: ;(10) ;(11) ;(12) in, The dimension representing each batch is The input audio or video features, Represents input Middle position The output features of 、 and represents the 1×1 convolution weight, and Represents the input audio and video features at position and vector, represents the similarity function calculated by dot product, represents the feature mapping function obtained by linear transformation, Represents the number of positions N, where N represents the total number of spatial positions .

[0047] The audio branch adopts a similar structural design to the video branch, and extracts audio features through four hybrid spatiotemporal convolutional networks. This design ensures the depth of feature extraction while maintaining computational efficiency.

[0048] Reference Figure 5 ,The steps of multimodal fusion using the interactive attention mechanism for the extracted feature information include: The interactive attention mechanism is designed to achieve deep information interaction between audio and visual modalities. Unlike the traditional self-attention mechanism, the interactive attention mechanism in this application only uses Q and K to calculate the correlation weight between modalities, without directly performing a weighted summation on V. Specifically, the features of one modality are used as Q and the features of the other modality are used as K, and the attention weight matrix is ​​calculated using the following formula: ; (13) Among them, and Represents the query and key of audio and video modes respectively, represents the dimension of the key, It is a normalization method.

[0049] Therefore, the fusion of audio and video can use the interactive attention gating mechanism to control which part of the information is more important at each moment: ;(14) ;(15) in, The function is a gating mechanism for adaptive adjustment of audio and video information based on the interactive attention mechanism, which determines the most important information features of audio and video information at a certain moment.

[0050] In the process of cross-modal feature fusion, Figure 5 As shown, taking the video branch as an example, the linear transformation information of the audio modal feature is used as Q, and the video modal feature information is used as K. Then, and After scaling and normalization, Softmax promotes competition in the attention matrix, highlighting the more important feature information of each modality, and obtaining the importance score of each key relative to each query. The video modality calculates the relative importance of each attribute of the video modality by querying the importance score of each attribute in the audio modality with the video modality, that is, the relevance weight. The same is true in the audio branch, so we obtain the relevance weight and , and then dropout is used to prevent overfitting. Finally, the output correlation weight matrix is ​​used to multiply the original feature information element by element to achieve mid-term fusion of multimodal features.

[0051] During cross-modal feature fusion, information from the video and audio modalities that is more similar, and vice versa, is highlighted. A multimodal fusion method using a mutual attention mechanism emphasizes information from one modality that is similar to the other, while suppressing dissimilar features from the two modalities. Therefore, only when features from both modalities exhibit high consistency do these features receive a greater weight in the final prediction, guiding the model to automatically focus on modality-independent features or features with strong cross-modal consistency, achieving multimodal fusion through correlation weighting.

[0052] Step 4: Define the loss function, train the multimodal sentiment classification model, and obtain the trained multimodal sentiment classification model.

[0053] Specifically, in this embodiment, using the fused features for training to complete the sentiment classification task includes the following steps: The sentiment analysis model maps the fused multimodal features to specific emotion categories to achieve final emotion recognition. The model mainly consists of two parts: global pooling and feature fusion, and linear classifier. After completing the multimodal feature extraction and fusion, the model performs global pooling operations on the high-dimensional features of the audio and visual modalities respectively. The output features of the audio branch and the visual branch are globally averaged pooled in the time series dimension, and the sequence features of the audio and video are compressed into a global feature vector of a fixed length. Subsequently, the pooled audio and visual features are spliced ​​in the feature dimension to form a joint multimodal feature representation. The fused feature fully integrates the global information of the audio and visual modalities, providing a discriminant basis for subsequent sentiment classification. After obtaining the fused feature, the model uses a fully connected linear classifier (Fully Connected Layer, FC) to discriminate the emotion category, that is: ; (16) Here, y represents the predicted sentiment classification.

[0054] Specifically, the fused features are input into a linear layer, which outputs a score for each emotion category. During the training phase, the model is optimized using a cross-entropy loss function. The final category scores output by the model are normalized using softmax to obtain a probability distribution for each emotion category. The category with the highest probability is selected as the final recognition result.

[0055] Simulation experiment results and analysis: According to the simulation experiment results in Table 1, we used a public facial expression dataset, a voice and intonation database, and collected patient data for testing. This method improved the emotion classification accuracy by an average of 4% in the full modality (AV) case. We also tested the classification accuracy when the model was fed only unimodal data. We found that the model trained with modality-missing data simulating real-world scenarios achieved up to 30% improvement in accuracy for audio / video unimodal (A / V) input, and up to 20% improvement in average accuracy (M).

[0056] Our proposed training method and system model maintain high emotion recognition performance even in scenarios where multiple modalities are missing (including situations where only a single modality is input or partial modal information is attenuated). In comparative experiments, our method significantly outperformed baseline models that did not employ our strategy, regardless of whether a single modality was missing or partial modal information was lost. This result shows that, in response to the problem of insufficient adaptability to modal loss in the background technology, the present invention effectively generates training samples covering multiple modal loss situations through soft and hard hybrid data enhancement, greatly improving the robustness and adaptability of the model in a real diagnosis and treatment environment; in response to the problems of data distribution differences and insufficient generalization ability, the multimodal collaborative enhancement mechanism effectively enhances the complementarity between different modalities while maintaining the temporal and semantic consistency between modalities, so that the model can better generalize to clinical collection data; in response to the problem of insufficient feature fusion depth, the hybrid spatiotemporal convolutional network combined with the interactive attention mechanism not only realizes multi-level modeling of local and global features, but also accurately captures cross-modal dependencies, significantly improving the discriminative power of fused features; at the same time, for the existing problem of weak data enhancement, the collaborative enhancement strategy of the present invention ensures the consistency between multimodal data while simulating real modal loss and partial attenuation scenarios, thereby effectively improving the stability and accuracy of the model in a changing environment. Overall, the method of the present invention has shown significant advantages in solving the four major problems described in the background technology, and provides solid technical support for the practical application of multimodal emotion recognition technology in the diagnosis and treatment of emotional and behavioral disorders in adolescents.

[0057] Table 1 Simulation experiment results;

[0058] Example 2 This embodiment provides a training system for a multimodal sentiment classification model, including: The data acquisition and feature extraction module is configured to: acquire video data for emotion classification, preprocess the video data, and obtain video features and audio features; The training data generation module is configured to: perform soft and hard mixed data enhancement processing on the obtained audio and video features to obtain enhanced audio and video features, and perform weighted fusion of the original audio and video features and the enhanced audio and video features in the batch dimension to generate multimodal missing training data; The feature fusion module is configured to extract video features and audio features from multimodal missing training data, perform cross-modal feature fusion on the extracted features to obtain fused features, and perform sentiment classification and recognition based on the fused features; The model training module is configured to: define a loss function, train the multimodal sentiment classification model, and obtain a trained multimodal sentiment classification model.

[0059] It should be noted that the above modules correspond to the steps in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules can be executed in a computer system as part of the system.

[0060] In further embodiments, there is also provided: An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed by the processor, wherein when the computer instructions are executed by the processor, the method in embodiment 1 is performed. For the sake of brevity, no further details are given here.

[0061] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0062] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method in embodiment 1 is completed.

[0063] The method in Example 1 can be directly executed by a hardware processor, or by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, it will not be described in detail here.

[0064] A computer program product includes a computer program, wherein the computer program implements the method in embodiment 1 when executed by a processor.

[0065] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions contained in program modules, which are executed in a device on a real or virtual processor of a target to perform the process / method described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided between program modules as needed. The machine-executable instructions for the program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0066] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the computer or other programmable data processing device, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0067] In the context of the present invention, computer program code or related data can be carried by any appropriate carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals include electrical, optical, radio, acoustic, or other forms of propagated signals, such as carrier waves, infrared signals, and the like.

[0068] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0069] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A training method for a multimodal sentiment classification model, characterized in that: The following steps are involved: Obtain video data for emotion classification, preprocess the video data, and obtain video features and audio features; Perform soft and hard mixed data enhancement processing on the obtained audio and video features to obtain enhanced audio and video features, and perform weighted fusion of the original audio and video features and the enhanced audio and video features in the batch dimension to generate multimodal missing training data; Extract video and audio features from multimodal missing training data, perform cross-modal feature fusion on the extracted features, obtain fused features, and perform sentiment classification and recognition based on the fused features; Define the loss function, train the multimodal sentiment classification model, and obtain the trained multimodal sentiment classification model.

2. The method for training a multimodal sentiment classification model according to claim 1, wherein: The soft-hard hybrid data enhancement includes soft enhancement and hard enhancement. Soft enhancement adopts a mild weak enhancement strategy to slightly transform the original data while keeping its main semantic information unchanged. Hard enhancement adopts a radical strong enhancement strategy to significantly change the representation of the original data and enhance the robustness of the model.

3. The method for training a multimodal sentiment classification model according to claim 2, wherein: The soft enhancement generates soft enhancement coefficients of audio and video features through a reward function, and iteratively updates the soft enhancement coefficients through an update function.

4. The method for training a multimodal sentiment classification model according to claim 3, wherein: The reward function includes an accuracy reward and a stability penalty. The accuracy reward reflects the classification accuracy of the current batch of the model, and the stability penalty is used to suppress the sharp fluctuation of the soft enhancement coefficient between adjacent batches.

5. The method for training a multimodal sentiment classification model according to claim 3, wherein: The reward function is: ； in, Represents the reward score of the current batch, Represents the recognition accuracy of the current batch, Represents the current audio mode missing enhancement coefficient, represents the audio and video modality missing enhancement coefficient of the previous batch, and λ is the coefficient of stability penalty.

6. The method for training a multimodal sentiment classification model according to claim 3, wherein: The update function is: ； Among them, α is the weight learning rate, represents the initial value of the enhancement coefficient, Represents the reward score of the current batch, Represents the reward score of the previous batch.

7. A training system for a multimodal sentiment classification model, characterized in that: include: The data acquisition and feature extraction module is configured to: acquire video data for emotion classification, preprocess the video data, and obtain video features and audio features; The training data generation module is configured to: perform soft and hard mixed data enhancement processing on the obtained audio and video features to obtain enhanced audio and video features, and perform weighted fusion of the original audio and video features and the enhanced audio and video features in the batch dimension to generate multimodal missing training data; The feature fusion module is configured to extract video features and audio features from multimodal missing training data, perform cross-modal feature fusion on the extracted features to obtain fused features, and perform sentiment classification and recognition based on the fused features; The model training module is configured to: define a loss function, train the multimodal sentiment classification model, and obtain a trained multimodal sentiment classification model.

8. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 6 is completed.

9. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images