Emotion recognition method and system based on multi-modal fusion data enhancement framework

Abstract

The invention discloses an emotion recognition method based on a multi-modal fusion data enhancement framework, and the method comprises the following steps: S1, extracting multi-modal input sequences comprising n physiological signals, carrying out the modal fusion of the multi-modal input sequences through a CNXAF model, and obtaining multi-modal fusion data; s2, a conditional self-attention generative adversarial network C-SAGAN is used to perform data enhancement on the multi-modal fusion data to obtain an enhanced data set, and the C-SAGAN comprises two components, namely a generator and a discriminator; and S3, performing emotion recognition on the enhanced data set by using a pre-trained improved ConvNeXt-t model. The invention further discloses an emotion recognition system based on the multi-modal fusion data enhancement framework. According to the invention, data fusion and data expansion can be carried out on the multi-modal physiological signals, and high-precision multi-modal emotion recognition is realized end to end.

Description

Technical Field

[0001] The present invention relates to the field of physiological signal emotion computing, and in particular to an emotion recognition method and system based on a multimodal fusion data enhancement framework. Background Art

[0002] Emotions play a key role in daily human interactions, influencing decision-making processes and shaping our perception of the world. With the growing interest in establishing emotional interactions between humans and computers, emotion recognition has become an important research field. Emotion recognition aims to develop systems that can recognize, interpret, process, and simulate human emotions. This process involves utilizing different data sources, including audio, text, facial expressions, and physiological signals. However, the efficacy of the first three methods may be affected because individuals may unconsciously or deliberately hide their true emotions. In contrast, physiological signals have higher objectivity, reliability, and biological relevance in emotion recognition. Therefore, physiological signals such as electroencephalography (EEG), electromyography (EMG), electrooculogram (EOG), and galvanic skin response (GSR) have received attention as major research areas in recent years.

[0003] In recent years, researchers have increasingly explored physiological signals as a means to perform emotion recognition. However, the complexity of human emotions poses a significant challenge to achieving optimal performance in emotion recognition using a single physiological signal, which is caused by the differences in perceived emotional cues across different tasks and individuals. In addition, single-signal methods are susceptible to noise interference, which reduces the credibility of the system. To address this challenge, some studies need to adopt multimodal fusion strategies to utilize multiple physiological signals to improve reliability and accuracy. However, compared with acquiring image and speech signals, collecting multiple physiological signal data faces obvious challenges. Insufficient data remains a persistent obstacle in emotion recognition tasks, especially in multimodal emotion recognition. However, current research on data augmentation mainly focuses on enhancing data from individual modalities, such as EEG, while ignoring the expansion of the entire multimodal dataset. And because there are significant interrelationships between multimodal fused physiological data, it is necessary to utilize information from all feature locations to create a complex generative network. In addition, simply generating unlabeled multimodal data is unlikely to significantly improve the performance of supervised models, as these models require sufficient complete data label pairs for training. Therefore, it is imperative to design methods to generate multimodal data while ensuring the provision of corresponding labels. Addressing these challenges will enhance the ability of multimodal physiological signals to demonstrate more powerful performance in emotion recognition tasks. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an emotion recognition method and a system based on a multimodal fusion data enhancement framework, which can perform data fusion and data expansion on multimodal physiological signals and realize high-precision multimodal emotion recognition in an end-to-end manner.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide an emotion recognition method based on a multimodal fusion data enhancement framework, comprising the following steps:

[0006] S1: extract multimodal input sequences including n physiological signals, perform modal fusion on these multimodal input sequences using the CNXAF model to obtain multimodal fusion data;

[0007] S2: Use a conditional self-attention generative adversarial network C-SAGAN to perform data enhancement on the multimodal fusion data to obtain an enhanced data set. The C-SAGAN network includes two components: a generator and a discriminator. The generator receives conditional information from the outside to guide the generator to generate multimodal data that meets the specified conditions; the discriminator also receives conditional information from the outside to identify the generated multimodal data and the real data;

[0008] S3: Use the pre-trained modified ConvNeXt-t model for emotion recognition on the augmented dataset.

[0009] In a preferred embodiment of the present invention, the CNXAF model includes a one-dimensional convolutional primary local feature extraction network based on independent modality, a deep separable convolutional intermediate local feature extraction network based on independent modality, a multi-scale convolutional intermediate global feature extraction network, and a self-attention ConvNeXt feature fusion network;

[0010] The independent modality-based one-dimensional convolutional primary local feature extraction network is used to generate primary local features of each modality from the extracted multimodal input sequence;

[0011] The independent modality-based deep separable convolutional intermediate local feature extraction network is used to extract intermediate local features of each modality;

[0012] The multi-scale convolutional intermediate global feature extraction network is used to extract intermediate global features including all modalities;

[0013] The self-attention ConvNeXt feature fusion network fuses and splices the intermediate local features and the intermediate global features by constructing a ConvNeXt network with an attention mechanism.

[0014] Furthermore, the independent modality-based deep separable convolution intermediate local feature extraction network consists of a channel-by-channel convolution network and a point-by-point convolution network; the main channel convolution network uses the same deep convolution kernel W for each input channel c of the convolution sequence of each modality. c Perform channel-by-channel convolution as shown in formula (1) to obtain the channel convolution feature map of each modality Where c represents the channel of the mode;

[0015]

[0016] In the formula, is the primary local feature, where M1, M2, ..., M n Represents different physiological signals, W c represents the convolution kernel of the cth channel of the depthwise convolution, Y c Represents the cth channel of the depth convolution output;

[0017] The point-by-point convolution network performs point-by-point convolution on the channel convolution feature map as shown in formula (2) to obtain the intermediate local features of each modality.

[0018]

[0019] Where C represents the total number of channels of each modality, W p Represents a 1×1 point-wise convolution kernel.

[0020] Furthermore, the multi-scale convolutional intermediate global feature extraction network concatenates the primary local features of each modality to obtain the primary global features. The obtained primary global features are processed through a multi-scale convolutional network as shown in formula (3) to obtain the intermediate global features that include all modalities:

[0021]

[0022] Where Concat represents the connection operation; W α ,W β ,W σ Represent the convolution kernel of each convolution respectively.

[0023] Furthermore, the self-attention ConvNeXt feature fusion network is constructed with the attention mechanism shown in equations (4) and (5):

[0024]

[0025] In the formula, is the learnable weight; Q Uni ,K Fus ,V FusUnimodal query, fusion of keys and values, respectively,

[0026]

[0027] Where, d k is the dimension of the attention head, used to scale the attention score, and Y is the input intermediate multimodal fusion data.

[0028] In a preferred embodiment of the present invention, the generator uses the self-attention mechanism shown in formula (6) to dynamically adjust attention to ensure that the model can pay attention to the emotional characteristics between modalities:

[0029]

[0030] In the formula, β j,i is the attention weight of position i to position j; exp() represents exponential operation; s ij represents the similarity between position i and position j; f and g represent attention weights, where f(x) = W f x,g(x)=W g x; j represents the synthesized area, and i represents the position.

[0031] In a preferred embodiment of the present invention, the training of the generator and the discriminator is achieved by the adversarial loss function shown in formula (7):

[0032]

[0033] In the formula, Represents the expectation of the data; p data Represents the distribution of data; L D and L G They represent the discriminator loss and generator loss respectively; D(.) and G(.) represent the outputs of the discriminator and generator respectively; Yr represents the label information, which is used to guide the judgment of the discriminator.

[0034] In a preferred embodiment of the present invention, the improved ConvNeXt-t model first uses a 7×7 convolution kernel to perform a convolution operation to reduce the resolution of the input data and extract low-level features, then uses Layer Normalization to normalize the data, and finally connects multiple ConvNeXt blocks using a residual structure to perform classification tasks.

[0035] In order to solve the above technical problems, another technical solution adopted by the present invention is: to provide an emotion recognition system based on a multimodal fusion data enhancement framework, using the emotion recognition method based on a multimodal fusion data enhancement framework as described in any one of claims 1 to 8, comprising:

[0036] The multimodal fusion data acquisition module is used to extract multimodal input sequences including n physiological signals, and perform modal fusion on these multimodal input sequences using the CNXAF model to obtain multimodal fusion data;

[0037] An enhanced data set acquisition module is used to perform data enhancement on the multimodal fusion data using a conditional self-attention generative adversarial network C-SAGAN to obtain an enhanced data set. The C-SAGAN network includes two components: a generator and a discriminator. The generator receives condition information from the outside to guide the generator to generate multimodal data that meets the specified conditions; the discriminator also receives condition information from the outside to identify the generated multimodal data and the real data;

[0038] Multimodal emotion recognition module for emotion recognition on augmented datasets using pre-trained ConvNeXt-t models.

[0039] In a preferred embodiment of the present invention, the CNXAF model in the multimodal fusion data acquisition module includes a one-dimensional convolution primary local feature extraction network based on independent modality, a deep separable convolution intermediate local feature extraction network based on independent modality, a multi-scale convolution intermediate global feature extraction network and a self-attention ConvNeXt feature fusion network;

[0040] The independent modality-based one-dimensional convolutional primary local feature extraction network is used to generate primary local features of each modality from the extracted multimodal input sequence;

[0041] The independent modality-based deep separable convolutional intermediate local feature extraction network is used to extract intermediate local features of each modality;

[0042] The multi-scale convolutional intermediate global feature extraction network is used to extract intermediate global features including all modalities;

[0043] The self-attention ConvNeXt feature fusion network fuses and splices the intermediate local features and the intermediate global features by constructing a ConvNeXt network with an attention mechanism.

[0044] Compared with the prior art, the present invention proposes an overall framework for emotion recognition, which seamlessly integrates the fusion, generation and recognition steps from multimodal fusion to emotion recognition. This comprehensive model architecture improves accuracy and a more detailed understanding of emotional states. The specific beneficial effects are reflected in:

[0045] (1) We design a CNXAF model that integrates unimodal and multimodal features to create a comprehensive multimodal representation. This global fusion mechanism ensures that information from all modalities is fully considered. Incorporating the attention mechanism into ConvNeXt enhances the model's ability to integrate physiological signals and emphasizes the importance of different modalities throughout the process.

[0046] (2) The present invention utilizes the self-attention mechanism of the C-SAGAN model to greatly enhance the ability of the generator and discriminator to capture complex relationships in multimodal data. This enhancement diversifies the dataset and ensures that the generated data retains the basic information of the original dataset, thereby improving fidelity and richness. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a framework and flow chart of the emotion recognition method and system based on the multimodal fusion data enhancement framework of the present invention;

[0048] Figure 2 It is the framework and flow chart of the CNXAF module;

[0049] Figure 3 It is the framework and flow chart of the generator in the C-SAGAN model network. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0051] See also Figure 1 , the embodiment of the present invention includes:

[0052] An emotion recognition method based on a multimodal fusion data enhancement framework comprises the following steps:

[0053] S1: Extract multimodal input sequence including n physiological signals where M1, M2, …, M n Representing different physiological signals, these multimodal input sequences are modally fused using the CNXAF model. The CNXAF model consists of four networks, including a one-dimensional convolutional primary local feature extraction network based on independent modalities, a deep separable convolutional intermediate local feature extraction network based on independent modalities, a multi-scale convolutional intermediate global feature extraction network, and a self-attention ConvNeXt feature fusion network. For details, see Figure 2 .

[0054] The one-dimensional convolutional primary local feature extraction network is composed of multiple one-dimensional convolutional networks that share the same output dimension. Specifically, the extracted multimodal input sequence is subjected to one-dimensional convolution to generate multiple convolution sequences. Each convolution sequence shares the same dimension D, which is the primary local features of each modality.

[0055] The deep separable convolutional intermediate local feature extraction network is implemented by a deep separable network (DSC network), which consists of a channel-by-channel convolutional network and a point-by-point convolutional network. The same deep convolution kernel W is used for each input channel c of the convolution sequence of each modality. c Perform channel-by-channel convolution as shown in formula (1) to obtain the channel convolution feature map of each modality Where c represents the channel of the mode.

[0056]

[0057] Where W c represents the convolution kernel of the cth channel of the depthwise convolution, Y c Represents the c-th channel of the depthwise convolution output.

[0058] Then the channel convolution feature map is convolved point by point as shown in formula (2) to obtain the intermediate local features of each modality

[0059]

[0060] Where C represents the total number of channels of each modality, W p Represents a 1×1 point-wise convolution kernel.

[0061] The multi-scale convolutional network (MSC) is used to extract mid-level global features. It consists of multiple convolutional networks with different convolutional kernels and a one-dimensional convolutional network of uniform dimension. These convolutional kernels can capture local and global features at different scales, thereby improving the model's perception of details and global patterns in multimodal signals. In addition, the MSC network is also combined with an inverse scaling mechanism to restore high-level abstract features to lower-scale representations by reverse mapping between multi-scale features. This inverse scaling can better maintain the balance of information between different scales and further enhance the ability to capture multi-dimensional features of the signal, thereby achieving a more accurate description of complex patterns. Specifically, the primary local features of each modality are spliced ​​to obtain the primary global features. The obtained features are processed through a multi-scale convolutional network as shown in formula (3) to obtain intermediate global features that include all modalities.

[0062]

[0063] Where Concat represents the connection operation; W α ,W β ,W σ Represent the convolution kernel of each convolution respectively.

[0064] The self-attention ConvNeXt feature fusion network constructs a ConvNeXt structure with an attention mechanism as shown in equations (4) and (5), and fuses and splices the intermediate local features and intermediate global features obtained in the above steps to obtain a feature representation The overall architecture of ConvNeXt is very similar to ResNet-50, consisting of a feature extraction layer, an intermediate layer with a stacked bottleneck structure of different dimensions, and a final high-dimensional feature classification layer, but modifications have been made in the details, see equations (4) and (5). The final high-level global features, i.e., the final multimodal fusion data, are obtained through the self-attention ConvNeXt feature fusion network.

[0065]

[0066] In the formula, is the learnable weight; Q Uni ,K Fus ,V Fus Unimodal query, fusion of keys and values, respectively,

[0067]

[0068] Where, d k is the dimension of the attention head, used to scale the attention score, and Y is the input intermediate multimodal fusion data.

[0069] S2: Use the conditional self-attention generative adversarial network C-SAGAN to perform data enhancement on the multimodal fusion data to obtain an enhanced data set, which is an enhanced data set of more data formed by expanding the generated multimodal fusion data to the original real data. The C-SAGAN network includes two components: the generator and the discriminator. Figure 1 The generator tries to deceive the discriminator so that it cannot distinguish the generated fusion data from the real fusion data, while the discriminator tries to distinguish the two.

[0070] The generator receives conditional information from the outside, which may be emotion tags, auxiliary information or other related data. In the present invention, emotion tags are used as conditional information. These conditional information will guide the generator to generate multimodal data that meets the specified conditions. Figure 3, the input conditional information is embedded into a low-dimensional vector so as to be combined with other inputs of the generator. The generator starts by calculating the attention score through a series of one-dimensional transposed convolutional layers, and then combines the conditional information with the attention score to gradually generate the initial version of the multimodal fusion data. At each stage of the generation process, the generator uses the self-attention mechanism shown in formula (6) to dynamically adjust the attention to ensure that the model can focus on the emotional characteristics between modalities.

[0071]

[0072] In the formula, β j,i is the attention weight of position i to position j; exp() represents exponential operation; s ij represents the similarity between position i and position j; f and g represent attention weights, where f(x) = W f x,g(x)=W g x; j represents the synthesized area, and i represents the position.

[0073] The discriminator network is similar to the generator. The discriminator also receives conditional information and embeds it into the attention score matrix calculated by the convolution layer. The discriminator then discriminates the generated data and the real data through the fully connected layer, thereby identifying the real data and the generated data. The training of the generator and the discriminator is achieved by the adversarial loss function shown in formula (7). Finally, during the training process, additional multimodal data samples are generated and combined with the real data samples to expand the training set.

[0074]

[0075] In the formula, Represents the expectation of the data; p data Represents the distribution of data; L D and L G They represent the discriminator loss and generator loss respectively; D(.) and G(.) represent the outputs of the discriminator and generator respectively; Yr represents the label information, which is used to guide the judgment of the discriminator.

[0076] S3: Use the pre-trained modified ConvNeXt-t model for emotion recognition on the augmented dataset.

[0077] The enhanced dataset is used for emotion recognition using a pre-trained improved ConvNeXt-t model. The improved ConvNeXt-t model is an advanced architecture based on the ResNet structure, and is upgraded by integrating the design ideas of Transformer and the optimization techniques of modern deep learning. Specifically, it first uses a 7×7 convolution kernel to perform a convolution operation to reduce the resolution of the input data and extract low-level features. Then, LayerNorm (LayerNormalization) is used to replace the traditional BatchNorm (Batch Normalization) to normalize the data. Finally, multiple ConvNeXt blocks are connected using a residual structure for classification tasks. The emotion classifier is trained by using an enhanced training dataset with labeled emotions. During the training process, the model parameters are continuously adjusted through the back-propagation algorithm to minimize the difference between the predicted emotion category and the true emotion category.

[0078] In order to quantitatively evaluate the results of the present invention, the present method is compared with several state-of-the-art emotion recognition methods using two common datasets (DEAP and WESAD).

[0079] Table 1

[0080]

[0081] Different enhancement factors represent different times of enlarging the original dataset.

[0082] Table 2

[0083]

[0084] Different enhancement factors represent different times of enlarging the original dataset.

[0085] Tables 1 and 2 show the emotion recognition accuracy achieved when data generated by different enhancement factors (0x, 0.5x, ... and 5x) are combined into the two datasets. The results show that when the enhancement factor exceeds 1, the recognition rate of the model on the DEAP dataset tends to stabilize at around 96%, with a maximum of (96.06±1.14)%, as shown in Table 1; the recognition rate of the model on the WESAD dataset is as high as (95.70±1.56)%, as shown in Table 2.

[0086] Table 3

[0087]

[0088] In addition, Table 3 shows the highest emotion recognition results of the comparative methods and the present invention on different data sets. The proposed model significantly improves the accuracy of emotion recognition compared with the comparative learning methods (Husformer, DAIL, MCMT, DGR-ERPS).

[0089] In summary, the present invention can solve the problem of insufficient data encountered when using multimodal physiological signals for emotion recognition. The framework can directly perform data expansion on the fused multimodal data, thereby solving the consequence of decreased accuracy caused by insufficient data in subsequent emotion recognition.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0091] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An emotion recognition method based on a multimodal fusion data enhancement framework, characterized in that: The following steps are involved: S1: extract multimodal input sequences including n physiological signals, perform modal fusion on these multimodal input sequences using the CNXAF model to obtain multimodal fusion data; S2: Use a conditional self-attention generative adversarial network C-SAGAN to perform data enhancement on the multimodal fusion data to obtain an enhanced data set. The C-SAGAN network includes two components: a generator and a discriminator. The generator receives conditional information from the outside to guide the generator to generate multimodal data that meets the specified conditions; the discriminator also receives conditional information from the outside to identify the generated multimodal data and the real data; S3: Use the pre-trained modified ConvNeXt-t model for emotion recognition on the augmented dataset.

2. The emotion recognition method based on the multimodal fusion data enhancement framework according to claim 1 is characterized in that: The CNXAF model includes a one-dimensional convolutional primary local feature extraction network based on independent modality, a deep separable convolutional intermediate local feature extraction network based on independent modality, a multi-scale convolutional intermediate global feature extraction network, and a self-attention ConvNeXt feature fusion network; The independent modality-based one-dimensional convolutional primary local feature extraction network is used to generate primary local features of each modality from the extracted multimodal input sequence; The independent modality-based deep separable convolutional intermediate local feature extraction network is used to extract intermediate local features of each modality; The multi-scale convolutional intermediate global feature extraction network is used to extract intermediate global features including all modalities; The self-attention ConvNeXt feature fusion network fuses and splices the intermediate local features and the intermediate global features by constructing a ConvNeXt network with an attention mechanism.

3. The emotion recognition method based on the multimodal fusion data enhancement framework according to claim 2 is characterized in that: The independent modality-based deep separable convolution intermediate local feature extraction network consists of a channel-by-channel convolution network and a point-by-point convolution network; the main channel convolution network uses the same deep convolution kernel W for each input channel c of the convolution sequence of each modality c Perform channel-by-channel convolution as shown in formula (1) to obtain the channel convolution feature map of each modality Where c represents the channel of the mode; In the formula, is the primary local feature, where M1, M2, ..., M n Represents different physiological signals, W c represents the convolution kernel of the cth channel of the depthwise convolution, Y c Represents the cth channel of the depth convolution output; The point-by-point convolution network performs point-by-point convolution on the channel convolution feature map as shown in formula (2) to obtain the intermediate local features of each modality. Where C represents the total number of channels of each modality, W p Represents a 1×1 point-wise convolution kernel.

4. The emotion recognition method based on the multimodal fusion data enhancement framework according to claim 2 is characterized in that: The multi-scale convolutional intermediate global feature extraction network concatenates the primary local features of each modality to obtain the primary global features The obtained primary global features are processed through a multi-scale convolutional network as shown in formula (3) to obtain the intermediate global features that include all modalities: Where Concat represents the connection operation; W α ,W β ,W σ Represent the convolution kernel of each convolution respectively.

5. The emotion recognition method based on the multimodal fusion data enhancement framework according to claim 2 is characterized in that: The self-attention ConvNeXt feature fusion network is constructed with the attention mechanism shown in equations (4) and (5): In the formula, is the learnable weight; Q Uni ,K Fus ,V Fus Unimodal query, fusion of keys and values, Where, d k is the dimension of the attention head, used to scale the attention score, and Y is the input intermediate multimodal fusion data.

6. The emotion recognition method based on the multimodal fusion data enhancement framework according to claim 1 is characterized in that: The generator uses the self-attention mechanism shown in formula (6) to dynamically adjust attention to ensure that the model can focus on the emotional characteristics between modalities: In the formula, β j,i is the attention weight of position i to position j; exp() represents exponential operation; s ij represents the similarity between position i and position j; f and g represent attention weights, where f(x) = W f x,g(x)=W g x; j represents the synthesized area, and i represents the position.

7. The emotion recognition method based on the multimodal fusion data enhancement framework according to claim 1 is characterized in that: The training of the generator and discriminator is achieved by the adversarial loss function shown in formula (7): In the formula, Represents the expectation of the data; p data Represents the distribution of data; L D and L G Represent the discriminator loss and generator loss respectively; D(.) and G(.) represent the outputs of the discriminator and generator respectively; Yr represents the label information, which is used to guide the judgment of the discriminator.

8. The emotion recognition method based on the multimodal fusion data enhancement framework according to claim 1 is characterized in that: The improved ConvNeXt-t model first uses a 7×7 convolution kernel to perform a convolution operation to reduce the resolution of the input data and extract low-level features, then uses Layer Normalization to normalize the data, and finally connects multiple ConvNeXt blocks using a residual structure to perform classification tasks.

9. An emotion recognition system based on a multimodal fusion data enhancement framework, using the emotion recognition method based on a multimodal fusion data enhancement framework as described in any one of claims 1 to 8, characterized in that: include: The multimodal fusion data acquisition module is used to extract multimodal input sequences including n physiological signals, and perform modal fusion on these multimodal input sequences using the CNXAF model to obtain multimodal fusion data; An enhanced data set acquisition module is used to perform data enhancement on the multimodal fusion data using a conditional self-attention generative adversarial network C-SAGAN to obtain an enhanced data set. The C-SAGAN network includes two components: a generator and a discriminator. The generator receives conditional information from the outside to guide the generator to generate multimodal data that meets the specified conditions. The discriminator also receives conditional information from the outside to distinguish the generated multimodal data from the real data; Multimodal emotion recognition module for emotion recognition on augmented datasets using pre-trained ConvNeXt-t models.

10. The emotion recognition system based on the multimodal fusion data enhancement framework according to claim 9, characterized in that: The CNXAF model in the multimodal fusion data acquisition module includes a one-dimensional convolution primary local feature extraction network based on independent modality, a deep separable convolution intermediate local feature extraction network based on independent modality, a multi-scale convolution intermediate global feature extraction network and a self-attention ConvNeXt feature fusion network; The independent modality-based one-dimensional convolutional primary local feature extraction network is used to generate primary local features of each modality from the extracted multimodal input sequence; The independent modality-based deep separable convolutional intermediate local feature extraction network is used to extract intermediate local features of each modality; The multi-scale convolutional intermediate global feature extraction network is used to extract intermediate global features including all modalities; The self-attention ConvNeXt feature fusion network fuses and splices the intermediate local features and the intermediate global features by constructing a ConvNeXt network with an attention mechanism.

Images