A method and system for emotion recognition combining spectrum sensing and multi-modal information fusion

Abstract

The application discloses a kind of joint spectrum sensing and multi-modal information fusion emotion recognition method and system, the emotion recognition method first obtains text, visual and audio data under dialogue scene and extracts modal initial features from it.Then, using frequency adapter executes mask perception local spectral filtering, and feature is explicitly decoupled into low-frequency component containing global semantics and high-frequency component containing instantaneous fluctuation, and the feature representation of spectral balance is reconstructed by dynamic gating mechanism.Next, frequency perception cross-modal attention mechanism is designed, while capturing the semantic correlation between modal, by amplifying the high-frequency residual of attention matrix and rebalancing the spectral energy of output feature, effectively offsetting the over-smoothing problem of deep network caused by Softmax.Finally, multi-level fusion is carried out based on spectral calibration feature to realize emotion prediction, which significantly improves the recognition performance of cross-modal conflict and sudden emotional change.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion analysis technology, specifically relating to an emotion recognition method and system that combines spectrum perception and multimodal information fusion. Background Technology

[0002] Multimodal emotion recognition in dialogue aims to decode complex interactive dynamics from textual, acoustic, and visual cues. Unlike unimodal emotion detection, multimodal emotion recognition requires the collaborative integration of information from different modalities. For example, a speaker might use a flat tone and expressionless face to utter positive words to express sarcasm—a phenomenon that can only be revealed through the complex interactions and conflicts between modalities.

[0003] To capture these multimodal dependencies, current state-of-the-art methods have largely shifted towards Transformer-based architectures. While Transformers are highly effective at capturing long-range context, pure attention models face a neglected theoretical bottleneck: spectral bias. Recent theoretical analyses in Visual Transformers (ViT) show that the self-attention mechanism essentially acts as a low-pass filter. As network depth increases, high-frequency (HF) components of feature maps are systematically suppressed, leading to over-smoothing or attention collapse, causing feature representations to tend towards uniformity.

[0004] To address the aforementioned challenges, this invention proposes a frequency-aware multimodal fusion emotion recognition method that reconstructs the multimodal fusion paradigm from a signal processing perspective. By explicitly performing spectral decoupling and calibration during the input and attention phases, it effectively overcomes the over-smoothing bottleneck of deep networks while preserving key high-frequency emotional cues. Summary of the Invention

[0005] The purpose of this invention is to address the low-pass filtering bias and high-frequency signal loss problems existing in the Transformer architecture of the prior art. It proposes an emotion recognition method and system that combines spectrum sensing and multimodal information fusion. This enables the capture of long-distance context during deep feature fusion and the retention of high-frequency signals representing sudden emotions and modal conflicts through a full-link spectrum calibration mechanism, thereby effectively improving the accuracy and robustness of emotion recognition.

[0006] In a first aspect, the present invention provides an emotion recognition method that combines spectrum sensing and multimodal information fusion, the method comprising:

[0007] An emotion recognition model is constructed; the emotion recognition model includes a frequency adaptation module, a frequency-aware cross-attention module, and a classifier connected in sequence; the frequency adaptation module is used to process input features; the frequency-aware cross-attention module is used to perform multimodal fusion and execute two-stage spectral calibration in the core fusion stage; the classifier is used to perform classification operations on the fusion results of the frequency-aware cross-attention module to obtain the output results of the emotion recognition model;

[0008] The frequency adaptation module includes multiple parallel frequency adapters, each used to process the input features of each mode. In the frequency adapter, after extracting low-frequency and high-frequency components from the input features through a deep convolutional layer, a dynamic gating mechanism is used to process the low-frequency and high-frequency components to obtain the spectral balance features output by the frequency adapter.

[0009] Collect multimodal dialogue data of the test dialogue; extract the initial modal features of each modal dialogue data, and input them into the emotion recognition model to obtain the emotion corresponding to the test dialogue.

[0010] Preferably, the method for obtaining the spectral balance feature is as follows: splicing low-frequency components and high-frequency components, and processing the splicing result through a cascaded fully connected layer and a Sigmoid activation function to obtain gating weights; multiplying the gating weights and their complementary weights element-wise with the low-frequency components and high-frequency components respectively, and fusing the multiplication results to obtain the spectral balance feature.

[0011] Preferably, the frequency-aware cross-attention module includes multiple parallel frequency-aware cross-attention sub-blocks, and each frequency-aware cross-attention sub-block corresponds to a frequency adapter. The frequency-aware cross-attention module uses the splicing result of the output features of multiple frequency-aware cross-attention sub-blocks as the output feature. Each frequency-aware cross-attention sub-block uses the spectral balance feature output by the corresponding frequency adapter as the main input feature, and the spectral balance features output by the other frequency adapters as auxiliary input features.

[0012] Preferably, in the frequency-aware cross-attention sub-block, the main input features are sequentially combined with the auxiliary input features to perform cross-modal cross-attention fusion operations, and the feature spectrum energy is rebalanced on the fusion results to obtain the intermediate fused features output by the frequency-aware cross-attention sub-block.

[0013] Preferably, the method for rebalancing the feature spectrum energy is as follows: performing local mean decomposition on the fusion result to obtain low-frequency and high-frequency components; and introducing learnable parameters. and Channel-level reweighting is performed on the frequency bands to obtain intermediate fusion features.

[0014] Preferably, the cross-modal cross-attention fusion method is as follows:

[0015] The auxiliary input features are mapped to a key matrix and a value matrix, and another input feature of the frequency-aware cross-attention sub-block is mapped to a query matrix. The standard cross-attention weight matrix is ​​calculated based on the query matrix and the key matrix. The standard cross-attention weight matrix is ​​decomposed into low-frequency components and high-frequency components, and a scaling factor is introduced to amplify the high-frequency components before fusing them with the low-frequency components to obtain the calibrated cross-attention weight matrix. The fused features are obtained based on the calibrated cross-attention weight matrix and the value matrix.

[0016] Preferably, the classifier comprises a fully connected layer, a multilayer perceptron, and a softmax activation function connected in sequence.

[0017] Preferably, the emotion recognition model is trained using a dataset containing multimodal dialogue data from different subjects. During training, an overall loss function is constructed to guide the parameter updates of the model. The overall loss function includes multi-class cross-entropy loss, low-frequency smoothing loss, and high-frequency sparsity loss. The multi-class cross-entropy loss is used to measure the difference between the classification result and the true label. The low-frequency smoothing loss is used to calculate the mean square error of the low-frequency components. The high-frequency sparsity loss is used to calculate the L1 norm of the high-frequency components.

[0018] Preferably, the modal initial features are extracted from the multimodal dialogue data using a convolutional neural network; the multimodal dialogue data includes text data, visual data, and audio data.

[0019] Secondly, the present invention provides an emotion recognition system that combines spectrum sensing and multimodal information fusion, used to perform the aforementioned emotion recognition method; the emotion recognition system includes a data acquisition module, a feature extraction module, a frequency adaptation module, a frequency sensing fusion module, and an emotion classification module; the data acquisition module is used to acquire multimodal dialogue data; the feature extraction module is used to extract original feature sequences of text, audio, and vision; the frequency adaptation module is used to decouple high and low frequency components and reconstruct spectral balance features during the input stage; the frequency sensing fusion module is used to amplify high-frequency residuals and balance spectral energy distribution in cross-modal interactions; and the emotion classification module is used to map the fused features to an emotion category probability space.

[0020] Thirdly, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the aforementioned emotion recognition method.

[0021] Fourthly, the present invention provides a readable storage medium storing a computer program; when executed by a processor, the computer program is used to implement the aforementioned emotion recognition method.

[0022] The beneficial effects of this invention are:

[0023] 1. In the emotion recognition model, this invention uses a frequency adapter to process the initial features of multimodal expressions. By explicitly decoupling the feature spectrum at the input stage, it overcomes the spectral bias problem in the Transformer architecture, thereby ensuring that transient high-frequency emotional cues (such as intonation bursts) are not smoothed out early.

[0024] 2. In the frequency-aware cross-attention module, this invention processes the input features through a two-stage spectral calibration strategy (high-frequency amplification of the attention matrix and reweighting of feature energy), which effectively counteracts the low-pass filtering effect of the Softmax function, enabling the model to capture modal inconsistencies and rapid sentiment shifts in deep features.

[0025] 3. This invention achieves interpretable frequency band decomposition by introducing low-frequency smoothing loss and high-frequency sparsity loss, which significantly outperforms the baseline model in emotion recognition tasks and exhibits stronger robustness and stability. Attached Figure Description

[0026] Figure 1 This is an overall flowchart of Embodiment 1 of the present invention.

[0027] Figure 2 This is a model framework diagram of Embodiment 1 of the present invention.

[0028] Figure 3 This is a schematic diagram of the frequency adapter in Embodiment 1 of the present invention.

[0029] Figure 4 This is a schematic diagram of the structure of the frequency-aware cross-attention sub-block in Embodiment 1 of the present invention. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings.

[0031] Example 1

[0032] like Figure 1 As shown, an emotion recognition method combining spectrum sensing and multimodal information fusion includes the following steps:

[0033] S1: Building the dataset

[0034] A dataset containing multimodal dialogue data from different subjects was constructed, with each subject's multimodal dialogue data serving as a sample. Each modal dialogue data point within each sample was labeled with an emotion type. The multimodal dialogue data included text, visual, and audio data. Initial modal features were extracted from each modal dialogue data point using a convolutional neural network to obtain the initial text features. Initial visual features and initial audio features ;in, This represents the number of samples in the dataset.

[0035] S2: Building an emotion recognition model

[0036] like Figure 2 As shown, the emotion recognition model comprises a frequency adaptation module, a frequency-aware cross-attention module, and a classifier connected in sequence. The frequency adaptation module processes the input features of each modality to preserve key high-frequency emotion signals. The frequency-aware cross-attention module performs multimodal fusion and executes two-stage spectral calibration in the core fusion stage. The classifier performs classification operations on the fusion results from the frequency-aware cross-attention module, yielding the output of the emotion recognition model.

[0037] S2.1: Frequency Adaptation Module

[0038] Figure 3 As shown, to prevent low-pass smoothing bias caused by deep networks, especially Softmax attention, a lightweight frequency band decoupling and energy redistribution are performed before fusion using a frequency adaptation module. The frequency adaptation module includes three parallel frequency adapters, used to process the initial text features, initial audio features, and initial visual features, respectively. The processing steps are as follows:

[0039] (1) Extract the local DC (DC) component of mask sensing as the low frequency (LF) component through a deep convolutional layer. It is represented as:

[0040]

[0041] Among them, LocalMean Achieved through depthwise convolution; For window size; These are the initial features of the modality.

[0042] This operation acts as a learnable "soft low-pass" filter, capturing slowly changing contextual semantics.

[0043] (2) Obtain the high frequency (HF) component by residual subtraction. It represents rapid changes deviating from the local mean, usually corresponding to emotional outbursts or sudden changes; high-frequency components. Represented as:

[0044]

[0045] (3) The spectral balance characteristics of the frequency adapter output are obtained by reconstructing the dynamic gating features. This balances the preservation of high-frequency cues with noise suppression; when the discourse semantics are stable, gating tends to favor low frequencies; when abrupt changes occur, gating amplifies high-frequency signals. Spectral balance characteristics. The specific acquisition process is as follows: splicing low-frequency components and high frequency components The concatenated result is then processed through a series of fully connected layers and a sigmoid activation function to obtain the gated weights. ; Gating weights The complementary weights of the gate weights are respectively related to the low-frequency components. and high frequency components Element-wise multiplication is performed, and the results are fused to obtain the spectral balance characteristics. The above process can be expressed mathematically as follows:

[0046]

[0047]

[0048] in, This represents the Sigmoid activation function; Indicates a fully connected layer; Indicates splicing.

[0049] S2.2: Frequency-aware cross-attention module

[0050] like Figure 4 As shown, the frequency-aware cross-attention module comprises three parallel frequency-aware cross-attention sub-blocks, each corresponding one-to-one with a frequency adapter. The frequency-aware cross-attention module uses the concatenated output features of the three frequency-aware cross-attention sub-blocks as its output feature. Each frequency-aware cross-attention sub-block uses the spectral balance feature output by its corresponding frequency adapter as its primary input feature c0, and the spectral balance features output by the other two frequency adapters as auxiliary input features c1 and c2.

[0051] In the frequency-aware cross-attention sub-block, a cross-modal cross-attention fusion operation is performed on the main input feature and the auxiliary input feature c1 to obtain the first fused feature. ; for the first fusion feature Perform a cross-modal cross-attention fusion operation with the auxiliary input feature c2 to obtain the second fused feature. To prevent the second fusion feature The spectral distribution is distorted again, affecting the second fusion feature. Perform eigenspectral energy rebalancing, i.e., rebalance the second fused feature. Local mean decomposition is performed to obtain low-frequency and high-frequency components, which are represented as follows:

[0052]

[0053]

[0054] Introducing learnable parameters and Channel-level reweighting is performed on the frequency band to obtain intermediate fusion features. This ensures that a balance is maintained between global semantics (low frequency) and transient difference cues (high frequency) during deep propagation. Intermediate fusion features. Represented as:

[0055]

[0056] The process of performing cross-modal cross-attention fusion is as follows:

[0057] (1) Modal projection

[0058] The auxiliary input features are mapped to key matrices through learnable linear projections. Sum matrix And another input feature is mapped to a query matrix through a learnable linear projection. It is used to model the interaction relationships between modalities.

[0059] (2) Attention matrix high-frequency amplification

[0060] Based on the query matrix Bond matrix Calculate the standard cross-attention weight matrix It is represented as follows:

[0061]

[0062] in, Let be the dimension of the key matrix; For padding mask.

[0063] (3) Due to the normalization properties of Softmax, the standard cross-attention weight matrix Essentially, it's a low-pass smoothing filter. To counteract this effect, the standard cross-attention weight matrix... Decomposed into low-frequency components and high frequency components It is represented as follows:

[0064]

[0065]

[0066] in, Low-frequency components The element value in; Standard cross-attention weight matrix The total number of valid positions (not obscured by the mask); This is the set of valid locations.

[0067] (4) Introduce a learnable scaling factor For high frequency components After amplification, it is combined with low-frequency components. The cross-attention weight matrix is ​​fused to obtain the calibrated cross-attention weight matrix. It is represented as:

[0068]

[0069] Based on the calibrated cross-attention weight matrix Sum matrix Obtain fusion features This operation enhances the "sharpness" of attention distribution while preserving non-uniform, differential cues.

[0070] S2.3: The classifier processes the output features of the frequency-aware cross-attention module through a fully connected layer, a multilayer perceptron, and a softmax activation function connected in sequence to obtain the classification result output by the emotion recognition model. .

[0071] S3: Train an emotion recognition model using a dataset, and then use the trained model to perform emotion recognition. During model training, construct an overall loss function. This guides the parameter updates of the model. The overall loss function includes multi-class cross-entropy loss, low-frequency smoothing loss, and high-frequency sparsity loss, expressed as:

[0072]

[0073] in, For multi-class cross-entropy loss; For low-frequency smoothing loss; For high-frequency sparsity loss; and It is a hyperparameter of the balanced frequency regularization term.

[0074] Cross-entropy loss is used to measure the difference between the classification result and the true label:

[0075]

[0076] The low-frequency smoothing loss is used to calculate the mean square error of low-frequency components, forcing them to change smoothly. It is expressed as:

[0077]

[0078] in, The number of dialogue sequences; The low-frequency component obtained in the frequency adapter;

[0079] The high-frequency sparsity loss is used to calculate the L1 norm of the high-frequency components, causing them to activate only during significant sentiment shifts, as shown below:

[0080]

[0081] in, This refers to the high-frequency components obtained from the frequency adapter.

[0082] S4: Method Evaluation

[0083] The emotion recognition method proposed in this embodiment was tested on two publicly available datasets related to multimodal emotion and widely recognized in the industry, and corresponding experiments were conducted. The datasets and parameter settings are as follows:

[0084] In emotion recognition, this invention uses the IEMOCAP and MELD datasets to verify the effectiveness of this embodiment.

[0085] (1) IEMOCAP: The IEMOCAP dataset contains approximately 12 hours of two-person dialogue videos, which are divided into 7433 statements and 151 dialogue segments. Each statement is labeled with one of six emotion tags: happy, sad, neutral, angry, excited, and frustrated. The dataset was labeled by multiple emotion experts to reduce subjective bias in the labeling process.

[0086] (2) MELD: The MELD dataset is a multi-role dialogue dataset containing 13,708 statements and 1,433 dialogues from the TV series Friends. Each statement is labeled with one of seven emotion categories: anger, disgust, fear, joy, neutrality, sadness, and surprise. Similar to IEMOCAP, MELD is also labeled by multiple emotion experts to ensure the accuracy and objectivity of the labeling results.

[0087] During testing, a comprehensive and detailed evaluation strategy was employed for the IEMOCAP dataset (six-class classification) and the MELD dataset (seven-class classification) to accurately measure the model's performance in sentiment classification tasks. Specifically, this invention calculates the F1 score for each sentiment category to evaluate the model's classification accuracy across different sentiment categories. Furthermore, to comprehensively reflect the overall model performance and consider the imbalance in class distribution within the dataset, this invention further calculates a weighted average F1 score. This evaluation method not only accurately characterizes the model's classification performance across each category but also effectively measures the impact of class imbalance on overall performance, thus providing a scientific basis for model optimization and improvement.

[0088] Test Result Analysis:

[0089] The proposed method was validated on the IEMOCAP dataset (six-class classification) and the MELD dataset (seven-class classification), and compared with existing state-of-the-art models. On the IEMOCAP dataset, the proposed method achieved superior performance in most sentiment categories, with an overall weighted F1 score of 73.05%, a 2.31% improvement over strong baseline models. The most significant improvements were observed in the Happiness and Excited categories, which are rich in high-frequency signals, with F1 scores of 63.52% and 78.69%, respectively. Furthermore, the proposed method also achieved an excellent score of 69.20% in the Frustrated category, which has fewer samples and ambiguous context, demonstrating stronger fine-grained discrimination ability. Especially under imbalanced data conditions, the proposed method still exhibits strong generalization ability in categories with scarce samples. On the MELD dataset, the proposed method achieved an overall weighted F1 score of 66.75%, surpassing existing state-of-the-art models. Especially under long-tailed data distribution conditions, the method of this invention performs exceptionally well on the low-resource categories Fear and Disgust, achieving F1 scores of 32.26% and 32.08%, respectively, representing a significant improvement over traditional models. Simultaneously, it maintains robust high performance on the dominant categories Neutral and Joy (79.38% and 65.90%, respectively). This method effectively overcomes the low-pass filtering bias in deep networks through a frequency-domain-aware end-link spectrum calibration mechanism. Even when dealing with modality conflicts or rapid sentiment shifts, it can still accurately recover high-frequency sentiment cues, thus significantly improving the accuracy and robustness of sentiment classification.

[0090] Example 2

[0091] A sentiment recognition system integrating spectrum sensing and multimodal information fusion includes a data acquisition module, a feature extraction module, a frequency adaptation module, a frequency-aware fusion module, and a sentiment classification module. The data acquisition module acquires multimodal dialogue data. The feature extraction module extracts raw feature sequences from text, audio, and visual data. The frequency adaptation module includes a mask-aware local filtering unit and a dynamic gating unit, responsible for decoupling high- and low-frequency components and reconstructing spectral balanced features during the input phase. The frequency-aware fusion module integrates an attention matrix calibration unit and a feature reweighting unit, responsible for amplifying high-frequency residuals and balancing spectral energy distribution in cross-modal interactions. The sentiment classification module maps the fused features to a sentiment category probability space.

[0092] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An emotion recognition method combining spectrum sensing and multimodal information fusion, characterized in that: The method includes: An emotion recognition model is constructed; the emotion recognition model includes a frequency adaptation module, a frequency-aware cross-attention module, and a classifier connected in sequence; the frequency adaptation module is used to process input features; the frequency-aware cross-attention module is used to perform multimodal fusion and execute two-stage spectral calibration in the core fusion stage; the classifier is used to perform classification operations on the fusion results of the frequency-aware cross-attention module to obtain the output results of the emotion recognition model; The frequency adaptation module includes multiple parallel frequency adapters, each used to process the input features of each mode. In the frequency adapter, after extracting low-frequency and high-frequency components from the input features through a deep convolutional layer, a dynamic gating mechanism is used to process the low-frequency and high-frequency components to obtain the spectral balance features output by the frequency adapter. Collect multimodal dialogue data of the test dialogue; extract the initial modal features of each modal dialogue data, and input them into the emotion recognition model to obtain the emotion corresponding to the test dialogue.

2. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 1, characterized in that: The method for obtaining the spectral balance feature is as follows: splicing low-frequency components and high-frequency components, and processing the splicing result through a cascaded fully connected layer and a Sigmoid activation function to obtain gating weights; multiplying the gating weights and their complementary weights element-wise with the low-frequency components and high-frequency components respectively, and fusing the multiplication results to obtain the spectral balance feature.

3. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 1, characterized in that: The frequency-aware cross-attention module includes multiple parallel frequency-aware cross-attention sub-blocks, and each frequency-aware cross-attention sub-block corresponds to a frequency adapter. The frequency-aware cross-attention module uses the splicing result of the output features of multiple frequency-aware cross-attention sub-blocks as the output feature. Each frequency-aware cross-attention sub-block uses the spectral balance feature output by the corresponding frequency adapter as the main input feature, and the spectral balance feature output by the other frequency adapters as the auxiliary input feature.

4. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 3, characterized in that: In the frequency-aware cross-attention sub-block, the main input features are sequentially combined with the auxiliary input features to perform cross-modal cross-attention fusion operations, and the feature spectrum energy is rebalanced on the fusion results to obtain the intermediate fused features output by the frequency-aware cross-attention sub-block.

5. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 4, characterized in that: The method for rebalancing the feature spectrum energy is as follows: local mean decomposition is performed on the fusion result to obtain low-frequency and high-frequency components; learnable parameters are introduced. and Channel-level reweighting is performed on the frequency bands to obtain intermediate fusion features.

6. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 4, characterized in that: The cross-modal cross-attention fusion method is as follows: The auxiliary input features are mapped to a key matrix and a value matrix, and another input feature of the frequency-aware cross-attention sub-block is mapped to a query matrix. The standard cross-attention weight matrix is ​​calculated based on the query matrix and the key matrix. The standard cross-attention weight matrix is ​​decomposed into low-frequency components and high-frequency components, and a scaling factor is introduced to amplify the high-frequency components before fusing them with the low-frequency components to obtain the calibrated cross-attention weight matrix. The fused features are obtained based on the calibrated cross-attention weight matrix and the value matrix.

7. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 1, characterized in that: The classifier comprises a fully connected layer, a multilayer perceptron, and a softmax activation function connected in sequence.

8. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 1, characterized in that: The emotion recognition model is trained using a dataset containing multimodal dialogue data from different subjects. During training, an overall loss function is constructed to guide the parameter updates of the model. The overall loss function includes multi-class cross-entropy loss, low-frequency smoothing loss, and high-frequency sparsity loss. The multi-class cross-entropy loss is used to measure the difference between the classification result and the true label. The low-frequency smoothing loss is used to calculate the mean square error of the low-frequency components. The high-frequency sparsity loss is used to calculate the L1 norm of the high-frequency components.

9. The emotion recognition method based on the joint spectrum sensing and multimodal information fusion according to claim 1, characterized in that: The initial modal features are extracted from the multimodal dialogue data using a convolutional neural network; the multimodal dialogue data includes text data, visual data, and audio data.

10. An emotion recognition system that combines spectrum sensing and multimodal information fusion, characterized in that: An emotion recognition method combining joint spectrum sensing and multimodal information fusion as described in claim 1 is used to execute the emotion recognition system comprising a data acquisition module, a feature extraction module, a frequency adaptation module, a frequency sensing fusion module, and an emotion classification module. The data acquisition module is used to acquire multimodal dialogue data; the feature extraction module is used to extract raw feature sequences from text, audio, and visual data; and the frequency adaptation module is used to decouple high- and low-frequency components and reconstruct spectral balance features during the input phase. The frequency-aware fusion module is used to amplify high-frequency residuals and balance spectral energy distribution in cross-modal interactions; The sentiment classification module is used to map the fused features to the sentiment category probability space.

Images