A cross-domain facial emotion recognition method based on cue learning

Abstract

This invention discloses a cross-domain emotion recognition method based on cue learning, belonging to the field of multimodal emotion recognition technology. In the feature alignment stage, this invention introduces a spatial-channel collaborative attention module within the CLIP multimodal framework to enhance the capture of micro-expression features in low-light blurred regions, achieving effective alignment of emotional features in the visible and low-light domains. In the recognition and inference stage, this method employs a semantically guided contextual cue learning approach, fully utilizing the knowledge of the pre-trained CLIP model to construct a text representation more adapted to the target domain features for each emotion category. A visual-text dual-path collaborative optimization framework is designed, achieving effective alignment and robust recognition of emotional features in the visible and low-light domains through cross-modal contrastive learning and domain adversarial loss. This invention significantly improves the model's generalization ability and recognition performance in cross-illumination domain scenarios.

Description

Technical Field

[0001] This application relates to the fields of multimodal emotion recognition and computer vision technology, and in particular to a cross-domain facial emotion recognition method based on cue learning. Background Technology

[0002] Facial emotion recognition is a key technology in the field of artificial intelligence, widely used in human-computer interaction, intelligent healthcare, and mental health monitoring. In recent years, with the rapid development of deep learning technology, especially convolutional neural networks and Transformer models, significant progress has been made in the accuracy of emotion recognition on standard visible light datasets. However, transferring the visible light domain to low-light scenarios still faces many challenges, severely limiting the practicality of the recognition system.

[0003] Existing emotion recognition models primarily rely on extracting rich facial texture, contour, and micro-expression features from visible light images. In recent years, visual-language models, represented by CLIP, have become a focus of research in cross-modal tasks. CLIP achieves cross-modal semantic alignment by jointly training image and text encoders through contrastive learning on large-scale image-text pairs, mapping images and text to a shared embedding space. Its core principle is to leverage zero-shot transfer capabilities, dynamically generating classifier weights by converting class labels into natural language-described text prompts, thus enabling the model to adapt to new visual classification tasks. While this approach has some feasibility, in cross-illumination domain facial emotion recognition, especially in transfer applications from the visible light domain to the infrared domain, the CLIP model relies heavily on fixed and artificially constructed text prompts (e.g., "a photo of [class]") for classification. Such fixed prompts cannot perceive and fully utilize the unique thermal radiation or low-light characteristics of infrared images, making it difficult to dynamically construct domain-sensitive semantic representations. As a result, in cross-domain scenarios, the semantic guidance effectiveness of text prompts on attenuated infrared visual features is reduced, and it cannot effectively correct deviations in cross-modal alignment, becoming a bottleneck for performance improvement. Summary of the Invention

[0004] Therefore, it is necessary to provide a cross-domain emotion recognition method based on cue learning to address the aforementioned technical problems.

[0005] The following technical solution is adopted in this specification: This specification provides a cross-domain emotion recognition method based on cue learning, including: Acquire facial emotion images under low-light conditions; An improved CLIP visual text encoder is constructed. The improved CLIP visual text encoder includes a visual encoder, a text encoder, and a classifier, with the classifier connected to the visual encoder and the text encoder respectively. The visual encoder consists of a spatial-channel collaborative attention module and a Transformer encoder. The text encoder consists of a word embedding layer and a text encoding layer. The word embedding layer replaces the fixed prompt word module with a semantically guided context optimization prompt module. The spatial-channel collaborative attention module extracts multi-semantic spatial features of key facial regions in the facial emotion image using multi-scale depthwise separable convolution to obtain spatial information calibration features. After channel attention weighting of the spatial information calibration features in the channel dimension, a Transformer encoder is used to encode visual enhancement features. The context optimization prompt module constructs prompts that adapt to the emotion category features for preset emotion category labels, and the text encoding layer obtains text adaptation features of facial emotions. The visual enhancement features and text adaptation features are then fused and input into the classifier, and the emotion category with the highest confidence is output as the final predicted emotion category.

[0006] Furthermore, the spatial-channel collaborative attention module includes: a spatial attention submodule and a channel attention submodule connected in sequence; The spatial attention submodule is used to process the input features. Perform average pooling on both the height and width dimensions to obtain the height-dimensional feature map. and width dimension feature map The height dimension feature map and width dimension feature map The features are evenly divided into 4 sub-features along the channel dimension, denoted as follows: and (i=1,2,3,4); Spatial information at different scales is extracted from the sub-features by four depthwise separable convolutions, and the extracted features are merged by concatenation and the sub-features are normalized independently by GroupNorm; The concatenated multi-scale features are then used to generate a spatial weight map containing facial micro-expression texture features by n×n convolution. The attention weight graph mentioned above is mathematically expressed as follows: ; ; ; in, This indicates a splicing operation along the feature channel; and This indicates a grouping and normalization generation operation based on the height and width directions; Use the Sigmoid activation function; This represents the spatial enhancement features generated through the joint attention mechanism; The channel attention submodule is used to process the spatially calibrated features. Spatial downsampling and normalization are performed; channel dependencies are modeled using a lightweight single-head self-attention mechanism to generate channel attention weight vectors; these channel attention weight vectors are then multiplied element-wise with the spatially calibrated feature maps to achieve channel-level feature recalibration, resulting in global channel feature vectors. The dimension of the global channel feature vector is compressed, and a lightweight single-head self-attention mechanism is used to model the inter-channel dependencies to obtain the channel attention score. The global channel feature vector is represented as: ; The channel attention score , is represented as: in, The projection matrix; This is a scaling factor used to prevent the gradient from vanishing due to an excessively large dot product result; Generates a size of The channel correlation matrix.

[0007] Furthermore, the visual enhancement feature is represented as: ; in, Y Indicates visual enhancement features; Represents spatially enhanced features; ⊗ represents the channel attention score; ⊗ represents the channel-by-channel weighted operation.

[0008] Furthermore, the acquisition of text-based facial emotion adaptation features specifically includes: By introducing a set of learnable continuous context vectors , to replace fixed prompt words; among which This is a hyperparameter representing the number of context tags; It is a dimension related to word embedding ; Based on context vector sequence and word embedding vector Construct for the first Complete input prompts for emotion categories: ;in No. Each emotion input is a word embedding vector obtained from the CLIP word embedding layer; In the template The placeholder embedding is replaced with the current learnable vector. and the constructed A tip They are fed into the text encoder respectively. Text classification features are obtained: ; Provided to the text encoder g The hint (⋅) is in the following form: Each of them It is a vector with the same dimension as the word embedding; M This is a hyperparameter used to specify the number of context tags and will provide hints. t Input to text encoder g (⋅).

[0009] Furthermore, the fusion process of the visual enhancement features and the text adaptation features specifically includes: The overall semantics of the emotion task are obtained by calculating the average vector of the text adaptation features; The visual enhancement features and the text adaptation features are mapped to a unified embedding space, and element-wise multiplication is performed to obtain a fused feature vector that incorporates bimodal information.

[0010] Furthermore, the classifier includes a first fully connected layer, a ReLU activation function, a Dropout layer, and a second fully connected layer connected in sequence.

[0011] Furthermore, the emotion category with the highest confidence in the output is used as the final predicted emotion category, specifically including: The lightweight classifier maps the fused features to a... dimensional output tensor ,get The raw scores for each emotion category; Based on the original score, through The function, obtained by Confidence of each emotion category The index of the sentiment category with the highest confidence level will be used as the final predicted sentiment category. ; The trainable independent parameters of the lightweight classifier , is represented as: ; The lightweight classifier consists of a multilayer perceptron. The multilayer sensor is composed of... The calculation formula is: ; The output tensor The calculation formula is: for Each of the samples ,That Confidence of each category The calculation formula is: ; Final prediction category , is represented as: ; in, , These are the weight matrix and bias vector of the first fully connected layer, respectively. , These are the weight matrix and bias vector of the second fully connected layer, respectively. This is a vector resulting from the fusion of visual and textual features. It is a sample For the The original score of the class; Indicates the confidence level for each emotion category. ,satisfy.

[0012] Furthermore, the cross-domain emotion recognition method is trained end-to-end using a hybrid loss function that includes domain adversarial loss to remove domain-specific interference factors such as illumination conditions, thereby enhancing the model's generalization ability to the target domain.

[0013] Furthermore, the hybrid loss function of the adversarial loss includes a main classification loss based on cross-entropy. Cross-modal contrast loss Domain confrontation losses ; The main classification loss based on cross-entropy By minimizing the difference between the predicted probability and the true label, the model's ability to distinguish emotion categories is ensured; and label smoothing is set to alleviate overfitting and prevent the model from being overconfident in the training samples. The cross-modal contrast loss By leveraging the semantic alignment properties of the CLIP pre-training space, the cosine similarity between image features and text features is constrained. The domain adversarial loss By using gradient inversion layers to adversarially train a deceptive domain classifier, the features cannot distinguish between the source domain and the target domain. The main classification loss based on cross-entropy , is represented as: ; in, The number of samples; The total number of emotion categories; This is a real label; To predict probabilities; The cross-modal contrast loss , is represented as: ; in, Indicates cosine similarity; For visual enhancement features; These are the text features corresponding to the real categories; For the first Emotional textual features; Temperature coefficient; The domain adversarial loss , is represented as: ; in, Expressing expectations; The output probability of the domain discriminator; and These represent the data distributions in the source and target domains, respectively. The total loss function is expressed as: ; in, =1.0, =0.5, =0.3, which is a hyperparameter.

[0014] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: The cross-domain emotion recognition method based on cue learning provided in this specification replaces the fixed and manually constructed text cue in the CLIP model with learnable continuous context vectors. This constructs a text representation adapted to the target domain features for the preset emotion category labels, obtaining text-adapted features, which are then combined with image features to obtain multimodal features, thus optimizing the understanding of emotion categories. The improved CLIP model, through training on learnable continuous context vectors, better constructs domain-sensitive semantic representations, corrects deviations in cross-modal alignment, and effectively addresses the domain differences and feature attenuation problems caused by infrared imaging, enabling the model to maintain strong generalization ability and robustness under extreme lighting conditions. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This document provides a flowchart illustrating a cross-domain facial emotion recognition method based on cue learning. Figure 2 This document presents a schematic diagram of the network framework for a cross-domain facial emotion recognition model based on cue learning. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0018] To mitigate the distribution discrepancy between labeled visible light data (source domain) and unlabeled infrared data (target domain), researchers have proposed unsupervised domain adaptation (UDA) methods. These methods (such as those based on adversarial learning or feature alignment) aim to narrow the distribution gap between the source and target domains in the feature space by learning domain-invariant features. However, the effectiveness of these methods is limited when the source domain (visible light) and target domain (infrared) have fundamentally different imaging principles. Specifically, infrared imaging relies on the thermal radiation of object surfaces, which weakens facial details (such as nasolabial folds and skin micro-textures) and blurs key micro-expression features dependent on subtle muscle movements. Furthermore, infrared images have a narrow dynamic range and are susceptible to noise and thermal conduction effects. These factors cause the "domain-invariant features" extracted by traditional visual encoders to often lose crucial details for emotion recognition, resulting in unsatisfactory robustness and accuracy in the infrared domain.

[0019] To address the aforementioned problems, this invention proposes a cross-domain facial emotion recognition method based on cue learning. The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a cross-domain emotion recognition method based on cue learning as described in this specification, which specifically includes the following steps: S101: Construct a spatial-channel collaborative attention module and embed it into the hierarchical feature extraction path of the CLIP visual encoder to enhance the ability to capture facial expression features under low light conditions and obtain visually enhanced features.

[0021] 1. In the spatial dimension, firstly, the input features... Perform average pooling along the height and width dimensions to calculate the mean values ​​in the height and width directions, thereby obtaining two new feature maps: a height-dimensional feature map and a width-dimensional feature map. and .

[0022] These two visual features are evenly divided into 4 groups of sub-features along the channel dimension: and (i=1,2,3,4), each part contains C / 4 channels. Then, four depth-shared 1D convolutions (with kernel sizes of 3, 5, 7, and 9) are used to process the sub-features of each part to extract spatial information at different scales.

[0023] The extracted features are merged by concatenation, and sub-features are independently normalized using GroupNorm (groups=4) to eliminate semantic interference across sub-features. After concatenation, multi-scale features are used to generate a spatial attention weight map using n×n convolution, the mathematical expression of which is:

[0024] in, This indicates a splicing operation along the feature channel. and This indicates a grouping and normalization generation operation based on the height and width directions. The activation function is Sigmoid; the joint attention mechanism generates spatially enhanced features. This module effectively aggregates local micro-expression features of facial features using bidirectional attention, while suppressing background noise in non-facial areas. The design corresponds to calculating the feature distribution through a gating mechanism and a 256-dimensional bottleneck layer, ultimately outputting the spatially calibrated attention distribution in the height and width dimensions. and .

[0025] Finally, the attention weights are multiplied element-wise with the original input features to calibrate the features using spatial information. Experiments show that the parallel structure of multi-scale convolution improves the model's response intensity in micro-expression regions such as the eyes and corners of the mouth.

[0026] 2. In the channel dimension, the overall importance of each feature channel is "evaluated" through calculation, and then this importance score is used to amplify useful feature channels and suppress ineffective or noisy feature channels.

[0027] First, analyze the output features of SMSA. Perform spatial downsampling, and then perform downsampling on the features. Perform normalization processing; The query is obtained through the convolutional layer. ),key( ) and value ( ), then , , Reshape and transform the data to perform attention calculations, calculate attention scores, and apply attention dropout layers. Relate attention score to value ( Multiplying these features yields channel-attention-weighted features. Then, average pooling is applied to these channel-attention-weighted features to obtain a one-dimensional channel attention vector. This vector represents the "global importance" assessment for each channel;

[0028] Will The channel attention scores are obtained through a gate function, Softmax. Applying this to the spatially attention-weighted feature map yields the final features. While retaining most of the original spatial information, the feature dimensions are compressed. Subsequently, a lightweight single-head self-attention mechanism is used to model inter-channel dependencies.

[0029] in, The projection matrix is ​​used, and channel recalibration is ultimately achieved through Sigmoid gating. This design, together with the LayerNorm operation of the text adapter, enables cross-modal parameter sharing, thereby improving the cross-modal feature similarity of the dataset.

[0030] S102: Construct a semantically guided context optimization prompt module. To address the issue of the visual feature distribution shift relative to the pre-training space caused by the collaborative attention module in step S101, a semantically guided context optimization prompt module is constructed.

[0031] The module is configured to: initialize a set of learnable continuous context vectors, and during joint training, dynamically adjust the weights of the continuous context vectors using backpropagation gradients, so that the text embedding vectors generated by them actively perform semantic alignment with the visual enhancement feature distribution output in step S101, and concatenate the adjusted continuous context vectors with preset emotion category labels to generate a calibrated text representation adapted to low-light enhancement features.

[0032] Traditional cross-domain methods often perform visual enhancement independently, directly reusing the fixed classification heads of pre-trained models. However, research has found that after introducing the SCSA module to perform depthwise separable convolution and channel recalibration on low-light images, the output visual features... While it retains more micro-expression details, its features have significantly deviated from the natural image distribution during CLIP model pre-training. At this point, if fixed discrete cue words ("a photo of a...") are continued to be used, the generated text feature anchors cannot accurately match the drifting visual features, resulting in a capped recognition accuracy.

[0033] The core idea of ​​context optimization is to parameterize the fixed context portion of the prompt words and introduce a set of learnable continuous context vectors. This is used to replace fixed prompt words. This is a hyperparameter representing the number of context markers. It is a dimension related to word embedding The same vector.

[0034] The specific construction process is as follows: based on the model's context vector sequence and word embedding vectors Construct for the first Complete input prompts for emotion categories : ,in It is the first The emotion input is obtained from the CLIP word embedding layer to obtain the word embedding vector. This method can automatically find an optimal set of "prefixes" in the continuous feature space to guide the text encoder g( Generate semantic anchors that best match the current visual feature distribution.

[0035] Finally, the constructed A tip The text is input into the text encoder to generate the final text classification features. During training, only the context vector is updated via backpropagation. This makes the generated text features It can achieve optimal alignment with low-light facial visual features enhanced by the collaborative attention module.

[0036] S103: Input the obtained fused feature vector into a separate lightweight classifier. This classifier is responsible for the final decision and outputs the prediction result, corresponding to the confidence level of the sentiment category.

[0037] The method first defines an "independent lightweight classifier," which is a multilayer perceptron. The trainable independent parameters of the classifier are: ; Classifier Fusion features Mapped to dimensional output tensor : This tensor This refers to the "raw prediction results," often called logits. It contains model pairs. The unnormalized raw scores of each emotion category.

[0038] for Each of the samples ,That Confidence vectors for each category It is calculated using the Softmax function: in, It is a sample For the The original score of the class; This represents the confidence level for each emotion category, satisfying... ; Final prediction category This is the category index with the highest confidence: S104: The vision-text dual-path collaborative optimization framework is trained end-to-end using a hybrid loss function that includes domain adversarial loss to remove domain-specific interference factors such as illumination conditions and enhance the model's generalization ability to the target domain. The method specifically involves a hybrid loss function consisting of three parts: a main classification loss based on cross-entropy. Cross-modal contrast loss Domain confrontation loss .

[0039] The main classification loss based on cross-entropy ensures the model's ability to distinguish sentiment categories by minimizing the difference between the predicted probability and the true label. Additionally, label smoothing is applied to mitigate overfitting and prevent the model from becoming overconfident in the training samples.

[0040] The double summation in the formula represents summing over all samples and all classes, calculating the difference between the model's predicted values ​​and the actual labels. The use of the negative sign and the logarithmic function is to convert the difference into a loss value; the smaller the loss value, the more accurate the model's prediction. The goal of this loss function is to minimize... That is, to improve the predictive accuracy of the model.

[0041] Cross-modal contrastive loss leverages the semantic alignment properties of the CLIP pre-training space to constrain the cosine similarity between image features and text features: in This is a temperature coefficient used for autofocusing difficult sample pairs. Force image features and text features to align in a shared space to bridge the modal gap between CLIP pre-training and emotion tasks.

[0042] Domain confrontation loss Adversarial training using gradient inversion layers (GRL) deceives the domain classifier, making features unable to distinguish between the source and target domains. Let the domain classifier D(⋅) consist of two fully connected layers, with its loss function being:

[0043] The total loss function is the weighted sum of the three: Through hyperparameters =1.0, =0.5, =0.3, achieving the best multi-objective optimization effect.

[0044] For cross-domain emotion recognition tasks, a two-stage adversarial training mechanism is designed: the first stage optimizes the visual branch using only source domain data. Through three-stage training in the code, a dynamic course learning strategy is adopted, gradually increasing the adversarial loss weights each epoch. The second stage involves unfreezing all model parameters and loading a mix of source and target domain data. Based on a random sampling strategy, data augmentation (color perturbation + random occlusion) is applied to the target domain samples, while a confidence threshold is used to filter pseudo-labels for training. Gradient backpropagation for the domain classifier is implemented using a PyTorch hook mechanism.

[0045] like Figure 1 and Figure 2 The model shown below has the following test results: Emotion recognition was performed using visible light and infrared facial emotion test sets, respectively. The model selected was the one that performed best on the validation set. Accuracy and F1-Score were used as evaluation criteria, and the models were tested on the RAF-DB and Oulu-CASIA datasets, respectively. The experimental results of this invention are shown in Tables 1 and 2. Compared with the baseline model CLIP, the method of this invention demonstrates superior facial emotion recognition performance.

[0046] Table 1. Comparison of experimental results of different methods on visible light datasets Table 2 Comparison of experimental results of different methods on the Oulu-CASIA infrared dataset. Compared with existing methods, the beneficial effects of the present invention are as follows: This invention addresses the problem of drastic performance degradation in traditional methods when migrating from the visible to the infrared domain. Compared to the baseline CLIP model, accuracy is improved by approximately 22.9%, and the F1 score by approximately 20.29%. This breakthrough performance directly demonstrates that the framework employed in this invention can effectively address the domain differences and feature attenuation issues introduced by infrared imaging, enabling the model to maintain strong generalization ability and robustness even under extreme lighting conditions.

[0047] This invention enhances cross-modal semantic adaptation capabilities. It employs a semantically guided context-optimized cueing method, replacing the fixed and manually constructed text cueing in the CLIP model with learnable continuous context vectors. This dynamic cueing learning can better construct domain-sensitive semantic representations and correct biases in cross-modal alignment.

[0048] This invention achieves synergistic optimization of visual and textual information, fully utilizing multimodal complementary information. It constructs a visual-text dual-path synergistic optimization framework. The attention optimization module is responsible for image feature extraction and cross-modal feature alignment, while the text optimization module is responsible for understanding emotion categories using a text semantic enhancement model. The combination of these two modules creates a complementary optimization effect.

[0049] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A cross-domain emotion recognition method based on cue learning, characterized in that, include: Acquire facial emotion images under low-light conditions; Construct an improved CLIP visual text encoder; The improved CLIP visual text encoder includes a visual encoder, a text encoder, and a classifier, with the classifier connected to both the visual encoder and the text encoder. The visual encoder consists of a spatial-channel collaborative attention module and a Transformer encoder. The text encoder consists of a word embedding layer and a text encoding layer, with the word embedding layer replacing the fixed prompt word module with a semantically guided context optimization prompt module. The spatial-channel collaborative attention module extracts multi-semantic spatial features of key facial regions in the facial emotion image using multi-scale depth-separable convolution to obtain spatial information calibration features. After channel attention weighting of the spatial information calibration features in the channel dimension, it encodes them through a Transformer encoder to obtain visual enhancement features. The context optimization prompt module constructs prompts that adapt to the emotion category features for preset emotion category labels, and obtains text adaptation features of facial emotions through the text encoding layer. The visual enhancement features are then fused with the text adaptation features and input into the classifier, which outputs the emotion category with the highest confidence as the final predicted emotion category.

2. The cross-domain emotion recognition method based on cue learning as described in claim 1, characterized in that, The spatial-channel collaborative attention module includes a spatial attention submodule and a channel attention submodule connected in sequence. The spatial attention submodule is used to process the input features. Perform average pooling on both the height and width dimensions to obtain the height-dimensional feature map. and width dimension feature map The height dimension feature map and width dimension feature map The features are evenly divided into 4 sub-features along the channel dimension, denoted as follows: and (i=1,2,3,4); Spatial information at different scales is extracted from the sub-features through four depthwise separable convolutions, and the extracted features are merged by concatenation and the sub-features are independently normalized by GroupNorm; The spliced ​​multi-scale features are then used to generate a spatial weight map containing facial micro-expression texture features through n×n convolution. The attention weight graph mentioned above is mathematically expressed as follows: ； ； ； in, This indicates a splicing operation along the feature channel; and This indicates a grouping and normalization generation operation based on the height and width directions; Use the Sigmoid activation function; This represents the spatial enhancement features generated through the joint attention mechanism; The channel attention submodule is used to process the spatially calibrated features. Spatial downsampling and normalization are performed; channel dependencies are modeled using a lightweight single-head self-attention mechanism to generate channel attention weight vectors; these channel attention weight vectors are then multiplied element-wise with the spatially calibrated feature maps to achieve channel-level feature recalibration, resulting in global channel feature vectors. The dimension of the global channel feature vector is compressed, and a lightweight single-head self-attention mechanism is used to model the inter-channel dependencies to obtain the channel attention score. The global channel feature vector is represented as: ； The channel attention score , is represented as: in, The projection matrix; This is a scaling factor used to prevent the gradient from vanishing due to an excessively large dot product result; Generates a size of The channel correlation matrix.

3. The cross-domain emotion recognition method based on cue learning as described in claim 1, characterized in that, The visual enhancement feature is represented as follows: ； in, Y Indicates visual enhancement features; Represents spatially enhanced features; ⊗ represents the channel attention score; ⊗ represents the channel-by-channel weighted operation.

4. The cross-domain emotion recognition method based on cue learning as described in claim 1, characterized in that, The acquisition of the text adaptation features of facial emotions specifically includes: By introducing a set of learnable continuous context vectors , to replace fixed prompt words; among which This is a hyperparameter representing the number of context tags; It is a dimension related to word embedding ; Based on context vector sequence and word embedding vector Construct for the first Complete input prompts for emotion categories: ;in No. Each emotion input is a word embedding vector obtained from the CLIP word embedding layer; In the template The placeholder embedding is replaced with the current learnable vector. and the constructed A tip They are fed into the text encoder respectively. Text classification features are obtained: ; Provided to the text encoder g The hint (⋅) is in the following form: Each of them It is a vector with the same dimension as the word embedding; M This is a hyperparameter used to specify the number of context tags and will provide hints. t Input to text encoder g (⋅).

5. The cross-domain emotion recognition method based on cue learning as described in claim 1, characterized in that, The fusion process of the visual enhancement features and the text adaptation features specifically includes: The overall semantics of the emotion task are obtained by calculating the average vector of the text adaptation features; The visual enhancement features and the text adaptation features are mapped to a unified embedding space, and element-wise multiplication is performed to obtain a fused feature vector that incorporates bimodal information.

6. The cross-domain emotion recognition method based on cue learning as described in claim 1, characterized in that, The classifier comprises a first fully connected layer, a ReLU activation function, a Dropout layer, and a second fully connected layer connected in sequence.

7. The cross-domain emotion recognition method based on cue learning as described in claim 1, characterized in that, The output emotion category with the highest confidence is used as the final predicted emotion category, specifically including: The lightweight classifier maps the fused features to a... dimensional output tensor ,get The raw scores for each emotion category; Based on the original score, through The function, obtained by Confidence of each emotion category The index of the sentiment category with the highest confidence level will be used as the final predicted sentiment category. ; The trainable independent parameters of the lightweight classifier , is represented as: ； The lightweight classifier consists of a multilayer perceptron. The multilayer sensor is composed of... The calculation formula is: ； The output tensor The calculation formula is: for Each of the samples ,That Confidence of each category The calculation formula is: ； Final prediction category , is represented as: ； in, , These are the weight matrix and bias vector of the first fully connected layer, respectively. , These are the weight matrix and bias vector of the second fully connected layer, respectively. This is a vector resulting from the fusion of visual and textual features. It is a sample For the The original score of the class; Indicates the confidence level for each emotion category. ,satisfy.

8. The cross-domain emotion recognition method based on cue learning as described in claim 1, characterized in that, By using a hybrid loss function that includes domain adversarial loss, the cross-domain emotion recognition method is trained end-to-end to remove domain-specific interference factors such as lighting conditions, thereby enhancing the model's generalization ability to the target domain.

9. The cross-domain emotion recognition method based on cue learning as described in claim 8, characterized in that, The adversarial loss's hybrid loss function includes a main classification loss based on cross-entropy. Cross-modal contrast loss Domain confrontation losses ; The main classification loss based on cross-entropy By minimizing the difference between the predicted probability and the true label, the model's ability to distinguish emotion categories is ensured; and label smoothing is set to alleviate overfitting and prevent the model from being overconfident in the training samples. The cross-modal contrast loss By leveraging the semantic alignment properties of the CLIP pre-training space, the cosine similarity between image features and text features is constrained. The domain adversarial loss By using gradient inversion layers to adversarially train a deceptive domain classifier, the features cannot distinguish between the source domain and the target domain. The main classification loss based on cross-entropy , is represented as: ； in, The number of samples; The total number of emotion categories; This is a real label; To predict probabilities; The cross-modal contrast loss , is represented as: ； in, Indicates cosine similarity; For visual enhancement features; These are the text features corresponding to the real categories; For the first Emotional textual features; Temperature coefficient; The domain adversarial loss , is represented as: ； in, Expressing expectations; The output probability of the domain discriminator; and These represent the data distributions in the source and target domains, respectively. The total loss function is expressed as: ； in, =1.0, =0.5, =0.3, which is a hyperparameter.

Images