Expression Movement Unit Detection Method Based on Dual Cross-Modal Attention

Abstract

The present invention discloses a method for detecting expression motion units based on dual cross-modal attention. It obtains refined multi-granularity visual feature representation through visual encoders, local perception attention matrices, and graph convolutional networks. It utilizes a multi-level encoding process to first model the semantic associations between words in AU descriptions, and then models the sentence-level semantic associations between different AU descriptions, effectively mining the rich semantic information in AU descriptions and significantly enhancing the expressive power of AU text features. It designs a dual cross-modal attention strategy that is both global and local in coordination to achieve deep interaction between visual and text modalities, helping the model to more comprehensively understand the complex semantic associations between visual and text modalities and enhance AU feature representation. Ultimately, by combining multi-granularity visual features and multi-level text features, as well as the synergistic effect of dual cross-modal attention, a powerful deep learning framework is constructed to effectively improve the performance of expression motion unit detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial expression movement unit detection, and mainly relates to a facial expression movement unit detection method based on dual cross-modal attention. Background Art

[0002] Facial expressions are a crucial way for humans to convey emotion. To comprehensively and objectively describe facial expressions, psychologists Ekman and others developed the Facial Action Coding System (FACS). Based on anatomical principles, this system decomposes complex and diverse facial expressions into quantifiable local facial muscle movements, known as action units (AUs). These action units, when combined in different ways, can accurately depict various facial expressions, thereby revealing an individual's emotional state. Therefore, action unit detection is a research focus in computer vision and affective computing. Its goal is to automatically identify the activation state of action units in images and analyze the emotional information conveyed by facial expressions. This automated facial expression detection technology shows broad application prospects in fields such as human-computer interaction and mental health assessment.

[0003] The task of detecting facial motion units faces challenges such as insufficient labeled data and significant individual differences. Specifically: 1) Data annotation for facial motion units requires specialized knowledge and is time-consuming, resulting in a limited number of labeled samples available for model training, severely hindering model performance; 2) Facial features vary across individuals and are easily affected by factors such as lighting, posture, and occlusion, further increasing the difficulty of facial motion unit detection.

[0004] Existing facial motion unit detection methods can be mainly divided into two categories: "association-based learning" and "region-based learning", as follows:

[0005] 1) Expression movement unit detection method based on association learning:

[0006] Publication numbers CN114842542A, CN116416667A, and CN117765596A are invention patent applications titled "Facial Action Unit Recognition Method and Apparatus Based on Adaptive Attention and Spatiotemporal Correlation," "Facial Action Unit Detection Method Based on Dynamic Correlation Information Embedding," and "A Facial Action Unit Detection Model Establishment Method Based on Multi-Task Learning." These patent applications primarily address the following: capturing dependencies between AUs through the construction of a graph attention network, a dynamically updated AU correlation graph, and an adaptive spatiotemporal graph convolutional neural network. These methods utilize graph neural networks to model inter-AU correlations, but rely heavily on the label distribution in the dataset, resulting in low generalization capabilities.

[0007] 2) Expression motion unit detection method based on region learning:

[0008] Publication numbers CN117576765A, CN115862120A, and CN111626113A are invention patent applications titled "A Method for Constructing a Facial Action Unit Detection Model Based on Hierarchical Feature Alignment," "A Facial Action Unit Recognition Method and Device Decoupled from a Separable Variational Autoencoder," and "A Facial Expression Recognition Method and Device Based on Facial Action Units." These patent applications primarily utilize convolutional neural networks, attention mechanisms, and other techniques to enhance the model's ability to perceive key local information. Due to the significant differences in facial features between individuals and the rapid and subtle changes in movements in local facial regions, these methods, while improving their ability to capture key local information, are prone to introducing noise, resulting in reduced detection accuracy. Summary of the Invention

[0009] In order to solve the problem of insufficient robustness caused by excessive reliance on limited labeled datasets in the existing technology, the present invention provides an expression motion unit detection method based on dual cross-modal attention.

[0010] In order to achieve the above object, the solution of the present invention is:

[0011] A method for detecting facial expression motion units based on dual cross-modal attention, comprising:

[0012] Step 1: Obtain a multimodal expression motion unit (AU) dataset D including image data and text data;

[0013] Step 2: Build a multimodal AU detection network;

[0014] Step 3: Divide the AU dataset D into a training set and a validation set, train and test the multimodal AU detection network, and obtain a multimodal AU detection model;

[0015] Step 4: Use the multimodal AU detection model to implement AU detection.

[0016] Preferably, the step of acquiring the multimodal expression motion unit AU data set includes:

[0017] Step 1.1, obtain an image dataset V including facial images and corresponding labels;

[0018] Step 1.2, based on the Facial Action Coding System (FACS) manual, collect text descriptions of AUs to obtain a text dataset T consisting of text descriptions of AUs;

[0019] Step 1.3: Integrate the image dataset V and the text dataset T to form an AU dataset D including image and text data.

[0020] Preferably, the step of constructing the multimodal AU detection network includes:

[0021] Construct a visual encoder to extract features from image data and obtain global visual features;

[0022] Construct N independent local perception attention matrices, take global visual features as input, and obtain corresponding N local visual features;

[0023] Construct an AU internal encoder, take text data as input, and obtain word-level feature representation; perform pooling on the word-level feature representation to obtain sentence-level feature representation; input the sentence-level feature representation into the AU interactive semantic encoder to obtain text feature representation;

[0024] Using global visual features as queries and text feature representations as keys and values, a cross-modal attention mechanism is used to calculate global interaction features.

[0025] Using local visual features as queries and text feature representations as keys and values, the cross-modal attention mechanism is used to calculate local interaction features.

[0026] The global interaction features, local interaction features and local visual features are fused to obtain fused features;

[0027] Construct an AU classifier, take the fusion features as input, and obtain the corresponding prediction probability.

[0028] Preferably, the loss is calculated using a cosine similarity difference loss function:

[0029]

[0030] Where I represents the identity matrix, represents the number of words in the word-level feature representation, Represents word-level feature representation.

[0031] Preferably, the cross-modal attention mechanism is defined as:

[0032] ,

[0033] in, is the modal feature of the query, are modal features as keys and values, , , is the learnable parameter matrix, is the scaling factor.

[0034] Preferably, the step of constructing the multimodal AU detection network further includes:

[0035] Perform a global average pooling operation on N local visual features to obtain a feature set;

[0036] Construct a graph neural network, using the local visual features of each AU as the node of the graph neural network, and define the cosine similarity between any two nodes as the edge of the graph neural network;

[0037] For each node in the graph neural network, select the K neighboring nodes with the greatest similarity, aggregate the neighboring node information through graph convolution, and obtain a refined feature representation of each node;

[0038] Construct an AU classifier, take the refined feature representation of the node as input, and obtain the corresponding prediction probability.

[0039] Preferably, an asymmetric loss function with weights is used Calculate the loss:

[0040]

[0041] in, 、 and Respectively represent The predicted probability, true value and loss weight of each AU.

[0042] Preferably, The calculation formula is: ,in Indicates the first The frequency of occurrence of AUs, Indicates the first The frequency of occurrence of AUs.

[0043] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned expression motion unit detection method based on dual cross-modal attention are implemented.

[0044] The present invention further provides an electronic device, comprising:

[0045] Memory for storing computer programs;

[0046] A processor is used to implement the steps of the above-mentioned expression movement unit detection method based on dual cross-modal attention when executing the computer program.

[0047] Compared with the existing technology, the significant advantages of the present invention are as follows: the present invention realizes the deep interaction between the AU visual modality and the text modality. In particular, it obtains a refined multi-granularity visual feature representation through a visual encoder, a local perception attention matrix, and a graph convolutional network; using a multi-level encoding process, it first models the semantic association between words in the AU description, and then models the sentence-level semantic association between different AU descriptions, effectively mining the rich semantic information in the AU description and significantly enhancing the expressive power of the AU text features; a dual cross-modal attention strategy of global and local collaboration is designed to achieve deep interaction between visual and text modalities, helping the model to more comprehensively understand the complex semantic associations between visual and text modalities and enhance AU feature representation. Ultimately, by combining multi-granularity visual features and multi-level text features, as well as the synergistic effect of dual cross-modal attention, a powerful deep learning framework is constructed to effectively improve the performance of expression motion unit detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the expression motion unit detection method based on dual cross-modal attention.

[0049] Figure 2 This is a schematic diagram of the multimodal AU detection network structure, where (A) is the training phase and (B) is the testing phase. DETAILED DESCRIPTION

[0050] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. The embodiments described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, the specific process of the facial expression movement unit detection method based on dual cross-modal attention of the present invention is as follows:

[0052] 1. Data preparation stage:

[0053] 1.1 Collecting facial expression motion unit dataset:

[0054] To construct the facial motion unit (AU) dataset, we first perform face detection and alignment on the video clips containing faces in the dataset. Specifically, each frame is processed using a face detection and alignment model to ensure accurate identification and alignment of the facial regions within the image. The processed facial images are then cropped to a standard size of 256×256 pixels to unify the input dimensions. Each image is annotated according to the label file in the dataset, forming an AU dataset consisting of image data V, which provides training and validation data for the subsequent AU detection task.

[0055] 1.2 Collect and expand the description of facial expression movement units:

[0056] Based on the Facial Action Coding System (FACS) manual, text descriptions of expression movement units were collected and organized. These descriptions cover rich semantic information such as facial area, intensity, category, and interdependence related to AUs. However, the number of AU descriptions is scarce, and each AU has only one original AU description. In order to further expand and diversify AU descriptions, a large language model (e.g., DeepSeek, GPT-4) is used to generate 50 or even more large-model augmented AU descriptions for each AU, combining the original AU descriptions with carefully designed prompts, and manually screening to ensure the quality of the generated text. Finally, the text data composed of these AU descriptions is generated. and AU dataset Integrate to form a multimodal AU dataset including image and text data .

[0057] 2. Model design phase: The specific design of the two-stage model is as follows:

[0058] 2.1 The overall model is denoted as , Includes vision encoder , AU internal encoder , AU interactive encoder , Graph Convolutional Network , dual cross-modal attention module , and AU classifier C. The model input is a multimodal AU dataset , including image data and text data .

[0059] 2.2 Extracting multi-granularity visual features. First, use the visual encoder For input image data Perform feature extraction and output global visual features In order to accurately capture the subtle expressions of local facial areas, we construct parameter-independent local perception attention matrix , each AU corresponds to a local perception attention matrix. The global visual features Input into these local perception attention matrices to extract the local visual features of the corresponding AU The process can be formally expressed as:

[0060] .

[0061] Local visual features Used to perceive local facial areas, laying the foundation for subsequent AU relationship modeling and cross-modal interaction.

[0062] 2.3 Building an AU Relationship Modeling Module Based on Graph Convolutional Network This module builds a graph neural network to model the dependencies between different AU visual features, thereby enhancing the semantic expression ability of local visual features. Local visual features Perform global average pooling operation to obtain feature set ,in Indicates the Secondly, the local visual features of each AU As a node of the graph neural network, define any two nodes and The cosine similarity between is the edge of the graph neural network, thus building a graph neural network. Then, for each node ,calculate and all other nodes Similarity , find out the previous nodes with the greatest similarity and constitute the node The set of neighbor nodes , which can be expressed as When the node belong , then the node and The connection relationship between Set to 1; otherwise, when the node Not ,but Set to 0. Use the connection relationship between nodes , aggregate neighbor node information through graph convolution to obtain the Refined feature representation of nodes The process can be expressed as:

[0063]

[0064] in, is a nonlinear activation function, and represents a differentiable function of a graph convolutional network layer, Representation node and The connection relationship between them, 1 means there is a connection, 0 means there is no connection.

[0065] Import node features To the AU classifier C, get the prediction probability of each AU Specifically, an AU classifier is constructed for each AU, which consists of a Learnable vectors of the same dimension The first The predicted probability of an AU can be expressed as:

[0066]

[0067] Among them, ReLU is a nonlinear activation function. Then, an asymmetric loss function with weights is used Calculate the loss:

[0068]

[0069] in, 、 and Respectively represent The predicted probability, true value and loss weight of each AU. Used to alleviate the label imbalance problem in the data set. The calculation formula is: ,in Indicates the first The frequency of occurrence of AUs.

[0070] 2.4 Building the AU Internal Encoder , AU interactive encoder The present invention first constructs the AU internal encoder Used to model the semantic relationship between words in AU description. Input AU description into AU internal encoder , to extract word-level feature representations for each AU description ={ },in Indicates the number of words in the AU description, each Represents the feature vector of a word. To further enhance the model's ability to identify different AU text features, a difference loss function based on cosine similarity is introduced:

[0071]

[0072] Where I represents the identity matrix. This loss function effectively enhances the specificity between AU text features by forcing the similarity matrix calculated from text features to present a diagonal dominant structure, that is, the values on the off-diagonal line of the matrix approach 0, which prompts the model to learn AU description features with significant discrimination. In order to obtain sentence-level feature representation, the word-level features are pooled to obtain the sentence-level feature representation of each AU description. ,in Indicates the On this basis, in order to further integrate the information between different AU descriptions, the sentence-level features Input to AU interaction semantic encoder , to fuse semantic information from other AU descriptions, further enhance the text features of each AU, and finally obtain a robust text feature representation: ( ).

[0073] 2.5 Building a Dual Cross-Modal Attention Module Achieve accurate cross-modal interaction. This module is composed of global and local cross-modal attention working together to model multi-granularity cross-modal interaction information and enhance AU feature representation. Given the modal features as the query (Query) , as a modal feature of key and value , then the cross-modal attention mechanism can be defined as:

[0074]

[0075] in, , , is the learnable parameter matrix, is the scaling factor ( Usually take the key value dimensions).

[0076] Global cross-modal attention is used to model the semantic dependency between global visual features and AU description features. Specifically, global visual features As a query, Text features of AUs As the key and value, the cross-modal attention mechanism is used to calculate the global interaction features: This design can eliminate the interference of other AU description information and focus on establishing exclusive semantic associations between global visual features and current AU text features. Local cross-modal attention focuses on the interaction between visual and text features of a single AU. Specifically, for the AU, based on local visual features As query, AU text features As keys and values, the cross-modal attention mechanism is used to calculate local interaction features: . The local cross-modal attention module is guided by local visual features and integrates the semantic information of the AU description text that uniquely corresponds to the visual features of the AU, thereby achieving fine cross-modal feature interaction and information query. Through the collaborative work of global and local cross-modal attention, multi-granularity cross-modal information interaction and fusion are achieved: the global module captures the semantic association between each AU description and the overall face. This process can be coupled with the mutual exclusion and symbiotic relationship between AUs to achieve preliminary query of AU activation status; the local module focuses on the fine interaction between AU-specific visual features and AU description text features. This process ensures that there is no contamination of other AU features during the cross-modal information fusion process through strict one-to-one input constraints, which helps to improve the semantic fidelity of the single AU feature representation and achieve accurate query of AU activation status.

[0077] 2.6 Global Interaction Features , local interaction features and local visual features Perform feature fusion to obtain fusion features for multimodal expression motion unit detection Then, the AU classifier C described in Section 2.3 is used to calculate the Fusion features of AUs The predicted probability , and adopts an asymmetric loss function with weights Calculate the detection task loss.

[0078] 2.7 Model The training is divided into two stages: the first stage uses an asymmetric loss function Specially trained visual encoder , graph neural network G and AU classifier C to establish preliminary visual feature extraction capabilities; based on the first stage training, the second stage further introduces three modules: AU internal encoder , AU interactive encoder , dual cross-modal attention module . This stage uses the overall loss function Train the model M, where Indicates the difference loss function added in this stage. The second stage jointly optimizes the three newly introduced modules and the visual encoder And AU classifier C. Update the model parameters by gradient descent method. First, execute steps 2.1, 2.2, and 2.3 to complete the first stage of training. Then execute steps 2.4, 2.5, and 2.6 to complete the second stage of training until convergence. Parameters The update strategy is as follows:

[0079]

[0080] in Represents the learning rate.

[0081] 2.8 The above steps are unified into a two-stage deep neural network framework to achieve optimized model training.

[0082] 3. Model training phase:

[0083] 3.1 Divide the multimodal emotion recognition dataset obtained in step 1.1 into Divide into training set and validation set ;

[0084] 3.2 Training set Input to the network model designed in step 2 In the model training, batch stochastic gradient descent method is used. There are two loss functions, namely weighted asymmetric loss , difference loss . The validation set is also used during the training phase. Verify the model training effect, that is, when the model In the validation set When good AU detection results are obtained and the performance does not improve in subsequent training iterations, the model training converges and the training process is stopped;

[0085] 3.3 Final training is completed to obtain the model .

[0086] 4. Model testing phase:

[0087] 4.1 The input data is the multimodal emotion recognition dataset D = {V, T} obtained by processing similarly to step 1.1, from which the test set is divided , and remove the large model augmented AU description in the text data. The model used in the test phase is model Visual encoder in , AU internal encoder , AU interactive encoder , dual cross-modal attention module , and AU classifier .

[0088] 4.2 Test Set Input the model obtained in step 3.3 To get the test set The F1 scores of the facial expression motion unit detection results for the BP4D and DISFA datasets are shown in Tables 1 and 2, respectively. The detection results show that the multimodal facial expression motion unit detection method based on visual and textual information proposed in this paper is effective, achieving good detection performance on both the BP4D and DISFA datasets.

[0089] Table 1 Expression motion unit detection results of BP4D dataset

[0090] AU1 AU2 AU4 AU6 AU7 AU10 AU12 AU14 AU15 AU17 AU23 AU24 Avg 54.1 49.7 63.3 79.3 79.8 84.5 88.8 68.5 57.0 62.6 53.1 56.8 66.5

[0091] Table 2 Expression motion unit detection results of DISFA dataset

[0092] AU1 AU2 AU4 AU6 AU9 AU12 AU25 AU26 Avg 55.7 58.4 75.4 51.0 56.5 74.8 93.9 63.8 66.2

[0093] To address the problem of insufficient robustness caused by the existing over-reliance on limited annotated datasets, the present invention proposes a multimodal expression motion unit detection method based on visual and textual information. The key points of this method are:

[0094] 1) Multi-granularity visual feature extraction: First, a visual encoder is used to extract features from the input image to obtain global visual features that contain overall facial information. Then, to accurately capture subtle facial expressions in local regions, a set of independently parameterized local-aware attention matrices are introduced. Each attention matrix is designed for a specific AU and focuses on the local region associated with that AU. By inputting global visual features into these matrices, local visual features of the corresponding AU can be independently extracted. To further enhance the expressive power of these local visual features, a graph convolutional network is constructed to model the dependencies between AUs. In this graph neural network, the local visual features of each AU are considered nodes, and the similarities between local visual features are considered edges. For each node, the top K most similar neighboring nodes are selected and their information is aggregated through graph convolution. This fully exploits the interdependencies between AUs, ultimately obtaining a refined representation of each node's features. In this multi-granularity visual feature extraction process, global visual features provide macroscopic information, while local visual features focus on microscopic details in local facial regions. The two complement each other to jointly describe multi-granularity facial information.

[0095] 2) Multi-level Text Feature Extraction: To effectively learn robust AU text features, a multi-level text feature extraction strategy is adopted, gradually modeling from the word level to the sentence level, deeply exploring the rich semantic information in AU descriptions. Since each AU has only one original description, direct learning may lead to insufficient text feature expression. Therefore, large language models (such as DeepSeek and GPT-4) are introduced to augment the original AU descriptions, generating more descriptions for each AU. Manual screening is performed to ensure text quality, thereby improving the diversity of AU descriptions. Specifically, word-level features are first extracted through the AU internal encoder, followed by pooling to obtain sentence-level features. To improve the specificity of text features, a difference loss function based on cosine similarity is introduced. This loss function ensures significant differences between AU text features, ultimately constructing highly discriminative AU-specific semantic representations. Then, through the AU interaction encoder, text information from other AUs is further integrated to strengthen sentence-level features, improve the expressiveness of text features, and effectively alleviate the sparsity problem of AU description data.

[0096] 3) Dual Cross-Modal Attention Promotes Cross-Modal Interaction: A dual cross-modal attention strategy, combining global and local attention, deeply mines semantic information from text descriptions, promoting precise interaction between the visual and textual modalities, and ultimately achieving more nuanced and robust feature representations. In global cross-modal attention, global visual features serve as the query, while the textual features of each expression motion unit (AU) serve as the key and value. By correlating the visual information of the entire facial region with the semantic information of the corresponding AU description, the model captures the overall emotional expression of the face image and outputs a global interaction feature. Local cross-modal attention, on the other hand, uses the local visual features of a specific AU as the query, with the corresponding textual features of the AU as the key and value. It focuses on the textual information associated with that AU, accurately matching local visual features with fine-grained semantic information to capture facial expression details and query the activation state of the current AU, ultimately outputting a local interaction feature. Through this synergistic and complementary dual cross-modal attention mechanism, the model fully understands the complex semantic connections between visual and textual modalities, effectively matching AU textual definitions with global and local visual features to accurately identify the activation state of the AU.

[0097] 4) Expression motion unit detection method based on dual cross-modal attention: Figure 2As shown in (A) and (B) of Figure 1, a two-stage training model is used to perform robust facial motion unit detection. The first stage focuses on extracting multi-granular visual features: first, a visual encoder is used to extract global visual features. Second, a set of local perceptual attention matrices is used to extract local visual features for each AU. A graph convolutional network is used to model the dependencies between AUs, enhancing the expressiveness of local features. This stage uses a weighted asymmetric loss function, L_wa, to optimize the local visual features of the AUs, resulting in refined feature representations. The second stage involves multi-level text feature extraction and cross-modal interaction using dual cross-modal attention. First, word-level text features are extracted using the AU internal encoder, and sentence-level features are obtained through average pooling. Then, a discrepancy loss function, L_dis, is introduced to enhance the specificity of text features. Finally, an AU interaction encoder is used to fuse semantic information from different AUs, further strengthening sentence-level features. Under this dual cross-modal attention strategy, visual features serve as queries, and AU text features serve as keys and values. Through global and local collaboration, visual and textual information is deeply mined, achieving precise cross-modal interaction. Finally, the refined local visual features extracted in the first stage are fused with the local and global interaction features obtained in the second stage to generate the final multimodal features, which are then fed into the AU classifier for AU prediction and classification. The classification loss is calculated using the asymmetric loss L_wa from the first stage. The model uses multiple loss functions to ensure the effective fusion of multimodal features, thereby improving the performance of expression motion unit detection.

[0098] In summary, the present invention proposes a multimodal expression motion unit detection model based on visual and textual information, which realizes the deep interaction between AU visual modality and textual modality. In particular, a refined multi-granularity visual feature representation is obtained through a visual encoder, a local perception attention matrix and a graph convolutional network; a multi-level encoding process is utilized to first model the semantic association between words in the AU description, and then model the sentence-level semantic association between different AU descriptions, effectively mining the rich semantic information in the AU description and significantly enhancing the expressive power of AU text features; a global and local collaborative dual cross-modal attention strategy is designed to achieve deep interaction between visual and textual modalities, helping the model to more comprehensively understand the complex semantic associations between visual and textual modalities and enhance AU feature representation. Finally, by combining multi-granularity visual features and multi-level textual features, as well as the synergistic effect of dual cross-modal attention, a powerful deep learning framework is constructed to effectively improve the performance of expression motion unit detection.

[0099] Based on the same technical solution, the present invention also provides an electronic device, comprising:

[0100] Memory for storing computer programs;

[0101] A processor is used to implement the steps of the above-mentioned expression movement unit detection method based on dual cross-modal attention when executing the computer program.

[0102] Based on the same technical solution, the present invention also proposes a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the steps of the above-mentioned method for detecting facial movement units based on dual cross-modal attention. The computer-readable storage medium may include any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0103] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

Claims

1. A facial expression motion unit detection method based on dual cross-modal attention, characterized in that: include: Step 1: Obtain a multimodal expression motion unit (AU) dataset D including image data and text data; Step 2: Build a multimodal AU detection network; Step 3: Divide the AU dataset D into a training set and a validation set, train and test the multimodal AU detection network, and obtain a multimodal AU detection model; Step 4: Utilize the multimodal AU detection model to implement AU detection; The steps of constructing the multimodal AU detection network include: Construct a visual encoder to extract features from image data and obtain global visual features; Construct N independent local perception attention matrices, take global visual features as input, and obtain corresponding N local visual features; Construct an AU internal encoder, take text data as input, and obtain word-level feature representation; perform pooling on the word-level feature representation to obtain sentence-level feature representation; input the sentence-level feature representation into the AU interactive semantic encoder to obtain text feature representation; Using global visual features as queries and text feature representations as keys and values, a cross-modal attention mechanism is used to calculate global interaction features. Using local visual features as queries and text feature representations as keys and values, the cross-modal attention mechanism is used to calculate local interaction features. The global interaction features, local interaction features and local visual features are fused to obtain fused features; Perform a global average pooling operation on N local visual features to obtain a feature set; Construct a graph neural network, using the local visual features of each AU as the node of the graph neural network, and define the cosine similarity between any two nodes as the edge of the graph neural network; For each node in the graph neural network, select the K neighboring nodes with the greatest similarity, aggregate the neighboring node information through graph convolution, and obtain a refined feature representation of each node; Construct an AU classifier, taking the fusion features and the refined feature representation of the nodes as input, and obtain the corresponding prediction probability; Use asymmetric loss function with weights Calculate the loss of visual encoder, graph neural network and AU classifier; introduce the difference loss function based on cosine similarity , joint weighted asymmetric loss function , calculate the loss of the multimodal AU detection network : ， ， ， in, 、 and Respectively represent The predicted probability, true value and loss weight of each AU, I represents the unit matrix, represents the number of words in the word-level feature representation, Represents word-level feature representation.

2. The method according to claim 1, characterized in that The step of acquiring the multimodal expression movement unit AU data set includes: Step 1.1, obtain an image dataset V including facial images and corresponding labels; Step 1.2, based on the Facial Action Coding System (FACS) manual, collect text descriptions of AUs to obtain a text dataset T consisting of text descriptions of AUs; Step 1.3: Integrate the image dataset V and the text dataset T to form an AU dataset D including image and text data.

3. The method according to claim 1, characterized in that The cross-modal attention mechanism is defined as: ， in, is the modal feature of the query, are modal features as keys and values, , , is the learnable parameter matrix, is the scaling factor.

4. The method according to claim 1, wherein For each node in the graph neural network, we select the K neighboring nodes with the greatest similarity and aggregate the neighboring node information through graph convolution to obtain a refined feature representation of each node, including: For each node in the graph neural network, select the K neighbor nodes with the greatest similarity to form the neighbor node set of each node; By utilizing the connection relationship between nodes and aggregating neighbor node information through graph convolution, a refined feature representation of each node is obtained; the connection relationship between nodes is expressed as: ， Where, For nodes and The connection relationship between For nodes The set of neighbor nodes.

5. The method according to claim 1, wherein The calculation formula is: ,in Indicates the first The frequency of occurrence of AUs, Indicates the first The frequency of occurrence of AUs.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which implements the steps of the method according to any one of claims 1 to 5 when executed by a processor.

Images