Facial action unit recognition method and system based on multi-modal fusion and text enhancement

Abstract

The invention discloses a facial action unit recognition method and system based on multi-modal fusion and text enhancement. The method comprises the following steps: extracting a facial image attribute set by a visual language model, constructing an adaptive text cue word based on the facial image attribute set, and processing the adaptive text cue word by adopting CLIP to obtain a text semantic feature; extracting facial key point features by the facial key point detection model; the facial expression recognition model extracts facial expression features; the image semantic feature fusion module processes the facial key point features and the facial expression features to obtain multi-modal image semantic features; inputting the multi-modal image semantic features and the text semantic features into a multi-modal feature fusion module to obtain multi-modal fusion features after text information enhancement; the recognition of the facial action unit of the facial image is realized. According to the method, the complementarity of cross-modal features and the robustness of joint representation can be enhanced, and the recognition precision of facial actions in a complex scene is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and deep learning, and particularly relates to a facial action unit recognition method, a storage medium, a computer device and a system based on multimodal fusion and text enhancement. Background Art

[0002] Facial action units are the basic facial movement components defined by the Facial Action Coding System and are also the smallest units for objectively and meticulously describing facial muscle activities. Each action unit corresponds to specific facial muscle movements (such as raising eyebrows, lifting the corners of the mouth, etc.), and complex facial expressions and microexpressions can be accurately represented by combining different action units. Therefore, the current recognition methods for facial action units are widely used in fields such as mental health detection, facial reconstruction and animation, and business negotiation.

[0003] In the prior art, facial action unit recognition methods mainly include methods based on single-image features, methods based on facial multi-task learning frameworks, and methods based on self-supervised training of unlabeled images, etc. Among them, the methods based on single-image features and facial multi-task learning have become the mainstream directions and achieved remarkable progress. However, these methods still have the following deficiencies:

[0004] (1) Data limitations: The cost of obtaining data labels for facial movement unit data is relatively high, and the data set has the characteristics of small scale and insufficient diversity, resulting in limited generalization ability of the model. At the same time, it is difficult to comprehensively represent complex facial dynamics by extracting single-modal features.

[0005] (2) Insufficient cross-modal collaboration: The depth of multi-task knowledge integration is insufficient, and the interaction mechanism design of multi-task features is relatively shallow, failing to fully explore the complementary value of indirect attributes (such as environment, identity).

[0006] (3) Limitations in artificial feature design: Relying on manually designed regional feature extraction rules, it is sensitive to identity differences and environmental interference, and the generalization ability significantly decreases under complex lighting conditions, lacking robustness. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned deficiencies existing in the prior art, and provide a facial action unit recognition method based on multimodal fusion and text enhancement. This method introduces text semantic information, and hierarchically interacts the attribute text semantic features with the multimodal image semantic features to generate multimodal fusion features enhanced by text information. The multimodal fusion features enhanced by text information have the complementarity of cross-modal features and the robustness of joint representation, can improve the context reasoning ability, and effectively improve the recognition accuracy of facial actions in complex scenarios.

[0008] Meanwhile, the second purpose of the present invention is to provide a computer device.

[0009] Meanwhile, the third object of the present invention is to provide a storage medium.

[0010] Meanwhile, the fourth object of the present invention is to provide a facial action unit recognition system based on multi-modal fusion and text enhancement.

[0011] The object of the present invention is achieved by the following technical solutions:

[0012] A facial action unit recognition method based on multi-modal fusion and text enhancement, comprising the following steps:

[0013] S1. Obtain the facial image data to be recognized, perform standardized preprocessing on the facial image data to be recognized, and obtain the preprocessed facial image data;

[0014] S2. Construct and train a vision-language model, a facial key point detection model, and a facial expression recognition model to obtain the trained vision-language model, facial key point detection model, and facial expression recognition model; input the preprocessed facial image data into the trained vision-language model, facial key point detection model, and facial expression recognition model;

[0015] S3. The vision-language model extracts a set of facial image attributes from the preprocessed facial image data, constructs a structured natural language description based on the set of facial image attributes, the structured natural language description includes a task-specific description, a subject-specific description, and a combined description, constructs an adaptive text prompt based on the task-specific description, subject-specific description, and combined description, and performs vectorization representation processing on the text semantic features of the adaptive text prompt by using a text-image alignment model to obtain text semantic features; the text-image alignment model uses the text encoder of the CLIP network model;

[0016] S4. The trained facial key point detection model extracts facial key point features from the preprocessed facial image data; the facial expression recognition model extracts facial expression features from the preprocessed facial image data; construct and train a multi-scale cross-attention feature fusion model, the multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module, use the image semantic feature fusion module to perform feature fusion processing on the facial key point features and facial expression features to obtain multi-modal image semantic features; input the multi-modal image semantic features and the text semantic features obtained in step S3 into the multi-modal feature fusion module for feature fusion processing to obtain multi-modal fusion features with enhanced text information;

[0017] S5. Use a classifier network to process the multi-modal fusion features enhanced from the text information, so as to realize the recognition and classification output of the facial action units of the facial image data to be recognized; the classifier network is trained by using an MLP network model.

[0018] Preferably, the normalization preprocessing includes facial key point alignment processing and normalization size adjustment processing.

[0019] Preferably, the facial image attribute set includes an identity attribute dimension and an environmental attribute dimension. The identity attribute dimension includes age, gender, race, beard status, and glasses wearing status. The environmental attribute dimension includes background environment, lighting condition, and data source. The specific representation of the facial image attribute set is as follows:

[0020] A s ={α i ∣i = 1, 2, …, 8}, α ∈ {info id , info env}

[0021] info id ={Age, Gender, Race, Beard, Hair, Glasses}

[0022] info env ={Background, Lighting}

[0023] Among them, info id is a set of identity attributes extracted for the individual identity information in the image. The identity attributes include age, gender, race, whether there is a beard, hair length, and whether glasses are worn; info env is a set of environmental attributes extracted for the environmental information in the image. The environmental attributes include background and lighting conditions.

[0024] Preferably, in step S3, the facial image attribute set implements a data screening mechanism based on a set frequency threshold, and combines an artificial verification process to ensure the accuracy of attribute description.

[0025] Preferably, the specific process of step S2 is as follows:

[0026] S21. Obtain historical facial image data. After performing facial key point alignment, normalization size adjustment, and data augmentation processing on the historical facial image data, construct a facial image data set and divide out the corresponding training set and test set;

[0027] S22. Use the training set obtained in step S21 to uniformly train the visual language model, facial key point detection model, and facial expression recognition model respectively, and use the test set to test the visual language model, facial key point detection model, and facial expression recognition model to obtain the trained visual language model, facial key point detection model, and facial expression recognition model; the visual language model uses the GPT-4o network model, and the facial key point detection model uses the HG-Net network model; the trained facial expression recognition model uses the IResNet50 network model trained based on the ArcFace loss function.

[0028] Preferably, the specific process of step S4 is as follows:

[0029] S41. Use the trained facial key point detection model to extract facial key point features from the preprocessed facial image data; the facial key point detection model uses the HG-Net network model; the representation of the facial key point features is as follows:

[0030]

[0031] where F LM is the facial key point feature, and C and D are the number of feature channels and the feature dimension respectively;

[0032] S42. Use the trained facial expression recognition model to extract facial expression features from the preprocessed facial image data; the trained facial expression recognition model uses the IResNet50 network model trained based on the ArcFace loss function; the specific representation of the facial expression features is as follows:

[0033]

[0034] The facial expression features have the same feature shape as the facial key point features;

[0035] S43. Construct and train a multi-scale cross-attention feature fusion model. The multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module. Both the image semantic feature fusion module and the multi-modal feature fusion module use the CFT network model; the CFT network model is composed of multiple stacked Cross FusionBlocks and is configured with three feature scale levels: Large, Medium, and Small.

[0036] S44. Use the image semantic feature fusion module to perform feature fusion processing on the facial key point features and facial expression features to obtain multi-modal image semantic features; the specific representation of the multi-modal image semantic features is as follows:

[0037]

[0038] S45. Input the multi-modal image semantic features and the text semantic features obtained in step S3 into the multi-modal feature fusion module for feature fusion processing to obtain multi-modal fusion features with enhanced text information.

[0039] Preferably, the working process of the multi-modal feature fusion module in step S45 is as follows:

[0040] The multi-modal feature fusion module adopts a CFT network model; the CFT network model is composed of multiple stacked Cross Fusion Blocks and is configured with three feature scale levels, namely Large, Medium, and Small, which is used to achieve the associated fusion of features at different scales through a hierarchical cross-attention mechanism; the Cross Fusion Block includes a normalization unit Norm, a cross-attention multi-head self-attention unit CAMHSA, a random dropout unit Dropout, a multi-layer perceptron mapper unit MLP, and a residual connection unit;

[0041] First, take the multi-modal image semantic features and the text semantic features as the dual-path features to be fused and input them into the cross-attention multi-head self-attention unit CAMHSA of the Cross Fusion Block of the CFT network model, and calculate their corresponding Query, Key, and Value matrices respectively; the calculation process of the cross-attention multi-head self-attention unit CAMHSA is as follows:

[0042]

[0043] where, z i is the multi-modal image semantic feature, z j is the text semantic feature, are all learnable projection matrices;

[0044] After that, exchange the Query matrices of the multi-modal image semantic features and the text semantic features, and realize cross-path interaction through scaled dot-product attention to output the obtained fusion features; the specific process is expressed as follows:

[0045]

[0046] where, z′1 and z′2 are the fusion features obtained by the mutual operation of the Query between the image semantic feature and the text semantic feature respectively, and are used as the output of the cross-attention multi-head self-attention unit CAMHSA;

[0047] Let the input dual-channel features be x1 and x2, then the working process of a single Cross Fusion Block is as follows:

[0048] x′1 = Norm(Norm(x1 + CAMHSA(x1, x2) + MLP(Norm(x1 + CAMHSA(x1, x2))))

[0049] x′2 = Norm(Norm(x2 + CAMHSA(x2, x1) + MLP(Norm(x2 + CAMHSA(x2, x1))))

[0050] where x′1 and x′2 are the fused features obtained after the cross-operation of the input features x1 and x2 respectively;

[0051] Then, the outputs of multiple Cross Fusion Blocks of the CFT network model are superimposed to obtain the dual-channel fused features F1 and F2;

[0052] Since the CFT network model is configured with three feature scale levels: Large, Medium, and Small; the parallel superposition of the CFT network models at the Large, Medium, and Small feature scale levels is as follows:

[0053] For the input of each group of CFTs, it is specifically expressed as:

[0054] F (l) = Concat(F1, F2)

[0055] F (m) = γ(F (l) )

[0056] F (s) = γ(F (m) )

[0057] where γ is the downsampling operation;

[0058] After being processed by the three-level CFT network model, a new multi-level fused feature F′ is obtained;

[0059] F′ (s) = CFT s (F (s) )

[0060] F′ (m) = CFT m (F (m) ) + ψ(F′ (s) )

[0061] F′ = F′ (l) = CFTl (F (l) ) + ψ(F' (m) )

[0062] where ψ is an upsampling operation;

[0063] Finally, the multi-modal fusion feature representation after text information enhancement is:

[0064] F image = CFT l&e (F exp , F lm )

[0065] where F image is the multi-modal fusion feature after text information enhancement.

[0066] A computer device includes a processor and a memory for storing programs executable by the processor. When the processor executes the programs stored in the memory, it implements the above-mentioned facial action unit recognition method based on multi-modal fusion and text enhancement.

[0067] A storage medium stores a program, which when executed by a processor, implements the above-mentioned facial action unit recognition method based on multi-modal fusion and text enhancement.

[0068] A facial action unit recognition system based on multi-modal fusion and text enhancement is used to implement the above-mentioned facial action unit recognition method based on multi-modal fusion and text enhancement. The system includes:

[0069] A facial image data acquisition module, which is used to acquire facial image data to be recognized and perform standardized preprocessing on the facial image data to be recognized to obtain preprocessed facial image data;

[0070] A visual language model construction module, which is used to construct and train a visual language model to obtain a trained visual language model;

[0071] A facial key point detection model construction module, which is used to construct and train a facial key point detection model to obtain a trained facial key point detection model;

[0072] A facial expression recognition model construction module, which is used to construct and train a facial expression recognition model to obtain a trained facial expression recognition model;

[0073] A facial image attribute set extraction module, which is used to extract a facial image attribute set from the preprocessed facial image data by using the trained visual language model;

[0074] A structured natural language description module for constructing a structured natural language description based on the set of facial image attributes; the structured natural language description includes a task-specific description, a subject-specific description, and a combined description;

[0075] An adaptive text prompt module for constructing an adaptive text prompt based on the task-specific description, subject-specific description, and combined description

[0076] A text semantic feature module for performing vectorized representation processing of the text semantic features of the adaptive text prompt using a graph-text alignment model to obtain text semantic features; the graph-text alignment model uses the text encoder of the CLIP network model;

[0077] A facial key point feature extraction module for extracting facial key point features from the preprocessed facial image data using a trained facial key point detection model;

[0078] A facial expression feature extraction module for extracting facial key point features from the preprocessed facial image data using a trained facial expression recognition model;

[0079] A multi-scale cross-attention feature fusion model construction module for constructing and training a multi-scale cross-attention feature fusion model, the multi-scale cross-attention feature fusion model including an image semantic feature fusion module and a multi-modal feature fusion module;

[0080] An image semantic feature fusion module for performing feature fusion processing on the facial key point features and facial expression features to obtain multi-modal image semantic features;

[0081] A multi-modal feature fusion module for performing feature fusion processing on the multi-modal image semantic features and the text semantic features to obtain multi-modal fusion features with enhanced text information;

[0082] A classifier network module for processing the multi-modal fusion features with enhanced text information to achieve recognition and classification output of the facial action units of the facial image data to be recognized.

[0083] The present invention has the following advantages and beneficial effects compared with the prior art:

[0084] (1) The facial action unit recognition method based on multi-modal fusion and text enhancement of the present invention introduces text semantic information, hierarchically interacts the attribute text semantic features with the multi-modal image semantic features to generate multi-modal fusion features enhanced by text information. The multi-modal fusion features enhanced by text information have the complementarity of cross-modal features and the robustness of joint representation, can improve the context reasoning ability, effectively improve the recognition accuracy of facial actions in complex scenarios, and at the same time reduce the dependence of traditional methods on artificial rules and data scale.

[0085] (2) The facial action unit recognition method based on multi-modal fusion and text enhancement of the present invention introduces facial expression features and facial key point features, and performs feature fusion based on the facial expression features and facial key points to obtain multi-modal image semantic features, fully mobilizes multi-task features to cooperate, enhances the global-local joint feature expression ability, and solves the problem of insufficient detail capture by single-task models.

[0086] (3) The facial action unit recognition method based on multi-modal fusion and text enhancement of the present invention introduces a dynamic semantic adaptation mechanism, extracts a set of facial image attributes through a vision-language model, constructs a structured natural language description based on the set of facial image attributes, and thus constructs an adaptive text prompt based on the task-specific description, subject-specific description, and combined description, and performs vectorization representation processing on the text semantic features of the adaptive text prompt by using a text-image alignment model to obtain text semantic features; thus, it can effectively alleviate the cross-scene semantic deviation problem and improve the accuracy of facial action unit recognition. Brief Description of the Drawings

[0087] Figure 1 is a schematic flowchart of the facial action unit recognition method based on multi-modal fusion and text enhancement provided in Embodiment 1 of the present invention;

[0088] Figure 2 is a schematic flowchart of the working processes of each network model provided in Embodiment 1 of the present invention;

[0089] Figure 3 is a schematic diagram of the set of facial image attributes and the structured natural language description provided in Embodiment 1 of the present invention;

[0090] Figure 4 is a schematic flowchart of the working process of the facial action unit recognition system based on multi-modal fusion and text enhancement provided in Embodiment 2 of the present invention.

[0091] Figure 5 is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention;

[0092] Figure 6Schematic diagram of a storage medium provided in Embodiment 4 of the present invention; Detailed implementation manners

[0093] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0094] In order to make the objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0095] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and through specific implementation manners.

[0096] Embodiment 1

[0097] As Figures 1 - 3 shown, a facial action unit recognition method based on multi-modal fusion and text enhancement includes the following steps:

[0098] S1. Obtain the facial image data to be recognized, perform standardized preprocessing on the facial image data to be recognized, and obtain the preprocessed facial image data;

[0099] Specifically, the standardized preprocessing includes facial key point alignment processing and standardized size adjustment processing.

[0100] S2. Construct and train a visual language model, a facial key point detection model, and a facial expression recognition model to obtain the trained visual language model, facial key point detection model, and facial expression recognition model; input the preprocessed facial image data into the trained visual language model, facial key point detection model, and facial expression recognition model;

[0101] The specific process of step S2 is as follows:

[0102] S21. Obtain the historical facial image data, perform facial key point alignment, standardized size adjustment, and data augmentation processing on the historical facial image data, construct a facial image data set, and divide the corresponding training set and test set;

[0103] Specifically, the data augmentation processing includes random horizontal flipping, color jittering, and image normalization processing.

[0104] S22. Use the training set obtained in step S21 to uniformly train the visual - language model, the facial key - point detection model, and the facial expression recognition model respectively, and use the test set to test the visual - language model, the facial key - point detection model, and the facial expression recognition model, to obtain the trained visual - language model, facial key - point detection model, and facial expression recognition model; the visual - language model uses the GPT - 4o network model, the facial key - point detection model uses the HG - Net network model; the trained facial expression recognition model uses the IResNet50 network model trained based on the ArcFace loss function.

[0105] S3. The visual - language model extracts a set of facial - image attributes from the pre - processed facial - image data, and constructs a structured natural - language description based on the set of facial - image attributes. The structured natural - language description includes a task - specific description, a subject - specific description, and a combined description. An adaptive text prompt is constructed based on the task - specific description, the subject - specific description, and the combined description. The adaptive text prompt is processed by a text - image alignment model for vector - representation processing of text semantic features to obtain text semantic features; the text - image alignment model uses the text encoder of the CLIP network model.

[0106] Specifically, the set of facial - image attributes includes an identity - attribute dimension and an environment - attribute dimension. The identity - attribute dimension includes age, gender, race, beard status, and glasses - wearing status. The environment - attribute dimension includes background environment, lighting condition, and data source. The specific representation of the set of facial - image attributes is as follows:

[0107] A s ={α i ∣i = 1,2,…,8},α∈{info id ,info env}

[0108] info id ={Age,Gender,Race,Beard,Hair,Glasses}

[0109] info env ={Background,Lighting}

[0110] Among them, info id is a set of identity attributes extracted for the individual identity information in the image. The identity attributes include age, gender, race, whether there is a beard, hair length, and whether glasses are worn; info env is a set of environment attributes extracted for the environment information in the image. The environment attributes include background and lighting conditions.

[0111] Specifically, the facial image attribute set described in step S3 implements a data screening mechanism based on a set frequency threshold and combines an artificial verification process to ensure the accuracy of attribute descriptions. The screening criteria for the frequency threshold are executed as follows:

[0112] For each attribute item, count its occurrence frequency, and use the highest frequency item as the preliminary selection result.

[0113]

[0114] Calculate the frequency values of each preliminary option. If it is higher than the preset frequency threshold hyperparameter τ, it is directly selected as the final output;

[0115] If the frequency of the preliminary option does not reach the threshold, trigger a secondary image attribute evaluation process, and recalculate the attribute confidence through a multi-modal feature fusion strategy to ensure the robustness of the result.

[0116] S4. The trained facial key point detection model extracts facial key point features from the preprocessed facial image data; the facial expression recognition model extracts facial expression features from the preprocessed facial image data; construct and train a multi-scale cross-attention feature fusion model, the multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module, use the image semantic feature fusion module to perform feature fusion processing on the facial key point features and facial expression features to obtain multi-modal image semantic features; input the multi-modal image semantic features and the text semantic features obtained in step S3 into the multi-modal feature fusion module for feature fusion processing to obtain multi-modal fusion features with enhanced text information;

[0117] The specific process of step S4 is as follows:

[0118] S41. Use the trained facial key point detection model to extract facial key point features from the preprocessed facial image data; the facial key point detection model uses the HG-Net network model; the representation of the facial key point features is as follows:

[0119]

[0120] where F LM is the facial key point feature, and C and D are the number of feature channels and feature dimensions respectively;

[0121] S42. Use the trained facial expression recognition model to extract facial expression features from the preprocessed facial image data; the trained facial expression recognition model uses the IResNet50 network model trained based on the ArcFace loss function; the specific representation of the facial expression features is as follows:

[0122]

[0123] The characteristic shape of the facial expression feature is the same as that of the facial key point feature;

[0124] S43. Construct and train a multi-scale cross-attention feature fusion model. The multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module. Both the image semantic feature fusion module and the multi-modal feature fusion module adopt a CFT network model. The CFT network model is composed of multiple stacked Cross Fusion Blocks and is configured with three feature scale levels: Large, Medium, and Small;

[0125] S44. Use the image semantic feature fusion module to perform feature fusion processing on the facial key point feature and the facial expression feature to obtain a multi-modal image semantic feature. The multi-modal image semantic feature is specifically represented as follows:

[0126]

[0127] Specifically, in this embodiment, the structure of the image semantic feature fusion module is the same as that of the multi-modal feature fusion module, and the working principle is also the same;

[0128] S45. Input the multi-modal image semantic feature and the text semantic feature obtained in step S3 into the multi-modal feature fusion module for feature fusion processing to obtain a multi-modal fusion feature with enhanced text information.

[0129] Specifically, the working process of the multi-modal feature fusion module in step S45 is as follows:

[0130] The multi-modal feature fusion module adopts a CFT network model. The CFT network model is composed of multiple stacked Cross Fusion Blocks and is configured with three feature scale levels: Large, Medium, and Small. This setting is used to realize the associated fusion of features at different scales through a hierarchical cross-attention mechanism. The Cross Fusion Block includes a normalization unit Norm, a cross-attention multi-head self-attention unit CAMHSA, a random dropout unit Dropout, a multi-layer perceptron mapper unit MLP, and a residual connection unit. This setting changes the original multi-head attention module MHSA of the CFT model so that it can input two-way features to be fused at the same time;

[0131] First, take the multi-modal image semantic features and text semantic features as the two-way features to be fused and input them into the Cross Fusion Block of the CFT network model into the Cross Attention Multi-Head Self-Attention Unit (CAMHSA), and calculate their corresponding Query, Key, and Value matrices respectively. The calculation process of the Cross Attention Multi-Head Self-Attention Unit (CAMHSA) is as follows:

[0132]

[0133] where z i is the multi-modal image semantic feature, and z j is the text semantic feature, both are learnable projection matrices;

[0134] After that, swap the Query matrices of the multi-modal image semantic features and text semantic features, and achieve cross-path interaction through scaled dot-product attention to output the obtained fused features. The specific process is as follows:

[0135]

[0136] where z′1 and z′2 are the fused features obtained by the mutual operation of the Query between the image semantic feature and the text semantic feature, and are used as the output of the Cross Attention Multi-Head Self-Attention Unit (CAMHSA);

[0137] Let the input two-way features be x1 and x2, then the working process of a single Cross Fusion Block is as follows:

[0138] x′1 = Norm(Norm(x1 + CAMHSA(x1, x2)) + MLP(Norm(x1 + CAMHSA(x1, x2))))

[0139] x′2 = Norm(Norm(x2 + CAMHSA(x2, x1)) + MLP(Norm(x2 + CAMHSA(x2, x1))))

[0140] where x′1 and x′2 are the fused features obtained by the mutual operation of the input features x1 and x2 respectively;

[0141] Then, stack the outputs of multiple Cross Fusion Blocks of the CFT network model to obtain the two-way fused features F1 and F2;

[0142] Considering the common requirements of facial action units for both overall and local facial information, in this embodiment, CFT modules of different scales are stacked in parallel to obtain different fusion results of the input features at different scales, so as to enhance the robustness of the model; the CFT network model is configured with three feature scale levels: Large, Medium, and Small; the process of stacking the CFT network models of the Large, Medium, and Small feature scale levels in parallel is as follows:

[0143] For the input of each group of CFTs, it is specifically expressed as:

[0144] F (l) = Concat(F1, F2)

[0145] F (m) = γ(F (l) )

[0146] F (s) = γ(F (m) )

[0147] where γ is the downsampling operation;

[0148] After being processed by the three-level CFT network model, a new multi-level fusion feature F' is obtained;

[0149] F' (s) = CFT s (F (s) )

[0150] F' (m) = CFT m (F (m) ) + ψ(F' (s) )

[0151] F' = F' (l) = CFT l (F (l) ) + ψ(F' (m) )

[0152] where ψ is the upsampling operation;

[0153] Finally, the multi-modal fusion feature after text information enhancement is expressed as:

[0154] F image = CFT l&e (F exp , F lm )

[0155] where F image is the multi-modal fusion feature after text information enhancement.

[0156] S5. Use a classifier network to process the multi-modal fusion features enhanced by the text information, and realize the recognition and classification output of the facial action units of the facial image data to be recognized; the classifier network is trained by using an MLP network model.

[0157] Embodiment 2

[0158] As Figure 4 shown, a facial action unit recognition system based on multi-modal fusion and text enhancement is used to implement the facial action unit recognition method based on multi-modal fusion and text enhancement described in Embodiment 1. The system includes:

[0159] A facial image data acquisition module, configured to acquire facial image data to be recognized, and perform standardized preprocessing on the facial image data to be recognized to obtain preprocessed facial image data;

[0160] A visual language model construction module, configured to construct and train a visual language model to obtain a trained visual language model; the visual language model uses a GPT-4o network model.

[0161] A facial key point detection model construction module, configured to construct and train a facial key point detection model to obtain a trained facial key point detection model; the facial key point detection model uses an HG-Net network model;

[0162] A facial expression recognition model construction module, configured to construct and train a facial expression recognition model to obtain a trained facial expression recognition model;

[0163] Specifically, in this embodiment, the trained facial expression recognition model uses an IResNet50 network model trained based on an ArcFace loss function;

[0164] A facial image attribute set extraction module, configured to extract a facial image attribute set from the preprocessed facial image data by using the trained visual language model;

[0165] A structured natural language description module, configured to construct a structured natural language description based on the facial image attribute set; the structured natural language description includes a task-specific description, a subject-specific description, and a combined description;

[0166] An adaptive text prompt module, configured to construct an adaptive text prompt based on the task-specific description, the subject-specific description, and the combined description

[0167] A text semantic feature module, which is used to perform vectorized representation processing of text semantic features on the adaptive text prompt using a text-image alignment model to obtain text semantic features; the text-image alignment model uses the text encoder of the CLIP network model.

[0168] A facial key point feature extraction module, which is used to extract facial key point features from the preprocessed facial image data using a trained facial key point detection model.

[0169] A facial expression feature extraction module, which is used to extract facial key point features from the preprocessed facial image data using a trained facial expression recognition model.

[0170] A multi-scale cross-attention feature fusion model construction module, which is used to construct and train a multi-scale cross-attention feature fusion model, and the multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module.

[0171] An image semantic feature fusion module, which is used to perform feature fusion processing on the facial key point features and facial expression features to obtain multi-modal image semantic features; both the image semantic feature fusion module and the multi-modal feature fusion module use the CFT network model.

[0172] A multi-modal feature fusion module, which is used to perform feature fusion processing on the multi-modal image semantic features and text semantic features to obtain multi-modal fusion features with enhanced text information.

[0173] A classifier network module, which is used to process the multi-modal fusion features with enhanced text information to realize the recognition and classification output of the facial action units of the facial image data to be recognized.

[0174] Specifically, in this embodiment, the classifier network is trained using the MLP network model.

[0175] Embodiment 3

[0176] As Figure 5 shown, this embodiment provides a computer device, which includes a processor 102, a memory, an input device 103, a display 104, and a network interface 105 connected through a system bus 101. Among them, the processor 102 is used to provide computing and control capabilities, the memory includes a non-volatile storage medium 106 and an internal memory 107. The non-volatile storage medium 106 stores an operating system, a computer program, and a database. The internal memory 107 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium 106. When the computer program is executed by the processor 102, the facial action unit recognition method described in Embodiment 1 is implemented.

[0177] Example 4

[0178] As Figure 6 shown, this embodiment provides a storage medium storing a program, which, when executed by a processor, implements the facial action unit recognition method based on multimodal fusion and text enhancement described in Embodiment 1.

[0179] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0180] In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in this embodiment, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable storage medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0181] The above computer-readable storage medium can be written in one or more programming languages or combinations thereof for executing the computer program of this embodiment. The above programming languages include object-oriented programming languages such as Java, Python, C++, and also include conventional procedural programming languages such as the C language or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0182] In summary, the facial action unit recognition method based on multi-modal fusion and text enhancement of the present invention introduces text semantic information, hierarchically interacts the attribute text semantic features with the multi-modal image semantic features, and generates multi-modal fusion features enhanced by text information. The multi-modal fusion features enhanced by text information have the complementarity of cross-modal features and the robustness of joint representation, can improve the context reasoning ability, effectively improve the recognition accuracy of facial actions in complex scenarios, and at the same time reduce the dependence of traditional methods on artificial rules and data scale. In addition, the present invention introduces facial expression features and facial key point features, and performs feature fusion based on the facial expression features and facial key points to obtain multi-modal image semantic features, fully mobilizes multi-task features to cooperate, enhances the global-local joint feature expression ability, and solves the problem of insufficient detail capture of single-task models. In addition, the present invention also introduces a dynamic semantic adaptation mechanism, extracts a set of facial image attributes through a vision-language model, constructs a structured natural language description based on the set of facial image attributes, and thus constructs an adaptive text prompt based on the task-specific description, subject-specific description, and combined description, and performs vectorization representation processing on the adaptive text prompt by using a text-image alignment model to obtain text semantic features; thereby effectively alleviating the cross-scene semantic deviation problem and further improving the accuracy of facial action unit recognition.

[0183] The above specific implementation manners are the preferred embodiments of the present invention and cannot limit the present invention. Any other changes or other equivalent replacement methods made without departing from the technical solution of the present invention are included in the protection scope of the present invention.

Claims

1. A facial action unit recognition method based on multimodal fusion and text enhancement, characterized in that, It includes the following steps: S1. Obtain the facial image data to be recognized, perform standardized preprocessing on the facial image data to be recognized, and obtain the preprocessed facial image data; S2. Construct and train a visual language model, a facial key point detection model, and a facial expression recognition model to obtain the trained visual language model, facial key point detection model, and facial expression recognition model; input the preprocessed facial image data into the trained visual language model, facial key point detection model, and facial expression recognition model; S3. The visual language model extracts a facial image attribute set from the preprocessed facial image data, constructs a structured natural language description based on the facial image attribute set, the structured natural language description includes a task-specific description, a subject-specific description, and a combined description, constructs an adaptive text prompt based on the task-specific description, subject-specific description, and combined description, and performs vectorization representation processing on the text semantic features of the adaptive text prompt using a text-image alignment model to obtain text semantic features; the text-image alignment model uses the text encoder of the CLIP network model; S4. The trained facial key point detection model extracts facial key point features from the preprocessed facial image data; The facial expression recognition model extracts facial expression features from the preprocessed facial image data; Construct and train a multi-scale cross-attention feature fusion model, the multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module, use the image semantic feature fusion module to perform feature fusion processing on the facial key point features and facial expression features to obtain multi-modal image semantic features; input the multi-modal image semantic features and the text semantic features obtained in step S3 into the multi-modal feature fusion module for feature fusion processing to obtain multi-modal fusion features with enhanced text information; S5. Use a classifier network to process the multi-modal fusion features with enhanced text information to realize the recognition and classification output of the facial action units of the facial image data to be recognized; the classifier network is trained using an MLP network model.

2. The facial action unit recognition method based on multi-modal fusion and text enhancement according to claim 1, characterized in that The standardized preprocessing includes facial key point alignment processing and standardized size adjustment processing.

3. According to the method for recognizing facial action units based on multi-modal fusion and text enhancement described in claim 1, the facial image attribute set includes an identity attribute dimension and an environmental attribute dimension, the identity attribute dimension includes age, gender, race, beard status, and glasses wearing status, the environmental attribute dimension includes background environment, lighting condition, and data source, and the specific representation of the facial image attribute set is as follows: A s = {α i | i = 1, 2, …, 8}, α ∈ {info id , info env} info id = {Age, Gender, Race, Beard, Hair, Glasses} info env = {Background, Lighting} Among them, info id A set of identity attributes extracted for the individual identity information in the image, where the identity attributes include age, gender, race, whether there is a beard, hair length, and whether glasses are worn; info env A set of environmental attributes extracted for the environmental information in the image, where the environmental attributes include background and lighting conditions.

4. According to the method for recognizing facial action units based on multi-modal fusion and text enhancement described in claim 1, in step S3, the facial image attribute set implements a data screening mechanism based on a set frequency threshold and combines an artificial verification process to ensure the accuracy of the attribute description.

5. The facial action unit recognition method based on multi-modal fusion and text enhancement according to claim 1, the specific process of step S2 is as follows: S21. Obtain historical facial image data. After performing facial key point alignment, standardized size adjustment, and data enhancement processing on the historical facial image data, construct a facial image dataset and divide it into corresponding training sets and test sets; S22. Use the training sets obtained in step S21 to uniformly train the visual language model, facial key point detection model, and facial expression recognition model respectively, and use the test sets to test the visual language model, facial key point detection model, and facial expression recognition model to obtain the trained visual language model, facial key point detection model, and facial expression recognition model; the visual language model uses the GPT-4o network model, and the facial key point detection model uses the HG-Net network model; the trained facial expression recognition model uses the IResNet50 network model trained based on the ArcFace loss function.

6. The facial action unit recognition method based on multi-modal fusion and text enhancement according to claim 1, the specific process of step S4 is as follows: S41. Extract facial key-point features from the preprocessed facial image data using the trained facial key-point detection model; The facial key point detection model uses the HG-Net network model; the representation of the facial key point features is as follows: Among them, F LM is the facial key point feature, and C and D are the number of feature channels and the feature dimension respectively; S42. Use the trained facial expression recognition model to extract facial expression features from the preprocessed facial image data; the trained facial expression recognition model uses the IResNet50 network model trained based on the ArcFace loss function; the specific representation of the facial expression features is as follows: The feature shape of the facial expression features is the same as that of the facial key point features; S43. Construct and train a multi-scale cross-attention feature fusion model. The multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module. Both the image semantic feature fusion module and the multi-modal feature fusion module use the CFT network model; the CFT network model is composed of multiple stacked Cross Fusion Blocks and is configured with three feature scale levels: Large, Medium, and Small; S44. Use the image semantic feature fusion module to perform feature fusion processing on the facial key point features and facial expression features to obtain multi-modal image semantic features; the specific representation of the multi-modal image semantic features is as follows: S45. Input the multi-modal image semantic features and the text semantic features obtained in step S3 into the multi-modal feature fusion module for feature fusion processing to obtain multi-modal fusion features with enhanced text information.

7. The working process of the multi-modal feature fusion module in step S45 of the facial action unit recognition method based on multi-modal fusion and text enhancement according to claim 6 is as follows: The multi-modal feature fusion module adopts a CFT network model; the CFT network model is composed of multiple stacked CrossFusion Blocks, and is configured with three feature scale levels: Large, Medium, and Small. This setting is used to achieve the associated fusion of features at different scales through a hierarchical cross-attention mechanism; the Cross Fusion Block includes a normalization unit Norm, a cross-attention multi-head self-attention unit CAMHSA, a random dropout unit Dropout, a multi-layer perceptron mapper unit MLP, and a residual connection unit; First, the multi-modal image semantic features and text semantic features are input as two-way features to be fused into the cross-attention multi-head self-attention unit CAMHSA of the Cross Fusion Block of the CFT network model, and their corresponding Query, Key, and Value matrices are calculated respectively; the calculation process of the cross-attention multi-head self-attention unit CAMHSA is as follows: Among them, z i is the multi-modal image semantic feature, z j is the text semantic feature, are both learnable projection matrices; After that, the Query matrices of the multi-modal image semantic features and text semantic features are exchanged, and cross-path interaction is realized through scaled dot-product attention, and the obtained fused features are output; the specific process is shown as follows: Among them, z′1 and z′2 are the fused features obtained after the mutual operation of the Query between the image semantic features and text semantic features, and are used as the output of the cross-attention multi-head self-attention unit CAMHSA; Let the input two-way features be x1 and x2, then the working process of a single Cross Fusion Block is as follows: x′1 = Norm(Norm(x1 + CAMHSA(x1, x2) + MLP(Norm(x1 + CAMHSA(x1, x2)))) x′2 = Norm(Norm(x2 + CAMHSA(x2, x1) + MLP(Norm(x2 + CAMHSA(x2, x1)))) Among them, x′1 and x′2 are the fused features obtained after the mutual operation of the input features x1 and x2; Then, the outputs of multiple Cross Fusion Blocks of the CFT network model are stacked to obtain the two-way fused features F1 and F2; Since the CFT network model is configured with three feature scale levels: Large, Medium, and Small; the CFT network models of the Large, Medium, and Small feature scale levels are stacked in parallel, and the process is as follows: For the input of each group of CFTs, it is specifically expressed as: F (l) = Concat(F1,F2) F (m) = γ9f (l) ) F (s) = γ(F (m) ) Among them, γ is a downsampling operation; After being processed by the three-level CFT network model, a new multi-level fused feature F′ is obtained; F′ (s) = CFT s 9F (s) ) F′ (m) = CFT m (F (m) ) + ψ(F′ (s) ) F′ = F′ (l) = CFT l (F (l) ) + ψ(F′ (m) ) Among them, ψ is an upsampling operation; Finally, the multi-modal fused feature after text information enhancement is expressed as: F image = CFT l&e (F exp , F lm ) Among them, F image is the multi-modal fusion feature after text information enhancement.

8. A computer device, comprising a processor and a memory for storing processor-executable programs, characterized in that, When the processor executes the program stored in the memory, it implements the facial action unit recognition method based on multi-modal fusion and text enhancement according to any one of claims 1-7.

9. A storage medium stores a program, characterized in that, When the described program is executed by a processor, it implements the facial action unit recognition method based on multi-modal fusion and text enhancement described in any one of claims 1-7.

10. A facial action unit recognition system based on multi-modal fusion and text enhancement, which is used to implement the facial action unit recognition method based on multi-modal fusion and text enhancement according to any one of claims 1-7, characterized in that, It includes: A facial image data acquisition module, configured to acquire facial image data to be recognized, and perform standardized preprocessing on the facial image data to be recognized to obtain preprocessed facial image data; A visual language model construction module, configured to construct and train a visual language model to obtain a trained visual language model; A facial key point detection model construction module, configured to construct and train a facial key point detection model to obtain a trained facial key point detection model; A facial expression recognition model construction module, configured to construct and train a facial expression recognition model to obtain a trained facial expression recognition model; A facial image attribute set extraction module, configured to extract a facial image attribute set from the preprocessed facial image data by using the trained visual language model; A structured natural language description module, configured to construct a structured natural language description based on the facial image attribute set; the structured natural language description includes a task-specific description, a subject-specific description, and a combined description; An adaptive text prompt module, configured to construct an adaptive text prompt based on the task-specific description, the subject-specific description, and the combined description A text semantic feature module, configured to perform vectorized representation processing of text semantic features on the adaptive text prompt by using a text-image alignment model to obtain text semantic features; the text-image alignment model uses the text encoder of the CLIP network model; A facial key point feature extraction module, configured to extract facial key point features from the preprocessed facial image data by using the trained facial key point detection model; A facial expression feature extraction module, configured to extract facial key point features from the preprocessed facial image data by using the trained facial expression recognition model; A multi-scale cross-attention feature fusion model construction module, configured to construct and train a multi-scale cross-attention feature fusion model, and the multi-scale cross-attention feature fusion model includes an image semantic feature fusion module and a multi-modal feature fusion module; An image semantic feature fusion module, configured to perform feature fusion processing on the facial key point features and the facial expression features to obtain multi-modal image semantic features; A multi-modal feature fusion module, configured to perform feature fusion processing on the multi-modal image semantic features and the text semantic features to obtain multi-modal fusion features with enhanced text information; A classifier network module, configured to process the multi-modal fusion features with enhanced text information to implement recognition and classification output of facial action units of the facial image data to be recognized.

Images