Multi-modal facial expression recognition method based on dynamic prompt learning

Abstract

The invention relates to a multi-modal facial expression recognition method based on dynamic prompt learning, belongs to the technical field of image and point cloud processing in computer vision, solves the problems of limited model view angle adaptation range and low heterogeneous modal feature cooperation efficiency in the prior art, and comprises the following steps: S1, preparing an image and point cloud expression recognition data set; s2, constructing a multi-modal expression recognition model which comprises an image branch, a point cloud branch, a cross-modal prompt fusion module and a full connection layer; s3, a double-level supervision mechanism is established to train the multi-modal expression recognition model, the trained multi-modal expression recognition model is obtained, and the double-level supervision mechanism comprises feature distance loss of a calculation clustering level and prediction classification loss of a sample level; and S4, inputting the to-be-recognized face image data and the point cloud data into the trained multi-mode expression recognition model for face expression recognition to obtain an expression recognition result.

Description

Technical Field

[0001] The present invention relates to the technical field of image and point cloud processing in computer vision, and in particular to a multimodal facial expression recognition method based on dynamic prompt learning. Background Art

[0002] Using computer technology to analyze facial features has always been an important application in the field of computer vision. Because facial expressions can reflect a person's inner emotions, facial expression recognition has become a key research direction in the field of facial analysis. This field has been committed to promoting the development of facial expression recognition methods. Existing expression recognition methods based on two-dimensional image data have been shown to have technical drawbacks such as illumination sensitivity (error fluctuations of up to ±40%) and strong posture dependence (profile recognition accuracy drops by more than 60%). Their recognition accuracy is typically below 70% in complex scenarios. Traditional three-dimensional recognition methods rely on manually designed low-order feature descriptors such as curvature features and normal vector features, which are unable to capture the complex, high-level details in three-dimensional facial data.

[0003] In recent years, the field has proposed the fusion of two-dimensional and three-dimensional data for expression recognition, which can combine the advantages of the two modalities, make up for their respective limitations, and improve overall performance. The deep learning-based DF-CNN projects three-dimensional data into geometric maps, such as depth maps, normal maps, and curvature maps, and then fuses them with two-dimensional texture maps for classification. However, projecting 3D data into such geometric representations often fails to capture complete 3D details, resulting in the loss of key shape features. In addition, relying solely on frontal view geometry limits the ability of deep networks to learn from multiple angles, ignoring the diverse and robust information provided by three-dimensional data at different viewing angles. The misalignment and heterogeneity of two-dimensional and three-dimensional representations hinder the effective collaboration of the two modalities, making it difficult for the model to fully utilize the complementarity of texture features and shape information. Summary of the Invention

[0004] In view of the above-mentioned problems of the prior art, the purpose of the present invention is to propose a multimodal facial expression recognition method based on dynamic prompt learning, which uses a cross-modal prompt fusion module located between each layer of the network to realize cross-modal adaptation and multimodal fusion operations. In addition, in view of the characteristics of fine-grained expression differences in expression recognition tasks, a two-level supervision mechanism combined with text prior knowledge is designed. The present invention can more effectively deal with challenging test samples and has good robustness in all categories. The present invention overcomes the technical defects of existing two-dimensional projection fusion methods, such as the loss of three-dimensional features, the limited adaptation range of the model perspective, and the low efficiency of the collaboration of heterogeneous modal features.

[0005] According to an embodiment of the present invention, a multimodal facial expression recognition method based on dynamic prompt learning is provided, comprising:

[0006] S1: Prepare image and point cloud expression recognition datasets, perform data preprocessing on the received raw image data and point cloud data, and perform data partitioning. The partitioned datasets include training set, validation set, and test set.

[0007] S2: Construct a multimodal expression recognition model, including an image branch, a point cloud branch, a cross-modal cue fusion module, and a fully connected layer. The image branch is used to extract expression-related texture features from the input image data, and the point cloud branch is used to extract expression-related geometric features from the input point cloud data. The image branch and the point cloud branch interact through the cross-modal cue fusion module, and the expression classification results are obtained through the fully connected layer.

[0008] S3: Establish a dual-level supervision mechanism to train the multimodal expression recognition model and obtain a trained multimodal expression recognition model. The dual-level supervision mechanism includes calculating the feature distance loss at the cluster level and the prediction classification loss at the sample level.

[0009] S4: Input the facial image data and point cloud data to be recognized into the trained multimodal expression recognition model to perform facial expression recognition and obtain the expression recognition result.

[0010] Optionally, S2 specifically includes:

[0011] S2.1: Establish an image branch, which consists of K transformer blocks, where the input image is divided into N fixed-size image blocks and projected into block embedding E0 as the initial block embedding of the input image branch, and construct a learnable classification label The input sequence is attached to the transformer block as the initial classification label, and is processed successively by each transformer block of the image branch to obtain the output K block embeddings;

[0012] S2.2: Construct K cross-modal cue fusion modules, each including a cross-modal adapter and a multimodal fusion part. Each cross-modal cue fusion module is placed before the input of each layer of the image branch and the point cloud branch to achieve multi-level fusion.

[0013] S2.3: Establish a point cloud branch. The point cloud branch includes K transformer blocks. The input point cloud is divided into M point cloud blocks and projected into a block embedding G0, which is used as the initial block embedding of the input point cloud branch. The block embeddings G1, G2…G are successively processed by each transformer block of the point cloud branch. i+1 ,…G K ;

[0014] S2.4: Establish a fully connected layer to pool the block embeddings obtained after the K transformer blocks passing through the point cloud branch and connect them with the classification labels to form features and classification prediction results for expression recognition.

[0015] Optionally, S2.1 specifically includes:

[0016] Establish an image branch, which includes K transformer blocks The input image is divided into N fixed-size image blocks and projected into block embeddings As the initial block embedding of the input image branch, where D1 is the feature dimension;

[0017] Building learnable classification labels The input sequence is appended to the transformer block as an initial classification tag to aggregate classification-related information;

[0018] Each transformer block in the image branch receives the block embedding and classification label output by the previous transformer block, and processes it to obtain expression-related texture features including the block embedding and classification label:

[0019]

[0020] Among them, [·,·] represents the connection operation, E i is the block embedding of the input to the i+1th transformer block, E i+1 is the block embedding output by the i+1th transformer block, is the classification label of the input to the i+1th transformer block, is the classification label output by the i+1th transformer block;

[0021] The block embeddings and classification labels output by each transformer block of the image branch are provided to the next transformer block as input, and the block embeddings output by each transformer block are provided to the corresponding cross-modal cue fusion module;

[0022] The image branch is processed by each transformer block in turn to obtain the block embedding E1, E2…E i+1 ,…E K , and classification tags

[0023] Optionally, S2.2 specifically includes:

[0024] A cross-modal adapter is established, including a convolutional layer, an up-projection linear layer, and a down-projection linear layer in the image block dimension, to perform cross-modal adaptation operations. The block embedding received from the image branch is processed to obtain the latent code, which is expressed as:

[0025] E i ′ =C i (E i )

[0026]

[0027] Among them, E i ′ represents the block embedding after changing the block dimension, C i Represents the convolutional layer of the i-th image block dimension, A i represents the hidden code output by the i-th cross-modal adapter, is the i-th up-projection linear layer, is the i-th down-projection linear layer, GELU(·) represents the activation function, i is the index of the cross-modal adapter and i = 0, 1, ... K-1;

[0028] Establish a multimodal fusion part to process the provided learnable prompt markup and combine it with the latent code output by each cross-modal adapter to obtain an updated fused prompt markup that incorporates modality information:

[0029]

[0030] Among them, P i represents the i-th group of learnable hint tags and represents the updated fusion prompt mark of the i-th group of multimodal fusion parts, D2 is the feature dimension, and T represents the number of prompt marks included in each group of learnable prompt marks;

[0031] The updated fused cue labels output by each cross-modal cue fusion module Provided to the corresponding transformer blocks in the point cloud branch respectively.

[0032] Optionally, S2.3 specifically includes:

[0033] Establish a point cloud branch, which includes K transformer blocks The input point cloud is divided into M point cloud blocks and projected into block embeddings As the initial block embedding of the input point cloud branch;

[0034] Building learnable classification labels The input sequence of the transformer block attached to the point cloud branch as the initial classification label;

[0035] Each transformer block of the point cloud branch receives the updated fusion hint tag from the corresponding cross-modal hint fusion module, as well as the block embedding and classification tag from the previous transformer block for processing:

[0036]

[0037] Among them, G i is the block embedding of the input to the i+1th transformer block, G i+1 is the block embedding output by the i+1th transformer block, is the classification label of the input to the i+1th transformer block, is the classification label output by the i+1th transformer block, is the interactive prompt tag output by the i+1th transformer block;

[0038] After each transformer block of the point cloud branch is processed successively, the final block embedding G is obtained K , interactive prompt mark and classification tags

[0039] Optionally, S2.4 specifically includes:

[0040] A fully connected layer is built to pool the block embeddings and interaction cue tags output by the last transformer block of the point cloud branch, and concatenate them with the classification tags output by the last transformer block of the point cloud branch to form features for expression recognition:

[0041]

[0042] Among them, Pooling(·) represents the maximum pooling layer, Indicates the interactive prompt mark output by the point cloud branch at the end, G K represents the block embedding of the last output of the point cloud branch, Represents the classification label of the final output of the point cloud branch;

[0043] The final classification prediction results are obtained based on the features of expression recognition:

[0044]

[0045] Here, Head(·) represents the classification head consisting of two fully connected layers.

[0046] Optionally, S3 specifically includes:

[0047] S3.1: Establish a cluster-level feature distance loss, which combines the center loss and the discriminant distribution-independent loss. The center loss minimizes the intra-class variance by pulling sample features toward their respective class centers, while the discriminant distribution-independent loss enhances inter-class discrimination by pushing features away from the centers of other classes.

[0048] S3.2: Establish sample-level prediction classification losses, including cross-entropy loss and weighted binary cross-entropy loss;

[0049] S3.3: The multimodal expression recognition model is trained through the cluster-level feature distance loss and the sample-level prediction classification loss to obtain a trained multimodal expression recognition model.

[0050] Optionally, S3.1 specifically includes:

[0051] Establish cluster-level feature distance loss:

[0052]

[0053] Among them, L Center represents the center loss, L DDA represents the discriminant distribution-independent loss, λ and γ are hyperparameters, p represents the sample index and p = 1, 2…B, B represents the sample batch size, f p represents the depth feature of the p-th sample, Y p represents the sample label of the p-th sample, Represents the sample label Y p The feature center of the category, c j Represents the feature center of other categories, j represents the index of the feature center of other categories and j = 1, 2...N c , N c Refers to the preset types of expressions that need to be classified.

[0054] Optionally, S3.2 specifically includes:

[0055] Judge and classify samples into simple samples and complex samples;

[0056] For simple samples, a naive cross entropy loss is applied to calculate the sample-level prediction classification loss;

[0057] For complex samples, binary cross entropy loss is calculated for each class to calculate the sample-level prediction classification loss.

[0058] Optionally, judging and classifying the sample includes:

[0059] Samples that meet both of the following conditions are judged and classified as simple samples: (1) Classification prediction results of samples and the true label Y p Consistent, (2) the predicted value of the expression type with the highest probability in the classification prediction results of the sample The predicted value of the expression type with the second highest probability Higher than the set threshold τ, that is p represents the sample index and p=1,2…B, B represents the sample batch size;

[0060] Samples that do not meet both of the above conditions are judged and classified as complex samples.

[0061] Compared with the prior art, the multimodal facial expression recognition method based on dynamic prompt learning provided by the embodiments of the present invention has at least the following beneficial effects:

[0062] (1) The two-dimensional and three-dimensional expression recognition method based on dynamic cross-modal fusion of prompt learning makes up for the defects of the existing algorithm's three-dimensional feature loss and limited model perspective adaptation range, and effectively improves the model's ability to mine multimodal features.

[0063] (2) The present invention provides a cross-modal cue fusion module, which is placed between each layer of the dual-branch network of the image branch and the point cloud branch. It realizes the interactive fusion of multimodal information through cross-modal adaptation and multimodal fusion operations, making up for the deficiency of existing algorithms in fully utilizing the complementarity of texture features and shape information.

[0064] (3) The present invention provides a dual-level supervision mechanism, which constrains the feature distribution at the cluster level and the prediction output at the sample level, thereby improving the test accuracy of the model on difficult categories and difficult samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention can be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0066] Figure 1 The present invention provides a flowchart of a multimodal facial expression recognition method based on dynamic prompt learning according to an embodiment of the present invention.

[0067] Figure 2 A schematic diagram of the architecture of a multimodal expression recognition model of a multimodal facial expression recognition method based on dynamic prompt learning provided according to an embodiment of the present invention.

[0068] Figure 3 Schematic diagram of the architecture of a cross-modal cue fusion module of a multimodal facial expression recognition method based on dynamic cue learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0070] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0071] A multimodal facial expression recognition method based on dynamic prompt learning according to an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0072] like Figures 1 to 3 As shown, a multimodal facial expression recognition method based on dynamic prompt learning provided in accordance with an embodiment of the present invention includes the following steps.

[0073] S1: Prepare image and point cloud expression recognition datasets. Preprocess the raw image and point cloud data and partition the data into training, validation, and test sets. The training and validation sets are used for model training. The training set is used to train the model, and the validation set is used to select the optimal model. The test set can be used to test the performance of the selected model.

[0074] Image and point cloud expression recognition datasets can be obtained from 3D scanners and color digital cameras in a controlled laboratory environment.

[0075] Data preprocessing includes: image data preprocessing, including one or more of image resizing, random cropping, random flipping, and image perturbation; and point cloud data preprocessing, including one or more of point cloud sampling, subsampling, random perturbation, and random scaling. This preprocessing extracts the xyz coordinates of the point cloud from the original data file, filters out some noise points, and normalizes all points to the unit sphere.

[0076] S2: Construct a multimodal expression recognition model, including image branch, point cloud branch, cross-modal hint fusion module and fully connected layer. Figure 2As shown, the image branch extracts expression-related texture features from the input image data, and the point cloud branch extracts expression-related geometric features from the input point cloud data. The image and point cloud branches interact through a cross-modal cue fusion module, and a fully connected layer is used to obtain expression classification results. Both the image and point cloud branches are based on the Transformer architecture. S2 specifically includes the following steps.

[0077] S2.1: Establish an image branch consisting of K transformer blocks Composition, such as Figure 2 ViT layer 1 to ViT layer K in the image branch are shown, where i is the index of the transformer block and block embedding and i=0,1,…K-1. In the example, the input image I is divided into N fixed-size image blocks, which are then projected into block embeddings. As the initial block embedding, where D1 is the feature dimension. Among them, the initial block embedding E0 passes through the first transformer block The processed blocks are embedded in E1 and E i After the transformer block Processing to obtain block embedding E i+1 , and so on. The number of image blocks N can be a hyperparameter and can be set to a value such as 32 or 64 according to experience.

[0078] A learnable classification tag can be constructed As the initial classification mark, it is attached to the input sequence of the transformer block to aggregate classification-related information and obtain expression-related texture features. This calculation process can be expressed as:

[0079]

[0080] Among them, [·,·] represents the connection operation.

[0081] The transformer block that branches the above image The output block embedding E i+1 and classification tags Provided to the next transformer block and embedding the block into E i+1 It is provided to the corresponding cross-modal prompt fusion module and then provided to the corresponding transformer block of the point cloud branch after further processing.

[0082] S2.2: Construct a cross-modal cue fusion module, including a cross-modal adapter and a multimodal fusion part. Figure 3As shown in Figure 2, the cross-modal cue fusion module performs two operations: cross-modal adaptation and multimodal fusion. Cross-modal adaptation is performed by the cross-modal adapter, which, in addition to the down-projection linear layer, nonlinear activation function, and up-projection linear layer of the naive adapter, also introduces a convolutional layer tailored to the input image block dimension to aggregate features from different blocks.

[0083] In S2.2, K cross-modal cue fusion modules can be constructed. Each cross-modal cue fusion module is located before the input of each layer of the corresponding image branch and point cloud branch. This design facilitates multi-level fusion, enabling continuous and dynamic integration of 2D and 3D information. In the cross-modal cue fusion module, the cross-modal cue fusion process is carried out in two steps: a cross-modal adaptation operation, in which the cross-modal adapter adjusts the 2D modality-specific information output by the image branch into a modality-independent unified expression representation to address domain differences between the modalities; and a multi-modal cue fusion operation, in which the multi-modal fusion part integrates a unified expression representation to achieve effective multi-modal feature fusion.

[0084] Each cross-modal adapter in S2.2 consists of a convolutional layer applied to the image block dimension, as well as the same up-projection linear layer, down-projection linear layer and non-linear activation function as the naive adapter. i The convolution operation of the image block dimension aggregates the N+1 tags (including N image blocks and 1 classification tag) in the image branch and converts it into a fused prompt tag group that integrates modality information. This model includes T fused cue markers. This aggregation is crucial for creating cues that capture cross-modal features in a unified manner, while the nonlinear activations within the cross-modal adapter enhance its ability to model complex inter-modal relationships. T is the number of cue markers added to each layer, a hyperparameter that can be empirically set to values ​​such as 1, 4, or 8. The total number of cue markers in the model is K*T.

[0085] The cross-modal adapter processes the block embeddings received from the image branch to obtain the latent code, which is expressed as:

[0086] E i ′ =C i (E i )

[0087]

[0088] Among them, A i represents the hidden code output by the i-th cross-modal adapter, C i represents the convolutional layer of the i-th image block dimension, E i ′ represents the block embedding after changing the block dimension, is the i-th up-projection linear layer, is the i-th down-projection linear layer, and GELU(·) represents the activation function. Figure 3 , i is the index of the cross-modal prompt fusion module, the above C i 、 and They are the convolutional layer, up-projection linear layer, and down-projection linear layer of the image patch dimension of the cross-modal adapter of the i-th cross-modal cue fusion module, respectively.

[0089] Continue to refer Figure 3 , and also provide a learnable set of hint tags P i It includes T learnable prompt tags, which are updated after being processed by each cross-modal prompt fusion module to Specifically, the multimodal fusion part of each cross-modal cue fusion module performs a multimodal cue fusion operation, marking the learnable cue P i and the hidden code A output by the cross-modal adapter i Combined, we get the updated fusion prompt mark that incorporates modal information:

[0090]

[0091] Among them, P i represents the i-th group of learnable hint tags, Represents the updated fusion prompt label of the i-th group of multimodal fusion parts.

[0092] In this step, the i-th cross-modal cue fusion module embeds the block E from the output of the i-th transformer block of the image branch. i , obtained through cross-modal adapter processing implicit code A i , by combining the multimodal fusion part with the learnable hint marker P i With hidden code A i , obtain and output the updated fusion prompt mark after combining and integrating the modal information Provided to the corresponding i+1th transformer block in the point cloud branch. Among them, for the i=0th cross-modal hint fusion module, it receives the initial block embedding E0 and the fused hint mark obtained after processing And provide it to the corresponding first transformer block in the point cloud branch.

[0093] S2.3: Establish point cloud branch, point cloud branch Consists of K transformer blocks Composition, such as Figure 2The ViT layer 1 to ViT layer K in the point cloud branch are shown, i is the index of the transformer block and block embedding of the point cloud branch and i = 0, 1, ... K-1. In the example, the input point cloud Q is divided into M point cloud blocks, which are then projected into block embeddings. As the initial block embedding, where D2 is the feature dimension. Among them, the initial block embedding G0 passes through the first transformer block The processed blocks are embedded in G1 and G i After the transformer block Processing is performed to obtain the block embedding G i+1 , and so on.

[0094] Building learnable classification labels The input sequence of the transformer block attached to the point cloud branch as the initial classification label to aggregate classification-related information. The learnable classification label is a learnable vector with parameters initialized using a normal distribution and continuously updated during training.

[0095] In addition, the transformer blocks of the point cloud branch receive the learnable cue labels P from the corresponding cross-modal cue fusion module respectively. i , the updated fusion prompt mark obtained by the cross-modal prompt fusion module And, the updated fusion hint mark,block is embedded into G i and classification tags Feed into the corresponding transformer block of the point cloud branch Process and get interactive prompt mark Block embedding G i+1 and classification tags This calculation process can be expressed as:

[0096]

[0097] Transformer block that branches the point cloud The output block embedding G i+1 and classification tags Combined updated fusion hint markers Provide to the next transformer block of the point cloud branch

[0098] After all K transformer blocks of the point cloud molecule are processed, the Kth interaction prompt mark is obtained and output. The Kth block embeds G K and the K-th classification label

[0099] S2.4: Establish a fully connected layer to pool the block embeddings and interaction cue tags obtained after the K transformer blocks of the point cloud branch, and concatenate them with the classification tags obtained after the K transformer blocks of the point cloud branch to form the feature f for expression recognition:

[0100]

[0101] Among them, Pooling(·) represents the maximum pooling layer, Represents the interactive prompt mark output by the last (Kth transformer block) of the point cloud branch, G K represents the block embedding output by the last (Kth transformer block) of the point cloud branch, Represents the classification label of the final output of the point cloud branch (K-th transformer block).

[0102] The final classification prediction result is obtained based on the feature f of expression recognition:

[0103]

[0104] Here, Head(·) represents the classification head consisting of two fully connected layers.

[0105] The above processing may include converting the expression feature f into a dimension through two layers of fully connected layers, and then normalizing it using the Softmax function to obtain the prediction result. Represents the model's predicted probability for the six expressions.

[0106] A multimodal expression recognition model is constructed in this way.

[0107] S3: Establish a dual-level supervision mechanism to train the multimodal expression recognition model and obtain a trained multimodal expression recognition model. Use the general model to initialize the image encoder, use the classification loss to constrain the parameter optimization direction, and use the backpropagation algorithm to iteratively update and optimize the model parameters until the model loss converges. Fix the parameters of the image encoder, use the general model to initialize the point cloud encoder, set the dual-level loss function, and use the backpropagation algorithm to iteratively update and optimize the point cloud encoder parameters until the model loss converges. This step can be used to train the multimodal expression recognition model using the training set obtained after data partitioning.

[0108] Specifically, the dual-level supervision mechanism used in the training model in step S3 includes two parts of loss: cluster-level feature distance loss and sample-level prediction classification loss. The core idea of ​​cluster-level feature distance loss is to minimize the distance between the sample feature and the center of the distribution of its corresponding category feature, while maximizing the distance between the centers of the distribution of other category features. This design enhances the separation of features between categories and the aggregation of features within categories, thereby providing a more global understanding of feature distribution and bringing more robust representation learning, which is especially suitable for subtle expression recognition. The core idea of ​​sample-level prediction classification loss is to use category-specific weights to provide stricter supervision for easily confused categories. In order to identify these easily confused or highly similar expression pairs, the text prior knowledge in the visual language model (CLIP) is used to obtain the relationship between the six expression categories in the special space. This step S3 specifically includes the following steps.

[0109] S3.1: Establish cluster-level feature distance loss. Cluster-level feature distance loss can be achieved by combining Center Loss and DDA Loss (discriminant distribution independent loss). The former minimizes the intra-class variance by pulling sample features toward the center of each class, while the latter enhances inter-class discrimination by pushing features away from the centers of other classes. For a given sample p and its corresponding deep feature f p and sample labels Y p , cluster-level feature distance loss L CFD Calculated by the following formula:

[0110]

[0111] in, Represents the sample label (target) Y p The feature center of the category, c j Represents the feature center of other categories, j represents the index of the feature center of other categories and j = 1, 2...N c , N c Refers to the preset types of facial expressions that need to be classified. It should be understood that the number of facial expression types that need to be classified can be set separately as needed. B is the sample batch size during training, and λ and γ are hyperparameters. L Center represents the center loss, L DDA represents the discriminant distribution-independent loss, p represents the sample index and p=1,2…B, f p represents the depth feature of the p-th sample, Y p represents the sample label of the p-th sample.

[0112] like Figure 2 As shown, in this embodiment, N is set c= 6. The six facial expression categories used for classification may include anger, disgust, fear, happiness, sadness, and surprise.

[0113] S3.2: Establish sample-level prediction classification loss, including combining cross-entropy loss and weighted binary cross-entropy loss.

[0114] The samples are judged and classified into simple samples and complex samples; for simple samples, the naive cross entropy loss is directly applied; for more challenging complex samples, the binary cross entropy loss is calculated for each class.

[0115] Specifically, the samples are judged and classified. When the samples meet the following two conditions, they are judged as simple samples: (1) Classification prediction results and the true label Y p Consistent, that is It represents the predicted probability value of the j-th facial expression in the classification prediction result of the p-th sample, and argmax is the function of the parameter set that makes the function reach the maximum value. represents the predicted probability value of the jth facial expression in the classification prediction result of the pth sample, (2) the predicted value of the expression type with the highest probability in the classification prediction result The predicted value of the expression type with the second highest probability Higher than the set threshold τ, that is Otherwise, the sample is complex.

[0116] For simple samples, the sample-level prediction classification loss is calculated using the naive cross entropy loss:

[0117]

[0118] in, is the predicted category of the p-th sample, y p is the one-hot label of the p-th sample, L CE (·,·) represents the cross entropy loss function. p N c Dimension label, except for the target category value of 1, the other category values ​​are 0.

[0119] Cross entropy loss L CE The definition is as follows:

[0120]

[0121] in, represents the probability value of the kth type of expression in the unique hot label of the pth sample, Represents the predicted probability value of the kth type of expression in the classification prediction result of the pth sample, where k is the index of the expression type and k=1,2…N c.

[0122] For complex samples, the sample-level prediction classification loss L SPC It can be calculated by binary cross entropy loss as follows:

[0123]

[0124] Among them, L BCE (·,·) represents the binary cross entropy loss function.

[0125] Binary cross entropy loss L BCE The definition is as follows:

[0126]

[0127] in, Indicates that the sample label is Y p , the loss weight of the kth type of expression.

[0128] The above weights N can be calculated by CLIP c Six types of similarity matrices between expressions in this embodiment Calculated. Similarity matrix The item in is the similarity s between the mth facial expression and the nth facial expression m,n , where m and n are the indices of facial expressions, indicating N c One of a number of facial expressions. m,n is calculated as follows:

[0129]

[0130] Among them, min is the minimum function, s m,k Represents the similarity between the mth expression and the kth expression, m = 1, 2…N c , n=1,2…N c , k=1,2…N c .

[0131] S3.3: The multimodal expression recognition model is trained using the above cluster-level feature distance loss and sample-level prediction classification loss to obtain a trained multimodal expression recognition model.

[0132] After the model training is completed, the expression classification test can be carried out. The pre-processed test data is input into the trained model, the expression recognition results are output, and the model performance test and analysis are carried out.

[0133] S4: Use the trained multimodal expression recognition model to perform facial expression recognition and obtain expression recognition results. The paired image data and point cloud data to be recognized are input into the trained multimodal expression recognition model. The output classification prediction result is the expression recognition result. The expression recognition result can be used in subsequent processing programs such as human-computer interaction, health monitoring, and security surveillance.

[0134] Example 1

[0135] To better understand the present invention, the following describes Example 1 of a multimodal facial expression recognition method based on dynamic prompt learning provided by an embodiment of the present invention. The method of Example 1 uses a two-dimensional and three-dimensional expression recognition method based on dynamic cross-modal fusion of prompt learning, and includes the following steps.

[0136] S1. Dataset preparation. Complete dataset selection, data processing, and data partitioning.

[0137] This example selects the general three-dimensional expression recognition datasets BU-3DFE and Bosphorus to verify the invented two-dimensional and three-dimensional expression recognition method.

[0138] Data processing includes data preprocessing and data enhancement. Specifically, preprocessing of two-dimensional image data refers to resizing. In this example, all images are resized to 256×256 pixels. Data enhancement includes random cropping, random flipping, and image perturbation. For fixed-size images, a 224×224 pixel image region is randomly cropped and a 50% probability of random horizontal flipping is performed. Image perturbation enhancement uses four image perturbation methods: random adjustment of contrast, random adjustment of brightness, random adjustment of hue, and random adjustment of saturation. Preprocessing of three-dimensional point cloud data refers to point cloud sampling. In this example, 3800 points are sampled from the original point cloud using a sampling method based on facial feature points. Data enhancement includes subsampling, random perturbation, and random scaling. Subsampling refers to randomly sampling the preprocessed 3800-point point cloud into a 1024-point point cloud. Random perturbation refers to adding a certain amount of Gaussian noise to the point cloud. Random scaling refers to randomly scaling the point cloud size.

[0139] Data partitioning was performed based on the criteria specified for each dataset. For the BU-3DFE dataset, 648 pairs of 2D and 3D data were selected for training, 480 pairs for validation, and 72 pairs for testing. For the Bosphorus dataset, 324 pairs of 2D and 3D data were selected for training and 36 pairs for testing.

[0140] S2. Construct a two-dimensional and three-dimensional expression recognition model. Figure 2As shown, the two-dimensional and three-dimensional expression recognition model includes an image branch, a point cloud branch, and a cross-modal prompt fusion module. The image branch uses the ViT-B network, which includes K=12 attention layers, and the input image is divided into non-overlapping 16×16 blocks. The point cloud branch uses the Vit-S network, which includes K=12 attention layers. The input point cloud uses the K-nearest neighbor algorithm with k=32 to obtain neighbor points and synthesize blocks. In the cross-modal prompt fusion module, the dimension of the low-dimensional feature is set to 192, and T=4 fusion prompts are generated. The specific model design is described in the above implementation method and will not be repeated here.

[0141] S3. Design a dual-level supervision strategy to train the expression recognition model. During training, use a cluster-level feature distance loss and a sample-level prediction classification loss. The specific process is described in the above implementation and will not be repeated here. In this example, set λ = 0.01, γ = 0.1, and τ = 0.2.

[0142] The backpropagation algorithm is used to update and optimize the network parameter weights until the model loss converges. In this example, model training and evaluation are completed on the PyTorch platform. The model is trained on an NVIDIA 3080-Ti GPU with a batch size of 32. The learning rate is 1×10 -4 The network is optimized by the AdamW optimizer with a decay rate of 0.05. The model is trained for 300 rounds, and the results of the best performing model on the validation set are finally reported.

[0143] S4. After model training is completed, an expression classification test is performed. The pre-processed test data is input into the trained model, and the model output is the expression recognition result.

[0144] Among them, the two-dimensional and three-dimensional expression recognition method using dynamic cross-modal fusion based on prompt learning achieved an expression recognition accuracy of 91.38% on the BU-3DFE dataset and an expression recognition accuracy of 92.50% on the Bosphorus dataset, both of which were higher than other comparison methods. The results of the image branch alone on the two datasets were 89.58% and 90.00%, and the results of the point cloud branch alone on the two datasets were 88.15% and 87.50%, which shows that the method proposed in the present invention effectively integrates the information of the two modalities and achieves performance that exceeds that of a single modality. Furthermore, the results of the model that does not use the cross-modal adapter on the two datasets were 90.40% and 92.50%, and the results of the model that does not use the dual-level supervision mechanism were 90.26% and 91.11%. The results show that the use of the present invention can effectively improve the accuracy of two-dimensional and three-dimensional expression recognition.

[0145] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0146] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0147] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multimodal facial expression recognition method based on dynamic prompt learning, characterized in that: include: S1: Prepare image and point cloud expression recognition datasets, perform data preprocessing on the received raw image data and point cloud data, and perform data partitioning. The partitioned datasets include training set, validation set, and test set. S2: Construct a multimodal expression recognition model, including an image branch, a point cloud branch, a cross-modal cue fusion module, and a fully connected layer. The image branch is used to extract expression-related texture features from the input image data, and the point cloud branch is used to extract expression-related geometric features from the input point cloud data. The image branch and the point cloud branch interact through the cross-modal cue fusion module, and the expression classification results are obtained through the fully connected layer. S3: Establish a dual-level supervision mechanism to train the multimodal expression recognition model and obtain a trained multimodal expression recognition model. The dual-level supervision mechanism includes calculating the feature distance loss at the cluster level and the prediction classification loss at the sample level. S4: Input the facial image data and point cloud data to be recognized into the trained multimodal expression recognition model to perform facial expression recognition and obtain the expression recognition result.

2. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 1, characterized in that: S2 specifically includes: S2.1: Establish an image branch, which consists of K transformer blocks, where the input image is divided into N fixed-size image blocks and projected into block embedding E0 as the initial block embedding of the input image branch, and construct a learnable classification label The input sequence is attached to the transformer block as the initial classification label, and is processed successively by each transformer block of the image branch to obtain the output K block embeddings; S2.2: Construct K cross-modal cue fusion modules, each including a cross-modal adapter and a multimodal fusion part. Each cross-modal cue fusion module is placed before the input of each layer of the image branch and the point cloud branch to achieve multi-level fusion. S2.3: Establish a point cloud branch. The point cloud branch includes K transformer blocks. The input point cloud is divided into M point cloud blocks and projected into a block embedding G0, which is used as the initial block embedding of the input point cloud branch. The block embeddings G1, G2…G are successively processed by each transformer block of the point cloud branch. i+1 ,…G K ; S2.4: Establish a fully connected layer to pool the block embeddings obtained after the K transformer blocks passing through the point cloud branch and connect them with the classification labels to form features and classification prediction results for expression recognition.

3. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 2, characterized in that: S2.1 specifically includes: Establish an image branch, which includes K transformer blocks The input image is divided into N fixed-size image blocks and projected into block embeddings As the initial block embedding of the input image branch, where D1 is the feature dimension; Building learnable classification labels The input sequence is appended to the transformer block as an initial classification tag to aggregate classification-related information; Each transformer block in the image branch receives the block embedding and classification label output by the previous transformer block, and processes it to obtain expression-related texture features including the block embedding and classification label: Among them, [·,·] represents the connection operation, E i is the block embedding of the input to the i+1th transformer block, E i+1 is the block embedding output by the i+1th transformer block, is the classification label of the input to the i+1th transformer block, is the classification label output by the i+1th transformer block; The block embeddings and classification labels output by each transformer block of the image branch are provided to the next transformer block as input, and the block embeddings output by each transformer block are provided to the corresponding cross-modal cue fusion module; The image branch is processed by each transformer block in turn to obtain the block embedding E1, E2…E i+1 ,…E K , and classification tags 4. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 3 is characterized in that: S2.2 specifically includes: A cross-modal adapter is established, including a convolutional layer, an up-projection linear layer, and a down-projection linear layer in the image block dimension, to perform cross-modal adaptation operations. The block embedding received from the image branch is processed to obtain the latent code, which is expressed as: AND' i =C i (AND i ) Among them, E′ i represents the block embedding after changing the block dimension, C i Represents the convolutional layer of the i-th image block dimension, A i represents the hidden code output by the i-th cross-modal adapter, is the i-th up-projection linear layer, is the i-th down-projection linear layer, GELU(·) represents the activation function, i is the index of the cross-modal adapter and i = 0, 1, ... K-1; Establish a multimodal fusion part to process the provided learnable prompt markup and combine it with the latent code output by each cross-modal adapter to obtain an updated fused prompt markup that incorporates modality information: Among them, P i represents the i-th group of learnable hint tags and represents the updated fusion prompt mark of the i-th group of multimodal fusion parts, D2 is the feature dimension, and T represents the number of prompt marks included in each group of learnable prompt marks; The updated fused cue labels output by each cross-modal cue fusion module Provided to the corresponding transformer blocks in the point cloud branch respectively.

5. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 4 is characterized in that: S2.3 specifically includes: Establish a point cloud branch, which includes K transformer blocks The input point cloud is divided into M point cloud blocks and projected into block embeddings As the initial block embedding of the input point cloud branch; Building learnable classification labels The input sequence of the transformer block attached to the point cloud branch as the initial classification label; Each transformer block of the point cloud branch receives the updated fusion hint tag from the corresponding cross-modal hint fusion module, as well as the block embedding and classification tag from the previous transformer block for processing: Among them, G i is the block embedding of the input to the i+1th transformer block, G i+1 is the block embedding output by the i+1th transformer block, is the classification label of the input to the i+1th transformer block, is the classification label output by the i+1th transformer block, is the interactive prompt tag output by the i+1th transformer block; After each transformer block of the point cloud branch is processed successively, the final block embedding G is obtained K , interactive prompt mark and classification tags 6. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 5 is characterized in that: S2.4 specifically includes: A fully connected layer is built to pool the block embeddings and interaction cue tags output by the last transformer block of the point cloud branch, and concatenate them with the classification tags output by the last transformer block of the point cloud branch to form features for expression recognition: Among them, Pooling(·) represents the maximum pooling layer, Indicates the interactive prompt mark output by the point cloud branch at the end, G K represents the block embedding of the last output of the point cloud branch, Represents the classification label of the final output of the point cloud branch; The final classification prediction results are obtained based on the features of expression recognition: Here, Head(·) represents the classification head consisting of two fully connected layers.

7. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 1, characterized in that: S3 specifically includes: S3.1: Establish a cluster-level feature distance loss, which combines the center loss and the discriminant distribution-independent loss. The center loss minimizes the intra-class variance by pulling sample features toward their respective class centers, while the discriminant distribution-independent loss enhances inter-class discrimination by pushing features away from the centers of other classes. S3.2: Establish sample-level prediction classification losses, including cross-entropy loss and weighted binary cross-entropy loss; S3.3: The multimodal expression recognition model is trained through the cluster-level feature distance loss and the sample-level prediction classification loss to obtain a trained multimodal expression recognition model.

8. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 7, characterized in that: S3.1 specifically includes: Establish cluster-level feature distance loss: Among them, L Center represents the center loss, L DDA represents the discriminant distribution-independent loss, λ and γ are hyperparameters, p represents the sample index and p = 1, 2…B, B represents the sample batch size, f p represents the depth feature of the p-th sample, Y p represents the sample label of the p-th sample, Represents the sample label Y p The feature center of the category, c j Represents the feature center of other categories, j represents the index of the feature center of other categories and j = 1, 2...N c , N c Refers to the preset types of expressions that need to be classified.

9. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 7, characterized in that: S3.2 specifically includes: Judge and classify samples into simple samples and complex samples; For simple samples, a naive cross entropy loss is applied to calculate the sample-level prediction classification loss; For complex samples, binary cross entropy loss is calculated for each class to calculate the sample-level prediction classification loss.

10. The multimodal facial expression recognition method based on dynamic prompt learning according to claim 9, characterized in that: Judging and classifying samples includes: Samples that meet both of the following conditions are judged and classified as simple samples: (1) Classification prediction results of samples and the true label Y p Consistent, (2) the predicted value of the expression type with the highest probability in the classification prediction results of the sample The predicted value of the expression type with the second highest probability Higher than the set threshold τ, that is p represents the sample index and p=1,2…B, B represents the sample batch size; Samples that do not meet both of the above conditions are judged and classified as complex samples.

Images