Facial expression recognition method, device and equipment

Abstract

The present application discloses a facial expression recognition method, device and equipment, which are applied to the technical field of large language models. The method includes: obtaining a video sequence to be tested and a question instruction associated with facial expression recognition, wherein the video sequence to be tested includes at least one frame of a user's facial image; processing at least one frame of a facial image to obtain a facial feature sequence of at least one frame of the facial image, which includes at least one of facial key points, visual enhancement features and query embedded features, wherein the visual enhancement feature is an enhanced coding feature associated with the user's face in the image, and the query embedded feature is a natural language feature with a newly added coding feature associated with the user's face; obtaining a codebook index through the video sequence to be tested and the question instruction; obtaining the user's facial expression information according to the codebook index and the facial feature sequence of at least one frame of the facial image in combination with a large language model. The present application improves the accuracy and feature extraction capability of the large language model in the facial expression recognition task.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to methods, devices and equipment for facial expression recognition. Background Art

[0002] Multimodal emotion recognition is a branch of emotion recognition, which realizes emotion recognition through facial expression recognition and other modalities such as language and text. Facial expression recognition is a crucial part of multimodal emotion recognition. Related technologies use multimodal large language models for facial expression recognition and have achieved certain success. However, multimodal large language models often extract facial features in user images through visual encoders such as CLIP, and their visual encoders are difficult to capture all facial features and are prone to losing detailed information of facial features, which in turn limits the feature extraction ability of multimodal large language models. The recognition accuracy of multimodal large language models in facial expression recognition tasks needs to be improved. Summary of the invention

[0003] The purpose of this application is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0004] To this end, the purpose of this application is to provide a facial expression recognition method, device and equipment.

[0005] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0006] On the one hand, the embodiment of the present application provides a facial expression recognition method, comprising the following steps:

[0007] Acquire a video sequence to be tested of a user and a question instruction associated with facial expression recognition, wherein the video sequence to be tested includes at least one frame of a user's facial image;

[0008] Processing at least one frame of a user's facial image to obtain a facial feature sequence of at least one frame of the user's facial image, wherein the facial feature sequence includes at least one of facial key points, visual enhancement features, and query embedded features, wherein the visual enhancement features are used to characterize the enhanced coding features associated with the user's face in the user's facial image, and the query embedded features are used to characterize the natural language features newly added with the coding features associated with the user's face;

[0009] Obtaining a codebook index according to the video sequence to be tested and the question instruction, wherein the codebook index is used to guide a pre-trained large language model to perform facial expression recognition;

[0010] The user's facial expression information is obtained according to the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model.

[0011] In addition, the facial expression recognition method according to the above embodiment of the present application may also have the following additional technical features:

[0012] Furthermore, in one embodiment, the processing of at least one frame of user facial image to obtain a facial feature sequence of at least one frame of user facial image includes:

[0013] For each frame of user facial image, firstly, feature extraction is performed on the user facial image to obtain visual coding features, facial coding features and facial key points, then the visual coding features are enhanced using the facial coding features to obtain visual enhancement features, and query embedding features are obtained based on the question instructions, the facial coding features and the visual coding features, and then a facial feature sequence of the user facial image is obtained based on the query embedding features, the facial key points and the visual enhancement features.

[0014] Furthermore, in one embodiment, the extracting features of the user's facial image to obtain visual coding features, facial coding features and facial key points includes:

[0015] The user's facial image is encoded to obtain the visual encoding feature.

[0016] Furthermore, in one embodiment, the extracting features of the user's facial image to obtain visual coding features, facial coding features and facial key points includes:

[0017] Performing facial detection and cropping processing on the user facial image to obtain a facial cropped image, wherein the facial cropped image is used to represent a human face in the user facial image;

[0018] Performing encoding processing on the facial cropped image to obtain the facial encoding features;

[0019] The facial coding features are decoded to obtain the facial key points.

[0020] Furthermore, in one embodiment, the enhancing process of the visual coding feature using the facial coding feature to obtain the visual enhancement feature includes:

[0021] Processing the visual coding feature based on a self-attention mechanism to obtain a first visual feature, and obtaining a second visual feature based on the first visual feature and the visual coding feature;

[0022] Processing the second visual feature and the facial encoding feature based on a cross attention mechanism to obtain a third visual feature, and obtaining a fourth visual feature based on the second visual feature and the third visual feature;

[0023] Processing the fourth visual feature based on a feedforward neural network to obtain a fifth visual feature, and obtaining a sixth visual feature based on the fourth visual feature and the fifth visual feature;

[0024] The sixth visual feature is pooled to obtain the visual enhancement feature.

[0025] Further, in one embodiment, obtaining the query embedding feature according to the question instruction, the facial coding feature and the visual coding feature includes:

[0026] Obtaining a sub-embedded feature according to the question instruction, the facial coding feature, the visual coding feature and a preset query vector;

[0027] The sub-embedding feature and the visual encoding feature are processed based on a content attention mechanism to obtain the query embedding feature.

[0028] Furthermore, in one embodiment, obtaining a facial feature sequence of the user's facial image according to the query embedding feature, the facial key points and the visual enhancement feature comprises:

[0029] Performing projection processing on the query embedding feature, the facial key points and the visual enhancement feature to obtain processed query embedding features, facial key points and visual enhancement features;

[0030] A facial feature sequence of the user's facial image is obtained according to the processed query embedding features, facial key points and visual enhancement features.

[0031] Further, in one embodiment, obtaining a codebook index according to the video sequence to be tested and the question instruction includes:

[0032] Selecting a frame of user facial image from the video sequence to be tested as an index facial image;

[0033] Obtaining facial coding features of the index facial image and performing decoding processing to obtain user attribute features;

[0034] The codebook index is obtained according to the user attribute feature and the question instruction.

[0035] Further, in one embodiment, obtaining the codebook index according to the user attribute feature and the question instruction includes:

[0036] The user attribute features, the question instructions and the preset prompt template data are input into a marker of a pre-trained large language model, the codebook index is obtained through the marker, and the prompt template data is used to guide the pre-trained large language model to use the user attribute features when performing facial expression recognition.

[0037] On the other hand, an embodiment of the present application provides a facial expression recognition device, comprising:

[0038] An acquisition module, used for acquiring a video sequence to be tested of a user and a question instruction associated with facial expression recognition, wherein the video sequence to be tested includes at least one frame of a user's facial image;

[0039] A first processing module is used to process at least one frame of user facial image to obtain a facial feature sequence of at least one frame of user facial image, wherein the facial feature sequence includes at least one of facial key points, visual enhancement features, and query embedded features, wherein the visual enhancement features are used to characterize the enhanced coding features associated with the user face in the user facial image, and the query embedded features are used to characterize the natural language features newly added with the coding features associated with the user face;

[0040] A second processing module is used to obtain a codebook index according to the video sequence to be tested and the question instruction, wherein the codebook index is used to guide a pre-trained large language model to perform facial expression recognition;

[0041] The third processing module is used to obtain the user's facial expression information according to the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model.

[0042] In another aspect, an embodiment of the present application provides an electronic device, including:

[0043] at least one processor;

[0044] at least one memory for storing at least one program;

[0045] When at least one program is executed by at least one processor, the at least one processor implements the above-mentioned facial expression recognition method.

[0046] The beneficial effects of the present application are as follows: a facial expression recognition method, apparatus and device are provided. First, a video sequence to be tested of a user and a question instruction associated with facial expression recognition are obtained, wherein the video sequence to be tested includes at least one frame of a user's facial image; then, the at least one frame of the user's facial image is processed to obtain a facial feature sequence of the at least one frame of the user's facial image, wherein the facial feature sequence includes at least one of facial key points, visual enhancement features and query embedded features, wherein the visual enhancement features are used to characterize the enhanced coding features associated with the user's face in the user's facial image, and the query embedded features are used to characterize the natural language features newly added with the coding features associated with the user's face; then, a codebook index is obtained according to the video sequence to be tested and the question instruction, and the codebook index is used to guide a pre-trained large language model to perform facial expression recognition; finally, according to the codebook index and the facial feature sequence of at least one frame of the user's facial image, in combination with the pre-trained large language model, the user's facial expression information is obtained. The present application can capture more detailed and rich facial features, effectively improving the feature extraction capability of the multimodal large language model. At the same time, the video sequence to be tested and its facial features are used as visual cues for facial expression recognition, and question instructions are used as the basis of natural language. Facial expression recognition is achieved by combining visual cues and natural language, effectively improving the accuracy of the multimodal large language model in facial expression recognition tasks, and achieving better facial expression recognition effects.

[0047] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of a facial expression recognition method provided by an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of a facial expression recognition method provided in an embodiment of the present application;

[0050] Figure 3 is a structural diagram of a facial information mining module provided in an embodiment of the present application;

[0051] Figure 4 This is a sample diagram of facial expression recognition provided by an embodiment of the present application;

[0052] Figure 5 is another example diagram of facial expression recognition provided by an embodiment of the present application;

[0053] Figure 6 is a structural diagram of a facial expression recognition device provided in an embodiment of the present application;

[0054] Figure 7 This is an example diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0057] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] Emotion recognition is a natural language processing (NLP) technology that aims to analyze emotional tendencies in text data. Since information such as sound, text, facial expressions and behavior can all be used as markers of emotion, and emotions are multimodal, multi-cue and multi-emotional in nature, most current emotion recognition technologies are dedicated to achieving emotion recognition in different modalities (such as text, speech, image, video and multimodal).

[0060] Multimodal emotion recognition is a branch of emotion recognition, which realizes emotion recognition through facial expression recognition and other modalities such as language, text, etc. For example, in video emotion analysis, combining multiple information such as language, facial expression and body posture can more comprehensively and accurately identify emotions.

[0061] Facial expression recognition is a crucial part of multimodal emotion recognition. Early facial expression recognition technology relied on manually designed feature extraction methods, such as identifying facial expressions by analyzing facial muscle movement information and the location of facial feature points. These methods often show certain limitations when dealing with complex expressions and facial expressions of different races, ages, and genders. With the development of deep learning technology, facial expression recognition methods based on neural network models such as Convolutional Neural Network (CNN) and Transformer have made significant progress, which can improve the accuracy and robustness of facial expression recognition.

[0062] In order to further improve the accuracy of facial expression recognition, related technologies use multimodal large language models (MLLMs) for facial expression recognition. Thanks to the rich prior knowledge in the multimodal large language model, the accuracy of facial expression recognition has been significantly improved, and this facial expression recognition method has achieved certain success.

[0063] However, the related technology still has the following defects:

[0064] First, facial expression recognition is usually divided into static facial expression recognition and dynamic facial expression recognition. Static facial expression recognition focuses on the recognition of expressions in static images, while dynamic facial expression recognition focuses on the recognition and analysis of continuous expressions in dynamic videos. Existing multimodal large language models perform poorly in handling these two tasks in the same framework.

[0065] Secondly, multimodal large language models usually extract facial features in user images through visual encoders such as CLIP. However, their visual encoders are difficult to capture all facial features and are prone to losing detailed information of facial features, which in turn limits the feature extraction capabilities of multimodal large language models. The effect of facial feature extraction needs to be improved.

[0066] Third, related technologies usually perform facial expression recognition based on pre-cropped facial images or video frames, which will weaken the overall understanding ability of the multimodal large language model and the sensitivity of the multimodal large language model to other visual cues, while limiting the future expansion of the multimodal large language model in multimodal emotion recognition tasks.

[0067] In view of this, the embodiments of the present application propose a facial expression recognition method, apparatus and device, aiming to improve the accuracy of a multimodal large language model in facial expression recognition tasks.

[0068] First, the implementation steps of the facial expression recognition method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0069] The facial expression recognition method provided in the embodiment of the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited thereto. Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0070] Reference Figure 1 , Figure 1 : is a flowchart of the facial expression recognition method provided by the present application, and the facial expression recognition method mainly includes the following steps S101-S104:

[0071] S101, obtaining a video sequence to be tested of a user and a question instruction associated with facial expression recognition, wherein the video sequence to be tested includes at least one frame of a user's facial image;

[0072] S102, processing at least one frame of the user's facial image to obtain a facial feature sequence of the at least one frame of the user's facial image, wherein the facial feature sequence includes at least one of facial key points, visual enhancement features, and query embedded features, wherein the visual enhancement features are used to characterize the enhanced coded features associated with the user's face in the user's facial image, and the query embedded features are used to characterize the natural language features newly added with the coded features associated with the user's face;

[0073] S103, obtaining a codebook index according to the video sequence to be tested and the question instruction, where the codebook index is used to guide a pre-trained large language model to perform facial expression recognition;

[0074] S104, obtaining the user's facial expression information according to the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model.

[0075] In an embodiment of the present application, first, a video sequence to be tested of a user and a question instruction associated with facial expression recognition are obtained, the video sequence to be tested comprising at least one frame of a user's facial image; then, the at least one frame of the user's facial image is processed to obtain a facial feature sequence of the at least one frame of the user's facial image, the facial feature sequence comprising at least one of facial key points, visual enhancement features, and query embedded features, the visual enhancement features being used to characterize the enhanced coding features associated with the user's face in the user's facial image, and the query embedded features being used to characterize the natural language features newly added with the coding features associated with the user's face; thereafter, a codebook index is obtained according to the video sequence to be tested and the question instruction, the codebook index being used to guide a pre-trained large language model to perform facial expression recognition; finally, according to the codebook index and the facial feature sequence of at least one frame of the user's facial image, in combination with the pre-trained large language model, the user's facial expression information is obtained.

[0076] Thus, on the one hand, the embodiment of the present application can be applied to both dynamic facial expression recognition (corresponding to a video sequence to be tested containing two or more frames of user facial images) and static facial expression recognition (corresponding to a video sequence to be tested containing one frame of user facial images); on the other hand, the embodiment of the present application processes at least one frame of user facial images, extracts feature information associated with the user's face from the user facial image and enhances it, thereby capturing more detailed and rich facial features, and effectively improving the feature extraction capability of the multimodal large language model; on the other hand, the embodiment of the present application uses the video sequence to be tested and its facial features as visual cues for facial expression recognition, uses question instructions as the basis of natural language, combines visual cues and natural language to achieve facial expression recognition, and effectively improves the accuracy of the multimodal large language model in facial expression recognition tasks, achieving better facial expression recognition effects.

[0077] In the above step S101, the user's video sequence to be tested and question instructions are obtained, and the question instructions are associated with facial expression recognition. The video sequence to be tested includes N frames of user facial images, N≥1, and the purpose is to use the video sequence to be tested and its facial features as visual cues for facial expression recognition, and use the question instructions as the basis of natural language, combining visual cues and natural language to achieve facial expression recognition.

[0078] When the above-mentioned video sequence to be tested includes one frame of user facial image, the embodiment of the present application corresponds to a static facial expression recognition task; or, when the above-mentioned video sequence to be tested includes at least two frames of user facial images, the embodiment of the present application corresponds to a dynamic facial expression recognition task.

[0079] The method for acquiring the above-mentioned video sequence to be tested can be set according to actual conditions, and the embodiment of the present application does not specifically limit this. For example, the user's face is photographed by an image acquisition device such as a camera to obtain the user's video sequence to be tested.

[0080] The above-mentioned question instructions associated with facial expression recognition can be set according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0081] For example, the question instruction associated with facial expression recognition may be “user instruction: What is the facial expression of this person?”; for another example, the question instruction associated with facial expression recognition may be “user instruction: What is the emotion of this person in the video?”, but is not limited to this.

[0082] In the above step S102, at least one frame of user facial image is processed to obtain a facial feature sequence of at least one frame of user facial image, aiming to extract feature information associated with the user's face from the user facial image and enhance it, thereby capturing more detailed and rich facial features, and effectively improving the feature extraction capability of the multimodal large language model.

[0083] The facial feature sequence may include at least one of facial key points, visual enhancement features and query embedding features.

[0084] In an example, the facial feature sequence includes three types of feature information: facial key points, visual enhancement features, and query embedding features.

[0085] The visual enhancement features are used to characterize the enhanced coding features associated with the user's face in the user's facial image.

[0086] The query embedding features are used to characterize natural language features that are newly added with encoding features associated with the user's face.

[0087] The above-mentioned facial key points are used to indicate the key points of the user's face. The type of the key points can be set according to the actual situation. The embodiment of the present application does not specifically limit this. For example, the facial key points may include but are not limited to eyes, nose, mouth and eyebrows.

[0088] In the above step S103, since the embodiment of the present application realizes facial expression recognition based on a multimodal large language model, and the codebook index is an indispensable input for the multimodal large language model, the present application constructs the codebook index of the multimodal large language model based on the video sequence to be tested and the question instruction to guide the multimodal large language model to complete facial expression recognition.

[0089] The above codebook index is used to guide the pre-trained large language model for facial expression recognition, that is, to guide the multimodal large language model to quickly find language patterns and text answers related to facial expression recognition.

[0090] In the above step S104, the user's facial expression recognition is performed based on the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model, and then the user's facial expression information is recognized.

[0091] The following will be combined Figure 2 To further explain the above steps, Figure 2 middle is the facial cropped image of the user’s facial image in the tth frame, V t is the user's facial image in the tth frame, is the visual encoding feature of the user's facial image in the tth frame, is the facial encoding feature of the user's facial image in the tth frame, is the facial key point of the user's facial image in the tth frame, AGR is the user attribute feature; Q t is the sub-embedding feature, is the query embedding feature after being processed by the projector, is the visual enhancement feature after being processed by the projector. are the facial key points processed by the projector, corresponding to the tth frame; T Q is the question instruction, T A is the prediction index; “Learnable” means that the parameters of this part can be adjusted during training, and “Frozen” means that the parameters of this part are not adjusted during training.

[0092] In some implementations, the above-mentioned obtaining of the user's video sequence to be tested and the question instruction associated with facial expression recognition may include:

[0093] Obtain the user's video data to be tested;

[0094] The video data to be tested is processed to obtain a video sequence to be tested.

[0095] In this implementation, the user's video data to be tested is obtained and processed, and the processing may include but is not limited to sampling processing and data augmentation, so as to obtain a video sequence to be tested, thereby effectively improving the generalization ability of the multimodal large language model.

[0096] The above-mentioned processing of the video data to be tested to obtain the video sequence to be tested may include firstly performing uniform sampling processing on the video data to be tested to obtain a sampling sequence, and then performing data augmentation on the sampling sequence to obtain the video sequence to be tested.

[0097] The above-mentioned uniform sampling process is performed on the video data to be tested to obtain a sampling sequence, which may include the video data to be tested for a preset duration T′ and a preset frame rate f′. The sampling interval τ is calculated by the preset sampling frequency f, and the corresponding index of each segmentation point is A total of N frames are extracted, where N is a maximum integer such that (N-1)τ<T′ and N is greater than or equal to 1, thereby obtaining a sampling sequence.

[0098] The above sampling interval is used to indicate the sampling segment, which can be calculated by the following formula (1):

[0099]

[0100] In formula (1), τ is the sampling interval, f is the sampling frequency, and f′ is the preset frame rate.

[0101] The above duration, frame rate and sampling frequency can be set according to actual conditions, and this embodiment does not impose any specific limitation on this. For example, the sampling frequency is 1fps.

[0102] The above data augmentation may include image scaling, image cropping, image flipping, etc., but is not limited thereto.

[0103] For example, image scaling of a sampling sequence may be scaling the sampling sequence using the same scaling ratio along the time dimension. The image scaling algorithm may include a nearest neighbor interpolation algorithm, but other algorithms such as a bilinear interpolation algorithm and a bicubic interpolation algorithm are also applicable. This embodiment does not specifically limit this.

[0104] For another example, image cropping of the sampling sequence may be performed by randomly selecting a rectangular area in the sampling sequence and deleting parts other than the rectangular area while ensuring the resolution of the sampling sequence. The rectangular area may be a user's face.

[0105] For another example, performing image flipping on the sampling sequence may be flipping the sampling sequence in at least one of horizontal, vertical and diagonal directions to change the direction of the sampling sequence.

[0106] In some implementations, the processing of at least one frame of user facial image to obtain a facial feature sequence of at least one frame of user facial image may include:

[0107] For each frame of user facial image, firstly, feature extraction is performed on the user facial image to obtain visual coding features, facial coding features and facial key points, then the visual coding features are enhanced using the facial coding features to obtain visual enhancement features, and query embedding features are obtained based on the question instructions, facial coding features and visual coding features, and then a facial feature sequence of the user facial image is obtained based on the query embedding features, facial key points and visual enhancement features.

[0108] In this embodiment, for each frame of the user's facial image, the following process is performed: first, feature extraction is performed on the user's facial image to obtain visual coding features, facial coding features and facial key points, so as to preliminarily extract features associated with the user's face in the user's facial image, wherein the facial key points are the first facial prior features; then, the visual coding features are enhanced using the facial coding features to capture more comprehensive and rich detail information in the visual coding features, thereby obtaining visual enhancement features as the second facial prior features; at the same time, query embedding features are obtained according to the question instructions, facial coding features and visual coding features, aiming to add the perception of facial visual content to natural language, thereby obtaining facial prior features that combine natural language and visual content, namely, query embedding features; finally, the above three facial prior features are processed, and finally a facial feature sequence of the user's facial image is obtained. By traversing N frames of user facial images, each frame of the user's facial image is processed, thereby obtaining a facial feature sequence of N frames of user facial images.

[0109] In this way, this embodiment extracts feature information associated with the user's face from each frame of the user's facial image, that is, queries embedded features, facial key points and visual enhancement features, thereby capturing more, more detailed and richer facial features, effectively improving the feature extraction capability of the multimodal large language model, and realizing facial expression recognition through three different facial prior features combined with codebook indexes, thereby improving the multimodal large language model's ability to understand facial expressions, and thereby improving the accuracy of the multimodal large language model in facial expression recognition tasks.

[0110] In some embodiments, the feature extraction of the user's facial image to obtain visual coding features, facial coding features, and facial key points may include:

[0111] The user's facial image is encoded to obtain visual encoding features.

[0112] In this embodiment, the user facial image is encoded and features associated with the user's face in the user facial image, namely, visual coding features, are preliminarily extracted, so that in subsequent steps, visual enhancement features and query embedding features are obtained based on the visual coding features.

[0113] The above-mentioned encoding process of the user's facial image to obtain the visual coding feature may include encoding the user's facial image through a preset visual encoder to obtain the visual coding feature.

[0114] The above-mentioned visual encoder can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0115] For example, the above visual encoder may include multiple stacked Transformer blocks.

[0116] The stacks may be connected sequentially, but are not limited thereto. It should be understood that when the multiple Transformer blocks are connected sequentially, the output of the previous Transformer block will be used as the input of the next Transformer block.

[0117] The above Transformer block may include a multi-head attention layer, a feed-forward neural network layer, and a normalization layer.

[0118] The above Transformer block adopts CLIP as its pre-trained weights.

[0119] The number of the above-mentioned Transformer blocks can be 24, that is, the visual encoder is composed of 24 layers of Transformer blocks connected in sequence, but is not limited to this.

[0120] The output of the above Transformer block can satisfy the following formula (2):

[0121]

[0122] In formula (2), Z l is the output of the lth Transformer block; is the intermediate output of the lth Transformer block; Z l-1 is the output of the l-1th Transformer block, which is also the input of the lth Transformer block; LayerNorm(·) is the layer normalization operation; FFN(·) is the operation of the Feed Forward Network (FFN); MultiHead(Q, K, V) is the operation of the multi-head attention layer, Q is the query matrix, K is the key matrix, and V is the value matrix.

[0123] The operation of the above multi-head attention layer can satisfy the following formula (3):

[0124] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O,

[0125]

[0126] In formula (3), Concat(·) is the concatenation process, W O is the preset parameter matrix, h is the number of self-attention layers, head i is the output of the i-th self-attention layer, Attention(·) is the operation of the self-attention layer, and W i Q , W i K and W i V is the linear mapping parameter of the i-th self-attention layer, d k is the feature dimension of K, and softmax(·) is the operation of the softmax function.

[0127] The above query matrix, key matrix and value matrix can satisfy the following formula (4):

[0128] Q=Z l-1 W Q , K=Z l-1 W K , V=Z l-1 W V (4);

[0129] In formula (4), W Q , W K and W V The preset linear mapping parameters.

[0130] The above linear mapping parameters, parameter matrix and feature dimensions can all be set according to actual conditions, and this embodiment does not specifically limit this.

[0131] In some embodiments, the above-mentioned feature extraction of the user's facial image to obtain visual coding features, facial coding features and facial key points may also include:

[0132] Performing facial detection and cropping processing on the user's facial image to obtain a facial cropped image;

[0133] Encoding the facial cropped image to obtain facial coding features;

[0134] The facial coding features are decoded to obtain facial key points.

[0135] In this embodiment, in order to preliminarily extract features associated with the user's face in the user's facial image, first, face detection and cropping are performed on the user's facial image to obtain a facial cropped image, aiming to crop a compact facial image from the user's facial image; then, the facial cropped image is encoded to extract facial features, namely, facial coding features, from the facial cropped image; then, the facial coding features are decoded to obtain the first facial prior features, namely, facial key points.

[0136] The above-mentioned facial cropped image is used to represent the human face in the user's facial image.

[0137] The encoding process of the cropped facial image to obtain the facial coding features may include encoding the cropped facial image using a pre-trained facial encoder to obtain the facial coding features.

[0138] The above-mentioned decoding process of facial coding features to obtain facial key points may include using a pre-trained facial decoder to decode the facial coding features to obtain facial key points.

[0139] In one example, a facial encoder of a preset facial prior network is used to encode a facial cropped image to obtain facial encoding features, and a facial decoder of a preset facial prior network is used to decode the facial encoding features to obtain facial key points, as shown in the following formula (5):

[0140]

[0141] In formula (5), is the facial encoding feature of the user's facial image in the tth frame, is the facial cropped image of the user's facial image in the tth frame; Operation of the facial encoder for the facial prior network; is the facial key points of the user's facial image in the tth frame; Operation of the face decoder for the face prior network.

[0142] The facial prior network may be FaceXFormer. When the facial prior network is FaceXFormer, the facial encoder includes a plurality of stacked Swin Transformer blocks, and the facial decoder is a decoder suitable for facial expression recognition tasks.

[0143] The stacks may be connected in sequence, but are not limited thereto. It should be understood that when the multiple Swin Transformer blocks are connected in sequence, the output of the previous Swin Transformer block will be used as the input of the next Swin Transformer block.

[0144] The number of the Swin Transformer blocks can be 12, but is not limited to this.

[0145] The above-mentioned facial prior network may further include a multi-scale fusion layer composed of a multi-layer perceptron (MLP), and the multi-scale fusion layer is used to fuse features of different scales to improve the effect of feature extraction.

[0146] In some implementations, performing facial detection and cropping on the user's facial image to obtain a facial cropped image may include:

[0147] Use a pre-trained facial detection network to perform facial detection on the user's facial image to obtain the face coordinates in the user's facial image;

[0148] The user's facial image is cropped based on the facial coordinates in the user's facial image to obtain a facial cropped image.

[0149] In this implementation, in order to address the problem that pre-cropped facial images or video frames will weaken the overall understanding ability of the multimodal large language model and the sensitivity of the multimodal large language model to other visual cues, first, a pre-trained facial detection network is used to perform facial detection on the user's facial image to obtain the face coordinates in the user's facial image, and then the user's facial image is cropped based on the face coordinates in the user's facial image to obtain a cropped facial image, thereby ensuring the overall understanding ability of the multimodal large language model and its sensitivity to other visual cues, which is conducive to ensuring the future expansion of the multimodal large language model in multimodal emotion recognition tasks.

[0150] The above-mentioned face detection network can be a multi-task convolutional neural network (Multi-Task Cascaded Convolutional Networks, MTCNN), but is not limited to this.

[0151] In some embodiments, the above-mentioned enhancement processing of the visual coding features using the facial coding features to obtain the visual enhancement features may include:

[0152] The visual encoding feature is processed based on the self-attention mechanism to obtain a first visual feature, and a second visual feature is obtained according to the first visual feature and the visual encoding feature;

[0153] The second visual feature and the facial encoding feature are processed based on a cross attention mechanism to obtain a third visual feature, and a fourth visual feature is obtained according to the second visual feature and the third visual feature;

[0154] Processing the fourth visual feature based on a feedforward neural network to obtain a fifth visual feature, and obtaining a sixth visual feature based on the fourth visual feature and the fifth visual feature;

[0155] The sixth visual feature is pooled to obtain a visual enhancement feature.

[0156] In this implementation, in order to obtain the second facial prior feature, the visual coding feature is used as the basic feature, and the facial coding feature is used as the auxiliary feature. The facial coding feature is used to supplement the visual coding feature with facial information, and then the second facial prior feature, namely the visual enhancement feature, is obtained. This can effectively solve the problem that the existing multimodal large language model cannot extract detailed facial information when understanding facial expressions, and capture more comprehensive and rich detail information in the visual coding feature, thereby improving the feature extraction capability of the multimodal large language model, which is conducive to achieving better facial feature extraction effects.

[0157] Specifically, the above enhancement processing is implemented by a preset face mining module, referring to Figure 3 The facial information mining module may include a self-attention layer, a cross-attention layer and a feedforward neural network layer. The self-attention layer is configured with a self-attention mechanism, the cross-attention layer is configured with a cross-attention mechanism, and the feedforward neural network layer is configured with a feedforward neural network as the name implies. Based on this, the implementation process of the above enhancement processing is as follows:

[0158] First, the visual coding feature is processed by the self-attention mechanism of the self-attention layer to obtain a first visual feature as the output of the self-attention layer, and a second visual feature is obtained based on the first visual feature and the visual coding feature.

[0159] The second visual feature is obtained according to the first visual feature and the visual coding feature. The second visual feature may be obtained by adding the first visual feature and the visual coding feature, but is not limited thereto.

[0160] Then, the second visual feature and the facial coding feature are used as input of the cross-attention layer, and the second visual feature and the facial coding feature are processed by the cross-attention mechanism of the cross-attention layer to obtain the third visual feature, and the fourth visual feature is obtained based on the second visual feature and the third visual feature.

[0161] The fourth visual feature is obtained according to the second visual feature and the third visual feature. The fourth visual feature may be obtained by adding the second visual feature and the third visual feature, but the present invention is not limited thereto.

[0162] Afterwards, the fourth visual feature is used as an input of the feedforward neural network layer, and the fourth visual feature is processed based on the feedforward neural network layer to obtain the fifth visual feature, and the sixth visual feature is obtained based on the fourth visual feature and the fifth visual feature.

[0163] The obtaining of the sixth visual feature according to the fourth visual feature and the fifth visual feature may include adding the fourth visual feature and the fifth visual feature to obtain the sixth visual feature, but is not limited thereto.

[0164] Finally, the sixth visual feature is pooled to obtain a visual enhancement feature.

[0165] The above pooling process may be a pooling operation along a spatial dimension, but is not limited thereto.

[0166] The implementation of the above facial information mining module can satisfy the following formula (6):

[0167]

[0168] In formula (6), is the visual enhancement feature, Pool(·) is the pooling process, FFN(·) is the operation of the feedforward neural network, XAttn(·) is the operation of the cross attention mechanism, SAttn(·) is the operation of the self-attention mechanism, is the visual encoding feature of the user's facial image in the tth frame, is the facial coding feature of the user's facial image in the tth frame. It should be understood that this formula does not represent an addition operation, but it does not affect the understanding of the facial information mining module by those skilled in the art.

[0169] In some implementations, obtaining the query embedding feature according to the question instruction, the facial encoding feature, and the visual encoding feature may include:

[0170] Obtain sub-embedded features according to the question instructions, facial encoding features, visual encoding features and a preset query vector;

[0171] The sub-embedding features and visual encoding features are processed based on the content attention mechanism to obtain the query embedding features.

[0172] In this implementation, in order to obtain the third facial prior feature, first, based on the question instructions, facial coding features, visual coding features and the preset query vector, feature information that combines natural language and coding features associated with the user's face, namely, sub-embedded features, is preliminarily obtained. Then, the sub-embedded features and visual coding features are processed through the content attention mechanism, aiming to enhance the perception of facial visual content by the feature information and improve the effect of feature extraction, thereby obtaining the third facial prior feature, namely, the query embedding feature.

[0173] The query vector is a parameter that can be set according to actual conditions, and the query vector is a learnable parameter, that is, the query vector is updated according to the gradient of the multimodal large language model when training the multimodal large language model.

[0174] The above-mentioned obtaining of sub-embedded features according to question instructions, facial coding features, visual coding features and a preset query vector may include taking the question instructions, facial coding features, visual coding features and a preset query vector as inputs of a preset query extraction module, and obtaining the sub-embedded features through the query extraction module.

[0175] The query extraction module may be a Q-Former, but is not limited thereto.

[0176] In one example, the question instruction, facial coding features, visual coding features and a preset query vector are input into the Q-Former, and the question instruction, facial coding features, visual coding features and the preset query vector are processed by the Q-Former to obtain a sub-embedded feature, and then the sub-embedded feature and the visual coding feature are processed based on the content attention mechanism to obtain a query embedded feature, as shown in the following formula (7):

[0177]

[0178] In formula (7), Q t is the sub-embedding feature, QFormer(·) is the operation of Q-Former, T Q For question instructions, is the visual encoding feature of the user's facial image in the tth frame, is the facial encoding feature of the user's facial image in the tth frame; is the query embedding feature, Pool(·) is the pooling process, ContextAttention(·) is the operation of the content attention mechanism, and softmax(·) is the operation of the softmax function.

[0179] In one embodiment, obtaining the facial feature sequence of the user's facial image based on the query embedding feature, facial key points and visual enhancement feature may include:

[0180] Projecting the query embedding features, facial key points and visual enhancement features to obtain processed query embedding features, facial key points and visual enhancement features;

[0181] According to the processed query embedding features, facial key points and visual enhancement features, a facial feature sequence of the user's facial image is obtained.

[0182] In this implementation, in order to map the query embedding features, facial key points and visual enhancement features to the target representation space of the multimodal large language model, the query embedding features, facial key points and visual enhancement features are projected, aiming to map the query embedding features, facial key points and visual enhancement features to the same target representation space through linear transformation to ensure the correctness of the data; after processing the query embedding features, facial key points and visual enhancement features, a facial feature sequence of the user's facial image is constructed as one of the inputs of the multimodal large language model based on the processed query embedding features, facial key points and visual enhancement features.

[0183] The above-mentioned processing of the query embedding features, facial key points and visual enhancement features to obtain the processed query embedding features, facial key points and visual enhancement features can include processing the query embedding features through a first projector to obtain the processed query embedding features; processing the facial key points through a second projector to obtain the processed facial key points; processing the visual enhancement features through a third projector to obtain the processed visual enhancement features.

[0184] The above projector can be a multi-layer perceptron, and the implementation of its projection processing is shown in the following formula (8):

[0185]

[0186] In formula (8), Represents the processed query embedding features, MLP T (·) is the operation of the first multilayer perceptron, Embed features for queries; Represents the processed visual enhancement features, MLP V (·) is the operation of the second multilayer perceptron, For visual enhancement features; Represents the processed facial key points, MLP L (·) is the operation of the third multilayer perceptron, The key points of the face.

[0187] The above-mentioned obtaining the facial feature sequence of the user's facial image based on the processed query embedded features, facial key points and visual enhancement features may include splicing the processed query embedded features, facial key points and visual enhancement features into the facial feature sequence of the user's facial image, but is not limited to this.

[0188] In some implementations, obtaining the codebook index according to the video sequence to be tested and the question instruction may include:

[0189] Selecting a frame of user facial image from the video sequence to be tested as an index facial image;

[0190] Obtaining facial coding features of the indexed facial image and performing decoding processing to obtain user attribute features;

[0191] According to the user attribute characteristics and question instructions, the codebook index is obtained.

[0192] In this embodiment, in order to obtain a codebook index for guiding a multimodal large language model to perform facial expression recognition, first, a frame of a user facial image is selected from a video sequence to be tested, and the selected user facial image is used as an index facial image; then, since the facial coding features of each frame of the user facial image have been obtained in the aforementioned steps, the facial coding features of the index facial image can be directly obtained, and the facial coding features of the index facial image are decoded to obtain user attribute features; finally, the user attribute features and question instructions are used to obtain a codebook index, so that the codebook index is integrated with attribute information related to the user and instructions associated with facial expression recognition, thereby improving the accuracy of the multimodal large language model in facial expression recognition tasks.

[0193] The selection of the above index facial images may be arbitrarily selected or may be selected according to a preset rule, and this embodiment does not impose any specific limitation on this.

[0194] The above-mentioned acquiring the facial coding features of the index facial image and performing decoding processing to obtain the user attribute features may include acquiring the facial coding features of the index facial image obtained in the aforementioned step, and using a facial decoder of a preset facial prior network to decode the facial coding features of the index facial image to obtain the user attribute features, as shown in the following formula (9):

[0195]

[0196] In formula (9), AGR is the user attribute feature, To index the facial encoding features of the face image, Operation of the face decoder for the face prior network.

[0197] The specific implementation of the above-mentioned facial prior network has been recorded in the aforementioned implementation manner, and this implementation manner will not be repeated here.

[0198] The above-mentioned user attribute features are used to characterize facial clues, which may include at least one of the user's age, gender and racial attributes. The above-mentioned age may be a specific value or at least one numerical range, and this embodiment does not make specific limitations on this; the above-mentioned gender includes either male or female; the above-mentioned racial attribute is used to characterize the user's skin color.

[0199] For example, the above user attribute characteristics may be "age: 40-49 years old, gender: male, race: white".

[0200] In some implementations, obtaining the codebook index according to the user attribute characteristics and the question instruction may include:

[0201] The user attribute features, question instructions and preset prompt template data are input into the tagger of the pre-trained large language model, and the codebook index is obtained through the tagger.

[0202] In this embodiment, in order to obtain the codebook index for guiding the multimodal large language model to perform facial expression recognition, after obtaining the user attribute features, the user attribute features, question instructions and preset prompt template data are input into the marker, which is a marker of a pre-trained large language model, to obtain the corresponding codebook index.

[0203] The above prompt template data is used to guide the pre-trained large language model to use user attribute features when performing facial expression recognition. For example, the above prompt template data can be "Depending on the specific problem, you can use or partially use the following information:".

[0204] It should be understood that the tokenizer of the large language model is an existing technology. For example, the Hugging face open source library provides a variety of large language models and their tokenizers, which will not be described in detail in this embodiment.

[0205] In some implementations, obtaining the user's facial expression information based on the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model may include:

[0206] Input the codebook index and the facial feature sequence of each frame of the user's facial image into the pre-trained large language model;

[0207] Recognize the user's facial expression through a large language model and obtain a prediction index;

[0208] According to the prediction index, the user's facial expression information is obtained.

[0209] In this embodiment, after obtaining the codebook index and the facial feature sequence of each frame of the user's facial image (including visual enhancement features, query embedded features and facial key points), the codebook index and the facial feature sequence of each frame of the user's facial image are input into a pre-trained large language model. The large language model will recognize the user's facial expression based on the input data and output the corresponding prediction index. Then, based on the prediction index, the user's facial expression information is obtained, thereby realizing the application of the large language model in the facial expression recognition task.

[0210] The above-mentioned inputting the codebook index and the facial feature sequence of each frame of the user's facial image into the pre-trained large language model may include concatenating the codebook index and the facial feature sequence of each frame of the user's facial image to obtain an input sequence and inputting it into the pre-trained large language model.

[0211] The above-mentioned recognition of the user's facial expression to obtain the prediction index can include first splitting the input sequence into multiple text segments (Tokens), where the text segment can be understood as the smallest unit for segmenting and encoding the input text, and then obtaining the probability of each text segment (Token) in the codebook, and then obtaining the prediction index (index ID) through a classification function and a preset sampling strategy.

[0212] The prediction formula of the above large language model can satisfy the following formula (10):

[0213]

[0214] In formula (10), p(T A |V,X F , X L , AGR, T Q ) is the prediction function of the large language model, T A is the prediction index, V is the video sequence to be tested, X L is the face cropped image, AGR is the user attribute feature, T Q is the problem instruction and Θ is a learnable parameter.

[0215] The above classification function may be a softmax function, but is not limited thereto.

[0216] The obtaining of the user's facial expression information according to the prediction index may include decoding the prediction index to obtain the user's facial expression information.

[0217] The facial expression information may be text information.

[0218] In some embodiments, the above method may further include the following steps:

[0219] Get the answer instruction corresponding to the predicted index;

[0220] Calculate the loss function value between the predicted index and its corresponding answer instruction;

[0221] The large language model is trained by backpropagation based on the loss function value.

[0222] In this embodiment, after completing one or more facial expression recognition tasks, for at least one prediction index, the answer instruction corresponding to the prediction index is obtained and the loss function value between the prediction index and the corresponding answer instruction is calculated, and then the large language model is back-propagated and trained according to the loss function value to update the model parameters of the large language model (i.e., tune the large language model), thereby improving the accuracy of the large language model in facial expression recognition tasks.

[0223] The above-mentioned answer instruction is used to indicate the real answer corresponding to the question instruction.

[0224] The above loss function may be a cross-entropy loss function (CELF), which is shown in the following formula (11):

[0225] L CE = -∑y′log(y) (11);

[0226] In formula (11), L CE Represents the loss function value between the predicted index and its corresponding answer instruction; y′ is the probability distribution of the answer instruction corresponding to the predicted index, which can be understood as the probability distribution of the true label; y is the probability distribution of the predicted index, which can be understood as the probability distribution of the predicted label.

[0227] The above back propagation training can be used to tune the low-rank adaptation parameters of the large language model.

[0228] In some embodiments, the above method may further include the following steps:

[0229] Back-propagation training is performed on at least one of the facial information mining module, the facial prior network, the query extraction module, the projector, and the module configured with the content attention mechanism according to the loss function value.

[0230] In this embodiment, after calculating the loss function value between the predicted index and its corresponding answer instruction, the loss function value is used to perform back-propagation training on at least one of the facial information mining module, the facial prior network, the query extraction module, the projector, and the module configured with the content attention mechanism to achieve tuning, thereby improving the accuracy of at least one of the facial information mining module, the facial prior network, the query extraction module, the projector, and the module configured with the content attention mechanism.

[0231] In some implementations, the above large language model can be set according to actual conditions, and this implementation is not specifically limited to this. It should be understood that the embodiments of the present application are applicable to any large language model, especially a multimodal large language model.

[0232] To facilitate the understanding of the above-mentioned facial expression recognition method of the present application, the actual application scenario of the above-mentioned facial expression recognition method of the present application is used as an example. The specific process of the above-mentioned facial expression recognition method is as follows: steps S201-S204.

[0233] S201, collect the user's video data to be tested and perform uniform sampling processing on the video data to be tested to obtain a sampling sequence, and then perform data augmentation on the sampling sequence to obtain a video sequence to be tested; wherein the video sequence to be tested includes N frames of user facial images, (N-1)τ<T′ and N≥1, and the data augmentation method can include at least one of image scaling, image cropping and image flipping.

[0234] S202, for each frame of the user's facial image, the following steps S01-S06 are performed to obtain a facial feature sequence of each frame of the user's facial image:

[0235] S01, encodes the user's facial image through a visual encoder composed of 24 stacked Transformer blocks to obtain visual encoding features;

[0236] S02, firstly, performing face detection on the user's face image through MTCNN to obtain the face coordinates in the user's face image, and performing cropping processing on the user's face image based on the face coordinates in the user's face image to obtain a face cropped image, and then performing encoding processing and decoding processing on the face cropped image through FaceXFormer, wherein the encoding processing obtains the face encoding features, and the decoding processing obtains the face key points;

[0237] S03, inputting the visual coding features and the facial coding features into a facial information mining module, using the facial coding features to supplement the facial information for the visual coding features, and obtaining visual enhancement features;

[0238] S04, inputting the question instruction, facial coding features, visual coding features and a preset query vector into Q-Former, processing the question instruction, facial coding features, visual coding features and the preset query vector through Q-Former to obtain a sub-embedded feature, and processing the sub-embedded feature and the visual coding feature based on a content attention mechanism to obtain a query embedded feature;

[0239] S05, processing the query embedding feature through a first multi-layer perceptron to obtain a processed query embedding feature, processing the facial key points through a second multi-layer perceptron to obtain a processed facial key points, and processing the visual enhancement feature through a third multi-layer perceptron to obtain a processed visual enhancement feature;

[0240] S06, concatenating the processed query embedding features, facial key points and visual enhancement features into a facial feature sequence of the user's facial image.

[0241] S203, first select a frame of user face image from the video sequence to be tested, and use the selected user face image as the index face image, then obtain the facial coding features of the index face image and decode the facial coding features of the index face image to obtain user attribute features, and then input the user attribute features, question instructions and preset prompt template data into a marker, which is a marker of a pre-trained large language model, to obtain a corresponding codebook index.

[0242] S204, concatenating the codebook index and the facial feature sequence of each frame of the user's facial image to obtain an input sequence and inputting it into a pre-trained large language model, and then recognizing the user's facial expression through the large language model to obtain a predicted index, and then decoding the predicted index to obtain the user's facial expression information.

[0243] Reference Figure 4 and Figure 5 , Figure 4 and Figure 5 These are all sample pictures of facial expression recognition provided in the embodiments of the present application. The facial expression recognition method provided in the present application can realize dynamic facial expression recognition and static facial expression recognition, and can accurately recognize the facial expression of the user in the video or image.

[0244] Secondly, the implementation method of the facial expression recognition device provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0245] Reference Figure 6 , the facial expression recognition device may include:

[0246] An acquisition module 301 is used to acquire a video sequence to be tested of a user and a question instruction associated with facial expression recognition;

[0247] A first processing module 302 is used to process at least one frame of user facial image to obtain a facial feature sequence of at least one frame of user facial image;

[0248] The second processing module 303 is used to obtain a codebook index according to the video sequence to be tested and the problem instruction;

[0249] The third processing module 304 is used to obtain the user's facial expression information according to the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model.

[0250] The above-mentioned facial expression recognition device executes the above-mentioned facial expression recognition method. The specific definition of the facial expression recognition device can be found in the definition of the facial expression recognition method above, which will not be repeated here.

[0251] Each module in the above-mentioned facial expression recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0252] Finally, refer to Figure 7 , an embodiment of the present application further provides an electronic device, which may include:

[0253] at least one processor 401;

[0254] At least one memory 402, for storing at least one program;

[0255] When at least one program is executed by at least one processor 401, at least one processor 401 implements the above-mentioned facial expression recognition method.

[0256] The memory 402, as a non-transient network system, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 402 may optionally include a memory 402 remotely arranged relative to the processor 401, and these remote memories 402 may be connected to the processor 401 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0257] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 402, and the processor 401 calls and executes the methods of the embodiments of this application.

[0258] The processor 401 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0259] In some embodiments, the electronic device further comprises:

[0260] Input / output interface, used to realize information input and output;

[0261] Communication interface, used to realize communication interaction between this device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0262] A bus that transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface, and communication interface);

[0263] The processor 401 , the memory 402 , the input / output interface and the communication interface can be connected to each other in communication within the device via a bus.

[0264] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0265] In summary, the embodiments of the present application fully explore the application of multimodal large language models in facial expression recognition in images or videos. Specifically:

[0266] In terms of dynamic and static facial expression recognition, the embodiments of the present application can be applied to both dynamic facial expression recognition (corresponding to a video sequence to be tested containing two or more frames of user facial images) and static facial expression recognition (corresponding to a video sequence to be tested containing one frame of user facial images), thereby improving the performance of a large multimodal language model in processing these two tasks in the same framework, and improving the scalability of a large multimodal language model in facial expression recognition and emotion understanding applications.

[0267] In terms of feature extraction, the embodiment of the present application extracts feature information associated with the user's face from each frame of the user's facial image, namely, query embedded features, facial key points and visual enhancement features, thereby capturing more, more detailed and richer facial features, reducing the risk of losing detailed information of facial features, and effectively improving the feature extraction capability of the multimodal large language model; and, the embodiment of the present application realizes facial expression recognition through three different facial prior features (namely, query embedded features, facial key points and visual enhancement features) combined with codebook indexing, which not only improves the multimodal large language model's ability to understand facial expressions, but also reduces the difficulty of combining multimodal emotional cues with natural language, ensuring that the multimodal large language model responds to the diversity of facial expression recognition, thereby improving the accuracy of the multimodal large language model in facial expression recognition tasks.

[0268] Regarding the extraction of the first facial prior feature (i.e., facial key points), the extraction of the first facial prior feature is performed based on a facial cropped image. Different from the existing pre-cropping method, the embodiment of the present application uses a pre-trained facial detection network to perform facial detection on the user's facial image, obtains the facial coordinates in the user's facial image, and implements cropping based on the facial coordinates. This not only improves the accuracy of the cropping process and ensures that a compact facial image is cropped from the user's facial image, but also improves the overall understanding ability of the multimodal large language model and its sensitivity to other visual cues, which is conducive to ensuring the future expansion of the multimodal large language model in multimodal emotion recognition tasks.

[0269] Regarding the extraction of the second facial prior feature (i.e., the visual enhancement feature), the embodiment of the present application uses the visual coding feature as the basic feature and the facial coding feature as the auxiliary feature, and uses the facial coding feature to supplement the visual coding feature with facial information, thereby obtaining the second facial prior feature, i.e., the visual enhancement feature. This can effectively solve the problem that the existing multimodal large language model cannot extract detailed facial information when understanding facial expressions, and capture more comprehensive and rich detail information in the visual coding feature, thereby improving the feature extraction capability of the multimodal large language model, which is conducive to achieving better facial feature extraction effects.

[0270] Regarding the extraction of the third facial prior feature (i.e., the query embedding feature), the embodiment of the present application processes the question instruction based on the facial coding features, visual coding features and the preset query vector to obtain feature information (i.e., sub-embedded features) that combines natural language and coding features associated with the user's face, and then enhances the perception of facial visual content through the content attention mechanism, thereby improving the effect of feature extraction.

[0271] In terms of codebook indexing, the embodiment of the present application constructs a codebook index through user attribute features and question instructions. The user attribute features are obtained by decoding the facial coding features of the user's facial image in the selected frame, so that the codebook index is integrated with user-related attribute information and instructions associated with facial expression recognition, thereby improving the accuracy of the multimodal large language model in facial expression recognition tasks, reducing the difficulty of combining multimodal emotional cues with natural language, and ensuring that the multimodal large language model responds to the diversity of facial expression recognition.

[0272] In terms of model tuning, the embodiment of the present application calculates the loss function of the real-time predicted results and the expected results, and updates the model parameters of at least one of the large language model, facial information mining module, facial prior network, query extraction module, projector and module configured with content attention mechanism based on the loss function value, thereby improving the performance of multiple modules and the large language model.

[0273] In summary, the embodiments of the present application can effectively improve the accuracy of the multimodal large language model in facial expression recognition tasks and achieve better facial expression recognition effects.

[0274] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0275] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0276] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0277] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0278] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0279] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0280] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0281] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0282] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A facial expression recognition method, characterized in that: The following steps are involved: Acquire a video sequence to be tested of a user and a question instruction associated with facial expression recognition, wherein the video sequence to be tested includes at least one frame of a user's facial image; Processing at least one frame of user facial image to obtain a facial feature sequence of at least one frame of user facial image, wherein the facial feature sequence includes facial key points, visual enhancement features, and query embedding features, wherein the visual enhancement features are used to characterize the enhanced coding features associated with the user face in the user facial image, and the query embedding features are used to characterize the natural language features newly added with the coding features associated with the user face; Obtaining a codebook index according to the video sequence to be tested and the question instruction, wherein the codebook index is used to guide a pre-trained large language model to perform facial expression recognition; Obtaining the user's facial expression information according to the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model; The processing of at least one frame of user facial image to obtain a facial feature sequence of at least one frame of user facial image includes: For each frame of user facial image, firstly, feature extraction is performed on the user facial image to obtain visual coding features, facial coding features and facial key points, then the visual coding features are enhanced using the facial coding features to obtain visual enhancement features, and query embedding features are obtained based on the question instructions, the facial coding features and the visual coding features, and then a facial feature sequence of the user facial image is obtained based on the query embedding features, the facial key points and the visual enhancement features.

2. The facial expression recognition method according to claim 1, characterized in that: The step of extracting features from the user's facial image to obtain visual coding features, facial coding features and facial key points includes: The user's facial image is encoded to obtain the visual encoding feature.

3. The facial expression recognition method according to claim 1, characterized in that: The step of extracting features from the user's facial image to obtain visual coding features, facial coding features and facial key points includes: Performing facial detection and cropping processing on the user facial image to obtain a facial cropped image, wherein the facial cropped image is used to represent a human face in the user facial image; Performing encoding processing on the facial cropped image to obtain the facial encoding features; The facial coding features are decoded to obtain the facial key points.

4. The facial expression recognition method according to claim 1, characterized in that: The method of enhancing the visual coding feature by using the facial coding feature to obtain a visual enhancement feature includes: Processing the visual coding feature based on a self-attention mechanism to obtain a first visual feature, and obtaining a second visual feature based on the first visual feature and the visual coding feature; Processing the second visual feature and the facial encoding feature based on a cross attention mechanism to obtain a third visual feature, and obtaining a fourth visual feature based on the second visual feature and the third visual feature; Processing the fourth visual feature based on a feedforward neural network to obtain a fifth visual feature, and obtaining a sixth visual feature based on the fourth visual feature and the fifth visual feature; The sixth visual feature is pooled to obtain the visual enhancement feature.

5. The facial expression recognition method according to claim 1, characterized in that: The step of obtaining a query embedding feature according to the question instruction, the facial coding feature and the visual coding feature comprises: Obtaining a sub-embedded feature according to the question instruction, the facial coding feature, the visual coding feature and a preset query vector; The sub-embedding feature and the visual encoding feature are processed based on a content attention mechanism to obtain the query embedding feature.

6. The facial expression recognition method according to claim 1, characterized in that: The step of obtaining a facial feature sequence of the user's facial image according to the query embedding feature, the facial key points and the visual enhancement feature comprises: Performing projection processing on the query embedding feature, the facial key points and the visual enhancement feature to obtain processed query embedding features, facial key points and visual enhancement features; A facial feature sequence of the user's facial image is obtained according to the processed query embedding features, facial key points and visual enhancement features.

7. The facial expression recognition method according to claim 1, characterized in that: The obtaining a codebook index according to the video sequence to be tested and the problem instruction includes: Selecting a frame of user facial image from the video sequence to be tested as an index facial image; Obtaining facial coding features of the index facial image and performing decoding processing to obtain user attribute features; The codebook index is obtained according to the user attribute feature and the question instruction.

8. The facial expression recognition method according to claim 7, characterized in that: The obtaining the codebook index according to the user attribute feature and the question instruction includes: The user attribute features, the question instructions and the preset prompt template data are input into a marker of a pre-trained large language model, the codebook index is obtained through the marker, and the prompt template data is used to guide the pre-trained large language model to use the user attribute features when performing facial expression recognition.

9. A facial expression recognition device, characterized in that: Applied to the facial expression recognition method according to any one of claims 1 to 8, the device comprises: An acquisition module, used for acquiring a video sequence to be tested of a user and a question instruction associated with facial expression recognition, wherein the video sequence to be tested includes at least one frame of a user's facial image; A first processing module is used to process at least one frame of user facial image to obtain a facial feature sequence of at least one frame of user facial image, wherein the facial feature sequence includes facial key points, visual enhancement features and query embedding features, wherein the visual enhancement features are used to characterize the enhanced coding features associated with the user face in the user facial image, and the query embedding features are used to characterize the natural language features newly added with the coding features associated with the user face; A second processing module is used to obtain a codebook index according to the video sequence to be tested and the question instruction, wherein the codebook index is used to guide a pre-trained large language model to perform facial expression recognition; The third processing module is used to obtain the user's facial expression information according to the codebook index and the facial feature sequence of at least one frame of the user's facial image in combination with a pre-trained large language model.

Images