Expression recognition method, device, electronic device and computer-readable storage medium

By extracting and encoding facial motion intensity data, the intensity feature matrix is ​​generated for camouflage expression recognition, which solves the problem of low accuracy of camouflage expression recognition in the prior art, and achieves high accuracy recognition of camouflage expressions.

CN114842523BActive Publication Date: 2024-11-12SOUTHWEST UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210259916.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-16
Publication Date
2024-11-12
Estimated Expiration
2042-03-16

AI Technical Summary

Technical Problem

The prior art has low accuracy when identifying camouflage expressions, and cannot effectively distinguish between multiple types of camouflage expressions and mixed expressions, resulting in inaccurate recognition results.

Method used

By extracting the facial motion intensity data of each frame of the image in the target video sequence, encode it into an intensity feature vector, and inputting it into a classifier for classification, an intensity feature matrix is ​​generated to identify camouflage expressions, and the accuracy of recognition is improved.

Benefits of technology

It effectively improves the recognition accuracy of camouflage expressions, can accurately identify various types of camouflage expressions and mixed expressions, and enhances the integrity of the recognition results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114842523B_ABST
    Figure CN114842523B_ABST
Patent Text Reader

Abstract

The present application provides an expression recognition method, device, electronic device and computer-readable storage medium, which relate to the field of expression recognition technology. The method comprises: extracting multiple facial motion intensity data from each frame image in a target video sequence; encoding the multiple facial motion intensity data to obtain an intensity feature vector corresponding to each frame image; inputting the intensity feature matrix determined by the multiple intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result. The present application can obtain the facial motion intensity data representing the facial expression muscle movement of a human face in multiple frames of images in a disguised expression data set, and encode to extract feature vectors, obtain the intensity feature matrix of the target video sequence according to the multiple feature vectors, classify the intensity feature matrix through a classifier, recognize the disguised expression appearing in the target video sequence, and obtain a recognition result, which effectively improves the accuracy of recognizing the disguised expression.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of expression recognition, and in particular to an expression recognition method, device, electronic device and computer-readable storage medium. Background Art

[0002] Facial expressions are the most direct and effective emotional patterns. By performing facial expression recognition on the captured facial images, we can get the facial expressions and judge the type of expression. Several common basic expressions include: anger, disgust, fear, happiness, sadness and surprise. However, humans sometimes deliberately use disguised expressions to cover up their true emotions for certain purposes. Disguised expressions are complex expressions and cannot be characterized by basic expressions.

[0003] Due to the complexity of disguised expressions under factors such as environment and individual differences, the existing recognition of disguised expressions has a high degree of similarity in facial features, and is unable to distinguish between multiple types of disguised expressions. It is also unable to recognize mixed expressions with similar states, resulting in a low accuracy rate in recognizing disguised expressions. Summary of the invention

[0004] In view of this, the purpose of the embodiments of the present application is to provide an expression recognition method, device, electronic device and computer-readable storage medium to improve the problem of low accuracy in recognizing disguised expressions in the prior art.

[0005] In order to solve the above problems, in a first aspect, the present application provides a method for facial expression recognition, the method comprising:

[0006] Extracting multiple facial motion intensity data from each frame of a target video sequence, wherein the target video sequence is any disguised expression video sequence in a disguised expression data set;

[0007] Encoding the plurality of facial motion intensity data to obtain an intensity feature vector corresponding to each frame of image;

[0008] The intensity feature matrix determined by the multiple intensity feature vectors is input into a classifier for classification to obtain a disguised expression recognition result.

[0009] In the above implementation, by acquiring facial motion intensity data that can represent the facial muscle movement when a person makes a disguised expression in each frame of the target video sequence, the disguised expression can be identified from the changing trend of the facial motion intensity data based on the facial motion intensity data. By encoding the facial motion intensity data, the local spatiotemporal features in each frame of the image can be extracted to obtain the corresponding intensity feature vector. By inputting the intensity feature matrix composed of multiple intensity feature vectors that represents the global spatiotemporal features in the target video sequence into a variety of classifiers, the intensity feature matrix can be used as the recognition data for the disguised expression recognition task to obtain the corresponding disguised expression recognition result. The present application extracts and encodes facial feature information, and uses facial motion intensity data to represent the facial muscle movement when the target person produces a disguised expression, which can effectively improve the accuracy of recognizing disguised expressions.

[0010] Optionally, encoding the plurality of facial motion intensity data to obtain an intensity feature vector corresponding to each frame of image includes:

[0011] Converting the plurality of facial motion intensity data corresponding to each frame of image into a plurality of sub-vectors;

[0012] The intensity feature vector representing the local spatiotemporal features of the disguised expression in each frame of image is generated according to the multiple sub-vectors.

[0013] In the above implementation, since the target video sequence contains multiple frames, the dimensions of the facial motion of the face in each frame can be encoded. By encoding the facial motion intensity data extracted from each frame, the intensity value can be converted into a sub-vector representing facial features in multiple dimensions, and the sub-vectors in multiple dimensions can generate an intensity feature vector representing the local spatiotemporal features of the disguised expression in the current frame. In this way, the local spatiotemporal features of the disguised expression in a single frame can be extracted, and the facial features can be extracted in real time when the expression of the target person in the target video sequence changes.

[0014] Optionally, the step of inputting the intensity feature matrix determined by the plurality of intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result includes:

[0015] Performing adjustment processing on the plurality of intensity feature vectors to obtain the intensity feature matrix representing the global spatiotemporal features of the disguised expression in the target video sequence;

[0016] The intensity feature matrix is ​​input into the classifier for classification to obtain the disguised expression recognition result.

[0017] In the above implementation, in order to better extract the spatiotemporal features of each video sequence, the intensity feature vectors representing the local spatiotemporal features of the disguised expression in a single frame image belonging to the same target video sequence can be adjusted to obtain an intensity feature matrix of uniform size that can represent the global spatiotemporal features of the disguised expression in the target video sequence, and the intensity feature matrix is ​​used as recognition data for the disguised expression recognition task of the target video sequence, and is input into a variety of classifiers for classification of the disguised expression to obtain the corresponding disguised expression recognition results. Recognition using the intensity feature matrix that represents the global spatiotemporal features can comprehensively recognize the target video sequence and improve the accuracy of the disguised expression recognition results.

[0018] Optionally, the adjusting process of the plurality of intensity feature vectors to obtain the intensity feature matrix representing the global spatiotemporal features of the disguised expression in the target video sequence includes:

[0019] Determine the target size;

[0020] Performing downsampling or interpolation processing on the multiple intensity feature vectors according to the target size to obtain multiple target intensity feature vectors;

[0021] The intensity feature matrix is ​​generated according to a plurality of the target intensity feature vectors.

[0022] In the above implementation, since the disguised expression data set may include multiple disguised expression video sequences, and the lengths of the multiple video sequences are not uniform, the intensity feature vectors belonging to the same target video sequence can be downsampled and interpolated by a determined target size to obtain multiple corresponding target intensity feature vectors, and then the multiple target feature vectors are encoded to generate an intensity feature matrix with uniform dimensions and sizes. Adjustment processing can be performed for video sequences of multiple different lengths to improve the comprehensiveness of the spatiotemporal features of the intensity feature matrix of each target video sequence.

[0023] Optionally, inputting the intensity feature matrix into the classifier for classification to obtain the disguised expression recognition result includes:

[0024] Inputting each of the intensity feature matrices into the classifier;

[0025] Based on the classifier, the intensity feature matrix is ​​identified to obtain multiple different types of disguised expression recognition data;

[0026] The disguised expression recognition result is determined according to the plurality of disguised expression data.

[0027] In the above implementation, the classifier can perform multi-layer processing on each input intensity feature matrix, and recognize the intensity feature matrix as the recognition data of the disguised expression recognition task, and can obtain disguised expression recognition data corresponding to a variety of different disguised expressions. By integrating and classifying a variety of different types of disguised expression data, the final disguised expression recognition result for each intensity feature matrix can be determined. Each type of disguised expression can be recognized, which improves the integrity of the disguised expression recognition result.

[0028] Optionally, extracting a plurality of facial motion intensity data from each frame of the target video sequence includes:

[0029] Determine a plurality of facial coding points in each frame of the target video sequence;

[0030] Screening the plurality of facial code points according to the intensity requirement to obtain a plurality of target code points that meet the intensity requirement;

[0031] Extracting a plurality of facial motion intensity data of a plurality of target coding points.

[0032] In the above implementation, when the facial intensity features of the target person are extracted, since the face of the target person in each frame image has multiple facial coding points, in order to improve the effectiveness of the extracted facial intensity features, the multiple facial coding points can be screened according to the intensity requirements to screen out multiple points that can represent a larger intensity of facial changes in the face as target coding points. The facial intensity features in the target coding points can be extracted to obtain multiple facial motion intensity data, thereby improving the effectiveness of the facial motion intensity data.

[0033] Optionally, determining a plurality of facial coding points in each frame of the target video sequence includes:

[0034] Determine the reference points of the face in each frame of the image;

[0035] Performing facial alignment on a plurality of the reference points to determine key areas of the human face;

[0036] A plurality of facial encoding points are determined from the key area.

[0037] In the above implementation, when detecting the face of the target person in the image, in order to eliminate the interference of image noise and head movement on intensity detection, points in the face that are less affected by expression muscle movement can be extracted as reference points to perform matrix transformation, align the face of the target person, determine the aligned key area, and determine multiple facial coding points in the key area. The position in the face can be calibrated during detection to improve the accuracy of the facial coding points, thereby improving the accuracy of facial intensity feature extraction.

[0038] In a second aspect, the present application further provides an expression recognition device, the device comprising:

[0039] An extraction module, used for extracting a plurality of facial motion intensity data in each frame of an image in a target video sequence, wherein the target video sequence is any disguised expression video sequence in a disguised expression data set;

[0040] An encoding module, used for encoding the plurality of facial motion intensity data to obtain an intensity feature vector corresponding to each frame of image;

[0041] The classification module is used to input the intensity feature matrix determined by the multiple intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result.

[0042] In the above implementation, facial motion intensity data that can represent the facial muscle movement when a person makes a disguised expression in each frame of the target video sequence is obtained through the extraction module, and the disguised expression can be identified from the change trend of the facial motion intensity data based on the facial motion intensity data. The facial motion intensity data is encoded by the encoding module, and the local spatiotemporal features in each frame of the image can be extracted to obtain the corresponding intensity feature vector. The intensity feature matrix composed of multiple intensity feature vectors that represents the global spatiotemporal features in the target video sequence is input into a variety of classifiers through the classification module, and the intensity feature matrix can be used as the recognition data of the disguised expression recognition task for recognition, and the corresponding disguised expression recognition result can be obtained.

[0043] In a third aspect, the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any implementation of the above-mentioned expression recognition method.

[0044] In a fourth aspect, the present application also provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps in any implementation of the above-mentioned expression recognition method are executed.

[0045] In summary, the present application provides an expression recognition method, device, electronic device and computer-readable storage medium, which uses facial movement intensity to characterize the facial muscle movement of the target person when the target person produces a disguised expression, extracts and encodes the feature information of the person's facial movement intensity, and recognizes the generated feature matrix through a classifier, thereby effectively improving the accuracy of recognizing disguised expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of an expression recognition method provided in an embodiment of the present application;

[0049] Figure 3 A detailed flowchart of step S300 provided in an embodiment of the present application;

[0050] Figure 4 A detailed flowchart of step S400 provided in an embodiment of the present application;

[0051] Figure 5 A detailed flowchart of step S410 provided in an embodiment of the present application;

[0052] Figure 6 A detailed flow chart of step S420 provided in an embodiment of the present application;

[0053] Figure 7 A detailed flow chart of step S200 provided in an embodiment of the present application;

[0054] Figure 8 A detailed flowchart of step S210 provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of the structure of an expression recognition device provided in an embodiment of the present application.

[0056] Icon: 100 - electronic device; 111 - memory; 112 - storage controller; 113 - processor; 114 - peripheral interface; 115 - input and output unit; 116 - display unit; 500 - expression recognition device; 510 - extraction module; 520 - encoding module; 530 - classification module. DETAILED DESCRIPTION

[0057] The technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.

[0058] In the expression of human facial expressions, several common basic expressions may include: anger (abbreviated as an or An), disgust (abbreviated as di or Di), fear (abbreviated as fe or Fe), happiness (abbreviated as ha or Fe), sadness (abbreviated as sa or Sa), surprise (abbreviated as su or Su). Humans sometimes deliberately use disguised expressions to cover up their true emotions for certain purposes. Disguised expressions are complex expressions and cannot be characterized by basic expressions.

[0059] In order to identify the real emotions of human beings, some methods for identifying disguised expressions have been proposed in the prior art. For example, the LBP (Local Binary Pattern) widely used in macro expressions is improved and extended to LBP-TOP (Local Binary Patterns from three orthogonal planes) of three orthogonal planes, and the global texture features of the disguised expression image sequence are extracted. Finally, the SVM (Support Vector Machine) is used to identify the disguised expressions. However, due to the complexity of disguised expressions under factors such as environment and individual differences, the existing recognition methods have a high degree of facial feature similarity during recognition, and are unable to distinguish between multiple types of disguised expressions. Moreover, mixed expressions with similar states cannot be recognized during recognition, resulting in a low accuracy rate in identifying disguised expressions.

[0060] Therefore, in order to solve the above problems, an embodiment of the present application provides an expression recognition method, which is applied to electronic devices. The electronic devices can be servers, personal computers (PCs), tablet computers, smart phones, personal digital assistants (PDAs), and other electronic devices with logical computing functions, which can recognize disguised expressions by extracting facial features.

[0061] Optionally, see Figure 1 , Figure 1 The block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may include a memory 111, a storage controller 112, a processor 113, a peripheral interface 114, an input and output unit 115, and a display unit 116. A person skilled in the art may understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0062] The memory 111, storage controller 112, processor 113, peripheral interface 114, input / output unit 115 and display unit 116 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute the executable module stored in the memory.

[0063] The memory 111 may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 113 executes the program after receiving the execution instruction. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application can be applied to the processor 113, or implemented by the processor 113.

[0064] The processor 113 may be an integrated circuit chip with signal processing capability. The processor 113 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0065] The peripheral interface 114 couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113 and the memory controller 112 can be implemented in a single chip. In other embodiments, they can be implemented by separate chips.

[0066] The input and output unit 115 is used to provide input data to the user. The input and output unit 115 can be, but is not limited to, a mouse and a keyboard.

[0067] The above-mentioned display unit 116 provides an interactive interface (such as a user operation interface) between the electronic device 100 and the user or is used to display image data for the user's reference. In this embodiment, the display unit can be a liquid crystal display or a touch display. If it is a touch display, it can be a capacitive touch screen or a resistive touch screen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more positions on the touch display, and the sensed touch operations are handed over to the processor for calculation and processing. In an embodiment of the present application, the display unit 116 can display multiple frames of images in a disguised expression data set, etc.

[0068] The electronic device in this embodiment can be used to execute each step in each expression recognition method provided in the embodiment of the present application. The implementation process of the expression recognition table method is described in detail below through several embodiments.

[0069] See also Figure 2 , Figure 2 A flow chart of an expression recognition method provided in an embodiment of the present application, the method may include the following steps.

[0070] Step S200: extract multiple facial motion intensity data in each frame image in the target video sequence.

[0071] In this application, when identifying disguised expressions, the identification data selected is a disguised expression data set obtained from a public database, and the target video sequence is any disguised expression video sequence in the disguised expression data set. The extracted facial motion intensity data can be data that can represent the intensity of facial motion units in a facial coding system, such as AU intensity, AU features or mixed features of AUs (Facial Action Units, automatic facial motion unit detection), and the facial motion intensity data that can represent the facial activities of the target person is obtained by encoding the movement of the facial muscles of the target person.

[0072] Optionally, when extracting facial motion intensity data of images in a target video sequence, cross-dataset automatic AUs detection technology can be used for extraction. For example, the automatic AUs detection in the open source open face is used to extract AUs and corresponding AU intensities. Detection can be performed through a real-time facial action unit intensity estimation and occurrence detection system based on appearance (gradient histogram) and geometric features (shape parameters and landmark positions), thereby improving the detection accuracy across data sets.

[0073] It is worth noting that the Masked Facial Expression Database (MFED) is a dataset for systematically studying masked expressions. Masked expressions are complex expressions compared to a variety of basic expressions. The Masked Facial Expression Database may include multiple recognition tasks, such as: classification tasks with or without masked expressions (2 categories), classification tasks of experienced emotions (6 categories, referred to as 6E), classification tasks of required expressions (6 categories, referred to as 6R), and classification tasks of 36 types of masked expressions (36 categories). For example, please refer to Table 1, which is a category detail of a masked expression dataset used. The masked expressions used in this application may be composed of 778 videos of 22 human subjects, including 12 females and 10 males. Each masked expression video is shot by a camera with a resolution of 1280*720 and a frame rate of 30 frames per second to ensure the uniformity of the video. The subjects are trained to be familiar with and demonstrate 6 basic expressions normally, and 36 categories of masked expressions are generated. There are 19-22 samples in each of the 36 expression categories, and 129-131 samples in each of the 6 major categories of disguised expressions. The number of samples is evenly distributed. During the collection of these data, all subjects received at least two weeks of training from researchers from the Institute of Psychology of the Chinese Academy of Sciences to ensure that when the video induces the 6 basic real expressions (emotions) hidden, they can correctly make the 6 required designated disguised expressions at the same time as receiving the instructions. The data set contains a total of 36 expression types (30 inconsistent expression types and 6 consistent basic expression types). Therefore, the data of the entire disguised expression data set can truly represent the disguised expressions produced in real scenes.

[0074] Table 1

[0075]

[0076] It is worth noting that the facial motion intensity data extracted in this application belongs to the secondary features of the shape features, which can build a facial perception computing model to distinguish the mixed expression categories in the disguised expression data set. And when using secondary features for recognition, since the secondary features cannot be traced back to the face image, it can effectively protect the user's privacy and improve the security during recognition.

[0077] Step S300, encoding the plurality of facial motion intensity data to obtain an intensity feature vector corresponding to each frame of image.

[0078] Among them, Aus can be combined with feature engineering. By encoding multiple facial motion intensity data of each frame image obtained, features in the facial motion intensity data are extracted to obtain an intensity feature vector that can represent the disguised expression in each frame image.

[0079] It is worth noting that, since the intensity feature vector can only represent the disguised expression in the frame image, the intensity feature vector is a feature vector of the local spatial feature of a single frame.

[0080] Step S400: inputting the intensity feature matrix determined by the plurality of intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result.

[0081] Among them, since the intensity feature vector is the feature vector of the local spatial features of a single frame, however, in disguised expressions, due to different environments, different individual differences and other factors, the intensity and speed of each person's disguised expressions are actually inconsistent. In order to distinguish the various disguised expressions in the target video sequence, considering that the local spatial features of a single frame image have too large a feature Pearson correlation coefficient when performing 36 classification tasks on disguised expressions, the feature similarity is too high, and the machine cannot recognize the classification of expressions. Therefore, the present application can also determine the intensity feature matrix that can characterize the global spatiotemporal features of the disguised expressions in the target video sequence based on multiple intensity feature vectors, and input the intensity feature matrix into the classifier for classification.

[0082] It is worth noting that the classifier can be a variety of models, such as CNN (Convolutional Neural Networks), 3DCNN, random forest, neural network, residual neural network, transform, etc., all of which can use the intensity feature matrix as the recognition data for the disguised expression recognition task and obtain the corresponding disguised expression recognition results.

[0083] For example, during classification, the entire sequence can be input to obtain 36 categories of disguised expressions, covering the problem of determining the presence or absence of classification and positioning of disguised expressions using local spatial features. It can also identify "mixed expressions" with very similar states in the process of evolution from real expression A to disguised expression B and from real expression B to disguised expression A. For example, the Pearson correlation coefficients of 15 pairs of disguised expressions of the same subject, such as (an-ha, ha-an), (an-fe, fe-an), (sa-su, su-sa), are calculated respectively, and the sum is divided by the total number of subjects to calculate the average correlation coefficient in order to improve the accuracy of classification.

[0084] For example, when using 3DCNN to identify the intensity feature matrix, you can use the built-in model in the torch library. When extracting optical flow features, you can use RAFT (Recurrent All Pairs Field Transforms) optical flow deep network. RAFT can extract features of each pixel from two input images, take the inner product of all feature vector pairs to build a 4D related volume, and the last two dimensions of the 4D volume are merged at multiple scales to build a set of multi-scale volumes, and the field is iteratively updated by a loop unit that performs a lookup on the related volume. Maintaining and updating a single fixed stream at high resolution can recover from incorrect predictions at coarse resolution.

[0085] exist Figure 1 In the illustrated embodiment, by extracting and encoding facial feature information and using facial movement intensity data to characterize the facial muscle movements of the target person when the target person produces a disguised expression, the accuracy of recognizing the disguised expression can be effectively improved.

[0086] Optionally, see Figure 3 , Figure 3 A detailed flow chart of step S300 is provided for an embodiment of the present application. Step S300 may also include steps S310-S320.

[0087] Step S310: convert the plurality of facial motion intensity data corresponding to each frame of image into a plurality of sub-vectors.

[0088] The intensity value representing the facial motion of the target person in the facial motion intensity data can be converted into multiple sub-vectors representing motion features. For example, when the target video sequence includes n frames of images, multiple facial motion intensity data in each frame of the image can be expressed as:

[0089]

[0090] Among them, au x They represent each facial motion intensity data respectively, and the dimension of the sub-vector corresponds to the number of facial motion intensity data.

[0091] Step S320: Generate the intensity feature vector representing the local spatiotemporal features of the disguised expression in each frame of image according to the multiple sub-vectors.

[0092] Among them, the algorithm is used to generate the intensity feature vector representing the local spatiotemporal characteristics of the disguised expression in the current frame from the sub-vectors of multiple dimensions. The obtained intensity feature vectors of multiple dimensions can be expressed as:

[0093]

[0094] exist Figure 3 In the illustrated embodiment, local spatiotemporal features of disguised expressions in a single-frame image can be extracted, and facial features can be extracted in real time when the expression of a target person in a target video sequence changes.

[0095] Optionally, see Figure 4 , Figure 4 A detailed flowchart of step S400 is provided in an embodiment of the present application. Step S400 may also include steps S410-S420.

[0096] Step S410: adjusting the plurality of intensity feature vectors to obtain the intensity feature matrix representing the global spatiotemporal features of the disguised expression in the target video sequence.

[0097] Among them, since the speed and intensity of activating the disguised expression of each target person in multiple video sequences in the disguised expression dataset are inconsistent, the video lengths of multiple video sequences are also inconsistent. In order to better extract the spatiotemporal features of each video sequence, the intensity feature vectors representing the local spatiotemporal features of the disguised expression in a single frame image belonging to the same target video sequence can be adjusted to obtain an intensity feature matrix of uniform size that can represent the global spatiotemporal features of the disguised expression in the target video sequence. For example, the intensity feature matrix can be recorded as:

[0098]

[0099] The intensity feature matrix can be determined according to the number of intensity feature vectors and the number of frames.

[0100] Step S420, inputting the intensity feature matrix into the classifier for classification to obtain the disguised expression recognition result.

[0101] Among them, the intensity feature matrix is ​​used as the recognition data for the disguised expression recognition task of the target video sequence, and is input into a variety of classifiers for classification processing of the disguised expression to obtain the corresponding disguised expression recognition results.

[0102] exist Figure 4 In the illustrated embodiment, the intensity feature matrix representing the global spatiotemporal features is used for recognition, so that the target video sequence can be fully recognized, thereby improving the accuracy of the disguised expression recognition result.

[0103] Optionally, see Figure 5 , Figure 5 A detailed flowchart diagram of step S410 is provided for an embodiment of the present application. Step S410 may also include steps S411-S413.

[0104] Step S411, determining the target size.

[0105] When processing multiple intensity feature vectors of video sequences of different lengths, a unified target size for recognition may be determined, and the target size may be the number of frames of a determined image.

[0106] Step S412: performing downsampling processing or interpolation processing on the multiple intensity feature vectors according to the target size to obtain multiple target intensity feature vectors.

[0107] The intensity feature vectors belonging to the same target video sequence can be downsampled and interpolated according to the determined target size, and a plurality of corresponding target intensity feature vectors can be obtained by reducing the intensity feature vector or increasing the sampling frequency of the intensity feature vector.

[0108] Step S413: generating the intensity feature matrix according to the plurality of target intensity feature vectors.

[0109] Among them, encoding processing is performed on multiple target feature vectors to generate an intensity feature matrix with uniform dimensions and sizes, and multiple video sequences of different lengths can be uniformly recognized.

[0110] exist Figure 5 In the illustrated embodiment, adjustment processing can be performed for video sequences of various lengths to improve the comprehensiveness of the spatiotemporal features of the intensity feature matrix of each target video sequence.

[0111] Optionally, see Figure 6 , Figure 6 A detailed flow chart of step S420 is provided for an embodiment of the present application. Step S420 may also include steps S421-S423.

[0112] Step S421: input each of the intensity feature matrices into the classifier.

[0113] Among them, the intensity feature matrix is ​​used as the recognition data of the disguised expression recognition task and input into the classifier for recognition.

[0114] Step S422: identifying the intensity feature matrix based on the classifier to obtain multiple different types of disguised expression recognition data.

[0115] Among them, since disguised expressions are divided into many different types, when the classifier classifies the intensity feature matrix, it can obtain disguised expression recognition data corresponding to a variety of different disguised expressions.

[0116] Optionally, different methods can be used for recognition. For example, in some cases, the first frame containing more real emotions (onset frame) is considered. This is because it is known that the first frame is the disguised expression that the subject has been asked to make after receiving the instruction, but it is still the frame with the most real expressions mixed in the target video sequence. Then the real emotion gradually weakens until the disguised expression that is asked to make dominates. In order to accurately determine whether the disguised expression has occurred, the initial frame (onset frame) to the top frame (apex frame) and the initial frame to the end frame (offsetframe) can be used for recognition. Since the AU intensity will tend to be stable after a large change, the obtained feature matrix is ​​also stable after the disguised expression intensity reaches the maximum. The frame with the largest change is marked as the apex frame. The change of the image frame is consistent with the trend of the detected AU intensity stability. Both the initial frame and the top frame can extract spatiotemporal features to reflect the movement process of the disguised expression, thereby obtaining a variety of different disguised expression data.

[0117] Step S423: determining the disguised expression recognition result according to the plurality of disguised expression data.

[0118] The various types of disguised expression data may be integrated, classified, etc., to determine the final disguised expression recognition result for each intensity feature matrix. For example, the disguised expression recognition result may be a probability distribution diagram, table, or other file format of various types of disguised expressions.

[0119] exist Figure 6 In the illustrated embodiment, each type of disguised expression can be identified, thereby improving the completeness of the disguised expression recognition result.

[0120] Optionally, see Figure 7 , Figure 7 A detailed flow chart of step S200 is provided for an embodiment of the present application. Step S200 may also include steps S210-S230.

[0121] Step S210: determining a plurality of facial coding points in each frame of the target video sequence.

[0122] When the facial action of the target person in each frame of the image is encoded according to the facial action encoding system, a plurality of facial encoding points can be obtained.

[0123] Step S220: screening the plurality of facial code points according to the intensity requirement to obtain a plurality of target code points that meet the intensity requirement.

[0124] Among them, in order to improve the effectiveness of the extracted facial intensity features, multiple facial coding points can be screened according to the intensity requirements to screen out multiple points representing larger facial change intensity as target coding points, and subtract some multiple points representing smaller facial change intensity, which can effectively reduce the calculation amount of coding points and improve the coding speed and extraction efficiency.

[0125] Optionally, the number of target coding points can be selected and adjusted according to demand and actual conditions. In the embodiment of the present application, 17 points can be used as target coding points.

[0126] Step S230, extracting a plurality of facial motion intensity data of a plurality of target coding points.

[0127] Among them, facial motion intensity data of multiple target coding points in the facial coding system can be extracted in a variety of ways.

[0128] exist Figure 7 In the illustrated embodiment, the validity of the facial motion intensity data is improved through screening.

[0129] Optionally, see Figure 8 , Figure 8 A detailed flow chart of step S210 is provided for an embodiment of the present application. Step S210 may also include steps S211-S213.

[0130] Step S211, determining the reference points of the face in each frame of image.

[0131] Among them, when detecting the face of the target person in the image, in order to eliminate the interference of image noise and head movement on intensity detection, points in the face that are less affected by expression muscle movement can be extracted from the facial coding system as reference points.

[0132] For example, DRFM (Discriminative Response Map Fitting) may be used to extract multiple reference points, and the multiple reference points may be multiple points such as the inner eye and the nasal spine which are less affected by the expression muscle movement.

[0133] Step S212: performing facial alignment on the plurality of reference points to determine key areas of the face.

[0134] Among them, by calculating the transformation matrix for multiple reference points, the face of the target person can be aligned and the key area for identification in the aligned face can be determined.

[0135] Step S213, determining a plurality of facial coding points from the key area.

[0136] exist Figure 8 In the illustrated embodiment, the position in the face can be calibrated during detection to improve the accuracy of the facial encoding points, thereby improving the accuracy of facial intensity feature extraction.

[0137] See also Fig. 9 , Fig. 9 The present invention provides a schematic diagram of the structure of an expression recognition device according to an embodiment of the present invention. The expression recognition device 500 may include:

[0138] An extraction module 510 is used to extract a plurality of facial motion intensity data in each frame of an image in a target video sequence, wherein the target video sequence is any disguised expression video sequence in a disguised expression data set;

[0139] An encoding module 520, configured to encode the plurality of facial motion intensity data to obtain an intensity feature vector corresponding to each frame of the image;

[0140] The classification module 530 is used to input the intensity feature matrix determined by the multiple intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result.

[0141] In an optional implementation, the encoding module 520 may further include a conversion submodule and a generation submodule;

[0142] A conversion submodule, used for converting the plurality of facial motion intensity data corresponding to each frame of image into a plurality of sub-vectors;

[0143] The generating submodule is used to generate the intensity feature vector representing the local spatiotemporal features of the disguised expression in each frame of the image according to the multiple subvectors.

[0144] In an optional implementation, the classification module 530 may further include an adjustment submodule and a classification submodule;

[0145] An adjustment submodule, configured to adjust the plurality of intensity feature vectors to obtain the intensity feature matrix representing the global spatiotemporal features of the disguised expression in the target video sequence;

[0146] The classification submodule is used to input the intensity feature matrix into the classifier for classification to obtain the disguised expression recognition result.

[0147] In an optional implementation, the adjustment submodule may further include a determination unit, a processing unit, and a generation unit;

[0148] A determination unit, used to determine the target size;

[0149] A processing unit, configured to perform downsampling or interpolation processing on the plurality of intensity feature vectors according to the target size to obtain a plurality of target intensity feature vectors;

[0150] A generating unit is used to generate the intensity feature matrix according to a plurality of the target intensity feature vectors.

[0151] In an optional implementation, the classification submodule may further include an input unit, a recognition unit, and a determination unit;

[0152] An input unit, used for inputting each of the intensity feature matrices into the classifier;

[0153] A recognition unit, used for recognizing the intensity feature matrix based on the classifier to obtain multiple different types of disguised expression recognition data;

[0154] The determination unit is used to determine the disguised expression recognition result according to the multiple disguised expression data.

[0155] In an optional implementation, the extraction module 510 may further include a code point determination submodule, a screening submodule, and an extraction submodule;

[0156] A coding point determination submodule, used to determine a plurality of facial coding points in each frame of the target video sequence;

[0157] A screening submodule, configured to screen the plurality of facial code points according to the intensity requirement to obtain a plurality of target code points that meet the intensity requirement;

[0158] The extraction submodule is used to extract the facial motion intensity data of the target coding points.

[0159] In an optional implementation, the code point determination submodule may further include a first determination unit, an alignment unit, and a second determination unit;

[0160] A first determining unit, used to determine the reference point of the face in each frame of image;

[0161] An alignment unit, used for performing facial alignment on the plurality of reference points to determine a key area of ​​the human face;

[0162] The second determining unit is used to determine a plurality of facial coding points from the key area.

[0163] Since the principle of solving the problem by the expression recognition device 500 in the embodiment of the present application is similar to that of the embodiment of the expression recognition method described above, the implementation of the expression recognition device 500 in this embodiment can refer to the description in the embodiment of the expression recognition method described above, and the repeated parts will not be repeated.

[0164] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps of any one of the expression recognition methods provided in the present embodiment are executed.

[0165] In summary, the embodiments of the present application provide an expression recognition method, device, electronic device and computer-readable storage medium, which use facial movement intensity to characterize the facial muscle movement of the target person when the target person produces a disguised expression, extract and encode the feature information of the person's facial movement intensity, and recognize the generated feature matrix through a classifier, thereby effectively improving the accuracy of recognizing disguised expressions.

[0166] In several embodiments provided in the present application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices according to the multiple embodiments of the present application. In this regard, each box in the block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram, and the combination of the block diagrams, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0168] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, this embodiment also provides a computer program instruction stored in a readable storage medium, and when the computer program instruction is read and executed by a processor, the steps in any method described in the block data storage method are executed. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Ran Dom Access Memory), disk or optical disk and other media that can store program codes.

[0169] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0170] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

[0171] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for identifying disguised facial expressions, characterized in that: The method comprises: Extracting multiple facial motion intensity data from each frame of a target video sequence, wherein the target video sequence is any disguised expression video sequence in a disguised expression data set; Encoding the plurality of facial motion intensity data to obtain an intensity feature vector corresponding to each frame of image; Inputting the intensity feature matrix determined by the plurality of intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result; The encoding of the multiple facial motion intensity data to obtain the intensity feature vector corresponding to each frame image includes: converting the multiple facial motion intensity data corresponding to each frame image into multiple sub-vectors; and generating the intensity feature vector representing the local spatiotemporal characteristics of the disguised expression in each frame image based on the multiple sub-vectors.

2. The method according to claim 1, characterized in that The step of inputting the intensity feature matrix determined by the plurality of intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result includes: Performing adjustment processing on the plurality of intensity feature vectors to obtain the intensity feature matrix representing the global spatiotemporal features of the disguised expression in the target video sequence; The intensity feature matrix is ​​input into the classifier for classification to obtain the disguised expression recognition result.

3. The method according to claim 2, characterized in that The adjusting process of the plurality of intensity feature vectors to obtain the intensity feature matrix representing the global spatiotemporal features of the disguised expression in the target video sequence includes: Determine the target size; Performing downsampling or interpolation processing on the multiple intensity feature vectors according to the target size to obtain multiple target intensity feature vectors; The intensity feature matrix is ​​generated according to a plurality of the target intensity feature vectors.

4. The method according to claim 2, characterized in that: The step of inputting the intensity feature matrix into the classifier for classification to obtain the disguised expression recognition result includes: Inputting each of the intensity feature matrices into the classifier; Based on the classifier, the intensity feature matrix is ​​identified to obtain multiple different types of disguised expression recognition data; The disguised expression recognition result is determined according to the plurality of disguised expression data.

5. The method according to claim 1, characterized in that The step of extracting a plurality of facial motion intensity data from each frame of an image in a target video sequence comprises: Determine a plurality of facial coding points in each frame of the target video sequence; Screening the plurality of facial code points according to the intensity requirement to obtain a plurality of target code points that meet the intensity requirement; Extracting a plurality of facial motion intensity data of a plurality of target coding points.

6. The method according to claim 5, characterized in that The step of determining a plurality of facial coding points in each frame of the target video sequence comprises: Determine the reference points of the face in each frame of the image; Performing facial alignment on a plurality of the reference points to determine key areas of the human face; A plurality of facial encoding points are determined from the key area.

7. A disguised expression recognition device, characterized in that: The device comprises: An extraction module, used for extracting a plurality of facial motion intensity data in each frame of an image in a target video sequence, wherein the target video sequence is any disguised expression video sequence in a disguised expression data set; An encoding module, used for encoding the plurality of facial motion intensity data to obtain an intensity feature vector corresponding to each frame of image; A classification module, used for inputting the intensity feature matrix determined by the plurality of intensity feature vectors into a classifier for classification to obtain a disguised expression recognition result; The encoding module includes: a conversion submodule and a generation submodule; the conversion submodule is used to convert the multiple facial motion intensity data corresponding to each frame image into multiple subvectors; the generation submodule is used to generate the intensity feature vector representing the local spatiotemporal characteristics of the disguised expression in each frame image based on the multiple subvectors.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps in the method according to any one of claims 1 to 6 are executed.

9. A computer-readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps in the method according to any one of claims 1 to 6 are executed.

Citation Information

Patent Citations

  • Effective micro-expression automatic identification method

    CN103440509A

  • Real-time expression feature extraction and identification method

    CN104123545A

  • Human face emotion recognition method in complex environment

    CN107423707A

  • Video micro-expression recognition method and device, computer equipment and storage medium

    CN111738160A

  • Expression recognition method and system based on facial expression coding system and electronic equipment

    CN112016368A