Micro-expression fitting method and system based on multiple feature points

Abstract

The present invention provides a micro-expression fitting method and system based on multiple feature points. The method comprises: obtaining a reference image and an emotional image, wherein the reference image refers to an image without a stimulus source, and the emotional image refers to an image with a predetermined stimulus source; inputting the reference image and the emotional image into a facial detection neural network for facial detection to obtain a reference facial image and an emotional facial image respectively; inputting the reference facial image and the emotional facial image into a facial feature point marking network for facial feature point extraction to obtain reference feature points and emotional feature points respectively; calculating a facial micro-feature vector according to the position information of the reference feature points and the position information of the emotional feature points, and outputting the facial micro-feature vector that meets a predetermined threshold as a facial motion unit; and fitting the facial motion unit using a recursive neural network model to obtain predicted emotional features.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a micro-expression fitting method based on multiple feature points, wherein the method is implemented based on non-contact micro-expression fitting based on deep learning. Background Art

[0002] Micro-expression recognition is a non-contact psychological activity analysis method that uses stimulus sources to cause facial muscle changes, extracts facial features through deep learning neural networks, fuses multiple features of the same dimension, and combines psychological semantic analysis to obtain psychological features. During the interrogation process, the non-contact micro-expression recognition method can reduce the difficulty of interrogation and assist interrogators in making judgments.

[0003] The existing micro-expression recognition methods generally input the entire facial image into a neural network for emotion clustering, and obtain seven basic emotions such as happiness, sadness, fear, anger, disgust, surprise, and contempt. As the number of classified emotions continues to increase, the accuracy will be greatly reduced. The existing methods have great limitations and play a very limited auxiliary role in the interrogation process. Summary of the invention

[0004] The purpose of the present invention is to provide a micro-expression fitting method and system based on multiple feature points, so that interrogators can obtain the emotional characteristics of the detected person more quickly and accurately under lower computing power.

[0005] The invention provides a micro-expression fitting method based on multiple feature points, the method comprising: obtaining a reference image and an emotional image, wherein the reference image refers to an image without a stimulus source, and the emotional image refers to an image with a predetermined stimulus source; inputting the reference image and the emotional image into a facial detection neural network to perform facial detection to obtain a reference facial image and an emotional facial image respectively; inputting the reference facial image and the emotional facial image into a facial feature point marking network to extract facial feature points to obtain reference feature points and emotional feature points respectively; calculating a facial micro-feature vector according to position information of the reference feature points and position information of the emotional feature points, and outputting the facial micro-feature vector that meets a predetermined threshold as a facial motion unit; and fitting the facial motion unit using a recursive neural network model to obtain predicted emotional features.

[0006] In an embodiment according to the inventive concept, the face detection neural network may include: a first convolution layer, which performs convolution calculation on the input image through a convolution kernel of a predetermined size and a predetermined step size; a second convolution layer, which performs depth-separable convolution calculation on the result output by the first convolution layer through the convolution kernel and the predetermined step size; a first downsampling layer, which performs downsampling calculation on the result output by the second convolution layer; a second downsampling layer, which performs downsampling calculation on the result output by the first downsampling layer; and a fully connected layer, which performs nonlinear fitting on the result output by the second downsampling layer to obtain a facial image corresponding to the input image. Wherein, the first downsampling layer and the second downsampling layer both include: a first basic unit, whose step size is 2, the width and height of the output result are half of the width and height of the input, and the number of channels of the output result is twice the number of channels of the input; and a second basic unit, whose step size is 1, and the width, height and number of channels of the output result are the same as the width, height and number of channels of the input.

[0007] In an embodiment according to the inventive concept, the method may further include the step of training the facial detection neural network through a facial detection training set: during the facial detection training, when the intersection-and-union ratio coefficient of the output face frame and the expected face frame no longer increases, or the number of iterations reaches a predetermined count, the training is terminated.

[0008] In an embodiment according to the inventive concept, the position information of the reference feature points and the position information of the emotional feature points may include two-dimensional position information or three-dimensional position information.

[0009] In an embodiment according to the inventive concept, the facial motion unit may include at least one of an eyebrow arch motion unit, an eyebrow center motion unit, an eye movement unit, a nose movement unit, a lip movement unit, and a mouth corner movement unit.

[0010] In an embodiment according to the inventive concept, the method may further include: constructing an emotion training set and an emotion test set based on training samples, the training samples including the facial motion units obtained based on the reference image and the emotion image and the emotion features corresponding to the facial motion units; constructing a recursive neural network model, each layer of the recursive neural network model shares the same network parameters; training the recursive neural network model through the emotion training set; evaluating the trained recursive neural network through the emotion test set; and based on the evaluation results, using the recursive neural network that meets a predetermined accuracy for predicting emotion features.

[0011] The present invention also provides a system for micro-expression fitting based on multiple feature points, the system comprising: an image acquisition unit, which acquires a reference image and an emotional image, wherein the reference image refers to an image without a stimulus source, and the emotional image refers to an image with a predetermined stimulus source; a facial detection unit, which inputs the reference image and the emotional image into a facial detection neural network for facial detection to obtain a reference facial image and an emotional facial image respectively; a feature point extraction unit, which inputs the reference facial image and the emotional facial image into a facial feature point labeling network for facial feature point extraction to obtain reference feature points and emotional feature points respectively; a facial motion acquisition unit, which calculates a facial micro-feature vector based on the position information of the reference feature points and the position information of the emotional feature points, and outputs the facial micro-feature vector that meets a predetermined threshold as a facial motion unit; and an emotion recognition unit, which fits the facial motion unit using a recursive neural network model to obtain predicted emotional features.

[0012] In an embodiment according to the inventive concept, the face detection unit includes: a first convolution layer, which performs convolution calculation on the input image through a convolution kernel of a predetermined size and a predetermined step size; a second convolution layer, which performs depth-separable convolution calculation on the result output by the first convolution layer through the convolution kernel and the predetermined step size; a first downsampling layer, which performs downsampling calculation on the result output by the second convolution layer; a second downsampling layer, which performs downsampling calculation on the result output by the first downsampling layer; and a fully connected layer, which performs nonlinear fitting on the result output by the second downsampling layer to obtain a facial image corresponding to the input image. Wherein, the first downsampling layer and the second downsampling layer both include: a first basic unit, whose step size is 2, the width and height of the output result are half of the width and height of the input, and the number of channels of the output result is twice the number of channels of the input; and a second basic unit, whose step size is 1, and the width, height and number of channels of the output result are the same as the width, height and number of channels of the input.

[0013] Another aspect of the present invention provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the micro-expression fitting method based on multiple feature points as described above is implemented.

[0014] Another aspect of the present invention provides a computer device, characterized in that the computer device includes: a processor; a memory storing a computer program, and when the computer program is executed by the processor, the micro-expression fitting method based on multiple feature points as described above is implemented.

[0015] According to one or more aspects of the present invention, a micro-expression fitting method and system based on multiple feature points predicts emotions based on a reference image and an emotional image. Since the facial motion units obtained based on the reference image and the emotional image are used for emotion fitting, more accurate emotional features can be obtained.

[0016] According to one or more aspects of the present invention, a micro-expression fitting method and system based on multiple feature points uses a multi-convolutional neural network joint model including a facial detection neural network and a facial feature point marking network to pre-process an input image to obtain facial feature points. The method and system can be optimized according to the interrogation scenario, so that a high accuracy rate can be achieved even with a lightweight model under low computing power.

[0017] According to one or more aspects of the present invention, in a micro-expression fitting method and system based on multiple feature points, the lightweight network model used in the facial detection neural network has a small number of parameters and low computational complexity, and can achieve fast and accurate feature extraction of images through a lightweight backbone network structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other aspects, features and advantages of the present disclosure will be more clearly understood through the following detailed description in conjunction with the accompanying drawings, in which:

[0019] Figure 1 It is a flow chart of the micro-expression fitting method based on multiple feature points;

[0020] Figure 2 is a flowchart of facial feature point extraction obtained through a facial detection neural network and a facial feature point labeling network;

[0021] Figure 3 is a schematic diagram of facial micro-feature vectors;

[0022] Figure 4 is a flow chart of generating a recursive neural network model for fitting of facial motion units through training samples; and

[0023] Figure 5 It is a block diagram of a system for micro-expression fitting based on multiple feature points. DETAILED DESCRIPTION

[0024] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent to those of ordinary skill in the art. For example, the order of operations described herein is merely an example and is not limited to the order set forth herein, but changes that will be apparent to those of ordinary skill in the art may be made in addition to the operations that must be performed in a particular order. In addition, in order to improve clarity and brevity, descriptions of features and structures that will be known to those of ordinary skill in the art may be omitted. The features described herein may be implemented in different forms and will not be construed as being limited to the examples described herein. More specifically, the examples described herein have been provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those of ordinary skill in the art.

[0025] Embodiments of the present invention are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the accompanying drawings, wherein the same reference numerals refer to the same components throughout. The embodiments will be described below with reference to the accompanying drawings in order to explain the present invention.

[0026] Figure 1 It is a flow chart of the micro-expression fitting method based on multiple feature points; Figure 2 is a flowchart of facial feature point extraction obtained through a facial detection neural network and a facial feature point labeling network; Figure 3 It is a schematic diagram of facial micro-feature vector.

[0027] Reference Figure 1In step S01, a reference image is obtained, and in step S02, an emotional image is obtained, wherein the reference image refers to an image without a stimulus source, and the emotional image refers to an image with a predetermined stimulus source. Specifically, the reference image may refer to a facial image of the person being detected that is obtained alone without a stimulus source or a stimulus. Selecting a facial image when the person being detected is relatively stable as a reference image will be more conducive to the prediction of emotions. In addition, the emotional image may refer to an image selected from a video (e.g., an interrogation video). For example, an emotional image may be an image of a person being detected responding to a predetermined stimulus source. For example, the predetermined stimulus source may refer to a conversation, exhibits, testimony, or a specific action of the interrogator that occurs during the interrogation process. It should be noted that the purpose of selecting a predetermined stimulus source rather than an arbitrary stimulus source is to exclude the interference of the expression of the person being detected that is irrelevant to the interrogation process on the emotion prediction and only predict the emotional image corresponding to the predetermined stimulus source. On the other hand, selecting a predetermined stimulus source can exclude the interference of unconscious or subconscious expressions on the emotion prediction. In particular, the above examples of predetermined stimulus sources are only examples and are not intended to be limiting. In another embodiment, the reference image and the emotional image may also be images selected from the same video according to the scene conditions. In an embodiment, the reference image and the emotional image may be two-dimensional image information, or may be three-dimensional image information with depth features captured and synthesized by dual cameras, multiple cameras or depth cameras.

[0028] In an embodiment of the present invention, a multi-convolutional neural network joint model is used to preprocess the input image to obtain facial feature points. The multi-convolutional neural network joint model includes a facial detection neural network and a facial feature point marking network improved based on the convolutional neural network. The steps of obtaining facial feature points will be further explained and illustrated below.

[0029] In step S03, the reference image and the emotional image are respectively input into the facial detection neural network for facial detection to obtain the reference facial image and the emotional facial image respectively. Figure 2 , input an image including a human face, and the image can output the position coordinate information of the upper left corner and the lower right corner of the human face through the face detection neural network. Then the image is cropped according to the position coordinate information. The specific structure and generation method of the face detection neural network will be further explained and illustrated below.

[0030] Table 1 BoxNet model structure

[0031]

[0032] As shown in Table 1 above, the facial detection neural network adopts a lightweight network model (or called a box network model, BoxNet model). The input image Image of the BoxNet model is an image of 320 wide × 180 high pixels, and the number of channels is 3. In an embodiment, the input image Image of the BoxNet model can be obtained by scaling the image of 1920 × 1080 pixels from the acquisition camera. The size of the input image Image described here is only an example. For example, in another embodiment, the size of the input image can be (230 ~ 400) × (130 ~ 225). The size of the input image should not be too large to avoid an increase in the amount of calculation resulting in a decrease in the detection speed; nor should it be too small to avoid a decrease in the accuracy.

[0033] The structure of the BoxNet model is: the first convolution layer Conv1, the second convolution layer S-Conv1, the first downsampling layer Block1, and the second downsampling layer Block2. The first convolution layer Conv1 performs convolution calculation on the input image through a convolution kernel of a predetermined size and a predetermined step size. For example, the size of the convolution kernel can be 3×3, the step size is 1, and the number of repetitions is 1. The size of the output result is 160×90, and the number of channels is 16.

[0034] The second convolution layer S-Conv1 can perform depth-separable convolution calculation on the output of the first convolution layer by using the same convolution kernel and predetermined step size as the first convolution layer Conv1. In this case, the size of the output result of the second convolution layer S-Conv1 is 80×45 and the number of channels is 24.

[0035] It should be noted that the size and step size of the convolution kernel of the first convolution layer Conv1 and the second convolution layer S-Conv1 can be appropriately adjusted according to the situation, and are not limited to the situation where the convolution kernel and the predetermined step size of the two are the same.

[0036] The first convolutional layer Conv1 and the second convolutional layer S-Conv1 may be referred to as First_Conv.

[0037] The first downsampling layer Block1 performs downsampling calculation on the output of the second convolutional layer, and the second downsampling layer Block2 performs downsampling calculation on the output of the first downsampling layer.

[0038] Among them, the first downsampling layer Block1 and the second downsampling layer Block2 both include: a first basic unit, whose step size is 2, the width and height of the output result are half of the width and height of the input, and the number of channels of the output result is twice the number of channels of the input; and a second basic unit, whose step size is 1, the width, height and number of channels of the output result are the same as the width, height and number of channels of the input.

[0039] For example, referring to Table 1, in the first downsampling layer Block1, the first basic unit is executed once with a stride of 2. Therefore, the output of the first basic unit after being executed once is half of its input, that is, 40×23 pixels, and the number of channels is twice that of the input, that is, 48. The width, height, and number of channels of the output of the second basic unit are the same as those of the output of the first basic unit. In the second downsampling layer Block2, the first basic unit is executed twice with a stride of 2. Therefore, the output of the first basic unit after being executed twice is one-quarter of its input, that is, 10×6 pixels, and the number of channels is four times that of the input, that is, 192. The width, height, and number of channels of the output of the second basic unit are the same as those of the output of the first basic unit.

[0040] The fully connected layer FC performs nonlinear fitting on the output of the second downsampling layer Block2 to obtain a facial image corresponding to the input image. For example, the downsampled 192-channel result is sent to a fully connected layer of 500 neural units and then to a fully connected layer of 4 neural units, and the extracted result (e.g., feature map) is nonlinearly fitted to output the coordinates of the upper left corner and the lower right corner of the face frame.

[0041] The first down-sampling layer Block1 and the second down-sampling layer Block2 may be referred to as Second_Conv.

[0042] In an embodiment, the method further comprises the step of training the face detection neural network by a face detection training set. During the training process, the number of iterations is controlled by setting parameters (e.g., epoch parameters, etc.), and the training is terminated when a specific condition is reached. For example, the intersection-over-union ratio coefficient (IOU_accuracy) between the output face frame and the expected face frame can be defined as a judgment condition for model training. For example, the intersection-over-union ratio coefficient can be the ratio of the intersection and union of the output face frame and the expected face frame position. The intersection-over-union ratio (IOU) parameter of the model is set to 0.1-0.5, preferably, it can be set to 0.3. That is, when it is greater than this value, the face position detection is deemed to be successful, and face tracking detection can be well implemented. In an embodiment, when the number of iterations is less than 300, the weight coefficients between the neurons are adjusted by the back propagation algorithm, and the next iteration is restarted; when the number of iterations is equal to 300 times, or the intersection-over-union ratio coefficient converges (e.g., no longer increases), the training is terminated.

[0043] In an example embodiment, the BoxNet model can be used for face recognition during an interrogation. The intensity of the ambient light source in the interrogation environment is stable, and there will be no movement, flickering, or changes in brightness of the light source. The surrounding walls are of a single color, and the person being tested sits on a chair, so the range of facial movement during the test is small. Therefore, in view of the characteristics of the interrogation environment, the BoxNet model uses an optimized network structure, omits the convolutional layer that copes with complex environmental backgrounds and light source changes, and optimizes the intersection-over-union ratio coefficient of the face frame. In this case, due to the small number of parameters and low computational complexity of the BoxNet model, it is possible to achieve fast and accurate feature extraction of images through a lightweight backbone network structure. Therefore, compared with the Haar cascade classifier based on OpenCV, this model can have a faster detection speed and higher detection accuracy.

[0044] Return to reference Figure 1 In step S04, the reference facial image and the emotional facial image are respectively input into the facial feature point labeling network to extract facial feature points to obtain reference feature points and emotional feature points respectively. Figure 2 Based on the facial image generated by the facial detection neural network, the facial feature point labeling network obtains feature points corresponding to the facial image. In an exemplary embodiment according to the present invention, the facial feature point labeling network can be a network structure of a shuffle unit, and the convolution module contained therein can adopt a ShuffleNet structure. The facial feature point labeling network takes a facial image (e.g., a reference facial image and an emotional facial image) as an input layer and outputs facial feature points (e.g., reference feature points and emotional feature points).

[0045] The facial feature point labeling network can use a high-performance convolutional neural network model, whose input image is a 160×160 pixel face image, and the output is the position information of 68 feature points of the face. The size of the input image described here is only an example. For example, in another embodiment, the size of the input image can be 110×110 to 210×210. Compared with the feature point extraction method of the commonly used computer vision tool library DLib, this model has a faster extraction speed and higher accuracy.

[0046] The recognition speed of the multi-convolutional neural network joint model is about 10ms on a host configured as i7-10700, and the accuracy rate is over 90% when tested using the Face Detection Data Set and Benchmark (FDDB) face detection algorithm evaluation standard.

[0047] In an embodiment, the position information of the reference feature points and the position information of the emotional feature points include two-dimensional position information. In the case of using dual cameras or multiple cameras, the position information of the reference feature points and the position information of the emotional feature points may include three-dimensional position information.

[0048] Return to reference Figure 1 In step S05, a facial micro-feature vector is calculated based on the position information of the reference feature point and the position information of the emotional feature point, and the facial micro-feature vector that meets a predetermined threshold is output as a facial motion unit.

[0049] Reference Figure 3 , the facial micro-feature vector can be defined as follows:

[0050] The eyebrow arch micro-feature 201 is a vertical displacement vector of the eyebrow arch feature point, which can represent the eyebrow arch motion unit;

[0051] The eyebrow center micro-feature 202 is a vertical displacement vector of the eyebrow center feature point, which can represent an eyebrow center motion unit;

[0052] The eye micro-feature 203 is a vertical displacement vector of an eye feature point, which can represent an eye movement unit, such as staring, squinting, closing eyes, etc.;

[0053] The nose micro-feature 204 is a vertical displacement vector of a nose feature point, which can represent a nose motion unit, for example, wrinkling the nose;

[0054] The lip micro-feature 205 is a vertical displacement vector of the lip feature point, which can represent a lip motion unit;

[0055] The mouth corner micro-feature 206 is the horizontal displacement vector and the vertical displacement vector of the mouth corner feature point, which can represent the mouth corner movement unit, for example, the mouth corner rises, the mouth corner drops, the mouth corner stretches, the mouth corner contracts, etc.

[0056] In the embodiment, taking the eyebrow micro-feature as an example, when the eyebrow micro-feature vector is upward and the length is greater than a predetermined threshold (for example, two pixels), it can be understood as triggering the upward movement of the eyebrow in the eyebrow motion unit, thereby realizing the conversion of machine language (changes in facial feature points) into facial semantics.

[0057] Return to reference Figure 1 In step S06, the facial motion unit is fitted using a recursive neural network model to obtain predicted emotional features. For example, the facial motion unit can be fitted using a recursive neural network to obtain seven basic emotions such as happiness, sadness, fear, anger, disgust, surprise, and contempt. In addition, because the facial motion unit is used for emotional fitting, complex emotions such as embarrassment, guilt, shyness, and pride can also be obtained.

[0058] Figure 4 is a flow chart of generating a recurrent neural network model for fitting facial motion units through training samples.

[0059] The micro-expression fitting method based on multiple feature points also includes the steps of constructing a recursive neural network (RNN) model and training it.

[0060] Reference Figure 4 In step S601, an emotion training set and an emotion test set are constructed based on training samples, wherein the training samples include facial motion units obtained based on reference images and emotion images and emotion features corresponding to the facial motion units. In an embodiment, the training samples may be images and emotion features of samples obtained in an interrogation environment.

[0061] In step S602, a recursive neural network model is constructed.

[0062] In step S603, the recursive neural network model is trained using the emotion training set. For example, in one iteration, the recursive neural network model is trained using 50 sets of training sets and outputs 600 emotion values. Among the 600 emotion values, the proportion of correct emotions to the total number of emotions is used to determine whether the classification requirement is met. For example, the prediction accuracy of the recursive neural network model is required to be greater than 90%.

[0063] In step S604, when the iteration condition is not met, for example, when the prediction accuracy is different from the prediction accuracy of the previous iteration (for example, greater than a predetermined accuracy change threshold), or when the predetermined maximum number of iterations is not reached, in step S605, the weight coefficients between the neurons are reversely adjusted to improve the accuracy of each iteration. In an exemplary embodiment, each layer of the recursive neural network model can share the same weight parameter, thereby reducing the amount of calculation, increasing the calculation speed, and making the model more generalizable. For example, a function encapsulated in the program code can be used for automatic adjustment. After adjusting the network parameters, return to step S603 for the next training.

[0064] Through continuous iteration, when the iteration conditions are met, for example, in step S604, when the prediction accuracy no longer changes (for example, less than or equal to a predetermined accuracy change threshold) or the number of iterations reaches a predetermined maximum number of iterations, the training of the recursive neural network model is terminated.

[0065] In step S606, it is determined whether the prediction accuracy of the recursive neural network model meets the classification requirements. If not, it is discarded; if it meets the classification requirements, it is output to the next step.

[0066] In step S607, the trained recursive neural network is evaluated using the emotion test set. Based on the evaluation result, the recursive neural network that meets the predetermined accuracy is used for fitting the facial motion unit. For example, when the emotion features given by the trained recursive neural network to the facial motion units in the emotion test set have a detection accuracy rate of more than 90%, the recursive neural network can be used for predicting the emotion features.

[0067] Figure 5 It is a block diagram of a system for micro-expression fitting based on multiple feature points.

[0068] exist Figure 5 In the embodiment, the system for micro-expression fitting based on multiple feature points includes: an image acquisition unit 100, a face detection unit 200, a feature point extraction unit 300, a face movement acquisition unit 400 and an emotion recognition unit 500.

[0069] The image acquisition unit 100 acquires a reference image and an emotional image. The image acquisition unit 100 may be configured to perform a reference Figure 1 The steps S01 and S02 are described above, and thus redundant descriptions are omitted here.

[0070] The face detection unit 200 inputs the reference image and the emotion image into the face detection neural network to perform face detection to obtain the reference face image and the emotion face image respectively. The face detection unit 200 may be configured to perform a reference Figure 1 This is similar to step S03 described in Table 1, so redundant description is omitted here.

[0071] The feature point extraction unit 300 inputs the reference facial image and the emotional facial image into the facial feature point labeling network to extract facial feature points to obtain reference feature points and emotional feature points respectively. The feature point extraction unit 300 may be configured to perform a reference Figure 1 The step S04 described above is described, and therefore redundant description is omitted here.

[0072] The facial motion acquisition unit 400 calculates a facial micro-feature vector according to the position information of the reference feature point and the position information of the emotional feature point, and outputs the facial micro-feature vector that meets a predetermined threshold as a facial motion unit. Figure 1 and Figure 2 The step S05 described above is described, and therefore redundant description is omitted here.

[0073] The emotion recognition unit 500 uses a recursive neural network model to fit the facial motion unit to obtain predicted emotion features. In addition, the emotion recognition unit 500 can also train the recursive neural network model according to the training samples. The emotion recognition unit 500 can be configured to perform a reference Figure 1 Step S06 described and referenced Figure 4 The steps S601 to S607 for training the recursive neural network model are described, so redundant descriptions are omitted here.

[0074] The method and system for micro-expression fitting based on multiple feature points are described above with reference to the accompanying drawings. The method and system for micro-expression fitting based on multiple feature points predict emotions based on a reference image and an emotional image. Since the facial motion units obtained based on the reference image and the emotional image are used for emotion fitting, more accurate emotional features can be obtained.

[0075] In addition, the micro-expression fitting method and system based on multiple feature points uses a multi-convolutional neural network joint model including a facial detection neural network and a facial feature point marking network to pre-process the input image to obtain facial feature points. The method and system can be optimized according to the interrogation scenario, so that a high accuracy rate can be achieved even with a lightweight model under low computing power.

[0076] On the other hand, in the micro-expression fitting method and system based on multiple feature points, the lightweight network model used in the facial detection neural network has a small number of parameters and low computational complexity, and can achieve fast and accurate feature extraction of images through a lightweight backbone network structure.

[0077] The above has been referred to Figures 1 to 5 A method and system for micro-expression fitting based on multiple feature points according to an exemplary embodiment of the present disclosure are described.

[0078] Figure 5 Each unit in the system for micro-expression fitting based on multiple feature points shown can be configured as software, hardware, firmware, or any combination of the above items to perform specific functions. For example, each unit may correspond to a dedicated integrated circuit, or to a pure software code, or to a module combining software and hardware. In addition, one or more functions implemented by each unit may also be uniformly executed by components in a physical entity device (e.g., a processor, a client, or a server, etc.).

[0079] In addition, refer to Figure 1 The method for micro-expression fitting based on multiple feature points described can be implemented by a program (or instruction) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the method for micro-expression fitting based on multiple feature points according to the present disclosure.

[0080] The computer program in the computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps. The contents of these additional steps and further processing have been described in reference to Figure 1 It is mentioned in the description of the related method, so it will not be repeated here to avoid repetition.

[0081] It should be noted that the various units in the device for micro-expression fitting based on multiple feature points according to the exemplary embodiment of the present disclosure can completely rely on the operation of the computer program to realize the corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding function.

[0082] on the other hand, Figure 5 The various units shown may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segment for performing the corresponding operation may be stored in a computer-readable medium such as a storage medium, so that the processor may perform the corresponding operation by reading and running the corresponding program code or code segment.

[0083] For example, an exemplary embodiment of the present disclosure may also be implemented as a computing device, which includes a storage component and a processor, wherein a set of computer executable instructions is stored in the storage component. When the set of computer executable instructions is executed by the processor, the method of micro-expression fitting based on multiple feature points according to the exemplary embodiment of the present disclosure is executed.

[0084] Specifically, the computing device can be deployed in a server or client, or can be deployed on a node device in a distributed network environment. In addition, the computing device can be a PC, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing the above instruction set.

[0085] Here, the computing device is not necessarily a single computing device, but may also be any device or circuit collection that can execute the above instructions (or instruction sets) individually or jointly. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0086] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0087] According to the exemplary embodiment of the present disclosure, some operations described in the method for micro-expression fitting based on multiple feature points can be implemented by software, some operations can be implemented by hardware, and in addition, these operations can also be implemented by a combination of software and hardware.

[0088] The processor may execute instructions or codes stored in one of the storage components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0089] The storage component may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage component may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage component and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the storage component.

[0090] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.

[0091] The method for micro-expression fitting based on multiple feature points according to an exemplary embodiment of the present disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operated according to non-exact boundaries.

[0092] Therefore, refer to Figure 1 The method for micro-expression fitting based on multiple feature points may be implemented by a system including at least one computing device and at least one storage device storing instructions.

[0093] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for executing a method for micro-expression fitting based on multiple feature points according to an exemplary embodiment of the present disclosure, and a computer executable instruction set is stored in the storage device. When the computer executable instruction set is executed by the at least one computing device, the execution reference Figure 1 The described method of micro-expression fitting based on multiple feature points.

[0094] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A micro-expression fitting method based on multiple feature points, the method comprising: Acquire a reference image and an emotional image, wherein the reference image refers to an image without a stimulus source, and the emotional image refers to an image with a predetermined stimulus source; The reference image and the emotional image are respectively input into a facial detection neural network for facial detection to obtain a reference facial image and an emotional facial image respectively; The reference facial image and the emotional facial image are respectively input into the facial feature point labeling network to extract facial feature points to obtain reference feature points and emotional feature points respectively; Calculating facial micro-feature vectors based on the position information of the reference feature points and the position information of the emotional feature points, and outputting the facial micro-feature vectors that meet a predetermined threshold as facial motion units; and The facial motion unit is fitted using a recursive neural network model to obtain predicted emotional features. The facial detection neural network is a BoxNet model, the facial feature point marking network is a convolutional neural network model using a ShuffleNet structure, and The recursive neural network model is trained using the following steps: Constructing an emotion training set and an emotion test set based on training samples, wherein the training samples include the facial motion units obtained based on the reference image and the emotion image and the emotion features corresponding to the facial motion units; Constructing a recursive neural network model, wherein each layer of the recursive neural network model shares the same network parameters; Train the recurrent neural network model using the sentiment training set; Evaluate the trained recurrent neural network using the sentiment test set; and According to the evaluation results, the recursive neural network that meets the predetermined accuracy is used for the prediction of emotional features.

2. The method according to claim 1, wherein the face detection neural network comprises: The first convolution layer performs convolution calculation on the input image through a convolution kernel of a predetermined size and a predetermined step size; A second convolution layer, performing a depth-separable convolution calculation on a result output by the first convolution layer by using the convolution kernel and the predetermined step size; The first downsampling layer performs downsampling calculation on the output of the second convolutional layer; The second downsampling layer performs downsampling calculation on the output of the first downsampling layer; as well as A fully connected layer performs nonlinear fitting on the output of the second downsampling layer to obtain a facial image corresponding to the input image. Wherein, the first downsampling layer and the second downsampling layer both include: A first basic unit, whose stride is 2, has an output result whose width and height are half of the input width and height, and has an output result whose number of channels is twice the number of input channels; and The second basic unit has a stride of 1, and the width, height, and number of channels of the output result are the same as the width, height, and number of channels of the input.

3. The method according to claim 2, further comprising: The step of training the face detection neural network through a face detection training set, wherein during the training process, when the intersection-over-union ratio coefficient of the output face frame and the expected face frame no longer increases, or the number of iterations reaches a predetermined count, the training is terminated. 4 . The method according to claim 1 , wherein the position information of the reference feature points and the position information of the emotional feature points comprise two-dimensional position information or three-dimensional position information.

5. The method according to claim 1, wherein the facial movement unit comprises: At least one of the eyebrow arch movement unit, the eyebrow center movement unit, the eye movement unit, the nose movement unit, the lip movement unit, and the mouth corner movement unit.

6. A system for micro-expression fitting based on multiple feature points, the system comprising: An image acquisition unit, which acquires a reference image and an emotional image, wherein the reference image refers to an image without a stimulus source, and the emotional image refers to an image with a predetermined stimulus source; A face detection unit, inputting the reference image and the emotion image into a face detection neural network for face detection to obtain a reference face image and an emotion face image respectively; A feature point extraction unit, which inputs the reference facial image and the emotional facial image into a facial feature point labeling network to extract facial feature points to obtain reference feature points and emotional feature points respectively; A facial motion acquisition unit calculates a facial micro-feature vector according to the position information of the reference feature point and the position information of the emotional feature point, and outputs the facial micro-feature vector that meets a predetermined threshold as a facial motion unit; and The emotion recognition unit uses a recursive neural network model to fit the facial motion unit to obtain predicted emotion features. The facial detection neural network is a BoxNet model, the facial feature point marking network is a convolutional neural network model using a ShuffleNet structure, and The recursive neural network model is trained using the following steps: Constructing an emotion training set and an emotion test set based on training samples, wherein the training samples include the facial motion units obtained based on the reference image and the emotion image and the emotion features corresponding to the facial motion units; Constructing a recursive neural network model, wherein each layer of the recursive neural network model shares the same network parameters; Train the recurrent neural network model using the sentiment training set; Evaluate the trained recurrent neural network using the sentiment test set; and According to the evaluation results, the recursive neural network that meets the predetermined accuracy is used for the prediction of emotional features.

7. The system according to claim 6, wherein the face detection unit comprises: The first convolution layer performs convolution calculation on the input image through a convolution kernel of a predetermined size and a predetermined step size; A second convolution layer, performing a depth-separable convolution calculation on a result output by the first convolution layer by using the convolution kernel and the predetermined step size; The first downsampling layer performs downsampling calculation on the output of the second convolutional layer; The second downsampling layer performs downsampling calculation on the output of the first downsampling layer; as well as A fully connected layer performs nonlinear fitting on the output of the second downsampling layer to obtain a facial image corresponding to the input image. Wherein, the first downsampling layer and the second downsampling layer both include: A first basic unit, whose stride is 2, has an output result whose width and height are half of the input width and height, and has an output result whose number of channels is twice the number of input channels; and The second basic unit has a stride of 1, and the width, height, and number of channels of the output result are the same as the width, height, and number of channels of the input.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the micro-expression fitting method based on multiple feature points as described in any one of claims 1 to 5 is implemented.

9. A computer device, characterized in that: The computer device comprises: processor; A memory storing a computer program, which, when executed by a processor, implements the micro-expression fitting method based on multiple feature points as claimed in any one of claims 1 to 5.

Images