Micro-expression extraction method and device, computer equipment and readable storage medium

Abstract

The invention provides a micro-expression extraction method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring a face image sequence comprising a plurality of image frames; performing feature extraction on each image frame to obtain a plurality of face features corresponding to each image frame; the face features at least comprise at least two of a face expression feature, a face action feature, a face posture feature and a face sight line feature; merging the plurality of face features and the time sequence information corresponding to each image frame to obtain time sequence feature information; and performing feature fusion on the multiple pieces of time sequence feature information to generate fused feature information for acquiring micro-expression information corresponding to the face image sequence, and completing micro-expression extraction of the face image sequence. According to the method, through extraction and fusion of multiple face features, the micro-expression of the customer can be quickly and accurately captured in face examination scenes such as loan, account opening and underwriting in the financial industry, so that the fraud risk is effectively reduced, and the face examination efficiency and the service quality are improved.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, which is applied in the financial field, and in particular to a micro-expression extraction method, device, computer equipment and readable storage medium. Background Art

[0002] In the financial field, face-to-face interviews are an important part of assessing the credit status and repayment ability of borrowers. Risk warnings can timely detect potential credit risks, fraud risks, etc., avoid the issuance of bad loans, and thus protect the asset security of financial institutions. In existing face-to-face interview scenarios, the interviewer observes the applicant's words, deeds, facial expressions, etc. during the interview assessment to determine whether the applicant is honest and trustworthy, and whether there is any suspicion of fraud. However, it is inevitable that the applicant's micro-expressions will not be observed through naked eye observation, resulting in failure to promptly identify risky users, leading to inevitable losses caused by incorrect loan issuance.

[0003] Therefore, there is an urgent need for a micro-expression extraction method to quickly determine whether the person being interviewed poses a risk in scenarios such as face-to-face interviews. Summary of the invention

[0004] This application provides a method, device, computer equipment and readable storage medium for extracting micro-expressions, which aims to solve the problem that in the existing face-to-face interview scenario, the interviewer judges whether the applicant is honest and trustworthy and whether there is any suspicion of fraud by observing the applicant's words, deeds, facial expressions, etc. during the interview assessment. However, it is inevitable that the applicant's micro-expressions will not be observed through naked eye observation, resulting in failure to timely confirm the risky users, resulting in the problem of inevitable losses caused by wrong loans. The method provided can quickly judge the customer's current emotional state through micro-expression extraction, which can effectively help enterprises improve service quality and customer satisfaction in the financial field.

[0005] In a first aspect, the present application provides a micro-expression extraction method, comprising:

[0006] Acquire a facial image sequence to be extracted, the facial image sequence including a plurality of image frames;

[0007] Perform feature extraction on each image frame to obtain multiple facial features corresponding to each image frame; the facial features include at least two of facial expression features, facial action features, facial posture features, and facial sight features;

[0008] Acquire the time sequence information corresponding to each image frame in the face image sequence, merge the multiple face features and the time sequence information corresponding to each image frame, and acquire the time sequence feature information corresponding to each image frame;

[0009] Fusing multiple time series feature information to generate fused feature information;

[0010] The micro-expression information corresponding to the facial image sequence is obtained according to the fused feature information, and the micro-expression extraction of the facial image sequence is completed.

[0011] In some embodiments, feature extraction is performed on each image frame to obtain multiple facial features corresponding to each image frame; the facial features include at least two of facial expression features, facial action features, facial posture features and facial sight line features, including: if the facial features include facial expression features, each image frame is input into a preset micro-expression extraction model, the micro-expression extraction model parses the image frame, and outputs the facial expression features; and / or, if the facial features include the facial action features, each image frame is input into a preset facial action unit model, the facial action unit model parses the image frame, and outputs the facial action features; and / or, if the facial features include the facial posture features, each image frame is input into a preset facial posture model, the facial posture model parses the image frame, and outputs the facial posture features; and / or, if the facial features include the facial sight line features, each image frame is input into a preset facial sight line model, the facial sight line model parses the image frame, and outputs the facial sight line features.

[0012] Exemplarily, facial features include at least facial expression features and facial action features. Before feature extraction is performed on each image frame, the method further includes: inputting each image frame into a micro-expression extraction model and a facial action unit model to be trained, respectively, to obtain predicted expression features output by the micro-expression extraction model and predicted action features output by the action unit model; obtaining supervisory feature information corresponding to the image frame; the supervisory feature information includes at least supervisory expression features and supervisory action features; and completing the training of the micro-expression extraction model and the facial action unit model according to the supervisory expression features, the supervisory action features, the predicted expression features and the predicted action features.

[0013] It should be noted that, in some embodiments, the training of the micro-expression extraction model and the facial action unit model is completed according to the supervised expression features, supervised action features, predicted expression features and predicted action features, including: calculating the first Euclidean distance between the predicted expression features and the corresponding supervised expression features; calculating the second Euclidean distance between the predicted action features and the corresponding supervised action features; calculating the loss function of the supervised expression features, supervised action features, predicted expression features, predicted action features and the first Euclidean distance and the second Euclidean distance; and completing the training of the micro-expression extraction model and the facial action unit model according to the loss function.

[0014] In some embodiments, it is characterized in that the timing information includes position coding information; the feature vectors corresponding to multiple facial features are spliced ​​to obtain a comprehensive feature vector; the timing information corresponding to each image frame in the facial image sequence is obtained, and the multiple facial features and timing information corresponding to each image frame are merged to obtain the timing feature information corresponding to each image frame, including: obtaining the position coding information corresponding to each image frame in the facial image sequence; splicing the feature vectors corresponding to multiple facial features to obtain a comprehensive feature vector; merging the comprehensive feature vector and the position coding information corresponding to each image frame to obtain the timing feature information corresponding to each image frame.

[0015] In some embodiments, before extracting features from each image frame, it also includes: calculating the effective frame ratio corresponding to each image frame; confirming an image frame whose effective frame ratio is less than a preset ratio as a target frame; and removing the target frame from multiple image frames.

[0016] In some embodiments, after obtaining micro-expression information corresponding to the facial image sequence based on the fused feature information, it also includes: obtaining a risk assessment result corresponding to the micro-expression information, the risk assessment result is used to confirm whether there is a preset suspicious action in the micro-expression information; if it is confirmed that there is a suspicious action in the micro-expression information based on the risk assessment result, generating warning information corresponding to the facial image sequence.

[0017] In a second aspect, the present application provides a micro-expression extraction device, comprising:

[0018] A sequence acquisition unit, used to acquire a facial image sequence to be extracted, wherein the facial image sequence includes a plurality of image frames;

[0019] A feature extraction unit, configured to extract features from each of the image frames to obtain a plurality of facial features corresponding to each of the image frames; the facial features include at least two of facial expression features, facial action features, facial posture features, and facial sight features;

[0020] A feature merging unit, used to obtain the time sequence information corresponding to each of the image frames in the face image sequence, and to merge the multiple face features and the time sequence information corresponding to each of the image frames to obtain the time sequence feature information corresponding to each of the image frames;

[0021] A feature fusion unit, used for fusing the plurality of time series feature information to generate fused feature information;

[0022] The expression extraction unit is used to obtain micro-expression information corresponding to the facial image sequence according to the fusion feature information, and complete the micro-expression extraction of the facial image sequence.

[0023] In a third aspect, the present application further provides a computer device, including:

[0024] Memory and processor;

[0025] The memory is used to store computer programs;

[0026] The processor is used to execute the computer program and implement the steps of the micro-expression extraction method described in the first aspect when executing the computer program.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the micro-expression extraction method described in the first aspect above.

[0028] The embodiment of the present application provides a method, device, computer equipment and readable storage medium for extracting micro-expressions. First, the method obtains a facial image sequence to be extracted, including multiple image frames. Then, feature extraction is performed on each image frame to obtain at least two of the following features: facial expression features: such as smiling, frowning, etc. Facial action features: such as nodding, shaking head, etc. Facial posture features: such as the tilt angle of the head, facial orientation, etc. Facial sight features: such as the gaze direction of the eyes, etc. Then, the multiple facial features corresponding to each image frame and its temporal features in the facial image sequence are merged to generate temporal feature information of each image frame. The temporal feature information reflects the dynamic changes of the face over time. Further, the multiple temporal feature information is feature fused to generate fused feature information. The purpose of feature fusion is to organically combine different types of features to form a comprehensive feature representation. Finally, according to the generated fused feature information, the micro-expression information corresponding to the facial image sequence is obtained to complete the micro-expression extraction. Micro-expression information can be used to judge the emotional state of the customer, such as happiness, anger, tension, etc. The proposed micro-expression extraction method can quickly and accurately capture customers' micro-expressions in face-to-face interview scenarios by extracting and fusing multiple facial features, thereby effectively reducing fraud risks and improving face-to-face interview efficiency and service quality. The automated and intelligent features of the method make it have broad application prospects in the financial field and other related fields.

[0029] The provided method has the following beneficial effects:

[0030] 1. Improve judgment accuracy and reliability: By extracting and integrating multiple facial features, customers’ expressions and emotional states can be judged more comprehensively and accurately, reducing omissions in naked eye observation.

[0031] 2. Reduce the risk of erroneous loan issuance: During the face-to-face review process, customers suspected of fraud can be identified in a timely manner through quick and accurate emotional state judgment, thereby reducing losses caused by erroneous loan issuance.

[0032] 3. Improve service quality: Financial institutions can use this method to improve the efficiency of face-to-face review, reduce the workload of reviewers, and improve service quality and customer experience. A fast and accurate face-to-face review process can reduce customer waiting time and improve their satisfaction.

[0033] 4. Multi-scenario application: This method is not only suitable for face-to-face interviews in the financial field, but can also be applied to other scenarios that require emotion recognition and behavior analysis, such as security monitoring, customer service, etc.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 is a schematic flow chart of the steps of a micro-expression extraction method provided in one embodiment of the present application;

[0037] Figure 2 is a schematic flow chart of the steps of another micro-expression extraction method provided by an embodiment of the present application;

[0038] Figure 3 This is a schematic flow chart of the steps of a method for generating early warning information provided by an embodiment of the present application;

[0039] Figure 4 is a structural schematic diagram of a micro-expression extraction device provided in one embodiment of the present application;

[0040] Figure 5 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.

[0041] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0044] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.

[0045] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0046] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0047] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0048] In the financial field, face-to-face interviews are an important part of assessing the credit status and repayment ability of borrowers. Risk warnings can timely detect potential credit risks, fraud risks, etc., avoid the issuance of bad loans, and thus protect the asset security of financial institutions. In existing face-to-face interview scenarios, the interviewer observes the applicant's words, deeds, facial expressions, etc. during the interview assessment to determine whether the applicant is honest and trustworthy, and whether there is any suspicion of fraud. However, by observing with the naked eye, the applicant's micro-expressions may not be observed, resulting in failure to promptly identify risky users, leading to inevitable losses caused by incorrect loan issuance.

[0049] Current risk warning algorithms in the industry use optical flow algorithms to extract motion features. However, this method is strongly affected by interference factors such as posture and lighting, resulting in less than ideal results in micro-expression extraction.

[0050] Therefore, there is an urgent need for a micro-expression extraction method to quickly determine whether the person being interviewed poses a risk in scenarios such as face-to-face interviews.

[0051] To solve the above problems, please refer to Figure 1 , Figure 1 : is a schematic flow chart of a micro-expression extraction method provided by an embodiment of the present application. The micro-expression extraction method can be implemented by a computer device, and the computer device can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0052] It should be noted that the acquisition of any information mentioned in the provided method is in compliance with relevant regulations and is carried out with the user's consent, and will not infringe on the user's privacy or violate relevant laws and regulations.

[0053] At the same time, the method provided by this application can be applied to scenarios such as the financial field, which requires frequent face-to-face interviews, and can also greatly improve the work efficiency of relevant reviewers in other scenarios that require review through face-to-face interviews. Therefore, the financial field is only a specific example of this application, and the specific application scope of the method provided by this application is not limited.

[0054] To solve the above problems, please refer to Figure 1 Specifically, Figure 1 As shown, the provided micro-expression extraction method includes steps S101 to S105. The details are as follows:

[0055] Step S101: Acquire a facial image sequence to be extracted, where the facial image sequence includes multiple image frames.

[0056] Specifically, the method uses a camera or other image acquisition device to record video clips of the applicant and then forms a facial image sequence after preprocessing. These devices can be ordinary webcams or professional cameras with high resolution and high frame rate. A series of image frames are extracted from real-time or offline recorded videos. Fixed time intervals (such as extracting one frame per second) or keyframe-based technology can be used to ensure that every important moment is recorded with an image frame. The acquired image frames are preprocessed, including but not limited to grayscale, normalization, cropping and alignment, to ensure image quality and consistency.

[0057] For example, in a video conference for a financial interview, a webcam is used to record the applicant's facial video and extract a frame every 0.5 seconds. The extracted image frames are pre-processed to crop and align the facial area for subsequent feature extraction.

[0058] The above step S101 helps to comprehensively analyze the changes in the applicant's expression by ensuring that the acquired image sequence contains facial information at multiple time points. At the same time, the preprocessing step can improve the image quality, reduce noise and interference, and make the subsequent feature extraction more accurate.

[0059] Step S102. Perform feature extraction on each image frame to obtain multiple facial features corresponding to each image frame; the facial features include at least two of facial expression features, facial movement features, facial posture features and facial sight features.

[0060] Specifically, a face detection algorithm (such as Haar features, HOG, and models based on deep learning such as MTCNN) is used to detect the face area in each image frame. Facial expression features such as smiling, frowning, staring, etc. are extracted. Facial action features such as nodding, shaking head, blinking, etc. are extracted. Facial posture features such as the tilt angle and rotation angle of the head are extracted. Eye tracking technology (such as an LSTM-based eye tracking model) is used to extract facial sight features such as the direction of the eyes, whether the eyes are looking straight, etc. By extracting multiple facial features, the facial expressions and behaviors of the applicant can be fully reflected, and the misjudgment caused by a single facial feature can be reduced. Using a deep learning model for feature extraction can improve the accuracy and robustness of feature extraction. At the same time, according to actual needs and computing power, facial features can be selected including at least two of facial expression features, facial action features, facial posture features, and facial sight features, or any combination of three or four items, and other facial features can be introduced to further enrich the content of facial features, so as to greatly enhance the flexibility of the method provided in the embodiment of the present application.

[0061] In some embodiments, feature extraction is performed on each image frame to obtain multiple facial features corresponding to each image frame, including: if the facial features include facial expression features, each image frame is input into a preset micro-expression extraction model, the micro-expression extraction model parses the image frame, and outputs the facial expression features.

[0062] By inputting each image frame into a preset micro-expression extraction model. The micro-expression extraction model parses the image frame and outputs facial expression features. This can be done by using an expression recognition model based on a convolutional neural network (CNN), such as VGGFace, ResNet, etc. The expression features output by the model may include but are not limited to basic expressions such as smiling, frowning, opening eyes, closing eyes, and their intensities. The image frames are preprocessed, such as cropping facial areas, image normalization, etc., to ensure the consistency and accuracy of the model input. The micro-expression extraction model extracts expression features in the image frame through convolutional layers, pooling layers, and fully connected layers. The extracted expression feature vector is output for use in subsequent steps.

[0063] In some embodiments, feature extraction is performed on each image frame to obtain multiple facial features corresponding to each image frame, including: if the facial features include facial action features, each image frame is input into a preset facial action unit model, the facial action unit model parses the image frame, and outputs the facial action features.

[0064] By inputting each image frame into a preset facial action unit model. The facial action unit model parses the image frame and outputs facial action features. For example, a deep learning-based action unit (AU) recognition model is used, such as FACET, OpenFace, etc. The action features output by the model may include but are not limited to specific action units such as eye opening and closing, eyebrow raising, and mouth corner upturning and their intensities. By preprocessing the image frame, such as facial key point detection and image alignment, the facial action unit model extracts action unit features in the image frame through convolutional layers, recurrent neural networks (RNN), etc. The extracted action unit feature vector is output for use in subsequent steps.

[0065] In some embodiments, feature extraction is performed on each image frame to obtain multiple facial features corresponding to each image frame, including: if the facial features include facial posture features, each image frame is input into a preset facial posture model, the facial posture model parses the image frame, and outputs the facial posture features.

[0066] By inputting each image frame into a preset face posture model. The face posture model parses the image frame and outputs the face posture features. For example, a face posture estimation model based on deep learning is used, such as a model based on 3DMM (3DMorphableModel). The posture features output by the model may include but are not limited to the pitch, left and right rotation, tilt angle, etc. of the head. After preprocessing the image frame, such as facial key point detection, image alignment, etc., the face posture model extracts the face posture features in the image frame through convolutional layers, fully connected layers, etc. The extracted head posture angle information is output for use in subsequent steps.

[0067] In some embodiments, feature extraction is performed on each image frame to obtain multiple facial features corresponding to each image frame, including: if the facial features include facial gaze features, each image frame is input into a preset facial gaze model, the facial gaze model parses the image frame, and outputs the facial gaze features.

[0068] By inputting each image frame into a preset face gaze model. The face gaze model parses the image frame and outputs the face gaze features. For example, an eye tracking model based on deep learning is used, such as an eye detection model based on YOLO (You On ly Look Once) and a face gaze estimation model based on regression. The gaze features output by the model may include the gaze direction of the eyes (such as looking at the camera, to the left, to the right, etc.). The image frame is first preprocessed, such as eye area cropping, image enhancement, etc. The face gaze model extracts the gaze features in the image frame through convolutional layers, regression layers, etc. The extracted gaze direction information is output for use in subsequent steps.

[0069] The above embodiment uses a preset special model (micro-expression extraction model, face action unit model, face posture model, face sight model), and the extraction of each feature is more professional and accurate. Each model is optimized for a specific feature, reducing the error of feature extraction and improving the accuracy of overall micro-expression extraction. By using a specially preset model, the expression features, action features, posture features and sight features of each image frame in the face image sequence are extracted respectively. This multi-feature extraction method can more comprehensively and accurately reflect the facial behavior and emotional state of the customer, thereby providing a more reliable basis for judgment during the face-to-face interview process. The beneficial effects of the method include improving the accuracy of feature extraction, enhancing the diversity and integrity of features, improving the scalability and flexibility of the system, reducing human intervention and subjectivity, and improving processing speed and efficiency. These advantages are particularly important in the financial field, and can effectively help enterprises improve service quality, improve customer satisfaction, and reduce fraud risks and economic losses.

[0070] Exemplarily, facial features include at least facial expression features and facial action features. Before feature extraction is performed on each image frame, the method further includes: inputting each image frame into a micro-expression extraction model and a facial action unit model to be trained, respectively, to obtain predicted expression features output by the micro-expression extraction model and predicted action features output by the action unit model; obtaining supervisory feature information corresponding to the image frame; the supervisory feature information includes at least supervisory expression features and supervisory action features; and completing the training of the micro-expression extraction model and the facial action unit model according to the supervisory expression features, the supervisory action features, the predicted expression features and the predicted action features.

[0071] Each image frame is input into the micro-expression extraction model and the facial action unit model to be trained to obtain the predicted features output by the model. The micro-expression extraction model outputs predicted expression features, such as smile, frown, etc. The facial action unit model outputs predicted action features, such as eye opening and closing, eyebrow raising, and mouth corner upturning.

[0072] Obtain supervised feature information corresponding to the image frame. The supervised feature information includes at least supervised expression features and supervised action features. The supervised expression features refer to correct expression features that are manually annotated or generated using a high-precision reference model. The supervised action features refer to correct action features that are manually annotated or generated using a high-precision reference model.

[0073] The micro-expression extraction model and the facial action unit model are trained based on supervised expression features, supervised action features, predicted expression features, and predicted action features. Supervised learning methods such as backpropagation and gradient descent are used to optimize model parameters by minimizing the difference between predicted features and supervised features.

[0074] First, each image frame is input into the micro-expression extraction model and the facial action unit model. Assume that there is an image frame containing the applicant's face, and input it into the micro-expression extraction model and the facial action unit model.

[0075] The micro-expression extraction model analyzes the image frame and outputs the predicted expression features; the facial action unit model analyzes the image frame and outputs the predicted action features. The micro-expression extraction model outputs the emotional state of the applicant in the image frame, such as "smile (intensity 0.8)". The facial action unit model outputs the specific action features of the applicant's eyes, eyebrows, corners of the mouth, etc. in the image frame, such as "left eyebrow raised (intensity 0.6)", "right eye wide open (intensity 0.7)", etc.

[0076] Obtain supervised feature information for each image frame, including supervised expression features and supervised action features. Manually annotate expression features or use a high-precision reference model to generate features such as "smile (intensity 0.9)". Manually annotate action features or use a high-precision reference model to generate features such as "left eyebrow raised (intensity 0.7)", "right eye wide open (intensity 0.8)", etc.

[0077] According to the supervised feature information and the predicted feature information, the model parameters are adjusted to make the predicted results closer to the supervised results. The difference between the predicted features and the supervised features is calculated using loss functions such as mean square error (MSE) or cross-entropy. The weights and biases of the model are updated through the back propagation algorithm to make the predicted results more accurate. The above process is repeated until the difference between the model's predicted features and the supervised features is minimized and the expected accuracy is achieved.

[0078] This example trains the micro-expression extraction model and the facial action unit model through supervised learning methods to ensure that these models can output high-precision prediction results during the feature extraction process. The specific steps include inputting the image frame into the model to obtain the prediction features, obtaining the supervision feature information, and completing the model training based on the supervision features and prediction features. The beneficial effects of this method include improving the accuracy of the model, enhancing the robustness of the model, reducing the workload of manual annotation, improving the automation of the system, and flexibly adapting to different application scenarios. These advantages are particularly important in financial audits, which can effectively reduce the risk of fraud and improve the accuracy and efficiency of audits.

[0079] It should be noted that, in some embodiments, the training of the micro-expression extraction model and the facial action unit model is completed according to the supervised expression feature, the supervised action feature, the predicted expression feature and the predicted action feature, including: calculating the first Euclidean distance between the predicted expression feature and the corresponding supervised expression feature; calculating the second Euclidean distance between the predicted action feature and the corresponding supervised action feature; calculating the loss function of the supervised expression feature, the supervised action feature, the predicted expression feature, the predicted action feature and the first Euclidean distance and the second Euclidean distance; completing the training of the micro-expression extraction model and the facial action unit model according to the loss function;

[0080] The expression of the loss function is:

[0081]

[0082] Where L is the loss function value, e is the facial expression feature, a is the facial action feature, M is the sum of the number of predicted expression features and predicted action features, and d is the number of predicted expression features and predicted action features. i is the first Euclidean distance or the second Euclidean distance, To predict facial features or action features, y i To supervise expression features or supervise action features.

[0083] By inputting each image frame into the micro-expression extraction model and the facial action unit model respectively. Assume that there is an image frame containing the applicant's face, and input it into the micro-expression extraction model and the facial action unit model.

[0084] The micro-expression extraction model analyzes the image frame and outputs the predicted expression features; the facial action unit model analyzes the image frame and outputs the predicted action features. For example, the micro-expression extraction model outputs the applicant's emotional state in the image frame, such as "smile (intensity 0.8)". The facial action unit model outputs the applicant's eyes, eyebrows, mouth corners and other specific action features in the image frame, such as "left eyebrow raised (intensity 0.6)", "right eye wide open (intensity 0.7)", etc.

[0085] By obtaining supervised feature information of each image frame, including supervised expression features and supervised action features. Expression features manually annotated or generated using a high-precision reference model, such as "smile (intensity 0.9)". Action features manually annotated or generated using a high-precision reference model, such as "left eyebrow raised (intensity 0.7)", "right eye wide open (intensity 0.8)", etc.

[0086] By calculating the Euclidean distance (L2 distance) between the predicted feature and the supervised feature. And for the action feature, calculate the Euclidean distance of the left eyebrow raising strength. If both are 0.1. According to the expression of the loss function, calculate the final loss value.

[0087] This embodiment performs supervised learning training on the micro-expression extraction model and the facial action unit model by calculating the Euclidean distance and a specific loss function. The specific steps include inputting the image frame into the model to obtain the predicted features, obtaining the supervised feature information, calculating the Euclidean distance, calculating the loss function, and adjusting the model parameters according to the loss function. The beneficial effects of this method include improving the accuracy of feature extraction, enhancing the robustness of the model, reducing the workload of manual annotation, improving the automation level of the system, flexibly adapting to different application scenarios, accurately measuring errors, and optimizing model performance. These advantages are particularly important in financial audits, which can effectively reduce the risk of fraud and improve the accuracy and efficiency of audits.

[0088] At the same time, the Euclidean distance can also be used as the loss function for the face posture model and the face sight line model to train the model, and the embodiment of the present application does not limit the type of loss function. At the same time, the loss functions of the micro-expression extraction model, the face action unit model, the face posture model and the face sight line model can be superimposed to obtain a total loss function, so as to train the micro-expression extraction model, the face action unit model, the face posture model and the face sight line model simultaneously according to the total loss function, wherein the weight corresponding to the loss function value of each model is set according to actual needs.

[0089] Step S103: Obtain the timing information corresponding to each image frame in the face image sequence, merge multiple face features and timing information corresponding to each image frame, and obtain the timing feature information corresponding to each image frame.

[0090] Specifically, multiple facial features in each image frame are spliced ​​into a 1-dimensional facial feature vector. Then the corresponding facial feature vector in each image frame and the corresponding timing information of the image frame in the complete facial image information are superimposed into a timing feature information. In the process of merging facial feature vectors, different facial features can be weighted, and different weights are assigned according to the importance of the features. By introducing timing information, feature fusion is performed on multiple feature information such as feature vectors and timing information, which effectively retains the time sequence information corresponding to each feature information, and improves the accuracy and robustness of emotional state judgment. This method has important application value in scenarios such as financial face-to-face audits, and can help companies more scientifically evaluate the emotional state of applicants and reduce the risk of erroneous loan grants.

[0091] In some embodiments, it is characterized in that the timing information includes position coding information; the feature vectors corresponding to multiple facial features are spliced ​​to obtain a comprehensive feature vector; the timing information corresponding to each image frame in the facial image sequence is obtained, and the multiple facial features and timing information corresponding to each image frame are merged to obtain the timing feature information corresponding to each image frame, including: obtaining the position coding information corresponding to each image frame in the facial image sequence; splicing the feature vectors corresponding to multiple facial features to obtain a comprehensive feature vector; merging the comprehensive feature vector and the position coding information corresponding to each image frame to obtain the timing feature information corresponding to each image frame.

[0092] For example, the expression features, action features, posture features, and sight features are respectively expressed in vector form, and then these vectors are spliced ​​into a comprehensive feature vector. The dimension of the comprehensive feature vector can be 1 or other values, and the embodiment of the present application does not limit this. At the same time, if a weight of 0.4 is assigned to the expression feature, a weight of 0.3 is assigned to the action feature, and a weight of 0.15 is assigned to the posture feature and the sight feature, respectively, to form a weighted comprehensive feature vector.

[0093] If the dimension of the comprehensive feature vector is 1. This means that multiple facial features (such as expression, action, posture, and sight) of each image frame are represented as a one-dimensional feature vector after feature extraction and merging. The temporal feature vector of each image frame can be represented as ft, where t is the index of the time frame.

[0094] In order to capture the time order of each image frame in the face image sequence, this embodiment introduces positional encoding. Positional encoding is a method of embedding time order information into a feature vector, which is commonly found in sequence models such as Transformer. Positional encoding can be generated in a variety of ways, such as sine and cosine functions, directly using frame index numbers, etc. The positional encoding vector can be represented as pt, where t is the index of the time frame. By combining each time series feature vector ft with its corresponding positional encoding information pt. The combination method can be simple vector concatenation, addition or multiplication, etc. A fusion model (such as Transformer, LSTM, etc.) can also be used to process the feature vector combined with the positional encoding to generate the final time series information.

[0095] By introducing position coding information as timing information, the timing information and comprehensive feature vector corresponding to each image frame are fused to form richer fusion features, effectively retaining the time order of the image frames, and improving the accuracy and robustness of emotional state judgment. At the same time, weighted processing can be used in the process of fusion feature formation to reduce the interference of unimportant features and improve the accuracy of overall judgment. This method has important application value in scenarios such as financial interviews, and can help companies more scientifically evaluate the emotional state of applicants and reduce the risk of erroneous loan issuance.

[0096] Step S104: Fusing multiple time series feature information to generate fused feature information.

[0097] Specifically, the temporal feature information of multiple image frames is modeled by using a time series model (such as LSTM, GRU) to capture the temporal trend of changes. The feature information of multiple image frames is aggregated to generate a comprehensive fusion feature information. Methods such as average pooling and maximum pooling can be used. Attention mechanisms can also be introduced to weight the feature information at different time points to highlight the features of key time points.

[0098] For example, the LSTM model is used to process the temporal feature information of multiple image frames to capture the changing trends of expressions and movements. The feature information output by the LSTM model is averaged and pooled to generate a comprehensive fusion feature information. The attention mechanism is introduced into the LSTM model to perform weighted processing on the feature information at different time points to highlight the applicant's key behaviors and expressions during the interview process.

[0099] Through time series modeling, we can capture the changing trends of facial expressions and behaviors, and more accurately reflect the dynamic emotional state of applicants. At the same time, the introduction of the attention mechanism can highlight the key behaviors and expressions during the interview process, reduce the interference of irrelevant information, and improve the accuracy of judgment.

[0100] Step S105: Obtain micro-expression information corresponding to the facial image sequence according to the fused feature information, and complete the extraction of micro-expression of the facial image sequence.

[0101] Specifically, the fused feature information is sentimentally classified using a preset micro-expression extraction model (such as support vector machine SVM, decision tree, deep learning model DNN, RNN) to determine the applicant's emotional state, such as honesty, tension, anxiety, etc. At the same time, a regression model is used to evaluate the intensity of the emotion of the fused feature information, such as the intensity of the smile, the degree of tension, etc. The results of the emotion classification or regression are post-processed, such as smoothing and thresholding, to ensure the stability and reliability of the final result.

[0102] For example, a DNN-based classification model is used as a micro-expression extraction model to determine whether the applicant is honest based on the fused feature information and output a binary classification result (honest / dishonest). A RNN-based regression model is used to evaluate the applicant's nervousness based on the fused feature information and output a value between 0 and 1. The output results of the DNN classification model are smoothed to ensure the continuity of the emotional state over a period of time; the output results of the RNN regression model are thresholded to mark applicants whose nervousness is below a certain threshold as low risk.

[0103] By analyzing the fusion feature information, the applicant's emotional state can be judged more accurately, helping the interviewer make a more scientific decision. Using the emotional regression model to evaluate the emotional intensity can reduce the misjudgment caused by the single output of the emotional classification model. Combined with automated emotional state judgment, the efficiency of the interview can be improved and the workload of the interviewer can be reduced.

[0104] The provided method has important application value in scenarios such as financial audits, and can help companies improve service quality, increase customer satisfaction, and reduce losses caused by erroneous loan issuance.

[0105] In some embodiments, Figure 2 As shown, before step S102 performs feature extraction on each image frame, steps S106 to S107 are also included.

[0106] Step S106: Calculate the effective frame ratio corresponding to each image frame.

[0107] A valid frame refers to a frame in which face detection and feature extraction are successful and feature quality is high in the current image frame. There are many ways to determine whether a frame is a valid frame, such as whether the detected face is complete, whether the feature points are clear, whether the lighting conditions are appropriate, etc.

[0108] For each image frame, calculate the percentage of valid frames. The specific method may include one or more of the following methods: 1. Use the confidence score of the face detection model to evaluate the quality of each frame. The higher the confidence score, the more valid the frame. 2. Use the confidence score of the feature point detection model to evaluate the quality of the feature points of each frame. The higher the confidence score, the more accurate the feature point detection. 3. Evaluate the lighting conditions of each frame to ensure that the image is under appropriate lighting for feature extraction. 4. Evaluate whether each frame is complete to ensure that the face area is not blocked or partially missing.

[0109] Assume that the MTCNN model is used to detect faces. Each detection box output by the MTCNN model has a confidence score. If the confidence score of a frame exceeds 0.8, the frame is considered to be a valid frame. A feature point detection model is used. Each feature point output by the model has a confidence score. If the average confidence score of the feature points in a frame exceeds 0.8, the frame is considered to be a valid frame. By calculating the brightness and contrast of the image frame, if the brightness and contrast are within the preset appropriate range, the frame is considered to be a valid frame. By detecting whether the face area is complete, not blocked or partially missing, if the face area is complete, the frame is considered to be a valid frame.

[0110] By calculating the effective frame ratio, we can ensure that the image frames used for feature extraction have high quality and reduce the impact of low-quality images on feature extraction results. At the same time, removing low-quality frames can reduce noise and interference and improve the accuracy of subsequent feature extraction and fusion. This ensures that the model performs more stably and accurately when processing high-quality data.

[0111] Step S107 . Identify an image frame whose effective frame ratio is less than a preset ratio as a target frame; and remove the target frame from the multiple image frames.

[0112] By setting a preset effective frame ratio threshold, such as 0.8. The threshold can be adjusted according to the specific application and data quality requirements. The image frame whose effective frame ratio is less than the preset ratio is identified as the target frame. The target frame refers to the low-quality image frame that needs to be removed. The target frame is removed from multiple image frames, and the image frame whose effective frame ratio is higher than or equal to the preset ratio is retained for subsequent feature extraction.

[0113] By calculating the effective frame ratio of each image frame and removing low-quality image frames before feature extraction, the image frames used for feature extraction are of high quality. This approach has important application value in scenarios such as financial face-to-face interviews, and can improve the accuracy and robustness of facial micro-expression extraction and reduce the risk of incorrect loan issuance.

[0114] In some embodiments, Figure 3As shown, after obtaining micro-expression information corresponding to the facial image sequence according to the fusion feature information in step S105, it also includes S108 to S109.

[0115] Step S108: Obtain risk assessment results corresponding to the micro-expression information. The risk assessment results are used to confirm whether there are preset suspicious actions in the micro-expression information.

[0116] The micro-expression information generated in step S105 is evaluated by using a risk assessment model to determine whether there is a preset suspicious action in the micro-expression information. The risk assessment model can be a model based on machine learning or deep learning, such as a support vector machine (SVM), a random forest, a convolutional neural network (CNN), or a recurrent neural network (RNN).

[0117] Define a set of preset suspicious actions, such as frequent blinking, rapid head movement, unnatural smile, etc. These actions may indicate that the applicant is nervous, anxious, or trying to deceive. Input micro-expression information into the risk assessment model, and the model outputs a risk assessment result, which can be a score or a binary label (suspicious / normal).

[0118] For example, a LSTM-based deep learning model is used, and the fused feature information generated in step S105 is input. The following suspicious actions are defined: frequent blinking (more than a certain number of times per minute), rapid head movement (head angle changes exceeding a certain threshold), unnatural smile (too short or too long duration, and lack of natural transition of expression change), erratic eyes (frequent changes in line of sight, long periods of time without looking straight). The fused feature information is input into the LSTM model, and the model outputs a risk assessment score, which indicates the probability that the applicant exhibits these suspicious actions.

[0119] By introducing a risk assessment model, the suspicious actions of applicants can be automatically detected and assessed, reducing the subjectivity and errors of manual judgment. The preset suspicious actions can help the system identify abnormal behaviors of applicants during the face-to-face review process and discover potential risks in a timely manner. At the same time, the risk assessment results provide a scientific basis for the reviewers, helping them make more accurate decisions and reduce the possibility of incorrect loan issuance.

[0120] Step S109: If it is confirmed according to the risk assessment result that there is suspicious action in the micro-expression information, generate warning information corresponding to the facial image sequence.

[0121] By setting a threshold for the risk assessment result, such as 0.8. If the risk assessment score exceeds the threshold, the applicant is considered to have suspicious actions. If suspicious actions are confirmed in the micro-expression information, the system generates corresponding warning information. The warning information may include the specific type of suspicious action, the time of occurrence, the risk assessment score, etc. The generated warning information is sent to the interviewer or the system log to remind the interviewer to pay attention to the applicant's behavior for further verification and judgment.

[0122] For example, the threshold of the risk assessment result is set to 0.6. Assuming that the risk assessment score output by the LSTM model is 0.7, which exceeds the set threshold of 0.6, the system generates a warning message: "The applicant has an unnatural smile, which occurs between 1 minute 30 seconds and 1 minute 45 seconds, and the risk assessment score is 0.7." The generated warning message is sent to the terminal device or system log of the interviewer, who can view the warning message and conduct further verification.

[0123] Furthermore, in combination with the above-mentioned embodiments, early warning information can be generated in real time during the face-to-face interview in combination with the micro-expression extraction results, helping the interviewer to promptly discover the applicant's suspicious behavior and improve the efficiency and accuracy of the face-to-face interview. By setting thresholds and providing detailed descriptions of early warning information, the possibility of misjudgment can be reduced, ensuring the fairness and scientific nature of the face-to-face interview. At the same time, the early warning information provides specific decision-making support for the interviewer, helping them to more comprehensively evaluate the applicant's integrity and improve the quality of decision-making.

[0124] The provided method has the following beneficial effects:

[0125] 1. Improve judgment accuracy and reliability: By extracting and integrating multiple facial features, customers’ expressions and emotional states can be judged more comprehensively and accurately, reducing omissions in naked eye observation.

[0126] 2. Reduce the risk of erroneous loan issuance: During the face-to-face review process, customers suspected of fraud can be identified in a timely manner through quick and accurate emotional state judgment, thereby reducing losses caused by erroneous loan issuance.

[0127] 3. Improve service quality: Financial institutions can use this method to improve the efficiency of face-to-face review, reduce the workload of reviewers, and improve service quality and customer experience. A fast and accurate face-to-face review process can reduce customer waiting time and improve their satisfaction.

[0128] 4. Multi-scenario application: This method is not only suitable for face-to-face interviews in the financial field, but can also be applied to other scenarios that require emotion recognition and behavior analysis, such as security monitoring, customer service, etc.

[0129] See also Figure 4 As shown, Figure 41 is a schematic diagram of the structure of the micro-expression extraction device 200 provided in an embodiment of the present application. The micro-expression extraction device 200 is used to perform the steps of the micro-expression extraction method shown in the above embodiments. The micro-expression extraction device 200 can be a single server or a server cluster, or the micro-expression extraction device 200 can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0130] like Figure 4 As shown, the micro-expression extraction device 200 includes:

[0131] A sequence acquisition unit 201 is used to acquire a facial image sequence to be extracted, wherein the facial image sequence includes a plurality of image frames;

[0132] A feature extraction unit 202 is used to extract features from each of the image frames to obtain a plurality of facial features corresponding to each of the image frames; the facial features include at least two of facial expression features, facial action features, facial posture features, and facial sight features;

[0133] A feature merging unit 203 is used to obtain the time sequence information corresponding to each of the image frames in the face image sequence, and to merge the multiple face features and the time sequence information corresponding to each of the image frames to obtain the time sequence feature information corresponding to each of the image frames;

[0134] A feature fusion unit 204 is used to fuse the multiple time series feature information to generate fused feature information;

[0135] The expression extraction unit 205 is used to obtain micro-expression information corresponding to the facial image sequence according to the fused feature information, and complete the micro-expression extraction of the facial image sequence.

[0136] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the micro-expression extraction device and each module described above can refer to the corresponding process in the micro-expression extraction method embodiments described in the above embodiments, and will not be repeated here.

[0137] The above-mentioned micro-expression extraction method can be implemented in the form of a computer program. The computer program can be used in Figure 4 Run on the device shown.

[0138] See also Figure 5 , Figure 5 1 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes a processor, a memory and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0139] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any micro-expression extraction method.

[0140] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0141] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any micro-expression extraction method.

[0142] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0144] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0145] Acquire a facial image sequence to be extracted, the facial image sequence including a plurality of image frames;

[0146] Perform feature extraction on each image frame to obtain multiple facial features corresponding to each image frame; the facial features include at least two of facial expression features, facial action features, facial posture features, and facial sight features;

[0147] Acquire the time sequence information corresponding to each image frame in the face image sequence, merge the multiple face features and the time sequence information corresponding to each image frame, and acquire the time sequence feature information corresponding to each image frame;

[0148] Fusing multiple time series feature information to generate fused feature information;

[0149] The micro-expression information corresponding to the facial image sequence is obtained according to the fused feature information, and the micro-expression extraction of the facial image sequence is completed.

[0150] In some embodiments, feature extraction is performed on each image frame to obtain multiple facial features corresponding to each image frame; the facial features include at least two of facial expression features, facial action features, facial posture features and facial sight line features, including: if the facial features include facial expression features, each image frame is input into a preset micro-expression extraction model, the micro-expression extraction model parses the image frame, and outputs the facial expression features; and / or, if the facial features include the facial action features, each image frame is input into a preset facial action unit model, the facial action unit model parses the image frame, and outputs the facial action features; and / or, if the facial features include the facial posture features, each image frame is input into a preset facial posture model, the facial posture model parses the image frame, and outputs the facial posture features; and / or, if the facial features include the facial sight line features, each image frame is input into a preset facial sight line model, the facial sight line model parses the image frame, and outputs the facial sight line features.

[0151] Exemplarily, facial features include at least facial expression features and facial action features. Before feature extraction is performed on each image frame, the method further includes: inputting each image frame into a micro-expression extraction model and a facial action unit model to be trained, respectively, to obtain predicted expression features output by the micro-expression extraction model and predicted action features output by the action unit model; obtaining supervisory feature information corresponding to the image frame; the supervisory feature information includes at least supervisory expression features and supervisory action features; and completing the training of the micro-expression extraction model and the facial action unit model according to the supervisory expression features, the supervisory action features, the predicted expression features and the predicted action features.

[0152] It should be noted that, in some embodiments, the training of the micro-expression extraction model and the facial action unit model is completed according to the supervised expression features, supervised action features, predicted expression features and predicted action features, including: calculating the first Euclidean distance between the predicted expression features and the corresponding supervised expression features; calculating the second Euclidean distance between the predicted action features and the corresponding supervised action features; calculating the loss function of the supervised expression features, supervised action features, predicted expression features, predicted action features and the first Euclidean distance and the second Euclidean distance; and completing the training of the micro-expression extraction model and the facial action unit model according to the loss function.

[0153] In some embodiments, it is characterized in that the timing information includes position coding information; the feature vectors corresponding to multiple facial features are spliced ​​to obtain a comprehensive feature vector; the timing information corresponding to each image frame in the facial image sequence is obtained, and the multiple facial features and timing information corresponding to each image frame are merged to obtain the timing feature information corresponding to each image frame, including: obtaining the position coding information corresponding to each image frame in the facial image sequence; splicing the feature vectors corresponding to multiple facial features to obtain a comprehensive feature vector; merging the comprehensive feature vector and the position coding information corresponding to each image frame to obtain the timing feature information corresponding to each image frame.

[0154] In some embodiments, before extracting features from each image frame, it also includes: calculating the effective frame ratio corresponding to each image frame; confirming an image frame whose effective frame ratio is less than a preset ratio as a target frame; and removing the target frame from multiple image frames.

[0155] In some embodiments, after obtaining micro-expression information corresponding to the facial image sequence based on the fused feature information, it also includes: obtaining a risk assessment result corresponding to the micro-expression information, the risk assessment result is used to confirm whether there is a preset suspicious action in the micro-expression information; if it is confirmed that there is a suspicious action in the micro-expression information based on the risk assessment result, generating warning information corresponding to the facial image sequence.

[0156] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the micro-expression extraction method described in the first aspect above.

[0157] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0158] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A micro-expression extraction method, characterized in that: include: Acquire a facial image sequence to be extracted; the facial image sequence includes a plurality of image frames; Performing feature extraction on each of the image frames to obtain a plurality of facial features corresponding to each of the image frames; the facial features include at least two of facial expression features, facial action features, facial posture features, and facial sight features; Acquire the time sequence information corresponding to each of the image frames in the face image sequence, merge the multiple face features corresponding to each of the image frames and the time sequence information, and acquire the time sequence feature information corresponding to each of the image frames; Fusing the plurality of time series feature information to generate fused feature information; The micro-expression information corresponding to the facial image sequence is obtained according to the fusion feature information, and the micro-expression extraction of the facial image sequence is completed.

2. The method according to claim 1, characterized in that The feature extraction is performed on each of the image frames to obtain a plurality of facial features corresponding to each of the image frames; the facial features include at least two of facial expression features, facial action features, facial posture features and facial sight features, including: If the facial features include facial expression features, each of the image frames is input into a preset micro-expression extraction model, the micro-expression extraction model analyzes the image frame and outputs the facial expression features; and / or, If the facial features include the facial action features, each of the image frames is input into a preset facial action unit model, the facial action unit model analyzes the image frame and outputs the facial action features; and / or, If the facial features include the facial posture features, each of the image frames is input into a preset facial posture model, the facial posture model analyzes the image frame and outputs the facial posture features; and / or, If the facial features include the facial sight line features, each of the image frames is input into a preset facial sight line model, and the facial sight line model analyzes the image frame and outputs the facial sight line features.

3. The method according to claim 2, characterized in that The facial features at least include facial expression features and facial action features. Before extracting features from each of the image frames, the method further includes: Input each of the image frames into a micro-expression extraction model and a facial action unit model to be trained, respectively, to obtain predicted expression features output by the micro-expression extraction model and predicted action features output by the action unit model; Obtaining supervisory feature information corresponding to the image frame; the supervisory feature information at least includes supervisory expression features and supervisory action features; The micro-expression extraction model and the facial action unit model are trained according to the supervised expression features, supervised action features, predicted expression features and predicted action features.

4. The method according to claim 3, characterized in that The training of the micro-expression extraction model and the facial action unit model is completed according to the supervised expression features, the supervised action features, the predicted expression features and the predicted action features, including: Calculating the first Euclidean distance between the predicted expression feature and the corresponding supervised expression feature; Calculating the second Euclidean distance between the predicted action feature and the corresponding supervised action feature; Calculating the loss function of the supervised expression feature, the supervised action feature, the predicted expression feature, the predicted action feature, the first Euclidean distance, and the second Euclidean distance; The training of the micro-expression extraction model and the facial action unit model is completed according to the loss function.

5. The method according to claim 1, characterized in that The timing information includes position coding information; the feature vectors corresponding to the multiple facial features are spliced ​​to obtain a comprehensive feature vector; the timing information corresponding to each image frame in the facial image sequence is obtained, and the multiple facial features and the timing information corresponding to each image frame are combined to obtain the timing feature information corresponding to each image frame, including: Obtaining position coding information corresponding to each of the image frames in the face image sequence; splicing feature vectors corresponding to a plurality of facial features to obtain a comprehensive feature vector; The comprehensive feature vector and the position coding information corresponding to each of the image frames are merged to obtain the temporal feature information corresponding to each of the image frames.

6. The method according to claim 1, characterized in that Before extracting features from each of the image frames, the method further includes: Calculate the effective frame ratio corresponding to each of the image frames; Confirming the image frame whose effective frame ratio is less than the preset ratio as the target frame; The target frame is removed from the plurality of image frames.

7. The method according to claim 1, characterized in that After acquiring the micro-expression information corresponding to the facial image sequence according to the fusion feature information, the method further includes: Obtaining a risk assessment result corresponding to the micro-expression information, wherein the risk assessment result is used to confirm whether there is a preset suspicious action in the micro-expression information; If it is confirmed according to the risk assessment result that the suspicious action exists in the micro-expression information, early warning information corresponding to the facial image sequence is generated.

8. A micro-expression extraction device, characterized in that: include: A sequence acquisition unit, used to acquire a facial image sequence to be extracted, wherein the facial image sequence includes a plurality of image frames; A feature extraction unit, configured to extract features from each of the image frames to obtain a plurality of facial features corresponding to each of the image frames; the facial features include at least two of facial expression features, facial action features, facial posture features, and facial sight features; A feature merging unit, used to obtain the time sequence information corresponding to each of the image frames in the face image sequence, and to merge the multiple face features corresponding to each of the image frames with the time sequence information to obtain the time sequence feature information corresponding to each of the image frames; A feature fusion unit, used for fusing the plurality of time series feature information to generate fused feature information; The expression extraction unit is used to obtain micro-expression information corresponding to the facial image sequence according to the fusion feature information, and complete the micro-expression extraction of the facial image sequence.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.

Images