Dynamic fine-grained expression detection method and system based on computer vision

Abstract

The invention provides a dynamic fine-grained expression detection method and system based on computer vision. The method comprises the following steps: acquiring a facial action video; taking the face action video as input, and executing a continuous feasible frame extraction algorithm to obtain all continuous feasible frame sequences in the video; and for each continuous feasible frame sequence, executing a dynamic action unit detection algorithm to obtain dynamic action unit strength, a static action maximum frame and a static action minimum frame of the left and right side parts corresponding to the current sequence, and completing detection of expression actions of the left and right side parts. By adopting a continuous feasible frame extraction algorithm and a dynamic action unit detection algorithm, the problems that a conventional expression detection method is difficult to accurately reflect facial movement only through a single-frame facial image and a conventional discrete expression detection result is difficult to accurately describe specific facial movement are solved; effects of efficient detection and accurate quantification of facial expressions are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of expression detection, and in particular to a dynamic fine-grained expression detection method and system based on computer vision. Background Art

[0002] Facial expression refers to the expression of various emotional states through changes in the eye muscles, facial muscles and mouth muscles. For example, eyes can not only convey emotions but also communicate thoughts. Facial expression is a very important non-verbal means of communication. Expression recognition refers to separating specific expression states from a given static image or dynamic video sequence to determine the psychological emotions of the identified object. Due to the diversity and complexity of facial expressions, and the involvement of physiology and psychology, expression recognition is very difficult. Therefore, compared with other biometric technologies such as fingerprint recognition, iris recognition, and face recognition, it has developed relatively slowly and is not widely used. However, expression recognition is of great value to human-computer interaction, so many research institutions and scholars at home and abroad are committed to research in this area.

[0003] Patent document CN115798007A discloses an end-to-end expression detection method and system, wherein the end-to-end expression detection method includes: constructing an expression detection data set; establishing an expression detection network model based on a feature pyramid network, using the expression detection network model to extract feature maps, collect semantic information and fuse features on the expression detection data set to obtain a facial expression heat map; the expression detection network model detects the face center point in the facial expression heat map according to a multi-task learning strategy; the expression detection network model performs IoU filtering on the face center point to identify the facial expression information corresponding to the face center point.

[0004] However, there are still some shortcomings in the existing expression detection technologies on the market. For example, conventional expression detection methods are difficult to accurately reflect facial movements through only a single frame of facial images, and conventional discrete expression detection results such as happiness, sadness, etc. are difficult to accurately describe specific facial movements. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a dynamic fine-grained expression detection method and system based on computer vision.

[0006] A dynamic fine-grained expression detection method based on computer vision provided by the present invention comprises:

[0007] Step S1: Collect facial action video;

[0008] Step S2: taking the facial action video as input, executing a continuous feasible frame extraction algorithm to obtain all continuous feasible frame sequences in the video;

[0009] Step S3: For each continuous feasible frame sequence, execute the dynamic action unit detection algorithm to obtain the dynamic action unit strength, static action maximum frame, and static action minimum frame of the left and right sides of the face corresponding to the current sequence, and complete the detection of facial expression movements on the left and right sides.

[0010] Preferably, the facial action video refers to a video containing the facial expression and action of a person, which is captured by a camera and in which the person faces the camera and makes a specific facial expression;

[0011] When capturing facial action videos, the subject does not need to keep his head still, and there are no requirements for the speed and frequency of facial movements.

[0012] Preferably, the step S2 comprises:

[0013] Step S2.1: Initialize a set of continuous feasible frame sequences and a temporary frame sequence;

[0014] Step S2.2: read the current frame of the video, detect the facial feature points on the current frame, and obtain the facial feature point set and head pose matrix of the current frame;

[0015] Step S2.3: Determine whether the current frame meets the feasible frame standard, if yes, execute step S2.4; if no, execute step S2.6;

[0016] Step S2.4: Determine whether the temporary frame sequence is empty. If so, directly add the current frame to the temporary frame sequence; if not, calculate the facial movement distance based on the facial feature point set corresponding to the first frame in the temporary frame sequence and the current frame. The calculation formula is as follows:

[0017]

[0018] in, represents the pixel coordinates of the facial feature point numbered i at the current frame time t, represents the pixel coordinates of the facial feature point with serial number i at the first frame time in the temporary frame sequence, C represents a subset of the set consisting of all facial feature point serial numbers, the subset is used to calculate the facial movement distance, and the number of facial feature point serial numbers contained in the set C is N;

[0019] Step S2.5: Determine whether the facial movement distance is less than a preset threshold value D. If so, the current frame is considered to meet the continuity of the temporary frame sequence and the current frame is added to the temporary frame sequence; if not, execute step S2.6;

[0020] Step S2.6: Check whether the duration of the temporary frame sequence is greater than a preset threshold. If so, add the temporary frame sequence to the continuous feasible frame sequence set and initialize the temporary frame sequence to be empty; if not, directly initialize the temporary frame sequence to be empty;

[0021] Step S2.7: Add the current frame to the temporary frame sequence S, and determine whether there is a next frame. If so, return the next frame as the current frame to execute step S2.2; if not, end the calculation process.

[0022] Preferably, the feasible frame standards include: constraints on the distance between the face and the camera, which are achieved by calculating the pixel distance between the eye feature points on the image; constraints on the horizontal and vertical deflection angles of the face, which are achieved through the head pose matrix; and constraints on blinking movements, which are achieved through the pixel distance between the upper and lower eyelid feature points on the image.

[0023] Preferably, the step S3 comprises:

[0024] Step S3.1: Calculate the static action intensity for each continuous feasible frame sequence; the calculation of the static action intensity sets a suitable index according to the specific task;

[0025] Step S3.2: According to the calculated static motion intensity of each frame, select the maximum frame and the minimum frame of the static motion intensity, calculate the static motion intensity difference between the two frames as the motion amplitude △a, and set the threshold A;

[0026] Step S3.3: Determine whether the current action amplitude △a is greater than the set threshold value A. If so, it indicates that there is a facial expression action corresponding to the task in the current continuous feasible frame sequence. At this time, the dynamic action unit strengths of the left and right sides are calculated according to the left and right side facial feature points in the maximum frame and the minimum frame of the static action intensity;

[0027] The calculation of the dynamic action unit strength sets a suitable indicator according to the specific task.

[0028] A dynamic fine-grained expression detection system based on computer vision provided by the present invention comprises:

[0029] Module M1: Collect facial action videos;

[0030] Module M2: taking the facial action video as input, triggering a continuous feasible frame extraction algorithm, and obtaining all continuous feasible frame sequences in the video;

[0031] Module M3: For each continuous feasible frame sequence, the dynamic action unit detection algorithm is triggered to obtain the dynamic action unit strength, static action maximum frame, and static action minimum frame of the left and right sides of the face corresponding to the current sequence, and complete the detection of facial expression movements on the left and right sides.

[0032] Preferably, the facial action video refers to a video containing the facial expression and action of a person, which is captured by a camera and in which the person faces the camera and makes a specific facial expression;

[0033] When capturing facial action videos, the subject does not need to keep his head still, and there are no requirements for the speed and frequency of facial movements.

[0034] Preferably, the module M2 comprises:

[0035] Module M2.1: Initialize the continuous feasible frame sequence set and the temporary frame sequence;

[0036] Module M2.2: Read the current frame of the video, detect the facial feature points on the current frame, and obtain the facial feature point set and head pose matrix of the current frame;

[0037] Module M2.3: Determine whether the current frame meets the feasible frame standard. If yes, trigger module M2.4; if no, trigger module M2.6;

[0038] Module M2.4: Determine whether the temporary frame sequence is empty. If so, directly add the current frame to the temporary frame sequence; if not, calculate the facial movement distance based on the facial feature point set corresponding to the first frame in the temporary frame sequence and the current frame. The calculation formula is as follows:

[0039]

[0040] in, represents the pixel coordinates of the facial feature point numbered i at the current frame time t, represents the pixel coordinates of the facial feature point numbered i at the first frame in the temporary frame sequence, C represents a subset of the set consisting of all facial feature point numbers, which is used to calculate the facial movement distance, and the number of facial feature point numbers contained in the set C is N.

[0041] Module M2.5: Determine whether the facial movement distance is less than a preset threshold D. If so, the current frame is considered to meet the continuity of the temporary frame sequence and the current frame is added to the temporary frame sequence; if not, trigger module M2.6;

[0042] Module M2.6: Check whether the duration of the temporary frame sequence is greater than a preset threshold. If so, add the temporary frame sequence to the continuous feasible frame sequence set and initialize the temporary frame sequence to be empty; if not, directly initialize the temporary frame sequence to be empty;

[0043] Module M2.7: Add the current frame to the temporary frame sequence S, and determine whether there is a next frame. If so, use the next frame as the current frame to trigger module M2.2; if not, end the calculation process.

[0044] Preferably, the feasible frame standards include: constraints on the distance between the face and the camera, which are achieved by calculating the pixel distance between the eye feature points on the image; constraints on the horizontal and vertical deflection angles of the face, which are achieved through the head pose matrix; and constraints on blinking movements, which are achieved through the pixel distance between the upper and lower eyelid feature points on the image.

[0045] Preferably, the module M3 comprises:

[0046] Module M3.1: Calculate the static action intensity for each continuous feasible frame sequence; the calculation of the static action intensity sets an appropriate indicator according to the specific task;

[0047] Module M3.2: According to the calculated static motion intensity of each frame, select the maximum frame and the minimum frame of static motion intensity, calculate the static motion intensity difference between the two frames as the motion amplitude △a, and set the threshold A;

[0048] Module M3.3: Determine whether the current action amplitude △a is greater than the set threshold A. If so, it indicates that there is a facial expression action corresponding to the task in the current continuous feasible frame sequence. At this time, the dynamic action unit strengths of the left and right sides are calculated according to the left and right side facial feature points in the maximum frame and the minimum frame of the static action intensity;

[0049] The calculation of the dynamic action unit strength sets a suitable indicator according to the specific task.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention solves the problem that conventional expression detection methods are difficult to accurately reflect facial movements through only a single-frame facial image, and the problem that conventional discrete expression detection results are difficult to accurately describe specific facial movements by adopting a continuous feasible frame extraction algorithm and a dynamic action unit detection algorithm, thereby achieving the effect of efficient detection and precise quantification of facial expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0053] Figure 1 The working method of the present invention is shown in the following figure;

[0054] Figure 2 It is a schematic diagram of the continuous feasible frame extraction algorithm flow of the present invention;

[0055] Figure 3 It is a schematic diagram of the dynamic action unit detection algorithm flow of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0057] The present invention is based on facial action videos, combined with a continuous feasible frame extraction algorithm and a dynamic action unit detection algorithm, to achieve accurate detection and quantification of left and right facial expression movements.

[0058] Embodiment 1

[0059] According to a computer vision-based dynamic fine-grained expression detection method provided by the present invention, Figure 1 As shown, including:

[0060] Step S1: Collecting facial action videos. The facial action videos refer to videos of a person facing a camera and making specific facial expressions, which are captured by a camera and contain the facial expressions and movements of the person. When collecting facial action videos, the person does not need to keep his head fixed, and there is no requirement for the speed and number of facial expressions and movements.

[0061] Step S2: Taking the facial action video as input, executing a continuous feasible frame extraction algorithm to obtain all continuous feasible frame sequences in the video. Step S2 includes:

[0062] Step S2.1: Initialize a set of continuous feasible frame sequences and a temporary frame sequence.

[0063] Step S2.2: Read the current frame of the video, and detect the facial feature points on the current frame to obtain the facial feature point set and the head pose matrix of the current frame. The facial feature point detection method includes a facial feature point detection algorithm. The facial feature point detection algorithm functions to input a facial image, detect all facial feature points on the image, and output them as a facial feature point set, while calculating and outputting the head pose matrix of the person on the image.

[0064] Step S2.3: Determine whether the current frame meets the feasible frame standard. If yes, execute step S2.4; if no, execute step S2.6. The feasible frame standard includes: for the constraint of the distance between the face and the camera, it is realized by calculating the pixel distance of the eye feature points on the image; for the constraint of the horizontal and vertical deflection angles of the face, it is realized by the head pose matrix; for the constraint of the blinking action, it is realized by the pixel distance of the upper and lower eyelid feature points on the image.

[0065] Step S2.4: Determine whether the temporary frame sequence is empty. If so, directly add the current frame to the temporary frame sequence; if not, calculate the facial movement distance based on the facial feature point set corresponding to the first frame in the temporary frame sequence and the current frame. The calculation formula is as follows:

[0066]

[0067] in, represents the pixel coordinates of the facial feature point numbered i at the current frame time t, represents the pixel coordinates of the facial feature point numbered i at the first frame in the temporary frame sequence, C represents a subset of the set consisting of all facial feature point numbers, which is used to calculate the facial movement distance, and the number of facial feature point numbers contained in the set C is N.

[0068] Step S2.5: Determine whether the facial movement distance is less than a preset threshold D. If so, the current frame is considered to meet the continuity of the temporary frame sequence and the current frame is added to the temporary frame sequence; if not, execute step S2.6.

[0069] Step S2.6: Check whether the duration of the temporary frame sequence is greater than a preset threshold. If so, add the temporary frame sequence to the continuous feasible frame sequence set and initialize the temporary frame sequence to empty; if not, directly initialize the temporary frame sequence to empty.

[0070] Step S2.7: Add the current frame to the temporary frame sequence S, and determine whether there is a next frame. If so, return the next frame as the current frame to execute step S2.2; if not, end the calculation process.

[0071] Specifically, in combination with Figure 2The continuous feasible frame extraction algorithm of the present invention is specifically described as follows: First, initialize the continuous feasible frame sequence set and initialize the temporary frame sequence. Subsequently, read the next frame of the video (the first frame) as the current frame, use the facial feature point detection algorithm to detect the facial feature points on the current frame, and obtain the facial feature point set of the current frame, as well as the head pose matrix. Based on the above information, determine whether the current frame meets the feasible frame standard. At this time, there are several situations as follows: a) If the feasible frame standard is met and the temporary frame sequence is empty, directly add the current frame to the temporary frame sequence. b) If the feasible frame standard is met and the temporary frame sequence is not empty, calculate the facial movement distance based on the facial feature point set corresponding to the first frame in the temporary frame sequence and the current frame. If the distance is less than a certain threshold, it can be considered that the current frame meets the continuity of the temporary frame sequence, and the current frame is added to the temporary frame sequence. c) If the above facial movement distance is greater than the threshold, or the current frame does not meet the feasible frame standard in the previous judgment, it indicates that the continuity of the temporary frame sequence has stopped. At this time, check whether the duration of the temporary frame sequence is greater than a certain threshold: if it is greater, add the temporary frame sequence to the continuous feasible frame sequence set, and initialize the temporary frame sequence to empty; if it is less, directly initialize the temporary frame sequence to empty. Then, add the current frame to the temporary frame sequence. At this point, the calculation for the current frame has been completed. At this time, determine whether there is a next frame in the video: if it exists, repeat the reading of the next frame of the video and its subsequent steps; if it does not exist, the program ends. The continuous feasible frame sequence set is all the continuous feasible frame sequences in the video.

[0072] Step S3: for each continuous feasible frame sequence, execute the dynamic action unit detection algorithm to obtain the dynamic action unit strength, static action maximum frame, and static action minimum frame of the left and right sides of the face corresponding to the current sequence, and complete the detection of the facial expression movements on the left and right sides. Step S3 includes:

[0073] Step S3.1: Calculate the static motion intensity for each continuous feasible frame sequence. The static motion intensity calculation method needs to set appropriate indicators according to the specific task. For example, for the mouth corner movement detection task, select multiple feature points of the left and right mouth corners and calculate the pixel distance d between the left and right mouth corners. mouth is the static action intensity, and the calculation formula is as follows:

[0074]

[0075] Among them, C l Represents the set of sequence numbers of the left corner of the mouth feature points, N l Represents the set C l The number of feature point numbers, C r Represents the set of feature point numbers of the right corner of the mouth, N r Represents the set C rThe number of feature point numbers.

[0076] Step S3.2: According to the calculated static motion intensity of each frame of the image, select the maximum frame and the minimum frame of the static motion intensity, calculate the static motion intensity difference between the two frames as the motion amplitude △a, and set the threshold A.

[0077] Step S3.3: Determine whether the current action amplitude △a is greater than the set threshold value A. If so, it indicates that there is a facial expression action corresponding to the task in the current continuous feasible frame sequence. At this time, the dynamic action unit strengths of the left and right sides are calculated according to the left and right facial feature points in the maximum frame and the minimum frame of the static action intensity. The calculation method of the dynamic action unit strength sets appropriate indicators according to the specific task. For example, for the mouth corner movement detection task, multiple feature points near the left corner of the mouth are selected, and the movement distance of these feature points between the maximum frame and the minimum frame of the static action intensity is calculated as the dynamic action unit strength of the left corner of the mouth. The same is true for the right corner of the mouth. The calculation formula is as follows:

[0078]

[0079] Among them, C l Represents the set of sequence numbers of the left corner of the mouth feature points, N l Represents the set C l The number of feature point numbers, t max and t min They respectively represent the time when the static action intensity is the maximum frame and the minimum frame.

[0080] Specifically, in combination with Figure 3 The dynamic action unit detection algorithm of the present invention is specifically described as follows: First, select a plurality of specific facial feature points according to a specific task to form a specific facial feature point set, thereby calculating the static action intensity of each frame image in a continuous feasible frame sequence. Subsequently, select the maximum frame and the minimum frame of the static action intensity, calculate the static action intensity difference between the two frames as the action amplitude, set a certain threshold, if the action amplitude is greater than the threshold, it indicates that there is a facial expression action corresponding to the task in the continuous feasible frame sequence, then at this time, the dynamic action unit intensity of the left and right sides is calculated according to the left and right side facial feature points in the maximum frame and the minimum frame of the static action.

[0081] Embodiment 2

[0082] The present invention also provides a dynamic fine-grained expression detection system based on computer vision. The dynamic fine-grained expression detection system based on computer vision can be implemented by executing the process steps of the dynamic fine-grained expression detection method based on computer vision, that is, those skilled in the art can understand the dynamic fine-grained expression detection method based on computer vision as a preferred implementation of the dynamic fine-grained expression detection system based on computer vision.

[0083] A dynamic fine-grained expression detection system based on computer vision provided by the present invention comprises:

[0084] Module M1: Collecting facial action videos. Facial action videos refer to videos of people facing the camera and making specific facial expressions, which are captured by the camera and contain the facial expressions and movements of the people. When collecting facial action videos, the people do not need to keep their heads still, and there are no requirements for the speed and number of facial expressions and movements.

[0085] Module M2: Take the facial action video as input, trigger the continuous feasible frame extraction algorithm, and obtain all continuous feasible frame sequences in the video. The module M2 includes: module M2.1: initialize the continuous feasible frame sequence set and initialize the temporary frame sequence; module M2.2: read the current frame of the video, and detect the facial feature points on the current frame to obtain the facial feature point set and head pose matrix of the current frame; module M2.3: determine whether the current frame meets the feasible frame standard, if so, trigger module M2.4; if not, trigger module M2.6; module M2.4: determine whether the temporary frame sequence is empty, if so, directly add the current frame to the temporary frame sequence; if not, calculate the facial movement distance based on the facial feature point set corresponding to the first frame in the temporary frame sequence and the current frame, and the calculation formula is as follows:

[0086]

[0087] in, represents the pixel coordinates of the facial feature point numbered i at the current frame time t, represents the pixel coordinates of the facial feature point numbered i at the first frame in the temporary frame sequence, C represents a subset of the set consisting of all facial feature point numbers, which is used to calculate the facial movement distance, and the number of facial feature point numbers contained in the set C is N.

[0088] Module M2.5: Determine whether the facial movement distance is less than a preset threshold D. If so, the current frame is considered to meet the continuity of the temporary frame sequence and the current frame is added to the temporary frame sequence; if not, module M2.6 is triggered; Module M2.6: Check whether the duration of the temporary frame sequence is greater than the preset threshold. If so, the temporary frame sequence is added to the continuous feasible frame sequence set and the temporary frame sequence is initialized to empty; if not, the temporary frame sequence is directly initialized to empty; Module M2.7: Add the current frame to the temporary frame sequence S and determine whether there is a next frame. If so, the next frame is used as the current frame to trigger module M2.2; if not, the calculation process ends.

[0089] The feasible frame standards include: constraints on the distance between the face and the camera, which are achieved by calculating the pixel distance between the eye feature points on the image; constraints on the horizontal and vertical deflection angles of the face, which are achieved through the head pose matrix; and constraints on blinking movements, which are achieved through the pixel distance between the upper and lower eyelid feature points on the image.

[0090] Module M3: For each continuous feasible frame sequence, trigger the dynamic action unit detection algorithm, obtain the dynamic action unit strength, static action maximum frame, and static action minimum frame of the left and right sides corresponding to the current sequence, and complete the detection of facial expression movements on the left and right sides. The module M3 includes: Module M3.1: Calculate the static action strength for each continuous feasible frame sequence; the calculation of the static action strength sets a suitable index according to the specific task; Module M3.2: According to the static action strength of each frame image calculated, select the maximum frame and the minimum frame of the static action strength, calculate the static action strength difference between the two frames as the action amplitude △a, and set the threshold A; Module M3.3: Determine whether the current action amplitude △a is greater than the set threshold A. If so, it indicates that there is a facial expression movement corresponding to the task in the current continuous feasible frame sequence. At this time, the dynamic action unit strength of the left and right sides is calculated according to the left and right facial feature points in the maximum frame and the minimum frame of the static action strength; the calculation of the dynamic action unit strength sets a suitable index according to the specific task.

[0091] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0092] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A dynamic fine-grained expression detection method based on computer vision, characterized in that: include: Step S1: Collect facial action video; Step S2: taking the facial action video as input, executing a continuous feasible frame extraction algorithm to obtain all continuous feasible frame sequences in the video; Step S3: For each continuous feasible frame sequence, execute the dynamic action unit detection algorithm to obtain the dynamic action unit strength, static action maximum frame, and static action minimum frame of the left and right sides of the face corresponding to the current sequence, and complete the detection of facial expression movements on the left and right sides.

2. The method for dynamic fine-grained expression detection based on computer vision according to claim 1, characterized in that: The facial action video refers to a video containing the facial expression and action of a person facing the camera and making a specific facial expression, which is captured by the camera; When capturing facial action videos, the subject does not need to keep his head still, and there are no requirements for the speed and frequency of facial movements.

3. The dynamic fine-grained expression detection method based on computer vision according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Initialize a set of continuous feasible frame sequences and a temporary frame sequence; Step S2.2: read the current frame of the video, detect the facial feature points on the current frame, and obtain the facial feature point set and head pose matrix of the current frame; Step S2.3: Determine whether the current frame meets the feasible frame standard, if yes, execute step S2.4; if no, execute step S2.6; Step S2.4: Determine whether the temporary frame sequence is empty. If so, directly add the current frame to the temporary frame sequence; if not, calculate the facial movement distance d based on the facial feature point set corresponding to the first frame in the temporary frame sequence and the current frame. The calculation formula is as follows: in, represents the pixel coordinates of the facial feature point numbered i at the current frame time t, represents the pixel coordinates of the facial feature point with serial number i at the first frame time in the temporary frame sequence, C represents a subset of the set consisting of all facial feature point serial numbers, the subset is used to calculate the facial movement distance, and the number of facial feature point serial numbers contained in the set C is N; Step S2.5: Determine whether the facial movement distance is less than a preset threshold value D. If so, the current frame is considered to meet the continuity of the temporary frame sequence and the current frame is added to the temporary frame sequence; if not, execute step S2.6; Step S2.6: Check whether the duration of the temporary frame sequence is greater than a preset threshold. If so, add the temporary frame sequence to the continuous feasible frame sequence set and initialize the temporary frame sequence to be empty; if not, directly initialize the temporary frame sequence to be empty; Step S2.7: Add the current frame to the temporary frame sequence S, and determine whether there is a next frame. If so, return the next frame as the current frame to execute step S2.2; if not, end the calculation process.

4. The method for dynamic fine-grained expression detection based on computer vision according to claim 3, characterized in that: The feasible frame standards include: constraints on the distance between the face and the camera, which are achieved by calculating the pixel distance between the eye feature points on the image; constraints on the horizontal and vertical deflection angles of the face, which are achieved through the head pose matrix; and constraints on blinking movements, which are achieved through the pixel distance between the upper and lower eyelid feature points on the image.

5. The method for dynamic fine-grained expression detection based on computer vision according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Calculate the static action intensity for each continuous feasible frame sequence; the calculation of the static action intensity sets a suitable index according to the specific task; Step S3.2: According to the calculated static motion intensity of each frame, select the maximum frame and the minimum frame of the static motion intensity, calculate the static motion intensity difference between the two frames as the motion amplitude △a, and set the threshold A; Step S3.3: Determine whether the current action amplitude △a is greater than the set threshold value A. If so, it indicates that there is a facial expression action corresponding to the task in the current continuous feasible frame sequence. At this time, the dynamic action unit strengths of the left and right sides are calculated according to the left and right side facial feature points in the maximum frame and the minimum frame of the static action intensity; The calculation of the dynamic action unit strength sets a suitable indicator according to the specific task.

6. A dynamic fine-grained expression detection system based on computer vision, characterized in that: include: Module M1: Collect facial action videos; Module M2: taking the facial action video as input, triggering a continuous feasible frame extraction algorithm, and obtaining all continuous feasible frame sequences in the video; Module M3: For each continuous feasible frame sequence, the dynamic action unit detection algorithm is triggered to obtain the dynamic action unit strength, static action maximum frame, and static action minimum frame of the left and right sides of the face corresponding to the current sequence, and complete the detection of facial expression movements on the left and right sides.

7. The dynamic fine-grained expression detection system based on computer vision according to claim 6, characterized in that: The facial action video refers to a video containing the facial expression and action of a person facing the camera and making a specific facial expression, which is captured by the camera; When capturing facial action videos, the subject does not need to keep his head still, and there are no requirements for the speed and frequency of facial movements.

8. The computer vision-based dynamic fine-grained expression detection system according to claim 6, characterized in that: The module M2 comprises: Module M2.1: Initialize the continuous feasible frame sequence set and the temporary frame sequence; Module M2.2: Read the current frame of the video, detect the facial feature points on the current frame, and obtain the facial feature point set and head pose matrix of the current frame; Module M2.3: Determine whether the current frame meets the feasible frame standard. If yes, trigger module M2.4; if no, trigger module M2.6; Module M2.4: Determine whether the temporary frame sequence is empty. If so, directly add the current frame to the temporary frame sequence; if not, calculate the facial movement distance d based on the facial feature point set corresponding to the first frame in the temporary frame sequence and the current frame. The calculation formula is as follows: in, represents the pixel coordinates of the facial feature point numbered i at the current frame time t, represents the pixel coordinates of the facial feature point with serial number i at the first frame time in the temporary frame sequence, C represents a subset of the set consisting of all facial feature point serial numbers, the subset is used to calculate the facial movement distance, and the number of facial feature point serial numbers contained in the set C is N; Module M2.5: Determine whether the facial movement distance is less than a preset threshold D. If so, the current frame is considered to meet the continuity of the temporary frame sequence and the current frame is added to the temporary frame sequence; if not, trigger module M2.6; Module M2.6: Check whether the duration of the temporary frame sequence is greater than a preset threshold. If so, add the temporary frame sequence to the continuous feasible frame sequence set and initialize the temporary frame sequence to be empty; if not, directly initialize the temporary frame sequence to be empty; Module M2.7: Add the current frame to the temporary frame sequence S, and determine whether there is a next frame. If so, use the next frame as the current frame to trigger module M2.2; if not, end the calculation process.

9. The dynamic fine-grained expression detection system based on computer vision according to claim 8, characterized in that: The feasible frame standards include: constraints on the distance between the face and the camera, which are achieved by calculating the pixel distance between the eye feature points on the image; constraints on the horizontal and vertical deflection angles of the face, which are achieved through the head pose matrix; and constraints on blinking movements, which are achieved through the pixel distance between the upper and lower eyelid feature points on the image.

10. The computer vision-based dynamic fine-grained expression detection system according to claim 6, characterized in that: The module M3 comprises: Module M3.1: Calculate the static action intensity for each continuous feasible frame sequence; the calculation of the static action intensity sets an appropriate indicator according to the specific task; Module M3.2: According to the calculated static motion intensity of each frame, select the maximum frame and the minimum frame of static motion intensity, calculate the static motion intensity difference between the two frames as the motion amplitude △a, and set the threshold A; Module M3.3: Determine whether the current action amplitude △a is greater than the set threshold A. If so, it indicates that there is a facial expression action corresponding to the task in the current continuous feasible frame sequence. At this time, the dynamic action unit strengths of the left and right sides are calculated according to the left and right side facial feature points in the maximum frame and the minimum frame of the static action intensity; The calculation of the dynamic action unit strength sets a suitable indicator according to the specific task.

Images