A method and system for detecting facial features based on machine vision

Abstract

The present invention discloses a method and system for detecting the state of facial features based on machine vision, the method comprising: collecting video frames or images within the machine field of view; obtaining the coordinates of key points of the face based on the Dlib face detection technology; constructing an eyebrow parameter model and judging the state of the left and right eyebrows respectively; obtaining the position state of the left and right pupils respectively according to the coordinates of the key points of the face; constructing an eyelid parameter model and judging the state of the left eyelid and the right eyelid respectively in combination with the position state of the left and right pupils; constructing a mouth parameter model and judging the state of the mouth; judging the state of the head according to the coordinates of the key points of the face; summarizing the state of the facial features and outputting it. The method of the present invention does not need to collect a large amount of sample data for model training, can simultaneously detect the state of each facial feature with good real-time performance, has high robustness even when wearing glasses, and has broad application prospects in the fields of remote robot control.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision and face recognition, and in particular to a method and system for detecting the state of facial features based on machine vision. Background Art

[0002] Facial feature state detection is an important part of facial expression recognition technology. By acquiring images or video streams through visual acquisition equipment, the face area can be automatically detected in the image, and then the facial features and state of the detected face can be located. In the prior art, a single classifier is mainly trained through deep learning or convolutional neural networks to detect a single facial feature, and multiple classifiers are required to detect all facial features. The training process is time-consuming and labor-intensive, and training parameters need to be optimized repeatedly. In addition, the accuracy of detection and positioning depends on the support of a large amount of sample data, and the generalization ability is poor. For difficult samples or application scenarios that are not included in the sample data set, accurate detection is often not possible. Summary of the invention

[0003] The existing facial features state detection technology overly relies on complex algorithm models such as convolutional neural networks and deep learning that require a large amount of sample data. The additional wearable devices required for detection also increase the complexity of detection, and the high computing power requirements when the algorithm model is running limit the application and development of related downstream technologies. To this end, the present invention provides a method and system for facial features state detection based on machine vision, and the specific technical solutions are as follows:

[0004] A method for detecting the state of facial features based on machine vision comprises the following steps:

[0005] Capture video frames or images within the machine’s field of view;

[0006] Get the location information of facial key points in video frames or images based on Dlib face detection technology;

[0007] Construct an eyebrow parameter model based on the position information of key points of the face, and determine the status of the left eyebrow and the right eyebrow respectively;

[0008] According to the position information of the key points of the face, the position states of the left eye pupil and the right eye pupil are respectively obtained;

[0009] An eyelid parameter model is constructed based on the position information of key points of the face, and the status of the left eyelid and the right eyelid are determined based on the position status of the left pupil and the right pupil respectively;

[0010] Construct a mouth parameter model based on the position information of key points of the face and determine the state of the mouth;

[0011] Determine the state of the head based on the position information of key points on the face;

[0012] Summarize the status of the left eyebrow, right eyebrow, left eyelid, right eyelid, mouth and head and output them.

[0013] Furthermore, the eyebrow parameter model includes a left eyebrow parameter model B left and right eyebrow parameter model B right , and there is

[0014]

[0015] Among them, P 21 and P 22 are the coordinates of the two key points corresponding to the right eyebrow, P 23 and P 24 are the coordinates of the two key points corresponding to the left eyebrow, P 37 and P 40 are the coordinates of the two key points corresponding to the right eye, P 43 and P 46 are the coordinates of the two key points corresponding to the left eye; set the first eyebrow threshold T B1 and the second state threshold T B2 , respectively judge the status of the left eyebrow and the right eyebrow: if B left >T B1 , then the left eyebrow is judged to be in the raised eyebrow state; if B left <T B2 , then the left eyebrow is judged to be in a frowning state; if T B1 ≥B left ≥T B2 , then the left eyebrow is judged to be in normal state; if B right >T B1 , then the right eyebrow is judged to be in the raised eyebrow state; if B right <T B2 , then the right eyebrow is judged to be in a frowning state; if T B1 ≥B right ≥T B2 , then the right eyebrow is judged to be in normal state.

[0016] Furthermore, the acquisition of the position state of the left eye pupil comprises the following steps:

[0017] S101, according to the facial key point P corresponding to the left eye 43 , P 44 , P 45 , P 46 , P 47 , P 48 The minimum bounding rectangle of the left eye is calculated based on the coordinates of the left eye, and an image L corresponding to the bounding rectangle area is intercepted in the original video frame or image;

[0018] S102, connecting the facial key points corresponding to the left eye in sequence to form a closed area, and intercepting the left eye image L corresponding to the closed area in the image Leye ;

[0019] S103: Left eye image L eye Binarize the image and get image L thresh And get the image L thresh The outline of

[0020] S104, traverse each contour, if its area ratio S and the minimum circumscribed rectangle aspect ratio L satisfy T1>L>T2 and S>T S , then the contour is determined to be a through hole, otherwise the output result is that there is no left pupil; the area ratio S is the pixel area of ​​the contour and the image L thresh The aspect ratio L is the aspect ratio of the minimum circumscribed rectangle of the outline; T S is the area ratio threshold, T1 and T2 are the first aspect ratio threshold and the second aspect ratio threshold respectively;

[0021] S105: The center of the smallest circumscribed circle of the through hole is determined as the center P of the left pupil. cl ;

[0022] S106, respectively calculate the center P of the left pupil cl To the key point P 43 and key point P 46 The distance d 1l and d 2l , determine the position of the left pupil: if d 1l / (d 1l +d 2l ) <T d , then the left pupil is considered to be facing inward; if d 2l / (d 1l +d 2l ) <T d , the left pupil is considered to be facing outward; if it is other cases, the left pupil is considered to be in the middle of the left eye.

[0023] Furthermore, the acquisition of the position state of the right eye pupil comprises the following steps:

[0024] S201, according to the facial key point P corresponding to the right eye 37 , P 38 , P 39 , P 40 , P 41 , P 42 The coordinates of the right eye are used to calculate the minimum bounding rectangle of the right eye, and an image R corresponding to the bounding rectangle area is intercepted in the original video frame or image;

[0025] S202, connecting the facial key points corresponding to the right eye in sequence to form a closed area, and intercepting the right eye image R corresponding to the closed area in the image R eye ;

[0026] S203, right eye image R eye Binarize the image and get image R thresh And get the image R thresh The outline of

[0027] S204, traverse each contour, if its area ratio S and the minimum circumscribed rectangle aspect ratio L satisfy T1>L>T2 and S>T S , then the contour is determined to be a through hole, otherwise the output result is that there is no right eye pupil;

[0028] S205: The center of the smallest circumscribed circle of the through hole is determined as the center P of the pupil of the right eye. cr ;

[0029] S206, respectively calculating the center P of the left pupil cr To the key point P 37 and key point P 40 The distance d 1r and d 2r , determine the position of the right pupil: if d 1r / (d 1r +d 2r ) <T d , then the right pupil is considered to be facing inward; if d 2r / (d 1r +d 2r ) <T d , the right pupil is considered to be facing outward; if it is other cases, the right pupil is considered to be in the middle of the right eye.

[0030] Furthermore, the image binarization in steps S103 and S203, and the determination of the binarization threshold thereof include the steps of:

[0031] S301, set the initial threshold nTh=5, the best threshold nBestTh=5, and the minimum weight minPer=1;

[0032] S302: perform bilateral filtering on the binarized eye image I, and then perform corrosion operation to obtain image I new ;

[0033] S303, using nTh to image I new Binarize and get image I th ;

[0034] S304, calculating image I thThe proportion per of black pixels in the total number of pixels in the image;

[0035] S305. Calculate abs(per - 0.48). If abs(per - 0.48) < minPer, then assign abs(per - 0.48) to minPer and assign nTh to nBestTh; otherwise, keep the values of minPer and nBestTh unchanged.

[0036] S306. Determine whether nTh is less than 100. If so, set nTh = nTh + 5 and return to S303 to continue execution; otherwise, output the current optimal threshold nBestTh.

[0037] S307. Repeat S301 - S306, respectively obtain the optimal thresholds of three consecutive video frames and take the average, and use the average value as the final binarization threshold.

[0038] Furthermore, the state judgment of the left eyelid and the right eyelid includes the following steps:

[0039] For the left eyelid, if no left pupil is output when obtaining the position state of the left eye pupil, it is determined that the left eyelid is in the closed state; otherwise, calculate the left eyelid parameter model E left =(||P 44 -P 48 ||+||P 45 -P 47 ||) / 2||P 43 -P 48 ||; if E left <T e then it is determined that the left eye is in the closed state; otherwise, it is in the open state.

[0040] For the right eyelid, if no right pupil is output when obtaining the position state of the right eye pupil, it is determined that the right eyelid is in the closed state; otherwise, calculate the right eyelid parameter model E right =(||P 38 -P 42 ||+||P 39 -P 41 ||) / 2||P 37 -P 42 ||; if E right <T e then it is determined that the right eye is in the closed state; otherwise, it is in the open state; where T e is the eye state threshold.

[0041] Furthermore, the mouth parameter model is M=(||P 51 -P 59 ||+||P 53 -P 57 ||) / 2||P49 -P 55 ||, where P 49 , P 51 , P 53 , P 55 , P 57 , P 59 is the coordinates of the facial key points corresponding to the mouth, if M>T m , the mouth is judged to be in an open state, otherwise it is in a closed state; where T m is the mouth posture threshold.

[0042] Furthermore, the state determination of the head includes the following steps:

[0043] The horizontal position of the head is determined by calculating the ratio of the length of the left and right eyebrows: the ratio of the length of the left eyebrow to the length of the right eyebrow If BR1>T br The head is considered to be rotated to the left in the horizontal direction, T br is the rotation threshold; the ratio of the length of the right eyebrow to the left eyebrow is If BR2>T br If the head is rotated to the right, the head is considered to be in the horizontal direction; if it is the other case, the head is considered to be in the middle position in the horizontal direction;

[0044] Determine the vertical state of the head:

[0045] ① Calculate the key point P1 to the key point P 17 Distance H: H = || P1-P 17 ||;

[0046] ②Calculate the key point P 28 To the key points P1 and P 17 The straight line P1 P 17 Distance V:

[0047]

[0048] ③Calculate the ratio R of H to V HV : R HV =H / V;

[0049] ④Calculate the key point P 28 The y coordinates of the key points P1 and P 17 The difference in y coordinates of the midpoints is ΔY:

[0050] ΔY=P 28 .y-(P1.yP 17 .y) / 2

[0051] Among them, P1.x and P1.y are the x-coordinate and y-coordinate of the key point P1, respectively.17 .x and P 17 .y are the key points P 17 The x- and y-coordinates of P 28 .x and P 28 .y are the key points P 28 The x- and y-coordinates of

[0052] If ΔY<0 and R HV >0.05, the head is considered to be in an upward position in the vertical direction; if ΔY>0 and R HV >0.05, the head is considered to be in a downward position in the vertical direction; otherwise, the head is considered to be in a middle position in the vertical direction.

[0053] A human face facial features state detection system based on machine vision, comprising

[0054] An image acquisition module is arranged on the robot and is used to acquire video frames or images within the field of view and send them to the face key point detection module;

[0055] The face key point detection module is connected to the image acquisition module and is used to detect the face key points of the received image or video frame based on the Dlib face detection technology;

[0056] The eyebrow state detection module is connected to the face key point detection module, and is used to obtain the state of the left eyebrow and the right eyebrow according to the above detection method, and send it to the facial features state output module;

[0057] The pupil position detection module is connected to the face key point detection module, and is used to obtain the position status of the left eye pupil and the right eye pupil according to the above detection method, and send it to the eyelid state detection module;

[0058] An eyelid state detection module, connected to the pupil position detection module, is used to determine the states of the left eyelid and the right eyelid respectively according to the above detection method, and send the results to the facial features state output module;

[0059] The mouth state detection module is connected to the face key point detection module, and is used to obtain the state of the mouth according to the above detection method and send it to the facial features state output module;

[0060] The head state detection module is connected to the face key point detection module, and is used to obtain the state of the head according to the above detection method and send it to the facial features state output module;

[0061] The facial features status output module is used to summarize the status information of the left eyebrow, right eyebrow, left eyelid, right eyelid, mouth and head, and output it to downstream devices for use.

[0062] A computer-readable storage medium stores a computer program, which is loaded by a central processing unit and executes the steps of the above-mentioned facial feature state detection method.

[0063] Beneficial effects of the present invention:

[0064] Compared with the existing facial features detection method based on deep learning, the method of the present invention does not need to collect a large amount of sample data for complex model training, and can detect the status of all facial features at the same time. The recognition and positioning of the pupil is also highly robust even when wearing glasses. In addition, the present invention has good real-time performance, which can reach 30fps, and has broad application prospects in the fields of remote robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0066] Figure 1 is a flow chart of facial features status detection provided by an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of 68 key points of Dlib face detection provided by an embodiment of the present invention;

[0068] Figure 3 It is a schematic diagram of the module composition of the facial features status detection system provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] like Figure 1As shown, the present invention provides a method for detecting the state of facial features based on machine vision, which mainly includes the following steps: first, using a camera arranged on a robot or other device module to collect images or videos within the field of view; then, detecting the face in the image or video frame based on the Dlib face detection technology, and obtaining the position information of 68 key points of the face; finally, calculating and obtaining the state information of the eyebrows, eyelids, pupils, mouth, and head based on the position information of the 68 key points. The facial features state information provided by this method can be used by downstream algorithms or devices, such as being combined with robots or virtual reality technology to enable robots or virtual people to simulate user facial expressions in real time. Among them, Dlib is a popular C++ toolkit for computer vision tasks, in which the face detection technology can extract 68 key points on a face image, such as Figure 2 As shown, these key points correspond to eyebrows, eyes, nose, mouth, and facial contours respectively.

[0071] Specifically, the facial features state detection in the present invention mainly includes eyebrow state detection, eyelid state detection, mouth state detection and head state detection, which will be described in detail below.

[0072] 1. Eyebrow status detection

[0073] When performing eyebrow state detection, it is necessary to construct a corresponding eyebrow parameter model. The left eyebrow parameter model B in the present invention left and right eyebrow parameter model B right Respectively expressed as:

[0074]

[0075] Among them, P 21 , P 22 are the coordinate values ​​of the two key points (the 21st key point and the 22nd key point) corresponding to the right eyebrow, P 23 , P 24 are the coordinate values ​​of the two key points (the 23rd key point and the 24th key point) corresponding to the left eyebrow, P 37 , P 40 are the coordinate values ​​of the two key points (the 37th key point and the 40th key point) corresponding to the right eye, P 43 , P 46 are the coordinate values ​​of the two key points corresponding to the left eye (the 43rd key point and the 46th key point).

[0076] Set the first eyebrow threshold T B1 and the second state threshold T B2 , respectively determine the status of the left and right eyebrows:

[0077] ①If B left >TB1 , then the left eyebrow is judged to be in the raised eyebrow state; if B left <T B2 , then the left eyebrow is judged to be in a frowning state; if T B1 ≥B left ≥T B2 , then the left eyebrow is judged to be in normal state. ② If B right >T B1 , then the right eyebrow is judged to be in the raised eyebrow state; if B right <T B2 , then the right eyebrow is judged to be in a frowning state; if T B1 ≥B right ≥T B2 , then the right eyebrow is judged to be in normal state.

[0078] In some embodiments, the first eyebrow state threshold T B1 and the second state threshold T B2 Take 2.7 and 2.2 respectively, and adjust them appropriately according to the actual situation in different application scenarios.

[0079] 2. Eyelid status detection

[0080] (1) Before performing eyelid status detection, pupil position information needs to be obtained

[0081] For the left pupil, its position detection includes the following steps:

[0082] S101, according to the key points P of the left eye 43 , P 44 , P 45 , P 46 , P 47 , P 48 The minimum bounding rectangle of the left eye is calculated based on the coordinate values ​​of the left eye, and an image L corresponding to the bounding rectangle area is intercepted in the original image or video frame;

[0083] S102, connecting the key points of the left eye in sequence to form a closed area, and intercepting the left eye image L corresponding to the closed area in the image L eye ;

[0084] S103: Left eye image L eye Binarize the image and get image L thresh And get the image L thresh The outline of

[0085] S104, traverse each contour, if its area ratio S and the minimum circumscribed rectangle aspect ratio L satisfy T1>L>T2 and S>T S , then the contour is judged as a through hole, otherwise the output result is that there is no left pupil; where the area ratio S is the pixel area of ​​the contour and the image L threshThe ratio of the pixel area, T S is the area ratio threshold, which is generally 0.23; the aspect ratio L is the aspect ratio of the minimum circumscribed rectangle of the outline, T1 is the first aspect ratio threshold, which is generally 2, and T2 is the second aspect ratio threshold, which is generally 0.5;

[0086] S105: The center of the smallest circumscribed circle of the through hole is determined as the center P of the left pupil. c ;

[0087] S106, respectively calculate the center P of the left pupil c To the key point P 43 and key point P 46 The distances d1 and d2,

[0088] If d1 / (d1+d2) <T d , then the left pupil is considered to be facing inward, T d is the distance threshold, usually 0.3,

[0089] If d2 / (d1+d2) <T d , then the left pupil is considered to be facing outwards,

[0090] In other cases, the left pupil is considered to be in the middle of the left eye.

[0091] Similar to the left eye pupil, the position detection of the right eye pupil specifically includes the following steps:

[0092] S201, according to the key points P of the right eye 37 , P 38 , P 39 , P 40 , P 41 , P 42 The minimum bounding rectangle of the right eye is calculated based on the coordinate values ​​of , and an image R corresponding to the bounding rectangle area is intercepted in the original image or video frame;

[0093] S202, connecting the key points of the right eye in sequence to form a closed area, and intercepting the right eye image R corresponding to the closed area in the image R eye ;

[0094] S203, right eye image R eye Binarize the image and get image R thresh And get the image R thresh The outline of

[0095] S204, traverse each contour, if its area ratio S and the minimum circumscribed rectangle aspect ratio L satisfy T1>L>T2 and S>T S , then the contour is determined to be a through hole, otherwise the output result is that there is no right eye pupil;

[0096] S205: The center of the smallest circumscribed circle of the through hole is determined as the center P of the pupil of the right eye. c ;

[0097] S206, respectively calculate the center P of the left pupil c To the key point P 37 and key point P 40 The distances d1 and d2,

[0098] If d1 / (d1+d2) <T d , then the right pupil is considered to be facing inwards,

[0099] If d2 / (d1+d2) <T d , then the right pupil is considered to be facing outwards,

[0100] In other cases, the right pupil is considered to be in the middle of the right eye.

[0101] In some embodiments, when acquiring the left eye image L eye and the right eye image R eye Previously, histogram equalization was performed on images L and R to enhance the image contrast.

[0102] The binarization in image processing has the direct effect of dividing all pixels in the grayscale image into two categories, replacing the pixel values ​​of pixels whose values ​​are greater than the binarization threshold with 1 (white pixels), and replacing the pixel values ​​of other pixels with 0 (black pixels). It is a widely used image processing method in image processing technology. The key to its application in different scenarios is the determination of the binarization threshold.

[0103] In some embodiments, for the image binarization in steps S103 and S203 above, the determination of the binarization threshold value specifically includes the following steps:

[0104] S301, set the initial threshold nTh=5, set the best threshold nBestTh=5, set the minimum proportion minPer=1;

[0105] S302: perform bilateral filtering on the binarized eye image I, and then perform corrosion operation to obtain image I new ;

[0106] S303, using nTh to image I new Binarize and get image I th ;

[0107] S304, calculating image I th The proportion of black pixels in the total pixels of the image per;

[0108] S305. Calculate abs(per - 0.48). If abs(per - 0.48) < minPer, assign abs(per - 0.48) to minPer and assign nTh to nBestTh; otherwise, keep the values of minPer and nBestTh unchanged.

[0109] S306. Determine whether nTh is less than 100. If so, set nTh = nTh + 5 and go back to S303 to continue execution; otherwise, output the current best threshold nBestTh.

[0110] S307. Repeat the above process to obtain the best thresholds for three consecutive video frames respectively, then take the average, and use the average value as the final binarization threshold.

[0111] (2) Based on the pupil position information and the eyelid model, determine the eyelid state

[0112] For the left eyelid, if no left pupil is detected during the left eye pupil detection, it is determined that the left eyelid is in the closed state; otherwise, calculate the left eyelid parameter model E left =(||P 44 -P 48 || + ||P 45 -P 47 ||) / 2||P 43 -P 48 ||; if E left <T e then it is determined that the left eye is in the closed state; otherwise, it is in the open state. Here, T e is the eye state threshold, generally 0.55.

[0113] For the right eyelid, if no right pupil is detected during the right eye pupil detection, it is determined that the right eyelid is in the closed state; otherwise, calculate the right eyelid parameter model E right =(||P 38 -P 42 || + ||P 39 -P 41 ||) / 2||P 37 -P 42 ||; if E right <T e then it is determined that the right eye is in the closed state; otherwise, it is in the open state.

[0114] III. Mouth state detection

[0115] When performing mouth state detection, a corresponding mouth parameter model needs to be constructed. The mouth parameter model constructed in the present invention is: M=(||P 51 -P 59 || + ||P 53 -P 57||) / 2||P 49 -P 55 ||, where P 49 , P 51 , P 53 , P 55 , P 57 , P 59 is the mouth key point, if M>T m , the mouth is judged to be open, otherwise it is closed. m is the mouth posture threshold, which is generally taken as 0.6.

[0116] 4. Head status detection

[0117] (1) Determine the horizontal position of the head by calculating the ratio of the lengths of the left and right eyebrows:

[0118] The ratio of the length of the left eyebrow to the right eyebrow is: If BR1>T br The head is considered to be rotated to the left in the horizontal direction, T br is the rotation threshold, which is generally set to 1.3.

[0119] The ratio of the length of the right eyebrow to the left eyebrow is: If BR2>T br The head is considered to be rotated to the right in the horizontal direction.

[0120] In other cases, the head is considered to be in a neutral position in the horizontal direction.

[0121] (2) Detect the vertical state of the head

[0122] ① Calculate the key point P1 to the key point P 17 Distance H: H = || P1-P 17 ||;

[0123] ②Calculate the key point P 28 To the key points P1 and P 17 The straight line P1 P 17 Distance V:

[0124]

[0125] ③Calculate the ratio R of H to V HV : R HV =H / V;

[0126] ④Calculate the key point P 28 The y coordinates of the key points P1 and P 17 The difference in y coordinates of the midpoints is ΔY:

[0127] ΔY=P 28.y-(P1.yP 17 .y) / 2

[0128] Among them, P1.x and P1.y are the x-coordinate and y-coordinate of the key point P1, respectively. 17 .x and P 17 .y are the key points P 17 The x- and y-coordinates of P 28 .x and P 28 .y are the key points P 28 The x- and y-coordinates of .

[0129] If ΔY<0 and R HV >0.05, the head is considered to be in an upward position in the vertical direction; if ΔY>0 and R HV >0.05, the head is considered to be in a downward position in the vertical direction; otherwise, the head is considered to be in a middle position in the vertical direction.

[0130] Based on the above five sense organs state detection method, the present invention also provides a five sense organs state detection system, such as Figure 3 As shown, the detection system mainly includes an image acquisition module, a face key point detection module, an eyebrow state detection module, a pupil position detection module, an eyelid state detection module, a mouth state detection module, a head state detection module and a facial features state output module. The image acquisition module is arranged on a robot or other device, and is used to collect images or videos within the field of view and send them to the face key point detection module. The face key point detection module performs face detection on the received image or video frame based on the Dlib face detection technology, and obtains the position information of 68 key points of the face. The eyebrow state detection module is connected to the face key point detection module, and is used to obtain the state information of the left and right eyebrows according to the above-mentioned eyebrow state detection method, and send it to the facial features state output module. The pupil position detection module is connected to the face key point detection module, and is used to obtain the position state of the left and right pupils according to the above-mentioned pupil position detection method, and send it to the eyelid state detection module. The eyelid state detection module uses the pupil position state and the eyelid model to judge the state of the left and right eyelids respectively and send it to the facial features state output module. The mouth state detection module and the head state detection module are respectively connected to the face key point detection module. The mouth state detection module is used to obtain the mouth state information according to the above-mentioned mouth state detection method and send it to the facial features state output module. The head state detection module is used to obtain the head state information according to the above-mentioned head state detection method and send it to the facial features state output module. The facial features state output module is used to summarize the state information of eyebrows, eyelids, mouth, and head, and output it to downstream devices for use.

[0131] The present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is loaded by a central processing unit and executes the steps of the above-mentioned facial feature status detection method.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting facial features based on machine vision, characterized in that: It includes the following steps: Collect video frames or images within the machine's field of view; Based on the Dlib face detection technology, obtain the position information of facial key points in the video frames or images; Construct an eyebrow parameter model according to the position information of facial key points, and respectively judge the states of the left eyebrow and the right eyebrow; Respectively obtain the position states of the left eye pupil and the right eye pupil according to the position information of facial key points; Construct an eyelid parameter model according to the position information of facial key points, and respectively judge the states of the left eyelid and the right eyelid in combination with the position states of the left eye pupil and the right eye pupil; Construct a mouth parameter model according to the position information of facial key points, and judge the state of the mouth; Judge the state of the head according to the position information of facial key points; Summarize the states of the left eyebrow, the right eyebrow, the left eyelid, the right eyelid, the mouth and the head and output; The obtaining of the position state of the left eye pupil includes the following steps: S101, according to the facial key point P corresponding to the left eye 43 , P 44 , P 45 , P 46 , P 47 , P 48 The coordinates of the left eye are used to calculate the minimum bounding rectangle of the left eye, and the image corresponding to the bounding rectangle area is intercepted in the original video frame or image. L ; S102, connecting the facial key points corresponding to the left eye in sequence to form a closed area, and L The left eye image corresponding to the closed area is intercepted ; S103: Left eye image Binarize the image and get the image And get the image The outline of S104, traverse each contour, if its area ratio S and the minimum enclosing rectangle aspect ratio L satisfy and , then the contour is determined to be a through hole, otherwise the output result is that there is no left eye pupil; the area ratio S is the pixel area of ​​the contour and the image The ratio of the pixel area, the aspect ratio L is the aspect ratio of the minimum circumscribed rectangle of the outline; is the area ratio threshold, and are respectively a first aspect ratio threshold and a second aspect ratio threshold; S105: The center of the smallest circumscribed circle of the through hole is determined as the center of the left eye pupil. P cl ; S106, respectively calculating the center of the left eye pupil P cl To the key point and key points Distance d 1l and d 2l , determine the position of the left pupil: if , then the left pupil is considered to be facing inward; if , the left pupil is considered to be facing outward; if it is any other case, the left pupil is considered to be in the middle of the left eye; is the distance threshold.

2. The method for detecting facial features of a person as claimed in claim 1, wherein: The eyebrow parameter model includes a left eyebrow parameter model and right eyebrow parameter model , and there is in, P 21 and P 22 are the coordinates of the two key points corresponding to the right eyebrow, P 23 and P 24 are the coordinates of the two key points corresponding to the left eyebrow, P 37 and P 40 are the coordinates of the two key points corresponding to the right eye, P 43 and P 46 are the coordinates of the two key points corresponding to the left eye; set the first eyebrow threshold and the second state threshold , respectively judge the status of the left eyebrow and the right eyebrow: if , then the left eyebrow is judged to be in the raised eyebrow state; if , then the left eyebrow is judged to be in a frowning state; if , then the left eyebrow is judged to be in normal state; if , then the right eyebrow is judged to be in the raised eyebrow state; if , then the right eyebrow is judged to be in a frowning state; if , then the right eyebrow is judged to be in normal state.

3. The method for detecting facial features of a person as claimed in claim 1, wherein: The obtaining of the position state of the right eye pupil includes the following steps: S201, according to the facial key point P corresponding to the right eye 37 , P 38 , P 39 , P 40 , P 41 , P 42 The coordinates of the right eye are used to calculate the minimum bounding rectangle of the right eye, and the image corresponding to the bounding rectangle area is intercepted in the original video frame or image. R ; S202, connect the facial key points corresponding to the right eye in sequence to form a closed area, and R The right eye image corresponding to the closed area is intercepted ; S203, right eye image Binarize the image and get the image And get the image The outline of S204, traverse each contour, if its area ratio S and the minimum enclosing rectangle aspect ratio L satisfy and , then the contour is determined to be a through hole, otherwise the output result is that there is no right eye pupil; S205: The center of the smallest circumscribed circle of the through hole is determined as the center of the pupil of the right eye. P cr ; S206, respectively calculating the center of the left eye pupil P cr To the key point and key points Distance d 1r and d 2r , determine the position of the right pupil: if , then the right pupil is considered to be facing inward; if , the right pupil is considered to be facing outward; if it is other cases, the right pupil is considered to be in the middle of the right eye.

4. The method for detecting facial features of a person as claimed in claim 3, wherein: For the image binarization in steps S103 and S203, the determination of its binarization threshold includes the following steps: S301. Set the initial threshold nTh = 5, the optimal threshold nBestTh = 5, and the minimum ratio minPer = 1; S302: perform bilateral filtering on the binarized eye image I, and then perform corrosion operation to obtain image I new ; S303, using nTh to image I new Binarize and get image I th ; S304, calculating image I th The proportion of black pixels in the total pixels of the image per; S305. Calculate abs(per - 0.48). If abs(per - 0.48) < minPer, then assign abs(per - 0.48) to minPer and assign nTh to nBestTh; otherwise, keep the values of minPer and nBestTh unchanged; S306. Judge whether nTh is less than 100. If so, let nTh = nTh + 5 and go back to S303 to continue execution; otherwise, output the current optimal threshold nBestTh; S307. Repeat S301~S306, respectively obtain the optimal thresholds of three consecutive video frames and take the average, and use the average value as the final binarization threshold.

5. The method for detecting facial features of a person as claimed in claim 3, characterized in that: The state judgment of the left eyelid and the right eyelid includes the following steps: For the left eyelid, if the output is no left pupil when the position state of the left pupil is obtained, the left eyelid is determined to be closed, otherwise the left eyelid parameter model is calculated. ;like If the left eye is in the closed state, then it is judged that the left eye is in the open state; otherwise, For the right eyelid, if the output is no right pupil when the position state of the right pupil is obtained, the right eyelid is determined to be closed, otherwise the right eyelid parameter model is calculated. ;like The right eye is judged to be closed, otherwise it is open; is the eye state threshold.

6. The method for detecting facial features of a person as claimed in claim 1, wherein: The mouth parameter model is , where P 49 , P 51 , P 53 , P 55 , P 57 , P 59 is the coordinates of the facial key points corresponding to the mouth, if , the mouth is judged to be in an open state, otherwise it is in a closed state; is the mouth posture threshold.

7. The method for detecting facial features of a person as claimed in claim 1, wherein: The state judgment of the head includes the following steps: The horizontal position of the head is determined by calculating the ratio of the length of the left and right eyebrows: the ratio of the length of the left eyebrow to the length of the right eyebrow ,like The head is considered to be rotated to the left in the horizontal direction. is the rotation threshold; the ratio of the length of the right eyebrow to the left eyebrow is ,like If the head is rotated to the right, the head is considered to be in the horizontal direction; if it is the other case, the head is considered to be in the middle position in the horizontal direction; Judge the state of the head in the vertical direction: ①Calculate key points P 1 to the key point P 17 Distance H : ; ②Calculate key points P 28 To the key point P 1 and P 17 The straight line formed P 1 P 17 Distance V : ③Calculation H and V Ratio R HV : R HV =H / V ; ④Calculate key points P 28 The y coordinates of the key points P 1 and P 17 The difference in the y coordinates of the midpoints : in, and Key points P The x- and y-coordinates of 1, and Key points P 17 The x- and y-coordinates of and Key points P 28 The x- and y-coordinates of like and R HV > 0.05, the head is considered to be in an upward position in the vertical direction; if and R HV > 0.05, the head is considered to be in a downward position in the vertical direction; otherwise, the head is considered to be in a middle position in the vertical direction.

8. A facial features state detection system based on the method according to any one of claims 1 to 7, characterized in that: It includes An image acquisition module, disposed on the robot, for collecting video frames or images within the field of view and sending them to the facial key point detection module; A facial key point detection module, connected to the image acquisition module, for performing facial key point detection on the received images or video frames based on the Dlib face detection technology; An eyebrow state detection module, connected to the facial key point detection module, for obtaining the states of the left eyebrow and the right eyebrow and sending them to the facial feature state output module; A pupil position detection module, connected to the facial key point detection module, for obtaining the position states of the left eye pupil and the right eye pupil and sending them to the eyelid state detection module; An eyelid state detection module, connected to the pupil position detection module, for judging the states of the left eyelid and the right eyelid and sending them to the facial feature state output module; A mouth state detection module, connected to the facial key point detection module, for obtaining the state of the mouth and sending it to the facial feature state output module; The head state detection module is connected to the face key point detection module to obtain the state of the head and send it to the facial features state output module; The facial features status output module is used to summarize the status information of the left eyebrow, right eyebrow, left eyelid, right eyelid, mouth and head, and output it to downstream devices for use.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded by a central processing unit and executes the steps of the facial feature state detection method described in any one of claims 1 to 7.

Images