Facial fatigue analysis method based on photometric stereo 3D reconstruction

Abstract

The present invention discloses a method for analyzing facial fatigue based on photometric stereoscopic three-dimensional reconstruction, comprising the following steps: Step 1, collecting multiple two-dimensional facial images of a person under test; Step 2, synthesizing the two-dimensional facial images of the person under test into a three-dimensional fine feature image of the person under test; Step 3, determining the position of the eyes in the two-dimensional facial image of the person under test and separating the dark circles; Step 4, extracting the two-dimensional coordinates of spots in the color facial image and using the two-dimensional coordinates to find the corresponding three-dimensional fine feature matrix Z of the face; Step 5, removing the overly dark and overly bright areas of the eye grayscale image, and then using Frangi filtering to obtain a red eye bloodshot image; Step 6, predicting the working time of the person under test. The present invention uses the fatigue characteristics and working time data of each person under test at different times to train a BP neural network, thereby reducing the possibility that the corresponding relationship between fatigue characteristics and working time is different due to different physical conditions of the person under test.

Description

Technical Field

[0001] The present invention relates to the technical field of facial fatigue analysis, and in particular to a facial fatigue analysis method based on photometric stereoscopic three-dimensional reconstruction. Background Art

[0002] Photometric stereo is a passive, single-view measurement method that reconstructs the fine three-dimensional facial features of an object under different lighting conditions using shadow, brightness, and reflection information from the object's image. Due to its high reconstruction accuracy and low cost, photometric stereo has been widely used in fields such as medical treatment, cultural heritage preservation, game development, and facial reconstruction. Photometric stereo can reconstruct the fine three-dimensional facial features of an object, providing more information for solving fatigue analysis problems.

[0003] The solution to the fatigue risk prediction problem is generally to analyze the facial video information of the person being tested to detect the fatigue state of the person being tested. This method has been widely used to detect the fatigue of the driver in real time while driving. Because when a driver is in a fatigued state, his face will show visual fatigue characteristics such as changes in pupil size and gaze direction, changes in head posture and facial expression, and changes in the temporal information of facial movements such as blinking and yawning, so the driver's fatigue state while driving can be identified by monitoring the above visual characteristics through video. However, this method requires a camera device to detect the fatigue characteristics of the driver's face in real time. It is easily affected by the complex lighting in the outdoor environment and cannot extract the fatigue feature information of the driver's face. It also cannot predict the driver's fatigue condition, and has certain limitations. Summary of the Invention

[0004] The purpose of the present invention is to provide a facial fatigue analysis method based on photometric stereoscopic three-dimensional reconstruction to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for analyzing facial fatigue based on photometric stereoscopic three-dimensional reconstruction, comprising the following steps:

[0006] Step 1: Collect multiple two-dimensional facial images of the person being measured;

[0007] Step 2: synthesizing the two-dimensional facial image of the person being measured into a three-dimensional fine feature image of the person being measured;

[0008] Step 3: Determine the position of the eyes in the two-dimensional facial image of the person being measured and separate the dark circles;

[0009] Step 4: Extract the two-dimensional coordinates of the spots in the color face image and use the two-dimensional coordinates to find the corresponding three-dimensional facial fine feature matrix Z;

[0010] Step 5: Remove the dark and bright areas of the eye grayscale image, and then use Frangi filtering to obtain the red eye bloodshot image;

[0011] Step 6: Predict the working hours of the person being tested.

[0012] Preferably, the multiple two-dimensional facial images of the person being measured are collected in step 1 and are taken under the illumination of different LED light sources.

[0013] Preferably, the three-dimensional fine feature image in step 2 is obtained by synthesizing multiple two-dimensional facial images of the person being measured using a photometric stereo vision algorithm.

[0014] Preferably, in step three, the Adaboost classification algorithm and the Haar-like feature with integral graph are used to find the position of the eyes in the captured two-dimensional facial image of the person being measured, and the clustering algorithm is used to separate the dark circles, and the grayscale difference C1 at the center point of the dark circle area and the non-dark circle area is calculated and stored.

[0015] Preferably, in step four, the Gaussian Laplace operator is used to extract the two-dimensional coordinates of the spots in the color face image, and the two-dimensional coordinates are used to find the corresponding three-dimensional facial fine feature matrix Z, and the singular value decomposition method is used to obtain quantized noise information. If the noise information is greater than the set value, the spot can be determined to be a facial defect, and the number is recorded as N.

[0016] Preferably, in step five, a threshold method is used to remove overly dark and overly bright areas of the eye grayscale image, and then a Frangi filter is used to obtain a red eye bloodshot image, which is compared with the size of the eye to obtain a ratio C3.

[0017] Preferably, in step six, C1, N, and C3 are substituted into a preset empirical formula or C1, N, and C3 are input into a trained BP neural network model to predict the working hours of the person being measured.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] Analyzing the fatigue characteristics of the person being measured before work and predicting the person's working hours can reduce the possibility of the person being measured getting tired while working, thereby preventing the occurrence of work accidents. In addition, the working hours of the person being measured are predicted by combining multiple dimensions such as the three-dimensional fine features of the person's face and color photo information obtained through photometric stereoscopic three-dimensional reconstruction, thereby improving the accuracy of the prediction. At the same time, in order to use the fatigue characteristics of the person being measured to more accurately predict their working hours, the present invention uses the fatigue characteristics and working time data of each person being measured at different times to train the BP neural network, thereby reducing the possibility that the correspondence between fatigue characteristics and working hours is different due to different physical conditions of the person being measured. In addition, because the noise on the face of the person being measured is removed, the accurate facial normal vector of the person being measured is restored as much as possible, thereby improving the accuracy of reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of the steps of a fatigue prediction method based on photometric stereo vision of the present invention;

[0021] Figure 2 This is a schematic diagram of the present invention taking a two-dimensional image of the face of a person being measured;

[0022] FIG3( a ) is a three-dimensional fine feature image of the face of a person under test restored in an embodiment of the present invention;

[0023] FIG3( b ) is a three-dimensional fine feature image of the face of the person being measured restored in an embodiment of the present invention;

[0024] Figure 4 A two-dimensional color face image captured in an embodiment of the present invention;

[0025] FIG5( a ) is a human eye image segmented using an Adaboost classifier in an embodiment of the present invention;

[0026] FIG5( b ) is an image of a human eye with black triangle labels removed from the interference information above the eyelashes in an embodiment of the present invention;

[0027] The white portion in FIG5( c ) is the dark circle area found by color clustering positioning in an embodiment of the present invention;

[0028] Figure 6 The speckle image is detected using the LOG operator and processed to remove interference information in an embodiment of the present invention;

[0029] FIG7( a ) is a three-dimensional fine feature image of a human face with facial defect information in an example of the present invention;

[0030] FIG7( b ) is a three-dimensional fine feature image of a human face using singular value decomposition to repair facial defect information in an example of the present invention;

[0031] FIG8( a ) is an input eye image with bloodshot information in an example of the present invention;

[0032] FIG8( b ) is a binary image obtained by filtering the original image through Frangi filtering in an example of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] See also Figure 1-8(b) The present invention provides a technical solution: a method for predicting facial fatigue based on photometric stereoscopic three-dimensional reconstruction, comprising the following steps:

[0035] Step 1: Collect multiple two-dimensional facial images of the person being measured taken under different LED light sources.

[0036] Step 2: Using a photometric stereo vision algorithm, the multiple acquired two-dimensional facial images of the subject are synthesized into a three-dimensional fine feature image of the subject's face;

[0037] Step 3: Use the Adaboost classification algorithm and the Haar-like feature with integral image to find the position of the eyes in the captured two-dimensional facial image of the person being measured, and use the clustering algorithm to separate the dark circles. Calculate and store the grayscale difference C1 at the center point of the dark circle area and the non-dark circle area.

[0038] Step 4: Use the Laplace of Gaussian (LOG) operator to extract the two-dimensional coordinates of the spots in the color face image, and use the two-dimensional coordinates to find the corresponding three-dimensional facial fine feature matrix Z. Use the singular value decomposition (SVD) method to obtain quantized noise information. If the noise information is greater than a set value, it can be determined that the spot is a facial defect. The number is recorded as N;

[0039] Step 5: Use the threshold method to remove the dark and bright areas of the eye grayscale image, then use Frangi filtering to obtain the red eye bloodshot image, and compare it with the size of the eye to obtain the ratio C3;

[0040] Step 6: Substitute C1, N, and C3 into a preset empirical formula or input C1, N, and C3 into a trained BP neural network model to predict the working hours of the person being measured.

[0041] The specific implementation of step one is as follows:

[0042] Fix the position of the CCD camera so that its shooting direction is as perpendicular as possible to the position to be measured;

[0043] Set up multiple LED light sources at different positions around the CCD camera so that the light from each LED light source can evenly illuminate the position to be measured;

[0044] Set the lighting mode of LED light source so that each LED light source can emit light independently;

[0045] The LED light sources are controlled to light up in sequence, and a two-dimensional image of the face of the person being measured is captured when each individual LED light source is lit. The number of captured two-dimensional images is the same as the number of LED light sources.

[0046] The specific implementation of step 2 is as follows:

[0047] Based on the captured two-dimensional image of the face of the person being measured, the surface normal vector of the face of the person being measured is calculated. When solving the surface normal vector of the face of the person being measured, due to the influence of some noise (such as highlights, shadows, etc.), there is a certain error between the solved surface normal vector of the face of the person being measured and the true value. The present invention provides a least squares method of a kernel function to minimize the influence of noise on the result. Experiments show that the more the face of the person being measured approximates the surface of the Lambertian body, the smaller the error of the obtained surface normal vector of the face of the person being measured. The specific implementation method is as follows:

[0048] According to the calibration data, the error function is constructed as follows:

[0049]

[0050] in is the i-th pixel, p is the total number of face pixels of the person being tested, q is the number of lights, I i is a q×1 grayscale pixel vector, L i is a 3×q light matrix, N i is the 3×1 surface normal vector, ρ i It is expressed as the reflectivity value of the pixel under each light. Represents ρ i Multiply each element of the vector matrix by The elements at the corresponding positions of the vector, L i and ρ i Obtained through calibration; construct the target optimization function:

[0051]

[0052] Perform k iterations to solve the above target optimization function:

[0053] Initialize w i,0 is the identity matrix of q×q, and the maximum number of iterations is k max , the minimum error is e min =10 -6 , N i,0 =0, c is a real number close to zero but greater than zero, here c = 0.01;

[0054] Solve the corresponding N for the kth iteration i,k ,in:

[0055]

[0056] Then the k-th generation error function is:

[0057]

[0058] Update weight function:

[0059]

[0060] For a q×1 vector, diag converts the diagonal element q×1 vector into a q×q diagonal matrix with only diagonal elements;

[0061] If ||N i,k -N i,k-1 ||≤e min or ||e i,k -e i,k-1 ||≤e min , then stop the iteration.

[0062] Calculating a gradient vector of the face of the person being measured based on the calculated surface normal vector of the face of the person being measured;

[0063] The three-dimensional fine features of the face of the person being measured are obtained by integrating the calculated gradient vector of the face of the person being measured.

[0064] The specific implementation of step three is as follows:

[0065] According to the Haar-like features with integral graph, Adaboost classifier is trained. Figure 4 The trained Adaboost classifier is used to locate the left and right eyes of the subject, and the image obtained is shown in Figure 5(a);

[0066] Use a low threshold of threshold = 0.1 to convert the grayscale images of the left and right eyes of the person being tested into binary images to obtain the eyelash position. The binary conversion formula used is:

[0067]

[0068] Where bw(i,j) is the pixel value of row i and column j in the binary image matrix of the eye of the person being tested, and gray(i,j) is the pixel value of row i and column j in the grayscale image matrix of the eye of the person being tested;

[0069] According to the coordinate positions of the black pixel points in the binary image of the eye of the person being tested, the least squares method is used to fit the linear equation to segment the RGB color image of the eye of the person being tested into images with black triangle labels, as shown in Figure 5(b);

[0070] The segmented RGB images of the left and right eyes of the subject are converted into Lab images, and the second and third channels are clustered using a clustering algorithm to obtain multiple images after color classification;

[0071] The clustered image is subjected to median filtering and closing operations to remove noise and smooth the image, and then converted into a binary image.

[0072] Filter out clustered images with black triangle labels;

[0073] The average geometric distance between the pixels of the filtered dark circle image and the linear equation is the smallest, and the image with the smallest distance is the image where the dark circle is located.

[0074] The specific implementation of step 4 is as follows:

[0075] The two-dimensional facial image of the subject is convolved using the Laplace of Gaussian (LOG) operator to obtain a smoothed facial image. The convolution expression formula is:

[0076]

[0077] Where B represents the input image, A is the smoothed facial image after convolution, and * represents the image convolution operation;

[0078] Find local extreme values ​​in the smoothed facial image of the person being tested and sort them. The 20 largest extreme value areas are regarded as spot areas.

[0079] Set the threshold to exclude the two-dimensional coordinates of n spots that are too large, too small, or too dark, such as the nostrils, Figure 6 As shown;

[0080] Using the found 2D coordinates of the spots, find each facial 3D fine feature matrix in the 3D fine features of the face after photometric 3D reconstruction

[0081] Perform singular value decomposition (SVD) on the facial 3D fine feature matrix Zi of the i-th spot to separate the noise information matrix. The absolute values ​​of all elements of the noise information matrix of the i-th spot are averaged to obtain Ri, as shown in Figure 7(a) and Figure 7(b). The red box area in Figure 7(a) is the 3D fine facial feature of the facial defect on the face of the subject, and the red box area in Figure 7(b) is the 3D fine facial feature of the facial defect after repair.

[0082] Set the threshold Tthreshold. If Ri>Tthreshold, the i-th spot is determined to be a facial defect, and the number of facial defects is N.

[0083] The number of facial defects N of the person being tested each day is stored in an Excel table.

[0084] The specific implementation of step five is as follows:

[0085] Input the test person's eyes or the test person's eyes photo matrix IRGB (RGB image) and I (grayscale image) cropped by Adaboost classifier, as shown in Figure 8(a), assuming

[0086] In order to eliminate the influence of highlights in the photo on the detection results, a higher threshold T is set. high =0.75, let max(I) be the value of the largest element in the photo matrix I, if I(i,j)>T high *max(I), Then I(i,j)=0, and the position of the highlight area is recorded as the set Ω high ;

[0087] In order to quickly find the position of the pupil and eliminate the influence of eyelashes on the subsequent operation, a lower threshold T is set. low =0.35, let max(I) be the largest element value in the photo matrix I, if I(i,j) <T low *max(I), Then I(i,j)=0, and the positions of pupil and eyelash are set Ω low , this part of the area can be considered as the area of ​​the eye except the white of the eye;

[0088] Apply Frangi filtering to the eye image matrix I to obtain the filtered image matrix I'. If (i, j)∈Ω low ∪Ω high , then I′(i,j)=0;

[0089] Convert I′ into a binary image matrix and find the largest connected area B, which stores I RGBand the position of certain pixels I, i.e., the red box area in Figure 8(b) in this example;

[0090] If the pixel at position B is in I RGB If the image matches the red range, then area B is the bloodshot area, and we proceed to the next step; if the pixel at position B is in I RGB If the image does not match the red range, delete B in the connected area and repeat the previous step. The processed image is then RGB encoded.

[0091] Let Ω be the set low The number of elements is n low , let the number of elements in set B be n red ,

[0092]

[0093] Store C3 in an Excel spreadsheet.

[0094] The specific implementation of step six is ​​as follows:

[0095] Arrange the data, put the data of the same person at different times together, and arrange the data in ascending order of time. Each person has i groups of data, and the data to be predicted is the jth group of data, and its characteristic value is C 1,j 、C 2,j 、C 3,j ;

[0096] Note C 2,1 =0 is the C2 of the first group of data, for the subject The group data are:

[0097] C 2,j =N j-1 -N j (9)

[0098] Input the actual working time of the person under test in group i, recorded as Rtime i ;

[0099] The C in each set of data 1,j 、C 2,j 、C 3,j As the input of the neural network, Rtime j As the output of the neural network, the BP neural network model is trained, where

[0100] The C that needs to be predicted 1,i 、C 2,i 、C 3,i Input the trained BP neural network model and output the value Rtime iThis is the predicted working time of the person being tested.

[0101] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0102] Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise expressly specified and defined, the terms "connection" and the like should be understood in a broad sense, such as "connection", which may be a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection, an indirect connection through an intermediate medium, or a communication between the internal parts of two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention according to specific circumstances.

Claims

1. A facial fatigue analysis method based on photometric stereo 3D reconstruction, characterized in that: The following steps are involved: Step 1: Collect multiple two-dimensional facial images of the person being measured; Step 2: synthesizing the two-dimensional facial image of the person being measured into a three-dimensional fine feature image of the person being measured; Step 3: Determine the position of the eyes in the two-dimensional facial image of the person being measured and separate the dark circles; Step 4: Extract the 2D coordinates of the spots in the color face image and use the 2D coordinates to find the corresponding 3D facial fine feature matrix ; Step 5: Remove the dark and bright areas of the eye grayscale image, and then use Frangi filtering to obtain the red eye bloodshot image; Step 6: Predict the working hours of the person being tested; In step 3, the Adaboost classification algorithm and the Haar-like feature with integral image are used to find the position of the eyes in the two-dimensional facial image of the person being tested, and the clustering algorithm is used to separate the dark circles, and the grayscale difference between the center point of the dark circle area and the non-dark circle area is calculated and stored. In step 4, the Gaussian Laplace operator is used to extract the two-dimensional coordinates of the spots in the color face image, and the two-dimensional coordinates are used to find the corresponding three-dimensional facial fine feature matrix , the singular value decomposition method is used to obtain quantized noise information. If the noise information is greater than the set value, the spot can be determined to be a facial defect. The number is recorded as N. In the step 5, the threshold method is used to remove the dark and bright areas of the eye grayscale image, and then the Frangi filter is used to obtain the red eye bloodshot image, which is compared with the size of the eye to obtain the ratio In step six, 、N、 Substitute the preset empirical formula or 、N、 Inputting the trained BP neural network model can predict the working hours of the person being tested.

2. The method for facial fatigue analysis based on photometric stereo 3D reconstruction according to claim 1, characterized in that: The multiple two-dimensional facial images of the person being measured are collected in step 1 and are taken under the illumination of different LED light sources.

3. The method for facial fatigue analysis based on photometric stereo 3D reconstruction according to claim 1, characterized in that: The three-dimensional fine feature image in step 2 is obtained by synthesizing multiple two-dimensional facial images of the person being measured through a photometric stereo vision algorithm.

Images