Face optimal frame selection method and device and electronic equipment

Abstract

The application provides a face optimal frame selection method and device and electronic equipment. The method calculates the quality score of a face picture by face detection confidence, face size, face Euler angle, face blocking degree and face blur degree. For the same ID face, the quality score calculated in the current frame picture is defined as the first quality score, the quality score calculated in the optimal frame determined before the current frame is defined as the second quality score, the first quality score is compared with the second quality score, if the first quality score is greater than or equal to the second quality score, the optimal frame is replaced by the current frame, otherwise, the optimal frame remains unchanged. When the time length of the face tracking model losing one ID face exceeds the preset time threshold, the face image of the optimal frame is output, the face optimal frame under the same monitoring picture can be automatically recognized and selected, and then the storage amount is reduced by storing the face optimal frame, and the face searching efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a face optimal frame selection method and device and electronic equipment. BACKGROUND

[0002] In order to protect public safety, face cameras are installed at places such as cell door entrances, shopping mall entrances and pedestrian paths to obtain pedestrian face information. This information is usually stored in the form of a video, which requires a large storage space and is not easy to find the required face information. SUMMARY

[0003] The present application aims to provide a face optimal frame selection method that can automatically identify and select a face optimal frame under the same monitoring screen, thereby reducing the storage amount by storing the face optimal frame and improving the face searching efficiency.

[0004] To achieve the above-mentioned purpose, the present application first provides a face optimal frame selection method, comprising the following steps:

[0005] Step S1: obtaining a training picture set, labeling the key point positions and attributes of the faces in each training picture in the training picture set, and training a face detection model using the labeled training picture set;

[0006] Step S2: receiving a video stream, extracting a to-be-detected image from the video stream and inputting the to-be-detected image into the trained face detection model to obtain first information of each face in the to-be-detected image, wherein the first information includes a detection frame boundary of the face, a key point position of the face, an attribute of the face, and a detection confidence of the face;

[0007] Step S3: obtaining second information of each face according to the first information of each face and a preset information conversion algorithm, wherein the second information includes an Euler angle of the face, a shielding degree value of the face, and a blur degree value of the face;

[0008] Step S4: tracking the detected faces by a face tracking model to determine the ID of each face, and determining the quality score of each face with each ID in each frame according to the second information of the face with each ID, the face detection confidence, and a preset quality score algorithm;

[0009] Step S5: for the face with the same ID, defining the quality score calculated in the current frame as a first quality score, defining the quality score calculated in the optimal frame determined before the current frame as a second quality score, comparing the first quality score with the second quality score, and if the first quality score is greater than or equal to the second quality score, replacing the optimal frame with the current frame, otherwise keeping the optimal frame unchanged;

[0010] Step S6: When the duration for which the face tracking model loses a face ID exceeds a preset duration threshold, output the face image of the optimal frame.

[0011] Optionally, the preset information conversion algorithm in step S3 includes:

[0012] Different face occlusion degree calculation formulas are selected according to the different attributes of the face in the first information. The attributes of the face in the first information include at least: the mask occlusion status of the face and the skin color area.

[0013] The occlusion degree of the face is calculated based on the determined formula for calculating the degree of face occlusion, the boundary of the face detection box, and the location of the key points of the face.

[0014] Optionally, different formulas for calculating the degree of face occlusion are selected based on the different attributes of the face in the first information. These formulas specifically include:

[0015] When the mask covers both the nose and mouth of the face, the formula for calculating the degree of facial occlusion is the first formula.

[0016] When the area of ​​a person's face covered by a mask is such that the mask covers the mouth but exposes the nose, the formula for calculating the degree of face occlusion is the second formula.

[0017] When the area of ​​a person's face covered by a mask is such that the mouth and nose are exposed, the formula for calculating the degree of face occlusion is the third formula.

[0018] When the area of ​​a person's face covered by a mask is not covered by a mask, the formula for calculating the degree of face occlusion is the fourth formula.

[0019] Alternatively, the first formula is: ;

[0020] The second formula is: ;

[0021] The third formula is: ;

[0022] The fourth formula is: ;

[0023] Where `occupy` represents the degree of occlusion of the face, `area_skin` represents the area of ​​the skin-colored region, `area_face` represents the face area, `x1` and `y1` are the x and y coordinates of the top-left corner of the face detection box, `x2` and `y2` are the x and y coordinates of the bottom-right corner of the face detection box, and `y`... le y re y n y lm and y rmA vertical coordinate of a left eye position, a vertical coordinate of a right eye position, a vertical coordinate of a nose tip position, a vertical coordinate of a left corner of a mouth, and a vertical coordinate of a right corner of a mouth of the face.

[0024] Optionally, the preset information conversion algorithm in the step S3 comprises:

[0025] According to the key point position of the face in the first information, the Euler angle of the face in the second information is calculated;

[0026] The face image in the detection frame boundary is extracted;

[0027] The face image in the detection frame boundary is detected by the cv2.Laplacian algorithm to obtain the blur degree value of the face;

[0028] Optionally, the preset quality score algorithm in the step S4 is:

[0029]

[0030] Wherein, the score is the quality score of the face of the ID in the frame picture, the conf is the detection confidence of the face of the ID in the frame picture, the size is the detection frame size of the face of the ID in the frame picture, the pitch, yaw and roll are the Euler angles of the face of the ID in the frame picture, the occupy is the occlusion degree value of the face of the ID in the frame picture, the blur is the blur degree value of the face of the ID in the frame picture, a1, a2, a3, a4, a5, a6 and a7 are proportional constants, and the values of a1, a2, a3, a4, a5, a6 and a7 are determined by the multivariate gradient descent method.

[0031] Optionally, the face training model is a retinaface network model with an attribute recognition head and an attribute recognition function, so that the face training model after training can detect the detection frame boundary of the face, the key point position of the face, the attribute of the face and the detection confidence of the face.

[0032] The face tracking model is a deep-sort network model in which the feature extraction network is improved from a resnet18 network to a mobilefacenet network.

[0033] The application also provides a face optimal frame selection device, comprising:

[0034] A training unit is configured to acquire a training picture set, mark the key point position and attribute of the face in each training picture in the training picture set, and train a face detection model by using the marked training picture set.

[0035] ​The detection unit is configured to receive a video stream, extract a to-be-detected image from the video stream, input the to-be-detected image into a trained face detection model, and obtain first information of each face in the to-be-detected image, wherein the first information includes a detection frame boundary of the face, a key point position of the face, a property of the face, and a detection confidence of the face.

[0036] The calculation unit is configured to obtain second information of each face according to the first information of the face and a preset information conversion algorithm, wherein the second information includes an Euler angle of the face, an occlusion degree value of the face, and a blur degree value of the face.

[0037] The tracking unit is configured to track the detected face by using a face tracking model, determine an ID of each face, and determine a quality score of each face with the ID in each frame according to the second information of the face with the ID in each frame, a face detection confidence, and a preset quality score algorithm.

[0038] The comparison unit is configured to, for a face with the same ID, define a quality score of the face calculated in a current frame as a first quality score, define a quality score of the face calculated in an optimal frame determined before the current frame as a second quality score, compare the first quality score with the second quality score, replace the optimal frame with the current frame if the first quality score is greater than or equal to the second quality score, or keep the optimal frame unchanged if the first quality score is less than the second quality score.

[0039] The output unit is configured to output a face image of the optimal frame when a time length for which the face tracking model loses a face with an ID exceeds a preset time length threshold.

[0040] The application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to make the processor execute steps of the method.

[0041] The present application provides a face optimal frame selection method, device and electronic equipment. The method comprises the following steps: obtaining a training picture set, labeling key point positions and attributes of faces in each training picture in the training picture set, and training a face detection model by using the labeled training picture set; receiving a video stream, extracting a to-be-detected image from the video stream, inputting the to-be-detected image into the trained face detection model, and obtaining first information of each face in the to-be-detected image, wherein the first information comprises a detection frame boundary of the face, a key point position of the face, an attribute of the face and a detection confidence of the face; obtaining second information of each face according to the first information of each face and a preset information conversion algorithm, wherein the second information comprises an Euler angle of the face, an occlusion degree value of the face and a blur degree value of the face; tracking the detected face by using a face tracking model, determining an ID of each face, and determining a quality score of each ID face in each frame picture according to the second information of each ID face in each frame picture, the face detection confidence and a preset quality score algorithm; for the face of the same ID, defining a quality score calculated in a current frame picture as a first quality score, a quality score calculated in an optimal frame determined before the current frame as a second quality score, comparing the first quality score with the second quality score, if the first quality score is greater than or equal to the second quality score, replacing the optimal frame with the current frame, otherwise keeping the optimal frame unchanged; when a time length of losing tracking of the face of one ID by the face tracking model exceeds a preset time threshold, outputting a face image of the optimal frame, which can automatically identify and select a face optimal frame under the same monitoring picture, and then reduce the storage amount by storing the face optimal frame and improve the face searching efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0042] For further understanding of the features and technical contents of the present application, please refer to the following detailed description of the present application and the attached drawings, however, the drawings are provided for reference and illustration only, and are not used to limit the present application.

[0043] In the drawings,

[0044] Figure 1 The flow chart of the face optimal frame selection method of the present application;

[0045] Figures 2-3 The flow chart of step S3 of the face optimal frame selection method of the present application;

[0046] Figure 4 The face key point position diagram of the face optimal frame selection method of the present application;

[0047] Figures 5-8 The different occlusion area diagram of the face of the face optimal frame selection method of the present application;

[0048] Figure 9A schematic diagram of the face optimal frame selection device of the present application;

[0049] Figure 10 A schematic diagram of the electronic device of the present application. DETAILED DESCRIPTION

[0050] For further illustrating the technical means adopted by the present application and its effects, the following will make a detailed description in combination with the preferred embodiments of the present application and its drawings.

[0051] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects of example implementations to those skilled in the art.

[0052] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0053] The block diagrams shown in the accompanying drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0054] The flowcharts shown in the accompanying drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0055] Referring to Figure 1 The present application provides a face optimal frame selection method, comprising the following steps:

[0056] Step S1, obtain a training picture set, label the key point positions and attributes of the face in each training picture in the training picture set, and train a face detection model using the labeled training picture set.

[0057] Specifically, as Figure 4As shown, the key point positions of the face include five positions, i.e., left eye le, right eye re, nose tip n, left mouth corner lm, and right mouth corner rm.

[0058] Specifically, the attributes of the face mainly include a mask occlusion state of the face and a skin color region, and the mask occlusion state of the face specifically includes whether the face is occluded by a mask and which regions of the face are occluded by the mask when the face is occluded by the mask, and the skin color region specifically refers to the position and size of a region in the training picture whose color is the same as or close to human skin color.

[0059] Of course, according to needs, the attributes of the face in some embodiments of the present application can further include the attributes of gender, beard, hat, and glasses of the face.

[0060] Further, the face training model is an improved retinaface network model, which is improved by adding an attribute recognition head and an attribute recognition function, so that the face training model after training can simultaneously detect the bounding box boundary of the face, the key point position, the attribute, and output the detection confidence, so as to facilitate subsequent calculation of the quality score of the optimal frame according to the key point position, so that the quality score of the optimal frame can be determined by simultaneously referring to the face detection confidence, the face size, the face Euler angle, the face occlusion degree, the face blur degree, and other factors, the determination of the quality score is more comprehensive, and the optimal frame is more accurate.

[0061] Step S2, receiving a video stream, extracting a to-be-detected image from the video stream, inputting the to-be-detected image into the trained face detection model, and obtaining first information of each face in the to-be-detected image, the first information including the bounding box boundary of the face, the key point position of the face, the attribute of the face, and the detection confidence of the face.

[0062] Step S3, obtaining second information of each face according to the first information of the face and a preset information conversion algorithm, the second information including the Euler angle of the face, the occlusion degree value of the face, and the blur degree value of the face.

[0063] Specifically, the preset information conversion algorithm in step S3 includes calculating the Euler angle of the face in the second information according to the key point position of the face in the first information, in detail, i.e., calculating three angles of pitch, yaw, and roll by using the five key point positions of the face.

[0064] Specifically, the preset information conversion algorithm in step S3 includes:

[0065] As shown in the formula (1), the face occlusion degree value is calculated according to the attribute of the face in the first information. Figure 2 As shown in the formula (2), the face occlusion degree value is calculated according to the attribute of the face in the first information.

[0066] The face occlusion degree value of the face is calculated according to a face occlusion degree value calculation formula, a face detection frame boundary, and a face key point position.

[0067] Specifically, the face attribute in the first information includes a mask occlusion state of the face and a skin color region.

[0068] According to different face attributes in the first information, different face occlusion degree value calculation formulas are selected.

[0069] As shown in the first formula, Figure 5 when the mask occlusion region of the face is with a mask and covers the nose and the mouth at the same time, the face occlusion degree value calculation formula is determined as the first formula.

[0070] As shown in the second formula, Figure 6 when the mask occlusion region of the face is with a mask and covers the mouth and exposes the nose, the face occlusion degree value calculation formula is determined as the second formula.

[0071] As shown in the third formula, Figure 7 when the mask occlusion region of the face is with a mask and exposes the mouth and exposes the nose, the face occlusion degree value calculation formula is determined as the third formula.

[0072] As shown in the fourth formula, Figure 8 when the mask occlusion region of the face is without a mask, the face occlusion degree value calculation formula is determined as the fourth formula.

[0073] Further, the first formula is:

[0074] The second formula is:

[0075] The third formula is:

[0076] The fourth formula is:

[0077] wherein, occupy is the face occlusion degree value, area_skin is the area of the skin color region, area_face is the area of the face, x1 and y1 are the horizontal coordinate and the vertical coordinate of the top left corner of the face detection frame, x2 and y2 are the horizontal coordinate and the vertical coordinate of the bottom right corner of the face detection frame, y le , y re , y n , y lm and y rm are the vertical coordinates of the left eye position, the right eye position, the nose tip position, the left corner of the mouth position, and the right corner of the mouth position of the face respectively.

[0078] ​​​​In addition, it is worth mentioning that in some embodiments of the present application, the determination of the face occlusion degree value also takes into account that when the face is occluded by other faces, the skin color extraction will be wrong, so the iou calculation of the face and other faces is added, and in general cases the face is occluded by the face standing in front of the face, so only the iou of the face and the face standing in front of the face is calculated, and the area of the face occluded by the face standing in front of the face is deducted from the area_face part of the above first to fourth formulas.

[0079] Specifically, as shown in Figure 3 the preset information conversion algorithm in step S3 includes:

[0080] extracting the face image within the detection frame boundary;

[0081] detecting the blur of the face image within the detection frame boundary by the cv2.Laplacian algorithm to obtain the blur degree value of the face.

[0082] Step S4, track the detected face by a face tracking model, determine the ID of each face, and determine the quality score of each ID face in the frame according to the second information of each ID face in each frame, the face detection confidence and the preset quality score algorithm.

[0083] Specifically, the face tracking model is an improved deep-sort network model, and the specific improvement is to replace the default feature extraction model resnet18 with mobilefacenet. The improved deep-sort network model is used for face tracking to obtain the ID of the face.

[0084] Step S5, for the face with the same ID, define the quality score calculated in the current frame as the first quality score, and the quality score calculated in the optimal frame determined before the current frame as the second quality score, compare the first quality score with the second quality score, if the first quality score is greater than or equal to the second quality score, replace the optimal frame with the current frame, otherwise keep the optimal frame unchanged.

[0085] Specifically, the preset quality score algorithm in step S4 is:

[0086] ;

[0087] Wherein, score is the quality score of the face of the ID in the frame picture, conf is the detection confidence of the face of the ID in the frame picture, size is the detection box size of the face of the ID in the frame picture, pitch, yaw and roll are the Euler angles of the face of the ID in the frame picture, occupy is the occlusion degree value of the face of the ID in the frame picture, blur is the blur degree value of the face of the ID in the frame picture, a1, a2, a3, a4, a5, a6, a7 are proportional constants.

[0088] Wherein, wherein the calculation of a1, a2, a3, a4, a5, a6, a7 is a multivariate linear regression problem, which can be solved by multivariate gradient descent method, and the specific steps are as follows:

[0089] Firstly, prepare N sets of face data of different IDs, wherein each set of data, i.e. each ID, has M pictures of different sizes, angles, occlusion degrees and blur degrees;

[0090] Then, the values of each set of data are calculated in advance, and are counted as follows:

[0091] id conf size pitch yaw roll occupy blur score 1_1 0.7 50 3 4 5 0.3 0.5 0.6 ... 1_M 0.8 60 6 7 8 0.6 0.7 0.5 ... N_1 0.8 60 6 7 8 0.6 0.7 0.5 ... N_M 0.8 60 6 7 8 0.6 0.7 0.5

[0092] Then, the multivariate gradient descent method is used to calculate a i1 , a i2 , a i3 , a i4 , a i5 , a i6 , a i7 of each group, wherein i=1, 2, …, N, and then the parameters of the N groups are averaged to obtain the final parameters, i.e.

[0093]

[0094] Wherein, i=1, 2, …, N represents N groups, and j=1, 2, …, 7 represents the parameter number.

[0095] Further, the calculation of the quality score of the face is finally completed.

[0096] Step S6, when the face tracking model loses the face of an ID for a time exceeding a preset time threshold, output the face image of the optimal frame.

[0097] In actual application, for steps S5 and S6, for the face of the same ID, the image in the current frame picture is defined as current_img, the image in the optimal frame determined before the current frame picture is defined as best_img, the image quality score of the image current_img in the current frame picture is defined as current_score, the image quality score of the image best_img in the optimal frame determined before the current frame picture is defined as best_score, if current_score>best_score, the optimal frame of the face of the ID and the corresponding face quality score are updated, that is, best_img=current_img and best_score=current_score, until the face of the ID leaves the monitoring picture, the optimal frame best_img is saved and output, so that the face optimal frame under the same monitoring picture can be automatically recognized and selected according to the application, the speed is fast, the face optimal frame quality is high, the quality score calculation considers the face detection confidence, the face size, the face Euler angle, the face occlusion degree, the face blur degree and other factors, is more comprehensive, and when the face occlusion degree is calculated, the face mask attribute, the skin color area, the key point and other information are considered, and is more accurate.

[0098] Please refer to Figure 9 The application further provides a face optimal frame selecting device, comprising:

[0099] A training unit 10 is configured to acquire a training picture set, label key point positions and attributes of faces in each training picture in the training picture set, and train a face detection model by using the labeled training picture set.

[0100] A detection unit 20 is configured to receive a video stream, extract a to-be-detected image from the video stream, input the to-be-detected image into the trained face detection model, and obtain first information of each face in the to-be-detected image, wherein the first information comprises a detection frame boundary of the face, a key point position of the face, an attribute of the face, and a detection confidence of the face.

[0101] A calculation unit 30 is configured to obtain second information of each face according to the first information of the face and a preset information conversion algorithm, wherein the second information comprises an Euler angle of the face, an occlusion degree value of the face, and a blur degree value of the face.

[0102] A tracking unit 40 is configured to track the detected faces by using a face tracking model, determine an ID of each face, and determine a quality score of each face of each ID in each frame picture according to the second information of the face of each ID in the frame picture, a face detection confidence, and a preset quality score algorithm.

[0103] The comparison unit 50 is configured to, for a face of the same ID, define a quality score calculated in a current frame as a first quality score, a quality score calculated in an optimal frame determined before the current frame as a second quality score, compare the first quality score with the second quality score, and replace the optimal frame with the current frame if the first quality score is greater than or equal to the second quality score, or keep the optimal frame unchanged if the first quality score is less than the second quality score.

[0104] The output unit 60 is configured to output a face image of the optimal frame when a time length for which the face tracking model loses a face of an ID exceeds a preset time length threshold.

[0105] Please refer to Figure 10 The present application also provides an electronic device, comprising a memory 100 and a processor 200, the memory 100 stores a computer program, and the computer program is executed by the processor 200 to make the processor 200 execute the steps of the above method.

[0106] In summary, the present application provides a face optimal frame selection method, device and electronic device. The method comprises the following steps: obtaining a training picture set, labeling key point positions and attributes of faces in each training picture in the training picture set, and training a face detection model using the labeled training picture set; receiving a video stream, extracting a to-be-detected image from the video stream, inputting the to-be-detected image into the trained face detection model, and obtaining first information of each face in the to-be-detected image, the first information comprising a detection frame boundary of the face, a key point position of the face, an attribute of the face and a detection confidence of the face; obtaining second information of each face according to the first information of each face and a preset information conversion algorithm, the second information comprising an Euler angle of the face, an occlusion degree value of the face and a blur degree value of the face; tracking the detected face by a face tracking model, determining an ID of each face, and determining a quality score of each face of each ID in each frame according to the second information of each face of each ID in each frame, the face detection confidence and a preset quality score algorithm; for a face of the same ID, defining a quality score calculated in a current frame as a first quality score, a quality score calculated in an optimal frame determined before the current frame as a second quality score, comparing the first quality score with the second quality score, and replacing the optimal frame with the current frame if the first quality score is greater than or equal to the second quality score, or keeping the optimal frame unchanged if the first quality score is less than the second quality score; and outputting a face image of the optimal frame when a time length for which the face tracking model loses a face of an ID exceeds a preset time length threshold. The face optimal frame selection method can automatically identify and select a face optimal frame in a same monitoring picture, and further reduce storage capacity by storing the face optimal frame and improve face searching efficiency.

[0107] The above description is only the preferred embodiment of the present application, and for those skilled in the art, various changes and modifications can be made according to the technical scheme and technical concept of the present application, and all these changes and modifications shall belong to the protection scope of the present application.

Claims

1. A method for selecting the optimal face frame, characterized in that, Includes the following steps: Step S1: Obtain the training image set, label the key point positions and attributes of the faces in each training image in the training image set, and train the face detection model using the labeled training image set. Step S2: Receive video stream, extract the image to be detected from the video stream and input it into the trained face detection model to obtain the first information of each face in the image to be detected. The first information includes the detection box boundary of the face, the key point position of the face, the attributes of the face and the detection confidence of the face. Step S3: Obtain the second information of each face based on the first information and a preset information conversion algorithm. The second information includes: the Euler angles of the face, the occlusion degree value of the face, and the blur degree value of the face. The preset information conversion algorithm includes: selecting different face occlusion degree calculation formulas according to different face attributes in the first information. The face attributes in the first information include at least: the mask occlusion state of the face and the skin color area; calculating the face occlusion degree value according to the determined face occlusion degree calculation formula, the detection box boundary of the face, and the key point position of the face; calculating the face occlusion degree value in the second information according to the key point position of the face in the first information; extracting the face image within the detection box boundary; performing blur detection on the face image within the detection box boundary using the cv2.Laplacian algorithm to obtain the blur degree value of the face. Step S4: Track the detected faces using a face tracking model, determine the ID of each face, and determine the quality score of each ID face in each frame based on the second information of each ID face in each frame, the face detection confidence, and the preset quality score algorithm. Step S5: For faces with the same ID, define the quality score calculated in the current frame as the first quality score, and the quality score calculated in the optimal frame determined before the current frame as the second quality score. Compare the first quality score and the second quality score. If the first quality score is greater than or equal to the second quality score, replace the optimal frame with the current frame; otherwise, keep the optimal frame unchanged. Step S6: When the duration for which the face tracking model loses a face ID exceeds a preset duration threshold, output the face image of the optimal frame.

2. The method for selecting the optimal face frame as described in claim 1, characterized in that, The specific formulas for calculating different degrees of facial occlusion based on the different attributes of the face in the first piece of information include: When the mask covers both the nose and mouth of the face, the formula for calculating the degree of facial occlusion is the first formula. When the area of ​​a person's face covered by a mask is such that the mask covers the mouth but exposes the nose, the formula for calculating the degree of face occlusion is the second formula. When the area of ​​a person's face covered by a mask is such that the mouth and nose are exposed, the formula for calculating the degree of face occlusion is the third formula. When the area of ​​a person's face covered by a mask is not covered by a mask, the formula for calculating the degree of face occlusion is the fourth formula.

3. The method for selecting the optimal face frame as described in claim 2, characterized in that, The first formula is: ; The second formula is: ; The third formula is: ; The fourth formula is: ; Where `occupy` represents the degree of occlusion of the face, `area_skin` represents the area of ​​the skin-colored region, `area_face` represents the face area, `x1` and `y1` are the x and y coordinates of the top-left corner of the face detection box, `x2` and `y2` are the x and y coordinates of the bottom-right corner of the face detection box, and `y`... le y re y n y lm and y rm These are the vertical coordinates of the left eye, right eye, nose tip, left corner of mouth, and right corner of mouth on the face.

4. The method for selecting the optimal face frame as described in claim 3, characterized in that, The preset quality score algorithm in step S4 is as follows: ； Wherein, score is the quality score of a face with ID in the frame, conf is the detection confidence of the face with ID in the frame, size is the detection box size of the face with ID in the frame, pitch, yaw, and roll are the Euler angles of the face with ID in the frame, occupy is the occlusion value of the face with ID in the frame, blur is the blur value of the face with ID in the frame, and a1, a2, a3, a4, a5, a6, and a7 are scaling constants, the values ​​of which are determined by multivariate gradient descent.

5. The method for selecting the optimal face frame as described in claim 1, characterized in that, The face training model is a retinaface network model that incorporates an attribute recognition head and an attribute recognition function, so that the trained face training model can simultaneously detect the bounding box of the face, the key point position of the face, the attributes of the face, and the detection confidence of the face. The face tracking model is a deep-sort network model that improves the feature extraction network from the ResNet18 network to the MobileFaceNet network.

6. A face optimal frame selection device, characterized in that, include: The training unit is used to acquire a training image set, annotate the key point locations and attributes of faces in each training image in the training image set, and train the face detection model using the annotated training image set. The detection unit is used to receive a video stream, extract the image to be detected from the video stream and input it into a trained face detection model to obtain the first information of each face in the image to be detected. The first information includes the detection box boundary of the face, the key point position of the face, the attributes of the face and the detection confidence of the face. The calculation unit is used to obtain second information of each face based on first information and a preset information conversion algorithm. The second information includes: Euler angles of the face, occlusion degree value of the face, and blur degree value of the face. The preset information conversion algorithm includes: selecting different face occlusion degree calculation formulas according to different face attributes in the first information, wherein the face attributes in the first information include at least: face mask occlusion state and skin color area; calculating the face occlusion degree value according to the determined face occlusion degree calculation formula, the detection box boundary of the face, and the key point position of the face; calculating the face Euler angles in the second information according to the key point position of the face in the first information; extracting the face image within the detection box boundary; and performing blur detection on the face image within the detection box boundary using the cv2.Laplacian algorithm to obtain the blur degree value of the face. The tracking unit is used to track detected faces using a face tracking model, determine the ID of each face, and determine the quality score of each face ID in each frame based on the second information of each face ID in each frame, the face detection confidence, and a preset quality score algorithm. The comparison unit is used to define the quality score calculated in the current frame as the first quality score for faces with the same ID, and the quality score calculated in the optimal frame determined before the current frame as the second quality score. The first quality score is compared with the second quality score. If the first quality score is greater than or equal to the second quality score, the optimal frame is replaced with the current frame; otherwise, the optimal frame is kept unchanged. The output unit is used to output the face image of the optimal frame when the face tracking model loses a face ID for a duration exceeding a preset duration threshold.

7. An electronic device, characterized in that, include: A memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1-5.

Images