Face data annotation methods, devices, electronic equipment and storage media

Abstract

This invention provides a face data annotation method, comprising: determining a first face image of the target face in an initial annotation image; determining the face pose information of the target face in the initial annotation image; performing a first correction on the first face image using a trained spatial transformation network to obtain a second face image; determining a reference bounding box of the target object in the initial annotation image based on the second face image; determining the aspect ratio of the corresponding target bounding box according to the face pose information; performing a second correction on the reference bounding box according to the aspect ratio of the target bounding box to obtain the final bounding box of the target face; replacing the initial face bounding box in the initial annotation image with the final bounding box to obtain the final annotated image of the target face. By performing spatial transformation correction and aspect ratio correction on the initial face bounding box in the initial annotation image, the annotation consistency of the face image can be improved, thereby improving the accuracy of the face detection results of the face detection model.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, apparatus, electronic device, and storage medium for annotating facial data. Background Technology

[0002] With the development of computer vision technology, face detection has become one of the important applications in the field of computer vision, and it has been widely used in security, face recognition, and intelligent monitoring. However, existing face detectors may exhibit inconsistent aspect ratios in their bounding boxes during the detection process, which affects subsequent face feature extraction and recognition. This is because the training data used to train the face detector has inconsistent annotations. Specifically, due to differences in the annotators' understanding of the annotation rules and the subjectivity of annotation, there are many annotation errors in face data annotation. For example, under the same face pose, the aspect ratios of the annotated face boxes may vary significantly. These annotation errors are carried over into the training of the face detector, resulting in the output of bounding boxes with inconsistent aspect ratios when used for face detection after training, thus affecting the face feature extraction and recognition results. Therefore, the existence of inconsistent aspect ratios in the existing training data affects the performance of the trained face detector and reduces the accuracy of the face detection results. Summary of the Invention

[0003] This invention provides a face data annotation method to address the problem that existing training data often has uniform aspect ratios in the bounding boxes, which negatively impacts the performance of trained face detectors and reduces the accuracy of face detection results. By performing spatial transformation correction and aspect ratio correction on the initial face bounding boxes in the initial annotation image, a final bounding box is obtained and used to replace the initial bounding boxes in the initial annotation image, resulting in a final annotation image. This improves the consistency of face image annotations. Using this final annotation image to train the face detector enhances its performance, effectively solves the problem of inconsistent aspect ratios in the output detection boxes, and ultimately improves the accuracy of face detection results.

[0004] In a first aspect, embodiments of the present invention provide a face data annotation method, the method comprising:

[0005] A first face image of the target face is determined in the initial annotation image, and the face pose information of the target face is determined in the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face through a trained face pose estimation model.

[0006] The first face image is corrected by a trained spatial transformation network to obtain a second face image, and the reference bounding box of the target object is determined in the initial annotation image based on the second face image.

[0007] The aspect ratio of the corresponding target bounding box is determined based on the face pose information, and the reference bounding box is corrected a second time based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face. Different face pose information corresponds to different aspect ratios of the bounding box.

[0008] The final annotation box replaces the initial face annotation box in the initial annotation image to obtain the final annotation image of the target face.

[0009] Optionally, the step of performing a first correction on the first face image using a trained spatial transformation network to obtain a second face image, and determining the reference bounding box of the target object in the initial annotation image based on the second face image, includes:

[0010] The first face image is spatially transformed by a trained spatial transformation network to obtain a second face image and a mapping matrix between the first face image and the second face image.

[0011] A reference face frame is determined in the second face image using preset face frame parameters;

[0012] The reference face bounding box is inversely mapped onto the initial labeled image using the mapping matrix to obtain the reference labeled box of the target object.

[0013] Optionally, before determining the aspect ratio of the corresponding target bounding box based on the face pose information, the face data annotation method further includes:

[0014] The trained face pose estimation model is used to perform face pose estimation on the target faces in all initially labeled images to obtain the face pose information of all target faces.

[0015] Determine the aspect ratio of all initial face bounding boxes, and calculate the average aspect ratio of all initial face bounding boxes under different face pose information;

[0016] The correspondence between each face pose information and the aspect ratio of the bounding box is determined based on the average aspect ratio of each face.

[0017] Optionally, the step of performing a second correction on the reference bounding box based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face includes:

[0018] Based on the facial pose information, determine the facial pose angle of the target face;

[0019] The reference frame is corrected a second time based on the aspect ratio of the target frame and the face pose angle to obtain the final frame of the target face.

[0020] Optionally, the step of performing a second correction on the reference bounding box based on the aspect ratio of the target bounding box and the face pose angle to obtain the final bounding box of the target face includes:

[0021] Based on the facial pose angle, determine the expected offset of the first left and right borders and the expected offset of the first top and bottom borders;

[0022] Based on the aspect ratio of the target annotation box, the expected offsets of the left and right borders and the expected offsets of the top and bottom borders are adjusted to obtain the second expected offsets of the left and right borders and the second expected offsets of the top and bottom borders.

[0023] The reference bounding box is corrected a second time based on the expected offsets of the second left and right borders and the expected offsets of the second top and bottom borders to obtain the final bounding box of the target face.

[0024] Secondly, embodiments of the present invention provide a method for training a face detection model, the method comprising the following steps:

[0025] Obtain a training dataset and a face detection model to be trained. The training dataset includes face-annotated images, which are annotated by the face data annotation method as described in any one of the embodiments of the present invention.

[0026] The face detection model to be trained is trained using the training dataset, and a trained face detection model is obtained after training.

[0027] Thirdly, embodiments of the present invention provide a face data annotation device, the face data annotation device comprising:

[0028] The first processing module is used to determine a first face image of the target face in the initial annotation image, and to determine the face pose information of the target face in the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face through a trained face pose estimation model.

[0029] The first correction module is used to perform a first correction on the first face image through a trained spatial transformation network to obtain a second face image, and to determine the reference bounding box of the target object in the initial annotation image based on the second face image.

[0030] The second correction module is used to determine the aspect ratio of the corresponding target annotation box based on the face pose information, and to perform a second correction on the reference annotation box based on the aspect ratio of the target annotation box to obtain the final annotation box of the target face. Different face pose information corresponds to different aspect ratios of the annotation box.

[0031] The second processing module is used to replace the initial face annotation box in the initial annotation image with the final annotation box to obtain the final annotation image of the target face.

[0032] Fourthly, embodiments of the present invention provide a training apparatus for a face detection model, the training apparatus for the face detection model comprising:

[0033] The acquisition module is used to acquire a training dataset and a face detection model to be trained. The training dataset includes face-annotated images, which are obtained by face data annotation method as described in any one of the embodiments of the present invention.

[0034] The training module is used to train the face detection model to be trained using the training dataset, and after training, a trained face detection model is obtained.

[0035] Fifthly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the face data annotation method provided in the embodiments of the present invention, or when the processor executes the computer program, it implements the steps in the face detection model training method provided in the embodiments of the present invention.

[0036] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps in the face data annotation method provided in the embodiments of the invention, or when the computer program is executed by a processor, it implements the steps in the face detection model training method provided in the embodiments of the invention.

[0037] In this embodiment of the invention, a first face image of the target face is determined from an initial annotation image, and the face pose information of the target face is determined from the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face using a trained face pose estimation model. The first face image is first corrected using a trained spatial transformation network to obtain a second face image, and a reference bounding box of the target object is determined in the initial annotation image based on the second face image. The aspect ratio of the corresponding target bounding box is determined according to the face pose information, and the reference bounding box is second corrected according to the aspect ratio of the target bounding box to obtain the final bounding box of the target face. Different face pose information corresponds to different aspect ratios of the bounding box. The final bounding box replaces the initial face bounding box in the initial annotation image to obtain the final annotation image of the target face. By performing spatial transformation correction and aspect ratio correction on the initial face bounding boxes in the initial labeled image, the final labeled boxes are obtained and used to replace the initial labeled boxes in the initial labeled image, resulting in the final labeled image. This improves the consistency of face image annotations. Using the final labeled image to train the face detector can improve the performance of the face detector, effectively solve the problem of inconsistent aspect ratios of the output detection boxes of the face detector, and thus improve the accuracy of face detection results of the face detection model. Attached Figure Description

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flowchart of a face data annotation method provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of a face annotation bounding box width and height consistency correction method provided in an embodiment of the present invention;

[0041] Figure 3 This is a flowchart of a training method for a face detection model provided in an embodiment of the present invention;

[0042] Figure 4 This is a flowchart of a method for improving the aspect ratio consistency of a face detector bounding box, provided by an embodiment of the present invention.

[0043] Figure 5This is a schematic diagram of the structure of a face data annotation device provided in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the structure of a training device for a face detection model provided in an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, Figure 1 This is a flowchart of a face data annotation method provided by an embodiment of the present invention. The face data annotation method includes the following steps:

[0048] 101. The first face image of the target face is determined in the initial labeled image, and the face pose information of the target face is determined in the initial labeled image.

[0049] In this embodiment of the invention, the above-described face data annotation method can be used in a face data annotation platform, which can be a platform built on a server or a distributed server. The face data annotation platform receives the original annotated image, corrects the original annotated image, and obtains the final annotated face image, which can be used to train a face detection model.

[0050] The initial labeled image includes the initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation on the target face through a trained face pose estimation model.

[0051] Specifically, the initial labeled image can be obtained manually by marking initial face bounding boxes in the face image. Alternatively, the initial labeled image can be directly downloaded from an open-source face annotation dataset. This embodiment of the invention does not limit the method of obtaining the initial labeled image.

[0052] The target faces mentioned above are those appearing in the initial annotation image. One or more target faces may exist in an initial annotation image. After obtaining the initial annotation image, image cropping can be performed within the initial annotation image based on the initial face bounding boxes to obtain the first face image.

[0053] After obtaining the first face image of the target face, face pose estimation can be performed on the first face image to obtain the face pose information of the target face. This face pose information can include the side angle, pitch angle, and rotation angle. The face pose estimation can be achieved using a face pose estimation model, such as Active Appearance Models, Active Shape Models, or SolvePnP.

[0054] 102. The first face image is corrected by the trained spatial transformation network to obtain the second face image, and the reference bounding box of the target object is determined in the initial annotation image based on the second face image.

[0055] In this embodiment of the invention, the aforementioned spatial transformation network can be an STN (spatial transformer network). This network can learn spatial transformation parameters of images or features based on the image alignment task, even without labeled keypoints, and spatially align the input image or learned features. This reduces the impact of geometric transformations such as rotation, translation, scaling, and distortion on tasks like classification and localization. The spatial transformation network can be pre-trained and downloaded from an open-source platform, or it can be trained using a first sample face dataset. The first sample face dataset can include sample images and corrected images.

[0056] The first face image is input into the aforementioned spatial transformation network, which aligns the first face image spatially to obtain the second face image, thus completing the first correction. Simultaneously, the mapping matrix of the second face image aligned from the first face image is recorded.

[0057] After performing a first correction on the first face image, a second face image is obtained. This second face image is the corrected image. A reference bounding box can be determined based on the second face image, and this box is mapped onto the initial annotation image to obtain the reference annotation box of the target object. Since the second face image is a spatially aligned face image, and the reference bounding box is also based on this spatially aligned box, the reference annotation box of the target object is also based on this spatially aligned box. The reference bounding box can be obtained by scaling the bounding box of the second face image, or a fixed box in the central region of the second face image can be used as the reference bounding box.

[0058] 103. Determine the aspect ratio of the corresponding target bounding box based on the face pose information, and perform a second correction on the reference bounding box based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face.

[0059] In this embodiment of the invention, the aspect ratio of the target bounding box corresponding to the face pose information can be determined by looking up a table. Different face pose information corresponds to different aspect ratios of the bounding box. Specifically, a bounding box aspect ratio mapping table can be constructed, which maintains the mapping relationship between face pose information and bounding box aspect ratios. After obtaining the face pose information of the target face, the aspect ratio of the bounding box corresponding to the face pose information is looked up in the bounding box aspect ratio mapping table, and the aspect ratio of the bounding box corresponding to the face pose information is determined as the target bounding box aspect ratio.

[0060] The mapping relationship between the aforementioned face pose information and the aspect ratio of the bounding boxes can be set empirically, or it can be determined based on the face pose information of the target faces in each initially annotated image in the initial annotation dataset and the initial aspect ratio of the target faces' initial bounding boxes. Specifically, the face pose information of all target faces is obtained by inferring the initial bounding boxes of all initially annotated images in the initial annotation dataset. The average aspect ratio of the initial bounding boxes of all target faces under different face poses is then calculated. Specifically, the average aspect ratio of all target faces under different poses in three dimensions—tilt, left and right sides, and in-plane rotation—can be calculated. Based on the average aspect ratio of the initial bounding boxes of target faces under different face poses, the mapping relationship between face pose information and the aspect ratio of the bounding boxes is determined.

[0061] After obtaining the aspect ratio of the target bounding box, this aspect ratio can be used as the correction expectation to adjust the reference bounding box, aligning its aspect ratio with that of the target bounding box. This completes the second correction, yielding the final bounding box of the target face. The final bounding box of the target face maintains a consistent aspect ratio across different face poses.

[0062] 104. Replace the initial face annotation box in the initial annotation image with the final annotation box to obtain the final annotation image of the target face.

[0063] After obtaining the final bounding boxes, the initial face bounding boxes are replaced in the initial labeled image to obtain the final labeled image of the target face. The final bounding boxes in the final labeled image undergo face space alignment correction and face pose aspect ratio correction, ensuring consistent aspect ratios across different face poses. Training the face detection model using the final labeled images allows the model to output face detection boxes with consistent aspect ratios, thereby improving the accuracy of face detection results.

[0064] In this embodiment of the invention, a first face image of the target face is determined from an initial annotation image, and the face pose information of the target face is also determined from the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face using a trained face pose estimation model. The first face image is then corrected using a trained spatial transformation network to obtain a second face image, and a reference bounding box of the target object is determined from the initial annotation image based on the second face image. The aspect ratio of the corresponding target bounding box is determined based on the face pose information, and the reference bounding box is corrected a second time based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face. Different face pose information corresponds to different aspect ratios of the bounding box. The final bounding box replaces the initial face bounding box in the initial annotation image to obtain the final annotation image of the target face. By performing spatial transformation correction and aspect ratio correction on the initial face bounding boxes in the initial labeled image, the final labeled boxes are obtained and used to replace the initial labeled boxes in the initial labeled image, resulting in the final labeled image. This improves the consistency of face image annotations. Using the final labeled image to train the face detector can improve the performance of the face detector, effectively solve the problem of inconsistent aspect ratios of the output detection boxes of the face detector, and thus improve the accuracy of face detection results of the face detection model.

[0065] Optionally, in the step of performing a first correction on the first face image using a trained spatial transformation network to obtain a second face image, and determining the reference bounding box of the target object in the initial annotation image based on the second face image, the trained spatial transformation network can be used to perform a spatial transformation on the first face image to obtain the second face image and a mapping matrix between the first and second face images; a reference face bounding box is determined in the second face image using preset face bounding box parameters; and the reference face bounding box is inversely mapped to the initial annotation image using the mapping matrix to obtain the reference bounding box of the target object.

[0066] In this embodiment of the invention, the first face image is input into the aforementioned spatial transformation network, and the first face image is spatially aligned through the spatial transformation network to obtain a second face image, thus completing the first correction. Simultaneously, the mapping matrix f(θ) of the second face image aligned from the first face image is recorded.

[0067] The preset face bounding box parameters mentioned above may include the center point and width / height of the face bounding box. The center point of the face bounding box is the center point of the second face image, and the width / height of the face bounding box are fixed values. After obtaining the second face image, the center point of the second face image is used as the center point of the reference face bounding box, and the width / height of the reference face bounding box is used as the width / height of the reference face bounding box. The reference face bounding box in the second face image is inversely mapped to the initial annotation image through the mapping matrix f(θ) to obtain the reference annotation box of the target object.

[0068] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a face annotation bounding box width and height consistency correction method provided in an embodiment of the present invention, as shown below. Figure 2 As shown, Figure 2 The large image on the left is the initial annotation image, which includes the initial face bounding box ABCD. The first face image is obtained by cropping the image within the initial face bounding box ABCD. Alternatively, the initial face bounding box can be expanded according to a certain expansion rule to obtain an expanded bounding box EFGH, and the first face image is obtained by cropping the image within the expanded bounding box EFGH. This expansion rule can be based on the center point of the initial face bounding box ABCD, expanding outwards in four directions: up, down, left, and right. After obtaining the first face image, it is spatially transformed and corrected using STN to obtain the corresponding mapping matrix f(θ) and the second face image efgh. In the second face image efgh, a reference face bounding box ijkl with a fixed central region is taken, and the corrected mapping matrix f(θ) is used to inversely map the reference face bounding box ijkl back to the initial annotation image, obtaining a new bounding box IJKL. The bounding box IJKL is the reference bounding box of the target object.

[0069] Optionally, before determining the aspect ratio of the corresponding target bounding box based on the face pose information, a trained face pose estimation model can be used to perform face pose estimation on the target faces in all initial labeled images to obtain the face pose information of all target faces; determine the aspect ratio of all initial face bounding boxes, and calculate the average aspect ratio of all initial face bounding boxes under different face pose information; determine the correspondence between each face pose information and the aspect ratio of the bounding box based on each average aspect ratio.

[0070] In this embodiment of the invention, a trained face pose estimation model is used to estimate the pose of target faces in all initially labeled images in the initial labeled dataset, obtaining face pose estimation information for all target faces. The average aspect ratio of the initial face bounding boxes for all target faces under different face poses is then calculated. Specifically, this can be done by calculating the average aspect ratio of all target faces under three dimensions: pitch, left / right profile, and in-plane rotation. Based on the average aspect ratio of the initial face bounding boxes for target faces under different face poses, the correspondence between face pose information and bounding box aspect ratios is determined. This correspondence is then used as a mapping relationship between face pose information and bounding box aspect ratios and added to a bounding box aspect ratio mapping table for maintenance.

[0071] It should be noted that since the pose estimation of the target face in all the initially annotated images in the initial annotation dataset is performed in advance, in step 101, the result of the pose estimation of the target face in all the initially annotated images in the initial annotation dataset can be reused to directly obtain the face pose information of the target face, without needing to perform face pose estimation on the target face again.

[0072] Optionally, in the step of performing a second correction on the reference annotation box based on the aspect ratio of the target annotation box to obtain the final annotation box of the target face, the face pose angle of the target face can be determined based on the face pose information; and the reference annotation box can be further corrected based on the aspect ratio of the target annotation box and the face pose angle to obtain the final annotation box of the target face.

[0073] In this embodiment of the invention, the aforementioned facial pose information includes angle information of the target face in three dimensions: pitch, left and right side profile, and in-plane rotation. The facial pose angles include the side profile angle, pitch angle, and in-plane rotation angle of the target face. Specifically, the side profile angle refers to the angle at which the face turns to the left or right, the pitch angle refers to the angle at which the face faces downwards or upwards, and the in-plane rotation angle refers to the angle at which the face rotates around the neck as a center point.

[0074] After determining the aspect ratio of the target bounding box, an adjustment strategy is determined based on the facial pose angle of the target face. This strategy is then used to adjust the reference bounding box to match the aspect ratio of the target bounding box, performing a second correction on the reference bounding box to obtain the final bounding box of the target face. Combined with... Figure 2 After obtaining the reference bounding box IJKL, the reference bounding box IJKL can be adjusted to match the aspect ratio of the target bounding box according to the adjustment strategy, so as to obtain the final bounding box MNPQ of the target face.

[0075] The above adjustment strategy can be to adjust at least one of the top border, bottom border, left border, and right border of the reference annotation box according to the face pose angle, so as to obtain the final annotation box that conforms to the face pose angle and the aspect ratio of the target annotation box.

[0076] Optionally, in the step of performing a second correction on the reference bounding box based on the aspect ratio of the target bounding box and the face pose angle to obtain the final bounding box of the target face, the expected offsets of the first left and right borders and the first expected offsets of the first top and bottom borders can be determined according to the face pose angle; the expected offsets of the left and right borders and the expected offsets of the top and bottom borders can be adjusted according to the aspect ratio of the target bounding box to obtain the expected offsets of the second left and right borders and the second expected offsets of the second top and bottom borders; and the reference bounding box can be further corrected based on the expected offsets of the second left and right borders and the second expected offsets of the second top and bottom borders to obtain the final bounding box of the target face.

[0077] In this embodiment of the invention, the aforementioned facial pose angles include the side profile angle, pitch angle, and in-plane rotation angle of the target face. The pitch angle and side profile angle range from -90° to 90°, and the in-plane rotation angle ranges from -180° to 180°. The width and height of the reference frame are w and h, respectively, and the pitch angle, side profile angle, and in-plane rotation angle of the target face are α, β, and θ, respectively.

[0078] In the second correction process, if only the profile angle is non-0° among the target face's facial pose angles, then only the adjustment strategy for the profile angle is considered. If only the pitch angle is non-0° among the target face's facial pose angles, then only the adjustment strategy for the pitch angle is considered. If only the in-plane rotation angle is non-0° among the target face's facial pose angles, then only the adjustment strategy for the in-plane rotation angle is considered. If at least two angles of the target face's facial pose angle are non-0°, then a multi-dimensional rotation angle adjustment strategy is considered.

[0079] Considering only the side profile angle: when the side profile angle is greater than 0° (to the left side of the face), fix the position of the right border, and the position of the left border is expected to be adjusted to the right by w*β / 90°×0.5 pixels; when the side profile angle is less than 0° (to the right side of the face), fix the position of the left border, and the position of the right border is expected to be adjusted to the left by w*β / 90°×0.5 pixels.

[0080] Considering only the pitch angle: when the pitch angle is greater than 0° (tilt up), the bottom border position is fixed, and the top border position is expected to be adjusted downwards by h*α / 90°×0.25 pixels; when the pitch angle is less than 0° (tilt down), the top border position is fixed, and the bottom border position is also expected to be adjusted upwards by h*α / 90×0.25 pixels.

[0081] Considering only in-plane rotation angles: the frame is only corrected based on the in-plane rotation angle when the rotation angle exceeds a set threshold (e.g., 20°). The scaling ratio of the rotated frame's width and height is calculated as follows:

[0082]

[0083] After rotation, the frame width w′ = w*r and the frame height h′ = h*r; the top and bottom borders need to be offset inward by (hh′) / 2 pixels respectively, and the left and right borders need to be offset inward by (ww′) / 2 pixels respectively.

[0084] Considering the angular changes in multiple dimensions: the expected offsets for the left and right borders are (ww′) / 2 and w*β / 90×0.5+(ww′) / 2, and the expected offsets for the top and bottom borders are (hh′) / 2 and h*α / 90×0.25+(hh′) / 2. The expected width of the face frame after the offset is w"=[ww*β / 90×0.5-(ww′)] and the height is h"=[hh′*α / 90×0.25-(hh′)].

[0085] Based on the principles of aspect ratio consistency and area consistency, the expected offset distances of the left and right borders and top and bottom borders are corrected:

[0086]

[0087] (w"-Δx)(h"-Δy)=w"×h

[0088] in The aspect ratio of the target bounding box is given. Δx and Δy can be calculated using the formula above; note that the calculated Δx and Δy can be negative.

[0089] From the above formula, we can conclude that for the left (or right) border that is offset only due to the rotation angle, the offset distance inward is (ww′) / 2, while the offset distance inward of the right (or left) border that is offset by the side angle is w*β / 90×0.5+(ww′) / 2-Δx.

[0090] For the top (or bottom) border that is offset only due to the rotation angle, the offset distance is (hh′) / 2, while the offset distance of the bottom (or top) border that needs to be superimposed with the pitch angle is h*α / 90×0.25+(hh′) / 2-Δy.

[0091] like Figure 3 The flowchart shown is a method for training a face detection model according to an embodiment of the present invention. The method for training a face detection model includes the following steps:

[0092] 301. Obtain the training dataset and the face detection model to be trained.

[0093] In this embodiment of the invention, the training dataset includes face-annotated images, which are obtained by annotating face data using any of the face data annotation methods described in this embodiment of the invention.

[0094] 302. Train the face detection model to be trained using the training dataset. After training, the trained face detection model is obtained.

[0095] In this embodiment of the invention, the training dataset includes final labeled images, which in turn include final labeled bounding boxes. The final labeled images in the training dataset can be input into the face detection model to be trained to obtain the output face bounding boxes of the final labeled images. The loss value between the output face bounding boxes of the final labeled images and the final labeled bounding boxes of the final labeled images is calculated. With minimizing the loss value as the optimization objective, the parameters of the face detection model to be trained are updated through backpropagation. The parameter update process is iterated until the loss value converges at the minimum loss value or the number of iterations reaches a preset number, at which point training stops, and a trained face detection model is obtained.

[0096] After obtaining a trained face detection model, it can be deployed to obtain a corresponding face detector. The image to be detected is input into the face detector, which outputs face detection results with consistent aspect ratios of the detection bounding boxes.

[0097] In this embodiment of the invention, a training dataset and a face detection model to be trained are obtained. The training dataset includes face annotation images, which are obtained by annotation using any of the face data annotation methods described in this embodiment. The face detection model to be trained is trained using the training dataset, and a trained face detection model is obtained after training. In the face data annotation method of this embodiment, the initial face annotation boxes in the initial annotation image are spatially transformed and the aspect ratio of the face pose is corrected to obtain the final annotation box, which replaces the initial annotation box in the initial annotation image to obtain the final annotation image. This improves the consistency of face image annotation. Using the final annotation image to train the face detector can improve the performance of the face detector, effectively solve the problem of inconsistent aspect ratios of the output detection boxes of the face detector, and thus improve the accuracy of the face detection results.

[0098] like Figure 4 As shown, Figure 4 This is a flowchart of a method for improving the aspect ratio consistency of a face detector bounding box according to an embodiment of the present invention, specifically including the following steps:

[0099] An STN face correction model is constructed. Sample face data is collected for the STN face correction model to obtain the first sample face data. The STN face correction model is trained using the first sample face data and then used for the first correction.

[0100] A face state estimation model is constructed. Sample face data and pose annotation are performed on the face state estimation model to obtain second sample face data. The face state estimation model is trained using the second sample face data and then used to calculate the face pose.

[0101] A face detection model is constructed by collecting sample face data and annotating face bounding boxes to obtain an initial labeled dataset. The initial face bounding boxes in the initially labeled images in the initial labeled dataset are corrected using an STN face correction model and a face pose estimation model to obtain a training dataset. The face detection model is trained using this training dataset. After training, the face detection model is deployed and used to detect faces in input images.

[0102] like Figure 5 As shown, an embodiment of the present invention provides a face data annotation device, which includes:

[0103] The first processing module 501 is used to determine a first face image of the target face in the initial annotation image, and to determine the face pose information of the target face in the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face through a trained face pose estimation model.

[0104] The first correction module 502 is used to perform a first correction on the first face image through a trained spatial transformation network to obtain a second face image, and to determine the reference bounding box of the target object in the initial annotation image based on the second face image.

[0105] The second correction module 503 is used to determine the aspect ratio of the corresponding target annotation box based on the face pose information, and to perform a second correction on the reference annotation box based on the aspect ratio of the target annotation box to obtain the final annotation box of the target face. Different face pose information corresponds to different aspect ratios of the annotation box.

[0106] The second processing module 504 is used to replace the initial face annotation box in the initial annotation image with the final annotation box to obtain the final annotation image of the target face.

[0107] Optionally, the first correction module 502 is further configured to perform spatial transformation on the first face image through a trained spatial transformation network to obtain a second face image and a mapping matrix between the first face image and the second face image; determine a reference face box in the second face image through preset face box parameters; and inversely map the reference face box to the initial annotation image through the mapping matrix to obtain the reference annotation box of the target object.

[0108] Optionally, the face data annotation device further includes:

[0109] The preprocessing module is used to perform face pose estimation on the target faces in all initially labeled images using a trained face pose estimation model, so as to obtain the face pose information of all target faces.

[0110] The aspect ratio calculation module is used to determine the aspect ratio of all initial face bounding boxes and calculate the average aspect ratio of all initial face bounding boxes under different face pose information.

[0111] The relationship determination module is used to determine the correspondence between each face pose information and the aspect ratio of the annotation box based on the average aspect ratio of each face.

[0112] Optionally, the second correction module 503 is further configured to determine the face pose angle of the target face based on the face pose information; and to perform a second correction on the reference frame based on the aspect ratio of the target frame and the face pose angle to obtain the final frame of the target face.

[0113] Optionally, the second correction module 503 is further configured to determine the expected offset of the first left and right borders and the expected offset of the first top and bottom borders based on the face pose angle; adjust the expected offset of the left and right borders and the expected offset of the top and bottom borders according to the aspect ratio of the target annotation box to obtain the expected offset of the second left and right borders and the expected offset of the second top and bottom borders; and perform a second correction on the reference annotation box based on the expected offset of the second left and right borders and the expected offset of the second top and bottom borders to obtain the final annotation box of the target face.

[0114] It should be noted that the face data annotation device provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that can perform face data annotation.

[0115] The face data annotation apparatus provided in this embodiment of the invention can implement all the processes implemented by the face data annotation method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0116] like Figure 6As shown, this embodiment of the invention provides a training device for a face detection model, which includes:

[0117] The acquisition module 601 is used to acquire a training dataset and a face detection model to be trained. The training dataset includes face-annotated images, which are obtained by face data annotation method as described in any one of the embodiments of the present invention.

[0118] The training module 602 is used to train the face detection model to be trained using the training dataset, and obtain the trained face detection model after training is completed.

[0119] It should be noted that the face detection model training device provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that can train face detection models.

[0120] The face detection model training device provided in this embodiment of the invention can implement all the processes of the face detection model training method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0121] See Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, it includes: a memory 702, a processor 701, and a computer program for a face data annotation method or a computer program for a face detection model training method stored on the memory 702 and capable of running on the processor 701, wherein:

[0122] The processor 701 calls the computer program for the face data annotation method stored in the memory 702 and executes the following steps:

[0123] A first face image of the target face is determined in the initial annotation image, and the face pose information of the target face is determined in the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face through a trained face pose estimation model.

[0124] The first face image is corrected by a trained spatial transformation network to obtain a second face image, and the reference bounding box of the target object is determined in the initial annotation image based on the second face image.

[0125] The aspect ratio of the corresponding target bounding box is determined based on the face pose information, and the reference bounding box is corrected a second time based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face. Different face pose information corresponds to different aspect ratios of the bounding box.

[0126] The final annotation box replaces the initial face annotation box in the initial annotation image to obtain the final annotation image of the target face.

[0127] Optionally, the processor 701 performs a first correction on the first face image using a trained spatial transformation network to obtain a second face image, and determines the reference bounding box of the target object in the initial annotation image based on the second face image, including:

[0128] The first face image is spatially transformed by a trained spatial transformation network to obtain a second face image and a mapping matrix between the first face image and the second face image.

[0129] A reference face frame is determined in the second face image using preset face frame parameters;

[0130] The reference face bounding box is inversely mapped onto the initial labeled image using the mapping matrix to obtain the reference labeled box of the target object.

[0131] Optionally, before determining the aspect ratio of the corresponding target bounding box based on the face pose information, the face data annotation method executed by the processor 701 further includes:

[0132] The trained face pose estimation model is used to perform face pose estimation on the target faces in all initially labeled images to obtain the face pose information of all target faces.

[0133] Determine the aspect ratio of all initial face bounding boxes, and calculate the average aspect ratio of all initial face bounding boxes under different face pose information;

[0134] The correspondence between each face pose information and the aspect ratio of the bounding box is determined based on the average aspect ratio of each face.

[0135] Optionally, the step of processor 701 performing a second correction on the reference bounding box based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face includes:

[0136] Based on the facial pose information, determine the facial pose angle of the target face;

[0137] The reference frame is corrected a second time based on the aspect ratio of the target frame and the face pose angle to obtain the final frame of the target face.

[0138] Optionally, the processor 701 performs a second correction on the reference bounding box based on the aspect ratio of the target bounding box and the face pose angle to obtain the final bounding box of the target face, including:

[0139] Based on the facial pose angle, determine the expected offset of the first left and right borders and the expected offset of the first top and bottom borders;

[0140] Based on the aspect ratio of the target annotation box, the expected offsets of the left and right borders and the expected offsets of the top and bottom borders are adjusted to obtain the second expected offsets of the left and right borders and the second expected offsets of the top and bottom borders.

[0141] The reference bounding box is corrected a second time based on the expected offsets of the second left and right borders and the expected offsets of the second top and bottom borders to obtain the final bounding box of the target face.

[0142] It should be noted that the electronic device provided in the embodiments of the present invention can be applied to devices such as smartphones, computers, and servers that can perform face data annotation methods.

[0143] The processor 701 calls the computer program storing the training method of the face detection model in the memory 702 and executes the following steps:

[0144] Obtain a training dataset and a face detection model to be trained. The training dataset includes face-annotated images, which are annotated by the face data annotation method as described in any one of the embodiments of the present invention.

[0145] The face detection model to be trained is trained using the training dataset, and a trained face detection model is obtained after training.

[0146] It should be noted that the electronic device provided in the embodiments of the present invention can be applied to devices such as smartphones, computers, and servers that can perform face detection model training methods.

[0147] The electronic device provided in this embodiment of the invention can implement all the processes of the face data annotation method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0148] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the face data annotation method or the face detection model training method provided in this invention, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0149] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0150] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for annotating facial data, characterized in that, The method includes the following steps: A first face image of the target face is determined in the initial annotation image, and the face pose information of the target face is determined in the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face through a trained face pose estimation model. The first face image is corrected by a trained spatial transformation network to obtain a second face image, and the reference bounding box of the target object is determined in the initial annotation image based on the second face image. The aspect ratio of the corresponding target bounding box is determined based on the facial pose information, and the reference bounding box is then corrected based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face. Different facial pose information corresponds to different aspect ratios of the bounding box. Specifically, a bounding box aspect ratio mapping table is constructed, which maintains the mapping relationship between facial pose information and bounding box aspect ratios. After obtaining the facial pose information of the target face, the aspect ratio of the bounding box corresponding to the facial pose information is found in the bounding box aspect ratio mapping table, and the aspect ratio of the bounding box corresponding to the facial pose information is determined. The aspect ratio of the target bounding box is defined; the facial pose angle of the target face is determined based on the facial pose information; the expected offsets of the first left and right borders and the first expected offsets of the first top and bottom borders are determined based on the facial pose angles; the expected offsets of the left and right borders and the expected offsets of the top and bottom borders are adjusted according to the aspect ratio of the target bounding box to obtain the expected offsets of the second left and right borders and the second expected offsets of the top and bottom borders; the reference bounding box is corrected a second time based on the expected offsets of the second left and right borders and the second expected offsets of the top and bottom borders to obtain the final bounding box of the target face. The final annotation box replaces the initial face annotation box in the initial annotation image to obtain the final annotation image of the target face.

2. The face data annotation method as described in claim 1, characterized in that, The step of performing a first correction on the first face image using a trained spatial transformation network to obtain a second face image, and determining the reference bounding box of the target object in the initial annotation image based on the second face image, includes: The first face image is spatially transformed by a trained spatial transformation network to obtain a second face image and a mapping matrix between the first face image and the second face image. A reference face frame is determined in the second face image using preset face frame parameters; The reference face bounding box is inversely mapped onto the initial labeled image using the mapping matrix to obtain the reference labeled box of the target object.

3. The face data annotation method as described in claim 1, characterized in that, Before determining the aspect ratio of the corresponding target bounding box based on the face pose information, the face data annotation method further includes: The trained face pose estimation model is used to perform face pose estimation on the target faces in all initially labeled images to obtain the face pose information of all target faces. Determine the aspect ratio of all initial face bounding boxes, and calculate the average aspect ratio of all initial face bounding boxes under different face pose information; The correspondence between each face pose information and the aspect ratio of the bounding box is determined based on the average aspect ratio of each face.

4. A method for training a face detection model, characterized in that, The training method for the face detection model includes the following steps: Obtain a training dataset and a face detection model to be trained. The training dataset includes face-annotated images, which are annotated using the face data annotation method as described in any one of claims 1 to 3. The face detection model to be trained is trained using the training dataset, and a trained face detection model is obtained after training.

5. A facial data annotation device, characterized in that, The face data annotation device includes: The first processing module is used to determine a first face image of the target face in the initial annotation image, and to determine the face pose information of the target face in the initial annotation image. The initial annotation image includes an initial face bounding box of the target face. The first face image is determined based on the initial face bounding box. The face pose information is obtained by performing face pose estimation processing on the target face through a trained face pose estimation model. The first correction module is used to perform a first correction on the first face image through a trained spatial transformation network to obtain a second face image, and to determine the reference bounding box of the target object in the initial annotation image based on the second face image. The second correction module is used to determine the aspect ratio of the corresponding target bounding box based on the face pose information, and to perform a second correction on the reference bounding box based on the aspect ratio of the target bounding box to obtain the final bounding box of the target face. Different face pose information corresponds to different aspect ratios of the bounding box. Specifically, a bounding box aspect ratio mapping table is constructed, which maintains the mapping relationship between face pose information and bounding box aspect ratios. After obtaining the face pose information of the target face, the aspect ratio of the bounding box corresponding to the face pose information is found in the bounding box aspect ratio mapping table, and the bounding box corresponding to the face pose information is then mapped. The aspect ratio of the bounding box is determined as the aspect ratio of the target annotation box; the facial pose angle of the target face is determined based on the facial pose information; the expected offsets of the first left and right borders and the first expected offsets of the first top and bottom borders are determined based on the facial pose angle; the expected offsets of the left and right borders and the expected offsets of the top and bottom borders are adjusted according to the aspect ratio of the target annotation box to obtain the expected offsets of the second left and right borders and the second expected offsets of the top and bottom borders; the reference annotation box is corrected a second time based on the expected offsets of the second left and right borders and the second expected offsets of the top and bottom borders to obtain the final annotation box of the target face; The second processing module is used to replace the initial face annotation box in the initial annotation image with the final annotation box to obtain the final annotation image of the target face.

6. A training device for a face detection model, characterized in that, The training device for the face detection model includes: The acquisition module is used to acquire a training dataset and a face detection model to be trained. The training dataset includes face-annotated images, which are obtained by the face data annotation method as described in any one of claims 1 to 3. The training module is used to train the face detection model to be trained using the training dataset, and after training, a trained face detection model is obtained.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the face data annotation method as described in any one of claims 1 to 3, or the steps in the face detection model training method as described in claim 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the face data annotation method as described in any one of claims 1 to 3, or, when executed by a processor, implements the steps in the face detection model training method as described in claim 4.

Images