Training method, device, computer equipment and storage medium for face key point detection model

Abstract

The present invention discloses a training method for a facial key point detection model, including: obtaining a three-dimensional facial model, determining facial key point annotations corresponding to the three-dimensional facial model at an initial position; determining the moving path of the three-dimensional facial model in three-dimensional space; photographing the three-dimensional facial model to obtain a new two-dimensional facial image; calculating facial key point annotations corresponding to the new two-dimensional facial image based on the position of the three-dimensional facial model in the moving path and the facial key point annotations corresponding to the initial position when photographing; adding the new two-dimensional facial image and the facial key point annotations corresponding to the new two-dimensional facial image to a training data set, and using the training data set to train a facial key point detection model. The facial key point detection model trained based on the training data set can accurately and stably identify facial key points in a video stream. In addition, a training device, computer equipment and storage medium for a facial key point detection model are also proposed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method, device, computer equipment and storage medium for a face key point detection model. Background Art

[0002] The function of facial key points is to accurately locate and segment the positions of various parts of the face, such as the eyes, eyebrows, the precise outer contours of the mouth, the outer contours of the face shape, etc. The main applications include face deformation (thin face, big eyes, etc.), virtual decoration, virtual makeup trial and animated films.

[0003] The accuracy of facial key point detection model detection depends on the training data set. The traditional training data set of facial key points is to randomly collect different face photos and manually mark the key points on the photos. Since the definition of facial key points is not precise and unique, the manually marked facial key points fluctuate within a certain range.

[0004] For video streams, the detection of facial key points requires not only high accuracy but also high stability. The ideal state is that the same key point detected in adjacent frames should be in the same position in the physical space, otherwise, a visual effect of key point jitter will appear.

[0005] However, at present, the training data of almost all models are manually annotated single-person images, without considering the positional relationship between consecutive video frames, which causes the position of the same key point in different video frames to deviate. The effect displayed in the continuous video stream is the shaking of the facial key points. Summary of the invention

[0006] Based on this, it is necessary to propose a training method, device, computer equipment and storage medium for a facial key point detection model to address the above problems. The training data set obtained in this way contains annotations of continuous video frames, and the facial key point detection model trained based on the training data set can accurately and stably identify facial key points in the video stream.

[0007] A training method for a facial key point detection model, comprising:

[0008] Acquire a three-dimensional face model, and determine facial key point annotations corresponding to the three-dimensional face model at an initial position, wherein the facial key point annotations are used to indicate position coordinates of the facial key points;

[0009] Determining a moving path of the three-dimensional face model in the three-dimensional space so that the three-dimensional face model moves according to the moving path;

[0010] When the three-dimensional face model moves along the moving path, photographing the three-dimensional face model to obtain a new two-dimensional face image;

[0011] The facial key point annotations corresponding to the new two-dimensional facial image are calculated based on the position of the three-dimensional facial model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position;

[0012] The new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image are added to the training data set, and the facial key point detection model is trained using the training data set.

[0013] A training device for a facial key point detection model, comprising:

[0014] Acquire a three-dimensional face model, and determine facial key point annotations corresponding to the three-dimensional face model at an initial position, wherein the facial key point annotations are used to indicate position coordinates of the facial key points;

[0015] Determining a moving path of the three-dimensional face model in the three-dimensional space so that the three-dimensional face model moves according to the moving path;

[0016] When the three-dimensional face model moves along the moving path, photographing the three-dimensional face model to obtain a new two-dimensional face image;

[0017] The facial key point annotations corresponding to the new two-dimensional face image are calculated based on the position of the three-dimensional face model in the moving path when taking pictures and the facial key point annotations corresponding to the initial position; the new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image are added to the training data set, and the facial key point detection model is trained using the training data set.

[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0019] Acquire a three-dimensional face model, and determine facial key point annotations corresponding to the three-dimensional face model at an initial position, wherein the facial key point annotations are used to indicate position coordinates of the facial key points;

[0020] Determining a moving path of the three-dimensional face model in the three-dimensional space so that the three-dimensional face model moves according to the moving path;

[0021] When the three-dimensional face model moves along the moving path, photographing the three-dimensional face model to obtain a new two-dimensional face image;

[0022] The facial key point annotations corresponding to the new two-dimensional facial image are calculated based on the position of the three-dimensional facial model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position;

[0023] The new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image are added to the training data set, and the facial key point detection model is trained using the training data set.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0025] Acquire a three-dimensional face model, and determine facial key point annotations corresponding to the three-dimensional face model at an initial position, wherein the facial key point annotations are used to indicate position coordinates of the facial key points;

[0026] Determining a moving path of the three-dimensional face model in the three-dimensional space so that the three-dimensional face model moves according to the moving path;

[0027] When the three-dimensional face model moves along the moving path, photographing the three-dimensional face model to obtain a new two-dimensional face image;

[0028] The facial key point annotations corresponding to the new two-dimensional facial image are calculated based on the position of the three-dimensional facial model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position;

[0029] The new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image are added to the training data set, and the facial key point detection model is trained using the training data set.

[0030] The training method, device, computer equipment and storage medium of the above-mentioned facial key point detection model, after determining the facial key point annotation corresponding to the initial position of the three-dimensional face model, during the movement of the three-dimensional face model, by continuously taking pictures, multiple new two-dimensional face images can be obtained, and the facial key point coordinates corresponding to each new two-dimensional face image are calculated, that is, the facial key point annotation corresponding to each new two-dimensional face image is obtained. The multiple new two-dimensional face images obtained by taking pictures during the movement of the three-dimensional face model constitute a continuous video frame, which is equivalent to annotating the facial key points of the continuous video frames. The annotation method is calculated based on the facial key point annotation corresponding to the initial position, and does not rely on manual annotation. The obtained facial key point annotation is consistent with the facial key point annotation corresponding to the initial position on the face, so it has stability. The obtained new two-dimensional face image with facial key point annotation is added to the training data set, and then the facial key point detection model is trained based on the training data set, so that the trained facial key point detection model can accurately and stably identify the facial key points in the video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] in:

[0033] Figure 1 is a flowchart of a method for training a facial key point detection model in an embodiment;

[0034] Figure 2 is a schematic diagram of a three-dimensional face model generated in an embodiment;

[0035] Figure 3A is a schematic diagram of translation of a three-dimensional face model in an embodiment;

[0036] Figure 3B is a schematic diagram of the rotation of a three-dimensional face model in an embodiment;

[0037] Figure 3C is a schematic diagram of a translation and rotation superposition in one embodiment;

[0038] Figure 4 is a schematic diagram of a translation path and a rotation path generated in one embodiment;

[0039] Figure 5is a flowchart of calculating the coordinates of key facial points corresponding to a new two-dimensional facial image in one embodiment;

[0040] Figure 6 is a structural block diagram of a training device for a facial key point detection model in an embodiment;

[0041] Figure 7 is a structural block diagram of a training device for a facial key point detection model in another embodiment;

[0042] Figure 8 The figure is a diagram of the internal structure of a computer device in an embodiment. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, a training method for a face key point detection model is proposed, and the training method for the face key point detection model can be applied to a terminal. This embodiment is illustrated by applying to a terminal. The training method for the face key point detection model specifically includes the following steps:

[0045] Step 102, obtaining a three-dimensional face model, determining facial key point annotations corresponding to the three-dimensional face model at an initial position, where the facial key point annotations are used to indicate position coordinates of facial key points.

[0046] Among them, face key points refer to feature points that reflect facial features, including eyebrows, eyes, nose, mouth and facial contours. The key areas of the face can be located based on the face key points. Therefore, the identification of face key points is a very important part of face image recognition. Face key point annotation refers to the annotation of the position of face key points, specifically expressed in coordinates. In a three-dimensional face model, face key point annotation refers to the position coordinates (three-dimensional coordinates) of face key points on the three-dimensional face model. In a two-dimensional face model, face key point annotation refers to the position coordinates (two-dimensional coordinates) of face key points on the two-dimensional face image.

[0047] Manual annotation of facial key points requires a lot of manpower and material resources, and the accuracy of annotation is unreliable, especially for continuous video frames. If manual annotation is used, the position of each annotation cannot be exactly the same, so it is easy for facial key points to jitter. In order to obtain more and stable two-dimensional face images with facial key points annotated, this embodiment innovatively proposes to obtain more two-dimensional face images with facial key annotations and stable facial key points based on a three-dimensional face model.

[0048] In one embodiment, a 3D face model is established based on a 2D face image using a 3D face reconstruction technique. Specifically, the pixels in the 2D face image are mapped to an initial 3D face model to obtain a 3D face model corresponding to the 2D face image. The generated 3D face model is a set H{h i (x i ,y i ,z i ,h i ,j i ,k i )}, where x i ,y i and z i is the three-dimensional space coordinate value, h i , j i and k i is the color (texture) information value at the coordinate, for example, the color value of the HSV or YUV color three-dimensional space. i and i The position of each point in the initial 3D face model is composed of three coordinates, which is the same as the corresponding 2D coordinates in the 2D face image. When mapping, the coordinate points on the 2D face image and the coordinates on the 3D face model are in a one-to-one correspondence. i and i After the value of , a point (x, y, z) can be uniquely determined on the 3D face model. Specifically, when x is known i and i After the value of , find the horizontal coordinate x on the 3D face model i and the ordinate y i The point is unique, and the z coordinate is determined based on this point. i , j i and k i It is also the same as the pixel value at the corresponding coordinate in the two-dimensional face image, such as Figure 2 FIG. 1 is a schematic diagram of a generated three-dimensional face model in one embodiment.

[0049] In another embodiment, a 3D scanner is directly used to scan the face to obtain a 3D face model, and then the 3D face model is annotated with facial key points. Specifically, a high-precision 3D scanner can be used to directly scan the face to obtain a 3D face model with texture information, and the 3D face model is annotated with facial key points, for example, the annotation can be performed manually or by facial key point recognition.

[0050] Step 104: determine a moving path of the three-dimensional face model in the three-dimensional space, so that the three-dimensional face model moves according to the moving path.

[0051] Among them, in order to obtain more training images with face annotations, the three-dimensional face model needs to be moved to facilitate taking pictures during the movement to obtain images from more angles. The movement path can be randomly generated or specified. The movement path includes: a translation path and a rotation path. The translation path refers to the horizontal movement of the three-dimensional face model, and the rotation path refers to the rotation path of the three-dimensional face model. For example, the rotation path from the front face to the side face. The translation path and the rotation path are performed simultaneously. The three-dimensional face model will rotate while translating. As shown in Figure 3, this is an embodiment. Figure 3A Schematic diagram of the translation of the 3D face model and Figure 3B Schematic diagram of the rotation of the 3D face model. Figure 3C It is the effect of the superposition of the two, that is, the effect of simultaneous translation and rotation.

[0052] Step 106: While the three-dimensional face model is moving along the moving path, the three-dimensional face model is photographed to obtain a new two-dimensional face image.

[0053] In the process of moving the 3D face model, the 3D face model is continuously photographed. It should be noted that the position of the camera is fixed and the 3D face model is changed. In this way, images of the 3D face model at different angles can be obtained, that is, different new 2D face images can be obtained. Multiple new 2D face images constitute continuous video frame images. The coordinates of the facial key points corresponding to each new 2D face image are obtained by subsequent calculation, which is equivalent to obtaining the facial key point annotations corresponding to the continuous video frame images.

[0054] Step 108, calculating and obtaining facial key point annotations corresponding to a new two-dimensional face image based on the position of the three-dimensional face model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position;

[0055] Among them, the 3D face model is marked with facial key points at the initial position. According to the position in the moving path, the translation vector and rotation matrix corresponding to the 3D face model from the initial position to the current position can be calculated, and then the coordinates of the facial key points corresponding to the 3D face model at this position can be calculated. When the 3D face model is projected onto a 2D plane, the x in the 2D face image i and i The same as the corresponding 2D coordinates in the 3D face model, except that the 3D coordinate z is removed. i Therefore, if the coordinates of the facial key points corresponding to the three-dimensional face model at the current position are known, the coordinates of the facial key points corresponding to the new two-dimensional face image can be determined, that is, the facial key point annotations corresponding to the new two-dimensional face image are determined.

[0056] Step 110: Add the new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image to a training data set, and use the training data set to train a facial key point detection model.

[0057] Among them, the obtained multiple new two-dimensional face images with face key points are added to the training data set, thereby expanding and improving the training data set, which is conducive to enabling the model trained according to the training data set to more accurately identify the face key points. Since the face key points in the generated multiple new two-dimensional face images with face key points are obtained by calculation, that is, the coordinates of the face key points on the obtained two-dimensional face images correspond to the same face key points of the three-dimensional face model, that is, the obtained face key point coordinates are consistent in physical space and point to the same point. Therefore, it is still stable even in the case of continuous video frames, so the trained model is more stable for the recognition of face key points in continuous video frames.

[0058] The training method of the above-mentioned facial key point detection model can obtain multiple new two-dimensional facial images by continuously taking photos during the movement of the three-dimensional facial model, and calculate the facial key point coordinates corresponding to each new two-dimensional facial image, that is, obtain the facial key point annotations corresponding to each new two-dimensional facial image. The multiple new two-dimensional facial images obtained by taking photos during the movement of the three-dimensional facial model constitute a continuous video frame. The annotation method is calculated based on the facial key point annotations corresponding to the initial position, and does not rely on manual annotation. The calculated facial key point annotations are consistent with the facial key point annotations corresponding to the initial position on the face, so they are stable. The obtained new two-dimensional facial images with facial key point annotations are added to the training data set, so that the facial key point detection model trained based on the training data set can accurately and stably identify the facial key points in the video stream.

[0059] In one embodiment, obtaining a three-dimensional face model and determining facial key point annotations corresponding to the three-dimensional face model include: obtaining a two-dimensional face image and facial key point annotations corresponding to the two-dimensional face image from a training data set; generating a three-dimensional face model corresponding to the two-dimensional face image based on the two-dimensional face image; and determining facial key point annotations mapped to the three-dimensional face model based on the facial key point annotations corresponding to the two-dimensional face image.

[0060] Among them, the corresponding three-dimensional face model is generated based on the existing two-dimensional face images in the training data set, and the facial key point annotations corresponding to the three-dimensional face model are obtained through mapping relationship calculation.

[0061] In one embodiment, the obtaining of the three-dimensional face model and determining the facial key point annotations corresponding to the three-dimensional face model include: scanning the face with a three-dimensional scanner to obtain the three-dimensional face model; and annotating the facial key points of the three-dimensional face model.

[0062] Among them, in order to obtain continuous video frame annotations, a three-dimensional scanner is directly used to scan the real face to obtain a three-dimensional face model, and then the three-dimensional face model is annotated with facial key points. In this way, multiple continuous video frame images can be obtained by taking pictures of the three-dimensional face model, and the facial key point annotations corresponding to the multiple continuous video frame images can be calculated. The facial key point annotations at the same position in different video frame images are consistent, thereby preventing jitter.

[0063] In one embodiment, the moving path includes: a translation path and a rotation path; determining the moving path of the three-dimensional face model in the three-dimensional space includes: randomly generating the translation path and rotation path of the three-dimensional face model in the three-dimensional space, so that the three-dimensional face model can move smoothly according to the translation path and the rotation path.

[0064] Among them, in order to obtain facial images at different angles, the three-dimensional face model needs to be in motion. In order to make the three-dimensional face model move, it is necessary to plan the movement path for the three-dimensional face model in advance. The movement path includes a translation path and a rotation path. The translation path refers to the horizontal movement of the three-dimensional face model, while the rotation path refers to the rotation path of the three-dimensional face model. The three-dimensional face model performs translation and rotation movements within the range based on the width, height and depth of the image. In order to ensure the continuity and smoothness of the movement, it is necessary to generate smooth translation paths and rotation paths for the three-dimensional face model in advance. Figure 4 As shown in the figure, L 1 and L 2 They are schematic diagrams of the generated translation path and rotation path. It can be understood that the translation path and rotation path are composed of points. Figure 4 The points on the translation path and rotation path are shown in Figure 1. The rotation path is the path of the face direction vector. The 3D face model rotates while translating, so the two are performed simultaneously. Figure 4 The three-dimensional face model is initially at point O, and the vector of the initial face direction is OO'. Then when the movement starts, the three-dimensional face model moves from point O to point A1. At the same time, the face direction changes from OO' to OB1. Similarly, when the three-dimensional model moves from A1 to A2, the face direction changes from OB1 to OB2.

[0065] By randomly generating the translation path and rotation path of the three-dimensional face model in the three-dimensional space, it is convenient to take pictures of the three-dimensional face model in the process of movement to obtain images at different angles. Moreover, since the path is randomly generated, the translation path and rotation path corresponding to different three-dimensional face models are different. In this way, images of different faces at different angles can be obtained, which is more conducive to enriching the training data set.

[0066] In one embodiment, the randomly generating a translation path and a rotation path of the three-dimensional face model in a three-dimensional space so that the three-dimensional model can move smoothly according to the translation path and the rotation path includes:

[0067] A plurality of points are randomly generated in a three-dimensional space according to a preset density, and the plurality of points are sorted according to a preset rule to obtain a sorted point sequence, wherein the preset rule is: randomly selecting a point from the plurality of points as a starting point, finding a point closest to the starting point as a second point, finding a point closest to the second point as a third point, and so on, until all points are traversed;

[0068] Perform smooth interpolation according to the sorted point sequence to obtain a smoothed translation path;

[0069] The depth coordinate of each point on the translation path is determined, and the rotation path is determined according to the depth coordinate of each point on the translation path, so that the depth coordinate of each point on the rotation path is greater than the depth coordinate of each point on the smooth path.

[0070] Among them, N points in the three-dimensional space are randomly generated according to a certain density. The reason for requiring a certain density is to keep the distance between points sufficient to ensure that the point cloud of the three-dimensional face model can move smoothly between these two points. Specifically, a point is randomly selected as the starting point, and then the point closest to it is found as the second point, and then the point closest to the second point is found as the third point, and so on. The generated N points are sorted to obtain a sequence of N points. Then the sorted N point sequence is smoothed and interpolated. For example, spline interpolation, Bezier curve interpolation, etc. can be used to make the sequence become a smooth path curve, which is called a "translation path", that is, the path composed of multiple points becomes a smooth path curve.

[0071] The generation of the rotation path can also be done in the above way, that is, first randomly generate N points in the three-dimensional space according to a certain density, and then connect the N points to form a smooth path curve. It should be noted that it is necessary to ensure that the depth coordinate, that is, the Z-axis coordinate, of each point on the generated rotation path is greater than the value on the corresponding translation path. Specifically, the following conditions are met: 2 =max(z 1 )+σ, where σ>0, z 2 Represents the depth coordinate on the rotation path, z 1 Represents the depth coordinate on the translation path. That is, the value in the Z direction on the rotation path must be greater than that on the translation path. The reason why the depth coordinate on the rotation path must be greater than the depth coordinate on the translation path is that the camera is fixed and in front of the initial 3D face model, so that the face can be captured at all times during the movement of the 3D face model, that is, the face is facing forward, and there will be no situation where the face is facing backward and the face cannot be captured.

[0072] In one embodiment, the above method also includes: performing diversified processing on the new two-dimensional face image to obtain two-dimensional face images in multiple scenarios, and adding the two-dimensional face images in the multiple scenarios and the corresponding face key point annotations to the training data set, and the diversified processing includes: replacing the background image, adjusting the image tone and adjusting the image brightness. At least one of the following.

[0073] Among them, in order to increase the diversity and generalization of the training data set, the existing two-dimensional face images are subjected to data enhancement processing, that is, diversification processing. In one embodiment, by adding different rendering methods, for example, changing the hue and brightness of the existing two-dimensional face image, a new two-dimensional face image is obtained. In another embodiment, the existing two-dimensional face image is cut out and then pasted into different background images to obtain training data for more scenes. Through the above-mentioned diversification processing, the diversity and generalization of the training data set are enriched, which is conducive to making the trained model robust, that is, adapting to the recognition of facial key points in various scenes.

[0074] like Figure 5 As shown, in one embodiment, the calculating of the facial key point coordinates corresponding to the new two-dimensional facial image based on the position of the three-dimensional facial model in the moving path when taking the photo includes:

[0075] Step 108A, determining the current position of the three-dimensional face model when taking the photo, and calculating the translation vector and rotation matrix corresponding to the initial position to the current position.

[0076] Among them, the facial key point annotations corresponding to the two-dimensional face image obtained by taking a photo are obtained by calculation. Specifically, it is necessary to first determine the current position of the three-dimensional face model when taking a photo. The current position includes: the position on the translation path and the position on the rotation path. Calculate the translation vector and rotation matrix corresponding to the initial position to the current position. The coordinates of the initial position can be set as the origin to facilitate calculation.

[0077] The calculation of the translation vector of the three-dimensional face model is relatively simple. It can be determined based on the three-dimensional coordinates of the current position and the three-dimensional coordinates of the initial position. Specifically, the translation vector is used to indicate the distance and relative direction between the initial position and the current position. The translation direction is the direction from the initial position to the current position, and the translation distance refers to the straight-line distance between the current position and the initial position. Since the three-dimensional face model is composed of many points, when calculating the position of the three-dimensional face model, a representative point can be selected to represent the position of the three-dimensional face model. For example, the position of the nose tip in the face is selected as the position representing the three-dimensional face model. Assuming that the initial position of the nose tip is (x0, y0, z0), and the current position of the nose tip on the translation path is (x1, y1 and z1), the translation vector is expressed as Y = (x1-x0, y1-y0, z1-z0). The translation vector can be divided into three directions of translation, namely, translation in the X-axis direction, translation in the Y-axis direction, and translation in the Z-axis direction.

[0078] The rotation matrix of the 3D face model is calculated as follows:

[0079] refer to Figure 4, assuming that the face direction vector of the 3D face model is rotated from the vector OO' at the initial position to the direction OB1, then the specific calculation process of the rotation matrix from the initial position OO' to OB1 can refer to the following method:

[0080] 1) First find the vector product (cross product) of the two vectors: And its unit vector is Where || is the modulus;

[0081] 2) The dot product of two vectors is also required:

[0082] 3) Then the angle between the two vectors is expressed as Where || is the modulus;

[0083] 4) According to the Rodriguez rotation formula, the rotation matrix corresponding to the initial position to the current position can be obtained as:

[0084]

[0085] Where I is the identity matrix and:

[0086]

[0087] Each point in the 3D face model corresponds to the same rotation matrix and translation vector from the initial position to the current position. Therefore, after calculating the rotation matrix and translation vector, the current position coordinates can be calculated based on the initial position coordinates of each point in the 3D face model.

[0088] Step 108B, calculating the position coordinates of the facial key points in the three-dimensional face model at the current position based on the translation vector, the rotation matrix and the facial key point annotations corresponding to the initial position.

[0089] Among them, the movement of each point in the three-dimensional face model corresponds to the same translation vector and rotation matrix. If the position coordinates of the key points of the face in the three-dimensional face model at the initial position are known, then the position coordinates of the key points of the face in the three-dimensional face model at the current position can be calculated based on the translation vector and the rotation matrix. Specifically, after the translation vector and the rotation matrix of the initial position and the current position are known, assuming that the coordinates of the key points of the face corresponding to the current position are A (x0, y0, z0), assuming that the rotation matrix is ​​represented by M and the translation vector is represented by Y, then after the key points of the face are rotated and translated, the coordinates of the key points at the current position are B, B = MA + Y, that is, the initial coordinates A are rotated and then translated.

[0090] Step 108C, determining facial key point labels corresponding to the new two-dimensional face image according to the position coordinates of the facial key points in the three-dimensional face model at the current position.

[0091] The coordinates of the facial key points corresponding to the two-dimensional plane are the same as the x and y coordinates of the facial key points in the three-dimensional face model. After the position coordinates of the facial key points in the three-dimensional face model are known, the position coordinates of the facial key points corresponding to the corresponding new two-dimensional face image can be determined, that is, the corresponding facial key point annotations are determined.

[0092] The above method can accurately calculate the facial key point annotations corresponding to the new two-dimensional face image. The obtained facial key point annotations point to the same position as the facial key points on the three-dimensional face model. In this way, no jitter will occur when identifying facial key points in continuous video frames, which greatly improves the stability of recognition.

[0093] like Figure 6 As shown, a training device for a facial key point detection model includes:

[0094] An acquisition module 602 is used to acquire a three-dimensional face model and determine facial key point annotations corresponding to the three-dimensional face model, where the facial key point annotations refer to position coordinates of the facial key points;

[0095] A first determining module 604 is used to determine a moving path of the 3D face model in the 3D space, so that the 3D face model moves according to the moving path;

[0096] A photographing module 606, configured to photograph the 3D face model to obtain a new 2D face image while the 3D face model moves along the moving path;

[0097] A calculation module 608 is used to calculate the coordinates of facial key points corresponding to the new two-dimensional facial image based on the position of the three-dimensional facial model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position, and use the coordinates of the facial key points corresponding to the new two-dimensional facial image as the facial key point annotations corresponding to the new two-dimensional facial image;

[0098] A second determining module 610 is used to determine the facial key point labels corresponding to the new two-dimensional facial image according to the facial key point coordinates corresponding to the new two-dimensional facial image;

[0099] The adding module 612 is used to add the new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image to the training data set, and use the training data set to train the facial key point detection model.

[0100] In one embodiment, the acquisition module 602 is also used to obtain a two-dimensional face image and facial key point annotations corresponding to the two-dimensional face image from a training data set; generate a three-dimensional face model corresponding to the two-dimensional face image based on the two-dimensional face image; and determine the facial key point annotations mapped to the three-dimensional face model based on the facial key point annotations corresponding to the two-dimensional face image.

[0101] In one embodiment, the acquisition module is further used to scan the face using a three-dimensional scanner to obtain a three-dimensional face model; and to mark key facial points on the three-dimensional face model.

[0102] In one embodiment, the moving path includes: a translation path and a rotation path; the determination module is also used to randomly generate the translation path and rotation path of the three-dimensional face model in the three-dimensional space, so that the three-dimensional model can move smoothly according to the translation path and rotation path.

[0103] In one embodiment, the determination module is also used to randomly generate multiple points in three-dimensional space according to a preset density, randomly select a point from the multiple points as a starting point, find the point closest to the starting point as a second point, find the point closest to the second point as a third point, and so on, sort the generated multiple points one by one to obtain a sorted point sequence; perform smooth interpolation according to the sorted point sequence to obtain a smoothed translation path; determine the depth coordinate of each point on the translation path, and determine the rotation path according to the depth coordinate of each point on the translation path, wherein the depth coordinate of each point on the rotation path is greater than the depth coordinate of each point on the smooth path.

[0104] In one embodiment, the calculation module is also used to determine the current position of the three-dimensional face model when taking a photo, calculate the translation vector and rotation matrix corresponding to the initial position to the current position; calculate the position coordinates of the facial key points in the three-dimensional face model at the current position based on the translation vector, the rotation matrix and the facial key point annotations corresponding to the initial position; determine the facial key point annotations corresponding to the new two-dimensional face image based on the position coordinates of the facial key points in the three-dimensional face model at the current position.

[0105] like Figure 7 As shown, in one embodiment, the training device of the above-mentioned facial key point detection model further includes:

[0106] The processing module 614 is used to perform diversified processing on the new two-dimensional face image to obtain two-dimensional face images in various scenes, and add the two-dimensional face images in various scenes and the corresponding face key point annotations to the training data set, wherein the diversified processing includes at least one of replacing the background image, adjusting the image tone and adjusting the image brightness.

[0107] Figure 8 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 8 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement the above-mentioned training method for the facial key point detection model. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may execute the above-mentioned training method for the facial key point detection model. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] In one embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the training method of the above-mentioned facial key point detection model.

[0109] In one embodiment, a computer device is proposed, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned training method for the facial key point detection model.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a facial key point detection model, characterized in that: include: Acquire a three-dimensional face model, and determine facial key point annotations corresponding to the three-dimensional face model at an initial position, wherein the facial key point annotations are used to indicate position coordinates of the facial key points; Determining a moving path of the three-dimensional face model in the three-dimensional space so that the three-dimensional face model moves according to the moving path; When the three-dimensional face model moves along the moving path, photographing the three-dimensional face model to obtain a new two-dimensional face image; The facial key point annotations corresponding to the three-dimensional face model at the current position when taking the photo are calculated based on the position of the three-dimensional face model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position, and the facial key point annotations corresponding to the new two-dimensional face image are determined according to the facial key point annotations corresponding to the three-dimensional face model at the current position when taking the photo; The new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image are added to a training data set, and the training data set is used to train a facial key point detection model.

2. The method according to claim 1, characterized in that The step of obtaining a three-dimensional face model and determining facial key point annotations corresponding to the three-dimensional face model includes: Obtaining a two-dimensional face image and facial key point annotations corresponding to the two-dimensional face image; Generating a three-dimensional face model corresponding to the two-dimensional face image based on the two-dimensional face image; Based on the facial key point annotations corresponding to the two-dimensional facial image, the facial key point annotations mapped to the three-dimensional facial model are determined.

3. The method according to claim 1, characterized in that The step of obtaining a three-dimensional face model and determining facial key point annotations corresponding to the three-dimensional face model includes: Scanning the human face with a 3D scanner to obtain a 3D human face model; The three-dimensional face model is labeled with key facial points.

4. The method according to claim 1, characterized in that: The moving path includes: a translation path and a rotation path; Determining a moving path of the three-dimensional face model in three-dimensional space includes: A translation path and a rotation path of the three-dimensional face model in a three-dimensional space are randomly generated so that the three-dimensional model can move smoothly according to the translation path and the rotation path.

5. The method according to claim 4, characterized in that The randomly generating a translation path and a rotation path of the three-dimensional face model in a three-dimensional space so that the three-dimensional model can move smoothly according to the translation path and the rotation path includes: A plurality of points are randomly generated in a three-dimensional space according to a preset density, and the plurality of points are sorted according to a preset rule to obtain a sorted point sequence, wherein the preset rule is: randomly selecting a point from the plurality of points as a starting point, finding a point closest to the starting point as a second point, finding a point closest to the second point as a third point, and so on, until all points are traversed; Perform smooth interpolation according to the sorted point sequence to obtain a smoothed translation path; The depth coordinate of each point on the translation path is determined, and the rotation path is determined according to the depth coordinate of each point on the translation path, wherein the depth coordinate of each point on the rotation path is greater than the depth coordinate of each point on the smooth path.

6. The method according to claim 1, characterized in that The method of calculating the facial key point annotations corresponding to the three-dimensional face model at the current position when taking the photo based on the position of the three-dimensional face model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position, and determining the facial key point annotations corresponding to the new two-dimensional face image according to the facial key point annotations corresponding to the three-dimensional face model at the current position when taking the photo, includes: Taking the position of the three-dimensional face model in the moving path when taking the photo as the current position of the three-dimensional face model when taking the photo, and calculating the translation vector and rotation matrix corresponding to the initial position to the current position; The facial key point annotations corresponding to the current position of the three-dimensional face model when taking the photo are calculated according to the translation vector, the rotation matrix and the facial key point annotations corresponding to the initial position; The facial key point annotations corresponding to the new two-dimensional face image are determined according to the facial key point annotations corresponding to the current position of the three-dimensional face model when taking the photo.

7. The method according to claim 1, characterized in that The method further comprises: The new two-dimensional face image is subjected to diversified processing to obtain two-dimensional face images in a variety of scenarios, and the two-dimensional face images in the various scenarios and the corresponding face key point annotations are added to the training data set, wherein the diversified processing includes: replacing the background image, adjusting the image tone and adjusting the image brightness.

8. A training device for a facial key point detection model, characterized in that: include: An acquisition module, used to acquire a three-dimensional face model, and determine the facial key point annotations corresponding to the three-dimensional face model, wherein the facial key point annotations refer to the position coordinates of the facial key points; A first determining module, used to determine a moving path of the 3D face model in a 3D space, so that the 3D face model moves according to the moving path; A photographing module, used for photographing the three-dimensional face model to obtain a new two-dimensional face image during the movement of the three-dimensional face model along the movement path; a calculation module, configured to calculate the facial key point annotations corresponding to the three-dimensional face model at the current position when taking the photo based on the position of the three-dimensional face model in the moving path when taking the photo and the facial key point annotations corresponding to the initial position, and determine the facial key point annotations corresponding to the new two-dimensional face image according to the facial key point annotations corresponding to the three-dimensional face model at the current position when taking the photo; A second determination module is used to determine the facial key point labels corresponding to the new two-dimensional facial image according to the facial key point coordinates corresponding to the new two-dimensional facial image; The adding module is used to add the new two-dimensional face image and the facial key point annotations corresponding to the new two-dimensional face image to the training data set, and use the training data set to train the facial key point detection model.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the training method of the facial key point detection model as described in any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the training method of the facial key point detection model as described in any one of claims 1 to 7.

Images