Face detection neural network and training method, face detection method, storage medium

Abstract

The present specification provides a face detection neural network, which includes: an encoding network, a first decoding network, a second decoding network and a prediction network; the encoding network is used to obtain samples of face images and distribute the output results to the first decoding network, the second decoding network and the prediction network; the first decoding network is used to extract the predicted coordinates of the sample key points according to the output results of the encoding network; the second decoding network is used to obtain the face posture change parameters of the predicted coordinates of the sample key points according to the output results of the encoding network; the prediction network is used to use the output results of the encoding network and the obtained predicted coordinates and face posture change parameters as input for training, and output a face model containing the three-dimensional coordinates of the sample key points.

Description

Background Art

[0002] In recent years, with the rapid development of deep learning technology, face detection technology has also made great progress. However, the existing face detection technology based on deep learning requires a large amount of training data with three-dimensional coordinates of key points, and the key points annotated in the same training data also need to be dense enough. However, the collection of three-dimensional face data and the annotation of key points are more difficult than traditional images, and require a lot of manpower, material and financial resources. Summary of the invention

[0003] In order to overcome the problem that the collection of three-dimensional face data and the annotation of key points in related technologies are more difficult than those of traditional images, and require a lot of manpower, material resources and financial resources, this specification provides a face detection neural network and its training method, a face detection method, a storage medium and a terminal device.

[0004] According to a first aspect of an embodiment of this specification, a neural network for face detection is provided, the neural network comprising:

[0005] encoding network, first decoding network, second decoding network and prediction network,

[0006] The encoding network is used to obtain samples of face images and distribute the output results to the first decoding network, the second decoding network and the prediction network, wherein the samples include face images with three-dimensional coordinates of key points annotated;

[0007] The first decoding network is used to extract the predicted coordinates of the sample key points according to the output result of the encoding network;

[0008] The second decoding network is used to obtain the facial posture change parameters of the predicted coordinates of the sample key points according to the output results of the encoding network;

[0009] The prediction network is used to train the output results of the encoding network and the acquired prediction coordinates and facial posture change parameters as input, and output a facial model containing the three-dimensional coordinates of the sample key points, and the number of key points outputted is greater than the number of key points of the three-dimensional coordinates marked in the sample.

[0010] According to a second aspect of an embodiment of this specification, a method for training a neural network for face detection is provided, the training method comprising:

[0011] Acquire a sample of a face image, wherein the sample includes a face image with three-dimensional coordinates of key points annotated;

[0012] Using the sample as input of the neural network to obtain predicted coordinates of key points and facial posture change parameters;

[0013] The neural network is trained based on the acquired predicted coordinates and facial posture change parameters so that the trained neural network outputs three-dimensional coordinates containing a first number of key points, wherein the first number is greater than the number of key points annotated with the three-dimensional coordinates in the sample.

[0014] According to a third aspect of the embodiments of this specification, a method for face detection is provided, the method being implemented by the encoding network and the prediction network obtained by training according to the first aspect of the embodiments of this specification, the method comprising:

[0015] The encoding network acquires an unlabeled face image for processing and outputs the processing result to the prediction network;

[0016] The prediction network outputs the three-dimensional coordinates of key points of the face image.

[0017] According to a fourth aspect of the embodiments of this specification, a computer storage medium is provided, wherein a computer program code is stored in the computer storage medium. When the computer program code is executed on a processor, the processor executes the training of the neural network described in the first aspect of the embodiments of this specification or the steps of the method described in the third aspect of the embodiments of this specification.

[0018] According to the fifth aspect of the embodiments of this specification, a terminal device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method described in the third aspect of the embodiments of this specification are implemented.

[0019] In an embodiment of the present application, a facial image sample annotated with three-dimensional coordinates of key points is used as an input to a neural network for face detection, and the neural network is trained by obtaining the predicted coordinates of key points and facial posture change parameters of the sample. By using the neural network for face detection, it is possible to input facial image samples annotated with only a small number of three-dimensional coordinates of key points into the neural network, and obtain the three-dimensional coordinates of a large number of key points of the sample. This can solve the problem in the prior art that the collection of three-dimensional face data and the annotation of key points are difficult and require a lot of manpower, material and financial resources.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0022] Figure 1 This is a system architecture diagram of a face detection neural network and its training method, and a face detection method, which are shown in this specification according to an exemplary embodiment.

[0023] Figure 2 It is a structural diagram of a neural network for face detection according to an exemplary embodiment of this specification.

[0024] Figure 3 is a flowchart of another method for obtaining predicted coordinates of key points according to an exemplary embodiment of the present specification.

[0025] Figure 4 This is a flowchart of a method for obtaining facial posture change parameters of key points according to an exemplary embodiment of this specification.

[0026] Figure 5 It is a flowchart of a method for training a prediction network according to an exemplary embodiment of the present specification.

[0027] Figure 6 This is a flowchart of a neural network training method for face detection according to an exemplary embodiment of this specification.

[0028] Figure 7 The flowchart of a method for applying a prediction network is shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0030] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] Face detection refers to the detection of the positions of key points on a face given a face image. The key points usually include eyebrows, eyes, nose, mouth, facial contours, etc. Face detection technology is the basic technology for application scenarios such as face dressing, beauty makeup, face effects, and face AR (Augmented Reality). Taking face effects as an example, in some live video applications, there will be some demand for three-dimensional animation effects, such as adding three-dimensional rabbit ears and pig masks to the faces in the video. These three-dimensional animation effects need to be based on the accurate positioning of the key points of the face, that is, for the key points of the face, the accurate three-dimensional coordinates of the key points can be obtained.

[0033] In recent years, with the development of deep learning technology, some face detection technologies based on deep learning have emerged. For example, a given face image is input, and the coordinates of the key points of the face are regressed through a convolutional neural network. For another example, a given face image is input, and the feature map corresponding to the key points of the face is regressed through a convolutional neural network, and the positions representing the key points of the face are determined from the feature map. However, the existing face detection technologies based on deep learning require a large amount of training data with three-dimensional coordinates of key points to be input into the training network, and the key points annotated in the same training data also need to be dense enough (that is, dense three-dimensional data is required). However, annotated three-dimensional face data is very precious, because the collection of three-dimensional face data and the annotation of key points are more difficult than traditional images, and require a lot of manpower, material resources and financial resources.

[0034] In view of the problem that the collection of three-dimensional face data and the annotation of key points in the prior art are more difficult than those of traditional images, and require a lot of manpower, material resources and financial resources, the present application proposes a face detection neural network and its training method, a face detection method, a storage medium and a terminal device.

[0035] Next, the embodiments of this specification are described in detail.

[0036] Figure 1 A schematic diagram of an exemplary system architecture of a face detection neural network and a training method thereof, and a face detection method to which the embodiments of the present application can be applied is shown.

[0037] like Figure 1 As shown, the system architecture 1000 may include one or more of terminal devices 1001, 1002, 1003, a network 1004, and a server 1005. The network 1004 is used to provide a medium for communication links between the terminal devices 1001, 1002, 1003 and the server 1005. The network 1004 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 1005 may be a server cluster composed of multiple servers.

[0039] Users can use terminal devices 1001, 1002, 1003 to interact with server 1005 through network 1004 to receive or send messages, etc. Terminal devices 1001, 1002, 1003 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, etc.

[0040] Server 1005 may be a server that provides various services. For example, server 1005 may provide a training of a neural network for face detection of the present application: server 1005 obtains a face image from terminal devices 1001, 1002, and 1003, and the face image may be a face image annotated with three-dimensional coordinates of a small number of key points (i.e., sparse three-dimensional data). The server trains the neural network for face detection based on the acquired face image. Based on the trained neural network, the server can output the three-dimensional coordinates of a large number of key points of the face image (i.e., dense three-dimensional data). For another example, server 1005 may provide a method for face detection of the present application for execution: server 1005 is pre-installed with all or part of the trained neural network for face detection, server 1005 obtains unannotated face images from terminal devices 1001, 1002, and 1003, and obtains the three-dimensional coordinates of the key points of the face image by all or part of the trained neural network for face detection.

[0041] Terminal devices 1001, 1002, and 1003 may be terminal devices that provide various services. For example, the terminal device may be a terminal device with an image acquisition unit, so the terminal device can acquire facial images for use by the terminal device itself or send them to other devices. For another example, by installing all or part of the neural network for face detection trained by server 1005 on terminal devices 1001, 1002, and 1003 in advance, the terminal device can obtain the three-dimensional coordinates of the key points of the facial image for the input unlabeled facial image.

[0042] However, it should be understood that for terminal devices 1001, 1002, 1003 whose computing power meets the training requirements, the training method of the neural network for face detection disclosed in the embodiment of the present application can also be executed by terminal devices 1001, 1002, 1003. All or part of the neural network trained by terminal devices 1001, 1002, 1003 can directly obtain the three-dimensional coordinates of the key points of the face image for the input unlabeled face image. Of course, it is also possible to configure all or part of the neural network trained by terminal devices 1001, 1002, 1003 on server 1005 in advance, so that server 1005 can obtain the three-dimensional coordinates of the key points of the face image for the input unlabeled face image.

[0043] like Figure 2 As shown, Figure 2 This is a structural diagram of a neural network for face detection according to an exemplary embodiment of this specification. The neural network for face detection includes: an encoding network, a first decoding network, a second decoding network and a prediction network; the encoding network is used to obtain a face image sample and send the output result to the first decoding network, the second decoding network and the prediction network, wherein the sample includes a face image with three-dimensional coordinates of key points annotated; the first decoding network extracts the predicted coordinates of the sample key points according to the output result of the encoding network; the second decoding network is used to obtain the face posture change parameters of the predicted coordinates of the sample key points according to the output result of the encoding network; the output result of the encoding network and the obtained predicted coordinates and face posture change parameters are used as input for training, and a face model containing the three-dimensional coordinates of the sample key points is output, and the number of key points output is greater than the number of key points of the three-dimensional coordinates annotated in the sample.

[0044] The training data of the neural network for face detection described in the present application may be face image samples annotated with three-dimensional coordinates of key points. It should be understood that the face image samples annotated with three-dimensional coordinates of key points may come from a public data set, or may be face images collected by a user and annotated with three-dimensional coordinates of key points. The present application does not place any restrictions on the source of the training data. The key points annotated in the face image samples may be the eyes, nose, mouth, facial contour, or other locations on the face, and the present application does not place any restrictions on this.

[0045] In one embodiment, when the training data used is a face image sample annotated with three-dimensional coordinates of key points, the activation function used in the neural network for face detection is a positive number corresponding to the three-dimensional coordinates of the key points, that is, the activation functions corresponding to the X-axis, Y-axis, and Z-axis are all positive numbers. For example, if the three-dimensional coordinates of a key point are x, y, and z, then the activation function corresponding to the key point is 1, 1, 1, indicating that the three-dimensional coordinates of the key point are all what the neural network needs to learn. Of course, those skilled in the art should understand that the activation functions corresponding to the X-axis, Y-axis, and Z-axis may be different and may be other positive numbers. For example, the three-dimensional coordinates of a key point are x, y, and z, and the corresponding activation functions can be 2, 2, and 10. The activation functions corresponding to the X-axis, Y-axis, and Z-axis are all greater than 0, indicating that the neural network needs to learn the three-dimensional coordinates of the key point. Among them, the activation function corresponding to the Z-axis coordinate is greater than the activation functions corresponding to the X-axis and Y-axis coordinates. This means that the learning of the Z-axis coordinate has a relatively large weight in the training process of the neural network for face detection, that is, through the training of the neural network for face detection, what we want to learn more is the Z-axis coordinate of the key point.

[0046] In one embodiment, in order to ensure better training effect, so that the trained face detection neural network has better generalization performance and more accurate prediction results for different input face images, the training data of the face detection neural network described in the present application can also be mixed data, and the mixed data includes face image samples marked with three-dimensional coordinates of key points and face image samples marked with two-dimensional coordinates of key points. For face images marked with two-dimensional coordinates of key points, since the key points do not have Z-axis coordinate values, the Z-axis coordinate values ​​of the key points marked with two-dimensional coordinates can be set to negative numbers, for example, set to -1, indicating that the Z-axis coordinates of the key points do not exist. Correspondingly, when the training data used is mixed data, for face image samples marked with three-dimensional coordinates of key points, the activation functions corresponding to the X-axis, Y-axis and Z-axis of the key points are the same as described above, and will not be repeated here. For face images marked with two-dimensional coordinates of key points, the activation functions corresponding to the X-axis and Y-axis are also the same as described above; the activation function corresponding to the Z-axis can be set to 0, indicating that the Z-axis coordinates of the key points are not what the face detection neural network needs to learn.

[0047] The above-mentioned mixed data is used as the training data of the neural network for face detection. The face image samples with the two-dimensional coordinates of the key points are marked. Although they do not have Z-axis coordinate values, they have X-axis and Y-axis coordinate values. The two-dimensional coordinate values ​​also contain certain useful information. Therefore, using mixed data for training can learn more useful information about the key points of the face image when there are insufficient face image samples with the three-dimensional coordinates of the key points, and can achieve better training effects on the neural network for face detection.

[0048] The encoding network of the neural network for face detection is used to obtain face image samples for preliminary processing, and send the results of the preliminary processing to other parts of the encoding network of the neural network for face detection. In one embodiment, the encoding network in the neural network for face detection can be a lightweight neural network. For example, the encoding network can be a Mobilenet series network designed by Google for deep learning applications on mobile and embedded terminals, including MobilenetV1, MobilenetV2 and MobilenetV3. The encoding network can also be other lightweight neural networks, such as SqueezeNet, ShuffleNet, Xception and other networks.

[0049] The neural network for face detection uses a lightweight neural network as the encoding network, which can save the volume of the neural network for face detection, improve the running speed of the neural network for face detection, facilitate the transplantation of part or all of the neural network for face detection to mobile terminals, and expand the application scope and application scenarios of the neural network for face detection.

[0050] In one embodiment, according to the output result of the encoding network, the predicted coordinates of the sample key points are extracted, which can be achieved by the first decoding network of the neural network for face detection.

[0051] like Figure 3 FIG. 1 is a schematic diagram of an embodiment of extracting predicted coordinates of key points of a face image through a first decoding network of a neural network for face detection. Figure 3In the method, the face image sample with annotation is input into the encoding network, and after the encoding network processes the face image sample, the result is sent to the first decoding network, and the first decoding network outputs the UV coordinate map. According to the preset mapping relationship between the UV coordinate map and the three-dimensional coordinates of the key points, the predicted coordinates of the key points can be extracted. Among them, UV is the abbreviation of U, V texture map, U represents the horizontal coordinate, and V is the vertical coordinate. The UV coordinate map is composed of pixel values ​​of three RGB channels, and each pixel channel represents the coordinates of the X-axis, Y-axis and Z-axis of the three-dimensional face key points respectively. Through the two-dimensional UV coordinate map, each point on the image in the UV coordinate system can be mapped to the surface of the three-dimensional face model, that is, each point on the three-dimensional face model has a unique point on the UV coordinate map. By presetting the mapping relationship between the UV coordinate map and the three-dimensional coordinates, the key point coordinates of the face image can be restored from the UV coordinate map.

[0052] When the face image samples input by the encoding network are only annotated with the three-dimensional coordinates of key points, the predicted coordinates of the key points are valid three-dimensional coordinates. When the face image samples input by the encoding network include face images annotated with the three-dimensional coordinates of key points and face images annotated with the two-dimensional coordinates of key points, the predicted coordinates corresponding to the samples annotated with the three-dimensional coordinates are valid three-dimensional coordinates, while the predicted coordinates corresponding to the samples annotated with the two-dimensional coordinates are pseudo three-dimensional coordinates, that is, the corresponding predicted coordinates do not have Z-axis coordinate values.

[0053] The encoding network and the first decoding network in the above embodiment may be pre-trained. The first decoding network may be a network such as a standard neural network (SNN), a convolutional neural network (CNN), or a recursive neural network (RNN). The pre-training of the encoding network and the first decoding network may be implemented using existing technologies, so that the encoding network and the first decoding network can obtain the predicted coordinates of the key points based on the facial image samples marked with the key point coordinates, which will not be described in detail here.

[0054] In one embodiment, the parameters of the encoding network and the first decoding network may be determined by training. As described above, the encoding network and the first decoding network may be networks of various forms, which will not be described in detail here. Here, an exemplary embodiment in which the parameters of the encoding network and the first decoding network are determined by training is given:

[0055] The encoding network obtains a sample of a human face image with annotations, and the first decoding network extracts the predicted coordinates of the sample key points according to the output result of the encoding network. The parameters of the encoding network and the first decoding network are jointly trained based on the first loss function until the first loss function meets the preset training conditions. The network parameters at this time are the parameters of the trained encoding network and the first decoding network. The preset training condition may be that the value of the first loss function is lower than a threshold, or that the first loss function converges, or other training conditions, which are not limited in this application.

[0056] In one embodiment, the first loss function may be a loss function based on the minimum absolute deviation, a loss function based on the minimum mean square deviation, or other forms of loss functions, which are not limited in this application. The first loss function may be the error between the predicted coordinates of the key point and the labeled coordinates corresponding to the key point, or other metrics to characterize the accuracy of the predicted coordinates of the key point. An exemplary first loss function may be:

[0057]

[0058] Among them, Loss1 is the first loss function, predict is the predicted coordinates of the key points output by the first decoding network, and label is the labeled coordinates of the key points of the face image sample input by the encoding network. It should be understood that the above first loss function is only an exemplary embodiment, and those skilled in the art can also design other first loss functions to train the encoding network and the first decoding network so that the predicted coordinates of the key points output by the first decoding network are accurate.

[0059] In one embodiment, according to the output result of the encoding network, obtaining the facial posture change parameters of the key points can be achieved through the second decoding network of the neural network of face detection.

[0060] The face posture change parameters are used to characterize the changes of face posture relative to the reference face model. The face posture change parameters may include shape parameters and expression parameters, and may also include texture parameters, etc. Since human faces have many commonalities, for example, they have a specific number of eyes, mouths, noses, and ears, and the relative positions of each part remain unchanged, and each part has a certain topological relationship, a standard three-dimensional face model with parameterized characterization can be established. By obtaining the changes of the predicted coordinates of the key points relative to the standard three-dimensional face model, the face posture change parameters can be obtained.

[0061] In one embodiment, the parameterized standard 3D face model may be a parameterized 3D face model obtained based on the 3DMM (3DMorphable Models) method. This method assumes that the 3D faces have been densely aligned, that is, all 3D faces can be represented by the same number of point clouds or facets, and points with the same sequence number represent the same semantics. In the case of dense alignment, each textured 3D face can be represented as:

[0062] S model =S 2 +∑α i S i (2)

[0063] T model =T 2 +∑β i T i (3)

[0064] Among them, S and T represent the average shape and average texture of the face. The discriminant characteristics of each face are reflected in a set of orthogonal bases S on the right side of the plus sign. i or T i The linear combination of S i and T i By collecting multiple 3D human head images and using the principal component analysis (PCA) method, we can obtain the principal components representing the facial shape and texture information, namely S i and T i . Different coefficients α i and β i , representing 3D faces of different shapes and textures. Based on the data of the BFM database, the above parametric representation face model is optimized to obtain another parametric representation standard 3D face model:

[0065] M=BFM mean +shape*shape std +exp*exp std (4)

[0066] Among them, BFM mean It is the average part of the face obtained based on the BFM database, shape std and exp stdis the feature vector of face shape and face expression, shape and exp are shape parameters and expression parameters respectively. As for BFM database, it is the prior art and will not be described here. In formula (4), by determining the shape and exp parameters, the standard 3D face can be transformed to obtain a 3D face model with posture (meaning with shape change and expression change).

[0067] Of course, those skilled in the art should understand that the parameterized standard 3D face model may also be a parameterized 3D face model established by other methods, for example, a parameterized 3D face model established by using a statistical method based on actually collected 3D face data or 3D face data obtained by other methods.

[0068] In one embodiment, the facial posture change parameters of the key points of the facial image can be obtained through a pre-trained second decoding network: the encoding network sends the output result to the second decoding network, and the second decoding network obtains the facial posture change parameters based on the output result of the encoding network. The second decoding network can be a standard neural network (Standard Neural Networks, SNN), a convolutional neural network (Convolutional Neural Networks, CNN), a recursive neural network (Recursive Neural Network, RNN), etc. The pre-training of the second decoding network can be implemented using existing technologies, which will not be described in detail here.

[0069] In one embodiment, the second decoding network may be determined through training. The training process of the second decoding network includes: based on the parameters of the encoding network and the second loss function, the second decoding network is trained until the second loss function meets the preset training conditions, and the parameters of the second decoding network are determined. The parameters of the encoding network may be determined by the training method described above, or may be trained in other ways, which is not limited in this application. The preset training condition may be that the value of the second loss function is lower than a threshold, or that the first loss function converges, or other training conditions, which are not limited in this application.

[0070] In one embodiment, the second loss function may be a loss function based on the minimum absolute deviation, a loss function based on the minimum mean square deviation, or other forms of loss functions, which are not limited in this application. The second loss function may be the error between the predicted coordinates of the key points and the true coordinates of the key points, or other metrics to characterize the accuracy of the face change parameters output by the second decoding network. An exemplary second loss function may be:

[0071]

[0072] Among them, Loss2 is the second loss function, predict is the predicted coordinates of the key points output by the first decoding network, mask is the activation function corresponding to the three-dimensional coordinates of the key points, affine represents the transformation matrix of the standard three-dimensional face model relative to the predicted coordinates, and the meanings of the remaining parameters are as described above.

[0073] like Figure 4 , which is a schematic diagram of acquiring face posture change parameters of key points of a face image by training a second decoding network. Figure 4 The schematic diagram shows the influence of the second loss function on the training, which is explained in conjunction with the expression formula (5) of the second loss function. The expression in brackets {} in formula (5) represents the real coordinates of the key points of the face image sample, which are obtained in the following way: according to the predicted coordinates of the key points obtained by the first decoding network and the standard three-dimensional face model, the transformation matrix of the standard three-dimensional face model relative to the predicted coordinates is calculated, and the transformation matrix represents the rotation, translation and scaling corresponding to the transformation of the standard three-dimensional face model to the three-dimensional face model corresponding to the predicted coordinates; the standard three-dimensional face model is transformed according to the transformation matrix to obtain the real coordinates of the key points. Among them, the standard three-dimensional face model is transformed according to the transformation matrix to obtain the real coordinates of the key points, and the parameters of the second decoding network are adjusted through the transformation matrix, so that the facial posture change parameters output by the second decoding network change, so that the posture of the standard three-dimensional face model changes. Taking the standard 3D face model represented by parameter in formula (4) as an example, the shape and exp parameters in formula (4) are changed by changing the parameters of the second decoding network, thereby changing the posture of the standard 3D face model represented by parameter. The parameterized expression is: M2 = {(bfm mean +shape*shape std +exp*exp std )*affine}, which is the content in the brackets in formula (5).

[0074] The second decoding network is trained based on the second loss function, that is, the parameters of the second decoding network are continuously adjusted to obtain the facial posture change parameters output by the second decoding network, and the standard three-dimensional face model is transformed until the standard three-dimensional face model is aligned with the predicted key points output by the first decoding network (that is, the error is small enough), and a three-dimensional face model with the same or similar predicted coordinate posture as the key points is obtained. The second decoding network completes the training, and the result output by the trained second decoding network is the facial posture change parameters.

[0075] Based on the facial posture change parameters output by the trained second decoding network, a 3D face model corresponding to the predicted coordinates of the key points of the input sample can be obtained. Based on the 3D face model, the 3D coordinates of any point on the model can be obtained. Therefore, the 3D coordinates of a large number of key points of the sample can be obtained.

[0076] In one embodiment, the parameters of the prediction network are determined based on the parameters determined by the encoding network and the third loss function training.

[0077] like Figure 5 As shown, according to the processing results of the face image samples output by the encoding network, combined with the obtained prediction coordinates and face posture change parameters, the parameters of the prediction network are trained based on the third loss function until the third loss function meets the preset training conditions. The network parameters at this time are the parameters of the trained prediction network, that is, the prediction network completes the training. The training of the prediction network is based on the third loss function, and the preset training conditions can be that the value of the third loss function is lower than the threshold, or the third loss function converges, or other training conditions, which are not limited in this application.

[0078] In one embodiment, the third loss function may be in the form of a loss function based on the minimum absolute deviation, or a loss function based on the minimum mean square deviation, or other forms of loss functions, which are not limited in this application. The content of the third loss function may be the error between the three-dimensional coordinates of the key points output by the prediction network and the true coordinates, or other metrics to characterize the accuracy of the three-dimensional coordinates of the key points output by the prediction network. An exemplary third loss function may be:

[0079]

[0080] Among them, Loss3 is the third loss function, predict pts To predict the three-dimensional coordinates of the key points output by the network, the meanings of the other parameters are as described above. As mentioned above, the expression in brackets {} in formula (5) actually represents the real coordinates of the key points. The acquisition method is the same as described above and will not be repeated here.

[0081] Based on the trained prediction network, it is possible to input an unlabeled face image into the encoding network. The encoding network sends the processing results to the prediction network, and the prediction network outputs the three-dimensional coordinates of the key points of the face image.

[0082] Those skilled in the art should understand that in order to make the trained model have better generalization ability and more accurate prediction results, while preventing the model from overfitting, the first, second and third loss functions may also add regularization terms to improve the overall performance of the trained model.

[0083] In one embodiment, the neural network of the face detection includes an encoding network, a first decoding network, a second decoding network and a prediction network. The encoding network, the first decoding network and the second decoding network can be pre-trained networks, or part of the network can be a trained network and part of the network can be an untrained network. For the specific training method of the untrained network and the designed loss function, please refer to the above description and will not be repeated here.

[0084] In one embodiment, the encoding network, the first decoding network, the second decoding network and the prediction network included in the neural network of the face detection may all be untrained networks. The following method may be used for the training order of each network:

[0085] Fixing preset parameters of the second decoding network and the prediction network, and jointly training the parameters of the encoding network and the first decoding network based on the first loss function;

[0086] The trained parameters of the encoding network are fixed, and the parameters of the second decoding network are trained based on the second loss function;

[0087] The trained parameters of the encoding network are fixed, and the parameters of the prediction network are trained based on the third function.

[0088] For the specific training methods and designed loss functions of each network, please refer to the previous article and will not be repeated here.

[0089] In the above embodiment, the face image sample annotated with the three-dimensional coordinates of key points is used as the input of the neural network for face detection, and the neural network is trained by obtaining the predicted coordinates of the key points of the sample and the parameters of the face posture change. By using the neural network for face detection, a three-dimensional face model corresponding to the predicted coordinates of the key points of the input sample can be obtained, and then the three-dimensional coordinates of a large number of key points of the sample can be obtained by inputting a face image sample annotated with only a small number of three-dimensional coordinates of key points into the neural network. In addition, based on the trained encoding network and prediction network, it is possible to input an unannotated face image into the encoding network, and the encoding network sends the processing results to the prediction network, and the prediction network outputs the three-dimensional coordinates of the key points of the face image, thereby realizing the detection of the three-dimensional coordinates of the key points of the unannotated face image.

[0090] like Figure 6 As shown, Figure 6 This is a flowchart of a neural network training method for face detection according to an exemplary embodiment of the present specification, comprising the following steps:

[0091] In step 602, a sample of a face image is obtained, wherein the sample includes a face image with three-dimensional coordinates of key points annotated;

[0092] In step 604, the sample is used as input of the detection model to obtain predicted coordinates of key points and facial posture change parameters;

[0093] In step 606, the detection model is trained based on the acquired predicted coordinates and facial posture change parameters so that the output result of the trained detection model is a three-dimensional coordinate containing a first number of key points, wherein the first number is greater than the number of key points marked with three-dimensional coordinates in the sample.

[0094] The above training method can be used in various neural networks that need to be trained based on the predicted coordinates of key points and face posture change parameters to obtain the three-dimensional coordinates of key points of face images. The present application does not limit the structure of the network trained using the training method. Of course, those skilled in the art should understand that the method can also be used in the training of the neural network for face detection described above.

[0095] In one embodiment, the encoding network, the first decoding network, and the second decoding network can be pre-trained networks, and the encoding network directly outputs the processing result of the input face image sample and sends it to the first decoding network, the second decoding network, and the prediction network; the first decoding network directly obtains the predicted coordinates of the key points, and the second decoding network directly obtains the face posture change parameters, and only the prediction network is trained. The specific training method of the prediction network has been described in detail above and will not be repeated here.

[0096] In one embodiment, the encoding network, the first decoding network, and the second decoding network may be untrained networks, and the training method described above may be used to train the encoding network, the first decoding network, the second decoding network, and the prediction network. The specific training method of the network has been described in detail above and will not be repeated here.

[0097] In addition, those skilled in the art should understand that the predicted coordinates of key points and facial posture change parameters can also be obtained through the above-mentioned training method. The acquisition method has been described in detail in the previous article and will not be repeated here.

[0098] When the neural network for face detection is trained, a parameter-determined encoding network and prediction network can be obtained. Based on the parameter-determined encoding network and prediction network, a face detection method can be implemented: the encoding network obtains an unlabeled face image; the prediction network outputs the three-dimensional coordinates of the key points of the face image. Based on the obtained three-dimensional coordinates of the key points of the face image, three-dimensional face reconstruction can be performed, such as Figure 7 shown.

[0099] The above-mentioned face detection method can be installed on a server to detect key points of a face image, or the trained encoding network and the first decoding network can be installed on a mobile terminal to realize face detection on the mobile terminal.

[0100] In this embodiment, by using the trained face model, it is possible to input an unlabeled face image into the encoding network and the prediction network, and obtain the three-dimensional coordinates of the key points of the face image. Based on the three-dimensional coordinates of the key points, the three-dimensional face can be further reconstructed.

[0101] Accordingly, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0102] Accordingly, the present application also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the program implements the steps of any of the above methods when executed by the processor.

[0103] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0104] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0105] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0106] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0107] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A neural network for face detection, characterized in that: The neural network comprises: encoding network, first decoding network, second decoding network and prediction network, The encoding network is used to obtain samples of face images and distribute the output results to the first decoding network, the second decoding network and the prediction network, wherein the samples include face images with three-dimensional coordinates of key points annotated; The first decoding network is used to obtain a UV coordinate map of the sample key points according to the output result of the encoding network, and extract the predicted coordinates of the sample key points based on the mapping relationship between the UV coordinate map and the three-dimensional coordinates; The second decoding network is used to obtain the face posture change parameters of the predicted coordinates of the sample key points according to the output result of the encoding network, including: According to the predicted coordinates of the key points obtained by the first decoding network and the standard three-dimensional face model, a transformation matrix of the standard three-dimensional face model relative to the predicted coordinates is obtained; The standard 3D face model is transformed according to the transformation matrix to obtain the real coordinates of the key points; According to the predicted coordinates of the key points and the real coordinates of the key points, the parameters of the second decoding network are trained based on the second loss function to determine the parameters of the second decoding network; The facial posture change parameters of the key points output by the trained second decoding network; The prediction network is used to train the output results of the encoding network and the acquired prediction coordinates and facial posture change parameters as input, and output a facial model containing the three-dimensional coordinates of the sample key points, and the number of key points outputted is greater than the number of key points of the three-dimensional coordinates marked in the sample.

2. The neural network according to claim 1, characterized in that The face image samples also include face images with two-dimensional coordinates of key points marked.

3. The neural network according to claim 1, characterized in that The encoding network is a lightweight neural network.

4. The neural network according to claim 1, characterized in that The parameters of the encoding network and the first decoding network are obtained in the following manner: The encoding network and the first decoding network are jointly trained based on a first loss function to determine parameters of the encoding network and parameters of the first decoding network.

5. The neural network according to claim 4, characterized in that The parameters of the second decoding network are obtained by: Based on the determined parameters of the encoding network and the second loss function, the second decoding network is trained to determine the parameters of the second decoding network.

6. The neural network according to claim 4, characterized in that The parameters of the prediction network are determined based on the parameters determined by the encoding network and the third loss function training.

7. The neural network according to claim 4, characterized in that The first loss function is the error between the predicted coordinates of the sample key points and the marked coordinates of the sample key points.

8. The neural network according to claim 1, characterized in that The standard three-dimensional face model is a parameterized three-dimensional face model obtained based on the 3DMM method.

9. The neural network according to claim 1, characterized in that The second loss function is the error between the predicted coordinates and the true coordinates of the key points.

10. The neural network according to claim 6, characterized in that The third loss function is the error between the three-dimensional coordinates of the key points output by the prediction network and the true coordinates.

11. A method for training a neural network for face detection, characterized in that: The training method comprises: Acquire a sample of a face image, wherein the sample includes a face image with three-dimensional coordinates of key points annotated; Using the sample as input of the neural network to obtain predicted coordinates of key points and facial posture change parameters; Training the neural network based on the acquired predicted coordinates and facial posture change parameters so that the trained neural network outputs three-dimensional coordinates containing a first number of key points, wherein the first number is greater than the number of key points annotated with the three-dimensional coordinates in the sample; The neural network for face detection includes an encoding network, a first decoding network, and a second decoding network. The encoding network is used to obtain samples of face images and distribute the output results to the first decoding network. The first decoding network is used to take the samples as input and obtain a UV coordinate map of the key points of the samples according to the output results of the encoding network; based on the mapping relationship between the UV coordinate map and the three-dimensional coordinates, the predicted coordinates of the key points of the samples are extracted; The second decoding network is used to obtain face posture change parameters based on the output result of the encoding network, including: obtaining a transformation matrix of the standard three-dimensional face model relative to the predicted coordinates according to the predicted coordinates of the key points obtained by the first decoding network and the standard three-dimensional face model; The standard 3D face model is transformed according to the transformation matrix to obtain the real coordinates of the key points; According to the predicted coordinates of the key points and the real coordinates of the key points, the parameters of the second decoding network are trained based on the second loss function to determine the parameters of the second decoding network; The trained second decoding network outputs the facial posture change parameters of the key points.

12. The training method according to claim 11, characterized in that: The training method further comprises: The encoding network and the first decoding network are jointly trained based on a first loss function to determine parameters of the encoding network and parameters of the first decoding network.

13. The training method according to claim 12, characterized in that: The training method further comprises: Based on the determined parameters of the encoding network and the second loss function, the second decoding network is trained to determine the parameters of the second decoding network.

14. The training method according to claim 13, characterized in that: The neural network for face detection further includes a prediction network, which is used to output three-dimensional coordinates containing a first number of key points based on the predicted coordinates of the key points and the face posture change parameters. The training method further includes: Based on the determined parameters of the encoding network and the third loss function, the prediction network is trained to determine the parameters of the prediction network.

15. A method for face detection, characterized in that: The method is implemented by the encoding network according to any one of claims 1 to 10 and the prediction network obtained by training, and the method comprises: The encoding network acquires an unlabeled face image for processing and outputs the processing result to the prediction network; The prediction network outputs the three-dimensional coordinates of key points of the face image.

16. The method for face detection according to claim 15, characterized in that: The encoding network and the trained prediction network are installed on a mobile terminal.

17. A computer storage medium having computer program code stored therein, characterized in that: When the computer program code is executed on a processor, the processor is caused to perform the training of the neural network of claim 1 or the steps of the method of claim 15.

18. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to claim 15 are implemented.

Images