Face key point detection model training method, live video image processing method and device

Abstract

The present application relates to the field of live broadcast and image processing technology, and provides a facial key point detection model training method, a live broadcast image processing method, an apparatus, a device and a medium. The present application can take into account both the accuracy and efficiency of facial key point detection. It includes: obtaining an average position based on the labeled position of each facial key point on each facial image in the data set, determining the principal component set of the facial images in the data set and obtaining the fitting coefficient of each image in each principal component, inputting the image into the facial key point detection model to be trained, obtaining the transformed facial image through a first spatial transformation network, obtaining its predicted fitting coefficient in each principal component through a coefficient prediction network, obtaining the predicted position of each facial key point based on the predicted fitting coefficient, each principal component and the average position, obtaining the predicted position of each facial key point on the facial image through a second spatial transformation network, and training the model based on a first loss that characterizes the consistency of the fitting coefficient and a second loss that characterizes the consistency of the position.

Description

Technical Field

[0001] The present application relates to the technical field of webcasting and image processing, and in particular to a method for training a facial key point detection model, a method, device, electronic device, and computer-readable storage medium for processing facial images for webcasting. Background Art

[0002] With the development of live streaming technology, image processing techniques are being applied to live streaming. For example, special effects such as beautification, makeup, and plastic surgery can be applied to facial images to enhance the dissemination of high-quality content shared during live streaming. Facial landmark detection algorithms are the foundation of these special effects. With the rapid development of deep learning, the positioning accuracy of facial landmark detection will continue to increase.

[0003] The deep learning-based facial key point detection algorithm needs to be trained on a large number of manually annotated facial image datasets. The current technology provides a method of integrating multiple datasets for model training and application, which generally requires a large amount of computation and is difficult to meet the real-time requirements of facial key point detection in fields such as live streaming. Summary of the Invention

[0004] Based on this, it is necessary to provide a training method for a facial key point detection model, a facial image processing method, device, electronic device and computer-readable storage medium for live broadcasting to address the above technical problems.

[0005] In a first aspect, the present application provides a method for training a facial key point detection model. The method comprises:

[0006] Obtaining the average position of each facial key point based on the marked position of each facial key point on each facial image in the facial image dataset;

[0007] Based on principal component analysis and annotated positions, a principal component set of facial images in the facial image dataset is determined, and a fitting coefficient corresponding to each principal component of each facial image is obtained; different principal components in the principal component set correspond to different morphological change dimensions of the face;

[0008] Inputting the facial image into a facial key point detection model to be trained, so that the facial key point detection model to be trained obtains a transformed facial image based on the facial image through a first spatial transformation network, obtains prediction fitting coefficients corresponding to each principal component based on the transformed facial image through a coefficient prediction network, obtains predicted positions of each facial key point on the transformed facial image based on the prediction fitting coefficients, each principal component, and an average position, and obtains predicted positions of each facial key point on the facial image based on the predicted positions of each facial key point on the transformed facial image through a second spatial transformation network;

[0009] Obtaining a first model loss representing the consistency between the predicted fitting coefficient and the fitting coefficient, and obtaining a second model loss representing the consistency between the predicted position and the marked position of each facial key point;

[0010] The facial key point detection model to be trained is trained according to the first model loss and the second model loss.

[0011] In one embodiment, obtaining the predicted position of each facial key point on the transformed facial image based on the predicted fitting coefficient, each principal component and the average position includes: obtaining the predicted position change of each facial key point on the transformed facial image based on the predicted fitting coefficient corresponding to each principal component of the transformed facial image and the principal components; obtaining the predicted position of each facial key point on the transformed facial image based on the predicted position change and the average position.

[0012] In one embodiment, determining the principal component set of facial images in the facial image dataset includes: performing a similarity transformation on the labeled positions of each facial key point on each facial image based on the average position of each facial key point to obtain the transformed position of each facial key point on each facial image; and performing principal component analysis based on the transformed position of each facial key point on each facial image to obtain the principal component set.

[0013] In one embodiment, performing principal component analysis based on the transformed positions of each facial key point on each facial image to obtain the principal component set includes: normalizing the transformed positions of each facial key point on each facial image relative to the center of the facial image to obtain the normalized transformed positions of each facial key point on each facial image; performing principal component analysis based on the normalized transformed positions of each facial key point on each facial image to obtain the principal component set.

[0014] In one embodiment, obtaining the fitting coefficient corresponding to each principal component of each facial image includes: fitting the principal component set using the normalized transformed position of each facial key point on each facial image to obtain the fitting coefficient corresponding to each principal component of each facial image.

[0015] In one embodiment, obtaining the first model loss characterizing the consistency between the predicted fitting coefficient and the fitting coefficient includes: obtaining the first model loss according to the difference between the predicted fitting coefficient and the fitting coefficient.

[0016] In one embodiment, the obtaining of the second model loss that characterizes the consistency between the predicted position and the annotated position of each facial key point includes: obtaining the difference between the predicted position and the annotated position corresponding to each facial key point, obtaining a mean of the differences based on the difference between the predicted position and the annotated position corresponding to each facial key point, and obtaining the second model loss based on the mean of the differences.

[0017] In a second aspect, the present application provides a method for processing facial images in live webcasts. The method comprises:

[0018] Obtain the face image to be processed in the live broadcast;

[0019] The facial key points on the face image to be processed are obtained by detecting the facial key point detection model trained by the training method of the facial key point detection model described above.

[0020] Applying special effects to the facial image to be processed based on each facial key point on the facial image to be processed.

[0021] In a third aspect, the present application provides a training device for a facial key point detection model. The device comprises:

[0022] A position acquisition module is used to obtain the average position of each facial key point based on the marked position of each facial key point on each facial image in the facial image dataset;

[0023] A principal component analysis module is configured to determine a principal component set of facial images in the facial image dataset based on principal component analysis and annotated positions, and obtain fitting coefficients corresponding to each principal component of each facial image; different principal components in the principal component set correspond to different morphological variation dimensions of the face;

[0024] An image input module, configured to input the facial image into a facial key point detection model to be trained, so that the facial key point detection model to be trained obtains a transformed facial image based on the facial image through a first spatial transformation network, obtains prediction fitting coefficients corresponding to each principal component based on the transformed facial image through a coefficient prediction network, obtains a predicted position of each facial key point on the transformed facial image based on the prediction fitting coefficients, each principal component, and an average position, and obtains a predicted position of each facial key point on the facial image based on the predicted position of each facial key point on the transformed facial image through a second spatial transformation network;

[0025] A loss acquisition module is used to acquire a first model loss representing the consistency between the predicted fitting coefficient and the fitting coefficient, and to acquire a second model loss representing the consistency between the predicted position and the marked position of each facial key point;

[0026] A model training module is used to train the facial key point detection model to be trained based on the first model loss and the second model loss.

[0027] In a fourth aspect, the present application provides a facial image processing device for live webcasting. The device comprises:

[0028] An image acquisition module is used to acquire the face image to be processed in the live broadcast;

[0029] A key point detection module, configured to detect and obtain facial key points on the face image to be processed using a trained facial key point detection model; the facial key point detection model is trained using the training device for the facial key point detection model described above;

[0030] The image processing module is used to apply special effects to the face image to be processed based on each facial key point on the face image to be processed.

[0031] In a fifth aspect, the present application provides an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] According to the annotated positions of each facial key point on each facial image in the facial image data set, the average position of each facial key point is obtained; based on principal component analysis and the annotated positions, the principal component set of the facial images in the facial image data set is determined, and the fitting coefficient corresponding to each principal component of each facial image is obtained; different principal components in the principal component set correspond to different morphological change dimensions of the face; the facial image is input into the facial key point detection model to be trained, so that the facial key point detection model to be trained obtains a transformed facial image according to the facial image through a first spatial transformation network, and obtains a transformed facial image according to the transformed facial image through a coefficient prediction network. Obtain a predicted fitting coefficient corresponding to each principal component, obtain a predicted position of each facial key point on the transformed facial image according to the predicted fitting coefficient, each principal component and the average position, obtain the predicted position of each facial key point on the facial image according to the predicted position of each facial key point on the transformed facial image through a second spatial transformation network; obtain a first model loss that characterizes the consistency between the predicted fitting coefficient and the fitting coefficient, and obtain a second model loss that characterizes the consistency between the predicted position of each facial key point and the labeled position; train the facial key point detection model to be trained according to the first model loss and the second model loss.

[0033] In a sixth aspect, the present application provides an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire a facial image to be processed in a live webcast; obtain facial key points on the facial image to be processed by detecting using a trained facial key point detection model; the facial key point detection model is trained according to the training method of the facial key point detection model described above; and apply special effects to the facial image to be processed based on the facial key points on the facial image to be processed.

[0035] In a seventh aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0036] According to the annotated positions of each facial key point on each facial image in the facial image data set, the average position of each facial key point is obtained; based on principal component analysis and the annotated positions, the principal component set of the facial images in the facial image data set is determined, and the fitting coefficient corresponding to each principal component of each facial image is obtained; different principal components in the principal component set correspond to different morphological change dimensions of the face; the facial image is input into the facial key point detection model to be trained, so that the facial key point detection model to be trained obtains a transformed facial image according to the facial image through a first spatial transformation network, and obtains a transformed facial image according to the transformed facial image through a coefficient prediction network. Obtain a predicted fitting coefficient corresponding to each principal component, obtain a predicted position of each facial key point on the transformed facial image according to the predicted fitting coefficient, each principal component and the average position, obtain the predicted position of each facial key point on the facial image according to the predicted position of each facial key point on the transformed facial image through a second spatial transformation network; obtain a first model loss that characterizes the consistency between the predicted fitting coefficient and the fitting coefficient, and obtain a second model loss that characterizes the consistency between the predicted position of each facial key point and the labeled position; train the facial key point detection model to be trained according to the first model loss and the second model loss.

[0037] In an eighth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0038] Acquire a facial image to be processed in a live webcast; obtain facial key points on the facial image to be processed by detecting using a trained facial key point detection model; the facial key point detection model is trained according to the training method of the facial key point detection model described above; and apply special effects to the facial image to be processed based on the facial key points on the facial image to be processed.

[0039] The above-mentioned training method of the facial key point detection model, the facial image processing method, device, equipment and medium for live broadcast on the Internet obtain the average position of each facial key point based on the annotated position of each facial key point on each facial image in the facial image data set, determine the principal component set of the facial images in the data set based on principal component analysis and the annotated position, and obtain the fitting coefficient corresponding to each principal component of each facial image. Different principal components in the principal component set correspond to different morphological change dimensions of the face, and then input the facial image into the facial key point detection model to be trained. The model obtains the transformed facial image through a first spatial transformation network, obtains the predicted fitting coefficient corresponding to each principal component of the transformed facial image through a coefficient prediction network, then obtains the predicted position of each facial key point on the transformed facial image based on the predicted fitting coefficient, each principal component and the average position, and then obtains the predicted position of each facial key point on the facial image based on the predicted position through a second spatial transformation network. The model is trained based on a first model loss that characterizes the consistency between the predicted fitting coefficient and the fitting coefficient and a second model loss that characterizes the consistency between the predicted position and the annotated position. This solution is based on the principal components obtained by principal component analysis of the face image dataset and the fitting coefficients corresponding to each principal component of each face image. Combined with the labeled positions and corresponding average positions of each facial key point on each face image in the dataset, the face key point detection model is trained so that it can accurately predict the fitting coefficients of the face image and based on this, the positions of each facial key point on the face image can be accurately predicted. The model structure is simple and efficient, and can take into account both the accuracy and efficiency of face key point detection. It can achieve accurate detection of face key points while meeting the real-time requirements of face key point detection in fields such as live streaming. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is an application scenario diagram of the relevant method in the embodiments of this application;

[0041] Figure 2 Schematic diagram of the process of training a facial key point detection model in an embodiment of the present application;

[0042] Figure 3 Schematic diagram of the average position of key points of each face in an embodiment of the present application;

[0043] Figure 4 Schematic diagram of the relationship between the principal components and key points of the face in the embodiment of this application;

[0044] Figure 5 Schematic diagram of similarity transformation in the embodiment of the present application;

[0045] Figure 6 Schematic diagram of the process of processing facial images by the facial key point detection model in an embodiment of the present application;

[0046] Figure 7 Schematic diagram of the process of the face image processing method for live webcasting in an embodiment of the present application;

[0047] Figure 8 This is a structural block diagram of a training device for a facial key point detection model according to an embodiment of the present application;

[0048] Figure 9 This is a structural block diagram of a face image processing device for live broadcasting in an embodiment of the present application;

[0049] Figure 10 This is a diagram of the internal structure of an electronic device in an embodiment of the present application;

[0050] Figure 11 This is a diagram of the internal structure of an electronic device in another embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] The training method of the face key point detection model and the face image processing method of the live broadcast provided in the embodiment of the present application can be applied to Figure 1 In the illustrated application scenario, the application scenario may include a terminal 110 and a server 120, wherein the terminal 110 communicates with the server 120 via a network. The terminal 110 may be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers, and the server 120 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0053] The following part is Figure 1 Based on the application scenarios shown, combined with various embodiments and related drawings, the training method of the facial key point detection model and the face image processing method for live broadcast of the present application are explained in sequence.

[0054] In one embodiment, Figure 2 As shown, the present application provides a method for training a facial key point detection model, which can be executed by the server 120 and can include the following steps:

[0055] Step S201 : obtaining the average position of each facial key point according to the marked position of each facial key point on each facial image in the facial image dataset.

[0056] Specifically, a facial image dataset may include multiple facial images and may also include the annotated locations of each facial key point on the multiple facial images, denoted as annotated locations. The multiple facial images are images containing faces, and the annotated locations of each facial key point on these facial images can be obtained through manual annotation. That is, the locations of each facial key point on the multiple facial images can be manually annotated to obtain the annotated locations. Once the annotation is completed, a facial image dataset can be obtained. The annotated locations can be represented by coordinates on the facial images. In a practical scenario, a relevant person can annotate the locations of each facial key point on terminal 110 to obtain a facial image dataset. Terminal 110 then transmits the facial image dataset to server 120 via a network, and server 120 obtains the facial image dataset. In the facial image dataset obtained by server 120, each facial image has the same number of facial key points. The total number of facial images included in the facial image dataset is denoted as N, and each facial image has L facial key points, where L is the total number of key points expected for a face. In this step, the server 120 may calculate the average position of each of the L facial key points based on the marked positions of the facial key points on each facial image in the facial image dataset.

[0057] Specifically, let the set of average positions of L facial key points be M, and the effect of marking the set M in the form of points on a blank image is as follows: Figure 3 As shown, the set M also contains the position information of L facial key points. In the set M, the average position of the i-th facial key point can be expressed as:

[0058]

[0059] in, Represents the labeled position of the i-th facial key point on the j-th face image.

[0060] Step S202 : Based on principal component analysis and the marked positions, a principal component set of the facial images in the facial image dataset is determined, and a fitting coefficient corresponding to each principal component of each facial image is obtained.

[0061] In this step, the server 120 determines the principal component set of the face images in the face image data set based on the principal component analysis and the annotated positions of each facial key point on each face image in the face image data set, and obtains the fitting coefficient corresponding to each principal component of each face image. The different principal components in the principal component set correspond to different morphological change dimensions of the face. Specifically, this step can perform principal component analysis (PCA, Principal Component Analysis) based on the coordinates of the annotated positions of each facial key point on each face image in the face image data set to obtain the principal component set. The principal component set contains multiple principal components, and different principal components correspond to different morphological change dimensions of the face. If the total number of facial key points is L, the position of each facial key point can be expressed as two coordinate values (x, y), then each principal component obtained by the principal component analysis is a 2*L dimensional vector, and the total number of principal components in the extracted principal component set is at most 2*L, and each principal component corresponds to a different morphological change dimension of the face, or each principal component controls different morphological change attributes of the face. In this regard, combined with Figure 4 To explain, Figure 4 The figure shows the position changes of each facial key point when different coefficients (+0.5, -0.5) are given to the first five principal components in the principal component set (which is reflected as the overall morphological change of the face). Among them, the first principal component can correspond to the morphological change dimension of the left and right rotation of the face, the second principal component can correspond to the morphological change dimension of the face looking up and down, the third principal component can correspond to the morphological change dimension of the face being fat or thin, and so on. It can be seen that each principal component can be used to control different morphological change attributes of the face. Then, in this step, the server 120 can also fit the obtained principal component set based on the annotated position of each facial key point on each face image in the face image dataset, thereby obtaining the fitting coefficient corresponding to each principal component of each face image in the face image dataset. In a specific implementation, the principal component set can be fitted based on the annotated position of each facial key point on a face image through a standard PCA process to obtain the fitting coefficients corresponding to each principal component of the principal component set of the face image. Among them, the total number of face images in the face image dataset is N, and the number of principal components in the obtained principal component set is P, then the PCA coefficient matrix can be fitted. The matrix dimension of the PCA coefficient matrix is NxP (N rows and P columns), and the matrix elements in the PCA coefficient matrix are the fitting coefficients corresponding to the corresponding principal components.

[0062] Furthermore, in some embodiments, determining the principal component set of the facial images in the facial image dataset in step S202 may include:

[0063] Based on the average position of each facial key point, a similarity transformation is performed on the marked position of each facial key point on each facial image to obtain the transformed position of each facial key point on each facial image; principal component analysis is performed based on the transformed position of each facial key point on each facial image to obtain the principal component set.

[0064] In this embodiment, the server 120 can perform a similarity transformation on the annotated position of each facial key point on each facial image in the facial image data set based on the average position of each facial key point, thereby obtaining the annotated position of each facial key point on each facial image after the similarity transformation, recorded as the transformed position, and then a principal component analysis can be performed based on the coordinates of the transformed position of each facial key point on each facial image to obtain a principal component set, so as to improve the accuracy and reliability of the trained model in detecting facial key points. Regarding the processing of similarity transformation, in a specific implementation, for all facial images in the facial image data set, the server 120 can calculate a similarity transformation matrix based on the position of each manually annotated facial key point, that is, the annotated position of each facial key point and the set M of the average position of the above-mentioned facial key points, such as Figure 5 As shown, the server 120 can straighten each facial image in the facial image dataset according to the similarity transformation matrix, and can also straighten each facial key point (marked position) on each facial image according to the similarity transformation matrix, thereby obtaining the transformed position of each facial key point on each facial image.

[0065] Based on this, in one embodiment, the principal component analysis in the above embodiment is performed based on the transformed positions of each facial key point on each facial image to obtain a principal component set, further comprising:

[0066] The transformed position of each facial key point on each facial image is normalized relative to the center of the facial image to obtain the normalized transformed position of each facial key point on each facial image; principal component analysis is performed based on the normalized transformed position of each facial key point on each facial image to obtain a principal component set.

[0067] In this embodiment, server 120 may perform a similarity transformation on the annotated positions of each facial key point on each facial image according to the aforementioned similarity transformation method to obtain the transformed positions of each facial key point on each facial image. Based on this, server 120 may normalize the transformed positions of each facial key point on each facial image to obtain the normalized transformed positions of each facial key point on each facial image. Server 120 may then perform principal component analysis based on the normalized transformed positions of each facial key point on each facial image to obtain a principal component set, thereby further improving the accuracy and reliability of facial key point detection by the trained model. Regarding the normalization process, in a specific implementation, server 120 may normalize the transformed positions of each facial key point on each facial image relative to the center of the facial image so that their position coordinates range from -1 to 1, thereby obtaining the normalized transformed positions of each facial key point on each facial image. As an example, suppose the width and height of the face image are 100 and 200 respectively, and the coordinates of the transformed position of the facial key point on it are (50, 200), then the normalized transformed position obtained by normalizing it relative to the center of the face image is (0, 1).

[0068] Based on the above embodiment, in one embodiment, obtaining the fitting coefficients corresponding to each principal component for each facial image in step S202 may further include: fitting the principal component set using the normalized transformed positions of each facial key point on each facial image to obtain the fitting coefficients corresponding to each principal component for each facial image. In this embodiment, specifically, the server 120 may use the coordinates of the normalized transformed positions of each facial key point on each facial image obtained after the above normalization process to fit the principal component set obtained in the above step to obtain the fitting coefficients corresponding to each principal component for each facial image in the facial image dataset, thereby improving the accuracy and reliability of facial key point detection by the trained model.

[0069] Step S203: Input the face image into the face key point detection model to be trained, so that the face key point detection model to be trained obtains a transformed face image based on the face image through the first spatial transformation network, obtains the predicted fitting coefficients corresponding to each principal component based on the transformed face image through the coefficient prediction network, obtains the predicted position of each face key point on the transformed face image based on the predicted fitting coefficients and each principal component and the average position, and obtains the predicted position of each face key point on the face image based on the predicted position of each face key point on the transformed face image through the second spatial transformation network.

[0070] This step is the process in which the server 120 inputs the face image into the face key point detection model to be trained and the model performs relevant processing based on the input face image. Figure 6 To illustrate this, Figure 6As shown, the server 120 first inputs the face image into the face key point detection model to be trained. The face key point detection model may include two spatial transformer networks (STN, Spatial Transformer Network), which are respectively recorded as the first spatial transformer network and the second spatial transformer network. The first spatial transformer network can be used to perform transformation processing related to the straightening of the face image, and the second spatial transformer network can be used to perform inverse transformation processing corresponding to the aforementioned straightening on the predicted face key point positions, so that the predicted face key point positions can be mapped back to the face image; the face key point detection model also includes a coefficient prediction network, which can be implemented based on the ResNet-18 model structure, which can extract image features and finally use a fully connected layer to predict the predicted fitting coefficients corresponding to each principal component of the transformed face image. Based on this, after the server 120 inputs the face image into the face key point detection model to be trained, the first spatial transformation network first obtains the transformed face image based on the face image, and then the transformed face image is passed to the coefficient prediction network, and the coefficient prediction network obtains the predicted fitting coefficients corresponding to the aforementioned principal components based on the transformed face image, and then the model can obtain the predicted positions of each facial key point on the transformed face image based on the predicted fitting coefficients and the average positions of the aforementioned principal components and the aforementioned facial key points, and input the predicted positions of each facial key point on the transformed face image into the second spatial transformation network, and the second spatial transformation network obtains the predicted positions of each facial key point on the face image input into the model based on the predicted positions of each facial key point on the transformed face image.

[0071] In one embodiment, obtaining the predicted position of each facial key point on the transformed facial image based on the predicted fitting coefficients, each principal component, and the average position in step S203 specifically includes:

[0072] According to the predicted fitting coefficients corresponding to the principal components of the transformed face image and each principal component, the predicted position changes of each facial key point on the transformed face image are obtained; according to the predicted position changes and the average position, the predicted position of each facial key point on the transformed face image is obtained.

[0073] In this embodiment, after the server 120 inputs the face image into the face key point detection model to be trained, it will be processed in sequence through the first spatial transformation network and the coefficient prediction network. The coefficient prediction network outputs the predicted fitting coefficients of the transformed face image corresponding to each principal component. Then, the server 120 can obtain the predicted position changes of each face key point on the transformed face image based on the predicted fitting coefficients corresponding to each principal component and each principal component of the transformed face image. The predicted position change can be the position change information of each face key point on the transformed face image relative to the average position of each face key point predicted by the model or called position offset information. Therefore, the server 120 can obtain the predicted position of each face key point on the transformed face image based on the predicted position change and the average position of each face key point, so as to achieve the purpose of efficiently and accurately predicting the position of each face key point on the transformed face image. Specifically, combined with Figure 6 After obtaining the predicted fitting coefficients of the transformed face image corresponding to each principal component output by the coefficient prediction network, the model can multiply the predicted fitting coefficients of the transformed face image corresponding to each principal component with the corresponding principal components and sum them up as the predicted position changes of each facial key point on the transformed face image, and then add the predicted position changes of each facial key point on the transformed face image with the average position of each facial key point to obtain the predicted position of each facial key point on the transformed face image.

[0074] Step S204: obtaining a first model loss that characterizes the consistency between the predicted fitting coefficient and the fitting coefficient, and obtaining a second model loss that characterizes the consistency between the predicted position and the annotated position of each facial key point.

[0075] Step S205: training the facial key point detection model to be trained according to the first model loss and the second model loss.

[0076] Steps S204 and S205 are steps in which the server 120 obtains the model loss and trains the facial key point detection model to be trained based on the model loss. The model loss can be used to calculate the gradient backpropagation based on the model loss to update the network parameters of the first spatial transformation network, the coefficient prediction network, and the second spatial transformation network in the model, thereby training the model. The model loss used for model training may include a first model loss and a second model loss. The first model loss is used to characterize the consistency between the predicted fitting coefficients predicted by the model and the fitting coefficients obtained in step S202, and the second model loss is used to characterize the consistency between the predicted positions of each facial key point predicted by the model and the manually annotated positions.

[0077] Specifically, for the first model loss L1, in one embodiment, obtaining the first model loss that characterizes the consistency between the predicted fitting coefficient and the fitting coefficient in step S204 may include: obtaining the first model loss according to the difference between the predicted fitting coefficient and the fitting coefficient. Among them, the absolute value of the difference between the predicted fitting coefficient and the fitting coefficient can be used to quantify the consistency between the predicted fitting coefficient and the fitting coefficient more simply and accurately. Therefore, in this embodiment, the absolute value of the difference between the predicted fitting coefficient and the fitting coefficient can be used as the first model loss L1. Specifically, it can be expressed by the following formula: L1 = |S gt -S pred |, where S gt represents the fitting coefficient obtained in step S202 above, S pred Represents the predicted fitting coefficient of the model prediction. For the second model loss L2, in one embodiment, obtaining the second model loss that characterizes the consistency between the predicted position and the annotated position of each facial key point in step S204 may include: obtaining the difference between the predicted position and the annotated position corresponding to each facial key point, obtaining a mean of the differences based on the difference between the predicted position and the annotated position corresponding to each facial key point, and obtaining the second model loss according to the mean of the differences. As mentioned above, this embodiment can obtain the absolute value of the difference between the predicted position and the annotated position corresponding to each facial key point, and calculate the mean of the absolute values of the differences based on the sum of the absolute values of the differences between the predicted position and the annotated position corresponding to each facial key point, combined with the total number of facial key points L, and then the mean of the absolute values of the differences can be used as the second model loss L2. Specifically, it can be expressed by this formula: Among them, g i represents the coordinates of the labeled position of the manually labeled i-th facial key point, p i Represents the coordinates of the predicted position of the i-th facial landmark predicted by the model.

[0078] For model training, as an example, server 120 may obtain a total model loss based on the sum of the first model loss and the second model loss. During the model training process, when server 120 determines that the total model loss is less than or equal to a preset total loss, server 120 may obtain a trained or so-called trained facial landmark detection model. As another example, during the model training process, when server 120 determines that the total model loss is less than or equal to a preset total loss threshold, and both the first model loss and the second model loss are less than or equal to their respective preset model loss thresholds, server 120 may obtain a trained facial landmark detection model.

[0079] The training method of the facial key point detection model of the present application obtains the average position of each facial key point based on the labeled position of each facial key point on each facial image in a facial image dataset, determines the principal component set of the facial images in the dataset and obtains the fitting coefficient corresponding to each principal component of each facial image based on principal component analysis and labeled positions, wherein different principal components in the principal component set correspond to different morphological change dimensions of the face, and then inputs the facial image into the facial key point detection model to be trained. The model obtains the transformed facial image through a first spatial transformation network, obtains the predicted fitting coefficient corresponding to each principal component of the transformed facial image through a coefficient prediction network, then obtains the predicted position of each facial key point on the transformed facial image based on the predicted fitting coefficient, each principal component and the average position, and then obtains the predicted position of each facial key point on the facial image based on the predicted position through a second spatial transformation network. The model is trained based on a first model loss that characterizes the consistency between the predicted fitting coefficient and the fitting coefficient and a second model loss that characterizes the consistency between the predicted position and the labeled position. This solution is based on the principal components obtained by principal component analysis of the face image dataset and the fitting coefficients corresponding to each principal component of each face image. Combined with the labeled positions and corresponding average positions of each facial key point on each face image in the dataset, the face key point detection model is trained so that it can accurately predict the fitting coefficients of the face image and based on this, the positions of each facial key point on the face image can be accurately predicted. The model structure is simple and efficient, and can take into account both the accuracy and efficiency of face key point detection. It can achieve accurate detection of face key points while meeting the real-time requirements of face key point detection in fields such as live streaming.

[0080] In one embodiment, Figure 7 As shown, this application provides a face image processing method for live broadcast, which can be applied to Figure 1 The terminal 110 or the server 120 shown, the method may include the following steps:

[0081] Step S701, obtaining a face image to be processed in a live webcast;

[0082] Step S702: Detect and obtain facial key points on the face image to be processed using a trained facial key point detection model, wherein the facial key point detection model can be trained according to the training method for facial key point detection model described in any of the above embodiments.

[0083] Step S703 : applying special effects to the facial image to be processed based on the facial key points on the facial image to be processed.

[0084] Specifically, the method provided in this embodiment is applied to Figure 1The terminal 110 shown is used for illustration, wherein the training of the facial key point detection model can be performed on the server 120. The server 120 can obtain the facial key point detection model by training according to the training method of the facial key point detection model as described in any of the above embodiments of the present application, and the server 120 sends the trained facial key point detection model to the terminal 110. In a live broadcast on the Internet, the terminal 110 can obtain the host's facial image as the facial image to be processed, and then obtain the facial key points on the facial image to be processed by detecting the trained facial key point detection model. Based on the facial key points on the facial image to be processed, special effects such as beauty and makeup can be applied to the facial image to be processed, and the facial image after the special effects are applied can be presented to the host and the audience.

[0085] In addition, the method provided in this embodiment is applied to Figure 1 The server 120 shown in FIG. 1 is used for illustration. During a live webcast, the terminal 110 can obtain the host's facial image and send it to the server 120. The server 120 uses the received host's facial image as the facial image to be processed. The server 120 then detects the facial key points on the facial image to be processed using a trained facial key point detection model. Based on these key points, special effects such as beauty and makeup are applied to obtain a facial image with the added effects. The facial image with the added effects can also be presented to the host and the audience.

[0086] The method of this embodiment can apply the training method of the facial key point detection model provided by this application to the facial key point detection in the face images of live broadcasts on the Internet. It can improve the accuracy of facial key point detection while meeting the real-time requirements of live broadcast scenarios for facial key point detection. On the basis of achieving efficient and accurate detection of facial key points, special effects such as beauty and makeup can be added, so that the added special effects can also have efficient and accurate effects, achieving the purpose of balancing the model calculation amount and effect, and better enabling facial key point detection technology to serve the live broadcast scenario.

[0087] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0088] Based on the same inventive concept, the embodiments of the present application further provide a related device for implementing the above-mentioned related methods. The implementation solution provided by the device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of one or more related device embodiments provided below can be referred to the limitations of the related methods above and will not be repeated here.

[0089] In one embodiment, Figure 8 As shown, a training device for a facial key point detection model is provided, and the device 800 may include:

[0090] A position acquisition module 801 is configured to obtain an average position of each facial key point based on the marked position of each facial key point on each facial image in the facial image dataset;

[0091] A principal component analysis module 802 is configured to determine a principal component set of facial images in the facial image dataset based on principal component analysis and annotated positions, and obtain fitting coefficients corresponding to each principal component of each facial image; different principal components in the principal component set correspond to different morphological variation dimensions of the face;

[0092] An image input module 803 is configured to input the facial image into a facial key point detection model to be trained, so that the facial key point detection model to be trained obtains a transformed facial image based on the facial image through a first spatial transformation network, obtains prediction fitting coefficients corresponding to each principal component based on the transformed facial image through a coefficient prediction network, obtains predicted positions of each facial key point on the transformed facial image based on the prediction fitting coefficients, each principal component, and an average position, and obtains predicted positions of each facial key point on the facial image based on the predicted positions of each facial key point on the transformed facial image through a second spatial transformation network;

[0093] The loss acquisition module 804 is configured to acquire a first model loss representing the consistency between the predicted fitting coefficient and the fitting coefficient, and acquire a second model loss representing the consistency between the predicted position and the annotated position of each facial key point;

[0094] The model training module 805 is used to train the facial key point detection model to be trained according to the first model loss and the second model loss.

[0095] In one embodiment, the image input module 803 is further used to obtain the predicted position changes of each facial key point on the transformed facial image based on the predicted fitting coefficients corresponding to each principal component of the transformed facial image and the principal components; and obtain the predicted position of each facial key point on the transformed facial image based on the predicted position changes and the average position.

[0096] In one embodiment, the principal component analysis module 802 is further used to perform a similarity transformation on the marked position of each facial key point on each facial image based on the average position of each facial key point to obtain the transformed position of each facial key point on each facial image; and perform principal component analysis based on the transformed position of each facial key point on each facial image to obtain the principal component set.

[0097] In one embodiment, the principal component analysis module 802 is further used to normalize the transformed position of each facial key point on each facial image relative to the center of the facial image to obtain the normalized transformed position of each facial key point on each facial image; and perform principal component analysis based on the normalized transformed position of each facial key point on each facial image to obtain the principal component set.

[0098] In one embodiment, the principal component analysis module 802 is further configured to fit the principal component set using the normalized transformed positions of the facial key points on each facial image to obtain the fitting coefficients corresponding to the principal components of each facial image.

[0099] In one embodiment, the loss acquisition module 804 is configured to obtain a first model loss according to a difference between the predicted fitting coefficient and the fitting coefficient.

[0100] In one embodiment, the loss acquisition module 804 is used to obtain the difference between the predicted position and the marked position corresponding to each facial key point, obtain the mean difference based on the difference between the predicted position and the marked position corresponding to each facial key point, and obtain the second model loss based on the mean difference.

[0101] In one embodiment, Figure 9 As shown, a face image processing device for live webcasting is provided, and the device 900 may include:

[0102] Image acquisition module 901, used to acquire the face image to be processed in the live broadcast;

[0103] A key point detection module 902 is configured to detect and obtain facial key points on the face image to be processed using a trained facial key point detection model; the facial key point detection model is trained using the training device for the facial key point detection model described above;

[0104] The image processing module 903 is configured to apply special effects to the facial image to be processed based on the facial key points on the facial image to be processed.

[0105] Each module in the above-mentioned related devices can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0106] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 10 As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data such as a facial image dataset. The network interface of the electronic device is used to communicate with external devices via a network connection. When the computer program is executed by the processor, a method for training a facial key point detection model and a method for processing facial images for live broadcast are implemented.

[0107] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with external devices in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for processing facial images for live broadcasting is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse.

[0108] Those skilled in the art will understand that Figure 10 and Figure 11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0109] In one embodiment, an electronic device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0113] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A training method for a facial key point detection model, characterized in that: The method comprises: Obtaining the average position of each facial key point based on the marked position of each facial key point on each facial image in the facial image dataset; Based on principal component analysis and annotated positions, a principal component set of facial images in the facial image dataset is determined, and a fitting coefficient corresponding to each principal component of each facial image is obtained; different principal components in the principal component set correspond to different morphological change dimensions of the face; Inputting the facial image into a facial key point detection model to be trained, so that the facial key point detection model to be trained obtains a transformed facial image based on the facial image through a first spatial transformation network, obtains prediction fitting coefficients corresponding to each principal component based on the transformed facial image through a coefficient prediction network, obtains predicted positions of each facial key point on the transformed facial image based on the prediction fitting coefficients, each principal component, and an average position, and obtains predicted positions of each facial key point on the facial image based on the predicted positions of each facial key point on the transformed facial image through a second spatial transformation network; Obtaining a first model loss representing the consistency between the predicted fitting coefficient and the fitting coefficient, and obtaining a second model loss representing the consistency between the predicted position and the marked position of each facial key point; The facial key point detection model to be trained is trained according to the first model loss and the second model loss.

2. The method according to claim 1, characterized in that Obtaining the predicted position of each facial key point on the transformed facial image according to the predicted fitting coefficient and the principal components and the average position includes: Obtaining predicted position changes of each facial key point on the transformed facial image based on the predicted fitting coefficients corresponding to each principal component of the transformed facial image and the principal components; The predicted position of each facial key point on the transformed facial image is obtained according to the predicted position change and the average position.

3. The method according to claim 1, characterized in that Determining a principal component set of facial images in the facial image dataset includes: Performing a similarity transformation on the marked positions of the facial key points on each facial image based on the average position of the facial key points to obtain the transformed positions of the facial key points on each facial image; Principal component analysis is performed based on the transformed positions of the facial key points on each facial image to obtain the principal component set.

4. The method according to claim 3, characterized in that The principal component analysis is performed based on the transformed positions of the facial key points on each facial image to obtain the principal component set, including: Normalizing the transformed positions of the facial key points on each facial image relative to the center of the facial image to obtain normalized transformed positions of the facial key points on each facial image; Principal component analysis is performed based on the normalized transformed positions of the facial key points on each facial image to obtain the principal component set.

5. The method according to claim 4, characterized in that The obtaining of the fitting coefficient corresponding to each principal component of each face image includes: The principal component set is fitted using the normalized transformed position of each facial key point on each facial image to obtain a fitting coefficient corresponding to each principal component of each facial image.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining of the first model loss representing the consistency between the predicted fitting coefficient and the fitting coefficient includes: obtaining the first model loss according to the difference between the predicted fitting coefficient and the fitting coefficient; The method of obtaining the second model loss that characterizes the consistency between the predicted position and the annotated position of each facial key point includes: obtaining the difference between the predicted position and the annotated position corresponding to each facial key point, obtaining a mean of the differences based on the difference between the predicted position and the annotated position corresponding to each facial key point, and obtaining the second model loss according to the mean of the differences.

7. A facial image processing method for live webcasting, characterized in that: The method comprises: Obtain the face image to be processed in the live broadcast; Obtaining facial key points on the face image to be processed by detecting a trained facial key point detection model; the facial key point detection model is trained using the method according to any one of claims 1 to 6; Applying special effects to the facial image to be processed based on each facial key point on the facial image to be processed.

8. A training device for a facial key point detection model, characterized in that: The device comprises: A position acquisition module is used to obtain the average position of each facial key point based on the marked position of each facial key point on each facial image in the facial image dataset; A principal component analysis module is configured to determine a principal component set of facial images in the facial image dataset based on principal component analysis and annotated positions, and obtain fitting coefficients corresponding to each principal component of each facial image; different principal components in the principal component set correspond to different morphological variation dimensions of the face; An image input module, configured to input the facial image into a facial key point detection model to be trained, so that the facial key point detection model to be trained obtains a transformed facial image based on the facial image through a first spatial transformation network, obtains prediction fitting coefficients corresponding to each principal component based on the transformed facial image through a coefficient prediction network, obtains a predicted position of each facial key point on the transformed facial image based on the prediction fitting coefficients, each principal component, and an average position, and obtains a predicted position of each facial key point on the facial image based on the predicted position of each facial key point on the transformed facial image through a second spatial transformation network; A loss acquisition module is used to acquire a first model loss representing the consistency between the predicted fitting coefficient and the fitting coefficient, and to acquire a second model loss representing the consistency between the predicted position and the marked position of each facial key point; A model training module is used to train the facial key point detection model to be trained based on the first model loss and the second model loss.

9. A face image processing device for live webcast, characterized in that: The device comprises: An image acquisition module is used to acquire the face image to be processed in the live broadcast; A key point detection module, configured to detect and obtain facial key points on the face image to be processed using a trained facial key point detection model; the facial key point detection model is trained using the apparatus according to claim 8; The image processing module is used to apply special effects to the face image to be processed based on each facial key point on the face image to be processed.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images