Face prediction model training and multi-task prediction method and device, and live broadcast system

Abstract

The application relates to a face prediction model training method and device, a face multi-task prediction method and device, a live broadcast system, computer equipment and a computer readable storage medium. The training method comprises the following steps: grouping face image sets according to prediction tasks, and performing data labeling on the face image groups respectively; a feature map pyramid network and a prediction task layer corresponding to each prediction task are constructed, the prediction task layers are connected to feature maps output by the feature map pyramid network respectively to obtain a face multi-task prediction model; loss functions of the prediction task layers of the face multi-task prediction model in prediction training are configured respectively; and the face multi-task prediction model is trained by using the labeled face images and according to the loss functions. According to the technical scheme, the prediction model performing multiple prediction tasks simultaneously can be trained, the efficiency of face multi-task prediction is improved, and the prediction demand in a live broadcast scene is particularly suitable, and the use demand of different real-time live broadcast scenes is met.

Description

Technical Field

[0001] This application relates to the field of live streaming technology, and in particular to a training method and apparatus for a face prediction model, a face multi-task prediction method and apparatus, a live streaming system, computer equipment, and a computer-readable storage medium. Background Technology

[0002] With the development of live streaming technology, various beauty filters, makeup effects, and plastic surgery effects are widely used in live streaming to enhance the dissemination of high-quality content. During live streaming, in order to accurately locate faces and add effects, facial landmark detection and segmentation are necessary. Therefore, algorithms related to facial landmark detection and segmentation are the foundation of live streaming and short video effects technologies such as beauty filters, makeup effects, and plastic surgery effects.

[0003] Currently, when performing 2D keypoint, 3D keypoint, and image segmentation tasks on facial images, face prediction models are typically trained for detection. Each prediction model can complete a corresponding prediction task. For example, a 2D keypoint model can be used to predict 2D keypoints, a 3D keypoint model can be used to predict 3D keypoints, and a segmentation model can be used to predict image segmentation. However, in live streaming scenarios, when multiple prediction tasks need to be completed, using these prediction models simultaneously cannot meet the high real-time requirements of live streaming. This will affect the computational efficiency of beautification and makeup algorithms in live streaming scenarios, making it difficult to meet the multi-task requirements of live streaming services and easily affecting the live streaming effect. Summary of the Invention

[0004] Therefore, it is necessary to address the aforementioned technical problems by providing a training method and apparatus for a face prediction model, a face multi-task prediction method and apparatus, a live streaming system, computer equipment, and a computer-readable storage medium, thereby realizing the multi-task prediction function of a single model and improving the real-time performance of live streaming.

[0005] Firstly, this application provides a method for training a face prediction model, comprising:

[0006] The face image set is grouped according to the prediction task, and the face image group is labeled according to each prediction task.

[0007] A feature map pyramid network and prediction task layers corresponding to each prediction task are constructed, and the prediction task layers are respectively connected to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model.

[0008] Configure the loss function for each prediction task layer of the face multi-task prediction model during prediction training;

[0009] The face multi-task prediction model is trained using the labeled face images and according to the loss function.

[0010] In one embodiment, constructing a feature map pyramid network and prediction task layers corresponding to each prediction task includes:

[0011] Construct a feature map pyramid network for extracting image features; wherein the feature map pyramid network takes a face image as input and outputs multiple feature maps with progressively increasing resolution;

[0012] Each prediction task layer is built after the feature map pyramid network according to the prediction task to be performed.

[0013] In one embodiment, the prediction task layers are respectively connected to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model, including:

[0014] For each prediction task layer, at least one feature map is selected from the feature maps output by the feature map pyramid network as the input image;

[0015] The prediction task layer is connected to the feature map based on the input image selected for each prediction task to obtain a face multi-task prediction model.

[0016] In one embodiment, training the face multi-task prediction model using the labeled face images and according to the loss function includes:

[0017] Read the labeled face images from each group and input them into the face multi-task prediction model respectively;

[0018] Calculate the prediction results of each prediction task output by the face multi-task prediction model under the combined influence of the various loss functions;

[0019] The parameters of the face multi-task prediction model are adjusted based on the prediction results until the prediction results output by the face multi-task prediction model meet the set target requirements.

[0020] In one embodiment, the prediction task includes: predicting 2D key points of a face image and predicting 3D key points of a face image;

[0021] The step of grouping and labeling the face images according to each prediction task includes:

[0022] In each face image grouped by 2D keypoints, several 2D keypoints are marked on the face region.

[0023] A frontal face image is rendered using a 3D base model of the face, and the correspondence between the vertices of the 3D base model and each 2D key point is determined. Based on the correspondence, several 2D key points are marked on the face area of ​​each face image in the group of 3D key point face images.

[0024] In one embodiment, rendering a frontal face image using a 3D base model of a face and determining the correspondence between the vertices of the 3D base model and each 2D keypoint includes:

[0025] Dimensionality reduction is performed on the expression basis and shape basis of the 3D facial base model;

[0026] A frontal face image is rendered using the average face of a 3D base model of a human face, and several 2D key points are marked on the frontal face image;

[0027] Calculate the 2D projection points of the 3D vertices on the frontal face image, and determine the projection point with the smallest distance to each 2D key point to obtain the correspondence between the vertices of the 3D base model and the 2D key points when facing the frontal face.

[0028] The 2D key points of the cheek contour in the frontal face image are adjusted to obtain the correspondence between the vertices of the 3D base model and the 2D key points when viewing the side profile.

[0029] In one embodiment, the prediction task further includes: predicting the pixel classification of the visible area of ​​the face in the face image and predicting the 3D mesh formed by the 3D facial key points;

[0030] The step of grouping and labeling the face images according to each prediction task further includes:

[0031] The face region is segmented from the background region in each face image group. The face region is labeled as foreground pixels and the background region is labeled as background pixels.

[0032] The connection relationships of 3D key points in face images are determined based on the vertex connection relationships of the 3D base model.

[0033] In one embodiment, the prediction task layer includes: a first prediction task layer for predicting 2D key points of a face image, a second prediction task layer for predicting pixel classification of the visible area of ​​a face in a face image, a third prediction task layer for predicting 3D key points of a face image, and a fourth prediction task layer for predicting the 3D mesh formed by the 3D key points of the face.

[0034] The feature map pyramid network of the image features outputs feature maps F1, F2, F3, and F4 with progressively increasing resolution;

[0035] The first and second prediction task layers are convolutional neural network layers, with feature map F4 as the input image; the third and fourth prediction task layers are linear layers, with feature map F1 as the input image.

[0036] In one embodiment, configuring the loss function of each prediction task layer of the face multi-task prediction model during prediction training includes:

[0037] Configure the loss function of the first prediction task layer as follows: in, This represents the coordinates of the i-th keypoint predicted by the multi-task face prediction model. This represents the coordinates of the i-th 2D keypoint.

[0038] Configure the loss function of the second prediction task layer as L seg =-(gt seg *logpred seg )+(1-gt seg )*log(1-pred seg ), where gt seg Foreground or background pixels, pred seg The prediction results of the face multi-task prediction model;

[0039] Configure the loss function for the third and fourth prediction task layers as follows: in, These are the 2D projections of 3D facial landmarks predicted by a multi-task face prediction model onto a face image. This represents the coordinates of the i-th 2D keypoint.

[0040] Secondly, this application provides a training device for a face prediction model, comprising:

[0041] The data annotation module is used to group the face image set according to the prediction task, and to annotate the face image group according to each prediction task.

[0042] The model building module is used to construct a feature map pyramid network and prediction task layers corresponding to each prediction task. The prediction task layers are connected to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model.

[0043] The function configuration module is used to configure the loss function of each prediction task layer of the face multi-task prediction model during prediction training.

[0044] The model training module is used to train the face multi-task prediction model using the labeled face images and according to the loss function.

[0045] Thirdly, this application provides a face multi-task prediction method, including:

[0046] Obtain the target face image of the broadcaster;

[0047] The target face image is input into the face multi-task prediction model; wherein the face multi-task prediction model is obtained using the training method of the face prediction model.

[0048] Based on the prediction task requirements, the corresponding target task prediction result is selected from the prediction results output by the face multi-task prediction model.

[0049] Fourthly, this application provides a face multitasking prediction device, comprising:

[0050] The image acquisition module is used to acquire the target face image of the anchor;

[0051] The model prediction module is used to input the target face image into the face multi-task prediction model; wherein the face multi-task prediction model is obtained using the training method of the face prediction model.

[0052] The result selection module is used to select the corresponding target task prediction result from the prediction results output by the face multi-task prediction model according to the prediction task requirements.

[0053] Fifthly, this application provides a live streaming system, including: a broadcaster's terminal, a viewer's terminal, and a live streaming server; wherein the broadcaster's terminal and the viewer's terminal are respectively connected to the live streaming server through a communication network;

[0054] The broadcaster's client is used to connect to the broadcaster in the live broadcast room and to collect the broadcaster's live video stream and upload it to the live broadcast server.

[0055] The live streaming server is used to forward live streams between the broadcaster and the audience and to send live video to the audience; it obtains the target face image of the broadcaster from the live video stream, uses the face multi-task prediction method to obtain the target task prediction result of the target face image and adds special effects.

[0056] The viewer terminal is used to connect to the live broadcast room and to receive and play the live video.

[0057] Sixthly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the training method for the face prediction model or the steps of the multi-task face prediction method.

[0058] In a seventh aspect, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the training method for the face prediction model or the steps of the multi-task face prediction method.

[0059] The technical solutions provided in the above embodiments, in the training of the face prediction model, group the face image set according to the prediction task and perform data annotation. By constructing a feature map pyramid network and prediction task layers corresponding to each prediction task, the feature map and the prediction task layers are connected to obtain a face multi-task prediction model. Then, the loss function of each prediction task layer is configured, and finally the face multi-task prediction model is trained using the annotated face images and the loss function. This technical solution can train a prediction model that performs multiple prediction tasks simultaneously, improves the efficiency of face multi-task prediction, and is particularly suitable for prediction needs in live streaming scenarios, meeting the usage needs of different real-time live streaming scenarios.

[0060] Furthermore, in multi-task face prediction, the target face image of the broadcaster is first acquired and then input into the multi-task face prediction model. Then, according to the prediction task requirements, the corresponding target task prediction result is selected from the prediction results output by the multi-task face prediction model. This technical solution can simultaneously predict and output prediction results for multiple tasks using the input target face image, thereby realizing the function of single-model prediction of multiple tasks. Multiple prediction results can be selected to add live broadcast effects according to the needs, better meeting the high requirements of real-time performance of algorithms in live broadcast business. Attached Figure Description

[0061] Figure 1 This is a schematic diagram illustrating an example of a live streaming business application scenario;

[0062] Figure 2 This is a flowchart of a face prediction model training method according to one embodiment;

[0063] Figure 3 This is a schematic diagram illustrating the 3D key point annotation results of an example face image;

[0064] Figure 4 This is a schematic diagram illustrating the conversion between 3D key points and 2D key points;

[0065] Figure 5 This is a schematic diagram of an example multi-task face prediction model.

[0066] Figure 6 This is a schematic diagram of the training device structure for a face prediction model according to one embodiment;

[0067] Figure 7 This is a flowchart of a face multi-task prediction method according to one embodiment;

[0068] Figure 8 This is a schematic diagram of the structure of a face multitasking prediction device according to one embodiment;

[0069] Figure 9 This is a schematic diagram of an example live streaming system structure;

[0070] Figure 10 This is a schematic diagram of the structure of a computer device according to one embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] The technical solutions provided in the embodiments of this application can be applied to, for example... Figure 1 In the application scenarios of the methods related to this application shown, Figure 1 This is a schematic diagram illustrating an example of a live streaming application scenario. The live streaming system can include a live streaming server, a broadcaster's client, and a viewer's client. The broadcaster's client and the viewer's client communicate with the live streaming server via a communication network, enabling real-time online streaming for both. The terminal devices for the broadcaster's client and the viewer's client can be, but are not limited to, various personal computers, laptops, smartphones, and tablets. The live streaming server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0073] The following describes an embodiment of the training method for the face prediction model of this application. This application can be applied to scenarios involving face information prediction and can simultaneously achieve multi-task prediction functions using a single model; see references. Figure 2 As shown, Figure 2 This is a flowchart of a training method for a face prediction model according to one embodiment, which may include the following steps:

[0074] S11, group the face image set according to the prediction task, and label the face image group according to each prediction task.

[0075] In this step, facial images containing faces can be collected from the Internet and other sources. These facial images are then grouped according to the prediction task to obtain facial image groups corresponding to different prediction tasks. Finally, these facial image groups are labeled according to different prediction tasks.

[0076] For example, if the prediction task includes predicting 2D keypoints and 3D keypoints of a face image, then correspondingly, 2D keypoint face image groups and 3D keypoint face image groups can be obtained in advance. When annotating the face image groups, the following can be included:

[0077] (1) Mark several 2D key points on the face part of each face image in the 2D key point face image group.

[0078] For 2D key points, they refer to key points in the visible area of ​​a face in a face image. Generally, when the face is in profile, the cheek contour points are not the cheek positions of the face in the real 3D world, but rather the positions in the visible area of ​​the 2D image that are closest to the cheeks of a real 3D face.

[0079] For example, 2D key points of the face image can be labeled on each face image in a group of 2D key point face images, such as labeling 300 2D key points on each face image.

[0080] (2) Render a frontal face image using a 3D base model of the face, and determine the correspondence between the vertices of the 3D base model and each 2D key point; mark several 2D key points on the face part of each face image in the 3D key point face image group according to the correspondence.

[0081] 3D key points refer to the key points of the real 3D face in the face image. Generally, in the case of a side profile, although this area is occluded and not visible in the face image, the cheek contour points are located in the cheek area of ​​the real 3D face.

[0082] For example, a frontal face image can be rendered using a 3D base model of a face, and then the correspondence between the vertices of the 3D base model and each 2D key point can be determined. When annotating the 3D key points, the data annotation of the 3D key points can be converted into 2D key points for annotation. The conversion between 3D and 2D key points can be achieved through the correspondence determined by the 3D base model.

[0083] As an example, the correspondence between the vertices of the 3D base model and each 2D key point can be determined using the BFM (Bfm facial model) 3D base model, thereby determining the transformation relationship between 3D and 2D, specifically including the following:

[0084] a. Reduce the dimensionality of the expression basis and shape basis of the 3D face base model.

[0085] For example, the BFM2019 face 3D base model (parametric model) can be used, and the expression basis and shape basis of the BFM2019 face 3D base model can be reduced to 80 dimensions using PCA (Principal Component Analysis) dimensionality reduction technology.

[0086] b. Render a frontal face image using the average face of the 3D base model of the face, and mark several 2D key points on the frontal face image.

[0087] For example, the meanshape of the face based on the BFM2019 face 3D base model is used to render a frontal face image. Referring to the annotation method of 2D key points, 300 2D key points are annotated on the frontal face image.

[0088] c. Calculate the 2D projection points of the 3D vertices on the frontal face image, and determine the projection point with the smallest distance to each 2D key point to obtain the correspondence between the vertices of the 3D base model and the 2D key points when facing the frontal face.

[0089] For example, calculate the 2D projection of the 3D vertex of each face image on the meanshape of the average face. For each of the 300 labeled 2D key points, find the projection point with the smallest distance. Then, in the case of a frontal face, obtain the correspondence between the 3D vertex of the projection point and the 2D key point.

[0090] refer to Figure 3 As shown, Figure 3 This is a schematic diagram of the 3D key point annotation result of an example face image. The "·" point is the annotated 2D key point, and the "×" point is the projection point of the corresponding 3D key point on the face image. It can be seen that the two are almost overlapping, thus obtaining the conversion relationship between 3D and 2D key points when facing the front.

[0091] d. Adjust the 2D key points of the cheek contour in the frontal face image to obtain the correspondence between the vertices of the 3D base model and the 2D key points when viewing the side profile.

[0092] For example, since the 3D key points and 2D key points of the cheek position of the face region differ greatly when the face image is rotated, the key points of the cheek contour of the face region are adjusted accordingly, so that the correspondence between the vertices of the 3D base model and the 2D key points can be obtained even when the face is viewed from the side, thereby obtaining the transformation relationship between 3D and 2D key points when viewed from the side.

[0093] For example, refer to Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the conversion between 3D and 2D keypoints; as shown, assuming a frontal face image, the 1999th 3D vertex v... 1999= (x,y,z) corresponds to the 2D keypoint pt1 = (w,h) of the cheek in the first face image, according to v 1999 The Y-axis coordinate can be used to define a horizontal line with a length of half the width of the face. All 3D vertices on this horizontal line are numbered and added to the 3D vertex set ω1 = {1999,…} of the 2D keypoint pt1. For the left side view of the face image, all numbered 3D vertices in the set ω1 are taken out. The 3D vertex with the largest x-coordinate corresponds to the 2D keypoint pt1 at this time. By analogy, the 2D keypoints corresponding to each 3D keypoint on the face image can be obtained.

[0094] As in the above embodiments, since 3D key points are converted into 2D key points for annotation, the 3D key point face image group can share a set of face images with the 2D key point face image group.

[0095] In one embodiment, the prediction task may further include predicting the pixel classification of the visible facial region in a face image and predicting the 3D mesh formed by 3D facial key points; accordingly, face region segmentation and face image groups can be obtained first, and data annotation of the face image groups may include the following:

[0096] (3) Segment the face region: The face region and background region on each face image group are segmented, and the face region is labeled as foreground pixels and the background region is labeled as background pixels.

[0097] Face region segmentation refers to the pixel-by-pixel classification of the visible area of ​​a face in a face image. The segmented image content generally only includes the visible area of ​​the face and does not include other background areas such as hair and clothing.

[0098] For example, the face region is first separated from the background region such as hair and clothing in each face image group. Then, the face region is labeled as a foreground pixel (pixel value of 1) and the background region is labeled as a background pixel (usually with a pixel value of 0), thereby separating the face region from the background region.

[0099] (4) Determine the 3D key point connection relationship of the face image group based on the vertex connection relationship of the 3D base model.

[0100] A 3D mesh is a 3D mesh representing a realistic 3D face in a face image. It consists of 3D keypoints and the connections between them. Since the vertex connections of the 3D base model are fixed, when annotating the 3D mesh, the connections between each 3D keypoint can be obtained using these vertex connections and the previously annotated 3D keypoints (which can be determined from 2D keypoints and their correspondences).

[0101] The above embodiments design prediction tasks for predicting 2D key points of face images, face region segmentation, and predicting 3D key points and their 3D meshes in face images; and by converting 3D key points into 2D key points for data annotation, it is convenient to perform multi-task model training in a single model.

[0102] S12, construct a feature map pyramid network and prediction task layers corresponding to each prediction task, and connect the prediction task layers to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model.

[0103] In this step, the process of building a face multi-task prediction model is to use Feature Pyramid Networks (FPN) to extract feature maps from the input face image, build corresponding prediction task layers according to each prediction task, and then connect the prediction task layers to the corresponding feature maps output by the Feature Pyramid Network according to the different feature map requirements of different prediction tasks, thereby building a face multi-task prediction model.

[0104] In one embodiment, the process of constructing the feature map pyramid network and the prediction task layers corresponding to each prediction task in step S12 may include the following:

[0105] S201, Construct a feature map pyramid network for extracting image features; wherein, the feature map pyramid network takes a face image as input and outputs multiple feature maps with gradually increasing resolution.

[0106] For example, a feature map pyramid network that constructs image features outputs feature maps F1, F2, F3, and F4 with progressively increasing resolution; wherein the resolution of feature maps F1 to F4 gradually increases.

[0107] S202, according to the various prediction tasks to be performed, build corresponding prediction task layers after the feature map pyramid network.

[0108] For example, as in the previous embodiment, four prediction tasks can be set: predicting 2D key points of a face image, predicting 3D key points of a face image, predicting pixel classification of the visible area of ​​the face in the face image, and predicting the 3D mesh formed by the 3D key points of the face.

[0109] Correspondingly, the prediction task layers include: a first prediction task layer for predicting 2D key points of a face image, a second prediction task layer for predicting pixel classification of the visible area of ​​the face in a face image, a third prediction task layer for predicting 3D key points of a face image, and a fourth prediction task layer for predicting the 3D mesh formed by the 3D key points of the face.

[0110] Preferably, the first and second prediction task layers can be convolutional neural network layers, and the third and fourth prediction task layers can be linear layers.

[0111] In the above embodiments, the process of connecting the prediction task layers to the feature maps output by the feature map pyramid network in step S12 to obtain the face multi-task prediction model can include the following:

[0112] S203, for each prediction task layer, select at least one feature map from the feature maps output by the feature map pyramid network as the input image.

[0113] For example, for the four prediction task layers mentioned above, different feature maps can be selected from the feature maps output by the feature map pyramid network as the input images for that prediction task layer. Preferably, the feature map with the highest resolution, F4, can be selected as the input image for the first and second prediction task layers, and the feature map with the lowest resolution, F1, can be selected as the input image for the third and fourth prediction task layers. Of course, other selection schemes can also be used, such as selecting feature maps F1 and F2 as the input images for the third and fourth prediction task layers, and selecting feature maps F3 and F4 as the input images for the first and second prediction task layers, etc. The specific settings can be configured according to the training requirements under different circumstances.

[0114] S204, Connect the prediction task layer to the feature map according to the input image selected for each prediction task to obtain a face multi-task prediction model.

[0115] like Figure 5 As shown, Figure 5 This is a schematic diagram of an example multi-task face prediction model. The first and second prediction task layers use feature map F4 as input, while the third and fourth prediction task layers use feature map F1 as input.

[0116] The first prediction task layer uses feature map F4 as the input image and employs the conv2d function of a convolutional neural network to obtain a heatmap. Then, the heatmap is used to calculate the coordinates of 2D key points using the soft-argmax algorithm.

[0117] The second prediction task layer is a convolutional neural network layer that takes feature map F4 as input image and uses the convolutional neural network operation function conv2d to predict the segmentation image of the face image.

[0118] The third prediction layer uses a linear function to predict the coefficients of the BFM base model and camera parameters, where the coefficients include shape coefficients W. id ∈R 80x1 The expression coefficient is W. exp ∈R 80x1 Camera parameters include translation parameters T∈R 3x1 and rotation parameters R∈R 3x1 The predicted set of 3D facial landmarks is V = meanshape + W. id *B id +W exp *B exp .

[0119] The fourth prediction task layer is obtained by using the prediction results of 3D face key points and the connection relationship of 3D key points from the third prediction task layer. That is, the 3D mesh is obtained by adding the connection relationship of predefined 3D vertex triangles to the 3D face key point set V.

[0120] The above embodiments provide a technical solution for building a face multi-task prediction model. This face multi-task prediction model includes several prediction tasks commonly used in current live streaming scenarios, especially covering face key point detection and segmentation algorithms commonly used in live streaming and short video special effects technologies such as beautification, makeup, and plastic surgery, which greatly improves the algorithm efficiency in live streaming scenarios.

[0121] S13, respectively configure the loss function of each prediction task layer of the face multi-task prediction model in prediction training.

[0122] In this step, based on the established face multi-task prediction model, the loss function of each prediction task layer in the prediction training is designed. Since the face multi-task prediction model is a single-model multi-task prediction structure, the loss functions of each prediction task layer affect each other during training, and the final output is that multiple prediction tasks output prediction results simultaneously.

[0123] In one embodiment, taking four prediction tasks as an example—predicting 2D keypoints of a face image, predicting 3D keypoints of a face image, predicting pixel classification of the visible facial region in a face image, and predicting the 3D mesh formed by the 3D face keypoints—the corresponding loss functions configured for each prediction task layer during prediction training can include the following:

[0124] Configure the loss function of the first prediction task layer as follows: in, This represents the coordinates of the i-th keypoint predicted by the multi-task face prediction model. This represents the coordinates of the i-th 2D keypoint.

[0125] Configure the loss function of the second prediction task layer as Lseg =-(gt seg *logpred seg )+(1-gt seg )*log(1-pred seg ), where gt seg This refers to the annotation information for foreground or background pixels, i.e., whether a pixel belongs to the foreground pixel (1) or the background pixel (0). seg The prediction results (0-1) of the face multi-task prediction model.

[0126] Configure the loss function for the third and fourth prediction task layers as follows: in, These are the 2D projections of 3D facial landmarks predicted by a multi-task face prediction model onto a face image. This represents the coordinates of the i-th 2D keypoint; where the correspondence between 3D keypoints and 2D keypoints is the correspondence between vertices of the 3D base model and 2D keypoints.

[0127] The above embodiments design loss functions for each prediction task layer during prediction training according to the prediction task. These loss functions influence each other during model training, thereby enabling the simultaneous output of prediction results for multiple prediction tasks.

[0128] S14, the face multi-task prediction model is trained using the labeled face images and according to the loss function.

[0129] In this step, the face images are grouped and trained under a set loss function to obtain a face multi-task prediction model that can be used for prediction in real-world scenarios.

[0130] In one embodiment, the training process may include the following:

[0131] S401, Read the labeled face images of each group and input them into the face multi-task prediction model respectively; specifically, input the labeled face images into the face multi-task prediction model respectively.

[0132] S402, calculate the prediction results of each prediction task output by the face multi-task prediction model under the combined influence of each of the loss functions; during the training process, each prediction task layer is trained and outputs prediction results simultaneously.

[0133] S402, adjust the parameters of the face multi-task prediction model (including the feature map pyramid network and each prediction task layer) according to the prediction results until the prediction results output by the face multi-task prediction model meet the set index requirements.

[0134] The above embodiment utilizes the loss function of multiple prediction task layers of a single model to train the model simultaneously, thereby enabling the face multi-task prediction model to perform multiple tasks simultaneously on a single face image, thus improving the efficiency of the model prediction algorithm.

[0135] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0136] Based on the same inventive concept, this application also provides an apparatus for implementing the related methods described above. The solution provided by this apparatus is similar to the solution described in the above methods. Therefore, the specific limitations in one or more related apparatus embodiments provided below can be found in the limitations of the related methods above, and will not be repeated here.

[0137] refer to Figure 6 As shown, Figure 6 This is a schematic diagram of a training device for a face prediction model according to one embodiment. The device includes:

[0138] The data annotation module 11 is used to group the face image set according to the prediction task, and to annotate the face image group according to each prediction task.

[0139] The model building module 12 is used to construct a feature map pyramid network and prediction task layers corresponding to each prediction task. The prediction task layers are connected to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model.

[0140] Function configuration module 13 is used to configure the loss function of each prediction task layer of the face multi-task prediction model during prediction training.

[0141] The model training module 14 is used to train the face multi-task prediction model using the labeled face images and according to the loss function.

[0142] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0143] The face prediction model training device of this embodiment can execute a face prediction model training method provided in the embodiments of this application. The implementation principle is similar. The actions performed by each module in the face prediction model training device in each embodiment of this application correspond to the steps in the face prediction model training method in each embodiment of this application. For detailed functional descriptions of each module of the face prediction model training device, please refer to the descriptions in the corresponding face prediction model training methods shown above. They will not be repeated here.

[0144] Based on the training scheme of the face prediction model provided in the foregoing embodiments, this application also provides a face multi-task prediction method, in which a face multi-task prediction model is obtained by using the training method of the face prediction model in any of the foregoing embodiments.

[0145] refer to Figure 7 As shown, Figure 7 This is a flowchart of a face multi-task prediction method according to one embodiment, including:

[0146] S21, Obtain the target face image of the anchor.

[0147] Specifically, in a live streaming scenario, the live streaming server can obtain the streamer's image uploaded by the streamer and perform facial recognition based on the streamer's image to obtain the streamer's target face image.

[0148] S22, input the target face image into the face multi-task prediction model.

[0149] Specifically, the target face image that needs to be processed by adding features is input into a pre-trained face multi-task prediction model to perform multi-task prediction and obtain the corresponding prediction results.

[0150] For example, you can input a target face image, and the face multi-task prediction model can output the prediction results of the face image's 2D key points, face image segmentation map, face image's 3D key points and 3D mesh.

[0151] S23, select the corresponding target task prediction result from the prediction results output by the face multi-task prediction model according to the prediction task requirements.

[0152] Specifically, the live streaming server can select the desired target task prediction result from multiple prediction results output by the face multi-task prediction model according to the usage requirements; for example, if it is currently necessary to obtain 2D key points of a face image and a face image segmentation map, these two output prediction results can be selected for use.

[0153] The technical solution provided in this application can use a single prediction model to simultaneously output prediction results such as 2D key points of a face, face segmentation map, 3D key points and its 3D mesh, which can significantly improve the computational efficiency of algorithms in scenarios using face key point detection algorithms, face segmentation algorithms, and face 3D key point and 3D mesh algorithms, and better meet the high real-time requirements of live streaming services.

[0154] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0155] Based on the same inventive concept, this application also provides an apparatus for implementing the related methods described above. The solution provided by this apparatus is similar to the solution described in the above methods. Therefore, the specific limitations in one or more related apparatus embodiments provided below can be found in the limitations of the related methods above, and will not be repeated here.

[0156] refer to Figure 8 As shown, Figure 8 This is a schematic diagram of a face multi-task prediction device according to an embodiment. The device includes:

[0157] Image acquisition module 21 is used to acquire the target face image of the anchor;

[0158] The model prediction module 22 is used to input the target face image into the face multi-task prediction model; wherein the face multi-task prediction model is obtained by the training method of the face prediction model in any of the above embodiments;

[0159] The result selection module 23 is used to select the corresponding target task prediction result from the prediction results output by the face multi-task prediction model according to the prediction task requirements.

[0160] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0161] The face multitasking prediction device of this embodiment can execute a face multitasking prediction method provided in the embodiments of this application. The implementation principle is similar. The actions performed by each module in the face multitasking prediction device in each embodiment of this application correspond to the steps in the face multitasking prediction method in each embodiment of this application. For detailed functional descriptions of each module of the face multitasking prediction device, please refer to the descriptions in the corresponding face multitasking prediction methods shown above. They will not be repeated here.

[0162] The following describes an implementation example of a live streaming system.

[0163] The live streaming system provided in this embodiment is based on... Figure 9 As shown, Figure 9 This is a schematic diagram of an example live streaming system, which includes: a broadcaster's client, a viewer's client, and a live streaming server; wherein the broadcaster's client and the viewer's client are connected to the live streaming server through a communication network.

[0164] For the broadcaster's end, it is used to connect to the live broadcast room and collect the broadcaster's live video stream and upload it to the live broadcast server; for the live broadcast server, it is used to forward the live stream between the broadcaster's end and the viewer's end and to send the live video to the viewer's end; it obtains the target face image of the broadcaster from the broadcaster's live video stream, uses the face multi-task prediction method of any of the above embodiments to obtain the target task prediction result of the target face image and adds special effects; for the viewer's end, it is used to connect to the live broadcast room and receive the live video for playback.

[0165] like Figure 9As shown, assuming viewers A, B, C... access the live stream through an app client to watch the streamer's broadcast, when the streamer needs to use beauty filters, makeup effects, etc., the multi-task face prediction model needs to be invoked to perform multi-task predictions of face-related information, such as 2D key points, face segmentation maps, 3D key points, and their 3D meshes. At this point, the streamer can upload the video stream to the live streaming server through their client. The live streaming server then calls the multi-task face prediction model to obtain the relevant prediction results and add the corresponding effects. The live streaming server can then generate a live video stream with added effects and distribute it to the clients of each viewer A, B, C... for playback. Because the above live streaming system uses the multi-task face prediction model of this application, it can simultaneously output prediction results for multiple tasks when performing multi-task predictions of face-related information, improving the model's algorithm efficiency and better serving beauty filters, makeup effects, and other special effects technologies in online live streaming services.

[0166] The following describes embodiments of the computer device and computer-readable storage medium of this application.

[0167] refer to Figure 10 As shown, Figure 10 This is a schematic diagram of an example computer device structure. This computer device can be used for live streaming server applications, or for applications on both the viewer and broadcaster ends. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as facial image datasets. The network interface connects to external devices via a communication network. When the computer program is executed by the processor, it implements the methods provided in the embodiments of this application.

[0168] Those skilled in the art will understand that the computer device structure provided in the above embodiments is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0169] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in the above embodiments. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application may include at least one of relational databases and non-relational databases. Non-relational databases may include, but are not limited to, blockchain-based distributed databases, etc. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., etc., and are not limited to these.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training a face prediction model, characterized in that, include: The face image set is grouped according to the prediction task, and data annotation is performed on the face image group according to each prediction task. This includes: annotating several 2D key points on the face part of each face image in the 2D key point face image group; reducing the dimensionality of the expression basis and shape basis of the face 3D basis model; rendering a frontal face image using the average face of the face 3D basis model, and annotating several 2D key points on the frontal face image; calculating the 2D projection points of the 3D vertices on the frontal face image, and determining the projection point with the smallest distance to each 2D key point to obtain the correspondence between the vertices of the 3D basis model and the 2D key points when facing the frontal face; adjusting the 2D key points of the cheek contour of the frontal face image to obtain the correspondence between the vertices of the 3D basis model and the 2D key points when facing the side; the prediction task includes: predicting the 2D key points of the face image and predicting the 3D key points of the face image. A feature map pyramid network and prediction task layers corresponding to each prediction task are constructed, and the prediction task layers are respectively connected to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model. Configure the loss function for each prediction task layer of the face multi-task prediction model during prediction training; The face multi-task prediction model is trained using the labeled face images and according to the loss function.

2. The training method for the face prediction model according to claim 1, characterized in that, Constructing the feature map pyramid network and the prediction task layers corresponding to each prediction task, including: Construct a feature map pyramid network for extracting image features; wherein the feature map pyramid network takes a face image as input and outputs multiple feature maps with progressively increasing resolution; Each prediction task layer is built after the feature map pyramid network according to the prediction task to be performed.

3. The training method for the face prediction model according to claim 2, characterized in that, The prediction task layers are respectively connected to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model, including: For each prediction task layer, at least one feature map is selected from the feature maps output by the feature map pyramid network as the input image; The prediction task layer is connected to the feature map based on the input image selected for each prediction task to obtain a face multi-task prediction model.

4. The training method for the face prediction model according to claim 1, characterized in that, The prediction task also includes: predicting the pixel classification of the visible area of ​​the face in the face image and predicting the 3D mesh formed by the 3D face key points; The step of grouping and labeling the face images according to each prediction task further includes: The face region is segmented from the background region in each face image group. The face region is labeled as foreground pixels and the background region is labeled as background pixels. The connection relationships of 3D key points in face images are determined based on the vertex connection relationships of the 3D base model.

5. A training device for a face prediction model, characterized in that, include: The data annotation module is used to group the face image set according to the prediction task, and to annotate the face image group according to each prediction task. This includes: annotating several 2D keypoints on the face portion of each face image in the 2D keypoint face image group; reducing the dimensionality of the expression basis and shape basis of the 3D face base model; rendering a frontal face image using the average face of the 3D face base model, and annotating several 2D keypoints on the frontal face image; calculating the 2D projection points of the 3D vertices on the frontal face image, and determining the projection point with the smallest distance to each 2D keypoint, thus obtaining the correspondence between the vertices of the 3D base model and the 2D keypoints for the frontal face; adjusting the 2D keypoints of the cheek contour of the frontal face image to obtain the correspondence between the vertices of the 3D base model and the 2D keypoints for the side view; the prediction task includes: predicting the 2D keypoints of the face image and predicting the 3D keypoints of the face image. The model building module is used to construct a feature map pyramid network and prediction task layers corresponding to each prediction task. The prediction task layers are connected to the feature maps output by the feature map pyramid network to obtain a face multi-task prediction model. The function configuration module is used to configure the loss function of each prediction task layer of the face multi-task prediction model during prediction training. The model training module is used to train the face multi-task prediction model using the labeled face images and according to the loss function.

6. A multi-task face prediction method, characterized in that, include: Obtain the target face image of the broadcaster; The target face image is input into the face multi-task prediction model; wherein the face multi-task prediction model is obtained using the training method of the face prediction model according to any one of claims 1-4; Based on the prediction task requirements, the corresponding target task prediction result is selected from the prediction results output by the face multi-task prediction model.

7. A face multi-task prediction device, characterized in that, include: The image acquisition module is used to acquire the target face image of the anchor; The model prediction module is used to input the target face image into the face multi-task prediction model; wherein the face multi-task prediction model is obtained by the training method of the face prediction model according to any one of claims 1-4; The result selection module is used to select the corresponding target task prediction result from the prediction results output by the face multi-task prediction model according to the prediction task requirements.

8. A live streaming system, characterized in that, include: The system includes a broadcaster's client, a viewer's client, and a live streaming server; wherein the broadcaster's client and the viewer's client are respectively connected to the live streaming server via a communication network. The broadcaster's client is used to connect to the broadcaster in the live broadcast room and to collect the broadcaster's live video stream and upload it to the live broadcast server. The live streaming server is used to forward live streams between the broadcaster and the audience and to send live video to the audience; it obtains the target face image of the broadcaster from the live video stream, uses the face multi-task prediction method described in claim 6 to obtain the target task prediction result of the target face image and adds special effects. The viewer terminal is used to connect to the live broadcast room and to receive and play the live video.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the training method for the face prediction model according to any one of claims 1-4 or the steps of the face multi-task prediction method according to claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the face prediction model according to any one of claims 1-4 or the steps of the face multi-task prediction method according to claim 6.

Images