Key point detection, model training, live streaming method and device, equipment and medium

Abstract

Embodiments of the present application disclose a key point detection, model training, live broadcast method, device, equipment and medium. The method comprises: receiving original image data, the original image data having face data; determining a key point detection model, the first n layers of the key point detection model being used to output edge texture features, the edge texture features being fused with texture features and edge features, and the edge texture features being used to generate a target heat map for the edges of the face data; inputting the original image data into the key point detection model, identifying face key points of the face data as target key points, so that the target key points are aligned with the target heat map. Embodiments of the present application can improve the accuracy of edge key point detection.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data processing, and in particular to a key point detection method, a model training method, a live broadcast method, device, equipment and medium. BACKGROUND

[0002] In recent years, face key point detection has been widely concerned in the field of computer vision.

[0003] For example, based on the key point detection result, functions such as face pose estimation, expression analysis or makeup special effect processing can be realized. Correspondingly, whether the detection result of the key point is accurate determines the reliability of the function realized based on the key point detection result.

[0004] At present, a regression-based method can be used for face key point detection, but the prediction result of the edge key point of the face contour is not accurate, and the contour cannot be accurately aligned. SUMMARY

[0005] Embodiments of the present application provide a key point detection method, a model training method, a live broadcast method, device, equipment and medium, which can improve the edge key point detection accuracy.

[0006] In a first aspect, embodiments of the present application provide a key point detection method, comprising:

[0007] receiving original image data, the original image data having face data;

[0008] determining a key point detection model, the first n layers of the key point detection model being used to output edge texture features, the edge texture features being fused with texture features and edge features, and the edge texture features being used to generate a target heat map for the edge of the face data;

[0009] inputting the original image data into the key point detection model, identifying the face key point of the face data as a target key point, so that the target key point aligns with the target heat map.

[0010] In a second aspect, embodiments of the present application provide a key point detection model training method, comprising:

[0011] receiving first sample image data, the first sample image data having face data;

[0012] inputting the first sample image data into the key point detection model to identify the face key point of the face data as a predicted key point;

[0013] extract a first edge texture feature of a network output of a first n layers in the key point detection model, the first edge texture feature being used for input into a pre-trained edge generation model to generate a heat map for a face edge of the first edge texture feature as a first predicted heat map, wherein the first n layers are obtained through common training with the edge generation model;

[0014] align the first predicted heat map with the predicted key point as supervision to train a network after the first n layers in the key point detection model.

[0015] In a third aspect, an embodiment of the present application provides a live streaming method, comprising:

[0016] collecting video data, the original image data in the video data having face data of a host user;

[0017] inputting the original image data into a preset key point detection model to identify a face key point of the face data as a target key point, so that the target key point is aligned with a target heat map, a network of a first n layers in the key point detection model being used to output an edge texture feature, the edge texture feature being fused with a texture feature and an edge feature, and being used to generate a target heat map for an edge of the face data;

[0018] performing business processing on the original image data according to the target key point to obtain target image data;

[0019] publishing video data containing the target image data in a live streaming room opened by the host user.

[0020] In a fourth aspect, an embodiment of the present application further provides a key point detection device, comprising:

[0021] an original image data acquisition module configured to receive original image data, the original image data having face data;

[0022] a key point detection model determination module configured to determine a key point detection model, a network of a first n layers in the key point detection model being used to output an edge texture feature, the edge texture feature being fused with a texture feature and an edge feature, and the edge texture feature being used to generate a target heat map for an edge of the face data;

[0023] a face key point identification module configured to input the original image data into the key point detection model to identify a face key point of the face data as a target key point, so that the target key point is aligned with the target heat map.

[0024] In a fifth aspect, an embodiment of the present application further provides a training device of a key point detection model, comprising:

[0025] The first sample image data acquisition module is configured to receive first sample image data, wherein the first sample image data contains face data.

[0026] The face key point recognition module is configured to input the first sample image data into the key point detection model to recognize face key points of the face data as predicted key points.

[0027] The first edge texture feature extraction module is configured to extract first edge texture features output by the first n layers of the key point detection model, and input the first edge texture features into a pre-trained edge generation model to generate a heat map for a face edge of the first edge texture features as a first predicted heat map, wherein the first n layers of the key point detection model are obtained through joint training with the edge generation model.

[0028] The key point detection model training module is configured to train the network after the first n layers of the key point detection model by aligning the first predicted heat map with the predicted key points as supervision.

[0029] In a sixth aspect, an embodiment of the present application further provides a live broadcast device, characterized in that comprising:

[0030] The video data acquisition module is configured to acquire video data, wherein original image data in the video data contains face data of a host user.

[0031] The live broadcast face key point recognition module is configured to input the original image data into a preset key point detection model to recognize face key points of the face data as target key points, so that the target key points are aligned with a target heat map, wherein the first n layers of the key point detection model are configured to output edge texture features, and the edge texture features are fused with texture features and edge features, and are configured to generate a target heat map for the edge of the face data.

[0032] The face image processing module is configured to perform business processing on the original image data according to the target key points to obtain target image data.

[0033] The target image data publishing module is configured to publish video data containing the target image data in a live broadcast room opened by the host user.

[0034] In a seventh aspect, an embodiment of the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the key point detection method according to any one of the embodiments of the present application, or the key point detection model training method according to any one of the embodiments of the present application, or the live streaming method according to any one of the embodiments of the present application.

[0035] In an eighth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the program is executed by a processor to implement the key point detection method according to any one of the embodiments of the present application, or the key point detection model training method according to any one of the embodiments of the present application, or the live streaming method according to any one of the embodiments of the present application.

[0036] The edge texture features are extracted from the face data in the image by the key point detection model, the extracted edge texture features can generate a target heat map for the edge of the face data, the key point detection model recognizes the face key points in the face data according to the extracted edge texture features, the face key points are aligned to the target heat map, that is, the face key points are aligned to the edge of the face data, the recognition accuracy of the edge key points of the face edge can be improved, the problem that the prediction result of the edge key points is inaccurate in the prior art is solved, the prediction result of the edge key points is improved, and the alignment accuracy of the face contour is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1a FIG. 1 is a flowchart of a key point detection method according to an embodiment of the present application;

[0038] Figure 1b FIG. 2 is a schematic diagram of a heat map according to an embodiment of the present application;

[0039] Figure 1c FIG. 3 is a schematic diagram of a key point detection model according to an embodiment of the present application;

[0040] Figure 2a FIG. 4 is a flowchart of a key point detection model training method according to an embodiment of the present application;

[0041] Figure 2b FIG. 5 is a schematic diagram of an n-layer network and an edge generation model according to an embodiment of the present application;

[0042] Figure 2c FIG. 6 is a schematic diagram of a standard key point according to an embodiment of the present application;

[0043] Figure 3a FIG. 7 is a flowchart of a live streaming method according to an embodiment of the present application;

[0044] Figure 3bis a schematic diagram of a prediction key point output by an unoptimized key point detection model applicable to the embodiment of the present application;

[0045] Figure 4 is a structural schematic diagram of a key point detection device in the fourth embodiment of the present application;

[0046] Figure 5 is a structural schematic diagram of a key point detection model training device in the fifth embodiment of the present application;

[0047] Figure 6 is a structural schematic diagram of a live broadcast device in the sixth embodiment of the present application;

[0048] Figure 7 is a structural schematic diagram of a computer device in the seventh embodiment of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0050] Embodiment one

[0051] Figure 1a is a schematic diagram of a flowchart of a key point detection method in the first embodiment of the present application. The present embodiment can be applicable to the case of identifying the key points of a face in a face image. The method can be executed by the key point detection device provided by the present embodiment. The device can be realized in the form of software and / or hardware, and can generally be integrated into a computer device. As shown in the figure, the method of the present embodiment specifically includes: Figure 1a

[0052] S110, receiving original image data, the original image data having face data therein.

[0053] The original image data includes face data. The original image data can be image data captured and collected by a user in real time, can be local image data specified by a user, or can be image data obtained by intercepting a frame of image in a video, etc. The face data can be real face data or virtual face data.

[0054] S120, determining a key point detection model, the first n layers of networks in the key point detection model being used to output edge texture features, the edge texture features being fused with texture features and edge features, and the edge texture features being used to generate a target heat map for the edges of the face data.

[0055] ​The key point detection model is used to identify the key points of a face in the original image data. The deep learning model is trained to generate the key point detection model. Specifically, the key point detection model can include ShufflenetV2, MobileNet, ShuffleNetV1, or Sception, etc.

[0056] The first n layers of the key point detection model are located at the front of the entire network. The first n layers are used to extract edge texture features in the face image. The network after the first n layers of the key point detection model determines the positions of the key points of the face in the original image data according to the edge texture features, as the key point detection result. Optionally, the first n layers include a convolutional layer and a max pooling layer. Determining the key point detection model is actually training the key point detection model.

[0057] The texture feature describes the surface information of the face in the original image data, and the edge feature describes the edge information of the face in the original image data. In fact, in addition to the edge feature and the texture feature, the first n layers of the network also output at least one of the color feature, the light intensity feature, and the shape feature. The edge texture feature is a combination of the edge feature and the texture feature. Accordingly, the edge texture feature describes the surface information and the edge information of the face. The edge texture feature is also used to generate a target heat map, which is used to describe the edge of the face in the original image data. In the heat map, only the edge key points are highlighted, and the display effect of the edge key points is not obvious. Since the influence of the edge key points on the surrounding pixel points in the heat map decreases as the distance increases, the pixel points that are greatly affected by the edge key points can also be regarded as edge key points in the heat map, that is, the pixel points within a predetermined range around the edge key points are all regarded as edge key points, and the display parameters (such as brightness or color, etc.) of the edge key points are adjusted to the parameter values that are different from the display parameters of the background, to generate a target heat map, so that the edge of the face in the original image data is significantly displayed in the target heat map. The size of the heat map is the same as that of the original image data. For example, as shown in FIG. 1, the left image is the original image data, and the right image is the target heat map matched with the original image data. In the target heat map, the edge of the face in the original image data can be clearly seen. The target heat map can include multiple face boundary lines. The face boundary lines can include at least one of the following: an outer contour, a left eyebrow, a right eyebrow, a nose bridge, a nose boundary, an upper left eyelid, a lower left eyelid, an upper right eyelid, a lower right eyelid, an upper edge of an upper lip, a lower edge of the upper lip, an upper edge of a lower lip, and a lower edge of the lower lip, etc. Figure 1b

[0058] Optionally, in the features output by the first n layers of the key point detection model, the data amount of the edge feature accounts for the largest proportion of the data amount of all the features.

[0059] ​The features output by the first n layers of the network can include edge features, texture features, color features, light intensity features, and shape features. Edge features have the largest data volume, indicating that they are the most important features among the features output by the first n layers of the network. In fact, the training process of the keypoint detection model continuously strengthens the ability of the first n layers of the network to extract edge features. The data volume of edge features output by a trained keypoint detection model is generally larger than that of other features.

[0060] By training the key point detection model, the data volume of edge features extracted by the first n layers of the network in the key point detection model is maximized, which can improve the ability of the first n layers of the network to extract edge features. Therefore, the key point detection model can perform facial key point detection on the original image data based on the edge features that can better represent the edges of the face, which can improve the accuracy of facial key point detection.

[0061] Optionally, the original image data is input into the key point detection model, and the facial key points of the facial data are identified as target key points, so that the target key points are aligned with the target heat map, including: inputting the original image data into the key point detection model, the key point detection model includes a first convolution layer, a pooling layer, a first shuffled network stage layer, a second shuffled network stage layer, a third shuffled network stage layer, a multi-scalar mean series layer and a second convolution layer; in the first convolution layer, a convolution operation is performed on the original image data to obtain initial features; in the pooling layer, a pooling operation is performed on the initial features to obtain edge texture features; in the first shuffled network In the stage layer, feature mapping is performed on the edge texture feature to obtain a first feature; in the second shuffled network stage layer, feature mapping is performed on the first feature to obtain a second feature, and the dimension of the second feature is higher than the dimension of the first feature; in the third shuffled network stage layer, feature mapping is performed on the second feature to obtain a third feature, and the dimension of the third feature is higher than the dimension of the second feature; in the multi-scalar mean series layer, the first feature, the second feature and the third feature are spliced ​​to obtain a fused feature; in the second convolutional layer, a convolution operation is performed on the fused feature to obtain the facial key points of the facial data, and use them as target key points.

[0062] Specifically, such as Figure 1c As shown, the key point detection model 100 structure includes a first convolutional layer 101, a pooling layer 102, a first shuffled network stage layer 103, a second shuffled network stage layer 104, a third shuffled network stage layer 105, a multi-scalar mean concatenation layer 106, and a second convolutional layer 107. Among them, the first convolutional layer 101 and the pooling layer 102 are the first n-layer network 110.

[0063] The convolutional layer is used to extract features, and the pooling layer is used to reduce the dimension of the features extracted by the first convolutional layer to obtain edge texture features. The pooling layer is used to perform maximum pooling on the features. In the trained key point detection model, the features output by the pooling layer are mainly edge features and texture features, and the proportion of other features is very small. Therefore, the features output by the pooling layer are edge texture features. The edge texture features are input to the first shuffle network stage layer, the first shuffle network stage layer is output to the second shuffle network stage layer, and the second shuffle network stage layer is output to the third shuffle network stage layer. The three shuffle network stage layers are all stage blocks of three ShuffleNetv2 networks, wherein each shuffle network stage layer is a convolution operation on the input data to form output data of different sizes. It should be noted that the input data of the convolution is a feature map matrix, the output data of the convolution is an image matrix, and the output data of different sizes is an image matrix of different sizes. The multi scalar mean concaten layer is used to concatenate the outputs of the three shuffle network stage layers, fuse the feature data output by the three shuffle network stage layers, and form a fused feature. The second convolutional layer is used to convolve the concatenated feature data to obtain key points as target key points.

[0064] The first convolutional layer and the maximum pooling layer are used to extract edge texture features using at least one parameter, and the parameter is a parameter in a pre-trained model for generating a heat map. Therefore, the low-dimensional edge texture features extracted using the above pre-trained parameters can well represent the edges of a face. The parameters include the elements in the matrix of the convolution kernel matrix in the first convolutional layer. Subsequent key point detection based on the edge texture features extracted by the above method can improve the alignment of the detected key points with the edges in the heat map, thereby realizing the alignment of the key points with the face contour.

[0065] The ShuffleNetV2 network divides the input feature map into two branches in the channel dimension and concatenates the outputs of the two branches into one feature element. The ShuffleNetV2 network includes a first shuffle network stage layer, a second shuffle network stage layer, a third shuffle network stage layer, and a multi scalar mean concaten layer. Moreover, the ShuffleNetV2 network is a lightweight neural network, specifically a neural network model with fewer parameters and lower computational cost. Using the ShuffleNetV2 network for high-dimensional feature extraction can reduce the computational resource consumption of the key point detection model and improve the efficiency of key point recognition. The dimensions of the first feature, the second feature, and the third feature increase in turn, and the sizes of the associated output feature maps decrease in turn. The third feature has the highest dimension, the output feature map associated with the third feature with the highest dimension has the smallest size, and the first feature with the lowest dimension has the largest size of the output feature map.

[0066] By configuring the key point detection model to include a convolutional layer output edge texture feature, low-dimensional features that significantly represent the edges of the face can be extracted, and then according to the low-dimensional features, the stage block of the ShuffleNetv2 network is further used for feature extraction to form high-dimensional features, which can improve the key point recognition efficiency, and the multi-scalar mean concatenation layer is used to concatenate the information of multiple high-dimensional features to form a fusion feature, reduce the information loss caused by the feature extraction operation, improve the representativeness of the feature data, and finally perform convolution based on the fusion feature to obtain the key point, which can accurately align the face contour and improve the recognition accuracy of the edge key point.

[0067] S130, input the original image data into the key point detection model, identify the face key point of the face data as the target key point, so that the target key point aligns with the target heat map.

[0068] The original image data is input into the key point detection model to obtain the face key point, i.e. the target key point, in the original image data. The target key point can include an edge key point. Specifically, the target key point is all the edge key point of the face.

[0069] The target key point can be aligned with the target heat map according to the standard position range of the edge key point determined by the target heat map, which is actually to continuously approach the standard position range of the edge key point, that is, to supervise and constrain the position of the target key point, to achieve accurate alignment of the contour and improve the accuracy of face edge detection.

[0070] The edge texture feature extracted from the face data in the image by the key point detection model can generate a target heat map for the edge of the face data. The key point detection model identifies the face key point in the face data according to the extracted edge texture feature, aligns the face key point with the target heat map, i.e. aligns the face key point with the edge of the face data, which can improve the recognition accuracy of the face edge key point, solve the problem of inaccurate prediction results of the edge key point in the prior art, improve the prediction results of the edge key point, and improve the alignment accuracy of the face contour.

[0071] Embodiment two

[0072] Figure 2a The flowchart of the training method of the key point detection model in the second embodiment of the present application can be used for training the key point detection model for identifying the face key point in the face image. The method can be executed by the training device of the key point detection model provided by the embodiment of the present application. The device can be realized by software and / or hardware, and generally can be integrated into a computer device. As shown in the figure, Figure 2aThe method of the embodiment shown specifically comprises:

[0073] S210, receiving first sample image data having face data.

[0074] The sample image data is used for training the key point detection model.

[0075] The embodiment of the application not described in detail can refer to the foregoing embodiment.

[0076] S220, inputting the first sample image data into the key point detection model to identify face key points of the face data as predicted key points.

[0077] The key point detection model at this time is an untrained deep learning model.

[0078] S230, extracting first edge texture features output by the first n layers of the key point detection model, the first edge texture features being used for input into a pre-trained edge generation model to generate a heat map for a face edge of the first edge texture features as a first predicted heat map, wherein the first n layers of the network are obtained through joint training with the edge generation model.

[0079] The edge generation model is used for generating a heat map according to edge texture features. The key point detection model is used for identifying face key points in sample image data. The first predicted heat map is used for determining a face edge in sample image data. Since the edge generation model is a trained model, the prediction accuracy of the face edge in the first predicted heat map reaches the training standard, that is, the face edge in the first predicted heat map is close to the real face edge.

[0080] The key point detection model and the edge generation model share the first n layers of the network, the edge generation model and the first n layers of the network can be a heat map generation model for a face edge, and the first n layers of the network and the edge generation model are jointly trained, which can be understood as training the heat map generation model, wherein the first n layers of the network are used as the first n layers of the network in the heat map generation model to detect first edge texture features according to sample image data. At the same time, the edge generation model is used for generating a first predicted heat map for a face edge in sample image data according to the first edge texture features. In fact, the first n layers of the network and the edge generation model are connected in series, and the first n layers of the network and the edge generation model can be jointly trained to obtain the first n layers of the network and the edge generation model trained at the same time. In the joint training process of the first n layers of the network and the edge generation model, the parameters in the first n layers of the network are constantly updated until the training is completed.

[0081] After the pre-n-layer network and the edge generation model are jointly trained, the edge texture features extracted by the pre-n-layer network can more accurately determine the facial edge contour. If the matrix corresponding to the edge texture features is displayed, the edge texture features can more significantly display the facial edge contour compared to before the training.

[0082] In fact, if an existing direct training key point detection model is used to detect key points of sample image data, the predicted key points obtained are not sensitive to facial contour edges and the like, and some predicted key points often cannot accurately align the contour. Since the key point detection model, which is a regression-based network, outputs a key point sequence, the key point sequence as a whole is closer to the standard key point, and is usually not particularly sensitive to whether the local key point fits the contour edge.

[0083] Therefore, the pre-n-layer network and the edge generation model can be jointly trained in advance, and the parameters of the trained pre-n-layer network can be used as the parameters of the pre-n-layer network of the key point detection model to be trained. The parameters can be used to extract features of a face image, which can enhance the response degree of edge texture feature extraction, so that the texture edge features include more features that can represent more rich facial edge contours.

[0084] Optionally, before the first sample image data is input into the key point detection model, the method further includes: obtaining second sample image data, the second sample image data having face data; inputting the second sample image data into the n-layer network to obtain second edge texture features output by the n-layer network; inputting the second edge texture features into a preset edge generation model to generate a heat map for a facial edge of the second sample image data as a second predicted heat map, and training the n-layer network and the edge generation model at the same time; locking parameters in the n-layer network when the n-layer network and the edge generation model are trained, and generating a key point detection model according to the n-layer network, the pre-n-layer network of the key point detection model being the n-layer network.

[0085] The second sample image data is used to train the n-layer network and the edge generation model at the same time. The n-layer network is actually a pre-n-layer network that is not trained. After the n-layer network and the edge generation model are trained, the n-layer network is used as the pre-n-layer network of the key point detection model to construct the key point detection model. The parameters in the n-layer network are locked to ensure that the parameters in the n-layer network are not updated during the training of the key point detection model, so that the edge texture features that can represent the facial edge are obtained during the training of the key point detection model.

[0086] By jointly training the n-layer network and the edge generation model, the parameters of the n-layer network capable of extracting more features capable of representing more rich face edge contours are trained and generated, and the edge generation model for generating the heat map for the face edge is trained and generated, and based on the parameters of the n-layer network, the key point detection model is constructed and trained, so that the trained key point detection model is aligned with the heat map generated by the edge generation model for the face edge, thereby improving the alignment of the predicted key points output by the key point detection model with the face contour, and improving the detection accuracy of the face edge key points.

[0087] Optionally, the second edge texture feature is input into a preset edge generation model to generate a heat map for the face edge of the second sample image data as a second predicted heat map, including: inputting the second edge texture feature into a preset edge generation model, the edge generation model including a third convolutional layer, a first hourglass network, a first message communication layer, a second hourglass network and a second message communication layer; in the third convolutional layer, the second edge texture feature is convolved to obtain a face edge feature; in the first hourglass network, the face edge feature is mapped to obtain a first intermediate heat map; in the first message communication layer, the first intermediate heat map is obtained and sent to the second message communication layer and the second hourglass network; in the second hourglass network, the first intermediate heat map is processed to obtain a second intermediate heat map; in the second message communication layer, the first intermediate heat map and the second intermediate heat map are obtained, and the second intermediate heat map is processed according to the first intermediate heat map to obtain a second predicted heat map.

[0088] Specifically, as shown in Figure 2b The edge generation model 200 structure includes a third convolutional layer 201, a first hourglass network 202, a first message communication layer 203, a second hourglass network 204 and a second message communication layer 205. The first convolutional layer 101 and the pooling layer 102 are the front n-layer network 110.

[0089] The convolutional layer is configured to perform convolution on the input data, and the hourglass network is configured to convert the input data into a heat map. The message passing layer (MPL) is configured to transmit data in the edge generation model. The intermediate heat map is the heat map output by the hourglass network. The number of hourglass networks can be increased to improve the recognition accuracy of the facial edge in the output heat map. The first message passing layer can transmit the intermediate result, i.e., the first intermediate heat map, to the second message passing layer, and the second intermediate heat map output by the second hourglass network is analyzed and processed to obtain the second predicted heat map, so that the second predicted heat map comprehensively analyzes the edge features of the first intermediate heat map and the edge features of the second intermediate heat map, reduces the feature loss, and improves the edge accuracy in the second predicted heat map.

[0090] The hourglass network includes a convolutional layer and a deconvolutional layer, or the hourglass network includes an encoder and a decoder. In addition, the number of hourglass networks can be one, three, four or more. When the number of hourglass networks is at least two, each hourglass network is followed by a message passing layer.

[0091] By configuring multiple hourglass networks and message passing layers as the edge generation model, the facial edge features can be accurately extracted, and the heat map of the accurate facial edge can be formed, so as to improve the extraction accuracy of the edge features of the n-layer network when training the n-layer network and the edge generation model at the same time.

[0092] Optionally, the simultaneous training of the n-layer network and the edge generation model includes: determining the facial key points marked in advance in the second sample image data as the standard key points matched by the second sample image data; using the standard key points matched by the second sample image data to generate a heat map of the facial edge of the second sample image data as a standard heat map; calculating the difference between the standard heat map and the second predicted heat map as a second sub-loss value; determining whether the change rate of the second sub-loss value is less than a preset second threshold value; if yes, determining that the simultaneous training of the n-layer network and the edge generation model is completed; if no, updating the parameters in the n-layer network and the parameters in the edge generation model, and returning to execute the inputting of the second sample image data into the n-layer network to obtain the second edge texture features output by the n-layer network.

[0093] The standard key points are accurate facial key points marked in advance. The (x, y) coordinate information file can be used to store the coordinate positions of the standard key points. For example, as shown in FIG. 2, the second sample image data is a facial image, and the standard key points are the coordinates of the facial key points in the facial image. Figure 2c Figure 2c ​The points in the standard heat map are standard key points, and the numbers next to the points are used to identify the standard key points, and different numbers represent different standard key points. Among them, the standard heat map formed by the standard key points can be used as the true value corresponding to the prediction result. According to the standard key points, the line method can be used to connect the points into a line, and the inflation and corrosion method can be used to make the line smoother to realize natural transition. And add the generated line to the background image to generate a standard boundary heat map. Among them, the background image of the standard boundary heat map and the predicted boundary heat map can be the same.

[0094] The second sub-loss value is used to evaluate the difference between the predicted value and the true value of the heat map in the edge generation model, wherein the image matrix corresponding to the standard heat map is the true value, and the image matrix corresponding to the predicted heat map is the predicted value. If the second sub-loss value is larger, it means that the prediction result of the edge generation model is less accurate; if the second sub-loss value is smaller, it means that the prediction result of the edge generation model is more accurate, and the n-layer network and the edge generation model are more in line with the expected training effect. Generally, within the range of the set training times, the second sub-loss value gradually decreases as the training times increase. In fact, the n-layer network and the edge generation model are in a series connection relationship, the accuracy of the prediction result of the edge generation model, the output of the n-layer network, is used as the input of the edge network generation model. Therefore, the n-layer network and the edge generation model can be regarded as a model, and the accuracy of the prediction result of the edge generation model can be regarded as the accuracy of the prediction result of the model, so that the accuracy of the prediction result of the edge generation model can indicate whether the edge texture features extracted by the n-layer network are accurate.

[0095] The change rate of the second sub-loss value is used to evaluate the change rate of the second sub-loss value. The second threshold is used to determine whether the n-layer network and the edge generation model are trained, specifically whether the second sub-loss value is stable or whether the change is very small. If the change rate of the second sub-loss value is fast, it means that the first sub-loss value will change with the training times; if the change rate of the second sub-loss value is slow, it means that the second sub-loss value gradually tends to be stable, and it can be considered that the n-layer network and the edge generation model are trained.

[0096] When not trained, updating the parameters of the n-layer network and the edge generation model can mean adjusting the parameters of the n-layer network and the edge generation model and continuing the next training.

[0097] By calculating the second sub-loss value and determining whether the n-layer network and the edge generation model are trained according to whether the second sub-loss value is stable or not, the prediction accuracy of the trained edge generation model can be improved, and the richness of the edge texture features of the trained n-layer network and the representativeness of the face edge can be improved.

[0098] Optionally, the computing the difference between the standard heat map and the second predicted heat map as a second sub-loss value comprises: generating a standard heat map matrix according to the standard heat map; and generating a predicted heat map matrix according to the second predicted heat map.

[0099] According to the standard heat map matrix and the predicted heat map matrix, a second sub-loss value LOSS r :

[0100] LOSS g =‖M e -N e ‖ 2

[0101] wherein Me is the predicted heat map matrix, Ne is the standard heat map matrix, and ||*|| is the Euclidean distance between two matrices. 2

[0102] By computing the Euclidean distance between the predicted heat map matrix and the standard heat map, the difference between the predicted value and the true value of the face key point can be accurately described, and as a training target, the prediction accuracy of the edge generation model for the heat map of the face edge can be improved.

[0103] In addition, after the n-layer network training is completed, the n-layer network generated key point detection model can be directly extracted, and the edge generation model can be deleted, without affecting the key point recognition process of the key point detection model. In fact, a separate edge generation model is used to learn the features of the face edge together with the n-layer network, and the heat map prediction result of the edge generation model for the face edge is not used as input by the key point detection model. The key point detection model only uses the trained n-layer network, and trains other networks in the key point detection model according to the trained n-layer network, which can help the input features of the key point detection model to focus on the edge information extraction, increase the response of the edge features in the low-dimensional features, and thus help the finally output key points to fall on the strong texture edge information, rather than on the existing experience value, so as to improve the prediction result of the face edge key point of the key point detection model in the case of increasing the parameter of the n-layer network for the edge feature extraction range and richness. Therefore, the edge generation model and the key point detection model are independent of each other. For the model, one more network will cause one more network prediction time-consuming and memory occupation. No matter on the mobile end or the server end, the smaller the model is, the better, the less time-consuming it is, and the smaller the CPU occupation is. Therefore, the configured key point detection model does not include the edge generation model, which can reduce the time consumption and memory consumption of the key point detection model, and improve the key point detection efficiency.

[0104] ​The key point detection model can be deployed on a mobile terminal, such as a mobile phone, a tablet computer, or a vehicle terminal. Optionally, the mobile terminal receives a face image processing request of a user, and obtains a target face image; the mobile terminal inputs the target face image into the key point detection model, and obtains face key points output by the key point detection model; and the mobile terminal performs image processing on the face key points according to the face image processing request. The image processing can include beauty processing and / or dynamic special effect processing.

[0105] S240, training a network after the first n layers of the key point detection model, with the first predicted heat map and the predicted key points being aligned as supervision.

[0106] Aligning the first predicted heat map and the predicted key points can mean that the predicted key points are constantly close to the matching edge key points in the first predicted heat map, that is, the coordinate position prediction result of the predicted key points in the sample image is constantly close to the predicted position of the edge key points matched with the first predicted heat map.

[0107] Training the network after the first n layers of the key point detection model actually means locking the parameters of the first n layers of the key point detection model which have been trained in advance. Thus, training the key point detection model only trains the network of the key point detection model other than the first n layers. Moreover, the first n layers are the frontmost network of the key point detection model. Training the network of the key point detection model other than the first n layers is equivalent to training the network after the first n layers of the key point detection model.

[0108] Training the network after the first n layers of the key point detection model with the first predicted heat map and the predicted key points being aligned as supervision is used to train the key point detection model, so that the predicted key points output by the key point detection model are constantly aligned with the edge key points matched with the first predicted heat map.

[0109] Optionally, the training the network after the first n layers of the key point detection model with the first predicted heat map and the predicted key points being aligned as supervision includes: determining the face key points labeled in the first sample image data in advance as standard key points matched with the first sample image data; calculating a difference between the standard key points matched with the first sample image data and the predicted key points as a first sub-loss value; judging whether a change rate of the first sub-loss value is less than a preset first threshold value; if yes, determining that the training of the key point detection model is completed; and if no, updating parameters of the network after the first n layers of the key point detection model, and returning to input the first sample image data into the key point detection model to identify face key points of the face data as predicted key points.

[0110] The standard key points are pre-labeled accurate face key points. The coordinate positions of the standard key points can be used as the true values corresponding to the prediction results. The first sub-loss value is used to evaluate the difference between the predicted values and the true values of the key points in the key point detection model, wherein the coordinate positions of the standard key points are the true values, and the coordinate positions of the predicted key points are the predicted values. If the first sub-loss value is larger, it indicates that the prediction result of the key point detection model is less accurate. If the first sub-loss value is smaller, it indicates that the prediction result of the key point detection model is more accurate, and thus the training effect expected is more likely to be achieved. Generally, within the range of the set training times, the first sub-loss value gradually decreases with the increase of the training times.

[0111] The change rate of the first sub-loss value is used to evaluate the change rate of the first sub-loss value. The first threshold is used to determine whether the training of the key point detection model is completed, specifically, whether the first sub-loss value is stable or has a very small change. If the change rate of the first sub-loss value is fast, it indicates that the first sub-loss value will change with the training times. If the change rate of the first sub-loss value is slow, it indicates that the first sub-loss value gradually approaches stability, and it can be considered that the training of the key point detection model is completed.

[0112] When the training is not completed, updating the parameters of the network after the first n layers of the key point detection model can mean adjusting the parameters of the network after the first n layers of the key point detection model and continuing the next training.

[0113] By calculating the first sub-loss value and determining whether the training of the key point detection model is completed according to whether the first sub-loss value is stable or changes, the prediction accuracy of the key point detection model generated by the training can be improved.

[0114] Optionally, the difference between the standard face key points and the predicted face key points is calculated as the first sub-loss value, which includes: calculating the first sub-loss value LOSS according to the standard face key points matched by the first sample image data and the predicted face key points according to the following formula r :

[0115]

[0116] wherein Pi is the coordinate of the predicted face key point, Li is the coordinate of the standard face key point matched by the first sample image data, and K is the total number of key points.

[0117] Specifically, Pi=(x'i, y'i) and Li=(xi, yi). Wherein (x'i, y'i) is the coordinate of the predicted face key point, (xi, yi) is the coordinate of the standard face key point matched by the first sample image data, and K is the total number of coordinates of the predicted face key points or the standard face key points.

[0118] By calculating the square difference between all predicted face key points and the matching face standard key, the difference between the predicted value and the true value of the face key point can be accurately described, and as a training target, the detection accuracy of the key point detection model can be improved.

[0119] The edge generation model and the first n-layer network are jointly trained, the response of the parameters obtained by training in the first n-layer network to the edge feature is enhanced, the richness of the edge feature in the edge texture feature is improved, and after the first n-layer network is trained, a key point detection model is constructed, and the first predicted heat map and the predicted key point are aligned as supervision, and the network after the first n-layer network in the key point detection model is trained using sample image data, so that the generated key point detection model can detect the face key point detection accuracy and the face edge key point detection accuracy, so that the recognition accuracy of the face edge key point in the key point detection model can be improved, the prediction result of the edge key point can be improved, and the alignment accuracy of the face contour can be improved.

[0120] Embodiment three

[0121] Figure 3a For the flowchart of the live method in the embodiment three of the application, the embodiment can be applied to the case that the pre-trained key point detection model is used for face key point recognition of the collected face image of the anchor user when the anchor user performs live broadcast, the method can be executed by the live device provided by the embodiment of the application, the device can be realized in software and / or hardware, and can be generally integrated into a computer device, which can include a server or a mobile terminal. As shown in FIG. 3, the method of the embodiment specifically includes:

[0122] S310, collecting video data, the original image data in the video data has face data representing an anchor user.

[0123] The embodiments of the application not described in detail can refer to the foregoing embodiments.

[0124] The video data can collect the data formed by the face image of the anchor user.

[0125] S320, inputting the original image data into a preset key point detection model, recognizing face key points of the face data as target key points, so that the target key points are aligned with a target heat map, the first n-layer network in the key point detection model is used for outputting edge texture features, the edge texture features are fused with texture features and edge features, and are used for generating a target heat map for the edge of the face data.

[0126] For example, the key points recognized in the existing face image are as shown in FIG. 8A. Figure 3b As shown in FIG. 8B, the heat map output by the trained edge generation model is as shown in FIG. 8C. Figure 1b The n-layer network is trained, and the n-layer network parameters are locked, and the key point detection model is trained. As can be seen, the n-layer network is independently trained, and the key point detection model is constructed based on the trained n-layer network. The key points of the predicted face edge of the trained key point detection model are more consistent with the contour boundary of the face.

[0127] S330, according to the target key point, the original image data is processed, and the target image data is obtained.

[0128] The business processing can include beautifying processing, face driving, virtual live broadcast, dynamic special effect processing (such as adding special effect images in the image), and face changing processing.

[0129] For example, the position and color parameter value of each target key point are adjusted for beautifying, and the processed target image data is generated. Or according to the preset standard face area, the position and color parameter value of each target key point are adjusted to generate a replacement face area; the face image is updated according to the replacement face area, and the processed face image is generated as the target image data.

[0130] S340, in the live broadcast room opened by the anchor user, the video data containing the target image data is published.

[0131] The target image data published in the live broadcast room can be sent to the live broadcast client used by the audience user entering the live broadcast room.

[0132] The embodiment of the application improves the recognition accuracy of the key points of the face of the anchor user by using the key point detection model that strengthens the extraction of edge texture features to recognize the key points of the face data of the anchor user, and processes the business based on the recognized key points to obtain the target image data and publish the target image data, which can improve the processing accuracy of the face of the anchor user in the anchor room and realize accurate processing of the details of the face of the anchor user.

[0133] Embodiment four

[0134] Figure 4 FIG. 1 is a schematic diagram of a key point detection device according to an embodiment of the application. The device of the embodiment four is a corresponding device for implementing the key point detection method provided in the above embodiments of the application, and can be realized in the form of software and / or hardware, and can be integrated in a computer device.

[0135] Correspondingly, the device of the embodiment can include:

[0136] The original image data acquisition module 410 is configured to receive original image data, wherein the original image data comprises face data;

[0137] The key point detection model determination module 420 is configured to determine a key point detection model, wherein the first n layers of the key point detection model are configured to output edge texture features, the edge texture features are fused with texture features and edge features, and the edge texture features are configured to generate a target heat map for edges of the face data.

[0138] The face key point recognition module 430 is configured to input the original image data into the key point detection model, recognize face key points of the face data as target key points, and align the target key points to the target heat map.

[0139] The edge texture features extracted from the face data in the image by the key point detection model can generate a target heat map for the edges of the face data, the key point detection model recognizes the face key points in the face data according to the extracted edge texture features, aligns the face key points to the target heat map, and realizes the alignment of the face key points to the edges of the face data, which can improve the recognition accuracy of the face edge key points, solve the problem of inaccurate prediction results of the edge key points in the prior art, improve the prediction results of the edge key points, and improve the alignment accuracy of the face contour.

[0140] Further, the face key point recognition module 430 comprises a key point detection unit configured to input the original image data into the key point detection model, wherein the key point detection model comprises a first convolution layer, a pooling layer, a first shuffle network stage layer, a second shuffle network stage layer, a third shuffle network stage layer, a multi-scalar mean concatenation layer, and a second convolution layer; the original image data is subjected to a convolution operation in the first convolution layer to obtain initial features; the initial features are subjected to a pooling operation in the pooling layer to obtain edge texture features; the edge texture features are subjected to feature mapping in the first shuffle network stage layer to obtain first features; the first features are subjected to feature mapping in the second shuffle network stage layer to obtain second features, wherein the dimension of the second features is higher than that of the first features; the second features are subjected to feature mapping in the third shuffle network stage layer to obtain third features, wherein the dimension of the third features is higher than that of the second features; the first features, the second features, and the third features are spliced in the multi-scalar mean concatenation layer to obtain fused features; and the fused features are subjected to a convolution operation in the second convolution layer to obtain face key points of the face data as target key points.

[0141] Further, in the features output by the first n layers of the key point detection model, the data amount of the edge feature accounts for the largest proportion of the data amount of all features.

[0142] The device can perform the key point detection method provided by the embodiments of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0143] Embodiment five

[0144] Figure 5 FIG. 1 is a schematic diagram of a device for training a key point detection model according to an embodiment of the application. The device of embodiment five is a device for implementing the training method of the key point detection model provided by the above-mentioned embodiments of the application. The device can be implemented in the form of software and / or hardware, and can generally be integrated into a computer device.

[0145] Correspondingly, the device of the embodiment can include:

[0146] The first sample image data acquisition module 510 is configured to receive first sample image data, wherein the first sample image data contains face data.

[0147] The face key point recognition module 520 is configured to input the first sample image data into the key point detection model to recognize face key points of the face data as predicted key points.

[0148] The first edge texture feature extraction module 530 is configured to extract first edge texture features output by the first n layers of the key point detection model, and input the first edge texture features into a pre-trained edge generation model to generate a heat map for the face edge of the first edge texture features as a first predicted heat map. The first n layers of the key point detection model are obtained by co-training with the edge generation model.

[0149] The key point detection model training module 540 is configured to train the network after the first n layers of the key point detection model under the supervision of aligning the first predicted heat map with the predicted key points.

[0150] The embodiment of the present application trains the edge generation model and the first n-layer network together, enhances the response of the parameters trained in the first n-layer network to the edge features, improves the richness of the edge features in the edge texture features, and after the training of the first n-layer network is completed, constructs a key point detection model, and takes the alignment of the first predicted heat map and the predicted key points as supervision, trains the network after the first n-layer network in the key point detection model using sample image data, can obtain more and richer edge texture features, so that the generated key point detection model can detect the face key points and the edge key points of the face to meet the face key point detection accuracy and the face edge key point detection accuracy, thereby improving the recognition accuracy of the edge key points of the face in the key point detection model, improving the prediction result of the edge key points, and improving the alignment accuracy of the face contour.

[0151] Further, the training device of the key point detection model further includes: an n-layer network and edge generation model co-training module, configured to, before inputting the first sample image data into the key point detection model, further include: obtaining second sample image data, the second sample image data having face data; inputting the second sample image data into the n-layer network to obtain second edge texture features output by the n-layer network; inputting the second edge texture features into a preset edge generation model to generate a heat map for the face edge of the second sample image data as a second predicted heat map, and training the n-layer network and the edge generation model at the same time; when the training of the n-layer network and the edge generation model is completed, locking the parameters in the n-layer network, and generating a key point detection model according to the n-layer network, the first n-layer network in the key point detection model being the n-layer network.

[0152] Further, the n-layer network and edge generation model co-training module includes: an edge generation model predicted heat map unit, configured to input the second edge texture features into a preset edge generation model, the edge generation model including a third convolutional layer, a first hourglass network, a first message communication layer, a second hourglass network and a second message communication layer; in the third convolutional layer, the second edge texture features are subjected to convolution operation to obtain face edge features; in the first hourglass network, the face edge features are subjected to feature mapping to obtain a first intermediate heat map; in the first message communication layer, the first intermediate heat map is obtained and sent to the second message communication layer and the second hourglass network; in the second hourglass network, the first intermediate heat map is processed to obtain a second intermediate heat map; in the second message communication layer, the first intermediate heat map and the second intermediate heat map are obtained, and the second intermediate heat map is processed according to the first intermediate heat map to obtain a second predicted heat map.

[0153] Further, the key point detection model training module 540 comprises a key point detection model training completion judging unit configured to determine the face key points marked in advance in the first sample image data as the standard key points matched by the first sample image data, calculate the difference between the standard key points matched by the first sample image data and the predicted key points as a first sub-loss value, judge whether the change rate of the first sub-loss value is less than a preset first threshold, if yes, determine that the key point detection model training is completed, and if no, update the parameters of the network after the first n layers of the key point detection model, and return to input the first sample image data into the key point detection model to identify the face key points of the face data as the predicted key points.

[0154] Further, the key point detection model training completion judging unit comprises a first sub-loss value calculation sub-unit configured to calculate the first sub-loss value LOSS according to the standard face key points matched by the first sample image data and the predicted face key points according to the following formula: r :

[0155]

[0156] wherein Pi is the coordinate of the predicted face key point, Li is the coordinate of the standard face key point matched by the first sample image data, and K is the total number of key points.

[0157] Further, the n-layer network and edge generation model joint training module comprises an n-layer network and edge generation model training completion judging unit configured to determine the face key points marked in advance in the second sample image data as the standard key points matched by the second sample image data, use the standard key points matched by the second sample image data to generate a standard heat map for the face edge of the second sample image data, calculate the difference between the standard heat map and the second predicted heat map as a second sub-loss value, judge whether the change rate of the second sub-loss value is less than a preset second threshold, if yes, determine that the n-layer network and the edge generation model are simultaneously trained to be completed, and if no, update the parameters in the n-layer network and the parameters in the edge generation model, and return to input the second sample image data into the n-layer network to obtain the second edge texture feature output by the n-layer network.

[0158] Further, the n-layer network and edge generation model joint training module comprises a second sub-loss value calculation sub-unit configured to generate a standard heat map matrix according to the standard heat map by the n-layer network and edge generation model training completion judging unit, and generate a predicted heat map matrix according to the second predicted heat map.

[0159] According to the following formula, a second sub-loss value LOSS is calculated according to the standard heat map matrix and the predicted heat map matrix r :

[0160] LOSS g =‖M e -N e ‖ 2

[0161] wherein Me is the predicted heat map matrix, Ne is the standard heat map matrix, and ||*|| is the Euclidean distance between two matrices. 2

[0162] The device can perform the training method of the key point detection model provided in the embodiments of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0163] Embodiment six

[0164] Figure 6 Fig. 1 is a schematic diagram of a live broadcast device according to an embodiment of the application. Embodiment six is a device for implementing the key point detection method provided in the above embodiments of the application, which can be implemented in software and / or hardware, and can generally be integrated into a computer device.

[0165] Correspondingly, the device of the embodiment can include:

[0166] The video data acquisition module 610 is configured to acquire video data, wherein the original image data in the video data has face data representing a host user;

[0167] The live broadcast face key point recognition module 620 is configured to input the original image data into a preset key point detection model, recognize face key points of the face data as target key points, so that the target key points are aligned with a target heat map, and the first n layers of the key point detection model are used to output edge texture features, wherein the edge texture features are fused with texture features and edge features, and are used to generate a target heat map for the edges of the face data.

[0168] The face image processing module 630 is configured to perform service processing on the original image data according to the target key points, and obtain target image data.

[0169] The target image data publishing module 640 is configured to publish video data containing the target image data in a live broadcast room opened by the host user.

[0170] ​The embodiment of the present application improves the recognition accuracy of the key points of the face of the anchor user by performing key point detection on the face data of the anchor user collected, and performs business processing based on the recognized key points to obtain target image data and publish the target image data, so that the processing precision of the face of the anchor user between anchors is improved, and the accurate processing of the face details of the anchor user is realized.

[0171] The device can perform the live broadcast method provided by the embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0172] Embodiment seven

[0173] Figure 7 A structural schematic diagram of a computer device provided by the seventh embodiment of the present application. Figure 7 A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present application is shown. Figure 7 The computer device 12 shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0174] As shown in Figure 7 The computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 can include but are not limited to one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components, including system memory 28 and processing unit 16. The computer device 12 can be a device connected to the bus.

[0175] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0176] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0177] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a Compact Disc Read-Only Memory (CD-ROM), a Digital Video Disc-Read Only Memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0178] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0179] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN)) through a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via the bus 18. It should be understood that although Figure 7Other hardware and / or software modules can be used in conjunction with computer device 12, as shown, including, but not limited to, microcode, device drivers, redundant processing units, redundant arrays of inexpensive disks (RAID) systems, tape drives, and data backup storage systems, etc.

[0180] Processing unit 16 performs various function applications and data processing by running programs stored in system memory 28, such as implementing the key point detection method, the key point detection model training method, or the live streaming method provided by any embodiment of the present application.

[0181] Embodiment eight

[0182] Embodiment eight of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided by all embodiments of the present application:

[0183] That is, the program is executed by the processor to implement: receiving original image data, the original image data having face data; determining a key point detection model, the first n layers of the key point detection model being used to output edge texture features, the edge texture features being fused with texture features and edge features, and the edge texture features being used to generate a target heat map for edges of the face data; inputting the original image data into the key point detection model to identify face key points of the face data as target key points, so that the target key points are aligned with the target heat map.

[0184] Or the program is executed by the processor to implement: receiving first sample image data, the first sample image data having face data; inputting the first sample image data into the key point detection model to identify face key points of the face data as predicted key points; extracting first edge texture features output by the first n layers of the key point detection model, the first edge texture features being used to be input into a pre-trained edge generation model to generate a heat map for edges of the face data as a first predicted heat map, wherein the first n layers of the key point detection model are obtained by co-training with the edge generation model; and training a network after the first n layers of the key point detection model with the first predicted heat map and the predicted key points being aligned as supervision.

[0185] Or the program is implemented when the processor executes: collect video data, the original image data in the video data has face data representing the anchor user; input the original image data into the preset key point detection model, identify the face key point of the face data as the target key point, so that the target key point aligns the target heat map, the first n layers of network in the key point detection model are used to output edge texture features, the edge texture features are fused with texture features and edge features, and the target heat map is generated for the edge of the face data; according to the target key point, the original image data is processed, and target image data is obtained; the video data containing the target image data is published in the live room opened by the anchor user.

[0186] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include: electrical connections having one or more wires, portable computer disks, hard disks, RAM, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fibers, portable CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0187] The computer readable signal medium can include a data signal propagating in a baseband or as part of a carrier wave, in which computer readable program code is carried. Such a propagated data signal can take on many forms, including but not limited to electro-magnetic signal, optical signal or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can transmit, propagate or transport program for use by or in connection with an instruction execution system, device or apparatus.

[0188] The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination thereof.

[0189] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a LAN or a WAN, or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0190] It is to be understood that the above description is directed to the preferred embodiments and that those skilled in the art will be able to devise various modifications which, although not specifically described herein, embody the principles of the application and are included within the spirit and scope of the application. Accordingly, while the preferred embodiments have been described above, those skilled in the art will understand that they are not to be limited to the preferred embodiments, but are to include all such embodiments falling within the scope of the application as defined by the appended claims.

Claims

1. A key point detection method, characterized in that: include: receiving original image data, wherein the original image data includes face data; Determine a key point detection model, wherein the first n layers of the network in the key point detection model are used to output edge texture features, wherein the edge texture features are a fusion of texture features and edge features, and the edge texture features are used to input into a pre-trained edge generation model to generate a target heat map for the edges of the facial data of the edge texture features; Inputting the original image data into the key point detection model, identifying the facial key points of the facial data as target key points, so that the target key points are aligned with the target heat map; The key point detection model includes a first convolutional layer, a pooling layer, a first shuffled network stage layer, a second shuffled network stage layer, a third shuffled network stage layer, a multi-scalar mean series layer, and a second convolutional layer; wherein the first convolutional layer and the pooling layer are the first n-layer network; the multi-scalar mean series layer is used to concatenate the outputs of the three shuffled network stage layers; The first n layers of the network are obtained by training together with the edge generation model, and the parameters of the trained first n layers of the network are used as the parameters of the first n layers of the network in the key point detection model to be trained.

2. The method according to claim 1, characterized in that The step of inputting the original image data into the key point detection model and identifying the facial key points of the facial data as target key points includes: Inputting the original image data into the key point detection model; In the first convolutional layer, a convolution operation is performed on the original image data to obtain initial features; In the pooling layer, a pooling operation is performed on the initial features to obtain edge texture features; In the first shuffle network stage layer, feature mapping is performed on the edge texture feature to obtain a first feature; In the second shuffle network stage layer, feature mapping is performed on the first feature to obtain a second feature, where the dimension of the second feature is higher than the dimension of the first feature; In the third shuffle network stage layer, feature mapping is performed on the second feature to obtain a third feature, where the dimension of the third feature is higher than the dimension of the second feature; In the multi-scalar mean concatenation layer, the first feature, the second feature, and the third feature are concatenated to obtain a fused feature; In the second convolutional layer, a convolution operation is performed on the fused features to obtain facial key points of the facial data, which are used as target key points.

3. The method according to claim 1, characterized in that Among the features output by the first n layers of the network in the key point detection model, the data volume of the edge features accounts for the largest proportion of the data volume of all features.

4. A training method for a key point detection model, characterized in that: include: receiving first sample image data, wherein the first sample image data includes face data; Inputting the first sample image data into the key point detection model to identify facial key points of the facial data as predicted key points; Extracting the first edge texture feature output by the first n-layer network in the key point detection model, the first edge texture feature is used to input into a pre-trained edge generation model to generate a heat map of the face edge for the first edge texture feature as a first predicted heat map, wherein the first n-layer network is obtained by training together with the edge generation model, and the parameters of the trained first n-layer network are used as the parameters of the first n-layer network in the key point detection model to be trained; the key point detection model includes a first convolutional layer, a pooling layer, a first shuffled network stage layer, a second shuffled network stage layer, a third shuffled network stage layer, a multi-scalar mean series layer and a second convolutional layer; wherein the first convolutional layer and the pooling layer are the first n-layer network; the multi-scalar mean series layer is used to concatenate the outputs of the three shuffled network stage layers; The network after the first n layers of the network in the key point detection model is trained using the alignment of the first predicted heat map with the predicted key points as supervision.

5. The method according to claim 4, characterized in that Before inputting the first sample image data into the key point detection model, the method further includes: Acquire second sample image data, wherein the second sample image data contains face data; Inputting the second sample image data into an n-layer network to obtain a second edge texture feature output by the n-layer network; Inputting the second edge texture feature into a preset edge generation model to generate a heat map for the face edge of the second sample image data as a second predicted heat map, and simultaneously training the n-layer network and the edge generation model; When the training of the n-layer network and the edge generation model is completed, the parameters in the n-layer network are locked, and a key point detection model is generated based on the n-layer network, where the first n layers of the network in the key point detection model are the n-layer network.

6. The method according to claim 5, characterized in that Inputting the second edge texture feature into a preset edge generation model to generate a heat map for the face edge of the second sample image data as a second predicted heat map includes: Inputting the second edge texture feature into a preset edge generation model, wherein the edge generation model includes a third convolutional layer, a first hourglass network, a first message communication layer, a second hourglass network, and a second message communication layer; In the third convolutional layer, a convolution operation is performed on the second edge texture feature to obtain a face edge feature; In the first hourglass network, feature mapping is performed on the edge features of the face to obtain a first intermediate heat map; In the first message communication layer, obtaining the first intermediate heat map, and sending the first intermediate heat map to the second message communication layer and the second hourglass network; In the second hourglass network, the first intermediate heat map is processed to obtain a second intermediate heat map; In the second message communication layer, the first intermediate heat map and the second intermediate heat map are obtained, and the second intermediate heat map is processed according to the first intermediate heat map to obtain a second predicted heat map.

7. The method according to claim 4, characterized in that The training of the network after the first n layers of the network in the key point detection model by aligning the first predicted heat map with the predicted key points as supervision includes: Determining facial key points pre-marked in the first sample image data as standard key points for matching the first sample image data; Calculating a difference between the standard key point matched by the first sample image data and the predicted key point as a first sub-loss value; Determining whether a rate of change of the first sub-loss value is less than a preset first threshold; If yes, it is determined that the key point detection model training is completed; If not, update the parameters of the network after the first n layers of the network in the key point detection model, and return to the step of inputting the first sample image data into the key point detection model to identify the facial key points of the facial data as predicted key points.

8. The method according to claim 7, characterized in that The calculating, as a first sub-loss value, a difference between the standard key point matched by the first sample image data and the predicted key point includes: According to the following formula, the first sub-loss value is calculated based on the standard facial key points and the predicted facial key points matched by the first sample image data: : Wherein, Pi is the coordinate of the predicted facial key points, Li is the coordinate of the standard facial key points matched by the first sample image data, and K is the total number of key points.

9. The method according to claim 5, characterized in that The simultaneously training the n-layer network and the edge generation model includes: determining facial key points pre-marked in the second sample image data as standard key points for matching the second sample image data; generating a heat map of the face edge of the second sample image data using the standard key points matched by the second sample image data as a standard heat map; Calculating the difference between the standard heat map and the second predicted heat map as a second sub-loss value; Determining whether the rate of change of the second sub-loss value is less than a preset second threshold; If so, it is determined that the n-layer network and the edge generation model are trained simultaneously; If not, update the parameters in the n-layer network and the parameters in the edge generation model, return to the step of inputting the second sample image data into the n-layer network, and obtain the second edge texture feature output by the n-layer network.

10. The method according to claim 9, characterized in that The calculating the difference between the standard heat map and the second predicted heat map as a second sub-loss value includes: Generating a standard heat map matrix according to the standard heat map; generating a prediction heat map matrix according to the second prediction heat map; According to the following formula, the second sub-loss value is calculated based on the standard heat map matrix and the predicted heat map matrix: : Among them, Me is the predicted heat map matrix, Ne is the standard heat map matrix, ||*|| 2 is the Euclidean distance between the two matrices.

11. A live broadcast method, characterized in that: include: Collecting video data, wherein original image data in the video data includes facial data representing a host user; Inputting the original image data into a preset key point detection model, identifying facial key points of the facial data as target key points, so that the target key points are aligned with the target heat map, the first n layers of the network in the key point detection model are used to output edge texture features, the edge texture features are integrated with texture features and edge features, and the edge texture features are used to input into a pre-trained edge generation model to generate a target heat map for the edges of the facial data with the edge texture features; Performing business processing on the original image data according to the target key points to obtain target image data; Publishing video data containing the target image data in a live broadcast room opened by the anchor user; The key point detection model includes a first convolutional layer, a pooling layer, a first shuffled network stage layer, a second shuffled network stage layer, a third shuffled network stage layer, a multi-scalar mean series layer, and a second convolutional layer; wherein the first convolutional layer and the pooling layer are the first n-layer network; the multi-scalar mean series layer is used to concatenate the outputs of the three shuffled network stage layers; The first n layers of the network are obtained by training together with the edge generation model, and the parameters of the trained first n layers of the network are used as the parameters of the first n layers of the network in the key point detection model to be trained.

12. A key point detection device, characterized in that: include: An original image data acquisition module, configured to receive original image data, wherein the original image data includes face data; A key point detection model determination module is used to determine a key point detection model, wherein the first n layers of the network in the key point detection model are used to output edge texture features, wherein the edge texture features are a fusion of texture features and edge features, and the edge texture features are used to input into a pre-trained edge generation model to generate a target heat map for the edges of the facial data of the edge texture features; A facial key point recognition module is configured to input the original image data into the key point detection model, identify facial key points of the facial data as target key points, and align the target key points with the target heat map; The key point detection model includes a first convolutional layer, a pooling layer, a first shuffled network stage layer, a second shuffled network stage layer, a third shuffled network stage layer, a multi-scalar mean series layer, and a second convolutional layer; wherein the first convolutional layer and the pooling layer are the first n-layer network; the multi-scalar mean series layer is used to concatenate the outputs of the three shuffled network stage layers; The first n layers of the network are obtained by training together with the edge generation model, and the parameters of the trained first n layers of the network are used as the parameters of the first n layers of the network in the key point detection model to be trained.

13. A training device for a key point detection model, characterized in that: include: A first sample image data acquisition module is configured to receive first sample image data, wherein the first sample image data includes face data; a facial key point recognition module, configured to input the first sample image data into the key point detection model to identify facial key points of the facial data as predicted key points; A first edge texture feature extraction module is used to extract the first edge texture feature output by the first n-layer network in the key point detection model, and the first edge texture feature is used to input into a pre-trained edge generation model to generate a heat map of the face edge of the first edge texture feature as a first predicted heat map, wherein the first n-layer network is obtained by training together with the edge generation model, and the parameters of the trained first n-layer network are used as the parameters of the first n-layer network in the key point detection model to be trained; the key point detection model includes a first convolutional layer, a pooling layer, a first shuffled network stage layer, a second shuffled network stage layer, a third shuffled network stage layer, a multi-scalar mean series layer and a second convolutional layer; wherein the first convolutional layer and the pooling layer are the first n-layer network; the multi-scalar mean series layer is used to concatenate the outputs of the three shuffled network stage layers; A key point detection model training module is used to train the network after the first n layers of the network in the key point detection model by aligning the first predicted heat map with the predicted key points as supervision.

14. A live broadcast device, characterized in that: include: A video data acquisition module is used to acquire video data, wherein the original image data in the video data contains facial data representing the anchor user; A live facial key point recognition module is configured to input the original image data into a preset key point detection model, identify the facial key points of the facial data as target key points, and align the target key points with a target heat map. The first n layers of the network in the key point detection model are configured to output edge texture features, which are a fusion of texture features and edge features. The edge texture features are configured to be input into a pre-trained edge generation model to generate a target heat map for the edges of the facial data with the edge texture features. A face image processing module is used to perform business processing on the original image data according to the target key points to obtain target image data; A target image data publishing module, configured to publish video data containing the target image data in the live broadcast room opened by the anchor user; The key point detection model includes a first convolutional layer, a pooling layer, a first shuffled network stage layer, a second shuffled network stage layer, a third shuffled network stage layer, a multi-scalar mean series layer, and a second convolutional layer; wherein the first convolutional layer and the pooling layer are the first n-layer network; the multi-scalar mean series layer is used to concatenate the outputs of the three shuffled network stage layers; The first n layers of the network are obtained by training together with the edge generation model, and the parameters of the trained first n layers of the network are used as the parameters of the first n layers of the network in the key point detection model to be trained.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the key point detection method as described in any one of claims 1-3, or the key point detection model training method as described in any one of claims 4-10, or the live broadcast method as described in claim 11.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the key point detection method as described in any one of claims 1-3, or the key point detection model training method as described in any one of claims 4-10, or the live broadcast method as described in claim 11.

Images