Network training and video frame processing method, device, equipment and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for network training and video frame processing. The video frame processing method includes: performing facial segmentation on a video frame containing a target face to obtain multiple partial facial images, wherein each partial facial image contains at least one facial part; inputting the partial facial images into a pre-trained local expression recognition network corresponding to the facial part, and obtaining a number of expression coefficients output by the local expression recognition network; and generating a virtual image consistent with the expression of the target face based on the expression coefficients output by each local expression recognition network. This method enables scenarios such as virtual live broadcasts and digital humans, greatly reducing the cost and threshold of virtual live broadcasts, improving the interactivity and fun of live broadcast services, and greatly expanding the application prospects and market potential of live broadcasts.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of live broadcast technology, and in particular to a method, apparatus, device, and storage medium for network training and video frame processing. Background Art

[0002] With the rapid development of two-dimensional virtual live streaming in various fields such as games and singing, the audience group of virtual images is gradually growing.

[0003] The technology behind virtual live streaming is facial expression capture. Among related technologies, facial expression capture technology relies on heavy helmet-mounted facial capture equipment and is expensive. If virtual live streaming is to be popularized, the cost of using this equipment will be high. Summary of the Invention

[0004] The present application provides a method, apparatus, device and storage medium for network training and video frame processing to solve the problem in the prior art of expression capture technology during virtual live broadcast, which relies on heavy helmet-mounted facial capture equipment, resulting in high equipment costs and limited promotion of virtual live broadcast.

[0005] In a first aspect, an embodiment of the present application provides a method for training an expression recognition network, the method comprising:

[0006] Perform facial segmentation on each facial image in the training sample set to obtain multiple sample partial images;

[0007] Grouping the plurality of sample partial images according to different facial parts to obtain a plurality of facial part sample sets;

[0008] Perform feature extraction on each sample local image in each facial part sample set;

[0009] Based on the feature set extracted from the sample set of each facial part, a preset neural network algorithm and a specified loss function are used for joint learning to obtain a local expression recognition network corresponding to each facial part, wherein each local expression recognition network is used to output an expression coefficient of a set dimension corresponding to its corresponding facial part.

[0010] In a second aspect, an embodiment of the present application further provides a method for processing a video frame, the method comprising:

[0011] Performing facial segmentation on a video frame containing a target face to obtain a plurality of partial facial images, wherein each partial facial image includes at least one facial part;

[0012] Inputting the partial facial images into pre-trained local expression recognition networks corresponding to the facial parts, respectively, and obtaining a number of expression coefficients output by the local expression recognition networks;

[0013] A virtual image consistent with the expression of the target face is generated according to a number of expression coefficients output by each local expression recognition network.

[0014] In a third aspect, an embodiment of the present application further provides a device for training an expression recognition network, the device comprising:

[0015] The sample image face segmentation module is used to perform face segmentation on each face image in the training sample set to obtain multiple sample partial images;

[0016] An image grouping module, configured to group the plurality of sample partial images according to different facial parts to obtain a plurality of facial part sample sets;

[0017] A feature extraction module is used to extract features from each sample local image in each facial part sample set;

[0018] The expression recognition network training module is used to perform joint learning based on the feature set extracted from the sample set of each facial part, using a preset neural network algorithm and a specified loss function to obtain the local expression recognition network corresponding to each facial part, wherein each local expression recognition network is used to output the expression coefficient of the set dimension corresponding to its corresponding facial part.

[0019] In a fourth aspect, an embodiment of the present application further provides a device for processing a video frame, the device comprising:

[0020] a video frame face segmentation module, configured to perform face segmentation on a video frame containing a target face to obtain a plurality of partial face images, wherein each partial face image contains at least one facial part;

[0021] An expression coefficient acquisition module, configured to input the partial facial images into a pre-trained local expression recognition network corresponding to the facial part, and obtain a number of expression coefficients output by the local expression recognition network;

[0022] The virtual image generation module is used to generate a virtual image consistent with the expression of the target face according to a number of expression coefficients output by each local expression recognition network.

[0023] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising:

[0024] one or more processors;

[0025] a storage device for storing one or more programs,

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the method of the first aspect or the second aspect mentioned above.

[0027] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method of the first aspect or the second aspect mentioned above is implemented.

[0028] The technical solution provided in this application has the following beneficial effects:

[0029] In this embodiment, facial segmentation is performed on a video frame containing a target face to obtain multiple partial facial images. A pre-trained local expression recognition network corresponding to each partial facial image is then obtained and fed into the corresponding local expression recognition network for real-time expression capture. The expression coefficients output by the local expression recognition network are then obtained, thereby capturing the target face's expression in the video frame in real time. Based on the expression coefficients output by each local expression recognition network, a virtual image is generated that matches the target face's expression. The virtual image's expression changes in sync with the target face's expression, enabling scenarios such as virtual live broadcasts and digital humans, enhancing the interactivity and fun of live broadcast services and greatly expanding the application prospects and market potential of live broadcasts.

[0030] In addition, this embodiment can capture the expression of the target face and determine that the virtual image makes an expression consistent with the target face by analyzing the video frames. It only needs to use the camera to capture the video frames. Compared with the helmet face capture equipment in the existing technology, it greatly reduces the cost and threshold of virtual live broadcast, and expands the audience and application scenarios of virtual live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of an embodiment of a method for training an expression recognition network provided in Example 1 of the present application;

[0032] Figure 2 This is a schematic diagram of a set of exemplary emoticon labels provided in Example 1 of the present application;

[0033] Figure 3 This is a schematic diagram of an example of calibration of calibration points provided in Example 1 of the present application;

[0034] Figure 4 This is a flowchart of a method embodiment of video frame processing provided in Example 2 of the present application;

[0035] Figure 5 This is a structural block diagram of an embodiment of an apparatus for expression recognition network training provided in the third embodiment of the present application;

[0036] Figure 6 This is a structural block diagram of an embodiment of a device for video frame processing provided in the fourth embodiment of the present application;

[0037] Figure 7 This is a structural diagram of an electronic device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0038] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present application and are not intended to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the present application, not all of the structures.

[0039] Example 1

[0040] Figure 1 This is a flowchart of an embodiment of a method for training an expression recognition network provided in the first embodiment of the present application. This embodiment can be applied to a live broadcast platform and executed by a server of the live broadcast platform. The method may include the following steps:

[0041] Step 110 , performing face segmentation on each facial image in the training sample set to obtain a plurality of sample partial images.

[0042] In one implementation, Apple's ARKit tool can be used to collect training sample data. For example, a designated app can be developed to call the Apple ARKit interface to collect data. During the sample data collection process, to improve prediction efficiency, the subject can be asked to make as many expressions as possible, covering multiple parts of the body, such as the eyes, eyebrows, nose, and mouth. This allows the neural network trained using this sample data to better capture a variety of expressions.

[0043] In one example, the collected training sample data may include several video segments, and each frame of the video can be used as a training sample, and can be pre-labeled with expression labels of several dimensions. For example, 52-dimensional expression labels can be used, and each dimensional expression label has an expression coefficient in the range of [0, 1]. Each dimensional expression label can represent a single expression, and its value (i.e., expression coefficient) represents the amplitude of the expression. The closer the expression coefficient is to 1, the greater the amplitude of the expression. For example, Figure 2The expression label diagram shows some expression labels, which can include from left to right: the inner eyelid area of ​​the left eye converges (eyeSquintLeft), the left eye looks down (eyeLookDownLeft), the chin is open (jawOpen), the degree of opening of the lips (mouthFunnel), the chin moves to the right (jawRight), the mouth closes (mouthClose), the lower lip does not move and the upper lip moves (mouthShrugUpper), the left eye looks inward (eyeLookInLeft), the left eye closes (eyeBlinkLeft), the right lower lip is compressed upward (mouthPressRight), and so on.

[0044] After obtaining a training sample set, facial recognition can be performed on each training sample, and the sample dataset can be expanded based on the recognized facial images. Specifically, after facial recognition, key points of various facial parts can be marked, such as the left eye, right eye, nose tip, and mouth. The facial image is then geometrically normalized based on these key points. First, the center point V of the facial image is determined. The image is rotated about V so that the left and right eyes are at the same horizontal position. The image is then translated so that the nose tip is centered. Following geometric normalization, illumination normalization is performed, and histogram equalization is performed on the facial image. Finally, the training set is expanded by horizontally flipping the facial image and applying six transformations to the original and flipped images: brightness increase, brightness decrease, contrast increase, contrast decrease, Gaussian blur, and Gaussian noise. This allows a single image to be expanded into 14 images.

[0045] In practice, before performing face segmentation, the size of each facial image can be uniformly adjusted to a specified size, for example, to 256*256. Then, face segmentation is performed according to facial parts to obtain multiple sample partial images, where each sample partial image can include one or more facial parts.

[0046] It should be noted that the above-mentioned face can be a human face or other animal face, and this embodiment does not limit this.

[0047] In one embodiment, step 110 may further include the following steps:

[0048] Step 110 - 1 : For each facial image, perform key point detection on the facial image to obtain key points of various facial parts.

[0049] In one implementation, a customized SDK (Software Development Kit) detection tool can be used to perform key point detection on each facial image to extract key point information of each facial part, where the facial parts may include the left eye, left eyebrow, right eyebrow, right eyebrow, nose, mouth, facial contour, etc.

[0050] Step 110 - 2 : performing facial segmentation on the facial image according to the key points of the multiple facial parts and a preset segmentation rule to obtain a plurality of sample partial images.

[0051] Step 110 - 3 : According to the segmentation rule, the sizes of the plurality of sample partial images are respectively adjusted to corresponding set sizes.

[0052] Exemplarily, the preset segmentation rules can be used to standardize the facial parts included in each partial image and the size of the partial images. For example, the segmentation rules can stipulate that the partial images obtained after segmentation include an upper left facial image, an upper right facial image, a middle facial image, and a lower facial image, wherein the upper left facial image includes the left eyebrow and the left eye area, the upper right facial image includes the right eyebrow and the right eye area, the middle facial image includes the nose area, and the lower facial image includes the mouth and chin area; and the corresponding sizes of the upper left facial image, the upper right facial image, the middle facial image, and the lower facial image are respectively: 48*48, 48*48, 48*48, and 96*96. Based on the above segmentation rules and the key points of each facial part detected, each facial image can be divided into 4 sample partial images, and their sizes are set to the corresponding set sizes.

[0053] Step 120 : Group the multiple sample partial images according to different facial parts to obtain multiple facial part sample sets.

[0054] For example, all the upper left facial images obtained by segmentation can be combined into an upper left face (i.e., the area of ​​the left eyebrow + left eye) sample set, all the upper right facial images can be combined into an upper right face (i.e., the area of ​​the right eyebrow + right eye) sample set, all the middle facial images can be combined into a middle face (i.e., the nose area) sample set, and all the lower facial images can be combined into a lower face (i.e., the mouth area + chin area) sample set.

[0055] Step 130 : extracting features from each sample local image in each facial part sample set.

[0056] In one implementation, a feature extraction network can be used to extract features from each facial part sample set. Different facial part sample sets can use different feature extraction networks for feature extraction, or all facial part sample sets can use the same feature extraction network for feature extraction, which is not limited in this embodiment.

[0057] Exemplarily, the feature extraction network may include a deep neural network, which may include the following parts: a first part consists of two convolutional layers, an activation function and a pooling layer; a second part consists of three convolutional layers, an activation function and a pooling layer; a third part consists of three convolutional layers, an activation function and a pooling layer; a fourth part consists of three convolutional layers, an activation function and a pooling layer; and a fifth part consists of two fully connected layers.

[0058] Step 140 , based on the feature set extracted from the sample set of each facial part, a preset neural network algorithm and a specified loss function are used for joint learning to obtain a local expression recognition network corresponding to each facial part.

[0059] In this embodiment, corresponding local expression recognition networks can be trained for different facial parts, such as the upper left facial part (including the left eyebrow and left eye), the upper right facial part (including the right eyebrow and right eye), the middle facial part (including the nose), and the lower facial part (including the mouth and chin). Each facial part corresponds to a local expression recognition network. Each local expression recognition network is configured to output an expression coefficient of a set dimension corresponding to its corresponding facial part, and a local expression recognition network is responsible for outputting a portion of the expression coefficients.

[0060] In one example, the preset neural network algorithm may include a lightweight ShuffleNet V2 convolutional neural network algorithm. The specified loss functions may include a vertex distance loss function, a vertex loss function, and a regression loss function L1 loss. During joint learning, each local expression recognition network may be converged based on the total loss value of the vertex distance loss value corresponding to the vertex distance loss function, the vertex loss value corresponding to the vertex loss function, and the regression loss value corresponding to the regression loss function.

[0061] In one embodiment, when the specified loss function is a vertex distance loss function, the vertex distance loss value corresponding to the vertex distance loss function is obtained as follows:

[0062] Step S1, obtaining the expression coefficients and expression labels output by each local expression recognition network.

[0063] In this embodiment, each local expression recognition network can output a certain number of expression labels and the expression coefficients corresponding to the expression labels. The number of expression labels output by each local expression recognition network can be pre-set. For example, assuming that the local expression recognition network includes an upper left face recognition network, an upper right face recognition network, a middle face recognition network, and a lower face recognition network, and the corresponding numbers of expression labels are 10, 10, 2, and 30 respectively, the specific number can be as follows:

[0064] The 10 expression labels of the upper left face recognition network are: eyeBlinkLeft, eyeLookDownLeft, eyeLookInLeft, left eye looking outward (eyeLookOutLeft), left eye looking up (eyeLookUpLeft), eyeSquintLeft, left eyelid widening (eyeWideLeft), left eyebrow moving downward (browDownLeft), left eyebrow moving upward (browInnerUp), left eyebrow moving upward (browOuterUpLeft);

[0065] The 10 expression labels of the upper right face recognition network are: right eye closed (eyeBlinkRight), right eye looking down (eyeLookDownRight), right eye looking inward (eyeLookInRight), right eye looking outward (eyeLookOutRight), right eye looking up (eyeLookUpRight), right eyelid area convergence (eyeSquintRight), right eyelid widening (eyeWideRight), right eyebrow outer side moving down (browDownRight), right eyebrow inner side moving up (browInnerUp), right eyebrow outer side moving up (browOuterUpRight);

[0066] The two expression labels of the middle face recognition network are: moving up around the left nostril (noseSneerLeft) and moving up around the right nostril (noseSneerRight);

[0067] The 30 expression labels of the lower face recognition network are: cheeks move outward (cheekPuff), left cheek moves up (cheekSquintLeft), right cheek moves up (cheekSquintRight), chin moves forward (jawForward), chin moves left (jawLeft), jawOpen, jawRight, mouthClose, left mouth moves back (mouthDimpleLeft), right mouth moves back (mouthDimpleRight), left mouth moves down (mouthFrownLeft), right mouth moves down (mouthFrownRight), mouthFunnel, lips move left (mouthLeft), left lower lip moves down (mouthLowerDownLeft), right lower lip moves down (mouthLowerDownRight) , compress the left lower lip upwards (mouthPressLeft), mouthPressRight, the degree of contraction of closed lips (mouthPucker), move both lips to the right (mouthRight), move the lower lip into the mouth (mouthRollLower), move the upper lip into the mouth (mouthRollUpper), move the lower lip outwards (mouthShrugLower), move the upper lip outwards (mouthShrugUpper), move the left side of the mouth upwards (mouthSmileLeft), move the right side of the mouth upwards (mouthSmileRight), move the left side of the mouth to the left (mouthStretchLeft), move the right side of the mouth to the left (mouthStretchRight), move the left upper lip up (mouthUpperUpLeft), move the right upper lip up (mouthUpperUpRight).

[0068] Step S2: generating first facial mesh data according to the expression coefficients output by each local expression recognition network, and generating second facial mesh data according to the expression labels output by each local expression recognition network.

[0069] After obtaining the expression labels and corresponding expression coefficients output by each local expression recognition network, the mesh data can be converted according to the expression labels and expression coefficients output by each local expression recognition network to generate the corresponding mesh. In computer graphics, the shape of a three-dimensional object can be defined by mesh data. The three-dimensional model contains mesh data and color information. The mesh is composed of triangular faces, and a triangular face contains three vertices. The mesh data structure includes: a vertex array and an index array; the vertex array stores the coordinate information of the vertices, and the index array stores the order of the vertices. When creating a triangular face in a three-dimensional coordinate system, you only need to traverse the index array and obtain the vertex coordinate information from the vertex array according to the index. Three indices correspond to three vertices, and three vertices constitute a triangular face.

[0070] In one implementation, the expression coefficients output by each local expression recognition network can be used as first 3DMM (3DMorphableModel) coefficients, which are then input into a 3DMM formula to generate the first facial mesh data. Similarly, the expression labels output by each local expression recognition network can be used as second 3DMM coefficients, which are then input into a 3DMM formula to generate the second facial mesh data.

[0071] Step S3: obtaining pre-calibrated calibration point pair information for each facial part.

[0072] Each calibration point pair information may include information about two calibration points. For example, the calibration point information may include a calibration point serial number. Figure 3 As shown, the calibration point pair information can be the information of the calibration points pre-calibrated on the left eye, right eye and mouth of the preset standard mesh, such as the upper and lower two calibration points of the left eye constitute a calibration point pair, the upper and lower two calibration points of the right eye constitute a calibration point pair, and the upper and lower two calibration points of the mouth constitute a calibration point pair.

[0073] Step S4: determining a first Euclidean distance corresponding to the first facial mesh data based on the calibration point pair information, and determining a second Euclidean distance corresponding to the second facial mesh data.

[0074] The first Euclidean distance corresponding to the first facial mesh data refers to the sum of the Euclidean distances between the two calibration points of each calibration point pair calculated based on the first facial mesh data; the second Euclidean distance corresponding to the second facial mesh data refers to the sum of the Euclidean distances between the two calibration points of each calibration point pair calculated based on the second facial mesh data.

[0075] Specifically, in one embodiment, each calibration point pair information includes a first calibration point serial number and a second calibration point serial number; the step of determining the first Euclidean distance corresponding to the first facial mesh data in step S4 may further include the following steps:

[0076] Step S4-1: Searching for the first and second calibration point numbers in each facial part of the first facial mesh data to determine first coordinates of the facial part corresponding to the first calibration point number in the first facial mesh data, and second coordinates of the facial part corresponding to the second calibration point number in the first facial mesh data.

[0077] For example, assuming that each face mesh data (including the first face mesh data and the second face mesh data) has a total of 1220 points, set the first calibration point number of the left eye (such as Figure 3 The number of the calibration point above the left eye is 650, and the number of the second calibration point (such as Figure 3 If the calibration point number below the middle left eye is 450, the coordinates of the calibration point with the serial number 650 can be searched in the upper left facial area of ​​the first facial grid data as the first coordinate of the upper left facial area, and the coordinates of the calibration point with the serial number 450 can be searched as the second coordinate of the upper left facial area.

[0078] Similarly, set the first calibration point number of the right eye (such as Figure 3 The number of the calibration point above the right eye is 888, and the number of the second calibration point (such as Figure 3 If the calibration point number below the middle right eye is 796, the coordinates of the calibration point with the serial number 888 can be searched in the upper right facial area of ​​the first facial grid data as the first coordinates of the upper right facial area, and the coordinates of the calibration point with the serial number 796 can be searched as the second coordinates of the upper right facial area.

[0079] Set the first calibration point number of the lips (such as Figure 3 The calibration point number above the middle lip is 250, and the second calibration point number (such as Figure 3 If the calibration point number below the middle lip is 174, the coordinates of the calibration point with the serial number 250 can be searched in the lower facial area of ​​the first facial grid data as the first coordinates of the lower facial area, and the coordinates of the calibration point with the serial number 174 can be searched as the second coordinates of the lower facial area.

[0080] Step S4-2: Calculate the Euclidean distance between the first coordinate and the second coordinate to obtain the Euclidean distance of the expression coefficient of the corresponding facial part.

[0081] After obtaining the first and second coordinates of each facial part, the Euclidean distance between the first and second coordinates of each facial part can be calculated for each facial part to obtain the Euclidean distance of the facial part's expression coefficient. For example, in the above example, for the upper left facial region, the Euclidean distance between the first coordinate of calibration point 650 and the second coordinate of calibration point 450 can be calculated to obtain the Euclidean distance of the expression coefficient of the upper left facial region; for the upper right facial region, the Euclidean distance between the first coordinate of calibration point 888 and the second coordinate of calibration point 796 can be calculated to obtain the Euclidean distance of the expression coefficient of the upper right facial region; for the lower facial region, the Euclidean distance between the first coordinate of calibration point 250 and the second coordinate of calibration point 174 can be calculated to obtain the Euclidean distance of the expression coefficient of the lower facial region.

[0082] Step S4-3: Calculate the sum of the Euclidean distances of the expression coefficients of the facial parts to obtain a first Euclidean distance corresponding to the first facial mesh data.

[0083] For example, the first Euclidean distance Dist_x corresponding to the first facial mesh data may be expressed by the following formula:

[0084] Dist_x = Dist_x_upper left face + Dist_x_upper right face + Dist_x_lower face

[0085] Among them, "Dist_x_upper-left-face" represents the Euclidean distance of the expression coefficient of the upper-left facial area, "Dist_x_upper-right-face" represents the Euclidean distance of the expression coefficient of the upper-right facial area, and "Dist_x_lower-face" represents the Euclidean distance of the expression coefficient of the lower facial area.

[0086] Similarly, the step of determining the second Euclidean distance corresponding to the second facial mesh data in step S4 may further include the following steps:

[0087] The first and second calibration point numbers are searched for each facial part in the second facial mesh data to determine a third coordinate of the facial part corresponding to the first calibration point number in the second facial mesh data, and a fourth coordinate of the facial part corresponding to the second calibration point number in the second facial mesh data is determined; a Euclidean distance between the third and fourth coordinates is calculated to obtain a label Euclidean distance of the corresponding facial part; and a sum of the label Euclidean distances of each facial part is calculated to obtain a second Euclidean distance corresponding to the second facial mesh data.

[0088] Specifically, the calculation method of the second Euclidean distance corresponding to the second facial mesh data is similar to the calculation method of the first Euclidean distance corresponding to the first facial mesh data. For similar processes, reference can be made to the process of calculating the first Euclidean distance.

[0089] For example, the coordinates of the calibration point numbered 650 can be searched in the upper left facial region of the second facial mesh data as the third coordinate of the upper left facial region, and the coordinates of the calibration point numbered 450 can be searched as the fourth coordinate of the upper left facial region; the coordinates of the calibration point numbered 888 can be searched in the upper right facial region of the second facial mesh data as the third coordinate of the upper right facial region, and the coordinates of the calibration point numbered 796 can be searched as the fourth coordinate of the upper right facial region; the coordinates of the calibration point numbered 250 can be searched in the lower facial region of the second facial mesh data as the third coordinate of the lower facial region, and the coordinates of the calibration point numbered 174 can be searched as the fourth coordinate of the lower facial region. The Euclidean distance between the third and fourth coordinates of each facial part is then calculated as the label Euclidean distance of the facial part. The label Euclidean distances of all facial parts can be added together to obtain the second Euclidean distance corresponding to the second facial mesh data, that is:

[0090] Dist_y = Dist_y_upper-left face + Dist_y_upper-right face + Dist_y_lower face

[0091] Among them, Dist_y is the second Euclidean distance, "Dist_y_UpperLeftFace" represents the label Euclidean distance of the upper left facial area, "Dist_y_UpperRightFace" represents the label Euclidean distance of the upper right facial area, and "Dist_y_LowerFace" represents the label Euclidean distance of the lower facial area.

[0092] Step S5: Calculate the absolute value of the difference between the first Euclidean distance and the second Euclidean distance to obtain an optimized vertex distance loss value.

[0093] Specifically, the vertex distance loss can be expressed as follows:

[0094] vertex distance loss=||Dist_x-Dist_y||

[0095] In this embodiment, by calculating the vertex distance loss, it is helpful to align the facial parts such as eyes and mouth whose opening and closing degree affects the visual perception with the corresponding label values ​​as much as possible.

[0096] In other embodiments, when the specified loss function is a vertex loss function, the vertex loss value Vertex loss corresponding to the vertex loss function can be obtained as follows:

[0097]

[0098] Among them, Mxj My represents the jth vertex in Mx (i.e., the first facial mesh data in the previous embodiment); j represents the j-th vertex in My (i.e., the second facial mesh data in the previous embodiment).

[0099] The intuitive meaning of vertex loss is to make the mesh obtained by converting the network output expression coefficient and the mesh obtained by label conversion as close as possible.

[0100] When the specified loss function is the regression loss function L1 loss, the regression loss value corresponding to the regression loss function L1 loss can be obtained as follows:

[0101]

[0102] Among them, x is the expression coefficient output by the local expression recognition network, y is the expression label corresponding to the expression coefficient output by the local expression recognition network, and n is the dimension corresponding to the current expression label.

[0103] Through the above three loss functions, multiple local expression recognition networks can be jointly trained using methods such as gradient backpropagation.

[0104] In this embodiment, facial images in a training sample set are segmented to obtain multiple sample partial images, which are then aggregated into multiple facial part sample sets based on different facial parts. Feature sets for each facial part are extracted from the different facial part sample sets. Joint learning is then performed using a preset neural network algorithm and a specified loss function based on these feature sets to simultaneously obtain a lightweight local expression recognition network for each facial part. Each local expression recognition network is configured to output an expression coefficient of a set dimension corresponding to its corresponding facial part. This improves the efficiency and accuracy of expression capture compared to using a single model to recognize the expression coefficient of the entire face.

[0105] Example 2

[0106] Figure 4 This is a flow chart of a method embodiment of video frame processing provided in the second embodiment of this application. This embodiment can be applied to a live broadcast platform and executed by the server of the live broadcast platform. It belongs to the stage of model reasoning of the above model. Figure 4 As shown, this embodiment may include the following steps:

[0107] Step 410 : performing face segmentation on the video frame containing the target face to obtain a plurality of partial face images, wherein each partial face image contains at least one facial part.

[0108] In this step, the process of performing face segmentation on the video frame containing the target face is similar to the process of performing face segmentation on the facial images in the training sample set in step 110. The specific process can refer to the process of step 110 and will not be repeated here.

[0109] In step 420 , the partial facial images are respectively input into a pre-trained local expression recognition network corresponding to the facial part, and a plurality of expression coefficients output by the local expression recognition network are obtained.

[0110] For example, assuming that the video frame containing the target face is segmented into multiple local facial images including an upper left facial image, an upper right facial image, a middle facial image and a lower facial image, the upper left facial image can be input into the upper left facial recognition network for expression recognition, and a 10-dimensional expression label and a corresponding expression coefficient output by the upper left facial recognition network are obtained; the upper right facial image is input into the upper right facial recognition network for expression recognition, and a 10-dimensional expression label and a corresponding expression coefficient output by the upper right facial recognition network are obtained; the middle facial image is input into the middle facial recognition network for expression recognition, and a 2-dimensional expression label and a corresponding expression coefficient output by the middle facial recognition network are obtained; the lower facial image is input into the lower facial recognition network for expression recognition, and a 30-dimensional expression label and a corresponding expression coefficient output by the lower facial recognition network are obtained.

[0111] Step 430 : generating a virtual image consistent with the expression of the target face based on the expression coefficients output by each local expression recognition network.

[0112] For example, after obtaining the 10-dimensional expression label and corresponding expression coefficient output by the upper left facial recognition network, the 10-dimensional expression label and corresponding expression coefficient output by the upper right facial recognition network, the 2-dimensional expression label and corresponding expression coefficient output by the middle facial recognition network, and the 30-dimensional expression label and corresponding expression coefficient output by the lower facial recognition network, these 52-dimensional expression labels and expression coefficients can be directly applied to the 3D virtual image to drive the 3D virtual image to make an expression consistent with the expression of the target face, and the expression of the 3D virtual image will change along with the expression of the target face.

[0113] In this embodiment, facial segmentation is performed on a video frame containing a target face to obtain multiple partial facial images. A pre-trained local expression recognition network corresponding to each partial facial image is then obtained and fed into the corresponding local expression recognition network for real-time expression capture. The expression coefficients output by the local expression recognition network are then obtained, thereby capturing the target face's expression in the video frame in real time. Based on the expression coefficients output by each local expression recognition network, a virtual image is generated that matches the target face's expression. The virtual image's expression changes in sync with the target face's expression, enabling scenarios such as virtual live broadcasts and digital humans, enhancing the interactivity and fun of live broadcast services and greatly expanding the application prospects and market potential of live broadcasts.

[0114] In addition, this embodiment can capture the expression of the target face and determine that the virtual image makes an expression consistent with the target face by analyzing the video frames. It only needs to use the camera to capture the video frames. Compared with the helmet face capture equipment in the existing technology, it greatly reduces the cost and threshold of live broadcast, and expands the audience and application scenarios of virtual live broadcast.

[0115] Example 3

[0116] Figure 5 This is a structural block diagram of an embodiment of an apparatus for training an expression recognition network provided in the third embodiment of the present application. The apparatus can be applied to a live broadcast platform and may include the following modules:

[0117] The sample image face segmentation module 510 is used to perform face segmentation on each face image in the training sample set to obtain a plurality of sample partial images;

[0118] An image grouping module 520 is configured to group the plurality of sample partial images according to different facial parts to obtain a plurality of facial part sample sets;

[0119] A feature extraction module 530 is used to extract features from each sample local image in each facial part sample set;

[0120] The expression recognition network training module 540 is used to perform joint learning based on the feature set extracted from the sample set of each facial part, using a preset neural network algorithm and a specified loss function to obtain a local expression recognition network corresponding to each facial part, wherein each local expression recognition network is used to output an expression coefficient of a set dimension corresponding to its corresponding facial part.

[0121] In one embodiment, the specified loss function includes a vertex loss function, a regression loss function, and a vertex distance loss function;

[0122] The expression recognition network training module 540 is specifically used to:

[0123] When performing joint learning, each local expression recognition network is converged based on the total loss value of the vertex distance loss value corresponding to the vertex distance loss function, the vertex loss value corresponding to the vertex loss function, and the regression loss value corresponding to the regression loss function.

[0124] In one embodiment, when the specified loss function is a vertex distance loss function, the expression recognition network training module 540 includes a vertex distance loss value determination submodule, and the vertex distance loss value determination submodule includes:

[0125] An expression coefficient and expression label acquisition unit, used to obtain the expression coefficient and expression label output by each local expression recognition network;

[0126] a mesh data generating unit, configured to generate first facial mesh data based on expression coefficients output by each local expression recognition network, and to generate second facial mesh data based on expression labels output by each local expression recognition network;

[0127] a calibration point pair information acquisition unit, configured to acquire calibration point pair information pre-calibrated for each facial part;

[0128] a Euclidean distance determining unit, configured to determine a first Euclidean distance corresponding to the first facial mesh data based on the calibration point pair information, and to determine a second Euclidean distance corresponding to the second facial mesh data;

[0129] The vertex distance loss value calculation unit is used to calculate the absolute value of the difference between the first Euclidean distance and the second Euclidean distance to obtain an optimized vertex distance loss value.

[0130] In one embodiment, each of the calibration point pair information includes a first calibration point serial number and a second calibration point serial number; and the Euclidean distance determination unit is specifically configured to:

[0131] searching for the first calibration point number and the second calibration point number in each facial part of the first facial mesh data to determine a first coordinate of the facial part corresponding to the first calibration point number in the first facial mesh data, and determining a second coordinate of the facial part corresponding to the second calibration point number in the first facial mesh data;

[0132] Calculating the Euclidean distance between the first coordinate and the second coordinate to obtain the Euclidean distance of the expression coefficient of the corresponding facial part;

[0133] The sum of the Euclidean distances of the expression coefficients of the facial parts is calculated to obtain a first Euclidean distance corresponding to the first facial mesh data.

[0134] In one embodiment, the Euclidean distance determination unit is further configured to:

[0135] searching for the first and second calibration point numbers in each facial part of the second facial mesh data to determine a third coordinate of the facial part corresponding to the first calibration point number in the second facial mesh data, and determining a fourth coordinate of the facial part corresponding to the second calibration point number in the second facial mesh data;

[0136] Calculating the Euclidean distance between the third coordinate and the fourth coordinate to obtain the label Euclidean distance of the corresponding facial part;

[0137] The sum of the label Euclidean distances of each facial part is calculated to obtain a second Euclidean distance corresponding to the second facial mesh data.

[0138] In one embodiment, the sample image face segmentation module 510 is specifically configured to:

[0139] For each facial image, performing key point detection on the facial image to obtain key points of multiple facial parts;

[0140] Performing facial segmentation on the facial image according to the key points of the multiple facial parts and a preset segmentation rule to obtain a plurality of sample partial images;

[0141] According to the segmentation rule, the sizes of the plurality of sample partial images are respectively adjusted to corresponding set sizes.

[0142] An apparatus for training an expression recognition network provided in an embodiment of the present application can execute an expression recognition network training method in the first embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0143] Example 4

[0144] Figure 6 This is a structural block diagram of an embodiment of a video frame processing device provided in the fourth embodiment of the present application. The device can be applied to a live broadcast platform and may include the following modules:

[0145] A video frame face segmentation module 610 is configured to perform face segmentation on a video frame containing a target face to obtain a plurality of partial face images, wherein each partial face image includes at least one facial part;

[0146] The expression coefficient acquisition module 620 is used to input the local facial images into a pre-trained local expression recognition network corresponding to the facial parts, and obtain a number of expression coefficients output by the local expression recognition network;

[0147] The virtual image generation module 630 is used to generate a virtual image consistent with the expression of the target face according to the expression coefficients output by each local expression recognition network.

[0148] A video frame processing device provided in an embodiment of the present application can execute a video frame processing method in embodiment 2 of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0149] Example 5

[0150] Figure 7 This is a structural diagram of an electronic device provided in Example 5 of the present application, such as Figure 7 As shown, the electronic device includes a processor 710, a memory 720, an input device 730, and an output device 740; the number of processors 710 in the electronic device can be one or more. Figure 7 In the figure, a processor 710 is used as an example; the processor 710, memory 720, input device 730 and output device 740 in the electronic device can be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0151] The memory 720, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the first or second embodiment of the present application. The processor 710 executes the software programs, instructions, and modules stored in the memory 720 to execute various functional applications and data processing of the electronic device, that is, to implement the method mentioned in the first or second embodiment of the method.

[0152] The memory 720 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal, etc. In addition, the memory 720 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include a memory remotely located relative to the processor 710, and these remote memories may be connected to the device / terminal / server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0154] Example 6

[0155] Embodiment 6 of the present application further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the method in the above method embodiment.

[0156] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present application, whose computer-executable instructions are not limited to the method operations described above, can also execute related operations in the method provided in any embodiment of the present application.

[0157] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present application can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0158] It is worth noting that in the embodiments of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0159] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for training an expression recognition network, characterized in that: The method comprises: Perform facial segmentation on each facial image in the training sample set to obtain multiple sample partial images; Grouping the plurality of sample partial images according to different facial parts to obtain a plurality of facial part sample sets; Perform feature extraction on each sample local image in each facial part sample set; Based on the feature set extracted from the sample set of each facial part, a preset neural network algorithm and a specified loss function are used for joint learning to obtain a local expression recognition network corresponding to each facial part, wherein each local expression recognition network is used to output an expression coefficient of a set dimension corresponding to its corresponding facial part; The specified loss function includes a vertex distance loss function, and the vertex distance loss value corresponding to the vertex distance loss function is obtained in the following manner: Obtaining expression coefficients and expression labels output by each local expression recognition network; generating first facial mesh data based on the expression coefficients output by each local expression recognition network, and generating second facial mesh data based on the expression labels output by each local expression recognition network; obtaining calibration point pair information pre-calibrated for each facial part; determining a first Euclidean distance corresponding to the first facial mesh data and a second Euclidean distance corresponding to the second facial mesh data based on the calibration point pair information; calculating the absolute value of the difference between the first Euclidean distance and the second Euclidean distance to obtain an optimized vertex distance loss value; Each of the local expression recognition networks is converged based on the vertex distance loss value.

2. The method according to claim 1, characterized in that The specified loss function also includes a vertex loss function and a regression loss function; The feature set extracted from the sample set of each facial part is jointly learned using a preset neural network algorithm and a specified loss function to obtain a local expression recognition network corresponding to each facial part, including: When performing joint learning, each local expression recognition network is converged based on the total loss value of the vertex distance loss value corresponding to the vertex distance loss function, the vertex loss value corresponding to the vertex loss function, and the regression loss value corresponding to the regression loss function.

3. The method according to any one of claims 1-2, characterized in that Each of the calibration point pair information includes a first calibration point serial number and a second calibration point serial number; and determining a first Euclidean distance corresponding to the first facial mesh data includes: searching for the first calibration point number and the second calibration point number in each facial part of the first facial mesh data to determine a first coordinate of the facial part corresponding to the first calibration point number in the first facial mesh data, and determining a second coordinate of the facial part corresponding to the second calibration point number in the first facial mesh data; Calculating the Euclidean distance between the first coordinate and the second coordinate to obtain the Euclidean distance of the expression coefficient of the corresponding facial part; The sum of the Euclidean distances of the expression coefficients of the facial parts is calculated to obtain a first Euclidean distance corresponding to the first facial mesh data.

4. The method according to claim 3, characterized in that The determining the second Euclidean distance corresponding to the second facial mesh data includes: searching for the first and second calibration point numbers in each facial part of the second facial mesh data to determine a third coordinate of the facial part corresponding to the first calibration point number in the second facial mesh data, and determining a fourth coordinate of the facial part corresponding to the second calibration point number in the second facial mesh data; Calculating the Euclidean distance between the third coordinate and the fourth coordinate to obtain the label Euclidean distance of the corresponding facial part; The sum of the label Euclidean distances of each facial part is calculated to obtain a second Euclidean distance corresponding to the second facial mesh data.

5. The method according to any one of claims 1-2, characterized in that The step of performing face segmentation on each facial image in the training sample set to obtain a plurality of sample partial images includes: For each facial image, performing key point detection on the facial image to obtain key points of multiple facial parts; Performing facial segmentation on the facial image according to the key points of the multiple facial parts and a preset segmentation rule to obtain a plurality of sample partial images; According to the segmentation rule, the sizes of the plurality of sample partial images are respectively adjusted to corresponding set sizes.

6. A method for video frame processing, characterized in that: The method comprises: Performing facial segmentation on a video frame containing a target face to obtain a plurality of partial facial images, wherein each partial facial image includes at least one facial part; Inputting the partial facial images into pre-trained local expression recognition networks corresponding to the facial parts, respectively, and obtaining a number of expression coefficients output by the local expression recognition networks; generating a virtual image consistent with the expression of the target face according to a number of expression coefficients output by each local expression recognition network; Wherein, the local expression recognition network is obtained by adopting the expression recognition network training method described in any one of claims 1-5.

7. A device for training an expression recognition network, characterized in that: The device comprises: The sample image face segmentation module is used to perform face segmentation on each face image in the training sample set to obtain multiple sample partial images; An image grouping module, configured to group the plurality of sample partial images according to different facial parts to obtain a plurality of facial part sample sets; A feature extraction module is used to extract features from each sample local image in each facial part sample set; An expression recognition network training module is used to perform joint learning based on the feature set extracted from the sample set of each facial part using a preset neural network algorithm and a specified loss function to obtain a local expression recognition network corresponding to each facial part, wherein each local expression recognition network is used to output an expression coefficient of a set dimension corresponding to its corresponding facial part; The specified loss function includes a vertex distance loss function; when the specified loss function is a vertex distance loss function, the expression recognition network training module includes a vertex distance loss value determination submodule, The vertex distance loss value determination submodule includes: an expression coefficient and expression label acquisition unit, which is used to obtain the expression coefficient and expression label output by each local expression recognition network; a mesh data generation unit, which is used to generate first facial mesh data according to the expression coefficient output by each local expression recognition network, and generate second facial mesh data according to the expression label output by each local expression recognition network; a calibration point pair information acquisition unit, which is used to obtain calibration point pair information pre-calibrated for each facial part; a Euclidean distance determination unit, which is used to determine the first Euclidean distance corresponding to the first facial mesh data based on the calibration point pair information, and determine the second Euclidean distance corresponding to the second facial mesh data; a vertex distance loss value calculation unit, which is used to calculate the absolute value of the difference between the first Euclidean distance and the second Euclidean distance to obtain an optimized vertex distance loss value; Each of the local expression recognition networks is converged based on the vertex distance loss value.

8. A device for video frame processing, characterized in that: The device comprises: a video frame face segmentation module, configured to perform face segmentation on a video frame containing a target face to obtain a plurality of partial face images, wherein each partial face image contains at least one facial part; An expression coefficient acquisition module, configured to input the partial facial images into a pre-trained local expression recognition network corresponding to the facial part, and obtain a number of expression coefficients output by the local expression recognition network; A virtual image generation module, configured to generate a virtual image consistent with the expression of the target face based on a number of expression coefficients output by each local expression recognition network; Wherein, the local expression recognition network is obtained by adopting the expression recognition network training method described in any one of claims 1-5.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images