Virtual expression generation method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a virtual expression generation method and device, electronic equipment and storage medium. The method comprises: obtaining a face region in an original image to obtain a target face image; obtaining a first face coefficient corresponding to the target face image; performing time domain correction processing on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain a target face coefficient; rendering an expression of the virtual image according to the target face coefficient to obtain a virtual expression. In this embodiment, by performing time domain correction processing on the first face coefficient, the expressions of adjacent original images in the video can have relevance and continuity, the reconstructed expression is more natural, and the viewing experience is improved; by transmitting the target face coefficient instead of image data to render the expression of the virtual image to obtain the virtual expression, the data transmission amount can be reduced, and the effect of real-time reconstruction of the virtual expression is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for generating virtual facial expressions. Background Technology

[0002] Three-dimensional (3D) modeling is a key issue in the field of machine vision, with 3D facial expression modeling being widely used in entertainment fields such as games, film and television special effects, and VR. Current mainstream methods for 3D virtual facial expression modeling are based on generating 3D virtual expressions from images. However, due to the complex structure of the human face and the coordinated movements of facial muscles, the process of facial expression changes is a complex non-rigid body motion. This places high demands on the acquisition equipment, acquisition environment, modeling equipment, and modeling process, making real-time performance difficult. Furthermore, when processing frames from video images, the correlation and continuity of facial expressions are ignored. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for generating virtual facial expressions to address the shortcomings of related technologies.

[0004] According to a first aspect of the present disclosure, a method for generating virtual facial expressions is provided, comprising:

[0005] Obtain the face region from the original image to obtain the target face image;

[0006] Obtain the first face coefficient corresponding to the target face image; the first face coefficient includes template expression coefficient and pose coefficient, the template expression coefficient is used to characterize the degree of matching between the face expression and each template, and the pose coefficient represents the rotation angle of the virtual image in three dimensions;

[0007] Temporal domain correction processing is performed on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain the target face coefficient;

[0008] The virtual avatar's expression is rendered based on the target face coefficients to obtain a virtual expression.

[0009] Optionally, the face region in the original image is obtained to obtain the target face image, including:

[0010] Perform face detection on the original image to obtain at least one face region contained in the original image;

[0011] Select a target face region from the at least one face region;

[0012] The target face region is corrected to obtain the target face image.

[0013] Optionally, selecting a target face region from the at least one face region includes:

[0014] When the number of face regions is one, the face region is determined to be the target face region;

[0015] When there are multiple face regions, a score value for each face region is calculated based on the region parameter data of each face region. The score value is used to represent the degree of distance of each face region from the central axis of the original image. The face region corresponding to the maximum score value is determined as the target face region.

[0016] Optionally, the region parameter data includes length, width, face area, and location data. A score value for each face region is calculated based on the region parameter data, including:

[0017] Obtain the difference between the horizontal coordinate of the middle position of each face region and half the width, as well as the absolute value of the difference;

[0018] Obtain the ratio of the absolute value of the difference to the width, and the product of the ratio and the constant 2;

[0019] Obtain the difference between the constant 1 and the product, and obtain the product of the difference corresponding to the product and the preset distance weight;

[0020] Obtain the ratio of the face area to the product of the length and width in each face region, and the square root of the ratio corresponding to the face area;

[0021] Obtain the product of the square root and the preset area weight, wherein the sum of the area weight and the distance weight is 1;

[0022] The sum of the product of the preset area weight and the product of the preset distance weight is calculated to obtain the score value of each face region.

[0023] Optionally, the target face region is corrected to obtain a target face image, including:

[0024] Determine the candidate square region corresponding to the target face region, and obtain the vertex coordinate data of the candidate square region;

[0025] An affine transformation is performed on the vertex coordinate data of the candidate square region and the vertex coordinate data of the preset square to obtain affine transformation coefficients; the vertex coordinate data of the preset square includes a specified origin.

[0026] The original image is subjected to an affine transformation using the affine transformation coefficients to obtain an affine transformed image.

[0027] Using the designated origin as a reference, a square region with a preset side length is extracted from the affine transformation image, and the image within the extracted square region is used as the target face image.

[0028] Optionally, obtaining the first face coefficient corresponding to the target face image includes:

[0029] The target face image is blurred and sharpened respectively to obtain at least one blurred image and at least one sharpened image;

[0030] Feature data are extracted from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image;

[0031] The original feature image, the blurred feature image, and the sharpened feature image are stitched together to obtain an initial feature image;

[0032] Obtain the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjust the initial feature image according to the importance coefficient to obtain the target feature image;

[0033] Based on the target feature image, the template expression coefficient and pose coefficient are determined to obtain the first face coefficient.

[0034] Optionally, obtaining the first face coefficient corresponding to the target face image includes:

[0035] The target face image is input into a preset face coefficient recognition network to obtain the first face coefficient corresponding to the target face image output by the preset face coefficient recognition network.

[0036] Optionally, the preset face coefficient recognition network includes: a blur sharpening module, a feature extraction module, an attention module, and a coefficient learning module;

[0037] The blurring and sharpening module performs blurring and sharpening processing on the target face image respectively, to obtain at least one blurred image and at least one sharpened image;

[0038] The feature extraction module extracts feature data from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image; and then stitches the original feature image, the blurred feature image, and the sharpened feature image together to obtain the initial feature image.

[0039] The attention module obtains the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjusts the initial feature image according to the importance coefficient to obtain the target feature image;

[0040] The coefficient learning module determines the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient.

[0041] Optionally, the attention module is implemented using a network model with a temporal attention mechanism or a spatial attention mechanism.

[0042] Optionally, the coefficient learning module is implemented using at least one network model selected from ResNet50, ResNet18, ResNet100, DenseNet, and YOLOv5.

[0043] Optionally, temporal domain correction processing is performed on the template expression coefficients and / or pose coefficients in the first face coefficients to obtain the target face coefficients, including:

[0044] Obtain the first face coefficient and a preset weight coefficient from the previous frame image before the original image; the sum of the weight coefficient of the previous frame image and the weight coefficient of the original image is 1;

[0045] The target face coefficient corresponding to the original image is obtained by weighted summing the first face coefficient of the original image and the first face coefficient of the previous frame image.

[0046] Optionally, after performing temporal domain correction processing on the template expression coefficients and / or pose coefficients in the first face coefficients, the method further includes:

[0047] Obtain a preset expression adaptation matrix; the expression adaptation matrix refers to the transformation relationship between two face coefficients with different template numbers;

[0048] The target face coefficient is obtained by multiplying the temporally corrected face coefficient with the expression adaptation matrix.

[0049] Optionally, the preset expression adaptation matrix is ​​obtained through the following steps:

[0050] Obtain the first preset coefficient corresponding to the sample image, wherein the first preset coefficient includes the coefficients of a first number of templates;

[0051] Obtain the second preset coefficient corresponding to the sample image, wherein the second preset coefficient includes the coefficients of a second number of templates;

[0052] The preset expression adaptation matrix is ​​obtained based on the first preset coefficient, the second preset coefficient, and the least squares method.

[0053] Optionally, the method further includes:

[0054] When no face region is detected in the original image, the detection continues in the next frame of the original image, and the virtual expression is obtained based on the target face coefficient of the previous frame of the original image.

[0055] or,

[0056] When no face region is detected in the original image and the duration exceeds a set duration threshold, a virtual expression is obtained based on a preset expression coefficient.

[0057] According to a second aspect of the present disclosure, a virtual facial expression generation apparatus is provided, comprising:

[0058] The target image acquisition module is used to acquire the face region in the original image and obtain the target face image;

[0059] The first coefficient acquisition module is used to acquire the first face coefficient corresponding to the target face image; the first face coefficient includes template expression coefficient and pose coefficient, the template expression coefficient is used to characterize the degree of matching between the face expression and each template, and the pose coefficient represents the rotation angle of the virtual image in three dimensions;

[0060] The target coefficient acquisition module is used to perform temporal domain correction processing on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain the target face coefficient;

[0061] The facial expression animation acquisition module is used to render the facial expressions of the virtual character based on the target face coefficients to obtain virtual facial expressions.

[0062] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0063] A camera is used to capture images and / or videos as raw images for the processor.

[0064] A 3D display for displaying virtual expressions of virtual avatars, or the images or videos, in real time;

[0065] Processor and memory for storing executable instructions;

[0066] The processor reads executable instructions from the memory to implement the steps of the method described in any of the first aspects.

[0067] According to a fourth aspect of the present disclosure, a chip is provided, comprising:

[0068] A processor and a memory for storing an executable program; the processor reads the executable program from the memory to implement the steps of the method described in any of the first aspects.

[0069] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer-executable program is stored, which, when executed, implements the steps of the method described in any of the first aspects.

[0070] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0071] As can be seen from the above embodiments, the solution provided in this disclosure can obtain the face region in the original image to obtain the target face image; then, obtain the first face coefficient corresponding to the target face image; subsequently, perform temporal domain correction processing on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain the target face coefficient; finally, render the expression of the virtual image according to the target face coefficient to obtain the virtual expression. Thus, by performing temporal domain correction processing on the first face coefficient in this embodiment, the expressions of adjacent original images in the video can be made correlated and continuous, making the reconstructed expression more natural and improving the viewing experience; furthermore, by transmitting the target face coefficient to render the expression of the virtual image to obtain the virtual expression, the amount of data transmission can be reduced compared to transmitting image data, achieving the effect of real-time reconstruction of virtual expressions.

[0072] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0073] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0074] Figure 1 This is a flowchart illustrating a virtual facial expression generation method according to an exemplary embodiment.

[0075] Figure 2 This is a flowchart illustrating an example of acquiring a target face image.

[0076] Figure 3 This is a flowchart illustrating an example of obtaining a target face region.

[0077] Figure 4 This is a flowchart illustrating a method for obtaining a score value for a face region according to an exemplary embodiment.

[0078] Figure 5This is a flowchart illustrating an example of acquiring a target face image.

[0079] Figure 6 This is a flowchart illustrating an embodiment of obtaining a first face coefficient.

[0080] Figure 7 This is a block diagram illustrating a face recognition network according to an exemplary embodiment.

[0081] Figure 8 This is a flowchart illustrating an embodiment of obtaining the coefficients of a target face.

[0082] Figure 9 This is a flowchart illustrating another method for obtaining the target face coefficient according to an exemplary embodiment.

[0083] Figure 10 This is a flowchart illustrating an example of obtaining an expression adaptation matrix.

[0084] Figure 11 This is a flowchart illustrating another method for obtaining an expression adaptation matrix, according to an exemplary embodiment.

[0085] Figure 12 This is a flowchart illustrating another method for generating virtual facial expressions according to an exemplary embodiment.

[0086] Figure 13 This is a block diagram illustrating a virtual facial expression generation device according to an exemplary embodiment.

[0087] Figure 14 This is a block diagram illustrating a server according to an exemplary embodiment. Detailed Implementation

[0088] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described below by way of example do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatus consistent with some aspects of this disclosure as detailed in the appended claims. It should be noted that, without conflict, the following embodiments and features in the implementation methods can be combined with each other.

[0089] 3D modeling is a key issue in the field of machine vision, with 3D facial expression modeling being widely used in entertainment fields such as games, film and television special effects, and VR. Current mainstream methods for 3D virtual facial expression modeling are based on generating 3D virtual expressions from images. However, due to the complex structure of the human face and the coordinated movements of facial muscles, the process of facial expression changes is a complex non-rigid body motion. This places high demands on the acquisition equipment, acquisition environment, modeling equipment, and modeling process, making real-time performance difficult. Furthermore, when processing individual frames of images in a video, the correlation and continuity of facial expressions are ignored.

[0090] To address the aforementioned technical problems, this disclosure provides a method for generating virtual facial expressions, which can be applied to electronic devices. Figure 1 This is a flowchart illustrating a virtual facial expression generation method according to an exemplary embodiment.

[0091] See Figure 1 A method for generating virtual facial expressions, comprising steps 11 to 14.

[0092] In step 11, the face region in the original image is obtained to obtain the target face image.

[0093] In this embodiment, the electronic device can communicate with a camera to acquire images and / or videos captured by the camera, and the camera's capture frame rate does not exceed 60fps; it can also read images and / or videos from a specified location. Considering that the electronic device processes one image or one frame of video at a time, the embodiments are described using the example of processing one image subsequently, and the processed image is referred to as the original image to distinguish it from other processed images.

[0094] In this embodiment, after acquiring the original image, the electronic device can obtain the face region in the original image, see [link to documentation]. Figure 2 This includes steps 21 to 23.

[0095] In step 21, the electronic device can perform face detection on the original image to obtain at least one face region contained in the original image.

[0096] In this step, the electronic device can use a preset face detection model to perform face detection on the original image. The preset face detection model can include, but is not limited to, models capable of object detection such as YOLOv5, ResNet18, R-CNN, and MobileNet. Those skilled in the art can select an appropriate model according to the specific scenario, and the corresponding solution falls within the protection scope of this disclosure. In this way, the preset face detection model can output at least one face region contained in the original image.

[0097] It should be noted that during face detection, the electronic device can record whether a face region is detected. When no face region exists, the marker can be set to -1; when a face region exists, the marker can be set to the number of face regions. Simultaneously, the region parameter data for each face region is recorded. This region parameter data includes length, width, face area, and position data.

[0098] For example, when there is only one face region, the region parameter data of the face region is [x, y, w, h, s], where x and y represent the horizontal and vertical coordinates of a specified point of the face region (such as the center point, top left vertex, bottom left vertex, top right vertex, or bottom right vertex), w and h represent the width and height of the face region, and s represents the area of ​​the face region. As another example, when there are n1 face regions (n1 is an integer greater than 1), the region parameter data of the n face regions is represented by a list, i.e., [[x1, y1, w1, h1, s1], [x2, y2, w2, h2, s2], ..., [xn1, yn1, wn1, hn1, sn1]].

[0099] In step 22, the electronic device can select a target face region from the at least one face region.

[0100] For example, when there is only one face region, the electronic device can identify that face region as the target face region.

[0101] For example, when there are multiple face regions (e.g., n1 is an integer greater than 1), the electronic device can select one of these multiple face regions as the target face region. See also Figure 3 In step 31, the electronic device can calculate a score for each face region based on the region parameter data of each face region. The score is used to indicate the degree of distance of each face region from the central axis of the original image. The central axis of the original image refers to the vertical line passing through the center point of the original image. For example, if the size of the original image is 1920*1080, then the line x = 960 can be used as the central axis of the original image.

[0102] In this example, see Figure 4 The electronic device acquires the score values ​​of each face region, including steps 41 to 46.

[0103] In step 41, the electronic device can obtain the difference between the horizontal coordinate of the midpoint of each face region and half its width, as well as the absolute value of the difference. For example, the absolute value of the difference is... Where xn1 represents the x-coordinate of the n1th face region, w represents the width of the n1th face region, and || represents the absolute value.

[0104] In step 42, the electronic device can obtain the ratio of the absolute value of the difference to the width, and the product of the ratio and a constant 2. For example, the product of the ratio and a constant 2 is...

[0105] In step 43, the electronic device can obtain the difference between the constant 1 and the product, and obtain the product of the difference corresponding to the product and a preset distance weight. For example, the product of the difference corresponding to the product and the preset distance weight is... α represents the preset distance weight, or the normalized value of the distance from the center of the face region to the central axis. This α is affected by the camera's acquisition distance, and in one example, the value of α is 0.2.

[0106] In step 44, the electronic device can obtain the ratio of the face area to the product of the length and width in each face region, and the square root of the ratio corresponding to the face area. For example, sn represents the area of ​​the face in the n1th face region, h represents the height of the n1th face region, and w represents the width of the n1th face region.

[0107] In step 45, the electronic device can obtain the product of the square root and a preset area weight, wherein the sum of the area weight and the distance weight is 1. For example, the product of the square root and the preset area weight is... 1-α represents the normalized value of the area of ​​the face relative to the area of ​​the original image.

[0108] In step 46, the electronic device can calculate the sum of the product of the preset area weight and the product of the preset distance weight to obtain the score value for each face region. For example, the score value for each face region is shown in the following formula (1):

[0109]

[0110] In step 32, the electronic device can determine the face region corresponding to the maximum value of the score as the target face region.

[0111] In this example, by identifying the face region with the highest score as the target face region, the face region that is closest to the central axis of the original image and has a large face area can be determined. This can closely resemble the scene of the object of interest in the shooting area during the actual image acquisition process, which helps to improve the accuracy of the acquired target face region.

[0112] In step 23, the electronic device can perform correction processing on the target face region to obtain a target face image.

[0113] In this step, see Figure 5The electronic device corrects the target face region, including steps 51 to 54.

[0114] In step 51, the electronic device can determine the candidate square region corresponding to the target face region and obtain the vertex coordinate data of the candidate square region. The electronic device can obtain the center point (x) of the target face region. n1 y n1 ), with the center point (x) n1 y n1 A square region is defined with ... Where scale is the magnification factor for the target face region, and its value is greater than 1. In one example, scale is 1.25. n1 h n1 These represent the width and height of the target face region, respectively. The electronic device can acquire the vertex coordinates of each vertex of the square region. For ease of description, the above square region will be referred to as the candidate square region.

[0115] In step 52, the electronic device can perform an affine transformation on the vertex coordinate data of the candidate square region and the vertex coordinate data of the preset square to obtain affine transformation coefficients; the vertex coordinate data of the preset square includes a specified origin.

[0116] In this step, the electronic device can store a preset square. The vertex coordinate data of this preset square includes a specified origin (0, 0) and a side length of a preset side length (e.g., 224 pixels). Taking the top-left vertex as the specified origin as an example, the vertex coordinate data of the four vertices of this preset square are the top-left corner (0, 0), the bottom-left corner (0, 224), the top-right corner (224, 0), and the bottom-right corner (224, 224).

[0117] In this step, the electronic device can perform an affine transformation on the candidate square region and the preset square, that is, establish an affine transformation relationship between each vertex of the preset square in the candidate square region to obtain the affine transformation coefficients. Alternatively, the electronic device can scale, translate, and rotate the candidate square region to obtain the preset square. Understandably, obtaining the affine transformation relationship between the two squares can refer to relevant technical solutions, which will not be elaborated upon here.

[0118] In step 53, the electronic device can perform an affine transformation on the original image using the affine transformation coefficients to obtain an affine transformed image.

[0119] In step 54, the electronic device can crop a square region with a preset side length from the affine transformation image using the specified origin as a reference, and use the image within the cropped square region as the target face image. For example, the electronic device can crop a square with a length and width of 224 from the (0,0) position in the affine transformation image to obtain the target face image.

[0120] Thus, in this example, by applying affine transformation to the target face region, compared to stretching or compressing the face region, the face region achieves better fidelity, resulting in better fidelity of facial expressions. This is beneficial for improving the accuracy of subsequent virtual expression generation. Alternatively, in this example, processing the original image into a high-fidelity normalized target face image improves the accuracy of the first face coefficients in step 12, as well as the realism and lifelikeness of the virtual expressions generated in step 14, thereby enhancing the interactive experience.

[0121] In step 12, the first face coefficient corresponding to the target face image is obtained; the first face coefficient includes template expression coefficient and pose coefficient, the template expression coefficient is used to characterize the degree of matching between the face expression and each template, and the pose coefficient represents the rotation angle of the virtual image in three dimensions.

[0122] In this step, the electronic device can acquire the first face coefficient corresponding to the target face image, see [link / reference]. Figure 6 This includes steps 61 to 65.

[0123] In step 61, the electronic device can perform blurring and sharpening processing on the target face image respectively to obtain at least one blurred image and at least one sharpened image.

[0124] Considering that the target face image is part of the original image and its features are not prominent, this step first extracts the overall features and / or detailed features of the target face image.

[0125] Taking the acquisition of overall contour features as an example, in this step, the electronic device can blur the target face image. The blurring algorithm used includes, but is not limited to, Gaussian Blur, Box Blur, Kawase Blur, Dual Blur, Bokeh Blur, Tilt Shift Blur, Iris Blur, Grainy Blur, Radial Blur, and Directional Blur. In one example, the Gaussian Blur algorithm is used to process the target face image to obtain at least one model image corresponding to the target face image.

[0126] Taking the acquisition of detailed contour features as an example, in this step, the electronic device can sharpen the target face image. The sharpening algorithm used includes, but is not limited to, the Robert operator, the Prewitt operator, the Sobel operator, the Laplacian operator, the Kirsch operator, etc. In one example, the Robert operator is used to process the target face image, thereby obtaining at least one sharpened image corresponding to the target face image.

[0127] In some examples, the above blurring and / or sharpening algorithms can also be implemented using neural networks (such as convolutional neural networks) in the field of machine vision, which can also produce blurred and / or sharpened images, and the corresponding schemes fall within the protection scope of this disclosure.

[0128] In this step, by blurring and sharpening the target face image, it is easier to use the overall contour features, detailed contour features, and original features of the target face image, thereby enriching the number and categories of features of the target face image and improving the accuracy of obtaining the first face coefficient in the subsequent process.

[0129] In step 62, the electronic device can extract feature data from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image. For example, the electronic device can perform at least one convolution operation on the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image.

[0130] In step 63, the electronic device can stitch together the original feature image, the blurred feature image, and the sharpened feature image to obtain an initial feature image. For example, the electronic device can stitch a blurred feature image after the original feature image; after the blurred feature image is stitched together, a sharpened feature image is stitched after the blurred feature image, until all feature images are stitched together, resulting in a feature image consisting of blurred features, original features, and sharpened features, which is referred to as the initial feature image.

[0131] In step 64, the electronic device can obtain the importance coefficients of each feature image in the initial feature image to the expression of the virtual character, and adjust the initial feature image according to the importance coefficients to obtain the target feature image. For example, the electronic device can obtain the importance coefficients of each feature image in the initial feature image to the expression of the virtual character through a temporal attention mechanism and / or spatial attention mechanism. Then, the electronic device can calculate the product of the above importance coefficients and the initial feature image to obtain the target feature image.

[0132] In this way, by adjusting the initial feature image through the importance coefficient in this step, the relatively important feature images can be highlighted while the relatively unimportant feature images are weakened, thereby improving the accuracy of the target feature image and thus improving the accuracy of the first face coefficient obtained in step 65.

[0133] In step 65, the electronic device can determine the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient.

[0134] In this step, the electronic device can store a preset set of facial expression templates, each template being called an expression base. The electronic device can match the fit between the target feature image and each expression base to determine the template expression coefficients and pose coefficients, thereby obtaining the aforementioned first face coefficients. Alternatively, by adjusting each expression base using the template expression coefficients in the first face coefficients, and by adjusting the spatial pose of each expression base using the pose coefficients, the aforementioned target feature image can be reconstructed.

[0135] In another embodiment, the electronic device may store a preset face coefficient recognition network. The electronic device can input the target face image into the preset face coefficient recognition network, and the preset face coefficient recognition network will output the first face coefficient corresponding to the target face image.

[0136] See Figure 7 The aforementioned preset face coefficient recognition network includes: a blurring and sharpening module 71, a feature extraction module 72, an attention module 73, and a coefficient learning module 74. Among these,

[0137] The blurring and sharpening module 71 performs blurring and sharpening processing on the target face image respectively, to obtain at least one blurred image and at least one sharpened image;

[0138] The feature extraction module 72 extracts feature data from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image; and then stitches the original feature image, the blurred feature image, and the sharpened feature image together to obtain the initial feature image.

[0139] Attention module 73 obtains the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjusts the initial feature image according to the importance coefficient to obtain the target feature image; this attention module is implemented using a network model with a temporal attention mechanism or a spatial attention mechanism.

[0140] The coefficient learning module 74 determines the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient. This coefficient learning module is implemented using at least one network model selected from ResNet50, ResNet18, ResNet100, DenseNet, and YoloV5. Technicians can choose the appropriate model based on the specific scenario, and the corresponding solution falls within the protection scope of this disclosure.

[0141] In step 13, the template expression coefficient and / or pose coefficient in the first face coefficient are subjected to temporal domain correction processing to obtain the target face coefficient; the target face coefficient is associated with the face coefficient of the previous frame of the original image.

[0142] In this step, see Figure 8 The electronic device can perform temporal domain correction processing on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain the target face coefficient, including steps 81 to 82.

[0143] In step 81, the electronic device can acquire the first face coefficient and a preset weight coefficient of the previous frame image preceding the original image; the sum of the weight coefficient of the previous frame image and the weight coefficient of the original image is 1.

[0144] In step 82, the electronic device can perform a weighted summation of the first face coefficient of the original image and the first face coefficient of the previous frame image to obtain the target face coefficient corresponding to the original image.

[0145] Thus, in this embodiment, the target face coefficient is obtained by weighted summation, which establishes a correlation between the face coefficients of the current original image and the previous frame. The larger the preset weight coefficient of the previous frame, the greater the proportion of the face coefficient of the previous frame in the subsequent target face coefficient, resulting in smoother parameters between the previous frame and the current original image. This leads to slower changes in the virtual expression corresponding to the current original image compared to the virtual expression corresponding to the previous frame. Conversely, the smaller the preset weight coefficient of the previous frame, the faster the parameters of the previous frame and the current original image change, resulting in faster changes in the virtual expression corresponding to the current original image compared to the virtual expression corresponding to the previous frame. Technicians can select appropriate preset weight coefficients based on specific scenarios to ensure that the expression changes between adjacent original images meet the requirements of the scenario. In one example, the preset weight coefficient of the previous frame is 0.4, and the weight coefficient of the current original image is 0.6.

[0146] It should be noted that when the current original image is the first frame of the video, there is no previous frame. The electronic device can directly use the first face coefficient of the first frame as the target face coefficient, that is, without performing temporal correction on the first face coefficient, thereby ensuring the accuracy of the expression in the first frame.

[0147] Considering the template used in step 65 and / or coefficient learning module 74 when obtaining the first face coefficients

[0148] The virtual emoji generation method disclosed herein uses a fixed set of emoji templates. "Fixed" means that each template in the set is fixed and the number of templates is fixed. Considering that different electronic devices may use different sets of emoji templates, it is necessary to adapt the first face coefficients obtained from different electronic devices, for example, adapting 64 emoji templates to 52 emoji templates. See [link to documentation]. Figure 9 The electronic device adapts the first face coefficient, including steps 91 to 92.

[0149] In step 91, the electronic device can acquire a preset expression adaptation matrix; the expression adaptation matrix refers to the conversion relationship between two face coefficients with different template numbers.

[0150] In this step, the electronic device can store a preset expression adaptation matrix. This preset expression adaptation matrix can be obtained through the following steps, see [link to relevant documentation]. Figure 10 and Figure 11 This includes steps 101 to 103.

[0151] In step 101, the electronic device can acquire the first preset coefficients corresponding to the sample image. The acquisition method can be found in [reference needed]. Figure 6 or Figure 7 The details of the illustrated embodiment will not be repeated here. The aforementioned first preset coefficient includes the coefficients of a first number (e.g., 64) of templates, which refers to the degree of fit between the target feature image and each template (or expression base) in the first number of templates.

[0152] In step 102, the electronic device can acquire the second preset coefficients corresponding to the sample image, wherein the second preset coefficients include the coefficients of a second number of templates. The acquisition method can be found in [reference needed]. Figure 6 or Figure 7 The details of the illustrated embodiment will not be repeated here. The aforementioned second preset coefficient includes the coefficients of a second number (e.g., 52) of templates, which refers to the degree of fit between the sample image and each template (or expression base) in the second number of templates.

[0153] In step 103, the electronic device can obtain the preset expression adaptation matrix based on the first preset coefficient, the second preset coefficient, and the least squares method.

[0154] In this step, the first preset coefficient is: The second preset coefficient is and and Linear relationship The sum of the squares of the differences between the first and second preset coefficients should be as small as possible, as shown in equation (3):

[0155]

[0156] In equation (3), J represents the sum of squares loss, and S∈R k×(j+1) Let k be the number of new expression bases (the second number), and j be the number of base expression data (the first number).

[0157] Calculating equation (3) yields S, as shown in equation (4):

[0158]

[0159] It should be noted that the linear relationship between the first and second preset coefficients is obtained through the following method, as analyzed below:

[0160] In this step, adjusting the first preset coefficient can be divided into adjusting the template expression coefficient and adjusting the pose coefficient. Considering that the pose coefficient has spatial physical meaning, it is only a transformation of different dimensions in space or a transformation of different coordinate systems, such as the conversion between radians and angles, clockwise and counterclockwise directions, and adaptation, etc., this part can refer to the transformation schemes in related technologies, and will not be elaborated here. Therefore, adjusting the first preset coefficient in this step refers to adjusting the template expression coefficient.

[0161] Understandably, considering that the representation of facial expressions in the spatial dimension can be viewed as the shape properties of a spatial geometry formed by multiple discrete vertices, as shown in Equation (5):

[0162] F=((x1,y1,z1),(x2,y2,z2),…,(x i ,y i ,z i ),…,(x m1 ,y m1 ,z m1 (5)

[0163] In equation (5), m1 represents the number of discrete vertices constituting a face, (x i ,y i ,z i ) represents the spatial coordinate data of the i-th vertex.

[0164] When too many discrete vertices are needed to depict facial expressions, the computational load on electronic devices is also relatively large, which is not conducive to generating animations. In this step, the electronic device can use Principal Component Analysis (PCA) to reduce dimensionality, so as to use the motion of low-dimensional discrete vertices to drive the high-dimensional model. After PCA processing, a matrix of eigenvectors can be obtained, that is, the principal component set, in which each principal component in the principal component set is orthogonal to the others, and each principal component serves as an expression basis. Therefore, the 3D expression of a face is a linear combination of a natural expression and the expression basis set, as shown in Equation (6):

[0165]

[0166] In equation (6), Represents a natural expression, that is, a face without any expression, or an initial face; P∈R n×m Let P be a matrix composed of m eigenvectors, considering that each eigenvector represents a blend shape in the application process; P represents a set of blend shapes. The coefficients are represented as facial feature vectors, such as the first preset coefficients or the first face coefficients.

[0167] The expression space, i.e., human facial expressions, can be represented by different natural expressions and different feature vectors, as shown in equation (7):

[0168]

[0169] In equation (7), basic and new represent the basic expression base space and the new expression base space, respectively, and P basic ∈R n×j P new ∈R n×k ,

[0170] Transforming equation (7), we obtain equation (8):

[0171]

[0172] In equation (8), C∈R k×j It is a mapping function between the base expression base and the new expression base.

[0173] Transforming equation (8), we obtain equation (9):

[0174]

[0175] In equation (9), Let the coefficients of the difference characteristic vector be represented. According to equations (8) and (9), equations (10) and (11) can be obtained:

[0176]

[0177]

[0178] Combining equations (7) and (11), we obtain equation (2):

[0179]

[0180] In step 92, the electronic device can calculate the product of the temporally corrected face coefficients and the expression adaptation matrix to obtain the target face coefficients. Thus, the target face coefficients in this step are corrected coefficients, achieving a transformation from different expression bases to other expression bases, thereby matching the target face coefficients with the corresponding expression bases and achieving the effect of expression transfer.

[0181] In step 14, the virtual image's expression is rendered based on the target face coefficient to obtain a virtual expression.

[0182] In this step, the electronic device can use the target face coefficients to render the expressions of the virtual character. For example, the electronic device can transmit the target face coefficients via UDP (User Datagram Protocol) broadcast. Then, the preset rendering program (such as Unity) renders the image when it receives the UDP data. Finally, the virtual expressions of the virtual character are displayed in real time using a 3D display.

[0183] In one embodiment, when no face region is detected in the original image, the electronic device can render the virtual avatar's expression based on the target face coefficients of the previous frame of the original image to obtain a virtual expression, thereby making the expressions of the virtual avatar in adjacent frames of the original image correlated and continuous. Furthermore, the electronic device can continue to detect the next frame of the original image, i.e., re-execute step 11.

[0184] In another embodiment, the electronic device can initiate a timer (or count) when no face region is detected in the original image. If the timer duration exceeds a set duration threshold (e.g., 3-5 seconds) and the electronic device still cannot detect a face region, it obtains a virtual expression based on a preset expression coefficient to display the initial expression of the virtual avatar. Furthermore, the electronic device can reduce the face detection frequency to save processing resources; for example, it can detect a face region every 3-5 frames of the original image until a face region is detected again, at which point it reverts to detecting a face region every frame of the original image.

[0185] Thus, the solution provided in this embodiment can obtain the face region in the original image to obtain the target face image; then, obtain the first face coefficient corresponding to the target face image; subsequently, perform temporal domain correction processing on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain the target face coefficient; finally, render the expression of the virtual image according to the target face coefficient to obtain the virtual expression. In this way, by performing temporal domain correction processing on the first face coefficient in this embodiment, the expressions of adjacent original images in the video can be made correlated and continuous, making the reconstructed expression more natural and improving the viewing experience; furthermore, by transmitting the target face coefficient to render the expression of the virtual image to obtain the virtual expression, the amount of data transmission can be reduced compared to transmitting image data, achieving the effect of real-time reconstruction of virtual expressions.

[0186] This disclosure provides a method for generating virtual facial expressions; see [link to previous section]. Figure 12 This includes steps 121 to 128.

[0187] In step 121, the model is initialized, and the model structure and parameters are loaded.

[0188] In step 122, the camera captures video at a frame rate not exceeding 60fps.

[0189] In step 123, face detection and correction involves using a preset face detection model to obtain all face regions in the video frame (i.e., the original image); selecting the best face based on the weighted values ​​of face size and face center position, and correcting it to make it a 224×224 pixel face image to meet the input requirements of the face coefficient recognition network.

[0190] In step 124, template expression coefficients are generated. The 224×224 pixel face image obtained in step 123 is sent to the face coefficient recognition network to obtain the first face coefficients, which are used to depict the expression and pose of the face.

[0191] In step 125, the adaptation correction mainly involves mapping the basic expression base coefficients to new expression base coefficients and the transformation of pose coefficients. The new expression base coefficients can be regarded as a linear combination of the basic expression base coefficients, so this process is only a matrix multiplication in the overall implementation process. The pose coefficients have clear physical meanings, and it is only necessary to fix the template pose according to the actual physical meaning.

[0192] In step 126, temporal correction takes into account the temporal correlation of facial expressions, rather than the independent reconstruction of expressions in each frame. Therefore, temporal correction of expression coefficients and pose coefficients is introduced to smooth the facial expression transformation process and improve the continuity and stability of 3D virtual expressions.

[0193] In step 127, rendering virtual expressions using the Unity program involves passing the processed expression coefficients and pose coefficients (i.e., the target face coefficients) into the Unity program via a UDP port to drive the motion of the established virtual expressions.

[0194] In step 128, the 3D display device is used to view 3D virtual expressions. Then, steps 122 to 127 are repeated to achieve real-time interaction with the 3D virtual expressions.

[0195] Based on the virtual expression generation method provided in this disclosure, this disclosure also provides a virtual expression generation device, see [link to relevant documentation]. Figure 13 The device includes:

[0196] The target image acquisition module 131 is used to acquire the face region in the original image to obtain the target face image;

[0197] The first coefficient acquisition module 132 is used to acquire the first face coefficient corresponding to the target face image; the first face coefficient includes template expression coefficient and pose coefficient, the template expression coefficient is used to characterize the degree of matching between the face expression and each template, and the pose coefficient represents the rotation angle of the virtual image in three dimensions.

[0198] The target coefficient acquisition module 133 is used to perform temporal domain correction processing on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain the target face coefficient; the target face coefficient is associated with the face coefficient of the previous frame of the original image;

[0199] The facial expression animation acquisition module 134 is used to render the facial expression of the virtual image based on the target face coefficient to obtain a virtual facial expression.

[0200] In one embodiment, the target image acquisition module includes:

[0201] The face region acquisition submodule is used to perform face detection on the original image and obtain at least one face region contained in the original image;

[0202] The target region acquisition submodule is used to select a target face region from the at least one face region;

[0203] The target image acquisition submodule is used to perform correction processing on the target face region to obtain the target face image.

[0204] In one embodiment, the target region acquisition submodule includes:

[0205] The first determining unit is configured to determine the face region as the target face region when the number of face regions is one.

[0206] The second determining unit is used to calculate a score value for each face region based on the region parameter data of each face region when there are multiple face regions, wherein the score value is used to represent the degree of distance of each face region from the central axis of the original image; and to determine the face region corresponding to the maximum score value as the target face region.

[0207] In one embodiment, the region parameter data includes length, width, face area, and location data, and the second determining unit includes:

[0208] The absolute value acquisition subunit is used to obtain the difference between the horizontal coordinate of the middle position of each face region and half the width, as well as the absolute value of the difference;

[0209] A ratio acquisition subunit is used to acquire the ratio of the absolute value of the difference to the width, and the product of the ratio and a constant 2;

[0210] The product acquisition subunit is used to obtain the difference between the constant 1 and the product, and to obtain the product of the difference corresponding to the product and the preset distance weight;

[0211] The square root acquisition subunit is used to obtain the ratio of the face area to the product of the length and width in each face region, and the square root of the ratio corresponding to the face area;

[0212] The product acquisition subunit is used to obtain the product of the square root and the preset area weight, wherein the sum of the area weight and the distance weight is 1;

[0213] The scoring value acquisition subunit is used to calculate the sum of the product of the preset area weight and the product of the preset distance weight to obtain the scoring value of each face region.

[0214] In one embodiment, the target image acquisition submodule includes:

[0215] The candidate region acquisition unit is used to determine the candidate square region corresponding to the target face region and obtain the vertex coordinate data of the candidate square region.

[0216] The affine coefficient acquisition unit is used to perform an affine transformation on the vertex coordinate data of the candidate square region and the vertex coordinate data of the preset square to obtain affine transformation coefficients; the vertex coordinate data of the preset square includes a specified origin;

[0217] An affine image acquisition unit is used to perform an affine transformation on the original image using the affine transformation coefficients to obtain an affine transformed image.

[0218] The target image acquisition unit is used to extract a square region with a preset side length from the affine transformation image based on the specified origin, and to use the image within the extracted square region as the target face image.

[0219] In one embodiment, the first coefficient acquisition module includes:

[0220] The image processing submodule is used to perform blurring and sharpening processing on the target face image respectively, to obtain at least one blurred image and at least one sharpened image;

[0221] The feature image acquisition submodule is used to extract feature data from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image.

[0222] The initial image acquisition submodule is used to stitch together the original feature image, the blurred feature image, and the sharpened feature image to obtain the initial feature image;

[0223] The target image acquisition submodule is used to acquire the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjust the initial feature image according to the importance coefficient to obtain the target feature image;

[0224] The face coefficient acquisition submodule is used to determine the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient.

[0225] In one embodiment, the first coefficient acquisition module includes:

[0226] The first coefficient acquisition submodule is used to input the target face image into a preset face coefficient recognition network to obtain the first face coefficient corresponding to the target face image output by the preset face coefficient recognition network.

[0227] In one embodiment, the preset face coefficient recognition network includes: a blur sharpening module, a feature extraction module, an attention module, and a coefficient learning module;

[0228] The blurring and sharpening module performs blurring and sharpening processing on the target face image respectively, to obtain at least one blurred image and at least one sharpened image;

[0229] The feature extraction module extracts feature data from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image; and then stitches the original feature image, the blurred feature image, and the sharpened feature image together to obtain the initial feature image.

[0230] The attention module obtains the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjusts the initial feature image according to the importance coefficient to obtain the target feature image;

[0231] The coefficient learning module determines the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient.

[0232] In one embodiment, the attention module is implemented using a network model with a temporal attention mechanism or a spatial attention mechanism.

[0233] In one embodiment, the coefficient learning module is implemented using at least one network model selected from ResNet50, ResNet18, ResNet100, DenseNet, and YoloV5.

[0234] In one embodiment, the target coefficient acquisition module includes:

[0235] The weight coefficient acquisition submodule is used to acquire the first face coefficient and the preset weight coefficient of the previous frame image before the original image; the sum of the weight coefficient of the previous frame image and the weight coefficient of the original image is 1;

[0236] The target coefficient acquisition submodule is used to perform a weighted summation of the first face coefficient of the original image and the first face coefficient of the previous frame image to obtain the target face coefficient corresponding to the original image.

[0237] In one embodiment, the device further includes:

[0238] The adaptation matrix acquisition module is used to acquire a preset expression adaptation matrix; the expression adaptation matrix refers to the transformation relationship between two face coefficients with different template numbers.

[0239] The target coefficient acquisition module is used to calculate the product of the face coefficient after temporal correction and the expression adaptation matrix to obtain the target face coefficient.

[0240] In one embodiment, the preset expression adaptation matrix is ​​obtained through the following steps:

[0241] Obtain the first preset coefficient corresponding to the sample image, wherein the first preset coefficient includes the coefficients of a first number of templates;

[0242] Obtain the second preset coefficient corresponding to the sample image, wherein the second preset coefficient includes the coefficients of a second number of templates;

[0243] The preset expression adaptation matrix is ​​obtained based on the first preset coefficient, the second preset coefficient, and the least squares method.

[0244] In one embodiment, the facial expression animation acquisition module is further configured to continue detecting the next frame of the original image when no face region is detected in the original image, and to acquire virtual facial expressions based on the target face coefficients of the previous frame of the original image;

[0245] or,

[0246] The facial expression animation acquisition module is also used to acquire virtual facial expressions based on preset facial expression coefficients when no face region is detected in the original image and the duration exceeds a set duration threshold.

[0247] It should be noted that the apparatus shown in this embodiment matches the content of the method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0248] In an exemplary embodiment, an electronic device is also provided, see [link to example]. Figure 14 ,include:

[0249] Camera 143 is used to acquire images and / or videos as raw images for the processor;

[0250] 3D display 144, used to display the virtual expressions of the virtual avatar, or the image or video in real time;

[0251] Processor 141; memory 142 for storing computer programs executable by the processor;

[0252] The processor is configured to execute a computer program in the memory to achieve, for example, Figures 1-12 The method described.

[0253] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as a memory including an executable computer program, which can be executed by a processor to achieve, for example... Figures 1-12 The method of the illustrated embodiment. The readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0254] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0255] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating virtual facial expressions, characterized in that, include: Obtain the face region from the original image to obtain the target face image; Obtaining the first face coefficient corresponding to the target face image includes: performing blurring and sharpening processing on the target face image respectively to obtain at least one blurred image and at least one sharpened image; extracting feature data from the target face image, each blurred image, and each sharpened image respectively to obtain an original feature image, a blurred feature image, and a sharpened feature image; stitching the original feature image, the blurred feature image, and the sharpened feature image to obtain an initial feature image; obtaining the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjusting the initial feature image according to the importance coefficient to obtain a target feature image; determining the template expression coefficient and pose coefficient according to the target feature image to obtain the first face coefficient; or, inputting the target face image into a preset face coefficient recognition network to obtain the first face coefficient corresponding to the target face image output by the preset face coefficient recognition network. The preset face coefficient recognition network includes: a blurring and sharpening module, a feature extraction module, an attention module, and a coefficient learning module. The blurring and sharpening module performs blurring and sharpening processing on the target face image to obtain at least one blurred image and at least one sharpened image. The feature extraction module extracts feature data from the target face image, each blurred image, and each sharpened image to obtain an original feature image, a blurred feature image, and a sharpened feature image. The original feature image, the blurred feature image, and the sharpened feature image are then stitched together to obtain an initial feature image. The attention module obtains the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjusts the initial feature image according to the importance coefficient to obtain a target feature image. The coefficient learning module determines the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient. The first face coefficient includes a template expression coefficient and a pose coefficient. The template expression coefficient is used to characterize the degree of matching between the face expression and each template, and the pose coefficient represents the rotation angle of the virtual character in three dimensions. The process of performing temporal domain correction on the template expression coefficient and / or pose coefficient in the first face coefficient to obtain the target face coefficient includes: acquiring the first face coefficient and a preset weight coefficient of the previous frame image before the original image; the sum of the weight coefficient of the previous frame image and the weight coefficient of the original image is 1; performing a weighted summation on the first face coefficient of the original image and the first face coefficient of the previous frame image to obtain the target face coefficient corresponding to the original image; the target face coefficient is associated with the face coefficient of the original image in the previous frame image. The virtual avatar's expression is rendered based on the target face coefficients to obtain a virtual expression.

2. The method according to claim 1, characterized in that, Extract the face region from the original image to obtain the target face image, including: Perform face detection on the original image to obtain at least one face region contained in the original image; Select a target face region from the at least one face region; The target face region is corrected to obtain the target face image.

3. The method according to claim 2, characterized in that, Selecting a target face region from at least one face region includes: When the number of face regions is one, the face region is determined to be the target face region; When there are multiple face regions, a score value for each face region is calculated based on the region parameter data of each face region. The score value is used to represent the degree of distance of each face region from the central axis of the original image. The face region corresponding to the maximum score value is determined as the target face region.

4. The method according to claim 3, characterized in that, The region parameter data includes length, width, face area, and location data. A score for each face region is calculated based on the region parameter data, including: Obtain the difference between the horizontal coordinate of the middle position of each face region and half the width, as well as the absolute value of the difference; Obtain the ratio of the absolute value of the difference to the width, and the product of the ratio and the constant 2; Obtain the difference between the constant 1 and the product, and obtain the product of the difference corresponding to the product and the preset distance weight; Obtain the ratio of the face area to the product of the length and width in each face region, and the square root of the ratio corresponding to the face area; Obtain the product of the square root and the preset area weight, wherein the sum of the area weight and the distance weight is 1; The sum of the product of the preset area weight and the product of the preset distance weight is calculated to obtain the score value of each face region.

5. The method according to claim 2, characterized in that, The target face region is corrected to obtain a target face image, including: Determine the candidate square region corresponding to the target face region, and obtain the vertex coordinate data of the candidate square region; An affine transformation is performed on the vertex coordinate data of the candidate square region and the vertex coordinate data of the preset square to obtain affine transformation coefficients; the vertex coordinate data of the preset square includes a specified origin. The original image is subjected to an affine transformation using the affine transformation coefficients to obtain an affine transformed image. Using the designated origin as a reference, a square region with a preset side length is extracted from the affine transformation image, and the image within the extracted square region is used as the target face image.

6. The method according to claim 1, characterized in that, The attention module is implemented using a network model with either a temporal attention mechanism or a spatial attention mechanism.

7. The method according to claim 1, characterized in that, The coefficient learning module is implemented using at least one network model selected from ResNet50, ResNet18, ResNet100, DenseNet, and YoloV5.

8. The method according to claim 1, characterized in that, After performing temporal domain correction processing on the template expression coefficients and / or pose coefficients in the first face coefficients, the method further includes: Obtain a preset expression adaptation matrix; the expression adaptation matrix refers to the transformation relationship between two face coefficients with different template numbers; The target face coefficient is obtained by multiplying the temporally corrected face coefficient with the expression adaptation matrix.

9. The method according to claim 8, characterized in that, The preset expression adaptation matrix is ​​obtained through the following steps: Obtain the first preset coefficient corresponding to the sample image, wherein the first preset coefficient includes the coefficients of a first number of templates; Obtain the second preset coefficient corresponding to the sample image, wherein the second preset coefficient includes the coefficients of a second number of templates; The preset expression adaptation matrix is ​​obtained based on the first preset coefficient, the second preset coefficient, and the least squares method.

10. The method according to claim 1, characterized in that, The method further includes: When no face region is detected in the original image, the detection continues in the next frame of the original image, and the virtual expression is obtained based on the target face coefficient of the previous frame of the original image. or, When no face region is detected in the original image and the duration exceeds a set duration threshold, a virtual expression is obtained based on a preset expression coefficient.

11. A virtual facial expression generation device, characterized in that, include: The target image acquisition module is used to acquire the face region in the original image and obtain the target face image; The first coefficient acquisition module is used to acquire the first face coefficient corresponding to the target face image; the first face coefficient includes template expression coefficient and pose coefficient, the template expression coefficient is used to characterize the degree of matching between the face expression and each template, and the pose coefficient represents the rotation angle of the virtual image in three dimensions; The target coefficient acquisition module is used to perform temporal domain correction processing on the template expression coefficient and / or the pose coefficient in the first face coefficient to obtain the target face coefficient; The facial expression animation acquisition module is used to render the facial expressions of the virtual character based on the target face coefficients to obtain virtual facial expressions; The target coefficient acquisition module includes: The weight coefficient acquisition submodule is used to acquire the first face coefficient and the preset weight coefficient of the previous frame image before the original image; the sum of the weight coefficient of the previous frame image and the weight coefficient of the original image is 1; The target coefficient acquisition submodule is used to perform a weighted summation of the first face coefficient of the original image and the first face coefficient of the previous frame image to obtain the target face coefficient corresponding to the original image. The first coefficient acquisition module includes: The image processing submodule is used to perform blurring and sharpening processing on the target face image respectively, to obtain at least one blurred image and at least one sharpened image; The feature image acquisition submodule is used to extract feature data from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image. The initial image acquisition submodule is used to stitch together the original feature image, the blurred feature image, and the sharpened feature image to obtain the initial feature image; The target image acquisition submodule is used to acquire the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjust the initial feature image according to the importance coefficient to obtain the target feature image; The face coefficient acquisition submodule is used to determine the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient; Alternatively, the first coefficient acquisition module includes: The first coefficient acquisition submodule is used to input the target face image into a preset face coefficient recognition network to obtain the first face coefficient corresponding to the target face image output by the preset face coefficient recognition network. The preset face coefficient recognition network includes: a blur sharpening module, a feature extraction module, an attention module, and a coefficient learning module; The blurring and sharpening module performs blurring and sharpening processing on the target face image respectively, to obtain at least one blurred image and at least one sharpened image; The feature extraction module extracts feature data from the target face image, each blurred image, and each sharpened image to obtain the original feature image, the blurred feature image, and the sharpened feature image; and then stitches the original feature image, the blurred feature image, and the sharpened feature image together to obtain the initial feature image. The attention module obtains the importance coefficient of each feature image in the initial feature image to the expression of the virtual character, and adjusts the initial feature image according to the importance coefficient to obtain the target feature image; The coefficient learning module determines the template expression coefficient and pose coefficient based on the target feature image to obtain the first face coefficient.

12. An electronic device, characterized in that, include: A camera is used to capture images and / or video as raw images for a processor. A 3D display for displaying virtual expressions of virtual avatars, or the images or videos, in real time; Processor and memory for storing executable instructions; The processor reads executable instructions from the memory to implement the steps of the method according to any one of claims 1 to 10.

13. A chip, characterized in that, include: Processor and memory for storing executable programs; The processor reads an executable program from the memory to implement the steps of the method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium having a computer-executable program stored thereon, characterized in that, When the executable program is executed, it performs the steps of the method according to any one of claims 1 to 10.

Images