Facial animation generation method, device, equipment and storage medium

Abstract

The present application provides a kind of face animation generation method, device, equipment and storage medium, wherein the method comprises: obtaining video stream data and line drawing image, and extracting driving sampling point to the video stream data;According to the position of the driving sampling point in the video stream data, the driving range of the driving sampling point is calculated;Heat map identification is carried out based on each frame video image in the video stream data, and corresponding heat map face information is generated;The face region of the line drawing image is driven by the driving sampling point based on the heat map face information and the driving range, and a face animation is generated.The present application can drive the line drawing image using the video stream data, provide more efficient animation production while retaining the processing of art, the premise of the quality of painting, and the final output video has better precision quality, and there is the facial micro-expression of the character in the video.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for generating facial animation. Background Technology

[0002] Current technologies for generating facial character animations primarily rely on frame-by-frame hand-drawn illustrations. While this allows artists to create characters with diverse artistic styles, animators then rig and fill in missing details after the artwork is completed. The final video content is then produced according to production requirements. Simulating subtle muscle expressions within thick paint requires highly skilled rigging techniques from the artists. Adding details to numerous gaps not only demands high skill from the animators but also incurs significant art costs, making it extremely difficult for mass-produced content. Other AI technologies employ a pixel-to-pixel approach. This method requires a high degree of similarity between the video content and the character illustration. If animation is to be driven by the video, the actor must resemble the illustration as closely as possible. This limits the artist's ability to freely create the desired facial features, sacrificing subjectivity and artistic refinement. Summary of the Invention

[0003] The main objective of this invention is to solve the technical problem that existing methods for generating facial portrait animations cannot preserve the subjectivity and artistic processing of art.

[0004] The first aspect of the present invention provides a method for generating facial animation, the method comprising: acquiring video stream data and a portrait image, and extracting driving sampling points from the video stream data; calculating the driving range of the driving sampling points based on their positions in the video stream data; performing heatmap recognition on each frame of video image in the video stream data to generate corresponding heatmap facial information; and driving the facial region of the portrait image using the driving sampling points based on the heatmap facial information and the driving range to generate facial animation.

[0005] A second aspect of the present invention provides a facial animation generation apparatus, comprising: an acquisition module for acquiring video stream data and a portrait image, and extracting driving sampling points from the video stream data; a range calculation module for calculating the driving range of the driving sampling points based on their positions in the video stream data; a heatmap generation module for performing heatmap recognition on each frame of video image in the video stream data to generate corresponding heatmap facial information; and a driving module for driving the facial region of the portrait image based on the heatmap facial information and the driving range using the driving sampling points to generate facial animation.

[0006] A third aspect of the present invention provides a facial animation generation apparatus, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the facial animation generation apparatus to perform the steps of the facial animation generation method described above.

[0007] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described facial animation generation method.

[0008] The aforementioned facial animation generation method, apparatus, device, and storage medium acquire video stream data and a portrait image, and extract driving sampling points from the video stream data; calculate the driving range of the driving sampling points based on their positions in the video stream data; perform heatmap recognition on each frame of the video image in the video stream data to generate corresponding heatmap facial information; and drive the facial region of the portrait image using the driving sampling points based on the heatmap facial information and the driving range to generate facial animation. In this method, by using video stream data to drive the portrait image, more efficient animation production can be provided while preserving artistic processing and painting quality. The final video output also has better precision quality and includes micro-facial expressions of the characters in the video.

[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the first embodiment of the facial animation generation method in this invention;

[0012] Figure 2 This is a schematic diagram of one embodiment of the facial animation generation device in this invention;

[0013] Figure 3 This is a schematic diagram of another embodiment of the facial animation generation device in this invention;

[0014] Figure 4This is a schematic diagram of one embodiment of the facial animation generation device in this invention. Detailed Implementation

[0015] This invention provides a method, apparatus, device, and storage medium for generating facial animation. The method involves acquiring video stream data and a portrait image, extracting driving sampling points from the video stream data, calculating the driving range of the driving sampling points based on their positions in the video stream data, performing heatmap recognition on each frame of the video stream data to generate corresponding heatmap facial information, and then driving the facial region of the portrait image using the driving sampling points based on the heatmap facial information and the driving range to generate facial animation. This method, by using video stream data to drive the portrait image, can provide more efficient animation production while preserving artistic processing and painting quality. The final video output also has better precision quality and includes micro-expressions of the characters' faces.

[0016] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the facial animation generation method in this invention includes:

[0018] 101. Acquire video stream data and a portrait image, and extract driving sampling points from the video stream data;

[0019] It is understood that the executing entity of this invention can be a facial animation generation device, a terminal, or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0020] In this embodiment, the acquisition of video stream data and portrait images, and the extraction of driving sampling points from the video stream data, specifically involves acquiring video stream data and portrait images, performing frame-by-frame processing on the video stream data to obtain multiple video images; performing facial key point detection on the multiple video images to obtain the facial key points corresponding to each video image and the position information of the facial key points; determining the facial region type of the corresponding facial key points based on the position information, and filtering the facial key points based on the facial region type; generating discrete points within a preset range from the filtered facial key points according to the corresponding facial region type, and using the facial key points and the corresponding discrete points as driving sampling points for the corresponding video images.

[0021] Specifically, video stream data can be real-time video captured by the camera of an electronic device, or video cached in the electronic device. Determining whether the video stream data contains multiple frames of video images of people is crucial. It should be noted that the content captured in the video stream data may include both human and non-human images. Therefore, to implement face-driven facial recognition, it is necessary to identify multiple frames of video images of people within the video stream data to facilitate subsequent face-driven facial recognition in the images to be processed. The facial region types include eyebrow, pupil, nose, and other facial features. The facial region type of the facial key points can be determined based on the positional relationship between them. Since facial key points also include facial boundaries, these boundaries are not affected by face-driven recognition. Facial key points excluding facial boundaries can be filtered out. Furthermore, since the actual facial key points may be a portion of a facial region rather than the center of the region, these points are a set of discrete points within that region.

[0022] In this embodiment, after acquiring video stream data and portrait images, and extracting driving sampling points from the video stream data, the method further includes: setting the filtered facial key points corresponding to the first frame of the multi-frame video images as facial alignment standard points; calculating the affine transformation matrix between the first frame of the video images and the video images other than the first frame of the video images based on the position information of the facial alignment standard points and the position information of the facial key points in the video images other than the first frame of the video images; and performing affine transformation on the pixels of each corresponding frame of the video images based on the affine transformation matrix to obtain multi-frame affine transformed video images.

[0023] Specifically, for any given image, due to differences in the resolution, focal length, object distance, framing, and photographic composition of the acquisition device, the location and proportion of the face region within the image often vary. Directly processing the entire image introduces a large amount of unstructured error, making model training difficult to converge and resulting in poor performance on new samples and poor generalization. Therefore, it is necessary to perform face alignment on the input video stream data beforehand, placing the target face region at a specific location in the image through rotation, translation, and scaling.

[0024] Specifically, face alignment can be achieved through affine transformation. The core of the transformation is the affine transformation matrix, which describes the specific parameters of the transformation process. The affine transformation matrix can be obtained by comparing the positional differences of specific pixels in an image before and after the transformation. For face alignment, the distribution of facial features is an excellent natural reference factor. Simply detect the positions of facial features in the original image, and then define the positions where the features should appear in the target image to obtain the transformation matrix. For example, let the selected facial key points corresponding to the first frame of the video image be set as the coordinates of the face alignment standard points.

[0025] Because the affine transformation matrix between points is:

[0026]

[0027] By calculating the values ​​of a1, a2, a3, a4, tx, and ty, and then using these values ​​to calculate the coordinates (x', y') of the point corresponding to (x, y), the pixel value at (x', y') in the new image is set to the pixel value at (x, y) in the original image. In this way, all the pixels in the entire image are moved using affine transformation. After the movement, some pixels in the new image are not located in positions that correspond to pixels in the old image. Therefore, we need to use interpolation to assign values ​​to these positions to achieve face alignment.

[0028] 102. Calculate the driving range of the driving sampling point based on its position in the video stream data;

[0029] In this embodiment, calculating the driving range of the driving sampling point based on its position in the video stream data includes: calculating the comprehensive driving sampling point of the multi-frame affine transformed video images; calculating the initial driving range between the driving sampling point in the first frame video image and the corresponding comprehensive driving sampling point using Wally noise as the initial influence factor; determining the actual influence factor between two video images by backpropagating the positional offset between the driving sampling point of the previous frame video image and the driving sampling point of the next frame video image, as well as the color offset between surrounding pixels, frame by frame starting from the first frame video image; calculating the actual driving range between two video images based on the actual influence factor; and summing the actual driving range between two video images in the video stream data and the initial driving range to obtain the driving range of the driving sampling point.

[0030] Specifically, for the comprehensive driving sampling points obtained after face alignment, the corresponding sampling points can be found in the first frame of the driving video. The two sampling points are used with Wally noise as the initial influence factor. Then, the position of the corresponding sampling point is obtained in the next frame. The actual factor is determined by backpropagation according to the offset of the point and the color offset of the surrounding pixels (the selected pixels have threshold limits and feathering boundaries). The driving range of the driving points is solved step by step in this iterative way.

[0031] 103. Perform heatmap recognition on each frame of video image in the video stream data to generate corresponding heatmap facial information;

[0032] In this embodiment, a heatmap can use color changes to reflect data information in a two-dimensional matrix or table. It can intuitively represent a certain attribute of the data value (such as size or density) with defined color intensities. The initial key points of the person can be the key points of the initial person's body. The corresponding heatmap is used to describe the probability of the initial key points of the person being located at each position in the first video frame. Specifically, based on the key points of the person in each frame of the video image, the probability of the key points of the person at each position in the corresponding video image can be determined, and a corresponding heatmap can be generated based on the probability and the position of the key points. Since the shape of the heatmap corresponding to each key point is different, the heatmaps corresponding to each key point can be unified into a region of a specified shape to obtain the facial information in the heatmap.

[0033] 104. Based on the heatmap facial information and the driving range, the driving sampling points drive the facial region of the portrait image to generate facial animation.

[0034] In this embodiment, the step of driving the facial region of the portrait image based on the heatmap facial information and the driving range through the driving sampling points to generate facial animation includes: generating an affine transformation function for each driving sampling point according to the heatmap facial information and the driving range; determining the facial region in the portrait image corresponding to the driving sampling point; and driving the facial region corresponding to the driving sampling point according to the affine transformation function to generate facial animation.

[0035] Specifically, for each driving sampling point, the corresponding affine transformation equation f() can be calculated. For each sampling point, there exists a heatmap A and a factor B{}, which can form a transformation function f(A,B{}). Since the affine transformation equation for each point is modified according to the backpropagation equation, the variables within the equation are modified. The actual initial transformation equation is f=δ·B{x,y}+factor·A, where δ is the factor of the 2D Gaussian equation for that driving sampling point, and the variable dimension d of the Gaussian equation is defaulted to 2. In practice, it can be finely modified according to the driving method.

[0036] In this embodiment, since using a heatmap method with driving sampling points to track facial features can cause information distortion in areas that are not drawn, the solution is to sample the areas that are actually distorted and lost from the driving video, mix the colors with the driven image, and then use the information from the driving video to supplement the image.

[0037] This step primarily aims to prevent teeth from disappearing during character mouth movements. It's important to note that the color blending in this step involves artistic adjustments. For example, the color samples for the teeth are taken from the shadows of the whites of the eyes and the lips in the original image. This ensures visual consistency (artists typically use similar colors for the whites of the eyes and teeth), while the lips are used to simulate the color inside the mouth. This method of supplementing missing elements in the image differs from other approaches.

[0038] In this embodiment, after generating a facial animation by driving the facial region of the portrait image based on the heatmap facial information through the driving sampling points, the method further includes: inputting the facial animation into a pre-trained SRCNN convolutional neural network for information reconstruction to obtain a new high-resolution facial animation.

[0039] Specifically, the SRCNN convolutional neural network includes a first convolutional network, a second convolutional network, and a third convolutional network. The step of inputting the facial animation into the pre-trained SRCNN convolutional neural network for information reconstruction to obtain a new high-resolution facial animation includes: inputting the facial animation into the pre-trained SRCNN convolutional neural network; extracting image patches from each frame of the facial animation image using the first convolutional network of the SRCNN convolutional neural network; extracting features from the extracted image patches using the second convolutional network and amplifying the extracted features; performing non-linear mapping on the features using the third convolutional network to reconstruct information from the corresponding image patches of the animation image, obtaining a new high-resolution animation image; and assembling a new high-resolution facial animation based on the new high-resolution animation image.

[0040] Specifically, after generating a portrait image using video stream data, the resolution of the generated facial animation is often too low. Therefore, it is necessary to restore the low video quality after the video is generated by signal reconstruction. In this embodiment, an SRCNN convolutional neural network is used. It is essentially a three-layer convolutional network. The first convolutional kernel extracts the content of the image block, the second layer calculates its features and amplifies them, and the last layer is a non-linear mapping of the features to reconstruct the information.

[0041] In this embodiment, before inputting the facial animation into a pre-trained SRCNN convolutional neural network for information reconstruction to obtain a new high-resolution facial animation, the method further includes: obtaining an initial SRCNN convolutional neural network and a training set corresponding to high-resolution and low-resolution images; inputting the low-resolution images from the training set into the initial SRCNN convolutional neural network, and processing the output image through the three-layer convolutional network of the initial SRCNN convolutional neural network; calculating the loss function between the high-resolution image and the corresponding processed image; determining whether the loss function is greater than a preset loss threshold; if so, backpropagating the initial SRCNN convolutional neural network according to the loss function, adjusting the network parameters of the initial SRCNN convolutional neural network, and re-inputting the low-resolution images from the training set into the initial SRCNN convolutional neural network; if not, the network training ends, and a trained SRCNN convolutional neural network is obtained.

[0042] In this embodiment, after backpropagating the initialized SRCNN convolutional neural network according to the loss function and adjusting the network parameters of the initialized SRCNN convolutional neural network, the method further includes: adding numerical precision constraints to the backpropagation function during the backpropagation process and adding corresponding edge resolution.

[0043] Specifically, in using SRCNN technology, it was found that because the library used is so close to real life, the final super-resolution image loses its painterly quality. In comparison, we can clearly see that the nose in SRCNN, due to the AI's training library, becomes remarkably close to a real person, with softer shadow boundaries and more realistic facial features. Our solution is to add numerical precision constraints to the backpropagation function of the SRCNN framework, while simultaneously outputting corresponding boundary calculations. These calculations are detected using the Sobel operator. The Sobel operator, calculated after multiple iterations, can be used to record the brushstroke boundary effects in the image. Finally, after the original image is generated, the image position is re-driven according to the boundary effects, which can be imagined as repainting the image with a brush. This makes the effect closer to that of a real digital painting.

[0044] In this embodiment, video stream data and a portrait image are acquired, and driving sampling points are extracted from the video stream data. Based on the position of the driving sampling points in the video stream data, the driving range of the driving sampling points is calculated. Heatmap recognition is performed on each frame of the video image in the video stream data to generate corresponding heatmap facial information. The facial region of the portrait image is driven by the driving sampling points based on the heatmap facial information and the driving range to generate facial animation. In this method, by using video stream data to drive the portrait image, more efficient animation production can be provided while preserving artistic processing and painting quality. The final video output also has better precision quality and includes micro-facial expressions of the characters in the video.

[0045] The facial animation generation method in the embodiments of the present invention has been described above. The facial animation generation apparatus in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 2 One embodiment of the facial animation generation device in this invention includes:

[0046] The acquisition module 201 is used to acquire video stream data and a portrait image, and to extract driving sampling points from the video stream data;

[0047] The range calculation module 202 is used to calculate the driving range of the driving sampling point based on the position of the driving sampling point in the video stream data;

[0048] The heatmap generation module 203 is used to perform heatmap recognition on each frame of video image in the video stream data and generate corresponding heatmap facial information.

[0049] The driving module 204 is used to drive the facial region of the portrait image based on the heatmap facial information and the driving range through the driving sampling points to generate facial animation.

[0050] In this embodiment of the invention, the facial animation generation device operates the aforementioned facial animation generation method. The device acquires video stream data and a portrait image, and extracts driving sampling points from the video stream data. Based on the position of the driving sampling points in the video stream data, it calculates the driving range of the driving sampling points. It performs heatmap recognition on each frame of the video image in the video stream data to generate corresponding heatmap facial information. The facial region of the portrait image is driven by the driving sampling points based on the heatmap facial information and the driving range to generate facial animation. In this method, by using video stream data to drive the portrait image, more efficient animation production can be provided while preserving artistic processing and painting quality. The final video output also has better precision quality and includes micro-expressions of the characters' faces.

[0051] Please see Figure 3 A second embodiment of the facial animation generation device in this invention includes:

[0052] The acquisition module 201 is used to acquire video stream data and a portrait image, and to extract driving sampling points from the video stream data;

[0053] The range calculation module 202 is used to calculate the driving range of the driving sampling point based on the position of the driving sampling point in the video stream data;

[0054] The heatmap generation module 203 is used to perform heatmap recognition on each frame of video image in the video stream data and generate corresponding heatmap facial information.

[0055] The driving module 204 is used to drive the facial region of the portrait image based on the heatmap facial information and the driving range through the driving sampling points to generate facial animation.

[0056] In this embodiment, the acquisition module 201 is specifically used for:

[0057] Acquire video stream data and a portrait image, and perform frame-by-frame processing on the video stream data to obtain multiple frames of video images;

[0058] Facial key point detection is performed on multiple frames of the video image to obtain the facial key points corresponding to each frame of the video image and the position information of the facial key points;

[0059] Based on the location information, the facial region type of the corresponding facial key points is determined, and the facial key points are filtered based on the facial region type.

[0060] The selected facial key points are used to generate discrete points within a preset range based on the corresponding facial region type, and the facial key points and the corresponding discrete points are used as driving sampling points for the corresponding video image.

[0061] In this embodiment, the facial animation generation device further includes a facial alignment module 205, which is specifically used for:

[0062] Set the filtered facial key points corresponding to the first frame of the multi-frame video image as the facial alignment standard points;

[0063] Based on the position information of the face alignment standard points and the position information of the face key points in the video images other than the first frame video image, calculate the affine transformation matrix between the first frame video image and the video images other than the first frame video image;

[0064] Based on the affine transformation matrix, perform affine transformation on the pixels of each corresponding video frame to obtain multi-frame affine transformed video images.

[0065] In this embodiment, the range calculation module 202 is specifically used for:

[0066] Calculate the comprehensive driving sampling points of the multi-frame affine transformed video images;

[0067] The initial driving range is calculated by using Wally noise as the initial influence factor between the driving sampling points in the first frame of the video image and the corresponding comprehensive driving sampling points.

[0068] Starting from the first video frame, the positional offset between the driving sampling points of the previous video frame and the driving sampling points of the next video frame, as well as the color offset between surrounding pixels, are backpropagated to determine the actual influence factor between the two video frames.

[0069] Calculate the actual driving range between two video frames based on the actual impact factor.

[0070] The driving range of the driving sampling point is obtained by summing the actual driving range between two video frames in the video stream data and the initial driving range.

[0071] In this embodiment, the driving module 204 is specifically used for:

[0072] Affine transformation function for each driving sampling point is generated based on the heatmap facial information and the driving range;

[0073] Determine the facial region in the portrait image that corresponds to the driving sampling point;

[0074] The facial region corresponding to the driving sampling point is driven by the affine transformation function to generate facial animation.

[0075] In this embodiment, the facial animation generation device further includes an information reconstruction module 206, which is used to: input the facial animation into a pre-trained SRCNN convolutional neural network for information reconstruction to obtain a new high-resolution facial animation.

[0076] In this embodiment, the SRCNN convolutional neural network includes a first convolutional network layer, a second convolutional network layer, and a third convolutional network layer;

[0077] The signal reconstruction module 206 is specifically used for:

[0078] The facial animation is input into a pre-trained SRCNN convolutional neural network, and image patches are extracted from each frame of the facial animation through the first layer of the SRCNN convolutional neural network.

[0079] The second convolutional network is used to extract features from the extracted image patches and amplify the extracted features.

[0080] The third convolutional network is used to perform nonlinear mapping on the features to reconstruct information from the corresponding image blocks of the animation image, thereby obtaining a new high-resolution animation image.

[0081] A new high-resolution facial animation is created based on the new high-resolution animated images.

[0082] In this embodiment, the facial animation generation device further includes a model training module 207, which is used for:

[0083] Obtain the initialization of the SRCNN convolutional neural network and the training set that corresponds one-to-one with the high-resolution and low-resolution images;

[0084] The low-resolution images in the training set are input into the initialized SRCNN convolutional neural network, and the processed images are output through the three-layer convolutional network of the initialized SRCNN convolutional neural network.

[0085] Calculate the loss function between the high-resolution image and the corresponding processed image;

[0086] Determine whether the loss function is greater than a preset loss threshold;

[0087] If so, backpropagation is performed on the initialized SRCNN convolutional neural network according to the loss function, the network parameters of the initialized SRCNN convolutional neural network are adjusted, and the low-resolution images in the training set are re-input into the initialized SRCNN convolutional neural network;

[0088] If not, then the network training ends, and the trained SRCNN convolutional neural network is obtained.

[0089] In this embodiment, the facial animation generation device further includes an edge resolution module 208, which is used for:

[0090] Numerical precision constraints are added to the backpropagation function during the backpropagation process, and corresponding edge calculations are added.

[0091] This embodiment, based on the previous embodiment, describes in detail the specific functions of each module and the unit composition of some modules. It acquires video stream data and a portrait image, and extracts driving sampling points from the video stream data. Based on the position of the driving sampling points in the video stream data, it calculates the driving range of the driving sampling points. It performs heatmap recognition on each frame of the video image in the video stream data to generate corresponding heatmap facial information. Based on the heatmap facial information and the driving range, it drives the facial region of the portrait image using the driving sampling points to generate facial animation. In this method, by using video stream data to drive the portrait image, it can provide more efficient animation production while preserving artistic processing and painting quality. The final video output has better precision quality and includes micro-facial expressions of the characters in the video.

[0092] above Figure 2 and Figure 3 The facial animation generation device in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The facial animation generation device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0093] Figure 4This is a schematic diagram of the structure of a facial animation generation device 400 provided in an embodiment of the present invention. The facial animation generation device 400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 433 or data 432. The memory 420 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the facial animation generation device 400. Furthermore, the processor 410 may be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the facial animation generation device 400 to implement the steps of the facial animation generation method described above.

[0094] The facial animation generation device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The illustrated facial animation generation device structure does not constitute a limitation on the facial animation generation device provided by the present invention. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0095] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the facial animation generation method.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating facial animation, characterized in that, The facial animation generation method includes: Acquire video stream data and a portrait image, and extract driving sampling points from the video stream data; Calculate the driving range of the driving sampling point based on its position in the video stream data; Perform heatmap recognition on each frame of video image in the video stream data to generate corresponding heatmap facial information; The facial region of the portrait image is driven by the driving sampling points based on the heatmap facial information and the driving range to generate facial animation; The step of driving the facial region of the portrait image based on the heatmap facial information and the driving range through the driving sampling points to generate facial animation includes: Affine transformation function for each driving sampling point is generated based on the heatmap facial information and the driving range; Determine the facial region in the portrait image that corresponds to the driving sampling point; The facial region corresponding to the driving sampling point is driven according to the affine transformation function to generate facial animation; The step of calculating the driving range of the driving sampling point based on its position in the video stream data includes: Calculate the comprehensive driving sampling points of the multi-frame affine transformed video images; The initial driving range is calculated by using Wally noise as the initial influence factor between the driving sampling points in the first frame of the video image and the corresponding comprehensive driving sampling points. Starting from the first video frame, the positional offset between the driving sampling points of the previous video frame and the driving sampling points of the next video frame, as well as the color offset between surrounding pixels, are backpropagated to determine the actual influence factor between the two video frames. Calculate the actual driving range between two video frames based on the actual impact factor. The driving range of the driving sampling point is obtained by summing the actual driving range between two video frames in the video stream data and the initial driving range.

2. The facial animation generation method according to claim 1, characterized in that, The process of acquiring video stream data and portrait images, and extracting driving sampling points from the video stream data includes: Acquire video stream data and a portrait image, and perform frame-by-frame processing on the video stream data to obtain multiple frames of video images; Facial key point detection is performed on multiple frames of the video image to obtain the facial key points corresponding to each frame of the video image and the position information of the facial key points; Based on the location information, the facial region type of the corresponding facial key points is determined, and the facial key points are filtered based on the facial region type. The selected facial key points are used to generate discrete points within a preset range based on the corresponding facial region type, and the facial key points and the corresponding discrete points are used as driving sampling points for the corresponding video image.

3. The facial animation generation method according to claim 2, characterized in that, After acquiring video stream data and portrait images, and extracting driving sampling points from the video stream data, the method further includes: Set the filtered facial key points corresponding to the first frame of the multi-frame video image as the facial alignment standard points; Based on the position information of the face alignment standard points and the position information of the face key points in the video images other than the first frame video image, calculate the affine transformation matrix between the first frame video image and the video images other than the first frame video image; Based on the affine transformation matrix, perform affine transformation on the pixels of each corresponding video frame to obtain multi-frame affine transformed video images.

4. The facial animation generation method according to any one of claims 1-3, characterized in that, After generating facial animation by driving the facial region of the portrait image based on the heatmap facial information using the driving sampling points, the method further includes: The facial animation is input into a pre-trained SRCNN convolutional neural network for information reconstruction, resulting in a new high-resolution facial animation.

5. The facial animation generation method according to claim 4, characterized in that, The SRCNN convolutional neural network includes a first convolutional network, a second convolutional network, and a third convolutional network; The step of inputting the facial animation into a pre-trained SRCNN convolutional neural network for information reconstruction to obtain a new high-resolution facial animation includes: The facial animation is input into a pre-trained SRCNN convolutional neural network, and image patches are extracted from each frame of the facial animation through the first layer of the SRCNN convolutional neural network. The second convolutional network is used to extract features from the extracted image patches and amplify the extracted features. The third convolutional network is used to perform nonlinear mapping on the features to reconstruct information from the corresponding image blocks of the animation image, thereby obtaining a new high-resolution animation image. A new high-resolution facial animation is created based on the new high-resolution animated images.

6. The facial animation generation method according to claim 5, characterized in that, Before inputting the facial animation into a pre-trained SRCNN convolutional neural network for information reconstruction to obtain a new high-resolution facial animation, the following steps are also included: Obtain the initialization of the SRCNN convolutional neural network and the training set that corresponds one-to-one with the high-resolution and low-resolution images; The low-resolution images in the training set are input into the initialized SRCNN convolutional neural network, and the processed images are output through the three-layer convolutional network of the initialized SRCNN convolutional neural network. Calculate the loss function between the high-resolution image and the corresponding processed image; Determine whether the loss function is greater than a preset loss threshold; If so, backpropagation is performed on the initialized SRCNN convolutional neural network according to the loss function, the network parameters of the initialized SRCNN convolutional neural network are adjusted, and the low-resolution images in the training set are re-input into the initialized SRCNN convolutional neural network; If not, then the network training ends, and the trained SRCNN convolutional neural network is obtained.

7. The facial animation generation method according to claim 6, characterized in that, After performing backpropagation on the initialized SRCNN convolutional neural network according to the loss function and adjusting the network parameters of the initialized SRCNN convolutional neural network, the method further includes: Numerical precision constraints are added to the backpropagation function during the backpropagation process, and corresponding edge calculations are added.

8. A facial animation generation device, characterized in that, The facial animation generation device includes: The acquisition module is used to acquire video stream data and a portrait image, and to extract driving sampling points from the video stream data; The range calculation module is used to calculate the driving range of the driving sampling point based on the position of the driving sampling point in the video stream data; The heatmap generation module is used to perform heatmap recognition on each frame of video image in the video stream data and generate corresponding heatmap facial information. The driving module is used to drive the facial region of the portrait image based on the heatmap facial information and the driving range through the driving sampling points to generate facial animation; The driving module is specifically used for: Affine transformation function for each driving sampling point is generated based on the heatmap facial information and the driving range; Determine the facial region in the portrait image that corresponds to the driving sampling point; The facial region corresponding to the driving sampling point is driven according to the affine transformation function to generate facial animation; The range calculation module is specifically used for: Calculate the comprehensive driving sampling points of the multi-frame affine transformed video images; The initial driving range is calculated by using Wally noise as the initial influence factor between the driving sampling points in the first frame of the video image and the corresponding comprehensive driving sampling points. Starting from the first video frame, the positional offset between the driving sampling points of the previous video frame and the driving sampling points of the next video frame, as well as the color offset between surrounding pixels, are backpropagated to determine the actual influence factor between the two video frames. Calculate the actual driving range between two video frames based on the actual impact factor. The driving range of the driving sampling point is obtained by summing the actual driving range between two video frames in the video stream data and the initial driving range.

9. A facial animation generation device, characterized in that, The facial animation generation device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the facial animation generation device to perform the steps of the facial animation generation method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the facial animation generation method as described in any one of claims 1-7.

Images