Open domain portrait animation method, system and device based on driving video and storage medium

Abstract

The invention relates to an open domain portrait animation method, system and device based on a driving video and a storage medium. The method comprises the following steps: acquiring key points of a reference image by using a preset appearance-guided key point matching module, the reference image being acquired in advance; based on the key points, a preset key point redirection module based on coordinates is utilized, a migration action is performed from a driving video to obtain a key point sequence of the reference image, and the driving video is acquired in advance; and inputting the key point sequence into a preset key point driven portrait animation model in an additional condition mode, and generating an open domain portrait animation. Compared with the prior art, the method has the advantages that the portrait in the open domain is effectively and accurately detected and redirected, and further data collection and training are not needed to improve generalization.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to an open-domain portrait animation method, system, device, and storage medium based on a driving video. Background Art

[0002] Portrait animation is a video processing method aimed at generating a dynamic and realistic video from a reference image by imitating the facial expressions in a driving video. This method is widely used in various application fields, such as film and television production, virtual characters and digital humans, short videos and filter applications, etc.

[0003] Regarding the current portrait animation methods, the video generation model based on the diffusion model has strong image generation capabilities and plays an important role in the development of portrait animation methods. Among the existing video generation models based on the diffusion model, the video generation model based on AniPortrait and the video generation model based on MOFA-Video mainly use FAN (Face Alignment Network) and 3DMM (3D Morphable Model) as the key point detector and action redirection module respectively to extract the key point driving sequences of the reference image, and inject these sequences into the video generation model for training to generate portrait animations. However, on the one hand, the existing methods rely on the key point detection model to recognize the facial key points of human characters and are mainly trained on human faces, resulting in difficulty in recognizing the facial features of non-human characters and poor performance in non-human character portrait animations; on the other hand, the existing methods cannot redirect the subtle movements of the driving video, and due to the significantly different facial features of non-human characters, it is also difficult for the existing methods to transfer the actions of the driving video to the reference image.

[0004] Therefore, the existing portrait animation methods have achieved certain results in human portrait animation performance, but they are still limited by the problems that key point detection and action redirection cannot be generalized to non-human characters, thereby reducing the accuracy of key point recognition and animation results in multi-type character scenarios. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned defects of the existing technology, which perform poorly in non-human character portrait animations and cannot provide accurate key point detection and corresponding action sequences for non-human characters, resulting in inaccurate portrait animation results. The present invention provides an open-domain portrait animation method, system, device, and storage medium based on a driving video, which can be applied to an open-domain portrait animation model driven by key points and can generate high-quality portrait animation videos according to any reference image without training.

[0006] The object of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided an open-domain portrait animation method based on a driving video, comprising the following steps: obtaining key points of a reference image by using a preset appearance-guided key point matching module, where the reference image is pre-obtained; based on the key points, using a preset coordinate-based key point redirection module to migrate actions from a driving video to obtain a key point sequence of the reference image, where the driving video is pre-obtained; inputting the key point sequence into a preset key point-driven portrait animation model in an additional condition manner to generate an open-domain portrait animation.

[0008] As a preferred technical solution, the running process of the appearance-guided key point matching module specifically includes: constructing k target images and performing key point marking; respectively inputting the marked target images and the reference image into a pre-trained diffusion model, extracting diffusion features of the t-th time step and the l-th layer through DDIM inversion, and extracting image appearance features as appearance guidance; upsampling the diffusion features to the original dimension of the image, and constructing a feature pair of key points at specific positions on the target image to match the features on the reference image to obtain the corresponding key point positions on the reference image.

[0009] As a preferred technical solution, the construction process of the target image specifically includes: constructing an appearance image library, where the appearance image library includes pictures of multiple different facial parts and different domains; matching the reference image with the nearest domain in the appearance image library, and then constructing the target image.

[0010] As a preferred technical solution, the running process of the coordinate-based key point redirection module includes a global redirection step and a local redirection step. The global redirection step is used to migrate global actions from the driving video to the corresponding reference image, and the local redirection step is used to migrate relevant actions and key point actions from the driving video to the corresponding reference image.

[0011] As a preferred technical solution, the global redirection step specifically includes: obtaining the key point sequence of the driving video; calculating the global rectangular coordinate system corresponding to each image in the driving video; based on the global rectangular coordinate system, calculating the global action, where the global action is the difference between the global rectangular coordinate systems of the m-th frame and the 0-th frame of the driving video; calculating the global rectangular coordinate system of the 0-th frame reference image; based on the global rectangular coordinate system of the 0-th frame reference image and the global action, calculating the global rectangular coordinate system of the m-th frame reference image; migrating the key points of the 0-th frame reference image to the global rectangular coordinate system of the corresponding m-th frame reference image to achieve global redirection.

[0012] As a preferred technical solution, the local redirection step specifically includes: dividing each image in the driving video and the reference image into corresponding multiple parts respectively; calculating the local rectangular coordinate systems of each part of each image in the driving video respectively; based on the local rectangular coordinate systems, calculating relevant actions, where the relevant actions include multiplying the difference between the origins of the local rectangular coordinate systems from the m-th frame to the 0-th frame of the driving video by a preset scaling value; calculating the local rectangular coordinate systems of each part of the reference image in the 0-th frame; based on the local rectangular coordinate systems of each part of the reference image in the 0-th frame and the relevant actions, calculating the local rectangular coordinate systems of each part of the reference image in the m-th frame; migrating the key points corresponding to each part of the reference image in the 0-th frame to the local rectangular coordinate systems of each part of the reference image in the m-th frame; based on a pre-constructed key point action model, migrating the key points corresponding to each part of the reference image in the 0-th frame to the local rectangular coordinate systems of each part of the reference image in the m-th frame again to complete local redirection.

[0013] As a preferred technical solution, the key point action model is expressed as:

[0014]

[0015] In the formula, represents the coordinates of the i-th key point in the m-th frame of the reference image in the local rectangular coordinate system, represents the coordinates of the i-th key point in the 0-th frame of the reference image in the local rectangular coordinate system, respectively represent the coordinates of the i-th key point in the 0-th frame and the m-th frame of the driving video in the local rectangular coordinate system.

[0016] According to the second aspect of the present invention, an open-domain portrait animation system based on a driving video is provided, including an appearance-guided key point matching module, a coordinate-based key point redirection module, and a portrait animation generation module; the appearance-guided key point matching module is used to obtain the key points of a reference image, where the reference image is pre-obtained; the coordinate-based key point matching module is used to migrate actions from the driving video based on the key points to obtain a key point sequence of the reference image, where the driving video is pre-obtained; the portrait animation generation module is used to input the key point sequence into a preset key point-driven portrait animation model in the form of additional conditions to generate an open-domain portrait animation.

[0017] According to the third aspect of the present invention, an open-domain portrait animation device based on a driving video is provided, including a memory, a processor, and a program stored in the memory, and the processor implements the method when executing the program.

[0018] According to the fourth aspect of the present invention, a storage medium is provided, on which a program is stored, and the method is implemented when the program is executed.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. In the portrait animation method provided by the present invention, the appearance-guided key point matching module is used to detect the key points of a portrait. This module can detect any portrait, including those of human or non-human characters, without being restricted by different facial features, and without the need for further data collection and training to improve generalization. At the same time, the coordinate-based key point redirection module is used to redirect the actions of the driving video to the reference image. This module can also redirect any action sequence of the driving video to any portrait without being restricted by different facial features and without the need for further data collection and training to improve generalization. Therefore, this method can effectively improve the accuracy of key point recognition and animation results in multi-type character scenarios;

[0021] 2. The appearance-guided key point matching module provided by the present invention adopts the strong semantic correlation between diffusion features, can accurately extract the key points of portraits in different domains, and improves the accuracy of key point detection;

[0022] 3. The coordinate-based key point redirection module provided by the present invention can redirect the key points of portraits in different domains without training by constructing global and local rectangular coordinate systems, realize the redirection of the minute actions of the driving video, and further improve the accuracy of key point detection and animation results;

[0023] 4. The present invention proposes a key point-driven open-domain portrait animation framework, which integrates the appearance-guided key point matching module and the coordinate-based key point redirection module. It can adapt to any driving video and reference image, realize the generation of open-domain portrait animation, and can directly utilize any pre-trained key point-driven portrait animation model without additional fine-tuning. This framework can effectively simplify the portrait animation process, making it more efficient and easy to apply. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic flow chart of the method provided by the present invention;

[0025] Figure 2 It is a schematic overall framework diagram of the method provided in Embodiment 1 of the present invention;

[0026] Figure 3 It is a schematic diagram of the construction of an appearance picture library provided in Embodiment 1 of the present invention;

[0027] Figure 4 It is an implementation process example of the coordinate-based key point redirection step provided in Embodiment 1 of the present invention;

[0028] Figure 5A schematic diagram of endpoints of different parts of a human face provided in Embodiment 1 of the present invention. Detailed implementation manners

[0029] Existing portrait animation methods are difficult to recognize the facial features of non - human characters, and it is difficult to migrate due to the inability to redirect the minute movements of the driving video. Moreover, in the generation pipeline of existing methods, collecting a large amount of non - human character information to improve the generalization of corresponding methods requires a huge amount of manpower and material resources, which is unrealistic. As a result, the accuracy of key - point recognition and animation results of existing methods decreases in multi - type character scenarios. Therefore, as Figure 1 shown, the present invention proposes an open - domain portrait animation method based on a driving video. This method first uses a preset appearance - guided key - point matching module to obtain the key points of a pre - acquired reference image. Then, based on the key points, it uses a preset coordinate - based key - point redirection module to migrate the actions from a pre - acquired driving video to obtain a key - point sequence of the reference image. Finally, the key - point sequence is input into a preset key - point - driven portrait animation model in an additional - condition manner to generate an open - domain portrait animation. The portrait animation method provided by the present invention aims to effectively detect the key points of portraits in the open domain without further data collection and training to improve generalization. At the same time, it can effectively perform action redirection for portraits in the open domain without further data collection and training to improve generalization. Based on this, the present invention can provide an open - domain portrait animation model without training, enabling users to directly generate high - quality portrait animations from any reference image and driving video, improving the practicality and efficiency of the model.

[0030] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and gives detailed implementation manners and specific operation processes. However, the protection scope of the present invention is not limited to the following embodiments.

[0031] Embodiment 1:

[0032] This embodiment provides an open - domain portrait animation method based on a driving video. Figure 2 One of the implementation processes is shown. This method mainly consists of three parts, namely, an appearance - guided key - point matching step, a coordinate - based key - point redirection step, and a key - point - driven open - domain portrait animation framework.

[0033] 1. Appearance - guided key - point matching step

[0034] This step uses a self-designed appearance-guided key-point matching module to detect the key points of any reference image. The appearance-guided key-point matching module performs feature matching through the strong semantic correlation of the intermediate layer features of the diffusion model. This module first constructs k target images and performs key-point marking, and then inputs the target image and the reference image into the pre-trained diffusion model respectively. Through the DDIM (Denoising Diffusion Implicit Models) inversion process, the diffusion features of the l-th layer at the t-th time step are extracted. As can be seen from Figure 2 both the target image and the reference image are input into the image feature extractor, and both go through the DDIM inversion. The image feature extractor outputs the image appearance features for appearance guidance, and together with the DDIM inversion results, the diffusion features of the l-th layer at the t-th time step are obtained.

[0035] The diffusion features are expressed as:

[0036]

[0037] In the formula, f tar represents the diffusion features of the target image, f ref represents the diffusion features of the reference image, F l represents the operation of extracting features from the l-th layer, ∈ θ represents the diffusion model, c represents the input of the text prompt, specifically: "a photo of a face", and c' is the image appearance feature, serving as appearance guidance. and are respectively calculated iteratively from the DDIM inversion formula in and :

[0038]

[0039] where z is obtained by extracting features of the image from the VAE (Auto-Encoding Variational Bayes) model.

[0040] After obtaining the diffusion features, they are upsampled to the original dimension of the image, and the feature pairs of specific positions on the target image are constructed to match the features on the reference image, obtaining the corresponding key-point positions on the reference image:

[0041]

[0042] In the formula, f ′ represents the upsampled diffusion features, p is the coordinate of the key point, d cosRepresents the cosine distance. The subscript ref corresponds to the reference image, the subscript tar corresponds to the target image, i represents the serial number of the key point, k represents the number of target images, j is the corresponding serial number of the target image among them, and p ref Represents the point in the reference feature map.

[0043] Finally, the key points of the reference image after matching are expressed as:

[0044]

[0045] In the formula, N represents the number of key points, and the superscript 0 represents the 0th frame.

[0046] Optionally, to improve the accuracy of matching, an appearance library can also be constructed. The appearance library contains pictures of five different domains of different facial parts. By matching the reference image with the nearest domain in the appearance library to construct the target image, the appearance difference between the two is reduced, and a more accurate match is obtained. Figure 3 Shows one of the constructions of the appearance library.

[0047] 2. Coordinate-based key point redirection steps

[0048] To solve the problem that the existing action redirection methods cannot capture tiny actions and cannot adapt to non-human characters, this step proposes a new coordinate-based key point redirection module, which realizes key point redirection through two steps: global and local. Among them, the global redirection step is used to migrate the global action in the driving video to the corresponding reference image. The global action refers to the coordinate change of the entire portrait. The local redirection step is used to migrate the relevant actions and key point actions in the driving video to the corresponding reference image. The relevant actions refer to the coordinate changes of each part divided in the portrait, and the key point actions are the specific coordinate changes of the points corresponding to each part.

[0049] Figure 4 Is an example of the implementation process of the coordinate-based key point redirection step.

[0050] (1) Global redirection step

[0051] After obtaining the key points of the reference image and the key point sequence of the driving video, the global redirection step specifically includes:

[0052] (1.1) Calculate the corresponding global rectangular coordinate system (O dri , θ dri ) for each portrait in the driving video. O represents the origin of the coordinate system, and θ represents the rotation angle. The calculation formula is as follows:

[0053]

[0054] In the formula, and represent two endpoints of the facial contour, and the subscript dri indicates that the parameter corresponds to the driving video.

[0055] (1.2) Based on the global rectangular coordinate system (O dri , θ dri ), calculate the global action.

[0056] In this embodiment, the global action is defined as the difference in the global rectangular coordinate system from the m-th frame to the 0-th frame of the driving video:

[0057]

[0058] (1.3) Calculate the global rectangular coordinate system of the reference image in the 0-th frame

[0059] (1.4) Based on the global rectangular coordinate system of the reference image in the 0-th frame and the global action ΔO dri , Δθ dri , calculate the global rectangular coordinate system of the reference image in the m-th frame. The calculation formula is as follows:

[0060]

[0061] (1.5) Finally, transfer the key points of the reference image in the 0-th frame to the global rectangular coordinate system of the corresponding reference image in the m-th frame, that is, transfer the key point coordinates from to to achieve global redirection.

[0062] (2) Local redirection steps

[0063] Local redirection is divided into related actions and key point actions.

[0064] For related actions, first divide the portrait into five parts: eyes, mouth, nose, eyebrows, and facial contour, and then calculate the local rectangular coordinate systems of these five parts respectively. The corresponding endpoints are as Figure 5 shown.

[0065] Then, perform local redirection according to the global redirection steps, but multiply a scaling value in front of the origin difference of the local rectangular coordinate system to limit the redirection of different parts to a reasonable position:

[0066]

[0067] In the formula, b represents the distance from the current face part to the face boundary.

[0068] For key point actions, in this embodiment, the key point actions are modeled as:

[0069]

[0070] In the formula, represents the coordinates of the i-th key point in the m-th frame of the reference image in the local rectangular coordinate system, represents the coordinates of the i-th key point in the 0-th frame of the reference image in the local rectangular coordinate system, respectively represent the coordinates of the i-th key point in the 0-th frame and the m-th frame of the driving video in the local rectangular coordinate system.

[0071] Based on this, the specific steps of local redirection include:

[0072] (2.1) For each image in the driving video and the reference image, they are respectively divided into corresponding five parts.

[0073] (2.2) Calculate the local rectangular coordinate systems of each part of each image in the driving video respectively.

[0074] (2.3) Based on the local rectangular coordinate systems obtained in step (2.2), calculate the relevant actions. In the relevant actions, the change Δθ of the rotation angle of the local rectangular coordinate system dri The calculation method remains unchanged, and the change of the origin ΔO dri needs to be corrected using the preset scaling value b ref / b dri for correction.

[0075] (2.4) Calculate the local rectangular coordinate systems of each part of the reference image in the 0-th frame.

[0076] (2.5) Based on the local rectangular coordinate systems of each part of the reference image in the 0-th frame and the relevant actions, calculate the local rectangular coordinate systems of each part of the reference image in the m-th frame. The calculation process refers to step (1.4).

[0077] (2.6) Correspondingly migrate the key points of each part of the reference image in the 0-th frame to the local rectangular coordinate system of each part of the reference image in the m-th frame;

[0078] (2.7) Based on the aforementioned key point action model, migrate the key points of each part of the reference image in the 0-th frame to the local rectangular coordinate system of each part of the reference image in the m-th frame again to complete the local redirection. Specifically: first calculate the coordinates of the key points of each part in the reference image in the 0-th frame in the local rectangular coordinate system according to the appearance-guided key point matching step then calculate the coordinates of the key points of each part in the 0-th frame and the m-th frame in the driving video in the local rectangular coordinate system and finally calculate the corresponding

[0079] According to the aforementioned global redirection step and local redirection step, the global action, related action, and key-point action are respectively migrated from the driving video to the reference image, obtaining the global Cartesian coordinate system, local Cartesian coordinate system, and key-point coordinates of the reference image after migration. Furthermore, based on the global Cartesian coordinate system, local Cartesian coordinate system, and key-point coordinates of the reference image, the key-point sequence of the final reference image is obtained. M represents the number of video frames.

[0080] The combined global and local redirection step provided in this embodiment can capture subtle actions and can be applied to any driving video and portrait animation without additional training.

[0081] 3. Key-Point Driven Open-Domain Portrait Animation Framework

[0082] In this embodiment, a key-point driven open-domain portrait animation framework is constructed, which supports integrating the appearance-guided key-point matching module and the coordinate-based key-point redirection module in the foregoing steps into any pre-trained key-point driven portrait animation model.

[0083] The process of the entire framework is as follows: First, the key-points of the reference image are obtained through the appearance-guided key-point matching module, then the key-point sequence of the reference image is obtained by migrating actions from the driving video through the coordinate-based key-point redirection module, and finally, the key-point sequence is input into any key-point driven portrait animation model in the form of an additional condition. This framework can adapt to any driving video and reference image to achieve open-domain portrait animation generation.

[0084] The method provided in this embodiment can be implemented based on any key-point driven animation model. When validating the effectiveness of this method, for appearance-guided key-point matching, the diffusion features of the 6th layer in step 301 can be extracted using the pre-trained parameters of StableDiffusion v1.5 and IP-Adapter, and 10 target images are set. In addition, 68 key-points are used to represent the human face, and the number of frames of each driving video is set to 64. Through extensive qualitative and quantitative experiments and evaluations on the portrait animation evaluation benchmark, it can be proved that this method is effective and superior in the portrait animation task, especially performing well in non-human character animations and being superior to existing portrait animation methods.

[0085] Optionally, different pre-trained video generation models are used as the base model, and the appearance-guided key-point matching module and the coordinate-based key-point redirection module proposed in the present invention are integrated to implement the method proposed in the present invention. For example, MOFA-Video is used as the base model. In addition, the method provided in the present invention can also be applied to fields such as key-point detection and action capture migration.

[0086] In summary, the method provided by the present invention can effectively detect the key points of any portrait, accurately capture the subtle movements of the driving video and transfer them to any portrait, and does not require any additional training. The proposed key-point-driven open-domain portrait animation framework can adapt to any driving video and reference image to achieve open-domain portrait animation generation.

[0087] Embodiment 2:

[0088] This embodiment provides an open-domain portrait animation system based on a driving video, including an appearance-guided key-point matching module, a coordinate-based key-point redirection module, and a portrait animation generation module. Among them, the appearance-guided key-point matching module is used to obtain the key points of the reference image, where the reference image is pre-acquired; the coordinate-based key-point matching module is used to migrate the actions from the driving video based on the key points to obtain the key-point sequence of the reference image, where the driving video is pre-acquired; the portrait animation generation module is used to input the key-point sequence into a preset key-point-driven portrait animation model in an additional-condition manner to generate an open-domain portrait animation. The specific implementation steps of each module are basically the same as the implementation processes of the steps in Embodiment 1 and will not be elaborated here.

[0089] Furthermore, this embodiment also provides an open-domain portrait animation device based on a driving video, including a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements one or more steps of the method in Embodiment 1 and will not be elaborated here. Among them, the processor includes a central processing unit (CPU), which can execute various appropriate actions and processes according to the computer program instructions stored in the read-only memory (ROM) or the computer program instructions loaded from the storage unit to the random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus. Multiple components in the device are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a disk, an optical disc, etc.; and a communication unit, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The processing unit executes the various methods and processes described above, such as one or more steps in the foregoing embodiments.

[0090] Furthermore, this embodiment also provides a storage medium, on which a program is stored. When the program is executed, one or more steps of the method in Embodiment 1 are implemented. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program codes can be executed entirely on the machine, partially on the machine, executed partially on the machine as an independent software package and partially on a remote machine, or executed entirely on a remote machine or server. In the context of the present invention, a computer-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should fall within the protection scope determined by the claims.

Claims

1. An open domain portrait animation method based on driving video, characterized in that The following steps are involved: Using a preset appearance-guided key point matching module, obtaining key points of a reference image, wherein the reference image is obtained in advance; Based on the key points, using a preset coordinate-based key point redirection module, migrating actions from a driving video to obtain a key point sequence of the reference image, wherein the driving video is obtained in advance; The key point sequence is input into a preset key point driven portrait animation model in the form of additional conditions to generate an open domain portrait animation.

2. The method for open domain portrait animation based on driving video according to claim 1, characterized in that: The operation process of the appearance-guided key point matching module specifically includes: Construct k target images and mark key points; The marked target image and the reference image are respectively input into the pre-trained diffusion model, the diffusion features of the lth layer at the tth time step are extracted through DDIM inversion, and the image appearance features are extracted as appearance guidance; The diffusion features are upsampled to the original dimensions of the image, and the features of the key points at specific positions on the target image are constructed to match the features on the reference image to obtain the corresponding key point positions on the reference image.

3. The method for open domain portrait animation based on driving video according to claim 2, characterized in that: The target image construction process specifically includes: constructing an appearance image library, wherein the appearance image library includes pictures of different domains of different facial parts; The reference image is matched to the nearest domain in the appearance gallery, thereby constructing the target image.

4. The method for open domain portrait animation based on driving video according to claim 1, characterized in that: The operation process of the coordinate-based key point redirection module includes a global redirection step and a local redirection step. The global redirection step is used to migrate global actions from the driving video to the corresponding reference image, and the local redirection step is used to migrate related actions and key point actions from the driving video to the corresponding reference image.

5. The method for open domain portrait animation based on driving video according to claim 4, characterized in that: The global redirection step specifically includes: Obtaining a key point sequence of the driving video; Calculate the global rectangular coordinate system corresponding to each image in the driving video; Based on the global rectangular coordinate system, a global action is calculated, where the global action is the difference between the global rectangular coordinate systems of the mth frame and the 0th frame of the driving video; Calculate the global rectangular coordinate system of the reference image of frame 0; Calculating a global rectangular coordinate system of the m-th frame reference image based on the global rectangular coordinate system of the 0-th frame reference image and the global action; The key points of the reference image of the 0th frame are migrated to the global rectangular coordinate system of the corresponding reference image of the mth frame to achieve global redirection.

6. The method for open domain portrait animation based on driving video according to claim 5, characterized in that: The local redirection step specifically includes: Divide each image in the driving video and the reference image into corresponding multiple parts respectively; Calculate the local rectangular coordinate system of each part of each image in the driving video respectively; Based on the local rectangular coordinate system, a related action is calculated, wherein the related action includes driving the multiplication of the difference between the origin of the local rectangular coordinate system of the mth frame and the 0th frame of the video by a preset scaling value; Calculate the local rectangular coordinate system of each part of the reference image of frame 0; Calculate the local rectangular coordinate system of each part of the m-th frame reference image based on the local rectangular coordinate system of each part of the 0-th frame reference image and the related action; Migrate the key points of each part of the reference image of the 0th frame to the local rectangular coordinate system of each part of the reference image of the mth frame; Based on the pre-built key point action model, the key points of each part of the reference image of the 0th frame are migrated to the local rectangular coordinate system of each part of the reference image of the mth frame to complete the local redirection.

7. The method for open domain portrait animation based on driving video according to claim 6, characterized in that: The key point action model is expressed as: In the formula, Represents the coordinates of the i-th key point in the m-th frame of the reference image in the local rectangular coordinate system, Represents the coordinates of the i-th key point in the 0th frame of the reference image in the local rectangular coordinate system, They respectively represent the coordinates of the i-th key point in the 0th frame and the mth frame in the driving video in the local rectangular coordinate system.

8. An open domain portrait animation system based on driven video, characterized in that It includes appearance-guided key point matching module, coordinate-based key point redirection module and portrait animation generation module; The appearance-guided key point matching module is used to obtain key points of a reference image, where the reference image is obtained in advance; The coordinate-based key point matching module is used to obtain a key point sequence of the reference image by migrating actions from a driving video based on the key points, wherein the driving video is obtained in advance; The portrait animation generation module is used to input the key point sequence into a preset key point driven portrait animation model in the form of additional conditions to generate an open domain portrait animation.

9. An open domain portrait animation device based on driving video, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 7 is implemented.

Images