Open-domain portrait animation method, system, device, and storage medium based on video-driven animation

Abstract

This invention relates to a method, system, apparatus, and storage medium for open-domain portrait animation based on driven video. The method utilizes a pre-defined appearance-guided keypoint matching module to acquire keypoints of a pre-acquired reference image. Based on these keypoints, a pre-defined coordinate-based keypoint retargeting module transfers motion from the driven video to obtain a keypoint sequence of the reference image. The driven video is also pre-acquired. The keypoint sequence is then input into a pre-defined keypoint-driven portrait animation model with additional conditions to generate an open-domain portrait animation. Compared to existing technologies, this invention offers advantages such as effectively and accurately detecting and retargeting portraits in the open domain without requiring further data collection and training to improve generalization.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an open-domain portrait animation method, system, apparatus, and storage medium based on driven video. Background Technology

[0002] Portrait animation is a video processing method that aims to generate dynamic and realistic videos from reference images by mimicking facial expressions that drive the video. This method is widely used in various application areas, such as film and television production, virtual characters and digital humans, short videos, and filter applications.

[0003] In current portrait animation methods, diffusion-based video generation models possess strong image generation capabilities and play a crucial role in the development of portrait animation techniques. Existing diffusion-based video generation models, such as AniPortrait and MOFA-Video, primarily use FAN (FaceAlignment Network) and 3DMM (3D Morphable Model) as keypoint detectors and motion retargeting modules, respectively, to extract keypoint-driven sequences from reference images. These sequences are then injected into the video generation model for training to generate portrait animations. However, on the one hand, existing methods rely on the keypoint detection model to identify keypoints on human faces, primarily training on human faces, making it difficult to recognize facial features of non-human characters and resulting in poor performance in portrait animation of non-human characters. On the other hand, existing methods cannot retarget subtle movements driving the video, and due to the significantly different facial features of non-human characters, they also struggle to transfer the driving video's movements to the reference image.

[0004] Therefore, while existing portrait animation methods have achieved certain results in human portrait animation performance, they are still limited by the inability of key point detection and motion redirection to generalize to non-human characters, which reduces the accuracy of key point recognition and animation results in multi-type character scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology, such as poor performance in non-human character portrait animation and inability to provide accurate key point detection and corresponding action sequences for non-human characters, resulting in inaccurate portrait animation results. The invention provides an open-domain portrait animation method, system, device, and storage medium based on driven video, which can be applied to key point-driven open-domain portrait animation models and can generate high-quality portrait animation videos from any reference image without training.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, an open-domain portrait animation method based on a driven video is provided, comprising the following steps: obtaining key points of a reference image using a preset appearance-guided key point matching module, wherein the reference image is pre-acquired; obtaining a key point sequence of the reference image by migrating motion from a driven video based on the key points using a preset coordinate-based key point relocation module, wherein the driven video is pre-acquired; and inputting the key point sequence into a preset key point-driven portrait animation model with additional conditions to generate an open-domain portrait animation.

[0008] As a preferred technical solution, the operation process of the appearance-guided key point matching module specifically includes: constructing k target images and marking key points; inputting the marked target images and the reference images into a pre-trained diffusion model, extracting the diffusion features of the l-th layer at time step t through DDIM inversion, and extracting image appearance features as appearance guidance; upsampling the diffusion features to the original dimension of the image, and constructing features of key points at specific locations on the target image to match features on the reference image to obtain the corresponding key point positions on the reference image.

[0009] As a preferred technical solution, the process of constructing the target image specifically includes: constructing an appearance image library, which includes multiple images of different facial parts and different domains; matching the nearest domain in the appearance image library to a reference image, thereby constructing the target image.

[0010] As a preferred technical solution, the operation process of the coordinate-based keypoint relocation module includes a global relocation step and a local relocation step. The global relocation step is used to migrate global actions from the driving video to the corresponding reference image, and the local relocation step is used to migrate related actions and keypoint actions from the driving video to the corresponding reference image.

[0011] As a preferred technical solution, the global redirection step specifically includes: obtaining the key point sequence of the driving video; calculating the global Cartesian coordinate system corresponding to each image in the driving video; calculating the global motion based on the global Cartesian coordinate system, wherein the global motion is the difference between the global Cartesian coordinate systems of the m-th frame and the 0-th frame of the driving video; calculating the global Cartesian coordinate system of the 0-th frame reference image; calculating the global Cartesian coordinate system of the m-th frame reference image based on the global Cartesian coordinate system of the 0-th frame reference image and the global motion; and migrating the key points of the 0-th frame reference image to the corresponding global Cartesian coordinate system of the m-th frame reference image to achieve global redirection.

[0012] As a preferred technical solution, the local retargeting step specifically includes: dividing each image in the driving video and the reference image into corresponding multiple parts; calculating the local Cartesian coordinate system of each part of each image in the driving video; calculating relevant actions based on the local Cartesian coordinate system, the relevant actions including multiplying the difference between the origin of the local Cartesian coordinate system from frame m to frame 0 of the driving video by a preset scaling value; calculating the local Cartesian coordinate system of each part of the reference image in frame 0; calculating the local Cartesian coordinate system of each part of the reference image in frame m based on the local Cartesian coordinate system of each part of the reference image in frame 0 and the relevant actions; transferring the key points of each part of the reference image in frame 0 to the local Cartesian coordinate system of each part of the reference image in frame m; and, based on the pre-constructed key point action model, transferring the key points of each part of the reference image in frame 0 to the local Cartesian coordinate system of each part of the reference image in frame m again, thus completing the local retargeting.

[0013] As a preferred technical solution, the key point action model is represented as follows:

[0014]

[0015] In the formula, This represents the coordinates of the i-th keypoint in the m-th frame of the reference image in the local Cartesian coordinate system. This represents the coordinates of the i-th keypoint in the local Cartesian coordinate system of the 0-th frame of the reference image. , These represent the coordinates of the i-th keypoint in the local Cartesian coordinate system in the 0th and m-th frames of the driving video, respectively.

[0016] According to a second aspect of the present invention, an open-domain portrait animation system based on driven video is provided, comprising an appearance-guided keypoint matching module, a coordinate-based keypoint retargeting module, and a portrait animation generation module; the appearance-guided keypoint matching module is used to acquire keypoints of a reference image, wherein the reference image is pre-acquired; the coordinate-based keypoint matching module is used to obtain a keypoint sequence of the reference image by migrating actions from the driven video based on the keypoints, wherein the driven video is pre-acquired; the portrait animation generation module is used to input the keypoint sequence into a preset keypoint-driven portrait animation model with additional conditions to generate an open-domain portrait animation.

[0017] According to a third aspect of the present invention, an open-domain portrait animation apparatus based on driven video is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described therein.

[0018] According to a fourth aspect of the present invention, a storage medium is provided having a program stored thereon, which, when executed, implements the method described thereon.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. In the portrait animation method provided by this invention, the appearance-guided key point matching module is used to detect key points of the portrait. This module does not require further data collection and training to improve generalization and can detect key points of any portrait, including human or non-human characters, without being limited by different facial features. At the same time, the coordinate-based key point redirection module is used to redirect the action of the driving video to the reference image. This module also does not require further data collection and training to improve generalization and can redirect any action sequence of the driving video to any portrait without being limited by different facial features. Therefore, this method can effectively improve the accuracy of key point recognition and animation results in multi-type character scenes.

[0021] 2. The appearance-guided key point matching module provided by this invention adopts the strong semantic correlation between diffusion features, which can accurately extract key points of portraits in different domains and improve the accuracy of key point detection.

[0022] 3. The coordinate-based keypoint relocation module provided by this invention can relocate portrait keypoints in different domains without training by constructing global and local Cartesian coordinate systems, thereby enabling the relocation of subtle movements in the video and further improving the accuracy of keypoint detection and animation results.

[0023] 4. This invention proposes a keypoint-driven open-domain portrait animation framework, which integrates an appearance-guided keypoint matching module and a coordinate-based keypoint relocation module. It can adapt to any driving video and reference image to achieve open-domain portrait animation generation. Moreover, it can directly utilize any pre-trained keypoint-driven portrait animation model without additional fine-tuning. This framework can effectively simplify the portrait animation process, making it more efficient and easier to apply. Attached Figure Description

[0024] Figure 1 A flowchart illustrating the method of this invention;

[0025] Figure 2 This is a schematic diagram of the overall framework of the method provided in Embodiment 1 of the present invention;

[0026] Figure 3 This is a schematic diagram of an appearance image library structure provided in Embodiment 1 of the present invention;

[0027] Figure 4This is an example of the implementation process of the coordinate-based keypoint redirection step provided in Embodiment 1 of the present invention;

[0028] Figure 5 This is a schematic diagram of the endpoints of different parts of a face provided in Embodiment 1 of the present invention. Detailed Implementation

[0029] Existing portrait animation methods struggle to recognize facial features of non-human characters and cannot redirect subtle movements driving the video, leading to poor transferability. Furthermore, collecting extensive non-human character information in the generation pipeline of existing methods to improve generalization requires significant human and material resources, which is impractical. Consequently, the accuracy of keypoint recognition and animation results in various character scenarios is reduced. Therefore, as... Figure 1 As shown, this invention proposes an open-domain portrait animation method based on driven video. The method first uses a pre-defined appearance-guided keypoint matching module to obtain keypoints from a pre-acquired reference image. Then, based on these keypoints, a pre-defined coordinate-based keypoint retargeting module transfers actions from the pre-acquired driven video to obtain a keypoint sequence for the reference image. Finally, the keypoint sequence is input into a pre-defined keypoint-driven portrait animation model with additional conditions to generate an open-domain portrait animation. The portrait animation method provided by this invention aims to effectively detect keypoints in open-domain portraits without requiring further data collection and training to improve generalization. Simultaneously, it effectively retargets actions in open-domain portraits without further data collection and training to improve generalization. Based on this, this invention provides a training-free open-domain portrait animation model, allowing users to directly generate high-quality portrait animations from any reference image and driven video, improving the model's practicality and efficiency.

[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0031] Example 1:

[0032] This embodiment provides an open-domain portrait animation method based on driven video. Figure 2 One implementation flow is illustrated. This method mainly consists of three parts: an appearance-guided keypoint matching step, a coordinate-based keypoint retargeting step, and a keypoint-driven open-domain portrait animation framework.

[0033] 1. Key point matching steps for appearance guidance

[0034] This step employs a self-designed appearance-guided keypoint matching module to detect keypoints in any reference image. The appearance-guided keypoint matching module performs feature matching based on the strong semantic correlation of features from the intermediate layers of the diffusion model. This module first constructs k target images and labels their keypoints. Then, it inputs both the target images and the reference image into a pre-trained diffusion model, extracting the diffusion features of the l-th layer at time step t through the DDIM (Denoising Diffusion Implicit Models) inversion process. Figure 2 As can be seen, both the target image and the reference image are input into the image feature extractor and are also subjected to DDIM inversion. The image feature extractor outputs the image appearance features for appearance guidance, which, together with the DDIM inversion results, yield the diffusion features of the l-th layer at time step t.

[0035] The diffusion characteristics are represented as follows:

[0036]

[0037]

[0038] In the formula, Indicates the diffusion characteristics of the target image. Indicates the diffusion characteristics of the reference image. This represents the operation of extracting features from the l-th layer. Let represent the diffusion model, c represents the text prompt input, specifically "a photo of a face", and c' represents the image appearance features, serving as appearance guidance. and From respectively and The formula is derived iteratively from the DDIM inversion formula:

[0039]

[0040] Here, z is the feature extracted from the image from the VAE (Auto-Encoding Variational Bayes) model.

[0041] After obtaining the diffusion features, they are upsampled to the original dimensions of the image, and the features of key points at specific locations in the target image are constructed and matched with the features in the reference image to obtain the corresponding key point locations in the reference image.

[0042]

[0043] In the formula, This represents the diffusion characteristics after upsampling. These are the coordinates of the key points. Represents the cosine distance, with the subscript 'ref' corresponding to the reference image and the subscript 'tar' corresponding to the target image. 'i' represents the keypoint index, 'k' represents the number of target images, and 'j' is the corresponding target image index. This represents a point in the reference feature map.

[0044] Finally, the key points of the matched reference image are represented as follows:

[0045]

[0046] In the formula, N represents the number of key points, and the superscript 0 indicates the 0th frame.

[0047] Optionally, to improve matching accuracy, an appearance image library can be constructed. This appearance image library contains images of five different facial features from different domains. By matching the reference image with the nearest domain within the appearance image library, the appearance difference between the two images can be reduced, resulting in a more accurate match. Figure 3 One of the appearance library constructions is shown.

[0048] 2. Keypoint Redirection Steps Based on Coordinates

[0049] To address the limitations of existing motion retargeting methods in capturing subtle movements and adapting to non-human characters, this paper proposes a novel coordinate-based keypoint retargeting module. This module performs keypoint retargeting through two steps: global and local. The global retargeting step migrates global motion from the driving video to the corresponding reference image. Global motion refers to the coordinate changes of the entire portrait. The local retargeting step migrates related motion and keypoint motion from the driving video to the corresponding reference image. Related motion refers to the coordinate changes of different parts of the portrait, while keypoint motion refers to the specific coordinate changes of points corresponding to each part.

[0050] Figure 4 This is an example of the implementation process for coordinate-based keypoint redirection.

[0051] (1) Global redirection steps

[0052] After obtaining the keypoints of the reference image and the keypoint sequence of the driving video, the global retargeting steps specifically include:

[0053] (1.1) Calculate the corresponding global Cartesian coordinate system for each portrait in the driving video. O represents the origin of the coordinate system, and θ represents the rotation angle. The calculation formula is as follows:

[0054]

[0055]

[0056] In the formula, and The two endpoints represent the facial contour, and the subscript dri indicates that the parameter corresponds to the driving video.

[0057] (1.2) Based on the global rectangular coordinate system Calculate global actions.

[0058] In this embodiment, the global action is defined as the difference between the global Cartesian coordinates of the m-th frame and the 0-th frame of the video:

[0059]

[0060]

[0061] (1.3) Calculate the global rectangular coordinate system of the reference image of frame 0. .

[0062] (1.4) Global Cartesian coordinate system based on the reference image of frame 0 and global actions , The global Cartesian coordinate system of the m-th frame reference image is calculated using the following formula:

[0063]

[0064]

[0065] (1.5) Finally, the key points of the reference image in frame 0 are migrated to the global Cartesian coordinate system of the corresponding reference image in frame m, that is, the key point coordinates are moved from... Migrate to This enables global redirection.

[0066] (2) Local redirection steps

[0067] Local redirection is divided into related actions and key point actions.

[0068] For the relevant actions, the portrait is first divided into five parts: eyes, mouth, nose, eyebrows, and facial contour. Then, the local Cartesian coordinate system of each of these five parts is calculated, with the corresponding endpoints as follows: Figure 5 As shown.

[0069] Next, perform local redirection following the global redirection steps, but multiply the difference between the origins of the local Cartesian coordinate system by a scaling value to constrain different parts to be redirected to a reasonable location:

[0070]

[0071] In the formula, b represents the distance from the current face portion to the face boundary.

[0072] For key point actions, this embodiment models key point actions as follows:

[0073]

[0074] In the formula, This represents the coordinates of the i-th keypoint in the m-th frame of the reference image in the local Cartesian coordinate system. This represents the coordinates of the i-th keypoint in the local Cartesian coordinate system of the 0-th frame of the reference image. , These represent the coordinates of the i-th keypoint in the local Cartesian coordinate system in the 0th and m-th frames of the driving video, respectively.

[0075] Based on this, the specific steps for partial redirection include:

[0076] (2.1) Each image in the driving video and the reference image are divided into five corresponding parts.

[0077] (2.2) Calculate the local rectangular coordinate system of each part of each image in the driving video.

[0078] (2.3) Based on the local rectangular coordinate system obtained in step (2.2), calculate the relevant actions. The relevant actions include the change in the rotation angle of the local rectangular coordinate system. The calculation method remains the same, but the origin changes. Preset scaling values ​​need to be used. Make corrections.

[0079] (2.4) Calculate the local rectangular coordinate system of each part of the reference image of frame 0.

[0080] (2.5) Based on the local rectangular coordinate system of each part of the reference image of frame 0 and related actions, calculate the local rectangular coordinate system of each part of the reference image of frame m. Refer to step (1.4) for the calculation process.

[0081] (2.6) Transfer the key points of each part of the reference image in frame 0 to the local rectangular coordinate system of each part of the reference image in frame m;

[0082] (2.7) Based on the aforementioned keypoint motion model, the keypoints of each part of the reference image in frame 0 are again transferred to the local Cartesian coordinate system of each part of the reference image in frame m, thus completing the local relocation. Specifically: First, the coordinates of the keypoints of each part in the reference image in frame 0 in the local Cartesian coordinate system are calculated according to the keypoint matching steps guided by appearance. Then calculate the coordinates of key points in the local Cartesian coordinate system for each part of the driving video in frame 0 and frame m. and Finally, the corresponding action model is calculated based on the key points. .

[0083] Based on the aforementioned global and local redirection steps, global actions, related actions, and keypoint actions are transferred from the driving video to the reference image, respectively, resulting in the global Cartesian coordinate system, local Cartesian coordinate system, and keypoint coordinates of the transferred reference image. Then, based on these coordinates, the final keypoint sequence of the reference image is obtained. M represents the number of video frames.

[0084] The global and local retargeting steps provided in this embodiment can capture minute movements and can be applied to any driving video and portrait animation without additional training.

[0085] 3. Keypoint-driven open-domain portrait animation framework

[0086] This embodiment constructs a keypoint-driven open-domain portrait animation framework, which supports the integration of appearance-guided keypoint matching modules and coordinate-based keypoint retargeting modules from the aforementioned steps into any pre-trained keypoint-driven portrait animation model.

[0087] The entire framework works as follows: First, the key points of the reference image are obtained through the appearance-guided key point matching module. Then, the key point sequence of the reference image is obtained by migrating motion from the driving video through the coordinate-based key point relocation module. Finally, the key point sequence is input into the arbitrary key point-driven portrait animation model with additional conditions. This framework can adapt to any driving video and reference image to achieve open-domain portrait animation generation.

[0088] The method provided in this embodiment can be implemented based on any keypoint-driven animation model. To verify the effectiveness of this method, for appearance-guided keypoint matching, the diffusion features of layer 6 at step 301 can be extracted using stable diffusion v1.5 and the pre-trained parameters of the ip-adapter, with 10 target images used. Additionally, 68 keypoints are used to represent the face, and the frame rate of each driving video segment is set to 64. Extensive qualitative and quantitative experiments on portrait animation benchmarks demonstrate the effectiveness and superiority of this method in portrait animation tasks, particularly excelling in non-human character animation, outperforming existing portrait animation methods.

[0089] Optionally, different pre-trained video generation models can be used as base models, integrating the appearance-guided keypoint matching module and coordinate-based keypoint retargeting module proposed in this invention to implement the method proposed in this invention. For example, MOFA-Video can be used as the base model. Furthermore, the method provided by this invention can also be applied to keypoint detection, motion capture transfer, and other fields.

[0090] In summary, the method provided by this invention can effectively detect key points of any portrait and accurately capture subtle movements of driving videos and transfer them to any portrait, without requiring any additional training. The proposed key point-driven open-domain portrait animation framework can adapt to any driving video and reference image, realizing open-domain portrait animation generation.

[0091] Example 2:

[0092] This embodiment provides an open-domain portrait animation system based on driven video, including an appearance-guided keypoint matching module, a coordinate-based keypoint retargeting module, and a portrait animation generation module. The appearance-guided keypoint matching module acquires keypoints from a pre-acquired reference image; the coordinate-based keypoint matching module obtains a keypoint sequence of the reference image by migrating motion from the driven video based on the keypoints; the driven video is also pre-acquired; and the portrait animation generation module inputs the keypoint sequence with additional conditions into a preset keypoint-driven portrait animation model to generate an open-domain portrait animation. The specific implementation steps of each module are basically the same as those in Embodiment 1, and will not be repeated here.

[0093] Furthermore, this embodiment also provides an open-domain portrait animation device based on driven video, including a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements one or more steps of the method in Embodiment 1, which will not be described again here. The processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from the memory unit into random access memory (RAM). Various programs and data required for device operation can also be stored in the RAM. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus. Multiple components in the device are connected to the I / O interfaces, including: input units, such as a keyboard, mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless transceivers, etc. The communication units allow the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit executes the various methods and processes described above, such as one or more steps in the foregoing embodiments.

[0094] Furthermore, this embodiment also provides a storage medium on which a program is stored, which, when executed, implements one or more steps of the method in Embodiment 1. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server. In the context of this invention, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0095] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An open-domain portrait animation method based on driven video, characterized in that, Includes the following steps: Using a preset appearance-guided key point matching module, key points of a reference image are obtained, wherein the reference image is pre-acquired; The operation of the appearance-guided key point matching module specifically includes: constructing k target images and marking key points; inputting the marked target images and the reference images into a pre-trained diffusion model, extracting the diffusion features of the l-th layer at time step t through DDIM inversion, and extracting image appearance features as appearance guidance; upsampling the diffusion features to the original dimension of the image, and matching the features of key points at specific locations on the target image with the features on the reference image to obtain the corresponding key point positions on the reference image; Based on the key points, a preset coordinate-based key point relocation module is used to migrate actions from the driving video to obtain the key point sequence of the reference image. The driving video is pre-acquired. The operation of the coordinate-based key point relocation module includes a global relocation step and a local relocation step. The global relocation step is used to migrate global actions from the driving video to the corresponding reference image, and the local relocation step is used to migrate related actions and key point actions from the driving video to the corresponding reference image. Here, the global action refers to the coordinate change of the entire portrait, the related action refers to the coordinate change of each part divided in the portrait, and the key point action is the specific coordinate change of the point corresponding to each part. The keypoint sequence is input into a preset keypoint-driven portrait animation model with additional conditions to generate an open-domain portrait animation.

2. The open-domain portrait animation method based on driven video according to claim 1, characterized in that, The process of constructing the target image specifically includes: Construct an appearance image library, which includes multiple images of different facial features and different domains; The target image is constructed by matching the nearest domain within the appearance library to the reference image.

3. The open-domain portrait animation method based on driven video according to claim 1, characterized in that, The global redirection step specifically includes: Obtain the key point sequence of the driving video; Calculate the global Cartesian coordinate system corresponding to each image in the driving video; Based on the global Cartesian coordinate system, calculate the global motion, where the global motion is the difference between the global Cartesian coordinate system of the driving video from frame m to frame 0. Calculate the global Cartesian coordinate system of the reference image in frame 0; Based on the global Cartesian coordinate system of the reference image of frame 0 and the global action, calculate the global Cartesian coordinate system of the reference image of frame m; The key points of the reference image in frame 0 are migrated to the global Cartesian coordinate system of the corresponding reference image in frame m, thus achieving global relocation.

4. The open-domain portrait animation method based on driven video according to claim 3, characterized in that, The local redirection step specifically includes: Each image in the driving video and the reference image are divided into corresponding parts; Calculate the local Cartesian coordinate system for each part of each image in the driving video; Based on the local rectangular coordinate system, relevant actions are calculated, including multiplying the difference between the origin of the local rectangular coordinate system from the m-th frame to the 0-th frame of the driving video by a preset scaling value. Calculate the local Cartesian coordinate system for each part of the reference image in frame 0; Based on the local Cartesian coordinate system of each part of the reference image of frame 0 and the related actions, calculate the local Cartesian coordinate system of each part of the reference image of frame m; The key points of each part of the reference image in frame 0 are transferred to the local Cartesian coordinate system of each part of the reference image in frame m. Based on the pre-built keypoint motion model, the keypoints of each part of the reference image in frame 0 are transferred to the local Cartesian coordinate system of each part of the reference image in frame m, thus completing the local relocation.

5. The open-domain portrait animation method based on driven video according to claim 4, characterized in that, The keypoint action model is represented as follows: In the formula, This represents the coordinates of the i-th keypoint in the m-th frame of the reference image in the local Cartesian coordinate system.

6. An open-domain portrait animation system based on driven video, characterized in that, This includes a keypoint matching module guided by appearance, a keypoint retargeting module based on coordinates, and a portrait animation generation module; The appearance-guided key point matching module is used to obtain key points of a reference image, which is pre-acquired. The operation of the appearance-guided key point matching module specifically includes: constructing k target images and marking key points; inputting the marked target images and the reference images into a pre-trained diffusion model, extracting the diffusion features of the l-th layer at time step t through DDIM inversion, and extracting image appearance features as appearance guidance; upsampling the diffusion features to the original dimension of the image, and matching the features of key points at specific locations on the target image with the features on the reference image to obtain the corresponding key point positions on the reference image; The coordinate-based keypoint matching module is used to obtain the keypoint sequence of the reference image by migrating actions from the driving video based on the keypoints, wherein the driving video is pre-acquired; the operation process of the coordinate-based keypoint relocation module includes a global relocation step and a local relocation step, wherein the global relocation step is used to migrate global actions from the driving video to the corresponding reference image, and the local relocation step is used to migrate related actions and keypoint actions from the driving video to the corresponding reference image; wherein, the global action refers to the coordinate change of the entire portrait, the related action refers to the coordinate change of each part divided in the portrait, and the keypoint action is the specific coordinate change of the point corresponding to each part; The portrait animation generation module is used to input the key point sequence into a preset key point-driven portrait animation model with additional conditions to generate an open-domain portrait animation.

7. An open-domain portrait animation device based on driven video, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-5.

8. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-5.

Images