Video generation method, device, equipment and storage medium

Abstract

The embodiment of the present application provides a video generation method, device, equipment and storage medium, the method comprising: obtaining N template expression images and an object image of an object to be processed, extracting expression drive signals corresponding to N groups of three-dimensional template facial key points from the N template expression images, and extracting original expression signals corresponding to the three-dimensional original facial key points of the object to be processed from the object image. According to the expression drive signals corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the N template expression images are transferred to the object image to obtain N initial expression images of the object to be processed. With the object features of the object to be processed in the object image as constraints, expression video data of the object to be processed is generated according to the N initial expression images. The accuracy of the expression video data can be improved by the present application.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video generation method, apparatus, device and storage medium. Background Art

[0002] Expression video generation technology may refer to the generation of expression video data that contains the expression movements of the object to be processed. Expression video generation technology is widely used in social scenarios, film and television entertainment scenarios, etc. For example, in film and television entertainment scenarios, expression video generation technology can be used to produce film and television works, design game characters, design anchor characters, etc.

[0003] At present, expression video data is mainly generated based on generative adversarial networks (GANs). In practice, it is found that due to the limited performance of generative adversarial networks and inaccurate representation of facial expressions and movements, the generated expression video data will contain unrealistic content (such as incorrect facial expressions and movements) and obvious artifacts (such as image blur and image distortion), which reduces the accuracy of the expression video data. Summary of the invention

[0004] The embodiments of the present application provide a video generation method, apparatus, device and storage medium, which can improve the accuracy of expression video data.

[0005] An embodiment of the present application provides a video generation method, including:

[0006] Obtain N template expression images and an object image of an object to be processed; N is an integer greater than 1, and each template expression image includes an expression action;

[0007] Extracting N groups of expression drive signals corresponding to the three-dimensional template facial key points from the N template expression images, and extracting the original expression signals corresponding to the three-dimensional original facial key points of the object to be processed from the object image; one template expression image corresponds to one group of expression drive signals of the three-dimensional template facial key points;

[0008] According to the expression drive signals corresponding to the N groups of facial key points of the three-dimensional templates and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the N template expression images are transferred to the object images to obtain the N initial expression images of the object to be processed;

[0009] Taking the object features of the object to be processed in the object image as constraints, the expression video data of the object to be processed is generated according to the N initial expression images.

[0010] An embodiment of the present application provides a video generation device, including:

[0011] An acquisition module is used to acquire N template expression images and an object image of an object to be processed; N is an integer greater than 1, and each template expression image contains an expression action;

[0012] An extraction module is used to extract N groups of expression drive signals corresponding to the three-dimensional template facial key points from the above-mentioned N template expression images, and to extract the original expression signals corresponding to the three-dimensional original facial key points of the above-mentioned object to be processed from the above-mentioned object image; one template expression image corresponds to a group of expression drive signals of the three-dimensional template facial key points;

[0013] A migration module, for migrating the expression actions corresponding to the N template expression images to the object image, according to the expression drive signals corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, to obtain the N initial expression images of the object to be processed;

[0014] A generating module is used to generate expression video data of the object to be processed based on the N initial expression images, taking the object features of the object to be processed in the object image as constraints.

[0015] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0016] On one hand, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0017] On the one hand, an embodiment of the present application provides a computer program product, including a computer program, and when the above-mentioned computer program is executed by a processor, the steps of the above-mentioned method are implemented.

[0018] In the present application, compared with the expression signal of the two-dimensional facial key point, the expression driving signal of the three-dimensional template facial key point can provide more information, that is, the expression driving signal of the three-dimensional template facial key point can accurately describe the expression action in the template expression image. Therefore, through N groups of expression driving signals of the three-dimensional template facial key points and the original expression signals of the three-dimensional original facial key points, the expression action in the N template expression images can be accurately transferred to the object image, and N initial expression images are obtained, thereby improving the migration accuracy of the expression action. Further, the object features of the object to be processed in the object image are used as constraints, and the expression video data of the object to be processed is generated according to the N initial expression images, so that the object features of the image in the expression video data are consistent with the object features of the object to be processed, avoiding the identity leakage of the object to be processed, and improving the accuracy of the expression video data, so that the generation effect is more natural. The identity leakage here can refer to the object features of the image in the expression video data being inconsistent with the object features of the object to be processed, that is, the object reflected by the image in the expression video data is not the object to be processed, such as the facial shape of the object to be processed is round, while the facial shape of the object in the image in the expression video data is square. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0020] Figure 1 is a structural schematic diagram of a video generation system provided in an embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of a video generation method provided in an embodiment of the present application. Figure 1 ;

[0022] Figure 3 This is a schematic diagram of a video generation method provided in an embodiment of the present application. Figure 2 ;

[0023] Figure 4 This is a schematic diagram of a video generation method provided in an embodiment of the present application. Figure 3 ;

[0024] Figure 5 It is a flowchart of a training method for an initial expression video prediction model provided in an embodiment of the present application;

[0025] Figure 6 It is a flowchart of a video generation method in a social scenario provided by an embodiment of the present application;

[0026] Figure 7is a structural schematic diagram of a video generating device provided in an embodiment of the present application;

[0027] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] To facilitate understanding, some nouns are briefly explained as follows:

[0030] (1) Facial key points, which can be used to reflect the key facial parts (such as eyes, mouth, eyebrows, etc.) and facial contours of an object (such as the object to be processed) in the image space. The image space may include a stereoscopic image space and a planar image space. The stereoscopic image space may refer to a space with three dimensions (such as length, width, and height), and the planar image space may refer to a space with two dimensions (such as length and width). The points used to reflect the key facial parts and facial contours of an object in the stereoscopic image space may be called three-dimensional facial key points, and the points used to reflect the key facial parts and facial contours of an object in the planar image space may be called two-dimensional facial key points.

[0031] For the sake of distinction, in this application, the three-dimensional facial key points extracted from the template expression image are referred to as the three-dimensional template facial key points, that is, the three-dimensional template facial key points can be points in the stereoscopic image space used to reflect the facial key parts and facial contours of the template object in the template expression image, and the template object can refer to the object corresponding to the template expression image. The template object can be a person, an animal, a virtual character, etc. The three-dimensional facial key points extracted from the object image of the object to be processed are referred to as the three-dimensional original facial key points, that is, the three-dimensional original facial key points can be points in the stereoscopic image space used to reflect the facial key parts and facial contours of the object to be processed in the object image.

[0032] (2) The expression prediction model may refer to a model used to generate expression video data. The expression video prediction model may be any one of a convolutional neural network model, a clustering model, a generative adversarial network model, and a diffusion model.

[0033] (3) A variational auto-encoder (VAE) can be used to project an image from a pixel space to a feature space, and then project the image from the feature space back to the image space. VAE can be divided into two parts: the part that can be used to project an image from an image space to a feature space can be called a VAE encoder, and the part that can be used to project an image from a feature space to an image space can be called a VAE decoder. The video decoder in the present application can be a VAE decoder.

[0034] In order to facilitate a clearer understanding of this application, first introduce Figure 1 , Figure 1 A schematic diagram of the structure of a video generation system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the structural diagram of the video generation system includes a server 10 and a terminal cluster. The terminal cluster may include multiple terminals, such as Figure 1 As shown, taking the terminal cluster including 3 terminals as an example, the terminal cluster can specifically include a first terminal, a second terminal, and a third terminal; it can be understood that the first terminal, the second terminal, and the third terminal can all be connected to the server through a network connection, so that each terminal can exchange data with the server through the network connection.

[0035] It is understandable that the server can be an independent physical server, or a server cluster or distributed system composed of at least two physical servers, or a cloud server that provides cloud services, cloud knowledge bases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can specifically refer to a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a screen speaker, a smart TV, a smart watch, etc., but is not limited to this. Each terminal and server can be directly or indirectly connected via wired or wireless communication. At the same time, the number of terminals and servers can be one or at least two, and this application does not limit this.

[0036] In the present application, the server 10 can be used to train an expression video prediction model, which can be obtained by adjusting the parameters in the initial expression video prediction model using the sample object image and M sample expression images of the sample object as training samples. The initial expression video prediction model can refer to the expression video prediction model to be trained.

[0037] In one embodiment, after the server 10 obtains the expression video prediction model through training, the expression video prediction model may be stored in the local memory of the server 10, and any terminal in the terminal cluster may generate expression video data by calling the expression video prediction model in the local memory of the server 10. For example, any terminal in the terminal cluster may send an expression video generation request to the server 10, and after receiving the expression video generation request, the server 10 generates expression video data of the object to be processed and returns the expression video data to the terminal.

[0038] In one embodiment, after training the expression video prediction model, the server 10 can send the expression video prediction model to each terminal in the terminal cluster. Each terminal can store the expression video prediction model in a local memory, and each terminal can generate expression video data through the expression video prediction model in the local memory.

[0039] In one embodiment, after the server obtains the expression video prediction model through training, the expression video prediction model can be sent to a cloud server. The cloud server can store the expression video prediction model in a local memory of the cloud server. Each terminal can call the expression video prediction model in the local memory of the cloud server to generate expression video data.

[0040] It should be noted that the process of generating expression video data based on the expression video prediction model and the training process of the expression video prediction model can be referred to below.

[0041] For further information, see Figure 2 , is a schematic diagram of a process of an image processing method provided in an embodiment of the present application Figure 1 .like Figure 2 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1 can be executed by the server in Figure 1 The terminal and server in the terminal cluster are used to perform collaboratively. The devices used to perform the image processing method in this application can be collectively referred to as computer devices. The method may include the following steps:

[0042] S101, obtaining N template expression images and an object image of an object to be processed.

[0043] In the present application, the computer device can obtain N template expression images by locally acquiring or downloading from the network; or obtain template expression video data by locally acquiring or downloading from the network, and perform frame extraction processing on the template expression video data in a frame extraction manner to obtain N template expression images. The object image of the object to be processed can be obtained by locally acquiring or downloading from the network.

[0044] The local acquisition may be any of the following: 1. Acquisition from a local storage of a computer device; 2. Acquisition through photography of a computer device; 3. Generation through software of a computer device.

[0045] The frame extraction method may refer to an interval frame extraction method or a continuous frame extraction method, and the interval frame extraction method may refer to extracting a template expression image from the template expression video data for each interval playback duration. For example, the playback duration of the template expression video data is 10 seconds, and the interval playback duration may be 0.5 seconds. The computer device may start playing from the 0th second, and extract a template expression image from the template expression video data for each interval of 0.5 seconds, to obtain 20 template expression images, that is, the playback time difference between each two adjacent template expression images is 0.5 seconds.

[0046] It should be noted that each template expression image contains expression actions, and the expression actions contained in each template expression image can refer to the facial expressions of the template objects corresponding to each template expression image. Among the N template expression images, at least two template expression images contain different expression actions, that is, the N template expression images can be used to reflect multiple expression actions. The template object and the object to be processed can be humans, animals, virtual characters, etc., and facial expressions can include smiling, frowning, staring, etc. Under different facial expressions, the position or shape of the key facial parts and facial contours of the template object will be different. The above-mentioned key facial parts can refer to the eyes, eyebrows, mouth, etc. of the template object. The number of template objects can be one or more. When the number of the above-mentioned template objects is one, each template expression image contains one expression action; when the number of the above-mentioned template objects is multiple, the expression actions of multiple template objects can be the same or different, that is, each template expression image can contain one or more expression actions.

[0047] It should be noted that the object to be processed may refer to all objects in the object image, or the object to be processed may refer to any specified object in the object image, the number of objects to be processed may be one or more, and the object image includes an expression of the object to be processed.

[0048] It should be noted that the present application can be used in social scenarios, film and television entertainment scenarios, and other scenarios. For example, in film and television entertainment scenarios, the template expression video data can be a completed film and television work. When character A in the film and television work needs to be replaced with character B, the N template expression images can refer to the images containing character A in the film and television work, that is, character A is the template object, character B is the object to be processed, and the image containing character B can be called an object image. Through the present application, the expression actions of character A are transferred to character B, so that a film and television work including character B can be generated, and the role exchange in the film and television work can be realized.

[0049] For example, in a social scenario, N template expression images may refer to N expression images that have been generated that include user C. When it is necessary to generate expression video data of user D, user C may become a template object, user D may be referred to as an object to be processed, N expression images that include user C may be referred to as N template expression images, and images that include user D may be referred to as object images. By migrating the expression actions of user C to user D through this application, personalized expression video data can be generated.

[0050] S102, extracting N groups of expression driving signals corresponding to the three-dimensional template facial key points from the N template expression images, and extracting original expression signals corresponding to the three-dimensional original facial key points of the object to be processed from the object image.

[0051] In the present application, the computer device can input each template expression image into the key point extraction model, and through the key point extraction model, recognize and extract the expression action in each template expression image to obtain the expression drive signal of the three-dimensional template facial key point corresponding to each template expression image. The computer device can also input the object image into the key point extraction model, and through the key point extraction model, recognize and extract the expression action in the object image to obtain the original expression signal of the three-dimensional original facial key point corresponding to the object image. Among them, the expression drive signal of the three-dimensional template facial key point and the original expression signal of the three-dimensional original facial key point accurately reflect the expression action, which is conducive to improving the accuracy of expression action migration.

[0052] Among them, the above-mentioned key point extraction model may include a MediaPipe model (a deep learning face detection model), a point cloud model, a stacked hourglass model, etc.

[0053] It should be noted that a set of expression driving signals corresponding to the three-dimensional template facial key points can be used to reflect the expression movements of the template object in the corresponding template expression image, and the original expression signals corresponding to the three-dimensional original facial key points are used to reflect the expression movements of the object to be processed in the object image.

[0054] The position information or shape size of the key facial parts and facial contour of the template object in the template expression image in the stereoscopic image space. The three-dimensional original facial key points can be points used to reflect the key facial parts and facial contour of the object to be processed in the object image. The original expression signal corresponding to the three-dimensional original facial key points can be used to reflect the position information or shape size of the key facial parts and facial contour of the object to be processed in the object image in the stereoscopic image space.

[0055] S103, according to the expression driving signals corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the N template expression images are transferred to the object image to obtain N initial expression images of the object to be processed.

[0056] In the present application, the computer device can transfer the expression actions corresponding to N template expression images to the object image according to the expression drive signals corresponding to N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, and obtain N initial expression images of the object to be processed. The expression actions in the N initial expression images can be made consistent with the expression actions in the N template expression images, such as the expression action of the object to be processed in the first initial expression image is the same as the expression action of the template object in the first template expression image, that is, an expression image with multiple expression actions of the object to be processed is generated, thereby improving the diversity of the expression actions of the object to be processed.

[0057] It should be noted that when the number of template objects in the template expression image and the number of objects to be processed in the object image are both one, based on the expression driving signals corresponding to N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the template objects in the N template expression images are migrated to the objects to be processed in the object image to obtain N initial expression images of the objects to be processed.

[0058] It should be noted that when the number of template objects in the template expression image is one and the number of objects to be processed in the object image is multiple, the expression movements corresponding to the template objects in the N template expression images are migrated to the objects to be processed in the object image according to the expression driving signals corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, so as to obtain N initial expression images, that is, the expression movements of the objects to be processed in any initial expression image are the same as the expression movements of the template objects in the corresponding template expression image, that is, the objects to be processed in the same initial expression image have the same expression movements.

[0059] It should be noted that when the number of template objects in the template expression image and the number of objects to be processed in the object image are both L, according to the expression drive signals corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the s-th template objects in the N template expression images are respectively migrated to the s-th object to be processed in the object image, until the expression actions of the L template objects are migrated, and N initial expression images are obtained. That is, the expression action of the s-th object to be processed in any initial expression image is the same as the expression action of the s-th template object in the template expression image, s is a positive integer less than or equal to L, and L is an integer greater than 1. The s-th object to be processed can be an object to be processed in the object image that has a similar relationship with the s-th template object, and the similar relationship can refer to the same or similar facial size, shape, position in the image, etc. of the object (object to be processed, template object).

[0060] It should be noted that when the number of template objects in the template expression image is S, the number of objects to be processed in the object image is L, and S is less than L, and S is an integer greater than 1, according to the expression drive signals corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to each template object in the N template expression images are migrated to the objects to be processed with similar relationships in the object image, until all L objects to be processed are migrated to obtain expression actions, and N initial expression images are obtained. That is, some objects to be processed can share the expression actions of the same template object.

[0061] It should be noted that the present application is mainly described by taking the example that the number of template objects in the template expression image and the number of objects to be processed in the object image are both one.

[0062] S104 , using the object features of the object to be processed in the object image as constraints, and generating expression video data of the object to be processed according to N initial expression images.

[0063] In the present application, a computer device can use the object features of the object to be processed in the object image as constraints to ensure that the identity of the object to be processed is not leaked, and perform smoothing on N initial expression images to obtain expression video data of the object to be processed, so as to make the expression movements of the object to be processed in the expression video data smoother, that is, the expression movements of the object to be processed are more natural, thereby improving the fluency of the expression video data.

[0064] It should be noted that the object features may include facial region features and body region features of the object to be processed. Among them, facial region features may include face contour, eye shape, eyeball color, ear shape, facial markings, etc., and body region features may include clothing style, hair style, body shape, height, etc. Taking the object features of the object to be processed in the object image as a constraint condition may mean controlling the object features of the object to be processed in each image in the expression video data to be consistent with the object features of the object to be processed in the object image, so as to avoid leakage of the identity of the object to be processed.

[0065] In the present application, compared with the expression signals of the two-dimensional facial key points, the expression driving signals of the three-dimensional template facial key points can provide more information, that is, the expression driving signals of the three-dimensional template facial key points can accurately describe the expression movements in the template expression image. Therefore, through N groups of expression driving signals of the three-dimensional template facial key points and the original expression signals of the three-dimensional original facial key points, the expression movements in the N template expression images can be accurately transferred to the object image, and N initial expression images can be obtained, thereby improving the migration accuracy of the expression movements. Furthermore, with the object features of the object to be processed in the object image as the constraint condition, the expression video data of the object to be processed is generated according to the N initial expression images, so that the object features of the image in the expression video data are consistent with the object features of the object to be processed, avoiding the identity leakage of the object to be processed, and improving the accuracy of the expression video data, making the generation effect more natural.

[0066] Please refer to Figure 3 , Figure 3 A schematic diagram of a video generation method provided in an embodiment of the present application Figure 2 The video generation method can be executed by a computer device, which may include Figure 1 The terminal device (such as the first terminal, the second terminal, etc.) or the server 10 shown. Figure 3 As shown, the method may include:

[0067] S201, obtaining N template expression images and an object image of an object to be processed.

[0068] S202, extracting N groups of expression driving signals corresponding to the three-dimensional template facial key points from the N template expression images, and extracting original expression signals corresponding to the three-dimensional original facial key points of the object to be processed from the object image.

[0069] S203: Obtain, from the N groups of three-dimensional template facial key points, three-dimensional reference facial key points corresponding to each group of three-dimensional template facial key points.

[0070] In the present application, a computer device can obtain a reference template expression image of the i-th template expression image, and use the three-dimensional template facial key points mentioned in the reference template expression image of the i-th template expression image as the three-dimensional reference facial key points of the i-th group of three-dimensional template facial key points.

[0071] Among them, the i-th group of three-dimensional template facial key points are the three-dimensional template facial key points extracted from the i-th template expression image, and the reference template expression image of the i-th template expression image may refer to a template expression image that appears before the appearance time of the i-th template expression image. In particular, the reference template expression image of the first template expression image is itself, the three-dimensional reference facial key points of the first group of three-dimensional template facial key points are the first group of three-dimensional template facial key points, and the first group of three-dimensional template facial key points refer to those extracted from the first template expression image. The first template expression image may be the template expression image that appears the earliest among the N template expression images, and i is an integer less than or equal to N and greater than 1.

[0072] It should be noted that each of the N template expression images corresponds to an appearance time, and the appearance time may be the shooting time or the playback time corresponding to the N template expression images, or the appearance time may be manually set.

[0073] S204, determining expression migration data corresponding to each group of 3D template facial key points according to the expression driving signals of each group of 3D template facial key points and the expression driving signals of the corresponding 3D reference facial key points.

[0074] It should be noted that the expression migration data corresponding to any set of facial key points of the three-dimensional template can be used for the expression movements in the corresponding template expression image, the movement change amount of the expression movement relative to the reference template expression image, that is, the movement change amount reflects the motion characteristics of the expression movement, and the movement change amount can include the facial shape deviation, translation deviation matrix and rotation matrix mentioned below.

[0075] It is understandable that the computer device can perform difference processing on the expression driving signals of each group of 3D template facial key points and the expression driving signals of the corresponding 3D reference facial key points to obtain the expression migration data corresponding to each group of 3D template facial key points.

[0076] Optionally, the above-mentioned expression drive signal includes a translation matrix, a rotation matrix and facial shape parameters. The translation matrix, the rotation matrix and the facial shape parameters can be used to correspond to the expression movements in the template expression image. The translation matrix is ​​used to reflect the translation distance of each group of three-dimensional template facial key points relative to the initial position, that is, the translation matrix is ​​used to reflect the translation distance of the facial key parts and facial contours of the template object in the stereoscopic image space. The rotation matrix is ​​used to reflect the rotation angle of each group of three-dimensional template facial key points relative to the initial position, and the rotation matrix is ​​used to reflect the rotation angle of the facial key parts and facial contours of the template object in the stereoscopic image space. The facial shape parameters can be used to reflect the depth information of each group of three-dimensional template facial key points, that is, the facial shape parameters can be used to reflect the facial shape, facial size, etc. of the facial key parts and facial contours of the template object in the stereoscopic image space.

[0077] The initial position may refer to the focus position of a shooting device used to shoot the template expression image, or the initial position may refer to the initial position of the template object in the stereoscopic image space, such as the initial position may refer to the position where the template expression image is shot.

[0078] In one embodiment, the above step S204 includes: the computer device can subtract the translation matrix of each group of three-dimensional template facial key points from the translation matrix of the corresponding three-dimensional reference facial key points, and the subtraction result is the translation deviation matrix corresponding to each group of three-dimensional template facial key points. The facial shape parameters of each group of three-dimensional template facial key points are subtracted from the facial shape parameters of the corresponding three-dimensional reference facial key points, and the subtraction result is the facial shape deviation corresponding to each group of three-dimensional template facial key points. The computer device can determine the facial shape deviation, translation deviation matrix and rotation matrix corresponding to each group of three-dimensional template facial key points as the expression migration data corresponding to each group of three-dimensional template facial key points.

[0079] It should be noted that the translation deviation matrix can be used to reflect the translation distance of each group of 3D template facial key points relative to the 3D reference facial key points, and the facial shape deviation can be used to reflect the facial change in the facial shape parameters of each group of 3D template facial key points relative to the facial shape parameters of the 3D reference facial key points, and the facial change reflects the differences in facial size and facial shape.

[0080] S205. Based on the expression migration data corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the N template expression images are migrated to the object image to obtain N initial expression images of the object to be processed.

[0081] In the present application, the expression movements corresponding to N template expression images are respectively transferred to the object image according to the expression transfer data corresponding to N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, thereby obtaining N initial expression images of the object to be processed. In contrast, the motion features of the expression movements corresponding to the N template expression images are transferred to the object image, so that the running features of the expression movements in the N initial expression images are kept consistent with the motion features of the expression movements corresponding to the N template expression images, so that the expression movements in the N initial expression images are kept consistent with the expression movements in the corresponding template expression images. At the same time, the facial features of the object to be processed itself will not be changed, such as the facial shape of the object to be processed will not change with the facial shape of the template object, thereby avoiding the identity leakage of the object to be processed.

[0082] In one embodiment, the above step S205 includes: the computer device can update the original expression signals corresponding to the three-dimensional original facial key points based on the expression migration data corresponding to the N groups of three-dimensional template facial key points, and the update can refer to rotation or movement, so as to obtain the updated expression signals corresponding to the N groups of three-dimensional updated facial key points of the object to be processed. Among them, the updated expression signals of a group of three-dimensional updated facial key points correspond to the expression actions in a template expression image. The computer device can project the updated expression signals corresponding to the above N groups of three-dimensional updated facial key points, that is, project the N groups of three-dimensional updated facial key points from the stereoscopic image space to the plane image space, and obtain the expression signals corresponding to the N groups of two-dimensional facial key points of the object to be processed. Then, according to the expression signals corresponding to the N groups of two-dimensional facial key points of the object to be processed, the N initial expression images of the object to be processed can be determined.

[0083] It should be noted that a set of updated expression signals of three-dimensional updated facial key points can be used to reflect the corresponding migrated expression movements of the object to be processed, and the migrated expression movements corresponding to the updated expression signals of a set of three-dimensional updated facial key points correspond to the expression movements in a template expression image.

[0084] It should be noted that the expression signals of a set of two-dimensional facial key points can be used to reflect the expression movements of the object to be processed in the corresponding initial expression image. The expression movements in the initial expression image corresponding to the expression signals of a set of two-dimensional facial key points correspond to the expression movements after migration corresponding to the updated expression signals of a set of three-dimensional updated facial key points. Therefore, the expression movements in the initial expression image corresponding to the expression signals of a set of two-dimensional facial key points correspond to the expression movements in a template expression image, so that the expression movements in N initial expression images are consistent with the expression movements in the corresponding template expression images.

[0085] It should be noted that the above-mentioned projection processing may refer to projecting the updated expression signals of the three-dimensional updated facial key points from the stereoscopic image space to the two-dimensional image space to obtain the expression signals of the corresponding two-dimensional facial key points in the two-dimensional image space.

[0086] In the present application, the computer device can update the original expression signal corresponding to the three-dimensional original facial key points according to the expression migration data corresponding to the N groups of three-dimensional template facial key points, and obtain the updated expression signal corresponding to the N groups of three-dimensional updated facial key points of the object to be processed. In contrast, the expression actions in the template expression images corresponding to the N groups of three-dimensional template facial key points are migrated to the object image relative to the movement change of the expression actions in the reference template expression image, so that the running characteristics of the migrated expression actions corresponding to the updated expression signals corresponding to the N groups of three-dimensional updated facial key points are consistent with the movement characteristics of the expression actions corresponding to the N template expression images. Migration based on the movement change does not change the facial shape of the object to be processed with the facial shape of the template object, and does not change the facial features of the object to be processed itself, so the identity leakage of the object to be processed can be avoided.

[0087] Further, the computer device can project the updated expression signals of the N groups of three-dimensional updated facial key points from the stereoscopic image space to the plane image space based on the updated expression signals corresponding to the N groups of three-dimensional updated facial key points, and obtain the N groups of two-dimensional facial key points of the object to be processed and the expression signals corresponding to the N groups of two-dimensional facial key points of the object to be processed. At the same time, the computer device can draw the first group of two-dimensional facial key points of the object to be processed based on the expression signals of the first group of two-dimensional facial key points of the object to be processed, and obtain the first initial expression image of the object to be processed. The above operations are performed in sequence until the N initial expression images of the object to be processed are obtained. Compared with the expression signals of the two-dimensional facial key points, the expression driving signals of the three-dimensional template facial key points can provide more information. Therefore, the data complexity of the three-dimensional key points is greater than that of the two-dimensional key points. When the expression signals of the three-dimensional key points corresponding to the initial expression image are used to generate the expression video data, the computer device consumes more computing resources. Projecting the expression signals of the three-dimensional key points into the expression signals of the two-dimensional key points, and using the expression signals of the two-dimensional key points corresponding to the initial expression image to generate the expression video data can reduce the computing resources consumed by the computer device and improve the generation efficiency of the expression video data.

[0088] Optionally, the three-dimensional reference facial key points corresponding to the above-mentioned N groups of three-dimensional template facial key points are all the first group of three-dimensional template facial key points, and the above-mentioned first group of three-dimensional template facial key points are extracted from the template expression image that appears earliest among the above-mentioned N template expression images; the above-mentioned expression migration data includes facial shape deviation, translation deviation matrix and rotation matrix; the above-mentioned original expression signal includes translation matrix and facial shape parameters.

[0089] It should be noted that when N template expression images are obtained by frame extraction of template expression video data, the first group of three-dimensional template facial key points are extracted from the template expression image that appears earliest in the template expression video data among the N template expression images.

[0090] In one embodiment, based on the expression migration data corresponding to the N groups of three-dimensional template facial key points, the original expression signals corresponding to the three-dimensional original facial key points are updated to obtain the updated expression signals corresponding to the N groups of three-dimensional updated facial key points of the object to be processed, including: the computer device can sum the translation deviation matrix corresponding to the i-th group of three-dimensional template facial key points and the translation matrix corresponding to the three-dimensional original facial key points to obtain the updated translation matrix corresponding to the i-th group of three-dimensional updated facial key points of the object to be processed. Wherein, i is a positive integer less than or equal to N. The computer device can sum the facial shape deviation corresponding to the i-th group of three-dimensional template facial key points and the facial shape parameters corresponding to the three-dimensional original facial key points to obtain the updated facial shape parameters corresponding to the i-th group of three-dimensional updated facial key points, and determine the rotation matrix corresponding to the i-th group of three-dimensional template facial key points as the updated rotation matrix corresponding to the i-th group of three-dimensional updated facial key points. At this time, the computer device can determine the updated translation matrix, updated facial shape parameters and updated rotation matrix corresponding to the i-th group of three-dimensional updated facial key points of the object to be processed as the updated expression signal corresponding to the i-th group of three-dimensional updated facial key points. This can make the amount of action change corresponding to the expression migration data between the updated expression signal corresponding to each group of three-dimensional updated facial key points and the updated expression signal corresponding to the first group of three-dimensional updated facial key points consistent with the amount of action change corresponding to the expression migration data of each group of three-dimensional template facial key points. The expression actions in N template expression images can be accurately migrated to the object image. The running characteristics of the migrated expression actions corresponding to the updated expression signals corresponding to the N groups of three-dimensional updated facial key points are consistent with the motion characteristics of the expression actions corresponding to the N template expression images. At the same time, the facial features of the object to be processed itself will not be changed, avoiding the identity leakage of the object to be processed.

[0091] The above-mentioned summation process may include addition process, weighted summation process, etc. This can avoid directly using the expression drive signals corresponding to the N groups of facial key points of the three-dimensional template, and adjust the original expression signals of the three-dimensional original facial key points, so that the identity of the object to be processed is leaked, and the migration accuracy of the expression action is improved.

[0092] Optionally, the three-dimensional reference facial key points corresponding to the first group of three-dimensional template facial key points are the above-mentioned first group of three-dimensional template facial key points, the three-dimensional reference facial key points corresponding to the j-th group of three-dimensional template facial key points are the j-1-th group of three-dimensional template facial key points, the above-mentioned first group of three-dimensional template facial key points are extracted from the template expression image with the earliest appearance time among the above-mentioned N template expression images, the appearance time of the template expression image corresponding to the j-1-th group of three-dimensional template facial key points is before the appearance time of the template expression image corresponding to the above-mentioned j-th group of three-dimensional template facial key points, and j is an integer greater than 1 and less than or equal to N.

[0093] In one embodiment, the original expression signals corresponding to the three-dimensional original facial key points are updated according to the expression migration data corresponding to the N groups of three-dimensional template facial key points to obtain the updated expression signals corresponding to the N groups of three-dimensional updated facial key points of the object to be processed, including: the computer can update the updated expression signals corresponding to the three-dimensional original facial key points according to the expression migration data corresponding to the first group of three-dimensional template facial key points to obtain the updated expression signals corresponding to the first group of three-dimensional updated facial key points of the object to be processed; according to the expression migration data corresponding to the jth group of three-dimensional template facial key points, the updated expression signals corresponding to the j-1th group of three-dimensional updated facial key points are adjusted to obtain the updated expression signals corresponding to the jth group of three-dimensional updated facial key points of the object to be processed.

[0094] For example, taking three template expression images as an example, the first group of three-dimensional template facial key points is extracted from the first template expression image, the second group of three-dimensional template facial key points is extracted from the second template expression image, and the third group of three-dimensional template facial key points is extracted from the third template expression image. At this time, the three-dimensional reference facial key points corresponding to the second group of three-dimensional template facial key points are the first group of three-dimensional template facial key points, and the three-dimensional reference facial key points corresponding to the third group of three-dimensional template facial key points are the first group of three-dimensional template facial key points.

[0095] In the present application, the computer device can use the expression migration data corresponding to the first group of three-dimensional template facial key points to update the expression signal of the three-dimensional original facial key points, and obtain the updated expression signal corresponding to the first group of three-dimensional updated facial key points. In contrast, the motion features of the expression actions corresponding to the first template expression image are migrated to the object image, so that the running features of the expression actions in the first initial expression image are consistent with the motion features of the migrated expression actions corresponding to the updated expression signal corresponding to the first group of three-dimensional updated facial key points.

[0096] Furthermore, the computer device can use the expression migration data corresponding to the second group of three-dimensional template facial key points to update the updated expression signals corresponding to the first group of three-dimensional updated facial key points, and obtain the updated expression signals corresponding to the second group of three-dimensional updated facial key points. That is, the motion features of the expression actions corresponding to the second template expression image are migrated to the expression actions corresponding to the updated expression signals corresponding to the second group of three-dimensional updated facial key points, so that the running features of the expression actions in the second initial expression image are consistent with the motion features of the expression actions after migration corresponding to the updated expression signals corresponding to the second group of three-dimensional updated facial key points. By performing the above operations in sequence, the updated expression signals corresponding to the N three-dimensional updated facial key points can be updated. Therefore, the running features of the expression actions after migration corresponding to the updated expression signals corresponding to the N groups of three-dimensional updated facial key points are consistent with the motion features of the expression actions corresponding to the N template expression images. Based on the amount of change in facial movements between each two adjacent template facial expression images, the facial movements of the updated expression signals corresponding to the previous set of three-dimensional updated facial key points are migrated. The facial shape of the object to be processed is transformed without changing the facial shape directly based on the template object, and the facial features of the object to be processed itself are not changed. Therefore, the identity leakage of the object to be processed can be avoided.

[0097] Optionally, the above-mentioned projection processing of the updated expression signals corresponding to the N groups of three-dimensional updated facial key points to obtain the expression signals of the N groups of two-dimensional facial key points of the object to be processed includes: extracting the three-dimensional eye key points in the k-th group of three-dimensional updated facial key points; k is a positive integer less than or equal to N; determining the k-th projection center point based on the coordinates of the three-dimensional eye key points in the k-th group of three-dimensional updated facial key points; and projecting the updated expression signals corresponding to the k-th group of three-dimensional updated facial key points with the relative distance between the k-th projection center point and the k-th group of three-dimensional updated facial key points as a restriction condition to obtain the expression signals of the k-th group of two-dimensional facial key points of the object to be processed.

[0098] It should be noted that the 3D eye key point may be a point used to reflect the eye of the object to be processed in the stereoscopic image space. The eye may include a pupil, an eye contour, etc. The point used to reflect the pupil of the object to be processed in the stereoscopic image space may be called a 3D pupil key point, and the point used to reflect the eye contour of the object to be processed in the stereoscopic image space may be called a 3D eye contour key point.

[0099] It should be noted that the relative distance may refer to the ratio of the distance between two or more three-dimensional facial key points to a certain reference distance, and the reference distance may be manually set or a default value in the computer device. The above-mentioned projection center point may be used to reflect the center or center of gravity of the facial area of ​​the object to be processed, and a three-dimensional coordinate axis (such as the X axis, Y axis, and Z axis) may be established with the projection center point as the origin, and the coordinates of each three-dimensional facial key point in the stereoscopic image space may be determined based on the three-dimensional coordinate axis.

[0100] In one embodiment, the three-dimensional eye key points are four three-dimensional eye contour key points as an example, and the four three-dimensional eye contour key points are respectively a three-dimensional left eye upper contour key point, a three-dimensional left eye lower contour key point, a three-dimensional right eye upper contour key point, and a three-dimensional right eye lower contour key point. The coordinates of the projection center point can be obtained by performing average processing on the four axis coordinates corresponding to the three-dimensional left eye upper contour key point, the three-dimensional left eye lower contour key point, the three-dimensional right eye upper contour key point, and the three-dimensional right eye lower contour key point.

[0101] For example, when the coordinates of the 3D left eye upper contour key point are (a1, b1, c1), the coordinates of the 3D left eye upper contour key point are (a2, b2, c2), the coordinates of the 3D right eye upper contour key point are (a3, b3, c3), and the coordinates of the 3D right eye lower contour key point are (a4, b4, c4), then the coordinates of the projection center point are ( ).

[0102] In one embodiment, taking the three-dimensional eye key points as two three-dimensional pupil key points as an example, the two three-dimensional pupil key points are respectively a three-dimensional left pupil key point and a three-dimensional right pupil key point. The coordinates of the projection center point can be obtained by performing average processing on the two axis coordinates corresponding to the three-dimensional left pupil key point and the three-dimensional right pupil key point.

[0103] For example, the coordinates of the 3D left pupil key point are (a5, b5, c5), and the coordinates of the 3D right pupil key point are (a6, b6, c6), then the coordinates of the projection center point are ( ).

[0104] In the present application, the computer device can obtain the three-dimensional eye key points in the first group of three-dimensional updated facial key points from the first group of three-dimensional updated facial key points, perform average processing on the coordinates of the three-dimensional eye key points in the first group of three-dimensional updated facial key points, and obtain the coordinates of the first projection center point; according to the updated expression signal corresponding to the first group of three-dimensional updated facial key points, the first group of three-dimensional updated facial key points is projected to obtain the initial expression signal of the first group of two-dimensional facial key points. The relative distance between the first projection center point and the first group of three-dimensional updated facial key points is calculated, and based on the relative distance between the first projection center point and the first group of three-dimensional updated facial key points, the initial expression signal of the first group of two-dimensional facial key points is adjusted to obtain the expression signal of the first group of two-dimensional facial key points. The above operations are performed in sequence until the expression signals of N groups of two-dimensional facial key points of the object to be processed are obtained. In this way, the relative distance between the expression signal of each group of two-dimensional facial key points and the projection center point in the expression signal of each group of two-dimensional facial key points can be made consistent with the relative distance between the corresponding projection center point and the corresponding three-dimensional updated facial key point, thereby avoiding the identity leakage of the object to be processed that may be caused during the projection process and improving the accuracy of the expression video data.

[0105] It should be noted that, based on the relative distance between the first projection center point and the first group of three-dimensional updated facial key points, the initial expression signal of the first group of two-dimensional facial key points is adjusted, and reference may be made to the description below.

[0106] Optionally, the updated expression signal includes facial shape parameters, a translation matrix and a rotation matrix; the updated expression signal corresponding to the kth group of three-dimensional updated facial key points is projected with the relative distance between the kth projection center point and the kth group of three-dimensional updated facial key points as a restriction condition to obtain the expression signal of the kth group of two-dimensional facial key points of the object to be processed, including: the computer device can multiply the translation matrix corresponding to the kth group of three-dimensional updated facial key points and the rotation matrix corresponding to the kth group of three-dimensional updated facial key points to obtain the transformation matrix corresponding to the kth group of three-dimensional updated facial key points. According to the transformation matrix corresponding to the kth group of three-dimensional updated facial key points, the facial shape parameters corresponding to the kth group of three-dimensional updated facial key points are mapped to obtain the initial expression signal of the kth group of two-dimensional facial key points of the object to be processed. According to the relative distance between the kth projection center point and the kth group of three-dimensional updated facial key points, the initial expression signal of the kth group of two-dimensional facial key points of the object to be processed is adjusted to obtain the expression signal of the kth group of two-dimensional facial key points of the object to be processed.

[0107] It should be noted that the translation matrix of each group of 3D updated facial key points can be used to reflect the translation distance of each group of 3D updated facial key points relative to the first group of 3D updated facial key points. That is, the translation matrix is ​​used to reflect the translation distance of the facial key parts and facial contours of the object to be processed in the stereoscopic image space. The rotation matrix of each group of 3D updated facial key points can be used to reflect the rotation angle of each group of 3D updated facial key points relative to the first group of 3D updated facial key points. That is, the rotation matrix is ​​used to reflect the rotation angle of the facial key parts and facial contours of the object to be processed in the stereoscopic image space. The facial shape parameters of each group of 3D updated facial key points are the depth information of each group of 3D updated facial key points. That is, the facial shape parameters can be used to reflect the facial shape, facial size, etc. of the facial key parts and facial contours of the object to be processed in the stereoscopic image space.

[0108] It should be noted that the above-mentioned mapping process may include perspective projection processing of the facial shape parameters corresponding to the kth group of three-dimensional updated facial key points, and may also include using a key point mapping model to process the facial shape parameters corresponding to the kth group of three-dimensional updated facial key points. The key point mapping model may include a multi-planar reconstruction (MPR) model, a surface shaded display (SSD) model, etc. The above-mentioned product processing may include a simple product processing, a weighted product processing, etc.

[0109] It can be understood that the computer device can perform product processing on the translation matrix corresponding to the first group of three-dimensional updated facial key points and the rotation matrix corresponding to the first group of three-dimensional updated facial key points to obtain the transformation matrix corresponding to the first group of three-dimensional updated facial key points. The computer device can input the transformation matrix corresponding to the first group of three-dimensional updated facial key points and the facial shape parameters corresponding to the first group of three-dimensional updated facial key points into the key point mapping model to obtain the initial expression signal of the first group of two-dimensional facial key points of the object to be processed. After determining the projection center point in the first group of two-dimensional facial key points, the relative distance between the projection center point in the first group of two-dimensional facial key points and the first group of two-dimensional facial key points can be calculated, and the relative distance between the first projection center point and the first group of three-dimensional updated facial key points can be calculated. The computer device can calculate the distance matching degree between the relative distance between the projection center point in the first group of two-dimensional facial key points and the first group of two-dimensional facial key points and the relative distance between the first projection center point and the first group of three-dimensional updated facial key points. When the distance matching degree is greater than the distance matching degree threshold, the computer device can adjust the initial expression signal of the first group of two-dimensional facial key points. When the distance matching degree between the relative distance between the projection center point in the first group of two-dimensional facial key points and the adjusted first group of two-dimensional facial key points and the relative distance between the first projection center point and the first group of three-dimensional updated facial key points is less than or equal to the distance matching degree threshold, the initial expression signal of the adjusted first group of two-dimensional facial key points can be determined as the expression signal of the first group of two-dimensional facial key points. The computer device can perform the above operations in sequence until the expression signals of N groups of two-dimensional facial key points of the object to be processed are obtained.

[0110] Optionally, the above-mentioned determining N initial expression images of the object to be processed based on the expression signals of the N groups of two-dimensional facial key points of the object to be processed comprises: screening out two-dimensional facial key points reflecting key facial parts of the object to be processed from each group of two-dimensional facial key points of the object to be processed; and determining the N initial expression images of the object to be processed based on the two-dimensional facial key points screened from each group of two-dimensional facial key points.

[0111] It should be noted that the key facial parts of the object to be processed may include the eyes, mouth, eyebrows, etc. of the object to be processed; the two-dimensional facial key points may include two-dimensional eye key points, two-dimensional mouth key points, two-dimensional eyebrow key points, etc. Among them, the points used to reflect the eyes of the object to be processed in the plane image space may be called two-dimensional eye key points, the points used to reflect the mouth of the object to be processed in the plane image space may be called two-dimensional mouth key points, and the points used to reflect the eyebrows of the object to be processed in the plane image space may be called two-dimensional eyebrow key points.

[0112] It is understandable that when the computer device draws N initial expression images of the object to be processed based on the expression signals of N groups of two-dimensional facial key points of the object to be processed, the two-dimensional eye key points, two-dimensional mouth key points and two-dimensional eyebrow key points of the object to be processed can be screened out from each group of two-dimensional facial key points of the object to be processed. Based on the coordinates corresponding to the two-dimensional eye key points, two-dimensional mouth key points and two-dimensional eyebrow key points of the object to be processed screened out from each group of two-dimensional facial key points, the N initial expression images of the object to be processed are drawn. In this way, the expression video prediction model can be helped to focus on key facial parts (such as eyes, mouth, eyebrows), and avoid the situation where the generation of the initial expression image is affected by large changes in facial contours when the expression movements change greatly, resulting in low accuracy of the expression video data.

[0113] S206 , using the object features of the object to be processed in the object image as constraints, and generating expression video data of the object to be processed according to the N initial expression images.

[0114] In one embodiment, the above-mentioned object features of the object to be processed in the object image are used as constraints, and the expression video data of the object to be processed is generated according to the N initial expression images, including: the computer device can extract features of the N initial expression images through an image encoder to obtain image features corresponding to the N initial expression images. Through an expression video prediction model, the object features of the object to be processed in the object image are used as constraints, and the image features corresponding to the N predicted expression images of the object to be processed are generated according to the image features corresponding to the N initial expression images. The computer device can decode the image features corresponding to the N predicted expression images of the object to be processed through a video decoder to generate the expression video data of the object to be processed.

[0115] It should be noted that the image encoder may include an intrinsic shape signature (ISS) encoder, a scale-invariant feature transform (SIFT) encoder, a simple learned keypoints (SiLK) encoder, etc. The expression video prediction model may be obtained by adjusting the parameters in the initial expression video prediction model using the sample object image and M sample expression images of the sample object as training samples. The initial expression video prediction model may be initialized based on the parameters of a stable video diffusion model. The video decoder may include a VAE decoder, an open multimedia alliance (AOMedia video codec1, AV1) decoder, etc.

[0116] It can be understood that the computer can input N initial expression images into the image encoder to obtain the image features corresponding to the N initial expression images. The object features of the object to be processed in the object image and the image features corresponding to the N initial expression images are input into the expression video prediction model, and the image features corresponding to the N initial expression images are smoothed while taking the object features of the object to be processed in the object image as constraints to ensure that the identity of the object to be processed is not leaked, so as to obtain the image features corresponding to the N predicted expression images of the object to be processed. Among them, the object features of the object to be processed in each predicted expression image are consistent with the object features of the object to be processed in the object image. The image features corresponding to the N predicted expression images are input into the video decoder, and the image features corresponding to the N predicted expression images are decoded to obtain the expression video data of the object to be processed.

[0117] Optionally, the above-mentioned expression video prediction model, with the object features of the above-mentioned object to be processed in the above-mentioned object image as constraints, generates the image features corresponding to the above-mentioned N predicted expression images of the object to be processed according to the image features corresponding to the above-mentioned N initial expression images, including: the computer device can generate image prompt words including the object features of the above-mentioned object to be processed in the above-mentioned object image. The computer device can add noise to the image features corresponding to the above-mentioned N initial expression images to obtain the noise image features corresponding to the above-mentioned N initial expression images. The computer device can denoise the noise image features corresponding to the above-mentioned N initial expression images according to the above-mentioned image prompt words through the expression video prediction model to obtain the image features corresponding to the above-mentioned N predicted expression images of the object to be processed.

[0118] It should be noted that the number of image prompt words may be one or more, and one image prompt word may include multiple object features of the object to be processed, or one image prompt word may include one object feature of the object to be processed.

[0119] In one embodiment, the computer device may split the object image of the object to be processed into multiple sub-images, extract object features of the object to be processed from each sub-image; and generate multiple image prompt words including the corresponding object features based on the object features corresponding to each sub-image.

[0120] In one embodiment, after extracting the object features of the object to be processed from each sub-image, the computer device may perform inductive processing on multiple image prompt words including the corresponding sub-object features to obtain image prompt words including the object features of the object to be processed. The inductive processing may refer to merging multiple identical or similar object features into one object feature. Similar object features may refer to object feature similarity between two object features being greater than an object feature similarity threshold. The object feature similarity threshold may be manually set, or the object feature similarity threshold may also be a default value in the computer device.

[0121] It should be noted that adding noise may refer to adding a certain amount of random noise to the image features corresponding to the N initial expression images, and denoising may refer to removing or reducing noise from the noise image features corresponding to the N initial expression images. The determination method of random noise may include probability density function, autocorrelation function, power spectrum density, etc. The above-mentioned addition may refer to simple summation, weighted summation, etc.

[0122] It is understandable that the computer device can extract the object features of the object to be processed in the object image from the object image, and generate an image prompt word including the object features of the object to be processed. The computer device can perform summation processing on the random noise and the image features corresponding to the N initial expression images, respectively, to generate the noise image features corresponding to the N initial expression images. The image prompt words and the noise image features corresponding to the N initial expression images are input into the expression video prediction model, and based on the object features in the image prompt words, the noise in the noise image features corresponding to the N initial expression images are gradually removed to obtain the image features corresponding to the N predicted expression images of the object to be processed. By inputting the image prompt words and using the image prompt words instead of the text prompt words in the stable video diffusion model, the model can reuse the pre-trained weights of the stable video diffusion model to improve the prediction accuracy of the initial expression image. The above-mentioned summation processing can refer to simple summation processing, weighted summation processing, and the like.

[0123] Optionally, the above-mentioned expression video prediction model is used to denoise the noise image features corresponding to the N initial expression images respectively according to the image prompt words to obtain the image features corresponding to the N predicted expression images of the object to be processed, including: weighted processing is performed on the noise image features corresponding to the N initial expression images of the object to be processed respectively through the self-attention layer of the expression video prediction model to obtain the weighted noise image features corresponding to the N initial expression images of the object to be processed respectively; denoising is performed on the weighted noise image features corresponding to the N initial expression images of the object to be processed respectively according to the image prompt words through the cross-attention layer of the expression video prediction model to obtain the denoised image features corresponding to the N initial expression images of the object to be processed respectively; fine-tuning is performed on the denoised image features corresponding to the N initial expression images of the object to be processed respectively according to the appearance times corresponding to the N initial expression images through the timing layer of the expression video prediction model to obtain the image features corresponding to the N predicted expression images of the object to be processed respectively.

[0124] It should be noted that there are multiple noise image features corresponding to each initial expression image, and each initial expression image includes multiple pixels, wherein one pixel corresponds to one noise image feature.

[0125] It should be noted that the expression video prediction model may include a self-attention layer, a cross-attention layer and a timing layer. The self-attention layer can be used to screen out important noise image features from multiple noise image features in each initial expression image, and the cross-attention layer can be used to sum the object features and important noise image features in the image prompt words. The timing layer can be used to perform fine-tuning based on the denoised image features corresponding to the N initial expression images, so that the expression movements corresponding to the N predicted expression images are smoother. Among them, fine-tuning may refer to adjusting the denoised image features corresponding to the N initial expression images within a preset feature adjustment range. The preset feature adjustment range may be a default value in the expression video prediction model, or the preset feature adjustment range may be manually set. The above-mentioned summation processing may include weighted summation processing, addition processing, and the like.

[0126] It can be understood that the computer device can input the noise image features corresponding to the N initial expression images into the self-attention layer of the expression video prediction model, calculate the feature similarity between each two noise image features in each initial expression image, and filter out the noise image features whose feature similarity is greater than the feature similarity threshold (i.e., important noise image features) from the noise image features corresponding to each initial expression image. The computer device can sum the filtered noise image features and the first feature weight; sum the noise image features other than the filtered noise image features in the noise image features corresponding to each initial expression image and the second feature weight to obtain the weighted noise image features corresponding to the N initial expression images. Among them, the first feature weight is greater than the second feature weight.

[0127] Furthermore, the computer device can input the weighted noise image features corresponding to the N initial expression images into the cross-attention layer of the expression video prediction model, and remove the noise in the weighted noise image features corresponding to the N initial expression images based on the object features in the image prompt words, so as to obtain the denoised image features corresponding to the N initial expression images. The computer device can also input the denoised image features corresponding to the N initial expression images into the timing layer of the expression video prediction model, obtain the appearance time corresponding to the N initial expression images, and fine-tune the denoised image features corresponding to the k-1th initial expression image and the denoised image features corresponding to the k+1th initial expression image, so that the expression movement corresponding to the kth predicted expression image is smoother; repeating the above operations in sequence can obtain the image features corresponding to the N predicted expression images of the object to be processed.

[0128] Please refer to Figure 4 , Figure 4 A schematic diagram of a video generation method provided in an embodiment of the present application Figure 3 .in, Figure 4 The system includes an object image 401 of an object to be processed, N initial expression images 404, a prompt word generator 402, an image encoder 405, an expression video prediction model 408 and a video decoder 410.

[0129] It should be noted that, with a self-attention layer, a cross-attention layer and a timing layer as a set of model layer structures, the expression video prediction model 408 includes multiple sets of model layer structures, wherein, taking a set of model layer structures as an example, the expression video prediction model 408 may include a self-attention layer 481, a cross-attention layer 482 and a timing layer 483. The prompt word generator 402 may be a contrastive language-image pre-training (CLIP) model.

[0130] Specifically, the computer device may input the object image 401 into the prompt word generator 402 to obtain an image prompt word 403 including the object features of the object to be processed, wherein the image prompt word 403 may include multiple object features. Figure 4 In the example, the image prompt word 403 may include three object features, and the three object features are respectively a first object feature, a second object feature, and a third object feature. For example, the first object feature may be the face contour of the object to be processed, the second object feature may be the ear shape of the object to be processed, and the third object feature may be the eye shape of the object to be processed.

[0131] Furthermore, after obtaining N initial expression images 404, the computer device can input the N initial expression images 404 into the image encoder 405 to obtain image features 406 corresponding to the N initial expression images respectively, and sum the image features 406 corresponding to the N initial expression images respectively with random noise 407 to obtain noise image features corresponding to the N initial expression images respectively.

[0132] Further, the computer device can input the noise image features corresponding to the N initial expression images into the expression video prediction model 408, and input the image prompt word 403 into the cross attention layer 482. Based on the self-attention layer 481, the cross attention layer 482 and the timing layer 483, the noise image features corresponding to the N initial expression images are denoised to obtain the image features 409 corresponding to the N predicted expression images of the object to be processed. The computer device can input the image features 409 corresponding to the N predicted expression images into the video decoder 410 to obtain N predicted expression images 411, and the expression video data of the object to be processed can be obtained based on the N predicted expression images 411.

[0133] Optionally, the training process of the expression video prediction model and the image encoder may include the following steps: the computer device may obtain a sample object image of the sample object and M sample expression images from the original video data; M is an integer greater than 1; the expression signals of M groups of two-dimensional facial key points corresponding to the M sample expression images are encoded by the initial image encoder to obtain the image features corresponding to the M sample expression images. The initial expression video prediction model is used to generate the image features corresponding to the M predicted expression images of the sample object according to the image features corresponding to the M sample expression images, with the object features of the sample object in the sample object image as constraints. The parameters in the initial expression video prediction model and the parameters in the initial image encoder are adjusted according to the M sample expression images of the sample object, the image features corresponding to the M sample expression images, and the image features corresponding to the M predicted expression images, to obtain the expression video prediction model and the image encoder.

[0134] It should be noted that the original video data may refer to sample video data used to train the initial expression video prediction model, wherein the original video data includes sample objects. The M sample expression images may be obtained by continuously extracting frames from the original video data. At this time, the sample object image may be any one of the M sample expression images.

[0135] In one embodiment, sample sub-video data can be intercepted from the original video data, and a continuous frame extraction operation can be performed on the sample sub-video data to obtain M sample expression images. At this time, the sample object image can be any image in the M sample expression images, or the computer device can also extract a frame of video image as the sample object image from the video data other than the sample sub-video data in the original video data.

[0136] It should be noted that the initial expression video prediction model may also include an initial self-attention layer, an initial cross-attention layer, and an initial timing layer. The parameters in the initial self-attention layer, the parameters in the initial cross-attention layer, and the parameters in the initial timing layer may all be initialized based on the parameters of the stable video diffusion model. Therefore, the parameters in the initial expression video prediction model may refer to the parameters in the initial self-attention layer, the parameters in the initial cross-attention layer, and the parameters in the initial timing layer, etc. The parameters in the initial image encoder may refer to network architecture parameters, regularization parameters, etc.

[0137] It can be understood that the computer device can obtain the original video data, perform continuous frame extraction operation on the original video data, and obtain M sample expression images; and extract a frame of video image from the original video data as the sample object image of the sample object. The computer device can extract the expression signals corresponding to M groups of two-dimensional facial key points from the M sample expression images, and input the expression signals of the M groups of two-dimensional facial key points corresponding to the M sample expression images into the initial image encoder. The computer device can obtain the image features corresponding to the M sample expression images based on the initial image encoder. Further, the object features of the sample object are obtained from the sample object image, and the image features corresponding to the M sample expression images and the object features of the sample object are input into the initial expression video prediction model. The object features of the sample object in the sample object image are used as constraints to ensure that the identity of the object to be processed is not leaked, and the image features corresponding to the M initial expression images are smoothed to obtain the image features corresponding to the M predicted expression images of the sample object. Among them, the object features of the sample object in each predicted expression image are consistent with the object features of the sample object in the sample object image.

[0138] Furthermore, the computer device can calculate the total loss (i.e., prediction loss) based on the M sample expression images of the sample object, the image features corresponding to the M sample expression images, and the image features corresponding to the M predicted expression images, and adjust the parameters in the initial expression video prediction model and the parameters in the initial image encoder based on the total loss to obtain the expression video prediction model and the image encoder. By training and adjusting the expression video prediction model and the image encoder, the image prediction accuracy of the expression video prediction model can be improved, the feature extraction accuracy of the image encoder can be improved, and thus the generation effect of the accuracy of the expression video data can be improved.

[0139] It should be noted that adjusting the parameters in the initial expression video prediction model and the parameters in the above-mentioned initial image encoder may include changing the parameters in the initial self-attention layer, changing the parameters in the initial cross-attention layer, changing the parameters in the initial timing layer, changing the network architecture parameters in the initial image encoder, etc.

[0140] Please refer to Figure 5 , Figure 5 A flowchart of a training method for an initial expression video prediction model provided in an embodiment of the present application is provided. Figure 5 It includes a sample object image 501 of a sample object, a sample sub-video 504, a prompt word generator 502, a video encoder 505, an initial image encoder 507, an initial expression video prediction model 511 and a video decoder 513.

[0141] It should be noted that, a group of model layer structures consists of an initial self-attention layer, an initial cross-attention layer and an initial timing layer. The initial expression video prediction model 511 includes multiple groups of model layer structures, wherein a group of model layer structures is taken as an example for explanation. The initial expression video prediction model 511 includes an initial self-attention layer 5111, an initial cross-attention layer 5112 and an initial timing layer 5113.

[0142] Specifically, the computer device can obtain the original video data, and extract the sample object image 501 and the sample sub-video 504 of the sample object from the original video data. The sample sub-video 504 is continuously framed to obtain M sample expression images. The computer device can input the sample object image 501 into the prompt word generator 502 to obtain the image prompt word 503 including the object features of the sample object, wherein the image prompt word 503 may include multiple object features. Figure 5 In the example, the image prompt word 503 may include three object features, and the three object features are the fourth object feature, the fifth object feature, and the sixth object feature. For example, the fourth object feature may be the face contour of the object to be processed, the fifth object feature may be the hair style of the object to be processed, and the sixth object feature may be the eye color of the object to be processed.

[0143] Further, the computer device can extract the expression signals of the M groups of two-dimensional facial key points corresponding to the M sample expression images from the M sample expression images, and determine the M sample initial expression images 506 based on the expression signals corresponding to the M groups of two-dimensional facial key points. The computer device can input the M sample initial expression images 506 into the initial image encoder 507 to obtain the image features 509 corresponding to the M sample initial expression images. The computer device can also input the sample sub-video 504 into the video encoder 505 to obtain the video features 508 corresponding to the sample sub-video 504; and sum the video features 508, the image features 509 corresponding to the M sample initial expression images, and the random noise 510 to obtain the model input features corresponding to the M sample initial expression images.

[0144] Further, the computer device can input the model input features corresponding to the M sample initial expression images into the initial expression video prediction model 511, and input the image prompt word 503 into the initial cross attention layer 5112; based on the initial self-attention layer 5111, the initial cross attention layer 5112 and the initial timing layer 5113, the model input features corresponding to the M sample initial expression images are denoised to obtain the image features 512 corresponding to the M predicted expression images of the sample object. The computer device can input the image features 512 corresponding to the M predicted expression images into the video decoder 513 to obtain M predicted expression images 514. Based on the M predicted expression images 514, the sample expression video data of the sample object can be obtained.

[0145] Furthermore, the computer device can calculate the recognition loss based on the M sample expression images, the image features corresponding to the M sample expression images, and the image features 512 corresponding to the M predicted expression images, and adjust the parameters of the initial cross-attention layer 5112, the parameters of the initial self-attention layer 5111, the parameters of the initial timing layer 5113, and the parameters in the initial image encoder 507 based on the recognition loss to obtain the expression video prediction model and image encoder.

[0146] Optionally, the computer may also determine the facial regions corresponding to the M sample expression images respectively based on the image features corresponding to the M sample expression images respectively. The facial recognition loss may be determined based on the facial regions corresponding to the M sample expression images respectively, the image features corresponding to the M sample expression images respectively, and the image features corresponding to the M predicted expression images respectively. The computer may also determine the expression image generation loss based on the image features corresponding to the M sample expression images respectively and the image features corresponding to the M predicted expression images respectively. Furthermore, the computer device may adjust the parameters in the initial expression video prediction model and the parameters in the initial image encoder according to the facial recognition loss and the expression image generation loss to obtain the expression video prediction model and the image encoder.

[0147] It should be noted that the facial region may refer to the region enclosed by connecting the facial contour key points of the sample object corresponding to each sample expression image end to end. The facial recognition loss may be used to reflect the recognition loss of the facial region of the sample object; the expression image generation loss may be used to reflect the recognition loss of the entire expression image.

[0148] It can be understood that the computer device can extract the two-dimensional facial key points corresponding to the M sample expression images from the M sample expression images, and filter out the two-dimensional facial contour key points of the sample object. The two-dimensional facial contour key points of the sample object in each sample expression image are connected end to end, and the area enclosed in each sample expression image is the facial area corresponding to each sample expression image. Based on the facial areas corresponding to the M sample expression images, the image features of the facial areas corresponding to the M sample expression images are filtered out from the image features corresponding to the M sample expression images. The computer device can filter out the image features of the facial areas corresponding to the M predicted expression images from the image features corresponding to the M predicted expression images; the facial loss function is used to calculate the facial recognition loss of the sample object according to the image features of the facial areas corresponding to the M sample expression images and the image features of the facial areas corresponding to the M predicted expression images. The computer device can input the image features corresponding to the M sample expression images and the image features corresponding to the M predicted expression images into the training loss function corresponding to the initial expression video prediction model to calculate the expression image generation loss. The computer device can sum the facial recognition loss and the expression image generation loss to obtain a total loss, and based on the total loss, adjust the parameters in the initial expression video prediction model and the parameters in the initial image encoder to obtain the expression video prediction model and the image encoder. The summation process may include addition processing, weighted summation processing, etc. By calculating the facial recognition loss, the initial expression video prediction model is prompted to pay more attention to the prediction effect of the facial area of ​​the sample object during the training stage, improve the fluency of the expression action corresponding to the predicted expression image, keep the object features in the predicted expression image consistent with the object features in the object image, and avoid the identity leakage of the object to be processed.

[0149] It should be noted that the facial loss function may include a mean square error loss function, a cross entropy loss function, an adversarial loss function, etc. The training loss function corresponding to the initial expression video prediction model may include a mean absolute error loss function, a feature loss function, a structural similarity index loss function, etc.

[0150] Optionally, the facial recognition loss is determined according to the facial regions corresponding to the M sample expression images, the image features corresponding to the M sample expression images, and the image features corresponding to the M predicted expression images, including: the computer device can determine the facial mask matrices corresponding to the M sample expression images according to the facial regions corresponding to the M sample expression images. Furthermore, the computer device can multiply the facial mask matrices corresponding to the M sample expression images with the image features corresponding to the M sample expression images to obtain the facial features corresponding to the M sample expression images. Next, the facial mask matrices corresponding to the M sample expression images can be multiplied with the image features corresponding to the M predicted expression images to obtain the facial features corresponding to the M predicted expression images; the facial features corresponding to the M sample expression images and the facial features corresponding to the M predicted expression images can be processed by mean square error to obtain the facial recognition loss.

[0151] It should be noted that the facial mask matrix can be used to identify the facial region of the sample object in the sample expression image, and the facial mask matrix includes multiple values, the number of values ​​in the facial mask matrix is ​​consistent with the number of pixels in the sample expression image, and one value corresponds to one pixel in the sample expression image. Among them, when a certain pixel in the sample expression image is located in the facial region of the sample object in the sample expression image, the value in the facial mask matrix corresponding to the pixel is 1; when a certain pixel in the sample expression image is not located in the facial region of the sample object in the sample expression image, the value in the facial mask matrix corresponding to the pixel is 0.

[0152] It can be understood that the computer device can determine the facial mask matrices corresponding to the M sample expression images according to the facial regions corresponding to the M sample expression images according to the above-mentioned method for determining the facial mask matrix. The facial mask matrix corresponding to the first sample expression image and the image features corresponding to the first sample expression image are multiplied to obtain the facial features corresponding to the first sample expression image; the above-mentioned operation steps are performed in sequence until the facial features corresponding to the M sample expression images are obtained. The facial mask matrix corresponding to the first sample expression image and the image features corresponding to the first predicted expression image are multiplied to obtain the facial features corresponding to the first predicted expression image; the above-mentioned operation steps are performed in sequence until the facial features corresponding to the M predicted expression images are obtained. The facial features corresponding to the M sample expression images and the facial features corresponding to the M predicted expression images are processed by mean square error to obtain the facial recognition loss. By calculating the facial recognition loss, the expression video prediction model is prompted to pay more attention to the prediction effect of the facial area during the training stage, thereby improving the fluency of the expression movements corresponding to the predicted expression images.

[0153] For example, a computer device obtains facial mask matrices corresponding to M sample expression images, image features corresponding to M sample expression images, and image features corresponding to M predicted expression images, and inputs the facial mask matrices corresponding to M sample expression images, image features corresponding to M sample expression images, and image features corresponding to M predicted expression images into formula (1), and the facial recognition loss can be obtained:

[0154] (1)

[0155] Among them, L in formula (1) FE is the face recognition loss, E in formula (1) represents the expectation, M in formula (1) is the face mask matrix, Z in formula (1) is the image feature corresponding to the sample expression image, and To predict the image features corresponding to the expression image. By transforming formula (1), we can get formula (2):

[0156] (2)

[0157] Among them, L in formula (2) FE is the face recognition loss, M in formula (2) i is the facial mask matrix corresponding to the i-th sample expression image, and Z in formula (2) i is the image feature corresponding to the i-th sample expression image, and the is the image feature corresponding to the i-th predicted expression image, and m in formula (2) is the number of M sample expression images. The computer device can input the facial mask matrix corresponding to the M sample expression images, the image features corresponding to the M sample expression images, and the image features corresponding to the M predicted expression images into formula (2), and the facial recognition loss can be obtained.

[0158] In one embodiment, the parameters in the initial expression video prediction model and the parameters in the initial image encoder are adjusted according to the facial recognition loss and the expression image generation loss to obtain the expression video prediction model and the image encoder, including: the computer device can sum the facial recognition loss and the expression image generation loss to obtain the total loss. When the total loss is greater than the loss threshold, the computer device can adjust the parameters in the initial expression video prediction model and the parameters in the initial image encoder according to the total loss. When the total loss obtained based on the adjusted initial label image prediction model and the adjusted initial image encoder is less than or equal to the loss threshold, the adjusted initial expression video prediction model is determined as the expression video prediction model. Therefore, the computer device can determine the adjusted initial image encoder as the image encoder. Among them, the summation process can include addition processing, weighted summation processing, etc.

[0159] It should be noted that the loss threshold may be a manually set value or a default value in a computer device.

[0160] It can be understood that the computer device can add the facial recognition loss to the expression image generation loss to obtain the total loss. When the total loss is greater than the loss threshold, it indicates that the prediction effect of the above-mentioned initial expression video prediction model is poor and the feature generation effect of the above-mentioned initial image encoder is poor. Therefore, the expression action corresponding to the predicted expression image is inconsistent with the expression action corresponding to the sample expression image. The computer device can adjust the parameters in the initial expression video prediction model and the parameters in the above-mentioned initial image encoder to obtain an adjusted initial image encoder and an adjusted initial image encoder. The computer device can train the adjusted initial image encoder and the adjusted initial image encoder based on training data (such as sample object images and sample expression images of sample objects). Until the total loss obtained based on the adjusted initial label image prediction model and the adjusted initial image encoder is less than or equal to the loss threshold, it indicates that the prediction effect of the above-mentioned initial expression video prediction model is good and the feature generation effect of the above-mentioned initial image encoder is good. Therefore, the expression action corresponding to the predicted expression image is consistent with the expression action corresponding to the sample expression image. The computer device can determine the adjusted initial expression video prediction model as the above-mentioned expression video prediction model, and can determine the adjusted initial image encoder as the image encoder.

[0161] In the present application, compared with the expression signal of the two-dimensional facial key point, the expression driving signal of the three-dimensional template facial key point can provide more information, that is, the expression driving signal of the three-dimensional template facial key point can accurately describe the expression action in the template expression image. Therefore, based on the expression migration data corresponding to each group of three-dimensional template facial key points and the original expression signal of the three-dimensional original facial key point, the expression action in N expression template images is migrated to the object image to obtain N initial expression images, so that the running characteristics of the expression action in the N initial expression images can be kept consistent with the motion characteristics of the expression action corresponding to the N template expression images, so that the expression action in the N initial expression images is kept consistent with the expression action in the corresponding template expression image. At the same time, the facial features of the object to be processed itself will not be changed, and the identity of the object to be processed is avoided. Further, the object features of the object to be processed in the object image are used as constraints, and the expression video data of the object to be processed is generated according to the N initial expression images, so that the object features of the image in the expression video data are kept consistent with the object features of the object to be processed, and the identity of the object to be processed is avoided. The accuracy of the expression video data can be improved and the generation effect is more natural.

[0162] Please refer to Figure 6 , Figure 6 The present invention provides a flow chart of a method for generating a video in a social scenario. The present invention can be applied to game scenarios, social scenarios, film and television entertainment scenarios, financial scenarios, and medical and health scenarios. In a social scenario, the video generation method can be jointly executed by a terminal and a server, and the terminal may include: Figure 1 The terminal device (such as the first terminal) shown in FIG. 1 may include a terminal device (such as the first terminal) as shown in FIG. 1 ; the server may include a terminal device (such as the first terminal) as shown in FIG. Figure 1 The server 10 shown in FIG. 10 includes a server 10 in which the expression video prediction model is stored in the server. The terminal can establish a social relationship with other terminals; other terminals may include Figure 1 The terminal device (such as the second terminal) shown in the figure. The terminal can generate expression video data (such as expression video in the social chat interface) based on the video generation method by calling the expression video prediction model stored in the server, and send the expression video data to other terminals. Therefore, the process steps of the video generation method in the social scenario are as follows:

[0163] S61. Prepare template expression video.

[0164] It is understandable that when a terminal needs to send expression video data to other terminals, the terminal can obtain the template expression video data by locally acquiring it or downloading it from the network.

[0165] S62: Prepare an object image of the object to be processed.

[0166] It can be understood that the terminal can obtain the object image of the object to be processed by locally acquiring it or downloading it from the network.

[0167] S63, acquiring expression driving signals corresponding to N groups of three-dimensional facial key points from the template expression video.

[0168] It can be understood that the terminal can send the template expression video data and the object image of the object to be processed to the server, and the server can continuously extract frames from the template expression video to obtain N template expression images. At the same time, the server can extract N groups of expression drive signals corresponding to three-dimensional facial key points from the N template expression images. Each template expression image contains expression movements; one template expression image corresponds to a group of expression drive signals of three-dimensional template facial key points.

[0169] S64, extracting three-dimensional original facial key points.

[0170] It can be understood that the server can extract the original expression signal of the three-dimensional original facial key points corresponding to the object image from the object image to be processed.

[0171] S65, obtaining N initial expression images based on the expression drive signals corresponding to the N groups of three-dimensional facial key points and the original expression signals of the three-dimensional original facial key points.

[0172] It can be understood that the server can adjust the original expression signals of the three-dimensional original facial key points based on the expression driving signals corresponding to N groups of three-dimensional facial key points, and migrate the expression actions corresponding to the N template expression images to the object image of the object to be processed, so as to obtain N initial expression images.

[0173] S66, extracting image prompt words.

[0174] It can be understood that the server can extract the object features of the object to be processed from the object image of the object to be processed, and generate an image prompt word including the object features of the object to be processed.

[0175] S67, inputting N initial expression images and image prompt words into the expression video prediction model.

[0176] It can be understood that the server can extract features from the initial expression images to obtain image features corresponding to the N initial expression images; add noise to the image features corresponding to the N initial expression images to obtain noise image features corresponding to the N initial expression images. The server can input the noise image features corresponding to the N initial expression images and the image prompt words into the expression video prediction model.

[0177] S68, using the predicted expression image obtained based on the expression video prediction model, and obtaining expression video data through a video decoder.

[0178] It can be understood that the server can obtain the image features corresponding to N predicted expression images through the expression video prediction model, and input the image features corresponding to the N predicted expression images into the video decoder to obtain the expression video data. After the server generates the expression video data, the expression video data can be sent to the terminal, and the terminal can send the expression video data to other terminals, which can enhance the fun of user social interaction.

[0179] In the present application, compared with the expression signals of the two-dimensional facial key points, the expression driving signals of the three-dimensional template facial key points can provide more information, that is, the expression driving signals of the three-dimensional template facial key points can accurately describe the expression movements in the template expression image. Therefore, through N groups of expression driving signals of the three-dimensional template facial key points and the original expression signals of the three-dimensional original facial key points, the expression movements in the N template expression images can be accurately transferred to the object image, and N initial expression images can be obtained, thereby improving the migration accuracy of the expression movements. Furthermore, with the object features of the object to be processed in the object image as the constraint condition, the expression video data of the object to be processed is generated according to the N initial expression images, so that the object features of the image in the expression video data are consistent with the object features of the object to be processed, avoiding the identity leakage of the object to be processed, and improving the accuracy of the expression video data, making the generation effect more natural.

[0180] See also Figure 7 , is a schematic diagram of the structure of a video generation device provided in an embodiment of the present application. Figure 7 As shown, the video generating device may include:

[0181] The acquisition module 711 is used to acquire N template expression images and an object image of an object to be processed; N is an integer greater than 1, and each template expression image includes an expression action;

[0182] The extraction module 712 is used to extract N groups of expression driving signals corresponding to the three-dimensional template facial key points from the above-mentioned N template expression images, and extract the original expression signals corresponding to the three-dimensional original facial key points of the above-mentioned object to be processed from the above-mentioned object image; one template expression image corresponds to one group of expression driving signals of the three-dimensional template facial key points;

[0183] A migration module 713 is used to migrate the expression actions corresponding to the N template expression images to the object image according to the expression drive signals corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, so as to obtain the N initial expression images of the object to be processed;

[0184] The generating module 714 is used to generate the expression video data of the object to be processed based on the N initial expression images, taking the object features of the object to be processed in the object image as constraints.

[0185] Optionally, the migration module 713 may also be used to perform the following operations:

[0186] From the N groups of 3D template facial key points, obtain the 3D reference facial key points corresponding to each group of 3D template facial key points;

[0187] Determine the expression migration data corresponding to each group of facial key points of the three-dimensional template according to the expression drive signal of each group of facial key points of the three-dimensional template and the expression drive signal of the corresponding three-dimensional reference facial key points;

[0188] According to the expression migration data corresponding to the above N groups of three-dimensional template facial key points and the original expression signals corresponding to the above three-dimensional original facial key points, the expression movements corresponding to the above N template expression images are migrated to the above object images to obtain the N initial expression images of the above object to be processed.

[0189] Optionally, the expression driving signal includes a translation matrix, a rotation matrix and facial shape parameters;

[0190] The migration module 713 can also be used to perform the following operations:

[0191] Performing a difference process on the translation matrix of each group of facial key points of the three-dimensional template and the translation matrix of the corresponding three-dimensional reference facial key points to obtain a translation deviation matrix corresponding to each group of facial key points of the three-dimensional template;

[0192] Performing a difference process on the facial shape parameters of each set of facial key points of the three-dimensional template and the facial shape parameters of the corresponding three-dimensional reference facial key points to obtain the facial shape deviations corresponding to each set of facial key points of the three-dimensional template;

[0193] The facial shape deviation, translation deviation matrix and rotation matrix corresponding to each group of facial key points of the three-dimensional template are determined as the expression migration data corresponding to each group of facial key points of the three-dimensional template.

[0194] Optionally, the migration module 713 may also be used to perform the following operations:

[0195] According to the expression migration data corresponding to the N groups of three-dimensional template facial key points, the original expression signals corresponding to the three-dimensional original facial key points are updated to obtain the updated expression signals corresponding to the N groups of three-dimensional updated facial key points of the object to be processed; the updated expression signals of one group of three-dimensional updated facial key points correspond to the expression actions in a template expression image;

[0196] Projecting the updated expression signals corresponding to the N groups of three-dimensional updated facial key points to obtain the expression signals corresponding to the N groups of two-dimensional facial key points of the object to be processed;

[0197] According to the expression signals respectively corresponding to the N groups of two-dimensional facial key points of the object to be processed, N initial expression images of the object to be processed are determined.

[0198] Optionally, the three-dimensional reference facial key points corresponding to the N groups of three-dimensional template facial key points are all the first group of three-dimensional template facial key points, and the first group of three-dimensional template facial key points are extracted from the template expression image that appears earliest among the N template expression images; the expression migration data includes facial shape deviation, translation deviation matrix and rotation matrix; the original expression signal includes translation matrix and facial shape parameters;

[0199] The migration module 713 can also be used to perform the following operations:

[0200] The translation deviation matrix corresponding to the i-th group of three-dimensional template facial key points and the translation matrix corresponding to the three-dimensional original facial key points are summed to obtain the updated translation matrix corresponding to the i-th group of three-dimensional updated facial key points of the object to be processed; i is a positive integer less than or equal to N;

[0201] The facial shape deviations corresponding to the i-th group of three-dimensional template facial key points and the facial shape parameters corresponding to the three-dimensional original facial key points are summed to obtain updated facial shape parameters corresponding to the i-th group of three-dimensional updated facial key points;

[0202] Determine the rotation matrix corresponding to the i-th group of three-dimensional template facial key points as the update rotation matrix corresponding to the i-th group of three-dimensional update facial key points;

[0203] The updated translation matrix, updated facial shape parameters and updated rotation matrix corresponding to the i-th group of three-dimensional updated facial key points of the object to be processed are determined as the updated expression signal corresponding to the i-th group of three-dimensional updated facial key points.

[0204] Optionally, the three-dimensional reference facial key points corresponding to the first group of three-dimensional template facial key points are the above-mentioned first group of three-dimensional template facial key points, the three-dimensional reference facial key points corresponding to the j-th group of three-dimensional template facial key points are the j-1-th group of three-dimensional template facial key points, the above-mentioned first group of three-dimensional template facial key points are extracted from the template expression image with the earliest appearance time among the above-mentioned N template expression images, the appearance time of the template expression image corresponding to the j-1-th group of three-dimensional template facial key points is before the appearance time of the template expression image corresponding to the j-th group of three-dimensional template facial key points, and j is an integer greater than 1 and less than or equal to N;

[0205] The migration module 713 can also be used to perform the following operations:

[0206] According to the expression migration data corresponding to the first group of three-dimensional template facial key points, the updated expression signals corresponding to the three-dimensional original facial key points are updated to obtain the updated expression signals corresponding to the first group of three-dimensional updated facial key points of the object to be processed;

[0207] According to the expression migration data corresponding to the jth group of three-dimensional template facial key points, the updated expression signal corresponding to the j-1th group of three-dimensional updated facial key points is adjusted to obtain the updated expression signal corresponding to the jth group of three-dimensional updated facial key points of the object to be processed.

[0208] Optionally, the migration module 713 may also be used to perform the following operations:

[0209] Extracting the three-dimensional eye key points from the kth group of three-dimensional updated facial key points; k is a positive integer less than or equal to N;

[0210] Determine the kth projection center point based on the coordinates of the three-dimensional eye key points in the kth group of three-dimensional updated facial key points;

[0211] Taking the relative distances between the k-th projection center point and the k-th group of three-dimensional updated facial key points as restriction conditions, projection processing is performed on the updated expression signals corresponding to the k-th group of three-dimensional updated facial key points to obtain the expression signals of the k-th group of two-dimensional facial key points of the object to be processed.

[0212] Optionally, the updated expression signal includes facial shape parameters, a translation matrix and a rotation matrix;

[0213] The migration module 713 can also be used to perform the following operations:

[0214] The translation matrix corresponding to the k-th group of three-dimensional updated facial key points is multiplied by the rotation matrix corresponding to the k-th group of three-dimensional updated facial key points to obtain the transformation matrix corresponding to the k-th group of three-dimensional updated facial key points;

[0215] According to the transformation matrix corresponding to the kth group of three-dimensional updated facial key points, mapping processing is performed on the facial shape parameters corresponding to the kth group of three-dimensional updated facial key points to obtain the initial expression signal of the kth group of two-dimensional facial key points of the object to be processed;

[0216] According to the relative distances between the kth projection center point and the kth group of three-dimensional updated facial key points, the initial expression signal of the kth group of two-dimensional facial key points of the object to be processed is adjusted to obtain the expression signal of the kth group of two-dimensional facial key points of the object to be processed.

[0217] Optionally, the migration module 713 may also be used to perform the following operations:

[0218] From each group of two-dimensional facial key points of the object to be processed, two-dimensional facial key points reflecting the facial key parts of the object to be processed are selected;

[0219] According to the two-dimensional facial key points screened from each group of two-dimensional facial key points, the N initial expression images of the object to be processed are determined.

[0220] Optionally, the generating module 714 may also be used to perform the following operations:

[0221] By using an image encoder, feature extraction is performed on the N initial expression images to obtain image features corresponding to the N initial expression images respectively;

[0222] By using the expression video prediction model, taking the object features of the object to be processed in the object image as constraints, and generating the image features corresponding to the N predicted expression images of the object to be processed respectively according to the image features corresponding to the N initial expression images respectively;

[0223] The image features corresponding to the N predicted expression images of the object to be processed are decoded by a video decoder to generate expression video data of the object to be processed.

[0224] Optionally, the generating module 714 may also be used to perform the following operations:

[0225] generating an image prompt word including the object features of the object to be processed in the object image;

[0226] Noising the image features corresponding to the N initial expression images to obtain noise image features corresponding to the N initial expression images;

[0227] Through the expression video prediction model, according to the above image prompt words, the noise image features corresponding to the above N initial expression images are denoised to obtain the image features corresponding to the above N predicted expression images of the object to be processed.

[0228] Optionally, the generating module 714 may also be used to perform the following operations:

[0229] Through the self-attention layer of the expression video prediction model, weighted processing is performed on the noise image features corresponding to the N initial expression images of the object to be processed, so as to obtain weighted noise image features corresponding to the N initial expression images of the object to be processed;

[0230] Through the cross attention layer of the expression video prediction model, according to the image prompt words, the weighted noise image features corresponding to the N initial expression images of the object to be processed are denoised to obtain the denoised image features corresponding to the N initial expression images of the object to be processed;

[0231] Through the timing layer of the above-mentioned expression video prediction model, the denoised image features corresponding to the N initial expression images of the above-mentioned object to be processed are fine-tuned according to the appearance times corresponding to the above-mentioned N initial expression images, so as to obtain the image features corresponding to the N predicted expression images of the above-mentioned object to be processed.

[0232] Optionally, the generating module 714 may also be used to perform the following operations:

[0233] By using an initial image encoder, encoding the expression signals of the M groups of two-dimensional facial key points corresponding to the M sample expression images, respectively, to obtain image features corresponding to the M sample expression images, respectively;

[0234] By using the initial expression video prediction model, taking the object features of the sample object in the sample object image as constraints, and according to the image features corresponding to the M sample expression images, respectively, generating image features corresponding to the M predicted expression images of the sample object;

[0235] According to the M sample expression images of the sample object, the image features corresponding to the M sample expression images respectively, and the image features corresponding to the M predicted expression images respectively, the parameters in the initial expression video prediction model and the parameters in the initial image encoder are adjusted to obtain the expression video prediction model and the image encoder.

[0236] Optionally, the generating module 714 may also be used to perform the following operations: determining the facial regions corresponding to the M sample expression images respectively according to the image features corresponding to the M sample expression images respectively;

[0237] Determine the facial recognition loss according to the facial regions respectively corresponding to the M sample expression images, the image features respectively corresponding to the M sample expression images, and the image features respectively corresponding to the M predicted expression images;

[0238] Determine the expression image generation loss according to the image features corresponding to the M sample expression images and the image features corresponding to the M predicted expression images;

[0239] According to the facial recognition loss and the expression image generation loss, the parameters in the initial expression video prediction model and the parameters in the initial image encoder are adjusted to obtain the expression video prediction model and the image encoder.

[0240] Optionally, the generating module 714 may also be used to perform the following operations:

[0241] Determine facial mask matrices corresponding to the M sample expression images, respectively, according to the facial regions corresponding to the M sample expression images;

[0242] Multiplying the facial mask matrices corresponding to the M sample expression images and the image features corresponding to the M sample expression images to obtain the facial features corresponding to the M sample expression images;

[0243] Multiplying the facial mask matrices corresponding to the M sample expression images and the image features corresponding to the M predicted expression images to obtain the facial features corresponding to the M predicted expression images;

[0244] The facial features corresponding to the M sample expression images and the facial features corresponding to the M predicted expression images are processed by mean square error to obtain the facial recognition loss.

[0245] Optionally, the generating module 714 may also be used to perform the following operations:

[0246] The above-mentioned face recognition loss and the above-mentioned expression image generation loss are summed to obtain the total loss;

[0247] When the total loss is greater than the loss threshold, adjusting the parameters in the initial expression video prediction model and the parameters in the initial image encoder according to the total loss;

[0248] When the total loss obtained based on the adjusted initial label image prediction model and the adjusted initial image encoder is less than or equal to the above-mentioned loss threshold, the adjusted initial expression video prediction model is determined as the above-mentioned expression video prediction model, and the adjusted initial image encoder is determined as the above-mentioned image encoder.

[0249] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0250] In the present application, compared with the expression signals of the two-dimensional facial key points, the expression driving signals of the three-dimensional template facial key points can provide more information, that is, the expression driving signals of the three-dimensional template facial key points can accurately describe the expression movements in the template expression image. Therefore, through N groups of expression driving signals of the three-dimensional template facial key points and the original expression signals of the three-dimensional original facial key points, the expression movements in N template expression images can be accurately transferred to the object image, and N initial expression images can be obtained, thereby improving the migration accuracy of the expression movements. Furthermore, taking the object features of the object to be processed in the object image as constraints, the expression video data of the object to be processed is generated according to the N initial expression images, so that the object features of the image in the expression video data are consistent with the object features of the object to be processed, avoiding the identity leakage of the object to be processed, and improving the accuracy of the expression video data, making the generation effect more natural.

[0251] See also Figure 8 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 8 As shown, the above-mentioned computer device 800 may refer to a server or a terminal, including: a processor 801, a network interface 804 and a memory 805. In addition, the above-mentioned computer device 800 may also include: a user interface 803, and at least one communication bus 802. Among them, the communication bus 802 is used to realize the connection and communication between these components. Among them, in some embodiments, the user interface 803 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 803 may also include a standard wired interface and a wireless interface. The network interface 804 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 805 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile Memory), such as at least one disk storage. The memory 805 may optionally also be at least one storage device away from the aforementioned processor 801. As Figure 8 As shown, the memory 805 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer program.

[0252] exist Figure 8 In the computer device 800 shown, the network interface 804 can provide a network communication function; the user interface 803 is mainly used to provide an input interface; and the processor 801 can be used to call a computer program stored in the memory 805 to execute:

[0253] Obtain N template expression images and an object image of an object to be processed; N is an integer greater than 1, and each template expression image includes an expression action;

[0254] Extracting N groups of expression drive signals corresponding to the three-dimensional template facial key points from the N template expression images, and extracting the original expression signals corresponding to the three-dimensional original facial key points of the object to be processed from the object image; one template expression image corresponds to one group of expression drive signals of the three-dimensional template facial key points;

[0255] According to the expression drive signals corresponding to the N groups of facial key points of the three-dimensional templates and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the N template expression images are transferred to the object images to obtain the N initial expression images of the object to be processed;

[0256] Taking the object features of the object to be processed in the object image as constraints, the expression video data of the object to be processed is generated according to the N initial expression images.

[0257] In the present application, compared with the expression signals of the two-dimensional facial key points, the expression driving signals of the three-dimensional template facial key points can provide more information, that is, the expression driving signals of the three-dimensional template facial key points can accurately describe the expression movements in the template expression image. Therefore, through N groups of expression driving signals of the three-dimensional template facial key points and the original expression signals of the three-dimensional original facial key points, the expression movements in the N template expression images can be accurately transferred to the object image, and N initial expression images can be obtained, thereby improving the migration accuracy of the expression movements. Furthermore, with the object features of the object to be processed in the object image as the constraint condition, the expression video data of the object to be processed is generated according to the N initial expression images, so that the object features of the image in the expression video data are consistent with the object features of the object to be processed, avoiding the identity leakage of the object to be processed, and improving the accuracy of the expression video data, making the generation effect more natural.

[0258] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data processing device mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the description of the data processing method in the corresponding embodiment above, so it will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0259] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on at least two computer devices at one location for execution, or executed on at least two computer devices distributed at at least two locations and interconnected through a communication network. At least two computer devices distributed at at least two locations and interconnected through a communication network may constitute a blockchain network.

[0260] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the central storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media card, SMC), a secure digital (Secure digital, SD) card, a flash card (flashcard), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the central storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0261] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish between contents in different media, rather than to describe a specific order. In addition, the terms "including" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other step units inherent to these processes, methods, devices, products, or equipment.

[0262] The data collection and processing in this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0263] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the description of the above-mentioned data processing method and decoding method in the corresponding embodiment above, so it will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the embodiment of the computer program product involved in the present application, please refer to the description of the method embodiment of the present application.

[0264] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0265] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable network connection device to generate a machine, so that the instructions executed by the processor of the computer or other programmable network connection device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable network-connected device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1These computer program instructions can also be loaded onto a computer or other programmable network connected device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0266] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video generation method, characterized in that: include: Obtain N template expression images and an object image of an object to be processed; N is an integer greater than 1, and each template expression image includes an expression action; Extracting expression drive signals corresponding to N groups of three-dimensional template facial key points from the N template expression images, and extracting original expression signals corresponding to the three-dimensional original facial key points of the object to be processed from the object image; A template expression image corresponds to a set of expression drive signals of key points of a three-dimensional template face; the expression drive signals include a translation matrix, a rotation matrix and facial shape parameters; From the N groups of 3D template facial key points, obtain the 3D reference facial key points corresponding to each group of 3D template facial key points; Performing a difference process on the translation matrix of each group of three-dimensional template facial key points and the translation matrix of the corresponding three-dimensional reference facial key points to obtain a translation deviation matrix corresponding to each group of three-dimensional template facial key points; Performing a difference process on the facial shape parameters of each group of three-dimensional template facial key points and the facial shape parameters of the corresponding three-dimensional reference facial key points to obtain facial shape deviations corresponding to each group of three-dimensional template facial key points; Determine the facial shape deviation, translation deviation matrix and rotation matrix corresponding to each group of facial key points of the three-dimensional template as the expression migration data corresponding to each group of facial key points of the three-dimensional template; According to the expression migration data corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the N template expression images are migrated to the object image to obtain N initial expression images of the object to be processed; Taking the object features of the object to be processed in the object image as constraints, expression video data of the object to be processed is generated according to the N initial expression images.

2. The method according to claim 1, characterized in that The method of migrating the facial expressions corresponding to the N sets of facial expression migration data respectively corresponding to the facial key points of the three-dimensional templates and the original facial expression signals corresponding to the three-dimensional original facial key points to obtain the N initial facial expression images of the object to be processed comprises: According to the expression migration data corresponding to the N groups of three-dimensional template facial key points, the original expression signals corresponding to the three-dimensional original facial key points are updated to obtain the updated expression signals corresponding to the N groups of three-dimensional updated facial key points of the object to be processed; the updated expression signals of one group of three-dimensional updated facial key points correspond to the expression actions in a template expression image; Performing projection processing on the updated expression signals corresponding to the N groups of three-dimensional updated facial key points, respectively, to obtain expression signals corresponding to the N groups of two-dimensional facial key points of the object to be processed, respectively; According to the expression signals respectively corresponding to the N groups of two-dimensional facial key points of the object to be processed, N initial expression images of the object to be processed are determined.

3. The method according to claim 2, characterized in that The three-dimensional reference facial key points corresponding to the N groups of three-dimensional template facial key points are all the first group of three-dimensional template facial key points, and the first group of three-dimensional template facial key points are extracted from the template expression image that appears first in the N template expression images; the expression migration data includes facial shape deviation, translation deviation matrix and rotation matrix; The original expression signal includes a translation matrix and facial shape parameters; The method of updating the original expression signals corresponding to the three-dimensional original facial key points according to the expression migration data respectively corresponding to the N groups of three-dimensional template facial key points to obtain the updated expression signals corresponding to the N groups of three-dimensional updated facial key points of the object to be processed comprises: The translation deviation matrix corresponding to the i-th group of three-dimensional template facial key points and the translation matrix corresponding to the three-dimensional original facial key points are summed to obtain the translation matrix corresponding to the i-th group of three-dimensional updated facial key points of the object to be processed; i is a positive integer less than or equal to N; The facial shape deviations corresponding to the i-th group of three-dimensional template facial key points and the facial shape parameters corresponding to the three-dimensional original facial key points are summed to obtain the facial shape parameters corresponding to the i-th group of three-dimensional updated facial key points; Determine the rotation matrix corresponding to the i-th group of three-dimensional template facial key points as the rotation matrix corresponding to the i-th group of three-dimensional updated facial key points; The translation matrix, facial shape parameters and rotation matrix corresponding to the i-th group of three-dimensional updated facial key points of the object to be processed are determined as the updated expression signal corresponding to the i-th group of three-dimensional updated facial key points.

4. The method according to claim 2, characterized in that: The three-dimensional reference facial key points corresponding to the first group of three-dimensional template facial key points are the first group of three-dimensional template facial key points, the three-dimensional reference facial key points corresponding to the j-th group of three-dimensional template facial key points are the j-1-th group of three-dimensional template facial key points, the first group of three-dimensional template facial key points are extracted from the template expression image with the earliest appearance time among the N template expression images, the appearance time of the expression template image corresponding to the j-1-th group of three-dimensional template facial key points is before the appearance time of the expression template image corresponding to the j-th group of three-dimensional template facial key points, and j is an integer greater than 1 and less than or equal to N; The method of updating the original expression signals corresponding to the three-dimensional original facial key points according to the expression migration data respectively corresponding to the N groups of three-dimensional template facial key points to obtain the updated expression signals corresponding to the N groups of three-dimensional updated facial key points of the object to be processed comprises: According to the expression migration data corresponding to the first group of three-dimensional template facial key points, the updated expression signals corresponding to the three-dimensional original facial key points are updated to obtain the updated expression signals corresponding to the first group of three-dimensional updated facial key points of the object to be processed; According to the expression migration data corresponding to the jth group of three-dimensional template facial key points, the updated expression signal corresponding to the j-1th group of three-dimensional updated facial key points is adjusted to obtain the updated expression signal corresponding to the jth group of three-dimensional updated facial key points of the object to be processed.

5. The method according to claim 2, characterized in that: The projecting process is performed on the updated expression signals corresponding to the N groups of three-dimensional updated facial key points to obtain the expression signals corresponding to the N groups of two-dimensional facial key points of the object to be processed, including: Extracting the three-dimensional eye key points from the kth group of three-dimensional updated facial key points; k is a positive integer less than or equal to N; Determine the kth projection center point based on the coordinates of the three-dimensional eye key points in the kth group of three-dimensional updated facial key points; Taking the relative distances between the kth projection center point and the kth group of three-dimensional updated facial key points as restriction conditions, projection processing is performed on the updated expression signals corresponding to the kth group of three-dimensional updated facial key points to obtain the expression signals of the kth group of two-dimensional facial key points of the object to be processed.

6. The method according to claim 5, characterized in that The updated expression signal includes facial shape parameters, a translation matrix and a rotation matrix; The method of performing projection processing on the updated expression signals corresponding to the kth group of three-dimensional updated facial key points using the relative distances between the kth projection center point and the kth group of three-dimensional updated facial key points as restriction conditions to obtain the expression signals of the kth group of two-dimensional facial key points of the object to be processed comprises: Performing product processing on the translation matrix corresponding to the kth group of three-dimensional updated facial key points and the rotation matrix corresponding to the kth group of three-dimensional updated facial key points to obtain the transformation matrix corresponding to the kth group of three-dimensional updated facial key points; According to the transformation matrix corresponding to the kth group of three-dimensional updated facial key points, mapping processing is performed on the facial shape parameters corresponding to the kth group of three-dimensional updated facial key points to obtain the initial expression signal of the kth group of two-dimensional facial key points of the object to be processed; According to the relative distances between the kth projection center point and the kth group of three-dimensional updated facial key points, the initial expression signals of the kth group of two-dimensional facial key points of the object to be processed are adjusted to obtain the expression signals of the kth group of two-dimensional facial key points of the object to be processed.

7. The method according to claim 2, characterized in that Determining N initial expression images of the object to be processed according to the expression signals of N groups of two-dimensional facial key points of the object to be processed includes: From each group of two-dimensional facial key points of the object to be processed, select two-dimensional facial key points reflecting the facial key parts of the object to be processed; According to the two-dimensional facial key points screened from each group of two-dimensional facial key points, N initial expression images of the object to be processed are determined.

8. The method according to claim 1, characterized in that The step of generating expression video data of the object to be processed based on the N initial expression images by taking the object features of the object to be processed in the object image as a constraint condition comprises: Performing feature extraction on the N initial expression images through an image encoder to obtain image features corresponding to the N initial expression images respectively; By using the expression video prediction model, taking the object features of the object to be processed in the object image as constraints, and generating image features corresponding to N predicted expression images of the object to be processed respectively according to the image features corresponding to the N initial expression images respectively; The image features respectively corresponding to the N predicted expression images of the object to be processed are decoded by a video decoder to generate expression video data of the object to be processed.

9. The method according to claim 8, characterized in that The method of using the facial expression video prediction model, taking the object features of the object to be processed in the object image as a constraint condition, and generating the image features corresponding to the N predicted facial expression images of the object to be processed respectively according to the image features corresponding to the N initial facial expression images respectively, comprises: generating an image prompt word including the object features of the object to be processed in the object image; Noising the image features respectively corresponding to the N initial expression images to obtain noise image features respectively corresponding to the N initial expression images; Through the expression video prediction model, according to the image prompt words, the noise image features corresponding to the N initial expression images are denoised to obtain the image features corresponding to the N predicted expression images of the object to be processed.

10. The method according to claim 9, characterized in that The method of denoising the noise image features respectively corresponding to the N initial expression images according to the image prompt words by using the expression video prediction model to obtain the image features respectively corresponding to the N predicted expression images of the object to be processed includes: Through the self-attention layer of the expression video prediction model, weighted processing is performed on the noise image features corresponding to the N initial expression images of the object to be processed, so as to obtain the weighted noise image features corresponding to the N initial expression images of the object to be processed; Through the cross attention layer of the expression video prediction model, according to the image prompt words, the weighted noise image features corresponding to the N initial expression images of the object to be processed are denoised to obtain the denoised image features corresponding to the N initial expression images of the object to be processed; Through the timing layer of the expression video prediction model, the denoised image features corresponding to the N initial expression images of the object to be processed are fine-tuned according to the appearance times corresponding to the N initial expression images, so as to obtain the image features corresponding to the N predicted expression images of the object to be processed.

11. The method according to claim 8, characterized in that The method further comprises: From the original video data, obtain a sample object image of the sample object and M sample expression images; M is an integer greater than 1; By using an initial image encoder, encoding expression signals of M groups of two-dimensional facial key points respectively corresponding to the M sample expression images, to obtain image features respectively corresponding to the M sample expression images; By using the initial expression video prediction model, taking the object features of the sample object in the sample object image as constraints, and according to the image features respectively corresponding to the M sample expression images, generating image features respectively corresponding to the M predicted expression images of the sample object; According to the M sample expression images of the sample object, the image features corresponding to the M sample expression images respectively, and the image features corresponding to the M predicted expression images respectively, the parameters in the initial expression video prediction model and the parameters in the initial image encoder are adjusted to obtain the expression video prediction model and the image encoder.

12. The method according to claim 11, characterized in that The method of adjusting the parameters in the initial expression video prediction model and the parameters in the initial image encoder according to the M sample expression images of the sample object, the image features respectively corresponding to the M sample expression images, and the image features respectively corresponding to the M predicted expression images, to obtain the expression video prediction model and the image encoder, comprises: Determine the facial regions corresponding to the M sample expression images respectively according to the image features corresponding to the M sample expression images respectively; Determine the facial recognition loss according to the facial regions respectively corresponding to the M sample expression images, the image features respectively corresponding to the M sample expression images, and the image features respectively corresponding to the M predicted expression images; Determining the expression image generation loss according to the image features corresponding to the M sample expression images and the image features corresponding to the M predicted expression images; According to the facial recognition loss and the expression image generation loss, the parameters in the initial expression video prediction model and the parameters in the initial image encoder are adjusted to obtain the expression video prediction model and the image encoder.

13. The method according to claim 12, characterized in that The determining of the facial recognition loss according to the facial regions respectively corresponding to the M sample expression images, the image features respectively corresponding to the M sample expression images, and the image features respectively corresponding to the M predicted expression images comprises: Determine facial mask matrices corresponding to the M sample expression images, respectively, according to facial regions corresponding to the M sample expression images; Multiplying the facial mask matrices corresponding to the M sample expression images and the image features corresponding to the M sample expression images to obtain the facial features corresponding to the M sample expression images; Multiplying the facial mask matrices corresponding to the M sample expression images and the image features corresponding to the M predicted expression images to obtain the facial features corresponding to the M predicted expression images; The facial features corresponding to the M sample expression images and the facial features corresponding to the M predicted expression images are processed by mean square error to obtain the facial recognition loss.

14. The method according to claim 12, characterized in that The method of adjusting the parameters in the initial expression video prediction model and the parameters in the initial image encoder according to the facial recognition loss and the expression image generation loss to obtain the expression video prediction model and the image encoder comprises: The facial recognition loss and the expression image generation loss are summed to obtain a total loss; When the total loss is greater than a loss threshold, adjusting parameters in the initial expression video prediction model and parameters in the initial image encoder according to the total loss; When the total loss obtained based on the adjusted initial label image prediction model and the adjusted initial image encoder is less than or equal to the loss threshold, the adjusted initial expression video prediction model is determined as the expression video prediction model, and the adjusted initial image encoder is determined as the image encoder.

15. A video generating device, characterized in that: include: An acquisition module is used to acquire N template expression images and an object image of an object to be processed; N is an integer greater than 1, and each template expression image contains an expression action; An extraction module, used to extract N groups of expression drive signals corresponding to the facial key points of the three-dimensional templates from the N template expression images, and to extract original expression signals corresponding to the three-dimensional original facial key points of the object to be processed from the object image; A template expression image corresponds to a set of expression drive signals of key points of a three-dimensional template face; the expression drive signals include a translation matrix, a rotation matrix and facial shape parameters; A migration module, used to obtain three-dimensional reference facial key points corresponding to each group of three-dimensional template facial key points from the N groups of three-dimensional template facial key points; Performing a difference process on the translation matrix of each group of three-dimensional template facial key points and the translation matrix of the corresponding three-dimensional reference facial key points to obtain a translation deviation matrix corresponding to each group of three-dimensional template facial key points; Performing difference processing on the facial shape parameters of each group of three-dimensional template facial key points and the facial shape parameters of the corresponding three-dimensional reference facial key points to obtain the facial shape deviations corresponding to each group of three-dimensional template facial key points; determining the facial shape deviations, translation deviation matrix and rotation matrix corresponding to each group of three-dimensional template facial key points as the expression migration data corresponding to each group of three-dimensional template facial key points; According to the expression migration data corresponding to the N groups of three-dimensional template facial key points and the original expression signals corresponding to the three-dimensional original facial key points, the expression actions corresponding to the N template expression images are migrated to the object image to obtain N initial expression images of the object to be processed; A generating module is used to generate expression video data of the object to be processed based on the N initial expression images, taking the object features of the object to be processed in the object image as constraints.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

17. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are performed.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images