Data processing method and device for virtual image, electronic equipment and medium

Abstract

The invention provides a data processing method and device for a virtual image, electronic equipment, a computer readable storage medium and a computer program product, and relates to the field of artificial intelligence, in particular to the technical fields of image processing, digital human and deep learning. According to the implementation scheme, a target image including the face of a target object is acquired; performing face key point extraction on the target image to obtain a first face key point image; obtaining a first group of expression coefficients based on the first face key point image and a preset group of expression bases; adjusting corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients; obtaining a second face key point image based on the second group of expression coefficients and the group of expression bases; and based on the expression image of the second face key point image and the target image, obtaining a corresponding first image after the facial expression of the target image is converted.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the fields of deep learning, image processing, and digital human technology, and specifically to a data processing method, device, electronic device, computer-readable storage medium, and computer program product for virtual images. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] With the development of artificial intelligence technology, virtual digital humans have been widely used in live broadcasts, news broadcasts, voice prompts and other fields. Usually, it is necessary to drive the virtual digital humans to make movements and expressions synchronized with the audio based on the audio you want to broadcast, and obtain the video driven by the audio. Generating realistic and expressive portrait videos from a single face image through audio drive has broad application prospects, covering multiple fields from digital media to games, film and television creation, etc. Summary of the invention

[0004] The present disclosure provides a data processing method, device, electronic device, computer-readable storage medium and computer program product for a virtual image.

[0005] According to one aspect of the present disclosure, a data processing method for a virtual image is provided, comprising: acquiring a target image including a face of a target object; extracting facial key points based on the target image to obtain a first facial key point image; obtaining a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients has a one-to-one correspondence with the set of expression bases; adjusting corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to a facial expression transformation of the target image; obtaining a second facial key point image based on the second group of expression coefficients and the preset set of expression bases; and obtaining a first image corresponding to a facial expression transformation of the target image based on the second facial key point image and the target image.

[0006] According to another aspect of the present disclosure, a model training method is provided, including: acquiring a first target image and a first label image including a face of a target object; extracting facial key points based on the first target image to obtain a first facial key point image; obtaining a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients corresponds to the set of expression bases one-to-one; adjusting corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the first target image after the facial expression is transformed; obtaining a second facial key point image based on the second group of expression coefficients and the preset set of expression bases; obtaining a first image corresponding to the target image after the facial expression is transformed through a video generation model based on the second facial key point image and the first target image; determining a first loss value through a preset loss function based on the first image and the first label image; and adjusting parameter values ​​of the video generation model based on the first loss value.

[0007] According to another aspect of the present disclosure, there is provided a data processing device for a virtual image, comprising: a first acquisition unit, configured to acquire a target image including a face of a target object; a first key point extraction unit, configured to perform facial key point extraction based on the target image to obtain a first facial key point image; a first calculation unit, configured to obtain a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients has a one-to-one correspondence with the set of expression bases; a second calculation unit, configured to adjust corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the target image after the facial expression is transformed; a second acquisition unit, configured to obtain a second facial key point image based on the second group of expression coefficients and the preset set of expression bases; and a first generation unit, configured to obtain a first image corresponding to the target image after the facial expression is transformed based on the second facial key point image and the target image.

[0008] According to another aspect of the present disclosure, a model training device is provided, comprising: a third acquisition unit, configured to acquire a first target image and a first label image including a face of a target object; a second key point extraction unit, configured to extract facial key points based on the first target image to obtain a first facial key point image; a third calculation unit, configured to obtain a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients corresponds to the set of expression bases one-to-one; and a fourth calculation unit, configured to adjust the corresponding expression coefficients in the first group of expression coefficients. to obtain a second set of expression coefficients corresponding to the facial expression of the first target image after the facial expression is transformed; a fourth acquisition unit is configured to obtain a second facial key point image based on the second set of expression coefficients and the preset set of expression bases; a second generation unit is configured to obtain a first image corresponding to the facial expression of the target image after the facial expression is transformed through a video generation model based on the second facial key point image and the first target image; a fifth calculation unit is configured to determine a first loss value through a preset loss function based on the first image and the first label image; and an adjustment unit is configured to adjust the parameter value of the video generation model based on the first loss value.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method described in the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method described in the present disclosure when executed by a processor.

[0012] According to one or more embodiments of the present disclosure, facial key points are directly operated based on expression bases and expression coefficients to obtain corresponding second facial key point images. Since the time and resources consumed in the subsequent calculation of the image corresponding to the target image after the facial expression is transformed are much greater than the generation of the second facial key point image, it is possible to judge the generation effect of subsequent images in advance through the second facial key point image and make timely adjustments, thereby saving computing resources.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2 A flow chart of a data processing method according to an embodiment of the present disclosure is shown;

[0017] Figure 3 A schematic diagram of a video generation model for generating a video according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A flow chart of a model training method according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0020] Figure 6 A structural block diagram of a model training device according to an embodiment of the present disclosure is shown; and

[0021] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0024] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0025] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0027] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable the method of data processing to be performed.

[0028] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0029] exist Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0030] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to input a target image, edit expression coefficients, or display an image / video, etc. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0031] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0032] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0033] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0034] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0035] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0036] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0037] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as images, expression base files, and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data from the databases in response to commands.

[0038] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0039] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0040] In early studies, researchers achieved face reconstruction by constructing a parametric face model (such as 3DMM). 3DMM can model features such as shape, expression, texture, and angle. However, the face model rendering algorithm based on 3DMM has poor performance and cannot generate high-precision texture, teeth and other detailed areas.

[0041] Recently, deep learning-based methods have been widely studied for their good video generation effects. The two most representative methods are: one is the GAN-based method (such as the StyleGAN series), and the other is the diffusion model-based method (such as Hallo, Follow-Your-Emoji, EchoMimic, Aniportrait, etc.). The portraits generated by the GAN-based method are more realistic, but the diversity is greatly affected by the data distribution, and the training process is unstable and prone to mode collapse. The diffusion model-based method can generate high-quality, high-resolution and more diverse portrait videos, but it requires more computing resources.

[0042] Therefore, according to an embodiment of the present disclosure, a data processing method for a virtual image is provided to generate a corresponding portrait video based on audio drive. Figure 2 A flow chart of a data processing method according to an embodiment of the present disclosure is shown. Figure 2As shown, method 200 includes: acquiring a target image including a face of a target object (step 210); extracting facial key points based on the target image to obtain a first facial key point image (step 220); obtaining a first group of expression coefficients corresponding to facial expressions in the target image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients corresponds to the set of expression bases one-to-one (step 230); adjusting corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the target image after the facial expression is transformed (step 240); obtaining a second facial key point image based on the second group of expression coefficients and the preset set of expression bases (step 250); and obtaining an image corresponding to the target image after the facial expression is transformed based on the second facial key point image and the target image (step 260).

[0043] According to the embodiments of the present disclosure, the facial key points are directly operated based on the expression base and the expression coefficient to obtain the corresponding second facial key point image. Since the time and resources consumed by the subsequent calculation for generating the image corresponding to the target image after the facial expression is transformed are much greater than the generation of the second facial key point image, it is possible to judge the generation effect of the subsequent image in advance through the second facial key point image and make timely adjustments, thereby saving computing resources.

[0044] In step 210, a target image including a target object's face is acquired.

[0045] In the present disclosure, the target object may not be limited to a person, but may also be an animal, or an anthropomorphic animal, an object, etc., which is not limited here. For example, taking the target object as a real person, the generated image may be an image obtained by transforming the facial expression of the person in the target image. In some examples, the target object is used to generate a virtual digital human. The virtual digital human may be a two-dimensional virtual digital human generated based on the target object.

[0046] In some embodiments, the generated image may include not only the face of the target object, but may further include a background area in the target image except the face of the target object.

[0047] In step 220, facial key points are extracted based on the target image to obtain a first facial key point image.

[0048] Specifically, in some examples, facial key point detection refers to locating the positions of key areas in a facial image through algorithms, such as eyebrows, eyes, nose, mouth, and facial contours. During the detection process, the system returns the coordinate information of these key points, thereby achieving detailed recognition and analysis of the target object's face.

[0049] In some examples, in the face key point detection, various suitable key point marking methods such as 68 points, 96 / 98 points, 106 / 186 points, etc. can be used for the face, which is not limited here. For example, when the face is marked with 68 points, the face key points are divided into internal key points and contour key points. The internal key points include eyebrows, eyes, nose, and mouth, a total of 51 key points, and the contour key points include 17 key points. Thus, a face key point image is obtained. The face key point image can include coordinate information or position information of each key point.

[0050] According to some embodiments, the first facial key point image and the second facial key point image include coordinate information of pupil key points.

[0051] In some examples, pupil key points can be further included in the face key point detection. For eye-related application scenarios, such as face recognition, expression change, eye movement tracking, etc., the accurate positioning of the pupil position is very important. Using two key points to represent the left and right pupils can provide more accurate position information, which helps to perform more detailed expression change analysis and processing later.

[0052] In some examples, the first image may be reconstructed into a 3D face using a 3D face reconstruction technique to obtain a facial key point image of the target object. For example, the 3D coordinates of the facial key points may be extracted using open source plug-ins such as media pipe and facenet. Alternatively, it is understood that the 2D coordinates of the facial key points may also be extracted using opencv, etc., which is not limited here.

[0053] In step 230, based on the first facial key point image and a preset set of expression bases, a first set of expression coefficients corresponding to the facial expression in the target image is obtained, wherein the first set of expression coefficients corresponds one-to-one to the set of expression bases.

[0054] In the present disclosure, a preset set of expression bases is composed of deformable networks representing different expressions, each of which is formed by changing the three-dimensional average face model of the animated image under different expressions, which can also be called other shapes of the basic shape, also known as deformation targets. For example, the basic shape can be a default shape, such as an expressionless face. Other shapes of the basic shape are used for blending / deformation, which are different expressions (such as left eye blinking, left eye blinking, chin to the left when pouting), which are collectively referred to as blending shapes or deformation targets.

[0055] In the present disclosure, the expression coefficient is used to represent the amplitude (or degree) of expression changes corresponding to other shapes (i.e., expression bases) of the basic shape. For example, if a certain expression base represents a mouth deformation based on a basic shape (such as an expressionless face) to represent an open mouth (jaw open), then the expression coefficient corresponding to the expression base can be used to represent the degree of mouth deformation (i.e., the size of the mouth opening). For example, the range corresponding to the expression coefficient is [0,1], where the expression coefficient equals 0 to indicate a closed mouth, and the expression coefficient equals 1 to indicate a fully open mouth.

[0056] For example, the driving of a virtual face model can be implemented based on formula (1):

[0057]

[0058] Wherein, B0 represents the average face model of the virtual face model, that is, the above basic shape, and the neutral face model is an expressionless face model. i is the i-th expression base among the n expression bases used to mix and form the virtual face model, α i Expression Base B i The expression coefficient.

[0059] In some examples, based on the first facial key point image and a preset set of expression bases, the first set of expression coefficients corresponding to the facial expression in the target image can be determined by formula (1). It is understandable that any other suitable set of expression bases and driving formulas can also be used to determine the first set of expression coefficients, which is not limited here.

[0060] In some examples, the preset set of expression bases represents a complete set of expression bases, that is, through this set of expression bases, various forms of facial expressions required can be obtained by mixing / deforming. For example, the set of expression bases may include 52 expressions of ARKit, each of which includes a corresponding expression name and an expression coefficient representing its current expression amplitude. Taking the expression base of ARKit representing the left eye wide open (eyeWideLeft) as an example, its expression coefficient gradually increases from 0 (ie, eyeWideLeft = 0) to 1 (ie, eyeWideLeft = 1), and in its corresponding facial model, it indicates that its left eye gradually changes from an unlabeled state to a wide open state.

[0061] In some examples, any set of expression coefficients may correspond one-to-one to a preset set of complete expression bases. When a facial expression does not involve a certain expression change of a certain part, such as the eyes have no expression, the expression coefficient corresponding to the expression change of the part in the set of expression coefficients may be 0.

[0062] In step 240, corresponding expression coefficients in the first set of expression coefficients are adjusted to obtain a second set of expression coefficients corresponding to the target image after the facial expression is transformed; and in step 250, a second facial key point image is obtained based on the second set of expression coefficients and the preset set of expression bases.

[0063] According to the above, the expression coefficient is used to represent the amplitude (or degree) of expression change corresponding to other shapes (i.e., expression base) of the basic shape. By changing the corresponding expression coefficient, the degree of expression change corresponding to the corresponding part of the face can be changed, thereby achieving the expression change effect.

[0064] According to some embodiments, the target image and the second facial landmark image have the same dimension.

[0065] In this embodiment, the dimensions of the target image and the second facial key point image are the same, that is, the data between the images used to finally generate the expression change image are aligned, which is more helpful to improve the generation effect of the expression change image.

[0066] According to some embodiments, based on the second facial key point image and the target image, obtaining a first image corresponding to the target image after the facial expression is transformed includes: generating an expression image based on the second facial key point image, wherein the expression image is generated based on the lines connecting facial key points related to the expression in the second facial key point image; and obtaining a first image corresponding to the target image after the facial expression is transformed based on the expression image and the target image.

[0067] It is understandable that when the second facial key point image used to generate the expression image has the same dimension as the target image, the expression image generated based on the second facial key point image and the target image also have the same dimension. For example, the target image and the second facial key point image are both two-dimensional images, or both are three-dimensional images. In this case, data alignment between the images used to finally generate the expression transformation image is more helpful to improve the generation effect of the expression transformation image.

[0068] Generally, the input target image is a two-dimensional image. Therefore, according to some embodiments, the target image and the second facial key point image are both two-dimensional key point images, and each expression base in the preset set of expression bases and the first facial key point image are both three-dimensional key point images.

[0069] It is understandable that 3D face data has one more dimension of depth information than 2D face data, so 3D face recognition has an advantage over 3D face recognition in terms of recognition accuracy and liveness detection accuracy. In the above embodiment, the expression base formed by the 3D key point image can achieve more refined expression changes.

[0070] Therefore, further, in order to improve the generation effect of expression transformation images, according to some embodiments, obtaining the second facial key point image based on the second group of expression coefficients and the preset group of expression bases includes: obtaining a third facial key point image after expression transformation of the target object's face based on the second group of expression coefficients and the preset group of expression bases, wherein the third facial key point image includes coordinate information of three-dimensional key points; and obtaining the second facial key point image based on the coordinate information of the three-dimensional key points and the preset first transformation matrix, wherein the second facial key point image includes coordinate information of two-dimensional key points.

[0071] In some examples, there is a mapping relationship between the 3D coordinate points of the face model and the 2D coordinate points of the corresponding image, which can be represented by a 3*4 affine transformation matrix. It should be noted that the 3D coordinate points of the face model and the 2D coordinate points of the corresponding image are normalized, and both have one more dimension than the original ones, that is, the 2D coordinate points X 2D =(X i ,Y i ,1) T , the three-dimensional coordinate point X of the face model 3D =(X i ,Y i ,Z i ,1) T . The two-dimensional to three-dimensional coordinate conversion is realized according to formula (2):

[0072] X 2D =P×X 3D Formula (2)

[0073] Among them, P is the affine transformation matrix that needs to be obtained, which can be applied to the three-dimensional coordinate point to obtain the coordinates of the corresponding two-dimensional point.

[0074] For example, when recognizing a face, by detecting the key points of the face (including eyes, nose, mouth, etc.), the three-dimensional key points of the face model and the two-dimensional key points of the corresponding image are obtained. Then, the affine transformation matrix (i.e., the first transformation matrix) is calculated based on the key point coordinates.

[0075] In some examples, any suitable algorithm may be used to solve the corresponding affine transformation matrix, including but not limited to the Gold Standard Algorithm. The affine transformation matrix obtained by solving the solution can be applied to the three-dimensional coordinate points to obtain the coordinates of the corresponding two-dimensional points, thereby obtaining the aligned face image.

[0076] In some examples, the affine transformation matrix between the 3D-2D key point coordinates of the corresponding facial model may be pre-calculated, and then the second facial key point image obtained after adjusting the expression coefficient may be dimensional transformed based on the affine transformation matrix.

[0077] According to some embodiments, obtaining a second facial key point image based on the second set of expression coefficients and the preset set of expression bases includes: obtaining a third facial key point image after performing expression transformation on the target object's face based on the second set of expression coefficients and the preset set of expression bases, wherein the third facial key point image includes coordinate information of three-dimensional key points; obtaining the second facial key point image based on the coordinate information of the three-dimensional key points and a preset second transformation matrix, wherein the preset second transformation matrix is ​​used to perform operations on the target object's face in three-dimensional space, wherein the operations include at least one of the following items: rotation, translation.

[0078] In this embodiment, rotation and movement control of the face can be achieved through rotation and translation operations, for example, fine-grained control effects such as raising the head 5 degrees and moving to the left 5 centimeters can be achieved.

[0079] In some examples, in order to estimate the correspondence between the two-dimensional and three-dimensional coordinate points, the scaling coefficient, the rotation matrix and the translation matrix can be further decomposed and derived through the above-mentioned affine transformation relationship. Through the corresponding rotation matrix and translation matrix, the rotation and translation operations of the target object face in the three-dimensional space can be realized.

[0080] According to some embodiments, the second facial key point image includes coordinate information of pupil key points. By controlling the pupil position, a refined expression change effect can be achieved.

[0081] As mentioned above, through facial key point detection, the positions of key areas in the facial image can be located, including eyebrows, eyes, nose, mouth, and facial contours. Usually, the key points in the eye area are generally the key points in the eye socket area. During the detection process, the system will return the coordinate information of these key points, such as the coordinate information of key points arranged in a circle along the eye socket.

[0082] According to some embodiments, the coordinate information of the pupil key point is determined based on the coordinate information of the orbital region key point in the second facial key point image.

[0083] In some examples, after obtaining the second facial key point image, the coordinate information of the pupil key point can be located and determined based on the coordinate information of the key points of the eye socket region in the second facial key point image. Here, the pupil key point position can be determined based on any suitable position and number of eye socket key points. For example, the coordinates of the pupil key point can be determined based on the key point coordinates at the four vertices of the upper, lower, left and right of the eye socket.

[0084] It is understandable that the pupil key point position is set to the middle position of the eye, and can also be set to other appropriate positions of the eye based on its transformed expression, and there is no limitation here.

[0085] According to some embodiments, adjusting corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the target image after the facial expression is transformed includes: adjusting corresponding expression coefficients in the first group of expression coefficients multiple times to obtain multiple second groups of expression coefficients, wherein each second group of expression coefficients in the multiple second groups of expression coefficients corresponds to a transformed facial expression.

[0086] Specifically, the first set of expression coefficients can be adjusted multiple times, and each adjustment corresponds to a facial expression of the target object. Through the multiple adjustments, a variety of different facial expressions of the target object can be achieved.

[0087] According to some embodiments, adjusting the corresponding expression coefficients in the first group of expression coefficients multiple times to obtain multiple second groups of expression coefficients includes: acquiring video data including the face of a first object; framing the video data to obtain multiple second images including the face of the first object; extracting facial key points for each of the multiple second images to obtain fourth facial key point images corresponding to each of the multiple second images; based on the fourth facial key point image and the preset set of expression bases, obtaining a third group of expression coefficients corresponding to each of the multiple second images; for each of the multiple second images, determining the difference between the corresponding expression coefficient in the third group of expression coefficients corresponding to the second image and the corresponding expression coefficient in the third group of expression coefficients corresponding to the third image to obtain a set of expression coefficient differences; for each of the second images, adjusting the corresponding expression coefficient in the first group of expression coefficients based on the set of expression coefficient differences corresponding to the second image to obtain the multiple second groups of expression coefficients.

[0088] In this embodiment, the facial expression of the target object is adjusted based on the facial expression of the first object in the video through a video-driven method, so as to achieve high-precision facial video generation based on the target image.

[0089] In the present disclosure, the acquired video data includes the face of a first object, and the video data includes multiple facial video frame images of the first object. The face in the facial video frame images can be in any posture and any expression, and it is only necessary to ensure that the face is clearly visible so that reliable facial features can be obtained.

[0090] In the above embodiment, the gap between the third group of expression coefficients corresponding to the third image and the first group of expression coefficients corresponding to the target image is the smallest. Wherein, the "gap" can be determined based on the sum of the corresponding set of expression coefficient differences or their weighted sum. For example, assuming that the third group of expression coefficients is {0.2, 0.3, 0.7, 0.1} and the first group of expression coefficients is {0.5, 0.4, 0.6, 0.1}, then the difference between the third group of expression coefficients and the first group of expression coefficients is {0.3, 0.1, 0.1, 0}, then the gap can be (0.3+0.1+0.1+0)=0.5. Alternatively, the gap can be determined based on the weighted sum of the differences between the set of expression coefficients.

[0091] In some examples, the weight value corresponding to the corresponding expression coefficient can be determined based on the degree of influence of the corresponding expression base on the expression change. For example, the weight corresponding to the expression base corresponding to the mouth is greater than the weight corresponding to the expression base corresponding to the eyes.

[0092] In some embodiments, for each of the second images, adjusting the corresponding expression coefficients in the first group of expression coefficients based on the difference value of the set of expression coefficients corresponding to the second image to obtain the multiple second groups of expression coefficients includes: for each of the second images, adjusting the corresponding expression coefficients in the first group of expression coefficients based on the difference value of the set of expression coefficients corresponding to the second image to obtain the multiple fourth groups of expression coefficients; determining the difference value between the corresponding expression coefficients in the third group of expression coefficients corresponding to the third image and the corresponding expression coefficients in the first group of expression coefficients corresponding to the target image; and obtaining the multiple second groups of expression coefficients based on the multiple fourth groups of expression coefficients and the expression coefficient differences calculated based on the third image and the target image.

[0093] Specifically, assuming that the above gap is A, the expression coefficient corresponding to the target image is a (assuming that only one table is included), and the expression coefficients corresponding to each of the multiple image frames of the video data are a sequence: [b, c, d, e, f, g] (assuming that each image frame also only includes the same expression), and assuming that the gap between e and a is the smallest, that is, A = ea. Then, the obtained multiple second groups of expression coefficients are a sequence: [a+(be)+A, a+(ce)+A, a+(de)+A, a+(ee)+A, a+(fe)+A, a+(ge)+A].

[0094] Therefore, according to some embodiments, obtaining a second facial key point image based on the second set of expression coefficients and the preset set of expression bases includes: obtaining a second facial key point image sequence based on the multiple second sets of expression coefficients and the preset set of expression bases, the images in the second facial key point image sequence corresponding one-to-one to the multiple second sets of expression coefficients.

[0095] In step 260, based on the second facial key point image and the target image, a first image corresponding to the target image after the facial expression is transformed is obtained.

[0096] According to some embodiments, based on the second facial key point image and the target image, obtaining a first image corresponding to the target image after the facial expression is transformed includes: generating an expression image based on the second facial key point image, wherein the expression image is generated based on the lines connecting facial key points related to the expression in the second facial key point image; and obtaining a first image corresponding to the target image after the facial expression is transformed based on the expression image and the target image.

[0097] In some examples, the expression image may include lines connecting the key points of the eyes, mouth, eyebrows, and the facial contour below the eyebrows or eyes. Generally, the nose part hardly changes or changes slightly in the change of facial expressions, so the nose part can be ignored in the expression image, and the nose key points are not connected.

[0098] In the above example of facial key point detection including pupil key points, the expression image may further include pupil key point information, thereby providing more accurate facial position information and facilitating more detailed analysis and processing in the future.

[0099] In the case where the second facial key point image sequence is generated as described above, according to some embodiments, obtaining a first image corresponding to a facial expression transformed for the target image based on the second facial key point image and the target image includes: generating an expression image sequence based on the second facial key point image sequence; and obtaining a plurality of first images corresponding to a facial expression transformed for the target image multiple times based on the expression image sequence and the target image, wherein the plurality of images correspond one-to-one to the expression images in the expression image sequence.

[0100] In some examples, the plurality of first images may also be feature maps.

[0101] According to some embodiments, a corresponding first image after the facial expression of the target image is transformed is obtained by the following operations: performing image feature extraction on the target image to obtain a first image feature; performing image feature extraction on the expression image to obtain a second image feature; inputting the first image feature and the second image feature into a preset diffusion model to obtain a third image feature; and based on the third image feature, obtaining a corresponding first image after the facial expression of the target image is transformed.

[0102] In the present disclosure, image feature extraction is the process of extracting useful information from an image. This information is usually expressed in the form of numerical values, vectors or symbols, and is not directly expressed as the image itself. These features can help computers "understand" the content of the image, thereby achieving image recognition and classification. Image features usually include geometric features, shape features, amplitude features, histogram features, and color features.

[0103] In some examples, an image encoder can be used to perform image feature extraction on a target image to obtain a first image feature. An image encoder is a component for processing visual information, which converts image data into a format that can be further analyzed by a model. This usually involves feature extraction, which is to extract useful information from an image, such as color, texture, shape, and object location.

[0104] According to some embodiments, performing image feature extraction on the target image to obtain a first image feature includes: inputting the target image into a variational autoencoder to obtain the first image feature.

[0105] In some examples, image feature extraction can be performed through a neural network based on deep learning, such as a convolutional neural network (CNN). CNN automatically learns features through a multi-layer network without manually setting feature extraction rules. VGG and ResNet are two well-known CNN architectures that extract image features through deep network structures.

[0106] It can be understood that in the embodiments of the present disclosure, image feature extraction can be achieved by any suitable method, which is not limited here.

[0107] According to some embodiments, the method further includes: performing video synthesis based on the multiple first images to obtain a video generated based on the target image.

[0108] According to some embodiments, obtaining, based on the multiple first images, a video generated based on the target image comprises: inputting the multiple first images into a variational self-decoder to obtain a video generated based on the target image.

[0109] According to some embodiments, performing video synthesis based on the multiple first images to obtain a video generated based on the target image includes: performing image feature extraction on the target image to obtain a first image feature; performing image feature extraction on the expression image sequence to obtain a second image feature sequence; inputting the first image feature and the second image feature sequence into an image generation module to obtain a fourth image feature sequence, wherein image features in the fourth image feature are image features generated based on the target image and corresponding to corresponding image features in the second image feature sequence; inputting the fourth image feature sequence into a video synthesis module to obtain the multiple first images, wherein a first image among the multiple first images is a feature map, wherein the video synthesis model is used to achieve smoothness of the video generated based on the multiple first images; and obtaining a video generated based on the target image based on the multiple first images.

[0110] In some examples, each of the fourth image features is an image feature generated based on the target image and corresponding to a corresponding image feature in the second image feature sequence. Each of the plurality of first images is a feature map.

[0111] According to some embodiments, the corresponding second image features are obtained through a preset linear attention network.

[0112] Figure 3 FIG. 2 shows a schematic diagram of a video generation model for generating a video according to an embodiment of the present disclosure. Figure 3 As shown, the target image is input into the image encoder to obtain the first image feature; the expression image sequence is input into the key point encoder to obtain the second image feature sequence; the first image feature and the second image feature sequence are sequentially input into the image generation module and the video synthesis module in the diffusion model to obtain multiple first images (i.e., feature maps). The multiple first images are passed through the image decoder to obtain a video generated based on the target image. In some examples, the main body of the diffusion model can adopt the Unet framework, the image generation module is responsible for the target object generation and image background supplement of a single image, and the video synthesis module is responsible for the smoothness and stability of the entire video to be generated.

[0113] According to the embodiments of the present disclosure, Figure 4As shown, a model training method 400 is also provided, including: obtaining a first target image and a first label image including a target object's face (step 410); extracting facial key points based on the first target image to obtain a first facial key point image (step 420); obtaining a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients corresponds to the set of expression bases one-to-one (step 430); adjusting the corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the first target image after the facial expression is transformed (step 440); obtaining a second facial key point image based on the second group of expression coefficients and the preset set of expression bases (step 450); obtaining a first image corresponding to the target image after the facial expression is transformed through a video generation model based on the second facial key point image and the first target image (step 460); determining a first loss value based on the first image and the first label image through a preset loss function (step 470); and adjusting the parameter value of the video generation model based on the first loss value (step 480).

[0114] According to some embodiments, the target image and the second facial landmark image have the same dimension.

[0115] In this embodiment, the dimensions of the target image and the second facial key point image are the same, that is, the data between the images used to finally generate the expression change image are aligned, which is more helpful to improve the generation effect of the expression change image.

[0116] According to some embodiments, based on the second facial key point image and the first target image, obtaining a first image corresponding to the target image after the facial expression is transformed through a video generation model includes: generating a first expression image based on the second facial key point image, wherein the first expression image is generated based on the lines connecting facial key points related to the expression in the second facial key point image; and inputting the first expression image and the target image into the video generation model to obtain the first image corresponding to the target image after the facial expression is transformed.

[0117] It is understandable that when the second facial key point image used to generate the expression image has the same dimension as the target image, the expression image generated based on the second facial key point image and the target image also have the same dimension. For example, the target image and the second facial key point image are both two-dimensional images, or both are three-dimensional images. At this time, data alignment between the images used to finally generate the expression transformation image is more helpful to improve the model learning and training effect.

[0118] Typically, the input target image is a two-dimensional image. Therefore, according to some embodiments, the first target image and the second facial key point image are both two-dimensional key point images, and each expression base in the preset set of expression bases and the first facial key point image are both three-dimensional key point images.

[0119] It is understandable that 3D face data has one more dimension of depth information than 2D face data, so 3D face recognition has an advantage over 3D face recognition in terms of recognition accuracy and liveness detection accuracy. In the above embodiment, the expression base formed by the 3D key point image can achieve more refined expression changes.

[0120] Therefore, further, in order to improve the generation effect of expression transformation images, according to some embodiments, obtaining the second facial key point image based on the second group of expression coefficients and the preset group of expression bases includes: obtaining a third facial key point image after expression transformation of the target object's face based on the second group of expression coefficients and the preset group of expression bases, wherein the third facial key point image includes coordinate information of three-dimensional key points; and obtaining the second facial key point image based on the coordinate information of the three-dimensional key points and the preset first transformation matrix, wherein the second facial key point image includes coordinate information of two-dimensional key points.

[0121] According to some embodiments, obtaining a second facial key point image based on the second set of expression coefficients and the preset set of expression bases includes: obtaining a third facial key point image after performing expression transformation on the target object's face based on the second set of expression coefficients and the preset set of expression bases, wherein the third facial key point image includes coordinate information of three-dimensional key points; obtaining the second facial key point image based on the coordinate information of the three-dimensional key points and a preset second transformation matrix, wherein the preset second transformation matrix is ​​used to perform operations on the target object's face in three-dimensional space, wherein the operations include at least one of the following items: rotation, translation.

[0122] According to some embodiments, the second facial key point image includes coordinate information of pupil key points. By controlling the pupil position, a refined expression change effect can be achieved.

[0123] According to some embodiments, the coordinate information of the pupil key point is determined based on the coordinate information of the orbital region key point in the second facial key point image.

[0124] According to some embodiments, the video generation model includes a first image encoder, a second image encoder, a diffusion model, and an image decoder. Inputting the first expression image and the first target image into the video generation model to obtain a second image corresponding to the target image after the facial expression is transformed includes: inputting the first target image into the first image encoder to obtain a first image feature; inputting the first expression image into the second image encoder to obtain a second image feature; inputting the first image feature and the second image feature into the diffusion model to obtain a third image feature; and based on the third image feature, obtaining a corresponding first image corresponding to the target image after the facial expression is transformed.

[0125] In some examples, image feature extraction can be performed by a neural network based on deep learning, that is, at least one of the first image encoder and the second image encoder can be a neural network based on deep learning, for example, a convolutional neural network (CNN). CNN automatically learns features through a multi-layer network without manually setting feature extraction rules. VGG and ResNet are two well-known CNN architectures that extract image features through deep network structures.

[0126] It can be understood that in the embodiments of the present disclosure, image feature extraction can be achieved by any suitable method, which is not limited here.

[0127] According to some embodiments, the first image encoder may include a variational self-encoder to obtain the first image feature. According to some embodiments, a video generated based on the target image may be obtained based on the plurality of first images. For example, the plurality of first images are input into a variational self-decoder to obtain a video generated based on the image.

[0128] Therefore, according to some embodiments, adjusting the parameter values ​​of the video generation model based on the first loss value includes: adjusting the parameter values ​​of the second image encoder and the diffusion model based on the first loss value.

[0129] According to some embodiments, the diffusion model includes an image generation module and a video synthesis module, wherein inputting the first image feature and the second image feature into the diffusion model to obtain a third image feature includes: inputting the first image feature and the second image feature into the image generation module to obtain a fourth image feature, wherein the fourth image feature is an image feature generated based on the target image and corresponding to the second image feature; and inputting the fourth image feature into the video synthesis module to obtain the third image feature, wherein the video synthesis model is used to achieve the smoothness of the video when generating the video based on multiple second image features.

[0130] According to some embodiments, adjusting the parameter values ​​of the video generation model based on the first loss value includes: adjusting the parameter values ​​of the second image encoder and the image generation module based on the first loss value.

[0131] According to some embodiments, the model training method according to the present disclosure also includes: obtaining multiple second expression images, a second target image including a target object's face, and multiple second label images corresponding one-to-one to the multiple second expression images, wherein each of the multiple second expression images is generated based on the connection lines of facial key points related to the expression in the corresponding facial key point image; inputting the second target image into the first image encoder to obtain a fifth image feature; inputting the multiple second expression images into the second image encoder to obtain a plurality of sixth image features; inputting the fifth image feature and the multiple sixth image features into the image generation module to obtain a plurality of seventh image features, wherein the multiple seventh image features correspond one-to-one to the multiple sixth image features; and inputting the multiple seventh image features into the video synthesis module to obtain a plurality of eighth image features; inputting the multiple eighth image features into the image decoder to obtain a plurality of second images; determining a second loss value based on the multiple second images and the multiple second label images by a preset loss function; and adjusting the parameter value of the video synthesis module based on the second loss value.

[0132] Specifically, in some examples, the training of the video generation model can be divided into two stages. That is, in the first stage, the second image encoder and the image generation module are trained based on a single-frame expression image; in the second stage, the video synthesis module is trained based on multiple-frame expression images.

[0133] According to some embodiments, the second image encoder comprises a linear attention network.

[0134] According to some embodiments, the preset loss function loss is determined based on the following formula:

[0135]

[0136] in, represents the first label image or the corresponding second label image among the multiple second label images, C represents the first image or the corresponding second image among the multiple second images, D represents the mouth mask image, and a and b are both preset hyperparameters.

[0137] In some examples, when When represents the corresponding second label image among the multiple second label images and C represents the corresponding second image among the multiple second images, the loss value corresponding to each image in the multiple second label images / second images can be calculated by the above formula, and the second loss value is obtained by adding the loss values ​​corresponding to the multiple second label images / second images, so as to adjust the parameter value of the video synthesis module based on the second loss value.

[0138] In this embodiment, in order to improve the clarity and stability of tooth generation, the mouth mask loss and the overall image loss are used to supervise the model training at the same time. In addition, in some examples, by setting appropriate a and b, the weight of the mouth mask loss can be increased to further improve the clarity and stability of tooth generation.

[0139] In the present disclosure, the model trained by the model training method described in any one of the above embodiments can be used to implement the data processing method described in any one of the embodiments of the present disclosure.

[0140] Here, the embodiments for implementing the model training method and the embodiments for implementing the data processing method, wherein the corresponding operations are similar, will not be described in detail here.

[0141] According to the embodiments of the present disclosure, Figure 5 As shown, a data processing device 500 for a virtual image is also provided, comprising: a first acquisition unit 510, configured to acquire a target image including a face of a target object; a first key point extraction unit 520, configured to extract facial key points based on the target image to obtain a first facial key point image; a first calculation unit 530, configured to acquire a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients has a one-to-one correspondence with the set of expression bases; a second calculation unit 540, configured to adjust corresponding expression coefficients in the first group of expression coefficients to acquire a second group of expression coefficients corresponding to the target image after the facial expression is transformed; a second acquisition unit 550, configured to acquire a second facial key point image based on the second group of expression coefficients and the preset set of expression bases; and a first generation unit 560, configured to acquire a first image corresponding to the target image after the facial expression is transformed based on the second facial key point image and the target image.

[0142] Here, the operations of the above-mentioned units 510-560 of the data processing device 500 are similar to the operations of the above-described steps 210-260, and are not described in detail here.

[0143] According to the embodiments of the present disclosure, Figure 6As shown, a model training device 600 is also provided, including: a third acquisition unit 610, configured to acquire a first target image and a first label image including the face of a target object; a second key point extraction unit 620, configured to extract facial key points based on the first target image to obtain a first facial key point image; a third calculation unit 630, configured to obtain a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset set of expression bases, wherein the first group of expression coefficients corresponds to the set of expression bases one by one; a fourth calculation unit 640, configured to adjust the corresponding expression coefficients in the first group of expression coefficients to obtain a first group of expression coefficients corresponding to the first facial key point image. Obtain a second set of expression coefficients corresponding to the facial expression after the first target image is transformed; a fourth acquisition unit 650 is configured to obtain a second facial key point image based on the second set of expression coefficients and the preset set of expression bases; a second generation unit 660 is configured to obtain a first image corresponding to the facial expression after the target image is transformed through a video generation model based on the second facial key point image and the first target image; a fifth calculation unit 670 is configured to determine a first loss value through a preset loss function based on the first image and the first label image; and an adjustment unit 680 is configured to adjust the parameter value of the video generation model based on the first loss value.

[0144] Here, the operations of the above-mentioned units 610-680 of the model training device 600 are similar to the operations of the steps 410-480 described above, and will not be repeated here.

[0145] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0146] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0147] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0148] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0149] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device 700. The input unit 706 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 707 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0150] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as method 200 or 400. For example, in some embodiments, method 200 or 400 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the method 200 or 400 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method 200 or 400 in any other appropriate manner (eg, by means of firmware).

[0151] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0153] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0156] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0157] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0158] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing method for a virtual image, comprising: Acquire a target image including a face of a target object; Extracting facial key points based on the target image to obtain a first facial key point image; Based on the first facial key point image and a preset set of expression bases, obtaining a first set of expression coefficients corresponding to the first facial key point image, wherein the first set of expression coefficients corresponds to the set of expression bases in a one-to-one manner; Adjusting corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the target image after the facial expression is transformed; Based on the second set of expression coefficients and the preset set of expression bases, obtaining a second facial key point image; as well as Based on the second facial key point image and the target image, a first image corresponding to the target image after the facial expression is transformed is obtained.

2. The method of claim 1, wherein: The target image and the second facial key point image have the same dimension.

3. The method of claim 2, wherein: The target image and the second facial key point image are both two-dimensional key point images, and each expression base in the preset set of expression bases and the first facial key point image are both three-dimensional key point images, wherein obtaining the second facial key point image based on the second set of expression coefficients and the preset set of expression bases comprises: Based on the second group of expression coefficients and the preset group of expression bases, obtaining a third facial key point image obtained after performing expression transformation on the target object's face, wherein the third facial key point image includes coordinate information of three-dimensional key points; and Based on the coordinate information of the three-dimensional key points and a preset first transformation matrix, the second facial key point image is obtained, wherein the second facial key point image includes the coordinate information of the two-dimensional key points.

4. The method according to any one of claims 1 to 3, wherein: Based on the second set of expression coefficients and the preset set of expression bases, obtaining a second facial key point image comprises: Based on the second group of expression coefficients and the preset group of expression bases, obtaining a third facial key point image obtained after performing expression transformation on the face of the target object, wherein the third facial key point image includes coordinate information of three-dimensional key points; Based on the coordinate information of the three-dimensional key points and a preset second transformation matrix, the second facial key point image is obtained, wherein the preset second transformation matrix is ​​used to perform operations on the target object's face in three-dimensional space, wherein the operations include at least one of the following items: rotation and translation.

5. The method of claim 1, wherein: The second facial key point image includes coordinate information of the pupil key point.

6. The method of claim 5, wherein: The coordinate information of the pupil key point is determined based on the coordinate information of the eye socket area key point in the second facial key point image.

7. The method of claim 1, wherein: Adjusting corresponding expression coefficients in the first set of expression coefficients to obtain a second set of expression coefficients corresponding to the target image after the facial expression is transformed includes: Performing multiple adjustments on corresponding expression coefficients in the first set of expression coefficients to obtain multiple second sets of expression coefficients, wherein corresponding second sets of expression coefficients in the multiple second sets of expression coefficients correspond to a transformed facial expression; Based on the second set of expression coefficients and the preset set of expression bases, obtaining a second facial key point image comprises: Based on the plurality of second groups of expression coefficients and the preset group of expression bases, a second facial key point image sequence is obtained, wherein images in the second facial key point image sequence correspond one-to-one to the plurality of second groups of expression coefficients.

8. The method of claim 7, wherein: The step of adjusting the corresponding expression coefficients in the first set of expression coefficients for multiple times to obtain multiple second sets of expression coefficients includes: acquiring video data including a face of a first subject; Frame the video data to obtain a plurality of second images including the face of the first object; Extracting facial key points from each of the plurality of second images to obtain fourth facial key point images corresponding to each of the plurality of second images; Based on the fourth facial key point image and the preset set of expression bases, obtaining a third set of expression coefficients corresponding to each of the plurality of second images; For each second image in the plurality of second images, determining a difference between a corresponding expression coefficient in the third set of expression coefficients corresponding to the second image and a corresponding expression coefficient in the third set of expression coefficients corresponding to the third image, so as to obtain a set of expression coefficient difference values; For each of the second images, adjusting corresponding expression coefficients in the first group of expression coefficients based on the group of expression coefficient differences corresponding to the second image, so as to obtain the plurality of second groups of expression coefficients; Among them, the difference between the third group of expression coefficients corresponding to the third image and the first group of expression coefficients corresponding to the target image is the smallest.

9. The method according to any one of claims 1 to 8, wherein: Based on the second facial key point image and the target image, obtaining a first image corresponding to the target image after the facial expression is transformed includes: generating an expression image based on the second facial key point image, wherein the expression image is generated based on lines connecting facial key points related to the expression in the second facial key point image; and Based on the expression image and the target image, a first image corresponding to the target image after the facial expression is transformed is obtained.

10. The method according to claim 7 or 8, wherein: Based on the second facial key point image and the target image, obtaining a first image corresponding to the target image after the facial expression is transformed includes: generating an expression image sequence based on the second facial key point image sequence; and Based on the expression image sequence and the target image, a plurality of first images corresponding to the target image after multiple facial expression changes are obtained, wherein the plurality of images correspond one-to-one to the expression images in the expression image sequence.

11. The method of claim 9, wherein: The corresponding first image corresponding to the target image after the facial expression is transformed is obtained by the following operations: Performing image feature extraction on the target image to obtain a first image feature; Performing image feature extraction on the expression image to obtain a second image feature; Inputting the first image feature and the second image feature into a preset diffusion model to obtain a third image feature; as well as Based on the third image feature, a corresponding first image corresponding to the target image after the facial expression is transformed is obtained.

12. The method of claim 10, further comprising: Video synthesis is performed based on the multiple first images to obtain a video generated based on the target image.

13. The method of claim 12, wherein: Performing video synthesis based on the multiple first images to obtain a video generated based on the target image includes: Performing image feature extraction on the target image to obtain a first image feature; Performing image feature extraction on the expression image sequence to obtain a second image feature sequence; Inputting the first image feature and the second image feature sequence into an image generation module to obtain a fourth image feature sequence, wherein image features in the fourth image feature sequence are image features generated based on the target image and corresponding to corresponding image features in the second image feature sequence; Inputting the fourth image feature sequence into a video synthesis module to obtain the plurality of first images, wherein a first image of the plurality of first images is a feature map, wherein the video synthesis model is used to achieve smoothness of a video generated based on the plurality of first images; and A video generated based on the target image is obtained based on the multiple first images.

14. The method of claim 13, wherein: Extracting image features from the target image to obtain a first image feature includes: inputting the target image into a variational autoencoder to obtain the first image feature; and Obtaining a video generated based on the target image based on the multiple first images includes: inputting the multiple first images into a variational self-decoder to obtain a video generated based on the target image.

15. The method as claimed in claim 11 or 13, wherein the corresponding second image features are obtained through a preset linear attention network.

16. A model training method, comprising: Acquire a first target image and a first label image including a face of a target object; Extracting facial key points based on the first target image to obtain a first facial key point image; Based on the first facial key point image and a preset set of expression bases, obtaining a first set of expression coefficients corresponding to the first facial key point image, wherein the first set of expression coefficients corresponds to the set of expression bases in a one-to-one manner; Adjusting corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the facial expression of the first target image after the facial expression is transformed; Based on the second set of expression coefficients and the preset set of expression bases, obtaining a second facial key point image; Based on the second facial key point image and the first target image, obtaining a first image corresponding to the target image after the facial expression is transformed through a video generation model; Based on the first image and the first label image, determining a first loss value by using a preset loss function; as well as The parameter values ​​of the video generation model are adjusted based on the first loss value.

17. The method of claim 16, wherein: The target image and the second facial key point image have the same dimension.

18. The method of claim 16, wherein: The first target image and the second facial key point image are both two-dimensional key point images, and each expression base in the preset set of expression bases and the first facial key point image are both three-dimensional key point images, wherein obtaining the second facial key point image based on the second set of expression coefficients and the preset set of expression bases comprises: Based on the second group of expression coefficients and the preset group of expression bases, obtaining a third facial key point image obtained after performing expression transformation on the target object's face, wherein the third facial key point image includes coordinate information of three-dimensional key points; and Based on the coordinate information of the three-dimensional key points and a preset first transformation matrix, the second facial key point image is obtained, wherein the second facial key point image includes the coordinate information of the two-dimensional key points.

19. The method according to any one of claims 16 to 18, wherein: Based on the second set of expression coefficients and the preset set of expression bases, obtaining a second facial key point image comprises: Based on the second group of expression coefficients and the preset group of expression bases, obtaining a third facial key point image obtained after performing expression transformation on the face of the target object, wherein the third facial key point image includes coordinate information of three-dimensional key points; Based on the coordinate information of the three-dimensional key points and a preset second transformation matrix, the second facial key point image is obtained, wherein the preset second transformation matrix is ​​used to perform operations on the target object's face in three-dimensional space, wherein the operations include at least one of the following items: rotation and translation.

20. The method of claim 16, wherein: The second facial key point image includes coordinate information of the pupil key point.

21. The method of claim 20, wherein: The coordinate information of the pupil key point is determined based on the coordinate information of the eye socket area key point in the second facial key point image.

22. The method of any one of claims 16 to 21, wherein: Based on the second facial key point image and the first target image, obtaining a first image corresponding to the target image after the facial expression is transformed by a video generation model includes: generating a first expression image based on the second facial key point image, wherein the first expression image is generated based on lines connecting facial key points related to expression in the second facial key point image; and The first expression image and the target image are input into a video generation model to obtain a first image corresponding to the target image after the facial expression is transformed.

23. The method of claim 22, wherein: The video generation model includes a first image encoder, a second image encoder, a diffusion model and an image decoder, and wherein the first expression image and the first target image are input into the video generation model to obtain a second image corresponding to the target image after the facial expression is transformed, comprising: Inputting the first target image into the first image encoder to obtain a first image feature; Inputting the first expression image into the second image encoder to obtain a second image feature; Inputting the first image feature and the second image feature into the diffusion model to obtain a third image feature; and Based on the third image feature, a corresponding first image is obtained after the facial expression of the target image is transformed.

24. The method of claim 23, wherein: Adjusting the parameter value of the video generation model based on the first loss value includes: adjusting the parameter values ​​of the second image encoder and the diffusion model based on the first loss value.

25. The method of claim 23 or 24, wherein: The diffusion model includes an image generation module and a video synthesis module, wherein inputting the first image feature and the second image feature into the diffusion model to obtain a third image feature includes: Inputting the first image feature and the second image feature into the image generation module to obtain a fourth image feature, wherein the fourth image feature is an image feature generated based on the target image and corresponding to the second image feature; and The fourth image feature is input into the video synthesis module to obtain the third image feature, wherein the video synthesis model is used to achieve the smoothness of the video when the video is generated based on multiple second image features.

26. The method of claim 25, wherein: Adjusting the parameter values ​​of the video generation model based on the first loss value includes: adjusting the parameter values ​​of the second image encoder and the image generation module based on the first loss value.

27. The method of any one of claims 23-26, further comprising: Acquire a plurality of second expression images, including a second target image of the target object's face, and a plurality of second label images corresponding one-to-one to the plurality of second expression images, wherein each of the plurality of second expression images is generated based on the connection lines of facial key points related to the expression in the corresponding facial key point image; Inputting the second target image into the first image encoder to obtain a fifth image feature; Inputting the plurality of second expression images into the second image encoder to obtain a plurality of sixth image features; Inputting the fifth image feature and the plurality of sixth image features into the image generation module to obtain a plurality of seventh image features, wherein the plurality of seventh image features correspond one-to-one to the plurality of sixth image features; and Inputting the plurality of seventh image features into the video synthesis module to obtain a plurality of eighth image features; Inputting the plurality of eighth image features into the image decoder to obtain a plurality of second images; Based on the plurality of second images and the plurality of second label images, determining a second loss value by a preset loss function; and The parameter value of the video synthesis module is adjusted based on the second loss value.

28. The method of any one of claims 23-24 and 27, wherein: The second image encoder comprises a linear attention network.

29. The method of claim 16 or 27, wherein: The preset loss function loss is determined based on the following formula: in, represents the first label image or the corresponding second label image among the multiple second label images, C represents the first image or the corresponding second image among the multiple second images, D represents the mouth mask image, and a and b are both preset hyperparameters.

30. A data processing device for a virtual image, comprising: A first acquisition unit, configured to acquire a target image including a face of a target object; A first key point extraction unit, configured to extract facial key points based on the target image to obtain a first facial key point image; A first calculation unit is configured to obtain a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset group of expression bases, wherein the first group of expression coefficients corresponds to the group of expression bases in a one-to-one correspondence; A second calculation unit is configured to adjust corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the target image after the facial expression is transformed; a second acquiring unit, configured to acquire a second facial key point image based on the second group of expression coefficients and the preset group of expression bases; as well as The first generating unit is configured to obtain, based on the second facial key point image and the target image, a first image corresponding to the target image after the facial expression is transformed.

31. A model training device, comprising: A third acquisition unit is configured to acquire a first target image including a face of a target object and a first label image; a second key point extraction unit, configured to extract facial key points based on the first target image to obtain a first facial key point image; a third calculating unit configured to obtain a first group of expression coefficients corresponding to the first facial key point image based on the first facial key point image and a preset group of expression bases, wherein the first group of expression coefficients corresponds to the group of expression bases in a one-to-one correspondence; a fourth calculation unit configured to adjust corresponding expression coefficients in the first group of expression coefficients to obtain a second group of expression coefficients corresponding to the facial expression after the first target image is transformed; a fourth acquisition unit, configured to obtain a second facial key point image based on the second group of expression coefficients and the preset group of expression bases; A second generating unit is configured to obtain, based on the second facial key point image and the first target image, a first image corresponding to the target image after the facial expression is transformed through a video generation model; a fifth computing unit, configured to determine a first loss value by using a preset loss function based on the first image and the first label image; as well as An adjustment unit is configured to adjust a parameter value of the video generation model based on the first loss value.

32. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 29.

33. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-29.

34. A computer program product comprising a computer program, wherein: The computer program implements the method of any one of claims 1 to 29 when executed by a processor.

Images