Image generation methods, apparatus, electronic devices and storage media
By acquiring and stitching together rendered images of facial triangular meshes, the problem of inconsistent perspectives in the generation of 3D virtual objects was solved, achieving multi-view consistency and identity preservation that are closer to real human faces.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
- Filing Date
- 2022-11-16
- Publication Date
- 2026-07-17
AI Technical Summary
Existing image generation methods struggle to guarantee consistency and identity preservation across multiple viewpoints when generating 3D virtual objects, resulting in inconsistent generated facial images from different perspectives.
By acquiring multiple face triangle meshes corresponding to a face image, including target face triangle meshes and local face triangle meshes, obtaining their respective sampling points and vertices, inputting them into a pre-trained face generation model, and using vectors representing skin texture to generate a rendered image, the local rendered images are finally stitched together to form the target rendered image.
It improves the controllability and consistency of generated images, making the generated target rendered images closer to the distribution of real human faces, and enhancing the multi-view consistency and identity preservation of generated images.
Smart Images
Figure CN115713609B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, specifically relating to an image generation method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the development of computer technology, image processing technology has also made significant progress in recent years. One important branch is the synthesis of virtual objects from different perspectives. Low-cost editing and synthesis of realistic virtual objects from different perspectives is crucial for 2D virtual object synthesis. 3D virtual object synthesis is a method based on editing the encoding vectors corresponding to virtual objects using neural networks. During 3D virtual object synthesis, it is necessary to ensure consistency and identity preservation across multiple perspectives, meaning that the virtual object must have the same structure from any perspective and remain unchanged across different perspectives. For example, with a face, the generated face must have the same structure from any perspective and represent the same person from different perspectives. However, current model rendering processes for generating 3D virtual objects often suffer from inconsistent perspectives. Therefore, how to solve this problem has become an urgent issue for engineers. Summary of the Invention
[0003] In view of the above problems, this application proposes an image generation method, apparatus, electronic device, and storage medium to improve the above problems.
[0004] In a first aspect, embodiments of this application provide an image generation method, the method comprising: acquiring a plurality of face triangle meshes corresponding to a face image, wherein the plurality of face triangle meshes include a target face triangle mesh and a plurality of local face triangle meshes, the plurality of local face triangle meshes being face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on face key points; acquiring a plurality of sampling points corresponding to each of the plurality of face triangle meshes; inputting the plurality of sampling points corresponding to each of the plurality of face triangle meshes, a plurality of vertices corresponding to each of the face triangle meshes, and a preset vector into a pre-trained face generation model, respectively, to acquire rendered images corresponding to each of the plurality of face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture; and stitching together the rendered images corresponding to each of the plurality of face triangle meshes to obtain a target rendered image corresponding to the face image.
[0005] Secondly, embodiments of this application provide an image generation apparatus, the apparatus comprising: a mesh acquisition unit, configured to acquire multiple face triangle meshes corresponding to a face image, wherein the multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes, the multiple local face triangle meshes being face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on face key points; a sampling unit, configured to acquire multiple sampling points corresponding to each of the multiple face triangle meshes; an image generation unit, configured to input the multiple sampling points corresponding to each of the multiple face triangle meshes, multiple vertices corresponding to each face triangle mesh, and a preset vector into a face generation model, respectively, to acquire rendered images corresponding to each of the multiple face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture; and a stitching unit, configured to stitch together the rendered images corresponding to each of the multiple face triangle meshes to obtain a target rendered image corresponding to the face image.
[0006] Thirdly, embodiments of this application provide an electronic device, including one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.
[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run.
[0008] This application provides an image generation method, apparatus, electronic device, and storage medium. First, multiple face triangle meshes corresponding to a face image are acquired. These multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes. The multiple local face triangle meshes are face triangle meshes corresponding to each face region obtained from the target face triangle mesh based on facial key points. Then, multiple sampling points corresponding to each of the multiple face triangle meshes are acquired. These sampling points, multiple vertices of each face triangle mesh, and a preset vector are input into a pre-trained face generation model. The model outputs rendered images corresponding to each of the multiple face triangle meshes. The preset vector is a vector representing skin texture. Finally, the rendered images corresponding to each of the multiple face triangle meshes are concatenated to obtain the target rendered image corresponding to the face image. By using the above method, local face triangular meshes are used to generate rendering images corresponding to local face parts. These rendering images are then stitched together to obtain the target rendering image. This method ensures that the generated target rendering image corresponds to the distribution of real faces, making it closer to the real image and improving the controllability and consistency of the generated image. Attached Figure Description
[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 This illustration shows an application scenario diagram of an image generation method proposed in an embodiment of this application;
[0011] Figure 2 This illustration shows an application scenario diagram of an image generation method proposed in an embodiment of this application;
[0012] Figure 3 A flowchart of an image generation method according to an embodiment of this application is shown;
[0013] Figure 4 A flowchart of an image generation method according to another embodiment of this application is shown;
[0014] Figure 5 A flowchart of an image generation method according to another embodiment of this application is shown;
[0015] Figure 6 This paper shows a structural block diagram of an image generation apparatus according to an embodiment of the present application;
[0016] Figure 7 This paper shows a structural block diagram of an image generation apparatus according to an embodiment of the present application;
[0017] Figure 8 This invention illustrates a structural block diagram of an electronic device or server for performing an image generation method according to an embodiment of the present invention.
[0018] Figure 9 This application shows a storage unit for storing or carrying program code that implements the image generation method according to the embodiments of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0020] With the development of computer technology, image processing technology has also made significant progress in recent years. One important branch is the synthesis of virtual objects from different perspectives. Low-cost editing and synthesis of realistic virtual objects from different perspectives is crucial for 2D virtual object synthesis. 3D virtual object synthesis is a method based on editing the encoding vectors corresponding to virtual objects using neural networks. During 3D virtual object synthesis, it is necessary to ensure consistency and identity preservation across multiple perspectives, meaning that the virtual object must have the same structure from any perspective and remain unchanged across different perspectives. For example, taking a face as an example, it is essential to ensure that the generated face has the same structure from any perspective and represents the same person from different perspectives.
[0021] However, the inventors discovered in their research on related image generation methods that, when generating face images, these methods rely on audio, stitched face images, and profile images. However, the goal of face image generation is generally to produce controllable and consistent face images from multiple angles. That is, based on a given input, one should be able to roughly determine the shape of the generated face image and generate face images with the same ID that exhibit high consistency across different viewpoints. The controllability and consistency of face images generated using the aforementioned methods cannot be guaranteed.
[0022] Therefore, the inventors have proposed the image generation method, apparatus, electronic device, and storage medium of this application. First, multiple face triangle meshes corresponding to a face image are obtained. These multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes. The multiple local face triangle meshes are face triangle meshes corresponding to each face region obtained from the target face triangle mesh based on facial key points. Then, multiple sampling points corresponding to each of the multiple face triangle meshes are obtained. These sampling points, multiple vertices of each face triangle mesh, and a preset vector are input into a pre-trained face generation model. The model outputs rendered images corresponding to each of the multiple face triangle meshes. The preset vector is a vector representing skin texture. Finally, the rendered images corresponding to each of the multiple face triangle meshes are stitched together to obtain the target rendered image corresponding to the face image. By using the above method, local face triangular meshes are used to generate rendering images corresponding to local face parts. These rendering images are then stitched together to obtain the target rendering image. This method ensures that the generated target rendering image corresponds to the distribution of real faces, making it closer to the real image and improving the controllability and consistency of the generated image.
[0023] In this embodiment of the application, the provided image generation method can be executed by an electronic device. In this manner, all steps of the image generation method provided in this embodiment can be performed by the electronic device. For example, as... Figure 1 As shown, the processor of electronic device 100 executes the process of acquiring multiple face triangle meshes corresponding to a face image. These multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes. The multiple local face triangle meshes are face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on facial key points. Multiple sampling points corresponding to each of the multiple face triangle meshes are acquired. The multiple sampling points corresponding to each of the multiple face triangle meshes, multiple vertices corresponding to each face triangle mesh, and a preset vector are input into a pre-trained face generation model to acquire the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model. The preset vector is a vector representing skin texture. The rendered images corresponding to each of the multiple face triangle meshes are then stitched together to obtain the target rendered image corresponding to the face image.
[0024] Furthermore, the image generation method provided in this application embodiment can also be executed by a server (cloud). Correspondingly, in this server-executed method, an electronic device can acquire multiple face triangle meshes corresponding to a face image and synchronously send these meshes to the server. The server then acquires multiple sampling points corresponding to each of the multiple face triangle meshes in real time. The multiple sampling points, multiple vertices, and preset vectors corresponding to each of the multiple face triangle meshes are input into a pre-trained face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model. The preset vectors are vectors representing skin texture. The rendered images corresponding to each of the multiple face triangle meshes are then stitched together to obtain the target rendered image corresponding to the face image.
[0025] Alternatively, the process can be performed collaboratively by an electronic device and a server. In this collaborative execution method, some steps of the image generation method provided in this application embodiment are performed by the electronic device, while other steps are performed by the server.
[0026] For example, such as Figure 2 As shown, the electronic device 100 can perform an image generation method including: acquiring multiple face triangle meshes corresponding to a face image; acquiring multiple sampling points corresponding to each of the multiple face triangle meshes; then, the server 200 performs the following: inputting the multiple sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, and a preset vector into a pre-trained face generation model to acquire the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture; and stitching the rendered images corresponding to each of the multiple face triangle meshes together to obtain the target rendered image corresponding to the face image.
[0027] It should be noted that in this method where electronic devices and servers work together, the steps performed by the electronic devices and servers are not limited to those described in the examples above. In practical applications, the steps performed by the electronic devices and servers can be dynamically adjusted according to the actual situation.
[0028] It should be noted that, in addition to being able to... Figure 1 and Figure 2 In addition to smartphones, the devices shown can also be in-vehicle systems, wearable devices, tablets, laptops, smart speakers, etc. Server 120 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers.
[0029] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0030] Please see Figure 3 The image generation method provided in this application embodiment is applied to, for example, Figure 1 or Figure 2 The method includes the following: (illustrated electronic device or server)
[0031] Step S110: Obtain multiple face triangle meshes corresponding to the face image, wherein the multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes, and the multiple local face triangle meshes are face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on face key points.
[0032] In this embodiment, the face triangular mesh, also known as a 3D mesh, refers to a mesh diagram representing the 3D topological structure of a face. The 3D topological structure can be obtained by triangulating a face image. The face triangular mesh consists of multiple vertex information and triangular face information. Each triangular face is a triangle composed of three vertices. The vertex information includes the coordinates of each vertex, which can be represented as (x, y, z).
[0033] The target face triangle mesh is the complete face triangle mesh corresponding to the face image. Multiple local face triangle meshes are face triangle meshes corresponding to each face feature, obtained by segmenting the target face triangle mesh. When segmenting the target face triangle mesh, segmentation can be performed according to facial key points corresponding to the face features included in the face image. The face features mainly refer to the facial features of the target face, such as the mouth, left eye, nose, right eye, ears, and eyebrows.
[0034] In one approach, after obtaining the target face triangle mesh corresponding to the face image, the face key points can be determined from the vertices included in the target face triangle mesh based on the correspondence between the vertices included in the target face triangle mesh and the face key points. Thus, the target face triangle mesh can be segmented based on the face key points.
[0035] It is known that each facial feature can be represented by multiple facial key points; for example, the mouth can be represented by facial key points representing the mouth. Therefore, when segmenting the target face triangle mesh based on these key points, the target face triangle mesh can be segmented according to the facial key points corresponding to each facial feature, resulting in multiple local face triangle meshes corresponding to each facial feature. Optionally, the size of each segmented local face triangle mesh can be determined based on the coordinates of each vertex among the multiple facial key points representing each facial feature. Specifically, the size of each local face triangle mesh can be determined based on the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, maximum y-coordinate, minimum z-coordinate, and maximum z-coordinate among the multiple facial key points representing each facial feature.
[0036] In one embodiment of this application, the multiple face triangle meshes corresponding to the face image can be pre-stored in a preset database or cloud server. When it is necessary to obtain the multiple face triangle meshes corresponding to the face image, the multiple face triangle meshes corresponding to the face image can be obtained from the preset database or cloud server.
[0037] As another approach, multiple face triangles corresponding to a face image can also be acquired in real time. When a grid data acquisition request is received, the face image is processed to obtain the target face triangle corresponding to the face image. Then, the target face triangle is segmented to obtain multiple local face triangles. The target face triangle and multiple local face triangles are then returned to the device that sent the grid data acquisition request.
[0038] In the embodiments of this application, the method of obtaining multiple face triangle meshes corresponding to a face image can also be other methods, and no specific limitation is made here.
[0039] Step S120: Obtain multiple sampling points corresponding to each of the multiple face triangle meshes.
[0040] In this embodiment, the multiple sampling points are points obtained on the line segment formed by the synthesized viewpoint of the acquired face image and the vertices included in each face triangular mesh. The synthesized viewpoint of the face image can be the acquisition point of the face image.
[0041] Since each face triangle contains multiple vertices, each face triangle can correspond to multiple line segments. When acquiring sampling points on each line segment, multiple sampling points can be acquired on each line segment to obtain multiple sampling points corresponding to each of the multiple face triangles.
[0042] Step S130: Input the multiple sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, and the preset vector into the pre-trained face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture.
[0043] In this embodiment, the pre-trained face generation model can be a Neural Radiance Fields (NERF) model. The NERF model is used to non-explicitly model a complex static scene using a neural network. After the network is trained, it can render clear scene images from any angle.
[0044] One approach is to input multiple sampling points, multiple vertices, and preset vectors corresponding to each face triangle mesh into a pre-trained face generation model when obtaining the rendered image corresponding to each face triangle mesh. The face generation model can then output the rendered image corresponding to each face triangle mesh. Specifically, multiple sampling points, multiple vertices, and preset vectors corresponding to the target face triangle mesh are first input into the pre-trained face generation model to obtain the rendered image corresponding to the target face triangle mesh output by the face generation model. Then, the vertices, multiple sampling points, and preset vectors corresponding to the local face triangle mesh for each face region are input into the pre-trained face generation model to obtain the rendered image corresponding to each face region output by the face generation model. By inputting the vertices of the local face triangle mesh corresponding to each face part, multiple sampling points of the local face triangle mesh, and preset vectors into a pre-trained face generation model, the rendered image corresponding to each face part output by the face generation model can be obtained. This ensures that the face parts in the generated rendered image correspond strictly to the points of the face parts on the face triangle mesh, thus guaranteeing the correspondence between the generated rendered image and the structure on the face triangle mesh.
[0045] Furthermore, in the embodiments of this application, the preset vector is a vector representing skin texture. The preset vector can be a vector representing skin texture of a face image obtained from a face image; the preset vector can also be a randomly obtained vector representing skin texture, for example, it can be a vector representing black skin texture; or it can be a vector representing yellow skin texture, without specific limitations here.
[0046] Step S140: The rendered images corresponding to the multiple face triangle meshes are stitched together to obtain the target rendered image corresponding to the face image.
[0047] In this embodiment, after obtaining the rendered image corresponding to each face part through the above method, the rendered image corresponding to each face part is stitched together with the complete rendered image corresponding to the face image to obtain the target rendered image. Specifically, when stitching the rendered image corresponding to each face part with the complete rendered image corresponding to the face image, the rendered image corresponding to each face part is placed at the corresponding face part in the complete rendered image corresponding to the face image.
[0048] This application provides an image generation method. First, multiple face triangle meshes corresponding to a face image are obtained. These multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes. The local face triangle meshes are face triangle meshes corresponding to each face region obtained from the target face triangle mesh based on facial key points. Then, multiple sampling points corresponding to each of the multiple face triangle meshes are obtained. These sampling points, multiple vertices of each face triangle mesh, and a preset vector are input into a pre-trained face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model. The preset vector is a vector representing skin texture. Finally, the rendered images corresponding to each of the multiple face triangle meshes are stitched together to obtain the target rendered image corresponding to the face image. By generating rendered images corresponding to local face regions using local face triangle meshes and then stitching these rendered images together to obtain the target rendered image, the generated target rendered image can correspond to the distribution of a real face, making it closer to the real image and improving the controllability and consistency of the generated image.
[0049] Please see Figure 4 The image generation method provided in this application embodiment is applied to, for example, Figure 1 or Figure 2 The method includes the following: (illustrated electronic device or server)
[0050] Step S210: Input the face image into a pre-trained 3D deformable face model to obtain the target face triangle mesh corresponding to the face image output by the 3D deformable face model.
[0051] In the embodiments of this application, 3D Morphable Models (3DMM) refers to a model that can reconstruct a face in three dimensions. The 3DMM model can find a set of orthonormal bases by training a set of face image samples, and linearly project the face onto the vector space composed of the orthonormal bases, so that each face image can be linearly represented by the set of orthonormal bases, and each three-dimensional face can be obtained by linear combination of many other faces, thereby transforming the face reconstruction process into using a linear combination of samples in the face database to represent the face that best matches the input face image.
[0052] In one approach, after acquiring a face image, the acquired face image can be directly input into a pre-trained 3D deformable face model, and then the 3D deformable face model can output the face triangle mesh corresponding to each face included in the face image.
[0053] In the embodiments of this application, target face triangular meshes with different expressions and shapes can be obtained through a three-dimensional deformable face model.
[0054] Step S220: Based on the target face triangle mesh, determine the facial key points corresponding to each facial part included in the face image.
[0055] As one method, based on the target face triangle mesh, determining the facial key points corresponding to each facial part included in the face image includes: inputting the target face triangle mesh into a pre-trained facial key point detection model, and obtaining the facial key points corresponding to each facial part included in the face image output by the facial key point detection model.
[0056] In this application embodiment, facial landmark detection, also known as facial landmark detection, localization, or face alignment, refers to locating the key areas of a face given a face image, including eyebrows, eyes, nose, mouth, and facial contours.
[0057] One approach is to use a pre-trained facial landmark detection model to determine the facial landmarks corresponding to each facial feature from the target face triangular mesh output by the 3D deformable face model. For example, the facial landmark detection model outputs 20 facial landmarks corresponding to the mouth, 9 facial landmarks corresponding to the nose, 12 facial landmarks corresponding to the eyes (6 for each eye), 10 facial landmarks corresponding to the eyebrows (5 for each eyebrow), and 17 contour landmarks.
[0058] The number of facial landmarks detected by the face landmark detection model can vary depending on the training dataset. For example, if the model is trained on the LFPW (Labeled Face Parts in the Wild) dataset, the output face image will contain 29 facial landmarks; if trained on the AFLW (Annotated Facial Landmarks in the Wild) dataset, the output face image will contain 21 facial landmarks; and if trained on the 300-W (300 Faces in the Wild Challenge), MVFW, or OCFW datasets, the output face image will contain 68 facial landmarks.
[0059] In this embodiment, the facial landmark detection model can be any one of the following: Active Shape Models (ASM), Active Appearance Models (AAM), Deep Convolutional Network (DCNN), Task-Constrained Deep Convolutional Network (TCDCN), Multi-task Cascaded Convolutional Networks (MTCNN), Tweaked Convolutional Neural Networks (TCNN), and Deep Alignment Networks (DAN). No specific limitation is made here.
[0060] Alternatively, the face image can be directly input into a pre-trained facial landmark detection model to obtain the facial landmarks corresponding to the face image output by the facial landmark detection model. Then, based on the facial landmarks corresponding to the face image output by the facial landmark detection model, the facial landmarks corresponding to each facial part in the face image can be determined.
[0061] Optionally, a correspondence between facial key points and vertices included in the facial triangular mesh can be established in advance. Then, after obtaining the target facial triangular mesh corresponding to the facial image through the 3D deformable facial model, the facial key points corresponding to the facial image can be determined directly from the target facial triangular mesh based on the pre-established correspondence between facial key points and vertices included in the facial triangular mesh. Furthermore, the facial key points corresponding to each facial part can also be determined.
[0062] Step S230: Based on the facial key points corresponding to each facial part, determine the facial triangle mesh corresponding to each facial part from the target facial triangle mesh to obtain the multiple local facial triangle meshes.
[0063] In this embodiment, based on the facial key points corresponding to each facial feature, a local facial triangle mesh corresponding to each facial feature is determined from the target facial triangle mesh. For example, as mentioned above, if the facial key point detection model outputs 20 facial key points corresponding to the mouth, then the facial triangle mesh corresponding to the mouth can be determined based on the coordinates of these 20 facial key points.
[0064] Step S240: Obtain the multiple line segments corresponding to each of the multiple face triangle grids, wherein the multiple line segments corresponding to each face triangle grid are line segments formed by the image acquisition device and each of the multiple vertices.
[0065] In this embodiment, the line segments formed by the image acquisition device and multiple vertices are actually line segments formed between a preset facial image synthesis viewpoint and multiple vertices. For example, if the target facial triangle mesh includes N vertices, then the image acquisition device will form N line segments with those N vertices. Since the local facial triangle mesh is obtained from the target facial triangle mesh, the number of line segments formed between the local facial triangle mesh and the image acquisition device will inevitably be less than N.
[0066] Step S250: Obtain a preset number of points from the multiple line segments corresponding to each face triangle grid to obtain multiple sampling points corresponding to each of the multiple face triangle grids.
[0067] In this embodiment, the preset number of points can be the points that need to be sampled, for example, the preset number of points can be set to M points. That is, M points are sampled from each line segment. For example, if the target face triangle mesh mentioned above includes N vertices, and the N vertices form N line segments with the image acquisition device, if M points are sampled on each line segment, then the multiple sampling points corresponding to the target face triangle mesh are N*M points.
[0068] One approach is to sample points on each line segment according to a preset line segment ratio, meaning that one point is sampled at the same location on each line segment. The line segment ratio is the ratio of the line segment formed between the sampling point and the image acquisition device to the line segment formed between the image acquisition device and the vertex.
[0069] Step S260: Input the multiple sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, and the preset vector into the pre-trained face generation model. Render the sampling points or vertices within each face region through the face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model.
[0070] In this embodiment, each local face triangle mesh can correspond to a separate face generation model; that is, one face generation model is used to generate a rendered image corresponding to one face region. For example, if a face image includes five face regions, then there are five corresponding face generation models. The vertices, sampling points, and preset vectors of the face triangle mesh corresponding to each face region are input into the corresponding face generation model, and the five face generation models can simultaneously output the rendered image corresponding to the corresponding face region.
[0071] In this embodiment of the application, a complete rendered image corresponding to a face image can also be generated using a separate face generation model.
[0072] In one approach, multiple face image generation models used to generate rendered images corresponding to facial features and face generation models used to generate complete rendered images corresponding to facial features can execute tasks in parallel; alternatively, after the face generation model used to generate complete rendered images corresponding to facial features has completed its task, multiple face image generation models used to generate rendered images corresponding to facial features can then execute their tasks in parallel, without any specific limitations.
[0073] In the embodiments of this application, during the process of generating the corresponding rendered image through the face generation model, the face generation model can first determine whether each point is within the corresponding face part. If it is, then the point is rendered.
[0074] Before inputting the vertices included in each face triangle mesh into the pre-trained face generation model, the coordinates of the vertices included in each face triangle mesh can be normalized to the range of -1 to 1.
[0075] Step S270: The rendered images corresponding to the multiple face triangle meshes are stitched together to obtain the target rendered image corresponding to the face image.
[0076] This application provides an image generation method that generates rendering images corresponding to local face parts using local face triangular meshes, and then stitches the generated rendering images corresponding to local face parts together to obtain a target rendering image. This method can make the generated target rendering image correspond to the distribution of real faces, making the generated target rendering image closer to the real image and improving the controllability and consistency of the generated image.
[0077] Please see Figure 5 The image generation method provided in this application embodiment is applied to, for example, Figure 1 or Figure 2 The method includes the following: (illustrated electronic device or server)
[0078] Step S310: Obtain the training image dataset, which includes multiple real face images and multiple rendered face images.
[0079] In this embodiment of the application, the training image dataset can be pre-stored in a preset database or cloud server. When it is necessary to obtain the training image dataset, the corresponding training image dataset can be obtained from the preset database or cloud server.
[0080] As one approach, since the face generation model used to generate the rendered image corresponding to each face triangle mesh can be different, the training image datasets used to train the face generation models for generating different rendered images can also be different. For example, the training image dataset for the model used to generate the rendered image corresponding to the local face triangle mesh corresponding to the mouth can include multiple real mouth images and multiple rendered mouth images.
[0081] Step S320: Input the training image dataset into the initial neural network, and iteratively train the initial neural network model until the training termination condition is met to obtain the face generation model.
[0082] One approach involves inputting the training image dataset into an initial neural network and iteratively training the initial neural network model until the training termination condition is met to obtain the face generation model. This includes: inputting the training image dataset into the discriminator and training the discriminator to obtain a trained discriminator; and training the initial neural network against the gradient based on the trained discriminator until the training termination condition is met to obtain the face generation model.
[0083] In the embodiments of this application, the training termination condition may be that the number of iterations reaches a preset number, or that the similarity between the rendered image output by the face generation model and the real image is greater than a preset similarity, etc., without specific limitations.
[0084] One approach is to use real face images and rendered face images from the training dataset as the training set for the discriminator, train the discriminator, and then use the trained discriminator to train the initial radiative neural field model inversely until the number of iterations meets the preset number, and the training termination condition is met, thus obtaining a face generation model for generating the rendered image corresponding to the face image.
[0085] As another approach, real face images and rendered face images from the training dataset are used as the training set for the discriminator. The discriminator is trained, and then the initial radiative neural field model is trained using the inverse gradient of the trained discriminator until the number of iterations meets the preset number. Once the training termination condition is met, a face generation model for generating rendered images corresponding to face parts is obtained.
[0086] Step S330: Obtain multiple face triangle meshes corresponding to the face image, wherein the multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes, and the multiple local face triangle meshes are face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on face key points.
[0087] Step S340: Obtain multiple sampling points corresponding to each of the multiple face triangle meshes.
[0088] Step S350: Input the multiple sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, and the preset vector into the pre-trained face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture.
[0089] In the embodiments of this application, while generating the corresponding rendered image, the face generation model can also output facial key point information of each facial part and mask image of each facial part, etc., without specific limitations.
[0090] When generating the mask image corresponding to each facial feature, the face generation model renders the triangular faces in the face triangle mesh and the triangular faces corresponding to the facial features onto the 2D image, thereby obtaining the mask image of the corresponding facial feature.
[0091] Optionally, the sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, the preset vector, and the vector representing the preset background features are respectively input into a pre-trained face generation model to obtain the rendered images with preset backgrounds corresponding to each of the multiple face triangle meshes output by the face generation model.
[0092] Step S360: The rendered images corresponding to the multiple face triangle meshes are stitched together to obtain the target rendered image corresponding to the face image.
[0093] This application provides an image generation method that generates rendering images corresponding to local face parts using local face triangular meshes, and then stitches the generated rendering images corresponding to local face parts together to obtain a target rendering image. This method can make the generated target rendering image correspond to the distribution of real faces, making the generated target rendering image closer to the real image and improving the controllability and consistency of the generated image.
[0094] Please see Figure 6 This application provides an image generation apparatus 400, which includes:
[0095] The mesh acquisition unit 410 is used to acquire multiple face triangle meshes corresponding to a face image. The multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes. The multiple local face triangle meshes are face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on face key points.
[0096] In one approach, the mesh acquisition unit 410 is specifically used to input the face image into a pre-trained three-dimensional deformable face model to obtain the target face triangle mesh corresponding to the face image output by the three-dimensional deformable face model; based on the target face triangle mesh, determine the facial key points corresponding to each face part included in the face image; based on the facial key points corresponding to each face part, determine the face triangle mesh corresponding to each face part from the target face triangle mesh to obtain the multiple local face triangle meshes.
[0097] Furthermore, the mesh acquisition unit 410 is also used to input the target face triangular mesh into a pre-trained face key point detection model to acquire the face key points corresponding to each face part included in the face image output by the face key point detection model.
[0098] The sampling unit 420 is used to acquire multiple sampling points corresponding to each of the multiple face triangle grids.
[0099] In one approach, the sampling unit 420 is specifically used to acquire multiple line segments corresponding to each of the multiple face triangle grids, wherein the multiple line segments corresponding to each face triangle grid are line segments formed by the image acquisition device and each of the multiple vertices; and to acquire a preset number of points from the multiple line segments corresponding to each face triangle grid to obtain multiple sampling points corresponding to each of the multiple face triangle grids.
[0100] Image generation unit 430 is used to input multiple sampling points corresponding to each of the multiple face triangle meshes, multiple vertices corresponding to each face triangle mesh, and a preset vector into the face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture.
[0101] In one approach, the image generation unit 430 is specifically used to input the multiple sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, and the preset vector into a pre-trained face generation model. The face generation model then renders the sampling points or vertices within each face region to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model.
[0102] The stitching unit 440 is used to stitch together the rendered images corresponding to the multiple face triangle meshes to obtain the target rendered image corresponding to the face image.
[0103] Please see Figure 7 The device 400 may further include:
[0104] The model training unit 450 is used to acquire a training image dataset, which includes multiple real face images and multiple rendered face images; the training image dataset is input into an initial neural network, and the initial neural network model is iteratively trained until the training termination condition is met, thereby obtaining the face generation model.
[0105] In one approach, the model training unit 450 is specifically used to input the training image dataset into the discriminator, train the discriminator to obtain a trained discriminator, and then train the initial neural network in reverse gradient based on the trained discriminator until the training termination condition is met to obtain the face generation model.
[0106] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.
[0107] The following will combine Figure 8 This application describes an electronic device or server provided.
[0108] Please see Figure 8Based on the image generation method and apparatus described above, this application also provides another electronic device or server 800 capable of executing the aforementioned image generation method. The electronic device or server 800 includes one or more (only one shown in the figure) processors 802, a memory 804, and a network module 806 coupled together. The memory 804 stores a program capable of executing the contents of the aforementioned embodiments, and the processor 802 can execute the program stored in the memory 804.
[0109] The processor 802 may include one or more processing cores. The processor 802 connects to various parts of the server 800 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 804, and by calling data stored in the memory 804. Optionally, the processor 802 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 802 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 802 and may be implemented separately using a communication chip.
[0110] The memory 804 may include random access memory (RAM) or read-only memory (ROM). The memory 804 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 804 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the electronic device or server 800 (such as phonebooks, audio and video data, chat log data, etc.).
[0111] The network module 806 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, thereby communicating with communication networks or other devices, such as audio playback devices. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM cards), memory, etc. The network module 806 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices through wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). For example, the network module 806 can interact with base stations.
[0112] Please refer to Figure 9 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 900 stores program code that can be called by a processor to execute the methods described in the above method embodiments.
[0113] The computer-readable storage medium 900 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 910 may be compressed, for example, in a suitable form.
[0114] This application provides an image generation method, apparatus, electronic device, and storage medium. First, multiple face triangle meshes corresponding to a face image are acquired. These multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes. The multiple local face triangle meshes are face triangle meshes corresponding to each face region obtained from the target face triangle mesh based on facial key points. Then, multiple sampling points corresponding to each of the multiple face triangle meshes are acquired. These sampling points, multiple vertices of each face triangle mesh, and a preset vector are input into a pre-trained face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model. The preset vector is a vector representing skin texture. Finally, the rendered images corresponding to each of the multiple face triangle meshes are stitched together to obtain the target rendered image corresponding to the face image. By using the above method, local face triangular meshes are used to generate rendering images corresponding to local face parts. These rendering images are then stitched together to obtain the target rendering image. This method ensures that the generated target rendering image corresponds to the distribution of real faces, making it closer to the real image and improving the controllability and consistency of the generated image.
[0115] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.
Claims
1. An image generation method, characterized in that, The method includes: Multiple face triangle meshes corresponding to a face image are obtained, wherein the multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes, and the multiple local face triangle meshes are face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on face key points; The acquisition of multiple face triangle meshes corresponding to the face image includes: The face image is input into a pre-trained 3D deformable face model to obtain the target face triangle mesh corresponding to the face image output by the 3D deformable face model. Based on the target face triangle mesh, determine the facial key points corresponding to each facial part included in the face image; Based on the facial key points corresponding to each facial part, the facial triangle mesh corresponding to each facial part is determined from the target facial triangle mesh to obtain the multiple local facial triangle meshes. Obtain multiple sampling points corresponding to each of the multiple face triangle meshes; The sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, and the preset vector are respectively input into a pre-trained face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture; The rendered images corresponding to the multiple face triangle meshes are stitched together to obtain the target rendered image corresponding to the face image.
2. The method according to claim 1, characterized in that, The step of inputting the sampling points corresponding to each of the plurality of face triangle meshes, the vertices corresponding to each face triangle mesh, and the preset vector into a pre-trained face generation model to obtain the rendered images corresponding to each of the plurality of face triangle meshes output by the face generation model includes: The sampling points corresponding to each of the multiple face triangle meshes, the multiple vertices corresponding to each face triangle mesh, and the preset vector are respectively input into a pre-trained face generation model. The face generation model renders the sampling points or vertices within each face region to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model.
3. The method according to claim 1, characterized in that, The step of determining the facial key points corresponding to each facial region included in the facial image based on the target facial triangular mesh includes: The target face triangular mesh is input into a pre-trained face landmark detection model to obtain the face landmarks corresponding to each face part included in the face image output by the face landmark detection model.
4. The method according to claim 1, characterized in that, Each face triangle mesh includes multiple vertices, and obtaining the multiple sampling points corresponding to each of the multiple face triangle meshes includes: Each of the multiple face triangle meshes is obtained by acquiring multiple line segments corresponding to each face triangle mesh, wherein the multiple line segments corresponding to each face triangle mesh are line segments formed by the image acquisition device and each of the multiple vertices respectively; A preset number of points are obtained from the multiple line segments corresponding to each face triangle grid to obtain multiple sampling points corresponding to each of the multiple face triangle grids.
5. The method according to claim 1, characterized in that, Before inputting the sampling points corresponding to each of the plurality of face triangle meshes, the vertices corresponding to each face triangle mesh, and the preset vector into the pre-trained face generation model to obtain the rendered images corresponding to each of the plurality of face triangle meshes output by the face generation model, the method further includes: Obtain a training image dataset, which includes multiple real face images and multiple rendered face images; The training image dataset is input into the initial neural network, and the initial neural network model is iteratively trained until the training termination condition is met, thus obtaining the face generation model.
6. The method according to claim 5, characterized in that, The initial neural network includes a discriminator. The process of inputting the training image dataset into the initial neural network and iteratively training the initial neural network model until the training termination condition is met, thereby obtaining the face generation model, includes: The training image dataset is input into the discriminator to train the discriminator and obtain a trained discriminator. The initial neural network is trained in reverse gradient based on the trained discriminator until the training termination condition is met, thus obtaining the face generation model.
7. An image generation apparatus, characterized in that, The device includes: A mesh acquisition unit is used to acquire multiple face triangle meshes corresponding to a face image. The multiple face triangle meshes include a target face triangle mesh and multiple local face triangle meshes. The multiple local face triangle meshes are face triangle meshes corresponding to each face part obtained from the target face triangle mesh based on facial key points. Acquiring multiple face triangle meshes corresponding to a face image includes: inputting the face image into a pre-trained 3D deformable face model to acquire the target face triangle mesh corresponding to the face image output by the 3D deformable face model; determining the facial key points corresponding to each face part included in the face image based on the target face triangle mesh; and determining the face triangle mesh corresponding to each face part from the target face triangle mesh based on the facial key points corresponding to each face part to obtain the multiple local face triangle meshes. The sampling unit is used to acquire multiple sampling points corresponding to each of the multiple face triangle grids; An image generation unit is used to input multiple sampling points corresponding to each of the multiple face triangle meshes, multiple vertices corresponding to each face triangle mesh, and a preset vector into a face generation model to obtain the rendered images corresponding to each of the multiple face triangle meshes output by the face generation model, wherein the preset vector is a vector representing skin texture. The stitching unit is used to stitch together the rendered images corresponding to the multiple face triangle meshes to obtain the target rendered image corresponding to the face image.
8. An electronic device, characterized in that, It includes one or more processors; one or more programs are stored in memory and configured to be executed by the one or more processors according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method according to any one of claims 1-6.