Image-based three-dimensional reconstruction and model training method, device and storage medium

Abstract

Embodiments of the present application provide a three-dimensional reconstruction method based on images, a model training method and device, and a storage medium. The three-dimensional reconstruction method based on images comprises: obtaining first, second and third frequency face shape features corresponding to a two-dimensional face image respectively, wherein the frequency is used to indicate the change rate of the face shape feature, the first frequency is lower than the second frequency, and the second frequency is lower than the third frequency; generating a corresponding basic three-dimensional face grid according to the first frequency face shape feature; and performing feature refinement on the basic three-dimensional face grid according to the second and third frequency face shape features to generate a target three-dimensional face grid; and generating a three-dimensional face model corresponding to the two-dimensional face image according to the target three-dimensional face grid. Through the embodiments of the present application, the generated three-dimensional face model is closer to the real face reflected by the two-dimensional face image, and the embodiments of the present application are suitable for application in the fields of virtual reality, augmented reality, three-dimensional modeling, etc.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image-based 3D reconstruction method, a 3D reconstruction model training method, an electronic device, and a computer storage medium. Background Technology

[0002] 3D reconstruction is a technique that uses data such as 2D images and laser point clouds to reconstruct the 3D structure of objects. 3D face reconstruction is an important branch of 3D reconstruction. It aims to reconstruct and restore the 3D structure of a face using 2D face images or laser point cloud data, and can be widely used in fields such as medicine, beauty, gaming, live streaming, video conferencing, and film production.

[0003] Currently, most 3D face reconstruction methods are based on single images. That is, a single face image is used to construct a corresponding 3D face mesh and texture, reconstructing the 3D structural information of the face. This method of 3D face reconstruction based on a single face image mostly utilizes statistical models of the face, obtaining a series of shape bases representing the main features of the face through methods such as principal component analysis, and then performing linear weighting to obtain the final 3D face. Although this method can roughly fit a face in a 2D image, it is limited to a coarse-grained, low-dimensional representation and cannot reconstruct detailed features such as facial depressions and wrinkles, making it difficult to apply to scenarios with high accuracy requirements. Summary of the Invention

[0004] In view of this, embodiments of this application provide a three-dimensional reconstruction scheme to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the present application, an image-based three-dimensional reconstruction method is provided, comprising: acquiring face shape features at a first frequency, a second frequency, and a third frequency corresponding to a two-dimensional face image, wherein the frequency is used to indicate the rate of change of the face shape features, the first frequency is lower than the second frequency, and the second frequency is lower than the third frequency; generating a corresponding basic three-dimensional face mesh based on the face shape features at the first frequency; refining the basic three-dimensional face mesh based on the face shape features at the second and third frequencies to generate a target three-dimensional face mesh; and generating a three-dimensional face model corresponding to the two-dimensional face image based on the target three-dimensional face mesh.

[0006] According to a second aspect of the present application, a method for training an image-based 3D reconstruction model is provided, comprising: acquiring training samples for training the 3D reconstruction model, wherein the training samples include at least: two-dimensional face image samples; inputting the two-dimensional face image samples into the 3D reconstruction model to obtain face shape features at a first frequency, a second frequency, and a third frequency corresponding to the two-dimensional face image samples, wherein the frequency is used to indicate the rate of change of the face shape features, the first frequency is lower than the second frequency, and the second frequency is lower than the third frequency; predicting and generating a corresponding basic 3D face mesh sample based on the face shape features at the first frequency; refining the features of the basic 3D face mesh sample based on the face shape features at the second and third frequencies to generate a target 3D face mesh sample; generating a 3D face model sample corresponding to the two-dimensional face image samples based on the target 3D face mesh sample; and training the 3D reconstruction model based on a rendered image of the 3D face model sample and a preset loss function.

[0007] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or second aspect.

[0008] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0009] According to the 3D reconstruction scheme provided in this application, when reconstructing a 3D face based on a single 2D face image, multiple levels and granularities of facial features are considered simultaneously, especially facial shape features, namely, first, second, and third frequency facial shape features. The first frequency facial shape features can express lower-dimensional features, allowing for the construction of a relatively coarse basic 3D face mesh. The second frequency facial shape features can characterize medium-granularity features in local facial regions such as the cheeks and cheekbones, allowing for a certain degree of refinement of the basic 3D face mesh, making it more closely resemble a real face. Furthermore, the third frequency facial shape features can characterize finer facial details such as wrinkles, corners of the eyes, and corners of the mouth, further refining the basic 3D face mesh and making the generated target 3D face mesh even closer to a real face. Based on this, rendering the target 3D face mesh yields the final 3D face model, which will approximate the real face reflected in the 2D face image to the greatest extent possible. Therefore, this 3D face model can be effectively applied to scenarios where high accuracy in 3D face reconstruction is required. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0011] Figure 1 A schematic diagram of an exemplary system for solutions applicable to embodiments of this application;

[0012] Figure 2A This is a flowchart illustrating the steps of an image-based 3D reconstruction method according to Embodiment 1 of this application.

[0013] Figure 2B for Figure 2A A schematic diagram of the structure of a 3D reconstruction model for 3D face reconstruction in the embodiment shown;

[0014] Figure 2C for Figure 2A A schematic diagram of a scenario example in the illustrated embodiment;

[0015] Figure 3A The following is a flowchart of the steps of an image-based 3D reconstruction model training method according to Embodiment 2 of this application;

[0016] Figure 3B for Figure 3A A schematic diagram of a scenario example in the illustrated embodiment;

[0017] Figure 4 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0019] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0020] Figure 1 An exemplary system applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example shown uses multiple user devices. It should be noted that the solution in this embodiment can be implemented either in the cloud server 102 or on a user device 106 with certain hardware and software capabilities.

[0021] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, when the cloud server 102 implements the scheme of the embodiments of this application, in some embodiments, the cloud server 102 can be used to generate a corresponding three-dimensional face model based on a single two-dimensional face image. As an optional example, in some embodiments, the cloud server 102 can extract face shape features at a first frequency, a second frequency, and a third frequency based on the two-dimensional face image, wherein the first frequency is lower than the second frequency, the second frequency is lower than the third frequency, and the frequency can indicate the rate of change of the face shape features; then, different three-dimensional face processing is performed according to the face shape features of different frequencies, such as generating a basic three-dimensional face mesh based on the face shape features of the first frequency, refining the basic three-dimensional face mesh based on the face shape features of the second and third frequencies to generate a refined target three-dimensional face mesh, and finally generating a three-dimensional face model based on the target three-dimensional face mesh. As another example, in some embodiments, the above-mentioned 3D face model is implemented through a 3D reconstruction model, and the cloud server 102 can also train the 3D reconstruction model. As another example, in some embodiments, the cloud server 102 can perform 3D face reconstruction based on the 3D face reconstruction request sent by the user device 106 and the corresponding 2D face image, and then send the reconstruction result back to the user device 106.

[0022] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user equipment 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user equipment 106 and the cloud server 102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0023] User equipment 106 may include any one or more user devices suitable for interacting with a user. In some embodiments, when user equipment 106 implements the scheme of the embodiments of this application, user equipment 106 can be used to generate a corresponding three-dimensional face model based on a single two-dimensional face image. As an optional example, in some embodiments, user equipment 106 can extract face shape features of a first frequency, a second frequency, and a third frequency based on the two-dimensional face image; then, perform different three-dimensional face processing according to the face shape features of different frequencies, such as generating a basic three-dimensional face mesh based on the face shape features of the first frequency, refining the basic three-dimensional face mesh based on the face shape features of the second and third frequencies to generate a refined target three-dimensional face mesh, and finally generating a three-dimensional face model based on the target three-dimensional face mesh. As another example, in some embodiments, when the cloud server 102 performs three-dimensional face reconstruction, user equipment 106 can send a three-dimensional face reconstruction request and a corresponding two-dimensional face image to the cloud server 102, and the cloud server 102 performs three-dimensional face reconstruction, and user equipment 106 obtains the three-dimensional face reconstruction result from the cloud server 102. User equipment 106 may include any suitable type of device. For example, in some embodiments, user equipment 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, and / or any other suitable type of user equipment.

[0024] Based on the above system, the following embodiments of this application illustrate the process of 3D face reconstruction and model training.

[0025] Example 1

[0026] This embodiment focuses on explaining the solution of this application embodiment from the perspective of the generation process of the three-dimensional face model.

[0027] Reference Figure 2A The diagram shows a flowchart of the steps of an image-based three-dimensional reconstruction method according to Embodiment 1 of this application.

[0028] The three-dimensional reconstruction method in this embodiment includes the following steps:

[0029] Step S202: Obtain the face shape features at the first frequency, second frequency, and third frequency corresponding to the two-dimensional face image.

[0030] In this embodiment, a two-dimensional face image refers to a two-dimensional image in which the face occupies the majority of the image. Preferably, it can be a two-dimensional face image with high image quality, such as an image whose clarity meets a certain clarity threshold. The higher the image quality, the more accurate the subsequently constructed three-dimensional face model will be. Furthermore, in this embodiment, the specific format of the two-dimensional face image is not limited. Unless otherwise specified, the solutions provided in this embodiment are all based on a single two-dimensional face image for three-dimensional face reconstruction.

[0031] Furthermore, in this embodiment, the frequency is used to indicate the rate of change of facial shape features, and can characterize the changes in facial geometry. The first frequency is lower than the second frequency, and the second frequency is lower than the third frequency. Because facial shape changes in space, the third frequency can characterize areas where facial geometry changes rapidly, such as wrinkles around the eyes; the second frequency, being slightly slower, can characterize areas where facial geometry changes more slowly, such as facial depressions; and the first frequency can characterize areas where facial geometry is relatively uniform or changes very slowly, such as the facial contour, etc. The specific setting of the frequency can be flexibly set by those skilled in the art based on the above criteria, as long as it can effectively characterize the corresponding geometric changes. This embodiment does not limit the specific value or range of the frequency. In one feasible approach, the first frequency of facial shape features may include one or more (two or more) of the following: facial illumination coefficient features, device coefficient features of the image acquisition device corresponding to the 2D facial image, facial pose coefficient features, facial shape coefficient features, facial expression coefficient features, and facial texture coefficient features; the second frequency of facial shape features may include features used to indicate changes in local facial regions; and the third frequency of facial shape features may include features used to indicate changes in facial facial features. Thus, facial features can be characterized in detail from multiple different levels and dimensions, providing effective data for subsequent 3D facial reconstruction.

[0032] Furthermore, in one feasible approach, this step can obtain the first, second, and third frequency facial shape features corresponding to the two-dimensional facial image through a three-dimensional reconstruction model used for three-dimensional facial reconstruction.

[0033] For example, an exemplary structure of the three-dimensional reconstruction model is as follows: Figure 2B As shown in the figure, the 3D reconstruction model includes: a first encoder, a first decoder corresponding to the first encoder, a second encoder, a second decoder and a third encoder corresponding to the second encoder, a feature fusion layer, and an output layer.

[0034] in:

[0035] The first encoder is used to extract face shape features of the first frequency from a two-dimensional face image to obtain face shape features of the first frequency.

[0036] The first decoder is used to generate a corresponding basic 3D face mesh based on the face shape features of the first frequency, and to generate a face texture map.

[0037] For example, the first encoder and the first decoder can be implemented using encoder and decoder structures in a three-dimensional face statistical model, such as a three-dimensional deformable face model (3DMM). However, this is not a limitation; other encoder-decoder structures that can implement the functions of the first encoder and the first decoder described above are also applicable to the embodiments of this application.

[0038] The second encoder is used to extract face shape features at the second and third frequencies from a two-dimensional face image to obtain the corresponding face shape features at the second and third frequencies.

[0039] The second encoder can have the same structure as the first decoder, or it can have a different structure.

[0040] The second decoder is used to generate the corresponding vertex offset map of the face based on the face shape features of the second frequency.

[0041] The third decoder is used to generate the corresponding normal displacement map of the face based on the face shape features of the third frequency.

[0042] For example, the second decoder and the third decoder can be decoders with the same structure or decoders with different structures, and can have the same or different structures as the first decoder.

[0043] The feature fusion layer is used to refine the features of the basic 3D face mesh based on the vertex offset map and normal displacement map, and generate the target 3D face mesh.

[0044] The output layer is used to generate a 3D face model corresponding to the 2D face image based on the target 3D face mesh and face texture map.

[0045] The specific functions of each part of the 3D reconstruction model will be illustrated with examples and steps below.

[0046] based on Figure 2B In the three-dimensional reconstruction model shown, in this step, the first encoder in the three-dimensional reconstruction model can be used to extract shape features from the two-dimensional face image to obtain the face shape features of the first frequency; the second encoder can be used to extract shape features from the two-dimensional face image to obtain the face shape features of the second frequency and the face shape features of the third frequency.

[0047] Step S204: Generate a basic three-dimensional face mesh based on the face shape features of the first frequency; and refine the basic three-dimensional face mesh based on the face shape features of the second and third frequencies to generate the target three-dimensional face mesh.

[0048] A 3D mesh is a type of data defined by a set of polygons, used to represent the topological and spatial structure of the surface contour of a 3D model. A 3D face mesh is a 3D mesh used to represent a 3D face model. In this embodiment, lower-frequency face shape features, i.e., first-frequency face shape features, are used to generate the corresponding 3D face mesh. For example, a 3D face statistical model, such as a 3DMM model, capable of extracting first-frequency face shape features can be used to generate the basic 3D face mesh.

[0049] 3DMM, or Deformable Face Model, is a 3D face representation model based on statistical methods. It represents the face structure using a unified topological 3D mesh and obtains principal component information of faces by statistically analyzing a large number of 3D face structures. Then, it uses low-dimensional coefficients to represent different faces. In practical applications, there are various algorithms for fitting 2D (2-dimensional) faces onto 3D (3-dimensional) models using 3DMM, all of which are applicable to the embodiments of this application and will not be detailed here.

[0050] However, the three-dimensional face mesh obtained through the 3DMM model, i.e. the basic three-dimensional face mesh, is relatively coarse. In order to fit the face more accurately, in this embodiment, the basic three-dimensional face mesh is further refined based on the face shape features of the second and third frequencies to obtain a fine three-dimensional face mesh, i.e. the target three-dimensional face mesh.

[0051] Based on this, in one feasible approach, this step can be implemented as follows: generating a corresponding basic 3D face mesh and face texture map based on the face shape features of the first frequency; generating a corresponding vertex offset map of the face based on the face shape features of the second frequency; generating a corresponding normal displacement map of the face based on the face shape features of the third frequency; and refining the basic 3D face mesh based on the vertex offset map and the normal displacement map to generate a target 3D face mesh.

[0052] When adopting such Figure 2BWhen the 3D reconstruction model shown performs 3D face reconstruction, the first decoder in the 3D reconstruction model can decode the face shape features of the first frequency to generate the corresponding basic 3D face mesh; and, based on the face shape features of the first frequency, generate the corresponding face texture map; the second decoder, targeting the face shape features of the second frequency, decodes the face shape features of the second frequency to generate the corresponding vertex offset map of the face; and the third decoder, targeting the face shape features of the third frequency, decodes the face shape features of the third frequency to generate the corresponding normal displacement map of the face. As mentioned above, in the embodiments of this application, there are no restrictions on the specific structure and algorithm implementation of each encoder and decoder, as long as the corresponding functions can be achieved. Through the encoder-decoder structure, the corresponding face shape features can be effectively extracted, and the required information map can be decoded by the decoder, making the overall solution efficient and the data accurate.

[0053] The vertex offset map of a face is a type of data used to identify and describe a face using vertices. The position and number of face vertices can be determined by conventional conventions or based on a 3D face mesh. For example, the vertices can be the vertices of a polygon, such as a triangle, that forms the basic 3D face mesh. The vertex offset map describes the 3D position of each vertex of the face, and based on this, offset information between different vertices can be obtained.

[0054] A normal displacement map is a texture map, but the points in this texture map do not correspond to color data such as R (Red), G (Green), and B (Blue) data (RGB color data is obtained by varying the R, G, and B color channels and superimposing them). Instead, they are based on coordinates along the x-axis (horizontal axis), y-axis (vertical axis), and z-axis (vertical axis), which define a normal direction. This allows for the description of more detailed geometric information about the face, such as wrinkles.

[0055] Based on this, the basic 3D face mesh can be deformed according to the vertex offset map to obtain the deformed 3D face mesh; and the normal map can be applied to the deformed 3D face mesh according to the normal displacement map to obtain the target 3D face mesh. The target 3D face mesh obtained in this way has more refined features and can more accurately represent the 3D face model.

[0056] The deformation of the basic 3D face mesh based on the vertex offset map can be achieved as follows: Based on the vertex offset map, the basic 3D face mesh is deformed through UV (U-axis and V-axis of the texture coordinate system, where U represents the distribution of texture in the horizontal coordinate and V represents the distribution of texture in the vertical coordinate) mapping to obtain the deformed 3D face mesh. UV is a two-dimensional texture coordinate system containing vertex component information of polygons and subdivided surface meshes, also known as the UV texture space. The UV texture space uses the letters U and V to indicate axes in two-dimensional space. The process of placing an image texture map on a 3D surface to create explicit UVs for the surface mesh is called UV mapping. However, as mentioned earlier, in this embodiment, the vertex offset map is not a texture map, but an offset map representing the offset of vertex coordinate positions. Therefore, based on the principle of UV mapping, the information carried in the vertex offset map can be mapped onto the basic 3D face mesh, realizing the deformation of the basic 3D face mesh and obtaining a more accurate 3D face mesh.

[0057] The normal map, on the other hand, can be a normal map applied to the basic 3D face mesh. Normal maps are special textures that can be applied to 3D surfaces, unlike previous textures which could only be used on 2D surfaces. As an extension of bump mapping, it gives each pixel on each plane a height value, containing a lot of detailed surface information, and can create a stereoscopic visual effect on the object's shape. In this embodiment, through the normal map, normals can be drawn at every point on the bumpy surface of the basic 3D face mesh. The direction of the normals is marked by the aforementioned xyz coordinates to depict more refined facial geometric details, such as wrinkles or local protrusions and depressions.

[0058] After refining the basic 3D face mesh using vertex offset map and normal displacement map, a more detailed target 3D face mesh that better matches the real face can be obtained.

[0059] Step S206: Generate a 3D face model corresponding to the 2D face image based on the target 3D face mesh.

[0060] For example, a 3D face model corresponding to a 2D face image can be generated based on the target 3D face mesh and the face texture map obtained based on the face shape features (especially the texture features) of the first frequency.

[0061] The following is a specific example to illustrate the above process, such as... Figure 2C As shown in the image.

[0062] Figure 2C In the example shown, a 3D reconstruction model is used to implement the above 3D reconstruction scheme. The structure of the 3D reconstruction model can be as follows: Figure 2B As shown in the image.

[0063] Based on this, when performing 3D face reconstruction, the encoder (i.e., the first encoder) of the 3DMM model in the 3D reconstruction model can first encode the 2D face image downsampled to 256x256, obtaining coefficients for shape, expression, texture, lighting, camera, pose, etc. (i.e., the first frequency face shape features) in different dimensions. Among them, the shape coefficient and expression coefficient can be used to obtain a relatively coarse face geometric mesh, i.e., the basic 3D face mesh, through the decoder of the 3DMM model.

[0064] However, the facial geometry mesh at this point (i.e., the basic 3D facial mesh) is only derived from low-dimensional shape and expression coefficients combined with the baseline weighting of the 3DMM shape base, which cannot accurately fit the facial contours and high-frequency details such as wrinkles. Therefore, in this example, while the encoder of the 3DMM model encodes the 2D facial image, a "detail encoder" (i.e., the second encoder) is used to encode the 2D facial image to obtain latent variables (i.e., the second and third frequency facial shape features) that contain mid- and high-frequency details of the face.

[0065] Then, the corresponding "mid-frequency decoder" (i.e., the second decoder) and "high-frequency decoder" (i.e., the third decoder) are used to decode these two latent variables representing mid- and high-frequency details, respectively, to obtain the vertex offset map and the normal displacement map. The vertex offset map represents the mid-frequency geometric information of the face and is used to perform moderate deformation on the basic face shape (a coarse basic 3D face mesh). For example, through UV mapping, the vertex offset map can be applied to each vertex in the coarse basic 3D face mesh to deform the face and obtain a more accurate facial contour. Correspondingly, the normal displacement map is applied to the normal map of the 3D model to depict more refined geometric details of the face, such as wrinkles and other local protrusions and depressions. Through the vertex offset map and the normal displacement map, the mid- and high-frequency details that low-frequency facial shape features cannot express can be compensated for, resulting in a more accurate reconstruction of the face and a more refined target 3D face mesh.

[0066] Finally, the refined 3D mesh of the target face can be combined with the texture map generated by the aforementioned texture coefficients to obtain the reconstructed 3D face model.

[0067] As can be seen, this embodiment, when reconstructing a 3D face based on a single 2D face image, considers multiple levels and granularities of facial features, especially facial shape features, namely, the first, second, and third frequencies of facial shape features. The first frequency facial shape features express lower-dimensional features, allowing for the construction of a relatively coarse basic 3D face mesh. The second frequency facial shape features characterize medium-granularity features in local facial regions such as the cheeks and cheekbones, enabling a certain degree of refinement of the basic 3D face mesh to better resemble a real face. Furthermore, the third frequency facial shape features characterize finer facial details such as wrinkles, corners of the eyes, and corners of the mouth, further refining the basic 3D face mesh and making the generated target 3D face mesh even closer to a real face. Based on this, rendering the target 3D face mesh yields the final 3D face model, which will approximate the real face reflected in the 2D face image as closely as possible. Therefore, this 3D face model can be effectively applied to scenarios where high accuracy in 3D face reconstruction is required.

[0068] Example 2

[0069] This embodiment focuses on the training of the 3D reconstruction model, combined with... Figure 3A and Figure 3B The solutions provided in the embodiments of this application will be described.

[0070] Reference Figure 3A The diagram shows a flowchart of the steps of an image-based 3D reconstruction model training method according to Embodiment 2 of this application.

[0071] The image-based 3D reconstruction model training method in this embodiment includes the following steps:

[0072] Step S302: Obtain training samples for training the 3D reconstruction model.

[0073] In this embodiment, the three-dimensional reconstruction model can be as follows: Figure 2B The model structure shown may include, in addition to a first encoder, a first decoder corresponding to the first encoder, a second encoder, a second decoder corresponding to the second encoder, and a third encoder, a discriminator corresponding to the second decoder, as well as a feature fusion layer and an output layer.

[0074] By using training samples, each part of the above-mentioned 3D reconstruction model can be trained to obtain the final model that can perform 3D face reconstruction based on 2D face images.

[0075] The training samples include at least two-dimensional face image samples, the format of which is not restricted.

[0076] Optionally, to better train the second encoder-decoder-discriminator branch, the training samples may also include ground truth values ​​of vertex offset maps of faces obtained from the 3D face model corresponding to the 2D face image samples. In this case, not only are there 2D face image samples, but also corresponding, relatively accurate 3D face models. Based on this 3D face model, features can be extracted using any appropriate method, including but not limited to neural network models, to obtain the vertex offset map of the face corresponding to the 3D face model. This map is then used as ground truth values ​​for training the subsequent 3D reconstruction model, especially the second encoder-decoder-discriminator branch in the 3D reconstruction model.

[0077] Step S304: Input the two-dimensional face image sample into the three-dimensional reconstruction model to obtain the face shape features of the first frequency, second frequency and third frequency corresponding to the two-dimensional face image sample respectively.

[0078] The frequency is used to indicate the rate of change of facial shape features. The first frequency is lower than the second frequency, and the second frequency is lower than the third frequency. The third frequency can characterize areas of rapid geometric change of the face, such as wrinkles at the corners of the eyes; the second frequency, being slightly slower, can characterize areas of slower geometric change of the face, such as facial depressions; the first frequency can characterize areas of relatively uniform or very slow-changing facial geometry, such as the outline of the face, etc. The specific setting of the frequency can be flexibly set by those skilled in the art based on the above benchmarks, as long as it can effectively characterize the corresponding geometric shape changes. The specific value or range of the frequency is not limited in the embodiments of this application. In one feasible approach, the facial shape features of the first frequency may include one or more of the following: facial illumination coefficient features, device coefficient features of the image acquisition device corresponding to the two-dimensional facial image, facial pose coefficient features, facial shape coefficient features, facial expression coefficient features, and facial texture coefficient features; the facial shape features of the second frequency may include features for indicating changes in local facial regions; and the facial shape features of the third frequency may include features for indicating changes in facial facial features. Therefore, facial features can be characterized in detail from multiple different levels and dimensions, providing effective data for subsequent 3D face reconstruction.

[0079] Specifically, in the 3D reconstruction model, the first encoder can extract the face shape features of the two-dimensional face image sample at the first frequency to obtain the face shape features of the first frequency; the second encoder can extract the face shape features of the two-dimensional face image sample at the second and third frequencies to obtain the corresponding face shape features of the second and third frequencies.

[0080] For example, such as Figure 3BAs shown, two-dimensional face images are input into the first encoder. Figure 3B The diagram shows the encoder of the 3DMM model, namely the 3DMM encoder, and the second encoder. Figure 3B The diagram illustrates a detail encoder. After processing by the 3DMM encoder, the first frequency facial shape features corresponding to the 2D face image are obtained, including: illumination coefficients (i.e., facial illumination coefficient features). Figure 3B The diagram shows (l) camera coefficients (i.e., the device coefficients of the image acquisition device corresponding to the two-dimensional face image). Figure 3B The diagram shows c), pose coefficients (i.e., facial pose coefficient features). Figure 3B The diagram in the image shows p), and the shape coefficient (i.e., the facial shape coefficient feature). Figure 3B The diagram in the image shows 's' and the expression coefficient (i.e., facial expression coefficient features). Figure 3B The diagram in the image shows e), texture coefficients (i.e., facial texture coefficient features), Figure 3B (t is used as an example in the diagram). After processing by the detail encoder, the second frequency facial shape features corresponding to the two-dimensional facial image are obtained. Figure 3B The middle frequency is represented by θ (mid-frequency detail latent variable); and the third frequency is the facial shape feature. Figure 3B The middle part is shown as (High-frequency details hidden variables).

[0081] These facial shape features at different frequencies will be input into different decoders to extract the corresponding features, feature maps, or information maps.

[0082] Step S306: Based on the face shape features of the first frequency, predict and generate the corresponding basic three-dimensional face mesh sample; and based on the face shape features of the second and third frequencies, refine the features of the basic three-dimensional face mesh sample to generate the target three-dimensional face mesh sample.

[0083] This includes: generating corresponding basic 3D face mesh samples and face texture map samples based on the face shape features of the first frequency; generating corresponding vertex offset map samples of the face based on the face shape features of the second frequency; generating corresponding normal displacement map samples of the face based on the face shape features of the third frequency; and refining the basic 3D face mesh samples based on the vertex offset map samples and normal displacement map samples to generate target 3D face mesh samples.

[0084] When combined with a 3D reconstruction model, the first decoder generates a corresponding basic 3D face mesh sample based on the face shape features of the first frequency; the second decoder generates a corresponding vertex offset map sample based on the face shape features of the second frequency; and the third decoder generates a corresponding normal displacement map sample based on the face shape features of the third frequency.

[0085] For example, such as Figure 3B As shown, the facial shape features at the first frequency were obtained, namely: illumination coefficient ( Figure 3B The diagram shows l), camera coefficient ( Figure 3B The diagram in the middle is shown as c), attitude coefficient ( Figure 3B The diagram in the image shows p), and the shape factor (p). Figure 3B The figure in the middle is s), and the expression coefficient ( Figure 3B The diagram in the middle is e), texture coefficients ( Figure 3B The second frequency of facial shape features is represented by t); Figure 3B The diagram illustrates θ (mid-frequency detail latent variable); and the third-frequency facial shape features, i.e. Figure 3B The diagram in the middle Following (high-frequency detail hidden variables), correspondingly, such as Figure 3B As shown, the 3DMM decoder (i.e., the first decoder) corresponding to the 3DMM encoder will be based at least on the shape coefficients ( Figure 3B The figure in the middle is s) and the expression coefficient ( Figure 3B (E) is shown in the diagram. The basic 3D face mesh sample is generated through decoding. Figure 3B The diagram illustrates a coarse mesh. The intermediate frequency decoder (i.e., the second decoder), corresponding to the detail encoder, is based on the second frequency facial shape features, i.e. Figure 3B The diagram illustrates θ (mid-frequency detail latent variable), which is used to decode and generate vertex offset map samples of the face. On one hand, this vertex offset map sample is input into the discriminator, where generative adversarial training is performed based on the aforementioned vertex offset map ground truth and generative adversarial loss function. Simultaneously, to ensure the smoothness of the face shape features, a smoothing loss function is also set for this branch. Through the generative adversarial loss function and the smoothing loss function, the detail decoder and mid-frequency decoder can better learn and analyze the second-frequency face shape features of the 2D face image, generating a more accurate vertex offset map. On the other hand, this vertex offset map sample, along with the normal displacement map sample, is applied to the base 3D face mesh sample. For example, ... Figure 3B As shown, this normal displacement map sample is the face shape feature obtained from the third frequency of the detail encoder output, i.e. Figure 3B The diagram in the middle After (high-frequency detail latent variables), they are decoded and generated by a high-frequency decoder (i.e., the third decoder). For this branch, a regularized loss function is also set to limit the detail encoder and high-frequency decoder of this branch to focus more on the subtle shape features of the face.

[0086] Based on the obtained vertex offset map samples and normal displacement map samples, the features and information represented by the vertex offset map samples and normal displacement map samples can be fused with the basic 3D face mesh through UV mapping and normal mapping, respectively, to generate the target 3D face mesh sample, such as... Figure 3B The “fine grid” shown.

[0087] Step S308: Generate a 3D face model sample corresponding to the 2D face image sample based on the target 3D face mesh sample.

[0088] After obtaining a more refined 3D face mesh sample, rendering can be performed based on it to generate the final 3D face model sample. For example, a 3D face model sample corresponding to a 2D face image sample can be generated based on the target 3D face mesh sample and the face texture map sample.

[0089] Among them, the face texture map samples can be obtained based on the texture coefficients extracted by the first encoder, such as Figure 3B As shown, after obtaining the "coarse and fine mesh," it can be rendered using a face texture map. Furthermore, it can be combined with lighting coefficients. Figure 3B The diagram shows l), camera coefficient ( Figure 3B The diagram in the middle is shown as c), attitude coefficient ( Figure 3B The diagram shows that p) together renders the target 3D face mesh to obtain the final rendered 3D face model sample.

[0090] Step S310: Train the 3D reconstruction model based on the rendered image of the 3D face model sample and the preset loss function.

[0091] During the model training phase, the loss function can effectively measure the training effect of the model. In the scheme of this embodiment, after obtaining the 3D face model sample, the rendered image corresponding to the 3D face model sample, the 2D face image sample, and the preset loss function can be used to train the first encoder, the first decoder, the second encoder, the second decoder, and the third decoder. That is, the loss result of the loss function will affect all branches of the 3D reconstruction model, causing the model parameters of each branch to be adjusted.

[0092] However, since each branch has its own corresponding function, each branch will be adjusted by both the overall model and its own branch during a single training iteration (using one training sample). Based on this, the training of the 3D reconstruction model can be implemented as follows: the first encoder and the first decoder are trained based on the rendered image, 2D face image samples, and the corresponding reconstruction error loss function; the second encoder and the second decoder are trained based on the rendered image, 2D face image samples, the reconstruction error loss function, vertex offset map samples, vertex offset map ground truth, the corresponding generative adversarial loss function, and the preset smoothing loss function; the second encoder and the third decoder are trained based on the rendered image, 2D face image samples, the reconstruction error loss function, and the preset regularization loss function.

[0093] For example, such as Figure 3B As shown, the overall network structure of the 3D reconstruction model is trained end-to-end, requiring only a large number of 2D face images during training, without the need for paired 3D models. The overall loss of the 3D reconstruction model comprises three parts:

[0094] The first part is the reconstruction error loss, namely L_rec = ||Rendered image of 3D face model sample - Input 2D face image sample||, which is used to evaluate the fitting difference between the 3D face model and the original 2D face image.

[0095] The second part is the loss for vertex offset map samples, which consists of two components. One component is the generative adversarial loss L_adv. For this part, a suitable model, such as a 3DMM model, and vertex offset maps can be pre-fitted onto real 3D face data to obtain the true vertex offset maps, i.e., the ground truth vertex offset maps. A discriminator is then used to judge the predicted vertex offset map samples, calculating the generative adversarial loss L_adv. In this approach, by using real 3D face data, prior information on the mid-frequency (i.e., second frequency) details of the face can be obtained, ensuring the reasonableness and realism of the predicted vertex offset map samples, thus achieving more realistic and reliable facial detail reconstruction. Furthermore, to ensure the smoothness of changes in facial geometry, a smoothing loss L_smo is also calculated for the vertex offset map samples.

[0096] The third part is the regularization loss L_reg for the normal displacement map samples. This loss is used to limit its size and guide it to focus on subtle features of the face geometry, such as wrinkles.

[0097] Therefore, the overall loss function of the 3D reconstruction model can be expressed as:

[0098] L_total=λ_rec x L_rec+λ_adv x L_adv+λ_smo x L_smo+λ_reg x L_reg

[0099] Here, the λ parameter corresponding to each loss represents the learnable weight coefficient. For example, λ_rec represents the weight coefficient corresponding to L_rec, λ_adv represents the weight coefficient corresponding to L_adv, λ_smo represents the weight coefficient corresponding to L_smo, and λ_reg represents the weight coefficient corresponding to L_reg.

[0100] However, this is not the only limitation. The aforementioned reconstruction error loss function can also be implemented as a Mean Absolute Error (MAE) loss function or a Mean Square Error (MSE) loss function, etc. Furthermore, in practical applications, those skilled in the art can add additional loss function components to the overall loss function as needed, such as a Learned Perceptual Image Patch Similarity (LPIPS) loss function, a face recognition loss function, and so on.

[0101] As can be seen, in this embodiment, during the training of the 3D reconstruction model, features at multiple levels and granularities are extracted from 2D face image samples to obtain face shape features at the first, second, and third frequencies. Then, based on these face shape features, the basic 3D face mesh samples are detailed, enabling the trained 3D reconstruction model to generate a 3D face model that more closely resembles a real face. This 3D reconstruction model can be trained using a self-supervised approach. The input is a clear, unobstructed 2D face image. Through a series of encoders and decoders, a fine 3D face mesh is obtained. Differentiable rendering is used to obtain the rendering result for this 3D face mesh. By self-supervising the rendering result and the input 2D face image samples, the 3D reconstruction model can be optimized. In the subsequent inference stage, the final differentiable rendering process is unnecessary; the final 3D face model can be obtained based on the target 3D face mesh and texture mapping.

[0102] Example 3

[0103] Reference Figure 4 The diagram shows a structural schematic of an electronic device according to Embodiment 3 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0104] like Figure 4As shown, the electronic device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0105] in:

[0106] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0107] Communication interface 404 is used to communicate with other electronic devices or servers.

[0108] The processor 402 is used to execute program 410, specifically to perform the relevant steps in any of the above method embodiments.

[0109] Specifically, program 410 may include program code that includes computer operation instructions.

[0110] Processor 402 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0111] The memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0112] The memory 406 is used to store the program 410. The program 410 has at least one executable instruction, which can be used to cause the processor 402 to perform the operation corresponding to the method described in any of the foregoing method embodiments.

[0113] The specific implementation of each step in program 410 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0114] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0115] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0116] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0117] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0118] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0119] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. An image-based 3D reconstruction method, characterized in that, include: The face shape features of a two-dimensional face image are obtained at a first frequency, a second frequency, and a third frequency, respectively. The frequency is used to indicate the rate of change of the face shape features. The first frequency is lower than the second frequency, and the second frequency is lower than the third frequency. The face shape features of the first frequency include one or more of face shape coefficient features and face expression coefficient features. The face shape features of the second frequency and the third frequency are obtained by feature extraction of the two-dimensional face image using a second encoder. Based on the face shape features of the first frequency, a corresponding basic 3D face mesh and face texture map are generated; the face shape features of the second frequency and the third frequency are decoded respectively to obtain the corresponding face vertex offset map and the corresponding face normal displacement map, and the basic 3D face mesh is refined based on the vertex offset map and the normal displacement map to generate the target 3D face mesh. The vertex offset map is obtained by decoding the face shape features of the second frequency through the second decoder. The vertex offset map is used to determine the offset information between different vertices of the face. The normal displacement map is used to mark the direction of the normals drawn by each point of the basic 3D face mesh based on the coordinates of the horizontal, vertical and vertical axes. The second encoder and the second decoder are obtained by training with face vertex offset map samples, face vertex offset map ground truth, corresponding generative adversarial loss function and preset smoothing loss function. The vertex offset map samples are generated based on the face shape features of the second frequency, and the vertex offset map ground truth is obtained based on the 3D face model corresponding to the 2D face image samples in the training samples. Based on the target 3D face mesh and the face texture map, a 3D face model corresponding to the 2D face image is generated.

2. The method according to claim 1, characterized in that, Based on the vertex offset map and the normal displacement map, the basic 3D face mesh is refined to generate the target 3D face mesh, including: The basic 3D face mesh is deformed according to the vertex offset map to obtain the deformed 3D face mesh; as well as, The normal mapping is performed on the deformed 3D face mesh according to the normal displacement map to obtain the target 3D face mesh.

3. The method according to claim 2, characterized in that, The step of deforming the basic 3D face mesh according to the vertex offset map to obtain the deformed 3D face mesh includes: Based on the vertex offset map, the basic 3D face mesh is deformed by UV mapping to obtain the deformed 3D face mesh.

4. The method according to any one of claims 1-3, characterized in that, The acquisition of facial shape features at first, second, and third frequencies corresponding to the two-dimensional facial image includes: The first encoder in the 3D reconstruction model extracts the shape features of the 2D face image to obtain the face shape features of the first frequency. The second encoder extracts shape features from the two-dimensional face image to obtain the face shape features at the second frequency and the face shape features at the third frequency.

5. The method according to claim 4, characterized in that, The step of generating a corresponding basic 3D face mesh and face texture map based on the face shape features of the first frequency includes: decoding the face shape features of the first frequency through the first decoder in the 3D reconstruction model to generate a corresponding basic 3D face mesh; and generating a corresponding face texture map based on the face shape features of the first frequency. The step of generating a vertex offset map of the corresponding face based on the face shape features of the second frequency includes: decoding the face shape features of the second frequency using a second decoder for the face shape features of the second frequency to generate a vertex offset map of the corresponding face. The step of generating a corresponding normal displacement map of the face based on the face shape features of the third frequency includes: decoding the face shape features of the third frequency using a third decoder to generate a corresponding normal displacement map of the face.

6. A method for training an image-based 3D reconstruction model, characterized in that, include: Obtain training samples for training the 3D reconstruction model, wherein the training samples include at least: 2D face image samples; The two-dimensional face image sample is input into the three-dimensional reconstruction model to obtain the face shape features corresponding to the two-dimensional face image sample at the first frequency, the second frequency and the third frequency, respectively. The frequency is used to indicate the rate of change of the face shape features. The first frequency is lower than the second frequency and the second frequency is lower than the third frequency. Based on the facial shape features of the first frequency, a corresponding basic three-dimensional facial mesh sample is generated; and based on the facial shape features of the second and third frequencies, the basic three-dimensional facial mesh sample is refined to generate a target three-dimensional facial mesh sample. Based on the target 3D face mesh sample, generate a 3D face model sample corresponding to the 2D face image sample; The 3D reconstruction model is trained based on the rendered image of the 3D face model sample and a preset loss function, wherein the trained 3D reconstruction model is used to execute the image-based 3D reconstruction method according to any one of claims 1 to 5.

7. The method according to claim 6, characterized in that, The process of generating a basic 3D face mesh sample based on the face shape features of the first frequency, and refining the basic 3D face mesh sample based on the face shape features of the second and third frequencies to generate a target 3D face mesh sample, includes: Based on the face shape features of the first frequency, generate corresponding basic 3D face mesh samples and face texture map samples; based on the face shape features of the second frequency, generate corresponding face vertex offset map samples; based on the face shape features of the third frequency, generate corresponding face normal displacement map samples. Based on the vertex offset map sample and the normal displacement map sample, the basic 3D face mesh sample is refined to generate the target 3D face mesh sample; The step of generating a 3D face model sample corresponding to the 2D face image sample based on the target 3D face mesh sample includes: generating a 3D face model sample corresponding to the 2D face image sample based on the target 3D face mesh sample and the face texture map sample.

8. The method according to claim 7, characterized in that, The three-dimensional reconstruction model includes: a first encoder, a first decoder corresponding to the first encoder, a second encoder, a second decoder corresponding to the second encoder, and a third decoder; in: The first encoder is used to extract face shape features at a first frequency from the two-dimensional face image sample to obtain face shape features at a first frequency. The first decoder is used to generate a corresponding basic three-dimensional face mesh sample based on the face shape features of the first frequency; The second encoder is used to extract face shape features at second and third frequencies from the two-dimensional face image sample to obtain the corresponding face shape features at second and third frequencies. The second decoder is used to generate a corresponding vertex offset map sample of the face based on the face shape features of the second frequency; The third decoder is used to generate a corresponding normal displacement map sample of the face based on the face shape features of the third frequency.

9. The method according to claim 8, characterized in that, The step of training the 3D reconstruction model based on the rendered image of the 3D face model sample and a preset loss function includes: Obtain the rendered image of the three-dimensional face model sample; Based on the rendered image, the two-dimensional face image samples, and the preset loss function, the first encoder, the first decoder, the second encoder, the second decoder, and the third decoder are all trained.

10. The method according to claim 9, characterized in that, The training samples also include: ground truth values ​​of vertex offset maps of the face obtained based on the three-dimensional face model corresponding to the two-dimensional face image samples; The step of training the first encoder, the first decoder, the second encoder, the second decoder, and the third decoder based on the rendered image, the two-dimensional face image samples, and a preset loss function includes: The first encoder and the first decoder are trained based on the rendered image, the two-dimensional face image sample, and the corresponding reconstruction error loss function; The second encoder and the second decoder are trained based on the rendered image, the two-dimensional face image samples, the reconstruction error loss function, the vertex offset map samples, the vertex offset map ground truth and the corresponding generative adversarial loss function, and the preset smoothing loss function. The second encoder and the third decoder are trained based on the rendered image, the two-dimensional face image sample, the reconstruction error loss function, and the preset regularization loss function.

11. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the method as described in any one of claims 1-10.

12. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images