A method and system for generating a three-dimensional asset of a character portrait

Abstract

The application discloses a kind of portrait three-dimensional asset generation method and system, belong to three-dimensional reconstruction technical field.It includes constructing dataset and portrait three-dimensional asset generation network;According to dataset, portrait three-dimensional asset generation network is trained and tested;Target two-dimensional figure image is obtained, and target three-dimensional portrait asset is obtained according to portrait three-dimensional asset generation network after processing target two-dimensional figure image after testing;The application combines diffusion generation model and neural radiation field architecture by portrait three-dimensional asset generation network, and only needs the portrait two-dimensional picture of identity feature, can generate realistic portrait three-dimensional asset.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional reconstruction, and particularly relates to a method and system for generating a three-dimensional asset of a portrait of a person. BACKGROUND

[0002] As a core element of digital content creation, a three-dimensional asset of a portrait of a person has wide and important application value in the fields of film and television production, game development, virtual reality, and metaverse. Traditionally, high-quality digital portrait assets are mainly obtained by using professional three-dimensional scanning equipment (such as structured light, laser radar, or a multi-camera array) and a time-consuming and costly manual post-editing process. Although such methods can obtain high-precision geometric and texture information, they have very high requirements for equipment, environment, and operating personnel, and are difficult to popularize to ordinary users or small and medium-sized enterprises.

[0003] At present, existing three-dimensional assets of a portrait of a person are mostly obtained by using a neural radiance field (NeRF) technology, but a single diffusion model is generally used for optimization, and the generated results often have problems such as excessive smoothing, missing details, and blurred skin texture, which further leads to low generation quality and low realism. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a method and system for generating a three-dimensional asset of a portrait of a person, which can solve the technical problems of low generation quality and low realism of a three-dimensional asset of a portrait of a person in the prior art.

[0005] To solve the above technical problems, the present application is implemented as follows:

[0006] In a first aspect, the embodiments of the present application provide a method for generating a three-dimensional asset of a portrait of a person, which comprises:

[0007] obtaining a plurality of two-dimensional portrait images of a person, and preprocessing the plurality of two-dimensional portrait images of a person to obtain a plurality of preprocessed two-dimensional portrait images of a person, and constructing a data set according to the plurality of preprocessed two-dimensional portrait images of a person;

[0008] constructing a three-dimensional asset generation network of a portrait of a person, wherein the three-dimensional asset generation network of a portrait of a person comprises a head feature extraction module, a feature mapping module, a mask generation module, a sampling optimization module, and a joint optimization module;

[0009] dividing the data set into a training set and a test set according to a preset ratio, and training and testing the three-dimensional asset generation network of a portrait of a person according to the training set and the test set, respectively;

[0010] Obtaining a target two-dimensional portrait image, processing the target two-dimensional portrait image according to the tested portrait three-dimensional asset generation network to obtain a target three-dimensional portrait asset.

[0011] As an optional implementation of the first aspect of the application, the portrait three-dimensional asset generation network processes the target two-dimensional portrait image to obtain a target three-dimensional portrait asset, specifically:

[0012] According to the head feature extraction module, the parameterized head feature of the target two-dimensional portrait image is extracted to obtain a parameterized head feature;

[0013] According to the feature mapping module, the parameterized head feature is mapped to a three-dimensional space to obtain a spatial head model;

[0014] According to the mask generation module, the spatial head model is processed by effectiveness sampling to obtain an effective region mask feature;

[0015] According to the sampling optimization module, the effective region mask feature is sampled to obtain an initial three-dimensional portrait body;

[0016] According to the joint optimization module, the initial three-dimensional portrait body is optimized in two stages to obtain the target three-dimensional portrait asset.

[0017] As an optional implementation of the first aspect of the application, the mask generation module processes the spatial head model by effectiveness sampling to obtain an effective region mask feature, specifically:

[0018] Calculate the distance from each spatial point in the three-dimensional space to the spatial head model to obtain a spatial feature mask corresponding to each spatial point;

[0019] According to each spatial feature mask, the exponential decay weight of the spatial point corresponding to each spatial feature mask is constructed;

[0020] Mark the spatial point with an exponential decay weight higher than a first preset threshold as a spatial effective point, and construct the effective region mask feature according to the spatial feature mask corresponding to all the spatial effective points.

[0021] As an optional implementation of the first aspect of the application, the sampling optimization module samples the effective region mask feature to obtain an initial three-dimensional portrait body, specifically:

[0022] Ray sampling processing is performed on each spatial pixel in the effective region mask feature to obtain a sampling ray corresponding to each spatial pixel;

[0023] comparing each of the sampling rays with a second preset threshold, marking the spatial pixels corresponding to the sampling rays higher than the second preset threshold as spatial key pixels;

[0024] retaining all the spatial key pixels and generating a two-dimensional mask according to all the spatial key pixels to obtain the initial three-dimensional portrait volume.

[0025] As an optional implementation of the first aspect of the application, the two-stage joint optimization includes a structure generation optimization stage and a texture enhancement optimization stage, and a dynamic weight mechanism is adopted to optimize the initial three-dimensional portrait volume, specifically:

[0026] In the high noise scale stage, the optimization weight setting of the structure generation optimization stage is greater than that of the texture enhancement optimization stage;

[0027] In the low noise scale stage, the optimization weight setting of the structure generation optimization stage is less than that of the texture enhancement optimization stage;

[0028] The structure generation optimization stage adopts a diffusion model with three-dimensional perception ability to process the initial three-dimensional portrait volume to generate a multi-view consistent geometric structure, and the texture enhancement optimization stage adopts a Stable Diffusion model to enhance the texture details and facial realism of the initial three-dimensional portrait volume;

[0029] Wherein, the optimization weight is obtained by a gating network, and the gating network adaptively adjusts the optimization weight according to the noise scale.

[0030] As an optional implementation of the first aspect of the application, the training and testing of the portrait three-dimensional asset generation network according to the training set and the test set are specifically:

[0031] A composite loss function is constructed, and the portrait three-dimensional asset generation network is iteratively trained according to the training set and the composite function;

[0032] The loss value in the training process of the portrait three-dimensional asset generation network is calculated according to the composite loss function;

[0033] The portrait three-dimensional asset generation network is back propagated according to the loss value to adjust the parameters of the portrait three-dimensional asset generation network until the parameters of the portrait three-dimensional asset generation network converge;

[0034] The portrait three-dimensional asset generation network with converged parameters is tested according to the test set, and the test result is evaluated;

[0035] If the evaluation result is qualified, the training and testing of the portrait three-dimensional asset generation network are completed, and if the evaluation result is not qualified, the parameters of the portrait three-dimensional asset generation network are adjusted for retraining until the evaluation result is qualified.

[0036] As an optional implementation form of the first aspect of the application, the composite loss function is represented by the following formula:

[0037]

[0038] ,

[0039] wherein, represents a composite loss function, represents an optimization strategy under different noise scales, represents an SDS loss function filtered by a spatial feature mask, represents a regression loss function, represents a weight of a regularization loss function, represents a regularization loss function, represents an SDS loss function, represents an element-wise multiplication, represents all spatial feature masks obtained by the mask generation module in the training process.

[0040] In a second aspect, the embodiments of the application provide a portrait three-dimensional asset generation system, which comprises:

[0041] a data set construction module configured to obtain a plurality of two-dimensional portrait images, pre-process the plurality of two-dimensional portrait images to obtain a plurality of pre-processed two-dimensional portrait images, and construct a data set according to the plurality of pre-processed two-dimensional portrait images;

[0042] a network construction module configured to construct a portrait three-dimensional asset generation network, wherein the portrait three-dimensional asset generation network comprises a head feature extraction module, a feature mapping module, a mask generation module, a sampling optimization module, and a joint optimization module;

[0043] a network training module configured to divide the data set into a training set and a test set according to a preset proportion, and train and test the portrait three-dimensional asset generation network according to the training set and the test set, respectively;

[0044] an execution processing module configured to obtain a target two-dimensional portrait image, process the target two-dimensional portrait image according to the tested portrait three-dimensional asset generation network, and obtain a target three-dimensional portrait asset.

[0045] In a third aspect, the embodiments of the present application provide an electronic device, comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0046] In a fourth aspect, the embodiments of the present application provide a readable storage medium, and the readable storage medium stores a program or instructions, and the program or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0047] In the embodiments of the present application, compared with the prior art, the following technical effects are achieved:

[0048] The present application combines the diffusion generative model with the neural radiance field architecture through the portrait three-dimensional asset generation network, and only needs a portrait two-dimensional picture with an identity feature to generate a realistic portrait three-dimensional asset. The portrait three-dimensional asset generation network deeply integrates the spatial feature mask, key sampling, and neural radiance field, so that the generation process has a three-dimensional space priori. The mask decays exponentially according to the distance of the human head surface, and determines the effective sampling area by ray. The redundant calculation of the background and the hollow area is fundamentally eliminated. Whether it is a front view or a large-angle side view, the training and rendering are focused on the effective voxels of the face. The generated asset maintains the true proportion in terms of geometric integrity and surface details, and completely alleviates the slow convergence, large noise, and side collapse phenomenon caused by uniform sampling of the traditional NeRF architecture. Secondly, the present application introduces a multi-stage diffusion model for joint optimization. The diffusion model with three-dimensional perception dominates the high-noise stage with a learnable gating weight, ensuring consistent geometry in multiple views. The Stable Diffusion model dominates the low-noise stage, focusing on skin texture and hair details. The two stages share the image embedding independent of the view as an identity signature, so that the core features are still locked when the continuous view, expression, or illumination changes, avoiding identity drift or excessive smoothing, and supporting real-time extraction of high-fidelity images at any angle after one-time optimization. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 FIG. 1 is a flowchart of a portrait three-dimensional asset generation method according to some embodiments of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0051] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in a "or" relationship.

[0052] The person portrait three-dimensional asset generation method and system provided by the embodiments of the present application will be described in detail below in conjunction with the drawings, specific embodiments and application scenarios.

[0053] Embodiments

[0054] A person portrait three-dimensional asset generation method includes the following steps:

[0055] S100: Obtain a plurality of two-dimensional person portrait images, and pre-process the plurality of two-dimensional person portrait images to obtain a plurality of pre-processed two-dimensional person images, and construct a data set according to the plurality of pre-processed two-dimensional person images;

[0056] It should be noted that a plurality of two-dimensional person portrait images in the public data set are collected, and the plurality of two-dimensional person portrait images are adjusted to have a resolution of 256x256 to obtain a plurality of pre-processed two-dimensional person images, and a data set is constructed according to the plurality of pre-processed two-dimensional person images.

[0057] S200: Construct a person portrait three-dimensional asset generation network, which includes a head feature extraction module, a feature mapping module, a mask generation module, a sampling optimization module, and a joint optimization module;

[0058] S300: Divide the data set into a training set and a test set according to a preset ratio, and train and test the person portrait three-dimensional asset generation network according to the training set and the test set, respectively;

[0059] As an optional implementation manner of the present embodiment, the person portrait three-dimensional asset generation network is trained and tested according to the training set and the test set in S300, specifically:

[0060] S310: Construct a composite loss function, and iteratively train the person portrait three-dimensional asset generation network according to the training set and the composite function;

[0061] S320: Calculate the loss value in the training process of the person portrait three-dimensional asset generation network according to the composite loss function;

[0062] S330: Perform back propagation on the character portrait three-dimensional asset generation network according to the loss value to adjust the parameters of the character portrait three-dimensional asset generation network until the parameters of the character portrait three-dimensional asset generation network converge;

[0063] S340: Test the character portrait three-dimensional asset generation network after the parameters converge according to the test set, and evaluate the test results;

[0064] S350: If the evaluation result is qualified, the training and testing of the character portrait three-dimensional asset generation network are completed, and if the evaluation result is not qualified, the parameters of the character portrait three-dimensional asset generation network are adjusted for retraining until the evaluation result is qualified.

[0065] As a specific embodiment of the present embodiment, the composite loss function is represented by the following formula:

[0066]

[0067]

[0068] wherein, represents the composite loss function, represents the optimization strategy under different noise scales, represents the SDS loss function filtered by the spatial feature mask, represents the regression loss function, represents the weight of the regularization loss function, represents the regularization loss function, represents the SDS loss function, represents element-wise multiplication, represents all spatial feature masks obtained by the mask generation module during the training process.

[0069] ​It should be noted that the data set is divided into a training set and a test set in a ratio of 4:1, and then the character portrait three-dimensional asset generation network is iteratively trained by the training set; in the training process, the loss value of the character portrait three-dimensional asset generation network during training is calculated by a composite loss function; the composite loss function includes three components, which are SDS (Score Distillation Sampling) loss function, regression loss function and regularization loss function; the SDS loss function is the core optimization target, and the mathematical essence of this loss function is to perform differential operation on the generation process of the pre-trained diffusion model; specifically, it is to calculate the transformation result of the Jacobian matrix under the guidance of the chain rule. This unique mathematical conversion mechanism realizes the effective migration of the strong generation ability and rich prior knowledge contained in the large-scale diffusion model into the parameter optimization process of the network through gradient backpropagation; in addition, the Score Distillation Sampling (SDS) loss function is optimized by a spatial feature mask, which is based on the good prior assumption of the character portrait in three-dimensional space. In order to improve the sampling efficiency, the SDS loss function is filtered by the spatial feature mask; the composite loss function sets two key hyperparameters and , 0.02, while is dynamically adjusted by a Sigmoid function based on noise scale, which can adaptively adjust the regularization strength during training, ensuring stable convergence during training and avoiding overfitting risk; then, the character portrait three-dimensional asset generation network is backpropagated by the loss value to adjust the parameters of the character portrait three-dimensional asset generation network until the parameters of the character portrait three-dimensional asset generation network converge; the Adam optimizer is used to optimize the network parameters, and the initial learning rate is set to 3e-4, and the cosine annealing strategy is used for dynamic adjustment; then, the character portrait three-dimensional asset generation network after parameter convergence is tested according to the test set, and the test results are evaluated; if the evaluation result is qualified, the training and testing of the character portrait three-dimensional asset generation network are completed, and if the evaluation result is not qualified, the parameters of the character portrait three-dimensional asset generation network are adjusted for retraining until the evaluation result is qualified.

[0070] S400: Obtain a target two-dimensional character image, process the target two-dimensional character image according to the character portrait three-dimensional asset generation network after testing, and obtain a target three-dimensional portrait asset.

[0071] As an optional implementation of the embodiment, the character portrait three-dimensional asset generation network in S400 processes the target two-dimensional character image to obtain a target three-dimensional portrait asset, specifically:

[0072] S410: parameterized feature extraction is performed on the target two-dimensional character image according to a head feature extraction module to obtain a parameterized head feature;

[0073] S420: The parameterized head feature is mapped to a three-dimensional space according to a feature mapping module to obtain a spatial head model;

[0074] S430: The spatial head model is processed by the mask generation module to obtain an effective region mask feature;

[0075] S440: The effective region mask feature is sampled by the sampling optimization module to obtain an initial three-dimensional portrait body;

[0076] S450: The initial three-dimensional portrait body is optimized in two stages by the joint optimization module to obtain a target three-dimensional portrait asset.

[0077] It should be noted that the character portrait three-dimensional asset generation network is a neural radiation field (NeRF) technology architecture, and the neural radiation field is used as the representation form of the finally generated character portrait three-dimensional asset, and the generation process is similar to the corresponding three-dimensional reconstruction process, which needs to sample the light corresponding to different angles and calculate the cumulative color. When the three-dimensional asset generation of the character portrait is specifically implemented in the embodiment, due to the unique prior distribution of the character portrait, all spatial points in the space are not equally important, and part of the light and part of the spatial position may not have the value of sampling, which prompts the embodiment to explore a sampling scheme more suitable for this specific scene; the mask generation module roughly judges the effective space related to the target to reduce the negative impact of useless spatial positions as much as possible, thereby enhancing the accuracy and effectiveness of the generation process, and further improving the generation quality; the specific processing process of the network is: first, the parameterized head feature is obtained by the head feature extraction module for parameterized feature extraction on the target two-dimensional character image; then, the spatial head model is obtained by the feature mapping module for mapping the parameterized head feature to the three-dimensional space; then, the effective region mask feature is obtained by the mask generation module for effective sampling processing on the spatial head model; subsequently, the sampling optimization module performs key sampling on the effective region mask feature to obtain an initial three-dimensional portrait body; finally, the target three-dimensional portrait asset is obtained by the joint optimization module for two-stage joint optimization of the initial three-dimensional portrait body.

[0078] According to an optional implementation of the embodiment, the mask generation module in S430 performs effective sampling processing on the spatial head model to obtain an effective region mask feature, specifically:

[0079] S431: calculate the distance from each spatial point in the three-dimensional space to the spatial human head model to obtain a spatial feature mask corresponding to each spatial point;

[0080] S432: construct an exponential decay weight of the spatial point corresponding to each spatial feature mask according to each spatial feature mask;

[0081] S433: mark the spatial point with an exponential decay weight higher than a first preset threshold as a spatial effective point, and construct an effective region mask feature according to the spatial feature mask corresponding to all spatial effective points.

[0082] It should be noted that first, the distance from each spatial point in the three-dimensional space to the spatial human head model is calculated to obtain a spatial feature mask corresponding to each spatial point; then an exponential decay weight of the spatial point corresponding to each spatial feature mask is constructed according to each spatial feature mask; finally, the spatial point with an exponential decay weight higher than a first preset threshold is marked as a spatial effective point, and an effective region mask feature is constructed according to the spatial feature mask corresponding to all spatial effective points; for the generation of a neural radiance field-based portrait three-dimensional asset, since it cannot be directly optimized like three-dimensional reconstruction, it needs to participate in the diffusion model, so the generation efficiency is not high, and in the stage with high noise, the entire space is often covered, which further leads to invalid sampling and generation. The designed spatial feature mask can effectively estimate the possible effectiveness of a spatial position in the rendering process considering the unique rendering mechanism of the neural radiance field. In the mask generation module, an effective threshold, i.e., a first preset threshold, is set, only the spatial position higher than the threshold can participate in the optimization of the subsequent sampling optimization module and the joint optimization module. In order to further effectively alleviate this problem and improve the generation efficiency.

[0083] As an optional implementation of the embodiment, the sampling optimization module in S440 performs key sampling on the effective region mask feature to obtain an initial three-dimensional portrait body, specifically:

[0084] S441: perform ray sampling processing on each spatial pixel in the effective region mask feature to obtain a sampling ray corresponding to each spatial pixel;

[0085] S442: compare each sampling ray with a second preset threshold, and mark the spatial pixel corresponding to the sampling ray higher than the second preset threshold as a spatial key pixel;

[0086] S443: retain all spatial key pixels, and perform two-dimensional mask generation according to all spatial key pixels to obtain an initial three-dimensional portrait body.

[0087] It is important to understand that, firstly, ray sampling is performed on each spatial pixel in the effective region mask features to obtain the sampling ray corresponding to each spatial pixel; then, each sampling ray is compared with a second preset threshold, and the spatial pixels corresponding to sampling rays that are higher than the second preset threshold are marked as spatial key pixels; finally, all spatial key pixels are retained, and a two-dimensional mask is generated based on all spatial key pixels to obtain the initial three-dimensional portrait. The second preset threshold can be set to the same as the first preset threshold in the mask generation module. By using the second preset threshold to perform focused sampling on the rendered sampling rays, the optimization efficiency is improved. At the same time, the corresponding two-dimensional mask is extracted from the rendering viewpoint. This two-dimensional mask will count the spatial validity under the corresponding viewpoint and set pixels below a certain threshold to zero. Only rays passing through the two-dimensional mask will be calculated, thereby further improving the generation efficiency of three-dimensional assets.

[0088] As an optional implementation of this embodiment, the two-stage joint optimization in S450 includes a structure generation optimization stage and a texture enhancement optimization stage, and a dynamic weighting mechanism is used to optimize the initial 3D portrait volume, specifically as follows:

[0089] In the high-noise-scale stage, the optimization weights of the structure generation optimization stage are set higher than those of the texture enhancement optimization stage; in the low-noise-scale stage, the optimization weights of the structure generation optimization stage are set lower than those of the texture enhancement optimization stage. The structure generation optimization stage uses a diffusion model with 3D perception capabilities to process the initial 3D portrait body and generate a consistent geometric structure from multiple perspectives. The texture enhancement optimization stage uses a Stable Diffusion model to enhance the texture details and facial realism of the initial 3D portrait body. The optimization weights are obtained by a gating network, which adaptively adjusts the optimization weights according to the noise scale.

[0090] It should be noted that using the initial diffusion model for generation will cause over-smoothing and cartoonization problems, which are caused by the generation diversity of the diffusion model itself. For the same object under the same perspective, the diffusion model will still generate images with different poses and features, which makes it difficult to fully utilize the high-quality characteristics of the diffusion model generation results, and instead causes great obstacles to training. If the diffusion model is made to have certain three-dimensional perception ability and can generate images under a specified perspective, it can greatly improve the generation quality, but this scheme usually causes the diffusion model to have a phenomenon of generation quality decline. The joint optimization module in the embodiment adopts a framework that combines the generation capabilities of the two to generate a three-dimensional asset of a portrait of a person. Based on these in-depth analyses, a technical route of multi-diffusion model joint optimization is designed. Specifically, a collaborative optimization system containing two specialized diffusion models is designed: the first model focuses on global structure generation and geometric detail optimization, i.e., a diffusion model with three-dimensional perception ability; the second model specializes in texture enhancement, i.e., a Stable Diffusion model. This division of labor and cooperation architecture not only significantly improves the generation quality, but also optimizes the generation efficiency through parallel computing. An optimization strategy for three-dimensional portrait generation based on multi-stage feature fusion is designed, and the core idea is to fully utilize the advantages of diffusion models in different denoising stages. Specifically, the dynamics characteristics of the diffusion model progressive denoising are analyzed in depth: in the high noise stage (T>0.7), the model mainly learns the generation of macro-geometric structure; in the medium noise stage (0.3<T<0.7), the model focuses on pose adjustment and structure refinement; and in the low noise stage (T<0.3), it focuses on optimizing texture details and micro features. Based on this finding, a dynamic weight adjustment mechanism is designed to adaptively control the participation ratio of the three-dimensional perception diffusion model and the Stable Diffusion model through a learnable gating network. The diffusion model with three-dimensional perception is mainly responsible for accurate modeling of geometric structure, while the Stable Diffusion model enhances the generation quality of texture details.

[0091] According to the portrait three-dimensional asset generation method, the diffusion generation model is combined with the neural radiation field architecture through the portrait three-dimensional asset generation network, and a realistic portrait three-dimensional asset can be generated only by using a two-dimensional portrait picture of a person with an identity feature; the portrait three-dimensional asset generation network deeply fuses the spatial feature mask, the key sampling, and the neural radiation field, so that the generation process has a three-dimensional space priori, the mask is exponentially attenuated according to the head surface distance, and the effective sampling area is determined by ray by ray, the redundant calculation of the background and the hollow area is fundamentally eliminated, whether it is a front view or a large-angle side view, the training and the rendering are focused on the effective voxels of the face, and the generated asset maintains a true proportion in terms of geometric integrity and surface details, thereby completely relieving the slow convergence, the large noise, and the side collapse phenomenon caused by the uniform sampling of the traditional NeRF architecture; secondly, the multi-stage diffusion model is introduced for joint optimization, the diffusion model with three-dimensional perception dominates the high-noise stage with a learnable gating weight, and the multi-view geometric consistency is ensured; the Stable Diffusion model dominates the low-noise stage, focuses on the skin texture and the hair details, and the image embedding unrelated to the view is shared as an identity signature in the two stages, so that the core appearance is still locked when the continuous view, the expression, or the illumination changes, identity drift or excessive smoothing is avoided, and real-time extraction of high-fidelity images at any angle is supported after one-time optimization.

[0092] It should be noted that the portrait three-dimensional asset generation method provided in the embodiments of the present application can be executed by a portrait three-dimensional asset generation system or a control module in the portrait three-dimensional asset generation system for executing the portrait three-dimensional asset generation method. In the embodiments of the present application, the portrait three-dimensional asset generation system is taken as an example to execute the portrait three-dimensional asset generation method, and the portrait three-dimensional asset generation method provided in the embodiments of the present application is described.

[0093] A portrait three-dimensional asset generation system comprises:

[0094] A data set construction module acquires a plurality of two-dimensional portrait images, pre-processes the plurality of two-dimensional portrait images to obtain a plurality of pre-processed two-dimensional portrait images, and constructs a data set according to the plurality of pre-processed two-dimensional portrait images;

[0095] A network construction module constructs a portrait three-dimensional asset generation network, wherein the portrait three-dimensional asset generation network comprises a head feature extraction module, a feature mapping module, a mask generation module, a sampling optimization module, and a joint optimization module;

[0096] A network training module divides the data set into a training set and a test set according to a preset proportion, and trains and tests the portrait three-dimensional asset generation network according to the training set and the test set, respectively;

[0097] The execution processing module: obtains a target two-dimensional figure image, processes the target two-dimensional figure image according to the figure portrait three-dimensional asset generation network after testing, and obtains a target three-dimensional portrait asset.

[0098] The figure portrait three-dimensional asset generation system in the embodiment of the present application can be a device, or a component, integrated circuit or chip in a terminal. The device can be a mobile electronic device, or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, tablet computer, notebook computer, palm computer, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), etc., which are not limited in the embodiment of the present application.

[0099] The virtual device in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not limited in the embodiment of the present application.

[0100] The figure portrait three-dimensional asset generation system provided in the embodiment of the present application can realize the method Figure 1 The processes and effects realized by the figure portrait three-dimensional asset generation system in the method embodiment are not repeated here to avoid repetition.

[0101] Optionally, the embodiment of the present application further provides an electronic device, which includes a processor, a memory, a program or instructions stored on the memory and executable on the processor. When the program or instructions are executed by the processor, the processes of the above-mentioned figure portrait three-dimensional asset generation method embodiment are realized, and the same technical effects are achieved. To avoid repetition, the processes are not repeated here.

[0102] The embodiment of the present application further provides a readable storage medium, which stores a program or instructions. When the program or instructions are executed by a processor, the processes of the above-mentioned figure portrait three-dimensional asset generation method embodiment are realized, and the same technical effects are achieved. To avoid repetition, the processes are not repeated here.

[0103] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0104] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the element. In addition, it should be noted that the scope of the methods and apparatus of the present embodiments are not limited by the order of the steps or the order of the functions performed in the steps, and can include performing the functions in different orders, or substantially simultaneously, or in reverse order, such as described, for example, the described methods can be performed in an order different from that described, and various steps can be added, omitted, or combined, in addition, features described with reference to certain examples can be combined in other examples.

[0105] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for making a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.

[0106] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative and not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims.

Claims

1. A method for generating three-dimensional portrait assets, characterized in that, The method includes: Multiple two-dimensional portrait images of people are acquired, and the multiple two-dimensional portrait images of people are preprocessed to obtain multiple preprocessed two-dimensional portrait images of people. A dataset is constructed based on the multiple preprocessed two-dimensional portrait images of people. A 3D portrait asset generation network is constructed, which includes a head feature extraction module, a feature mapping module, a mask generation module, a sampling optimization module, and a joint optimization module. The dataset is divided into a training set and a test set according to a preset ratio, and the portrait 3D asset generation network is trained and tested using the training set and the test set respectively, specifically as follows: Construct a composite loss function, and iteratively train the 3D asset generation network for human portraits based on the training set and the composite loss function; The composite loss function is expressed as follows: ， in, Represents the composite loss function. These represent the optimization strategy parameters under different noise scales. This represents the SDS loss function after spatial feature masking. Represents the regression loss function. The weights represent the weights of the regularization loss function. This represents the regularization loss function. Represents the SDS loss function. This indicates element-wise multiplication. This represents all spatial feature masks obtained by the mask generation module during training; A target 2D human image is acquired, and then processed using the tested 3D portrait asset generation network to obtain the target 3D portrait asset. Specifically: The parameterized features of the target two-dimensional human image are extracted by the head feature extraction module to obtain parameterized head features; The parameterized head features are mapped to a three-dimensional space using the feature mapping module to obtain a spatial head model. The effective region mask features are obtained by performing validity sampling processing on the spatial head model according to the mask generation module. The sampling optimization module performs focused sampling on the mask features of the effective region to obtain an initial three-dimensional portrait. The initial 3D portrait body is subjected to two-stage joint optimization by the joint optimization module to obtain the target 3D portrait asset; the two-stage joint optimization includes a structure generation optimization stage and a texture enhancement optimization stage.

2. The method for generating three-dimensional portrait assets according to claim 1, characterized in that, The step of performing validity sampling processing on the spatial head model based on the mask generation module to obtain valid region mask features is as follows: Calculate the distance from each spatial point in the three-dimensional space to the spatial human head model, and obtain the spatial feature mask corresponding to each spatial point; Based on each spatial feature mask, construct an exponential decay weight for the spatial point corresponding to each spatial feature mask; Spatial points whose exponential decay weight is higher than a first preset threshold are marked as valid spatial points, and the valid region mask features are constructed based on the spatial feature masks corresponding to all valid spatial points.

3. The method for generating three-dimensional portrait assets according to claim 1, characterized in that, The initial 3D portrait is obtained by focusing on sampling the mask features of the effective region according to the sampling optimization module. Specifically: Perform ray sampling on each spatial pixel in the effective region mask feature to obtain the sampling ray corresponding to each spatial pixel; Each sampling ray is compared with a second preset threshold, and the spatial pixels corresponding to the sampling rays that are higher than the second preset threshold are marked as spatial key pixels; All the spatial key pixels are retained, and a two-dimensional mask is generated based on all the spatial key pixels to obtain the initial three-dimensional portrait.

4. The method for generating three-dimensional portrait assets according to claim 1, characterized in that, The two-stage joint optimization includes a structure generation optimization stage and a texture enhancement optimization stage, and a dynamic weighting mechanism is used to optimize the initial 3D portrait volume, specifically as follows: In the high-noise scale stage, the optimization weights of the structure generation optimization stage are set to be greater than those of the texture enhancement optimization stage; In the low-noise scale stage, the optimization weights of the structure generation optimization stage are set smaller than those of the texture enhancement optimization stage; The structure generation optimization stage uses a diffusion model with three-dimensional perception capabilities to process the initial three-dimensional portrait body and generate a geometric structure consistent from multiple perspectives. The texture enhancement optimization stage uses a stable diffusion model to enhance the texture details and facial realism of the initial three-dimensional portrait body. The optimization weights are obtained by a gating network, which adaptively adjusts the optimization weights according to the noise scale.

5. The method for generating three-dimensional portrait assets according to claim 1, characterized in that, The process of training and testing the 3D portrait asset generation network based on the training and test sets, respectively, specifically involves: The loss value during the training process of the portrait 3D asset generation network is calculated based on the composite loss function. The loss value is used to backpropagate the 3D portrait asset generation network to adjust the parameters of the 3D portrait asset generation network until the parameters of the 3D portrait asset generation network converge. The human portrait 3D asset generation network after parameter convergence is tested using the test set, and the test results are evaluated. If the evaluation result is satisfactory, the training and testing of the portrait 3D asset generation network are completed. If the evaluation result is unsatisfactory, the parameters of the portrait 3D asset generation network are adjusted and retrained until the evaluation result is satisfactory.

6. A system for generating three-dimensional portrait assets, used to implement the method for generating three-dimensional portrait assets according to any one of claims 1-5, characterized in that, The system includes: Dataset construction module: acquire multiple two-dimensional portrait images of people, preprocess the multiple two-dimensional portrait images of people to obtain multiple preprocessed two-dimensional portrait images of people, and construct a dataset based on the multiple preprocessed two-dimensional portrait images of people; Network construction module: Constructs a 3D asset generation network for human portraits, which includes a head feature extraction module, a feature mapping module, a mask generation module, a sampling optimization module, and a joint optimization module; Network training module: Divide the dataset into training set and test set according to a preset ratio, and train and test the human portrait 3D asset generation network according to the training set and test set respectively; The execution processing module acquires a target two-dimensional human image, processes the target two-dimensional human image according to the tested human portrait three-dimensional asset generation network, and obtains the target three-dimensional portrait asset.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of a method for generating a three-dimensional human portrait asset as described in any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of a method for generating a three-dimensional portrait asset as described in any one of claims 1-5.

Images