A single-image 3D portrait generation method and system based on mixed prior and noise resampling

Abstract

The application provides a single-image 3D portrait generation method and system based on a hybrid prior and noise resampling, and relates to the technical field of computer vision. The application generates multiple plane features according to a first portrait image obtained by using a pre-trained first model, combines the plane features with decoder parameters and super-resolution parameters corresponding to the first model to obtain initial parameters. A second portrait image is input into a pre-trained second model, and the parameters of the second model are fine-tuned so that the second model has the ability to generate a three-dimensional face structure corresponding to the second portrait image. The initial parameters and the fine-tuned second model are combined to determine target parameters. With the aid of the target parameters, a hybrid prior diffusion model and a multi-view noise resampling strategy are used to generate a three-dimensional face texture, which can provide a clear and executable technical path for single-image 3D portrait generation.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for generating 3D human portraits from a single image based on hybrid priors and noise resampling. Background Technology

[0002] In the fields of digital entertainment, virtual social networking, and film and television production, the demand for single-image 3D portrait generation is prominent. For example, virtual social platforms need to generate highly realistic 3D virtual avatars based on users' single selfies, while film and television productions need to convert actors' single portraits into 3D character models that can be adjusted from multiple angles. These scenarios require 3D portraits to match the features and textures of the original image, support multi-view rendering, and adapt to different poses, expressions, and lighting, placing certain demands on generation efficiency and generalization ability.

[0003] Currently, the mainstream solution for this requirement is the traditional neural radiation field generation method. This method takes a single 2D portrait image as input, constructs a neural radiation field model to learn the mapping relationship between pixel color and spatial position, and renders 3D portraits from different perspectives using a ray casting algorithm, achieving the conversion from a single image to a 3D portrait. It has already been applied in scenarios with low detail requirements.

[0004] However, this approach has significant drawbacks. It lacks effective geometric and photometric priors, struggles to capture structural changes caused by differences in viewing angles, is prone to overfitting local features, and loses subtle details such as skin texture and hair strands. Furthermore, the model's expressive power is limited; it is prone to contour distortion and facial feature imbalance when processing complex poses or facial expressions, has weak noise handling, and is prone to blurring and inconsistencies in detail when rendering from multiple perspectives, failing to meet high-fidelity requirements. Summary of the Invention

[0005] The purpose of this application is to provide a single-image 3D portrait generation method and system based on hybrid prior and noise resampling, so as to solve the problems of detail loss and overfitting in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a single-image 3D portrait generation method based on hybrid prior and noise resampling, comprising:

[0007] A pre-trained first model is used to generate multiple planar features based on the acquired first portrait image; the multiple planar features, along with the decoder parameters and super-resolution parameters corresponding to the first model, are combined into initial parameters.

[0008] By inputting the second portrait image into the pre-trained second model, the model parameters of the second model are fine-tuned, enabling the second model to generate a three-dimensional face structure corresponding to the second portrait image.

[0009] Based on the initial parameters and the fine-tuned second model, determine the target parameters;

[0010] The target parameters are used to generate three-dimensional face texture by mixing a prior diffusion model and a multi-view noise resampling strategy.

[0011] Optionally, the target parameters are used to generate three-dimensional face texture by mixing a prior diffusion model and a multi-view noise resampling strategy, including:

[0012] Based on the target parameters, an explicit branch processing, an implicit branch processing, and a geometric prior fusion are performed by mixing a prior diffusion model to generate a texture representation under a new view angle;

[0013] Based on the texture representation, a multi-view noise resampling strategy is used to initialize and optimize the anchor point noise to determine a noise distribution of resampling;

[0014] Based on the texture representation and the noise distribution of resampling, the target parameters are optimized by using an SDS loss and a multi-view noise resampling loss to generate three-dimensional face texture.

[0015] Optionally, based on the target parameters, an explicit branch processing, an implicit branch processing, and a geometric prior fusion are performed by mixing a prior diffusion model to generate a texture representation under a new view angle, including:

[0016] An explicit branch processing flow of the mixed prior diffusion model is used to determine a feature vector of each 3D face at a corresponding pixel position based on the target parameters by adjusting the generated detail size through a deformable container structure, and the feature vectors are spliced to generate a projection reference image, which is input as an explicit condition to control the generation of a new view angle;

[0017] An implicit branch processing flow of the mixed prior diffusion model is used to obtain a driving image based on the target parameters, apply a spatial attention mechanism to the driving image to highlight a foreground region in the current view, splice a fusion feature map obtained by activating features of the foreground region through a nonlinear layer, use the fusion feature map for texture understanding, and implicitly pass the result of texture understanding to the explicit branch flow through a residual block;

[0018] A geometric prior fusion module of the mixed prior diffusion model is used to obtain geometric prior information under a current view angle based on the target parameters, the geometric prior information including a rendered rough image and a normal map, the geometric prior information is input as a condition, and the geometric prior information is implicitly fused with a reference texture through a residual block to obtain a texture representation under a new view angle.

[0019] Optionally, based on the texture representation, anchor point noise is initialized and consistency optimized by using a multi-view noise resampling strategy to determine a resampled noise distribution, including:

[0020] Based on the texture representation, a geometric prior is obtained, the noise of the driving image is mapped to a three-dimensional point cloud according to the geometric prior, and the noise of the three-dimensional point cloud is rendered to a target view to complete the initialization of the anchor point noise.

[0021] In the training process of SDS, the initialized anchor point noise is resampled and updated according to multi-view gradient consistency to realize consistency optimization and alignment of multi-view representation.

[0022] Optionally, based on the texture representation and the resampled noise distribution, the target parameters are optimized by using SDS loss and multi-view noise resampling loss to generate a three-dimensional face texture, including:

[0023] The texture representation and the resampled noise distribution are input into an optimization process to perform the following operations:

[0024] SDS loss is used as a core loss function to iteratively optimize and update the target parameters by distilling prior knowledge of a two-dimensional diffusion model to a three-dimensional space.

[0025] At the same time, multi-view noise resampling loss is used to resample and update noise distribution based on multi-view gradient consistency, gradually align texture representations of different views, and ensure the similarity of texture details.

[0026] The optimized target parameters are rendered to generate a high-fidelity and detailed three-dimensional face texture.

[0027] Optionally, after determining the target parameters according to the initial parameters and the second model after fine-tuning, the method further includes:

[0028] A dynamic multi-view consistency supervision mechanism is introduced, a lightweight auxiliary discriminant network is constructed based on the target parameters and a plurality of predefined camera views, semantic consistency analysis of each view rendering image is performed through the auxiliary discriminant network to generate a multi-view consistency confidence map.

[0029] The consistency confidence map is used as an adaptive weight and fused into the multi-view noise resampling strategy of the hybrid prior diffusion model, the loss components of different views in the hybrid prior diffusion model are weighted calculated according to the adaptive weight, and the target parameters are updated through the weighted loss components.

[0030] Optionally, after generating the three-dimensional face texture, the method further includes:

[0031] input the three-dimensional face texture and the fine-tuned three-dimensional face structure into a differentiable renderer, render the three-dimensional face texture and the fine-tuned three-dimensional face structure through the differentiable renderer under multiple real light environments, and obtain rendered images under multiple light conditions;

[0032] calculate a high-order semantic difference between the rendered images and the input portrait images in a perceptual feature space, and construct a perceptual optimization loss according to the high-order semantic difference;

[0033] use the perceptual optimization loss to optimize a texture coordinate mapping of the three-dimensional face texture through a back propagation algorithm, perform nonlinear transformation and fine adjustment on the texture coordinate mapping, and output an optimized three-dimensional portrait model.

[0034] In a second aspect, the present application provides a single-image 3D portrait generation system based on a hybrid prior and noise resampling, comprising:

[0035] An acquisition module is configured to generate multiple plane features according to an acquired first portrait image by using a pre-trained first model, and combine the multiple plane features, decoder parameters corresponding to the first model, and super-resolution parameters into initial parameters.

[0036] A fine-tuning module is configured to input a second portrait image into a pre-trained second model to fine-tune model parameters of the second model, so that the second model can generate a three-dimensional face structure corresponding to the second portrait image.

[0037] A generation module is configured to determine target parameters according to the initial parameters and the fine-tuned second model.

[0038] An optimization module is configured to generate a three-dimensional face texture by using the target parameters, a hybrid prior diffusion model, and a multi-view noise resampling strategy.

[0039] In a third aspect, the present application provides an electronic device, comprising:

[0040] A memory is configured to store a computer program.

[0041] A processor is configured to implement the steps of the single-image 3D portrait generation method based on a hybrid prior and noise resampling when the computer program is executed.

[0042] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can implement the steps of the single-image 3D portrait generation method based on a hybrid prior and noise resampling when the computer program is executed by a processor.

[0043] The single-image 3D portrait generation method based on mixed prior and noise resampling provided in the application can quickly obtain basic features and model configuration and provide initial support for subsequent 3D generation by using a pre-trained first model to generate a plurality of plane features according to a first portrait image and combining the plane features with decoder parameters and super-resolution parameters of the first model into initial parameters; the second model can generate a corresponding three-dimensional face structure and accurately match the geometric shape of the target portrait, thereby improving the accuracy of the 3D structure; the target parameters can be determined according to the initial parameters and the second model after fine-tuning, so as to integrate the basic features and accurate geometric information and form core parameters suitable for 3D texture generation; and the three-dimensional face texture can be generated by using the target parameters, combining a mixed prior diffusion model and a multi-view noise resampling strategy, so as to take into account texture details and multi-view consistency and improve the realism of the 3D portrait.

[0044] Further, based on the target parameters, the new-view texture representation is obtained by performing explicit branch, implicit branch processing and geometric prior fusion through the mixed prior diffusion model, the resampling noise distribution is determined by initializing and optimizing the anchor point noise, and finally the three-dimensional face texture is generated by optimizing the target parameters using the SDS loss and the multi-view noise resampling loss. Through the multi-branch fusion and noise optimization strategy, the detail richness, multi-view consistency and adaptability to the geometric structure of the three-dimensional face texture are effectively improved, and the realism and reliability of the 3D portrait generation are further enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A flowchart of a single-image 3D portrait generation method based on mixed prior and noise resampling provided by an embodiment of the present application is shown in the figure.

[0047] Figure 2 A specific implementation flowchart of a single-image 3D portrait generation method based on mixed prior and noise resampling provided by an embodiment of the present application is shown in the figure.

[0048] Figure 3 A structure diagram of a single-image 3D portrait generation system based on mixed prior and noise resampling provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0049] In the fields of digital entertainment, virtual socialization, film production, etc., the traditional neural radiance field generation method of single image 3D portrait generation has obvious shortcomings: on the one hand, due to the lack of effective geometric and photometric priors, it is difficult to capture the structural changes brought by different observation angles, and it is easy to overfit local features of the image, resulting in the loss of fine details such as skin texture and hair; on the other hand, the model expression ability is limited, when processing portrait images with complex poses or expressions, the face contour is often deformed, the proportion of the five organs is unbalanced, and the noise processing ability is weak, which is easy to produce image blur and inconsistent details when rendering multiple views, and cannot meet the demand of high-fidelity 3D portrait generation.

[0050] In view of the above problems, the present application provides a single image 3D portrait generation method based on hybrid prior and noise resampling. The method realizes optimization through four core processes: first, a pre-trained model is used to generate plane features with rich details and combine them into initial parameters; then another pre-trained model is fine-tuned to adapt to the three-dimensional structure of a specific portrait; then the two are combined to determine high-quality target parameters; finally, a hybrid prior diffusion model and a multi-view noise resampling strategy are used to generate three-dimensional face texture. This scheme not only supplements geometric and photometric priors through pre-trained models and parameter combination to avoid detail loss and overfitting, but also improves the model's ability to adapt to complex portraits with model fine-tuning and noise resampling strategy, ensuring consistent multi-view texture, fundamentally solving the problems of traditional schemes such as lack of details, structural distortion, and poor multi-view effect, and generating high-fidelity 3D portraits that meet the needs.

[0051] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0052] The core of the present application is to provide a single image 3D portrait generation method based on hybrid prior and noise resampling, and a specific embodiment of the method is shown in the flowchart as Figure 1 The method comprises:

[0053] S101, using a pre-trained first model to generate a plurality of plane features according to the obtained first portrait image; combining the plurality of plane features and the decoder parameters and super-resolution parameters corresponding to the first model into initial parameters;

[0054] In this step, the pre-trained first model refers to a model trained in advance on a large amount of image data and capable of extracting image features, which can capture detail and texture information from a portrait image; the multiple plane features refer to data extracted from the first portrait image and capable of reflecting the geometric shape and surface texture of the image in different directions; the decoder parameters are parameters matched with the first model and used for converting the plane features into a three-dimensional space density distribution; the super-resolution parameters are parameters used for improving image clarity and restoring details; and the initial parameters refer to a parameter set obtained by integrating the plane features, the decoder parameters and the super-resolution parameters.

[0055] In a specific embodiment, a first portrait image is first obtained, and a clear front-facing face image is usually selected to ensure the completeness of feature extraction. The image is input into a pre-trained first model, and the training process of the model needs to collect a large-scale portrait image dataset containing different races, ages and expressions. The dataset is iteratively trained through a convolutional neural network (CNN) to minimize the image feature extraction error, and the network weight is continuously adjusted to finally form a model capable of stably extracting facial features.

[0056] When the model processes the input first portrait image, the pre-processing module is used to remove background interference in the image and focus on the face area. Then, the feature extraction layer is used to gradually analyze the facial information. The bottom layer network captures surface details such as skin texture, lip color and eyebrow shape, and the middle layer network extracts geometric features such as eye position and facial contour. Finally, the feature mapping module is used to arrange these features into multiple plane features corresponding to different viewing angles.

[0057] Then, the decoder parameters and the super-resolution parameters matched with the first model are called, and the two types of parameters and the generated multiple plane features are integrated in a preset format to form initial parameters required for subsequent processes.

[0058] For example, if a 3D image of a virtual social platform is to be generated for a user, a clear front-facing selfie of the user is first obtained as a first portrait image. The image is input into a pre-trained first model based on a Res Net architecture, which has been trained through a dataset containing 100,000 different portraits and is capable of stably extracting facial features. The model first removes the background in the selfie and only retains the face area of the user. Then, the bottom layer network captures the delicate texture of the user's skin, the direction of the dark brown eyebrows and the light red lip color, and the middle layer network extracts geometric features such as the round facial contour, medium eye distance and slightly straight nose bridge of the user. Finally, 3 sets of plane features are generated, corresponding to the facial features in the front, left side 30° and right side 30° viewing angles.

[0059] Subsequently, the decoder parameters of the model and the super-resolution parameters are called, the decoder parameters are used for subsequent conversion of the planar features into three-dimensional density data, and the super-resolution parameters are used to improve the definition of the skin texture of the 3D image. Integrating the three groups of planar features and the two types of parameters forms the initial parameters. The above example is only one example of the present application, and in actual application, the angle of the first portrait image, the size of the training data set of the model, and the number of output planar features can be set according to requirements, and the present application does not limit this.

[0060] S102, by inputting the second portrait image into the pre-trained second model, the model parameters of the second model are fine-tuned to enable the second model to generate a three-dimensional face structure corresponding to the second portrait image;

[0061] The second portrait image is an input image for adapting the model to the unique structure of a specific portrait, containing exclusive structural information such as the facial contour and the proportion of the features of the specific portrait, and is the core basis for fine-tuning the model parameters. The pre-trained second model is a model that has been trained in advance on a large portrait data set and has the basic ability to generate a general three-dimensional face structure, which can be used as an initial model for subsequent adaptation to a specific portrait. The model parameters are variables inside the second model for calculation and feature learning, and after fine-tuning, they can enable the model to accurately match the structural features of a specific portrait. The three-dimensional face structure is a collection of information that can reflect the three-dimensional shape of the face, including facial depth, feature spatial position, and contour curve, and is the core geometric framework for 3D portrait generation.

[0062] In a specific embodiment, the second portrait image is first obtained, and images containing different angles of the specific portrait, such as front and side views, are usually selected to fully cover its facial structural features. These images are input into the pre-trained second model. The training process of this model needs to first construct a large-scale portrait data set containing various facial poses and expressions. The UNet network architecture is used to generate a three-dimensional structure that conforms to the geometric laws of real faces as the goal. Through multiple rounds of iterative training to adjust the network parameters, a pre-trained model with the ability to generate a general three-dimensional face structure is finally formed.

[0063] The model fine-tuning stage needs to introduce a loss function to measure the difference between the generated result and the second portrait image. Here, the SDS (Score Distillation Sampling) loss function is used, and the core formula is:

[0064]

[0065] wherein, is the SDS loss value, represents the expected operation, is a standard normal distribution noise sampled in the middle of the diffusion process, is the noise of the model prediction, is the image feature at the moment, is the noise of the model prediction, denotes the L2 norm. This formula guides the model to learn the structural features of the second portrait image by calculating the difference between the model prediction noise and the real noise.

[0066] The specific fine-tuning process is as follows: the second portrait image is input into the pre-trained model, and the model first generates an initial three-dimensional face structure; the error between the structure and the second portrait image is calculated by the above SDS loss function; according to the error value, the internal parameters of the model are adjusted through the back propagation algorithm, and the error is gradually reduced; repeat the process, usually iterate 30-50 rounds, until the three-dimensional face structure generated by the model completely matches the structural features of the second portrait image, the fine-tuning is completed, and the model has the ability to generate corresponding exclusive three-dimensional face structure.

[0067] In another specific embodiment, the second portrait image can be selected to contain a specific expression of a specific person, such as a smiling or frowning image, to adapt to the dynamic structure requirement. The pre-trained second model can use an improved U Net architecture, and an attention mechanism module is added to the network to enhance the feature learning ability of key areas such as eyes and mouth. In addition to the SDS loss, a geometric consistency loss is introduced during fine-tuning to ensure the coherence of the generated three-dimensional structure under different viewing angles, further improving the structure accuracy.

[0068] S103, determining a target parameter according to the initial parameter and the fine-tuned second model;

[0069] The target parameter is a core parameter set for generating adaptive three-dimensional face texture formed by integrating the basic features of the initial parameter and the exclusive structure generation ability of the fine-tuned second model.

[0070] In a specific embodiment, the core information of the initial parameter and the fine-tuned second model is first extracted. The plane features, decoder parameters, and super-resolution parameters are parsed from the initial parameter to clarify the role of these parameters in texture detail capture, three-dimensional density conversion, and clarity improvement; at the same time, the key parameters related to the three-dimensional structure of a specific person in the fine-tuned second model are read, such as the network layer weights that control the facial contour and the position of the five organs, to determine the adaptation logic of the model to the exclusive geometric structure.

[0071] The parameter fusion stage uses a feature structure mapping method to first establish the spatial correspondence between the plane features in the initial parameter and the three-dimensional structure of the fine-tuned model, ensuring that the plane features can be accurately mapped to the corresponding area of the three-dimensional structure. This process needs to verify the mapping accuracy through similarity calculation, and the core formula is:

[0072]

[0073] in, Indicates the first Planar features With three-dimensional structure each region similarity, It is the dot product of the two. and They are and The L2 norm. When the similarity value is greater than a preset threshold, the mapping relationship is considered valid; the threshold is usually set to 0.85.

[0074] Assume a certain skin texture feature in the initial parameters. The vector is After fine-tuning, the three-dimensional structure of the facial skin area The vector is Then the dot product ; ; Similarity If the value is greater than the threshold of 0.85, the mapping relationship is valid.

[0075] After mapping, the decoder parameters in the initial parameters are adjusted to adapt to the density distribution of the fine-tuned 3D structure, avoiding reconstruction distortion caused by parameter incompatibility. Finally, through multiple rounds of parameter collaborative verification, the matching degree between the preliminary 3D structure generated by the fused parameters and the second portrait image is compared. If the matching degree reaches 90% or more, it can be determined as the target parameter.

[0076] In another specific implementation, parameter fusion can introduce a lightweight parameter adaptation network. This network is constructed with 5-8 convolutional and activation layers and pre-trained on a dataset containing different portrait parameters. The optimization objective is the matching degree of the generated structure after parameter fusion. It can automatically learn the fusion weights of the initial parameters and the fine-tuned model parameters, reducing manual adjustment costs. Simultaneously, multi-view matching detection can be added during the validation phase to ensure that the target parameters can adapt to specific portrait structures under different viewpoints, further improving parameter reliability.

[0077] S104. Using the target parameters, a three-dimensional face texture is generated by a hybrid prior diffusion model and a multi-view noise resampling strategy.

[0078] In this step, the mixed prior diffusion model is a model including an explicit branch, an implicit branch and a geometric prior fusion module, which is used to generate a texture representation under a new view in combination with target parameters, and takes into account the texture details and the adaptability of geometric structure; the multi-view noise resampling strategy is a strategy for initializing and optimizing the consistency of anchor point noise, which is used to ensure the uniformity of texture details under different views; the three-dimensional face texture is the final output image data containing skin texture, hair shape and other details, which needs to be accurately matched with the three-dimensional face structure, and is the core element of 3D portrait visual presentation.

[0079] Optionally, as shown in Figure 2 Step S104 can specifically include the following steps:

[0080] S1041, based on the target parameters, performing explicit branch processing, implicit branch processing and geometric prior fusion through a mixed prior diffusion model to generate a texture representation under a new view;

[0081] In step S1041, the explicit branch processing flow of the mixed prior diffusion model is used to determine the feature vector of each 3D face at the corresponding pixel position based on the target parameters and the generated detail size of the deformable container structure, and the feature vectors are spliced to generate a projection reference image, which is input as an explicit condition to control the generation of a new view; the implicit branch processing flow of the mixed prior diffusion model is used to obtain a driving image based on the target parameters, apply a spatial attention mechanism to the driving image to highlight the foreground region in the current view, splice the feature maps of the foreground region after nonlinear layer activation to obtain a fusion feature map, use the fusion feature map for texture understanding, and implicitly pass the results of texture understanding to the explicit branch flow through a residual block; the geometric prior fusion module of the mixed prior diffusion model is used to obtain geometric prior information under the current view based on the target parameters, the geometric prior information including a rendered rough image and a normal map, the geometric prior information being input as a condition, and the geometric prior information and the reference texture being implicitly fused through a residual block to obtain a texture representation under a new view.

[0082] In the above steps, the deformable container structure is used to adjust the size of the generated details according to the texture requirements of different regions under the new perspective; the feature vector is data reflecting the color, gloss, and other attributes of the 3D surface at the corresponding pixel position; the projection reference image is an image generated after stitching the feature vectors, used to control the generation direction of the new perspective texture; the spatial attention mechanism is used to highlight the foreground region in the driving image, weaken the background, and focus on the core texture information; the fused feature map is data obtained by stitching the foreground region feature map after activation, used to more accurately understand the texture features; the geometric prior information includes the rendered coarse image and normal map, used to enhance the fit between the texture and the geometric structure; the texture representation is the basic data of the new perspective texture obtained by fusing the explicit and implicit branch results and the geometric prior, containing the texture details and structural mapping relationship.

[0083] In one specific implementation, explicit branching is first performed. Based on the target parameters, the size of the details is adjusted using a deformable container structure: for the facial skin area, the detail size is set to 1×1 pixels to present a delicate texture; for the hair area, the detail size is set to 3×3 pixels to present hair strand layers.

[0084] Next, the feature vector of each 3D surface at the corresponding pixel position is determined. For example, for a certain 3D surface corresponding to the pixels of the cheek area, the feature vector of that area is extracted: skin color RGB value: 255,224,189, glossiness 0.3. Then, the feature vectors of all 3D surfaces are stitched together according to their spatial positions to generate a projection reference image. This image is used as an explicit conditional input to control the generation direction of new perspectives, such as the 45° texture on the right.

[0085] Secondly, implicit branching is performed. A driving image is obtained based on the target parameters, typically a multi-angle portrait image corresponding to the target parameters. A spatial attention mechanism is applied to the driving image, calculated by weighting pixels. The facial region weight is set to 0.9, and the background region weight is set to 0.1, thus highlighting the foreground area and minimizing background interference. Feature maps of the foreground region, such as the skin texture around the eyes and eyebrow shape, are activated through a ReLU nonlinear layer. The activation formula is: This activation formula highlights effective texture features and filters out invalid information by retaining pixel values ​​greater than 0 in the feature map and setting pixel values ​​less than or equal to 0 to 0. For example, if a pixel value in the feature map is -0.2, it will be 0 after activation; if it is 0.5, it will remain 0.5 after activation.

[0086] The activated feature maps are then concatenated by channel to obtain a fused feature map. The fused feature map is used to understand the texture, such as identifying the thickness and direction of eyebrows. The texture understanding results are implicitly passed to the explicit branch through residual blocks. The residual blocks can preserve the original feature information, avoid gradient vanishing, and assist the explicit branch in adjusting the texture generation details.

[0087] Finally, the geometric prior is fused. The geometric prior information in the current view is obtained based on the target parameters: a rough image is rendered to present the facial contour, and a normal map is generated to reflect the orientation of the nose bridge, jaw, etc. For example, the normal direction of the nose bridge area is 45°, indicating that the area is inclined upward. The geometric prior information is input as a condition, and the projection reference image generated by the residual block and the texture understanding result transmitted by the implicit branch are implicitly fused. For example, the orientation information of the nose bridge area in the normal map is combined with the texture features of the nose bridge in the projection reference image to ensure that the texture fits the three-dimensional structure of the nose bridge, and finally the texture representation in the new view is generated.

[0088] S1042、based on the texture representation, using a multi-view noise resampling strategy, initializing and consistency optimizing the anchor point noise to determine the resampled noise distribution;

[0089] In the step S1042, the process can include the following steps: obtaining the geometric prior based on the texture representation, mapping the noise of the driving image to the three-dimensional point cloud according to the geometric prior, and rendering the noise of the three-dimensional point cloud to the target view to complete the initialization of the anchor point noise; in the training process of SDS, the initialized anchor point noise is resampled and updated according to the multi-view gradient consistency to realize the consistency optimization and alignment of multi-view representation.

[0090] In the above steps, the anchor point noise is the basic noise data used to affect the generation of texture details, and the initial state needs to match the geometric structure; the three-dimensional point cloud is a set of three-dimensional space points converted from the driving image noise according to the geometric prior, used to establish the association between noise and three-dimensional structure; the target view is the view of the texture to be generated, which is commonly front, left 45°, and right 45°; the multi-view gradient consistency means that the change rule of noise gradient in different views needs to be consistent, and the difference in gradient direction and size needs to be within a reasonable range, which is used to ensure the uniformity of multi-view texture details; the resampled noise distribution is the distribution state of the optimized anchor point noise, which is used for subsequent target parameter optimization to ensure the texture consistency.

[0091] In a specific embodiment, first, anchor point noise initialization is performed. Based on the texture representation generated in S1041, geometric priors are extracted therefrom, including depth of each region of the face and contour information. According to the geometric priors, noise of the driving image is mapped to a three-dimensional point cloud, and the noise of the driving image is subject to a standard normal distribution. For example, the noise value of a certain pixel in the driving image is 0.8, and the corresponding region in the texture representation is the skin region of the face, and the depth value of the region is 5. Then, the noise value is mapped to the coordinates x=10, y=8, and z=5 of the three-dimensional point cloud. Subsequently, the noise of the three-dimensional point cloud is rendered to the target view through a ray casting algorithm, and the target view can be set as the front view, the left side 45° view, and the right side 45° view. The initialization of the anchor point noise is completed, and the initial noise is ensured to match the three-dimensional structure.

[0092] Secondly, anchor point noise consistency optimization is performed. In the SDS training process, the initialized anchor point noise is resampled and updated according to the multi-view gradient consistency. The gradient of the anchor point noise in different target views is calculated, and the gradient formula is as follows:

[0093]

[0094] wherein, is the anchor point noise value, is the three-dimensional coordinate. If the gradient of a certain anchor point in the front view is , the gradient of the anchor point in the left side view is , and the difference between the two is too large.

[0095] The L2 norm of the gradient difference is calculated, and the calculation process is as follows: If the value is greater than the preset threshold value 0.5, the noise value of the point is resampled: the noise value 0.5 is resampled from the standard normal distribution, and the L2 norm of the gradient difference after updating is 0.35, which is less than the threshold value 0.5. The consistency optimization and alignment of the multi-view representation are achieved, and the resampled noise distribution is finally determined.

[0096] In another specific embodiment, the anchor point noise initialization can introduce a noise filtering mechanism to remove abnormal noise in the driving image caused by shooting interference, such as noise caused by sudden changes in brightness; the consistency optimization can increase the view weight, and the gradient consistency requirement of the core view such as the front view is higher, and the requirement of the secondary view such as the rear side is appropriately relaxed, further improving the texture quality of the core view.

[0097] S1043, based on the texture representation and the resampled noise distribution, the target parameters are optimized using the SDS loss and the multi-view noise resampling loss, and a three-dimensional face texture is generated.

[0098] In step S1043, the texture representation and the resampled noise distribution can be input into an optimization process, and the following operations can be performed: using SDS loss as a core loss function, iteratively updating and optimizing the target parameters by distilling prior knowledge of a two-dimensional diffusion model to a three-dimensional space; and using multi-view noise resampling loss, resampling and updating the noise distribution based on multi-view gradient consistency, gradually aligning the texture representations of different views, and ensuring the similarity of texture details; and rendering the optimized target parameters to generate a high-fidelity and detailed three-dimensional face texture.

[0099] In the above steps, the SDS loss is a core loss function for optimizing the target parameters by distilling prior knowledge of a two-dimensional diffusion model to a three-dimensional space to improve texture fineness; the multi-view noise resampling loss is an auxiliary loss function for optimizing the noise distribution based on multi-view gradient consistency to ensure the similarity of texture details of different views; the iterative updating and optimization refers to a process of calculating loss and adjusting parameters for multiple rounds to gradually reduce the difference between the texture and the ideal effect; and the high-fidelity three-dimensional face texture is the generated texture after optimization, which has the characteristics of rich details, consistent multi-views, and accurate adaptation to geometric structure.

[0100] In a specific embodiment, the texture representation generated in S1041 and the resampled noise distribution determined in S1042 are first input into an optimization process.

[0101] In the first step, the SDS loss is used to optimize the target parameters, and the SDS loss formula is as follows:

[0102]

[0103] wherein, is the SDS loss value, represents an expected operation, is noise sampled from a standard normal distribution , such as a sampling value of is a texture representation feature at time t in the diffusion process, such as , is a feature vector of a certain skin area is a model-predicted noise, such as a predicted value of represents an L2 norm. The single-round loss is calculated as follows: The target parameters are adjusted through a backpropagation algorithm to reduce the loss, and after 30 iterations, the SDS loss is reduced from the initial 0.08 to below 0.01, and the texture fineness is significantly improved.

[0104] In the second step, the multi-view noise resampling loss is used for optimization. The multi-view noise resampling loss formula is as follows:

[0105]

[0106] in, This represents the multi-view noise resampling loss value. For different number of perspectives, for example, M represents the front and N represents the left. They are respectively The gradient of anchor point noise from the perspective, for example, for for .

[0107] Calculate single-round losses: By adjusting the noise distribution and target parameters through backpropagation, after 20 iterations, the loss decreased from the initial 0.3 to below 0.1, and the consistency of texture from different perspectives was improved.

[0108] Finally, the optimized target parameters are rendered using a differentiable renderer to generate a high-fidelity 3D face texture: the skin area shows a delicate pore texture, the hair area shows clear hair strand layers, and the texture details remain consistent from the front, left 45° and other viewpoints.

[0109] In another specific implementation, the optimization process can introduce a loss weight adjustment mechanism. In the early stage, SDS loss is given a higher weight, such as 0.7, to prioritize improving texture detail. In the later stage, multi-view noise resampling loss is given a higher weight, such as 0.6, to prioritize ensuring consistency. In the rendering stage, texture smoothing can be added to remove local noise spots that still exist after optimization, further improving the texture visual effect.

[0110] For example, to optimize the generation of 3D face textures for virtual idols: the texture representation and resampling noise distribution are input into the optimization process. In the first round of SDS loss calculation, the sampling noise... The model prediction noise is 0.7. The value is 0.6, and the loss value is In the calculation of multi-view noise resampling loss, the front and right sides... The view gradients are respectively loss value After 30 iterations, the SDS loss decreased to 0.008, and the multi-view loss decreased to 0.09. The rendered 3D facial texture of the virtual idol was obtained: the skin exhibited a natural sheen and fine pores, the brown hair displayed clear, curly strands, and the texture showed no significant difference from different viewpoints. The above example is merely one illustration of this application. In practical applications, the number of iterations, loss weights, rendering parameters, etc., can all be set according to requirements, and this application does not impose any limitations on them.

[0111] The application generates three-dimensional face texture by combining the prior diffusion model and the multi-view noise resampling strategy. The generated three-dimensional face texture has rich details and can clearly present subtle features such as skin texture and hair shape. The texture consistency under different views is guaranteed, and the details are not disjointed when the view is switched. At the same time, the adaptability of the texture to the three-dimensional face structure is significantly improved, effectively solving the problems of rough texture, inconsistent views, and poor structure fitting in traditional solutions, greatly improving the visual fidelity of 3D portraits, and making them more meet the needs of high-resolution 3D portraits in digital entertainment, virtual social interaction, film production and other scenarios, expanding the application range and practical value of single-image 3D portrait generation technology.

[0112] Optionally, after determining the target parameter according to the initial parameter and the second model after fine tuning, the method further comprises:

[0113] A dynamic multi-view consistency supervision mechanism is introduced, and a lightweight auxiliary discriminant network is constructed based on the target parameter and a plurality of predefined camera views. The auxiliary discriminant network performs semantic consistency analysis on the rendered images of each view to generate a multi-view consistency confidence map. The consistency confidence map is used as an adaptive weight and is fused into the multi-view noise resampling strategy of the hybrid prior diffusion model. The adaptive weight is used to weight the loss components of different views in the hybrid prior diffusion model, and the target parameter is updated based on the weighted loss components.

[0114] The dynamic multi-view consistency supervision mechanism is an auxiliary optimization mechanism for improving the consistency of 3D portrait texture and structure under multi-view. By analyzing the semantic matching degree of the rendered results of each view in real time, the model loss calculation weight is dynamically adjusted to ensure the uniformity of the generation effect under different views.

[0115] The plurality of predefined camera views are typical views set in advance according to the requirements of the application scenario, covering the common display angles of 3D portraits. The lightweight auxiliary discriminant network is a neural network with simple structure and low computational cost, which is used for semantic analysis of multi-view rendered images to avoid increasing the computational burden of the overall process.

[0116] The semantic consistency analysis is to compare the feature matching degree of the same semantic region in different view images through the network to determine the consistency level between the views. The consistency confidence map is an image output by the auxiliary discriminant network, which reflects the semantic consistency degree between the views. The numerical value of different regions represents the consistency level of the region under multi-view.

[0117] The adaptive weight is a weight value generated based on a consistency confidence map, and is used to adjust the importance of a loss component. A lower consistency corresponds to a higher weight to strengthen the optimization of the area.

[0118] In a specific embodiment, a dynamic multi-view consistency supervision mechanism is first introduced, and a lightweight auxiliary discriminant network is constructed based on target parameters and a plurality of predefined camera views. First, the required predefined camera views for the application scenario are determined, for example, in a virtual idol live streaming scenario, four views of front, 30° side, 60° side, and slightly upward are set. Based on the target parameters, preliminary view rendering images are generated under each predefined view.

[0119] When constructing the lightweight auxiliary discriminant network, a simplified convolutional layer and pooling layer structure is used: including 3 convolutional layers (convolution kernel size 3x3), 2 pooling layers (pooling kernel size 2x2), and 1 fully connected layer, the total amount of network parameters is controlled within 1 million, and the calculation time is reduced. The training process of this network needs to prepare a dataset containing multi-view portraits in advance, aiming to accurately judge the semantic consistency of different views, using cross-entropy loss function for iterative training, and adjusting the network weight until the consistency judgment accuracy reaches the preset standard.

[0120] Next, the auxiliary discriminant network is used to analyze the semantic consistency of the view rendering images. The rendering images of each predefined view are input into the auxiliary discriminant network, which first performs semantic segmentation on each image to label key semantic regions such as eyes, nose, and mouth; then calculates the feature similarity of the same semantic regions between different views, the higher the similarity, the stronger the consistency. Then, a multi-view consistency confidence map is generated based on the consistency results of each semantic region, with different values representing the consistency level, and the higher the value, the higher the consistency.

[0121] After that, the consistency confidence map is used as an adaptive weight and is integrated into the multi-view noise resampling strategy of the hybrid prior diffusion model. First, the values in the consistency confidence map are converted into corresponding weight values in proportion, to ensure that areas with lower consistency correspond to higher adaptive weights. For example, if the value of a certain area in the confidence map is 0.3, the consistency is low, and the weight is converted to 0.8; if the value of a certain area is 0.9, the consistency is high, and the weight is converted to 0.2. Then, the adaptive weight is integrated into the loss calculation link of the multi-view noise resampling strategy, replacing the previous fixed weight setting, so that the strategy can dynamically adjust the optimization focus according to the consistency level of each area.

[0122] Finally, the loss components of different views in the hybrid prior diffusion model are calculated by weighting according to the adaptive weights, and the target parameters are updated by the weighted loss components. First, the loss components of the hybrid prior diffusion model are calculated under each predefined view to reflect the error between the generated result and the ideal effect under that view. Then, the loss components of each view are multiplied by the adaptive weights of the corresponding region to obtain the weighted loss components. Taking the weighted loss components as the optimization target, the target parameters are adjusted by the back propagation algorithm to gradually improve the multi-view matching degree of the low consistency region, and the optimization of the target parameters is completed.

[0123] For example, in the virtual idol live scene, four views of front view, 30° side view, 60° side view, and slightly upward view are predefined. First, the loss components of the hybrid prior diffusion model under each view are calculated (the loss function formula has been described in detail before, and will not be repeated here): the front view hair outline loss component is 0.4, the 30° side view is 0.5, the 60° side view is 0.55, and the slightly upward view is 0.6. These values reflect the error between the generated effect and the ideal effect of the hair outline under each view.

[0124] Combined with the previously converted adaptive weights, the hair outline region weight is 0.9 and the eye region weight is 0.1. The loss components of each view are multiplied by the corresponding weights to obtain the weighted loss components: the front view is 0.4x0.9=0.36, the 30° side view is 0.5x0.9=0.45, the 60° side view is 0.55x0.9=0.495, and the slightly upward view is 0.6x0.9=0.54.

[0125] Taking these weighted loss components as the optimization target, the back propagation algorithm is started, and the algorithm automatically locates the network layer weight in the target parameters that controls the generation of the hair shape. According to the size of the weighted loss, the parameter value is adjusted: the slightly upward view weighted loss 0.54 is the largest, and the corresponding parameter adjustment amplitude is the largest; the front view weighted loss 0.36 is the smallest, and the adjustment amplitude is the smallest. After 5 rounds of iterative updates, the hair outline loss components of each view are calculated again, all of which are reduced to below 0.3, the shape matching degree of the hair outline under different views is significantly improved, and the optimization of the target parameters is completed. The above example is only one example of the present application, and in actual application, the initial value of the loss component, the number of iterative updates, the parameter adjustment amplitude, etc. can be set according to the requirements, and the present application does not limit this.

[0126] Through the optimization step, the present application accurately locates the region with insufficient consistency under multiple views and optimizes it, avoiding the blindness of fixed weight optimization. Under the premise of not significantly increasing the calculation cost, the texture and structure consistency of the 3D portrait under different views is improved, avoiding the problems of shape disconnection and texture breakage when switching views, and enhancing the scene adaptability and visual coherence of the 3D portrait.

[0127] Optionally, after the three-dimensional face texture is generated, the method further comprises:

[0128] The three-dimensional face texture and the fine-tuned three-dimensional face structure are jointly input into a differentiable renderer, rendering is performed by the differentiable renderer under a plurality of real light environments, and a plurality of light conditions are obtained. The rendering images are calculated in a perceptual feature space with an input portrait image, a perceptual optimization loss is constructed according to the high-order semantic difference, the texture coordinate mapping of the three-dimensional face texture is optimized by using the perceptual optimization loss through a back propagation algorithm, nonlinear transformation and fine adjustment are performed on the texture coordinate mapping, and an optimized three-dimensional portrait model is output.

[0129] In the above steps, the differentiable renderer is a rendering tool supporting gradient back propagation, which can calculate the error gradient of the image and the target in the rendering process, and provide a basis for subsequent parameter optimization; the plurality of real light environments are light scenes commonly seen in reality, covering possible display light conditions of the 3D portrait; the rendering images under the plurality of light conditions are 3D portrait images generated by the differentiable renderer in different light environments, reflecting the influence of light on the visual presentation of texture and structure.

[0130] The perceptual feature space is a feature dimension constructed based on human visual perception, including semantic information and style features of the image; the high-order semantic difference is a difference in a complex semantic layer calculated in the perceptual feature space, such as the coordination of skin color and shade, the rationality of facial shadow distribution, etc.; the perceptual optimization loss is a loss function constructed based on the high-order semantic difference, used to quantify the gap between the rendering effect and the ideal effect, and guide the optimization of texture parameters.

[0131] The texture coordinate mapping is a coordinate correspondence relationship of accurately matching the three-dimensional face texture to the corresponding positions of the three-dimensional face structure, which determines the fitting degree of the texture on the three-dimensional structure; the nonlinear transformation is a nonlinear adjustment operation on the texture coordinate mapping, used to correct local deviations in the coordinate mapping; and the optimized three-dimensional portrait model is a complete 3D portrait model that can maintain visual authenticity under a plurality of light conditions after light adaptation and texture coordinate optimization.

[0132] In a specific embodiment, rendering under a plurality of light environments is first performed. A plurality of real light parameters are collected, including indoor warm light, outdoor noon strong light, side light, backlight, etc., and are imported into the differentiable renderer. The three-dimensional face texture generated before and the fine-tuned three-dimensional face structure are imported into the renderer according to the coordinate correspondence relationship, the renderer calculates the reflection, refraction and shadow distribution of light on the face according to different light parameters, and generates rendering images under each light condition. For example, under the outdoor strong light environment, the rendering image presents obvious highlights in the nose bridge area and clear shadows in the chin area, simulating the real light effect.

[0133] Then, the high-order semantic difference is calculated and the perceptual optimization loss is constructed. First, the features of the rendered image and the input portrait image in the perceptual feature space are extracted by the pre-trained visual perception model, including skin tone, facial shadow transition curve, and feature contour sharpness. Then, the high-order semantic difference of the two images in these feature dimensions is calculated, and the core formula is:

[0134]

[0135] wherein, is the high-order semantic difference value, is the number of perceptual feature dimensions, is the perceptual feature value of the rendered image in the th dimension, is the perceptual feature value of the input portrait image in the th dimension, is the L2 norm.

[0136] Calculation example: select 3 core perceptual feature dimensions: skin tone, shadow transition, and contour sharpness, . The feature values of the rendered image are (skin tone), (shadow transition), (contour sharpness); the feature values of the input portrait image are .

[0137] Calculate the L2 norm of the difference in each dimension:

[0138] Skin tone: ;

[0139] Shadow transition: ;

[0140] Contour sharpness: ;

[0141] High-order semantic difference .

[0142] Based on the high-order semantic difference, the perceptual optimization loss is constructed, and the loss formula is:

[0143]

[0144] wherein, is the perceptual optimization loss value, is the weight coefficient (usually set to 10). Substituting the example data, .

[0145] Finally, texture coordinate mapping optimization is performed. A perceptual optimization loss is used as an optimization objective, a gradient of the loss with respect to the texture coordinate mapping is calculated through a back propagation algorithm, and areas that need to be adjusted in the coordinate mapping are determined. Nonlinear transformation is performed on the texture coordinate mapping of these areas, and the transformation formula is:

[0146]

[0147] wherein, is the adjusted texture coordinate, is the original texture coordinate, is a nonlinear transformation coefficient, controls the bending degree, controls the translation amplitude, controls the offset.

[0148] For example, for the area where the transition of the nose bridge shadow is unnatural, the original coordinate is set to , and the adjusted coordinate . Meanwhile, fine adjustment is performed on the detail areas such as the eye and lip, to ensure the matching accuracy of the texture and structure in local details. After multiple rounds of iterative optimization, the optimized three-dimensional portrait model is output.

[0149] Through the optimization step, the application effectively solves the problem of visual distortion of traditional 3D portraits under different lighting conditions, so that the model can adapt to various display scenarios in reality. The optimized three-dimensional portrait model can maintain visual authenticity and naturalness under various lighting conditions, the fitting accuracy of the texture and geometric structure is significantly improved, the scene adaptability and visual expressiveness of the 3D portrait are significantly enhanced, and the scene requirements of high lighting adaptation requirement such as film and television, virtual social interaction are met.

[0150] Figure 3 is a structural schematic diagram of a specific embodiment of a single-image 3D portrait generation system based on hybrid prior and noise resampling provided by the application, referring to Figure 3 The system can include:

[0151] The acquisition module 31 is configured to generate a plurality of plane features from a first portrait image acquired by using a pre-trained first model; and combine the plurality of plane features, a decoder parameter corresponding to the first model, and a super-resolution parameter into initial parameters.

[0152] The fine-tuning module 32 is configured to input a second portrait image into a pre-trained second model to fine-tune model parameters of the second model, so that the second model can generate a three-dimensional face structure corresponding to the second portrait image.

[0153] The generation module 33 is configured to determine target parameters according to the initial parameters and the fine-tuned second model.

[0154] An optimization module 34 is configured to generate a three-dimensional face texture by mixing a priori diffusion model and multi-view noise resampling strategy using the target parameter.

[0155] The single-image 3D portrait generation system based on mixed priori and noise resampling of the embodiments of the present application is used to implement the single-image 3D portrait generation method based on mixed priori and noise resampling as described above, and therefore the specific embodiments of the single-image 3D portrait generation system based on mixed priori and noise resampling can be seen from the embodiment part of the single-image 3D portrait generation method based on mixed priori and noise resampling as described above, and the specific embodiments can be referred to the description of the corresponding embodiment part, which will not be described here.

[0156] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the single-image 3D portrait generation method based on mixed priori and noise resampling as described above.

[0157] The present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the single-image 3D portrait generation method based on mixed priori and noise resampling as described above.

[0158] In an exemplary embodiment, the computer readable storage medium as described above can include but is not limited to: a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0159] The embodiments of the present application also provide a computer program product, and the computer program product comprises a computer program, and the computer program is executed by a processor to implement the steps of any one of the single-image 3D portrait generation method embodiments based on mixed priori and noise resampling.

[0160] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in a general manner in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0161] The above describes in detail a single image 3D portrait generation method and system based on a hybrid prior and noise resampling provided by the present application. In this paper, specific examples are used to illustrate the principles and implementation modes of the present application. The above description of the embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A single-image 3D portrait generation method based on hybrid prior and noise resampling, characterized in that, include: The first pre-trained model generates multiple planar features based on the acquired first portrait image; The multiple planar features and the decoder parameters and super-resolution parameters corresponding to the first model are combined into initial parameters; By inputting the second portrait image into the pre-trained second model, the model parameters of the second model are fine-tuned, enabling the second model to generate a three-dimensional face structure corresponding to the second portrait image. Based on the initial parameters and the fine-tuned second model, determine the target parameters; Using the target parameters, a three-dimensional face texture is generated by a hybrid prior diffusion model and a multi-view noise resampling strategy. Specifically, using the target parameters, a 3D face texture is generated through a hybrid prior diffusion model and a multi-view noise resampling strategy, including: Based on the target parameters, a new perspective texture representation is generated by using a hybrid prior diffusion model to perform explicit branching, implicit branching, and geometric prior fusion. Based on the texture representation, a multi-view noise resampling strategy is adopted to initialize and optimize the consistency of anchor point noise, and determine the noise distribution of resampling. Based on the texture representation and the noise distribution of the resampled data, the target parameters are optimized using SDS loss and multi-view noise resampling loss to generate a three-dimensional face texture.

2. The method according to claim 1, characterized in that, Based on the target parameters, a new perspective texture representation is generated through explicit branching, implicit branching, and geometric prior fusion using a hybrid prior diffusion model, including: Using the explicit branching process of the hybrid prior diffusion model, based on the target parameters, the generated detail size is adjusted through a deformable container structure to determine the feature vector of each 3D surface at the corresponding pixel position, and the feature vectors are concatenated to generate a projection reference image. The projection reference image is then used as an explicit conditional input to control the generation of new perspectives. Using the implicit branching process of the hybrid prior diffusion model, a driving image is obtained based on the target parameters. A spatial attention mechanism is applied to the driving image to highlight the foreground region in the current view. The feature map of the foreground region is activated by a nonlinear layer and then stitched together to obtain a fused feature map. The fused feature map is used for texture understanding, and the result of texture understanding is implicitly passed to the explicit branching process through residual blocks. The geometric prior fusion module of the hybrid prior diffusion model is used to obtain geometric prior information under the current viewpoint based on the target parameters. The geometric prior information includes a rendered coarse image and a normal map. The geometric prior information is used as a conditional input, and the geometric prior information is implicitly fused with the reference texture through residual blocks to obtain a texture representation under the new viewpoint.

3. The method according to claim 1, characterized in that, Based on the texture representation, a multi-view noise resampling strategy is adopted to initialize and optimize the consistency of anchor point noise, and determine the noise distribution of resampling, including: Based on the texture representation, a geometric prior is obtained. According to the geometric prior, the noise of the driving image is mapped to the 3D point cloud, and the noise of the 3D point cloud is rendered to the target view to complete the initialization of anchor point noise. During the training of SDS, the initial anchor noise is resampled and updated based on the consistency of multi-view gradients to achieve consistency optimization and alignment of multi-view representations.

4. The method according to claim 1, characterized in that, Based on the texture representation and resampled noise distribution, the target parameters are optimized using SDS loss and multi-view noise resampling loss to generate a 3D face texture, including: Input the texture representation and the resampled noise distribution into the optimization process and perform the following operations: Using SDS loss as the core loss function, the target parameters are iteratively optimized and updated by distilling the prior knowledge of the two-dimensional diffusion model into three-dimensional space. Simultaneously, by utilizing multi-view noise resampling loss, and based on multi-view gradient consistency, the noise distribution is resampled and updated, gradually aligning the texture representations from different views to ensure the similarity of texture details; The optimized target parameters are rendered to generate a high-fidelity and detailed 3D face texture.

5. The method according to claim 1, characterized in that, After determining the target parameters based on the initial parameters and the fine-tuned second model, the method further includes: A dynamic multi-view consistency supervision mechanism is introduced. Based on the target parameters and multiple predefined camera views, a lightweight auxiliary discriminant network is constructed. The auxiliary discriminant network performs semantic consistency analysis on the rendered images from each view to generate a multi-view consistency confidence map. The consistency confidence map is used as an adaptive weight and integrated into the multi-view noise resampling strategy of the hybrid prior diffusion model. The loss components of different views in the hybrid prior diffusion model are weighted according to the adaptive weight, and the target parameters are updated through the weighted loss components.

6. The method according to claim 1, characterized in that, After generating the 3D face texture, the process also includes: The three-dimensional face texture and the finely adjusted three-dimensional face structure are input together into a differentiable renderer. Under various realistic lighting environments, the differentiable renderer is used to render the image under multiple lighting conditions. Calculate the higher-order semantic difference between the rendered image and the input portrait image in the perceptual feature space, and construct a perceptual optimization loss based on the higher-order semantic difference; Using the aforementioned perceptual optimization loss, the texture coordinate mapping of the 3D face texture is optimized through the backpropagation algorithm. The texture coordinate mapping is then subjected to nonlinear transformation and fine-tuning to output the optimized 3D portrait model.

7. A single-image 3D portrait generation system based on hybrid prior and noise resampling, characterized in that, include: The acquisition module is used to generate multiple planar features based on the acquired first portrait image using a pre-trained first model. The multiple planar features and the decoder parameters and super-resolution parameters corresponding to the first model are combined into initial parameters; The fine-tuning module is used to fine-tune the model parameters of the second model by inputting the second portrait image into the pre-trained second model, so that the second model can generate a three-dimensional face structure corresponding to the second portrait image. The generation module is used to determine the target parameters based on the initial parameters and the fine-tuned second model; The optimization module is used to generate three-dimensional face textures by utilizing the target parameters and employing a hybrid prior diffusion model and a multi-view noise resampling strategy. Specifically, using the target parameters, a 3D face texture is generated through a hybrid prior diffusion model and a multi-view noise resampling strategy, including: Based on the target parameters, a new perspective texture representation is generated by using a hybrid prior diffusion model to perform explicit branching, implicit branching, and geometric prior fusion. Based on the texture representation, a multi-view noise resampling strategy is adopted to initialize and optimize the consistency of anchor point noise, and determine the noise distribution of resampling. Based on the texture representation and the noise distribution of the resampled data, the target parameters are optimized using SDS loss and multi-view noise resampling loss to generate a three-dimensional face texture.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the single-image 3D portrait generation method based on hybrid prior and noise resampling as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the single-image 3D portrait generation method based on hybrid priors and noise resampling as described in any one of claims 1 to 6.

Images