Layer focusing illustration single-view 3D reconstruction method and system

Abstract

The invention provides a layer focusing illustration single-view 3D reconstruction method, which comprises the following steps of: performing region segmentation on an original illustration single view in a data set through SAM to obtain a region text pair fine-tuning CLIP model comprising the original illustration single view and a layer channel of the original illustration single view; extracting an unnatural contour line, semantic information and layer information from each original illustration single view in the data set T; selecting a basic element model to initialize a 3D point cloud according to a role posture in a single view, segmenting the human body point cloud into different parts through point cloud segmentation, carrying out noise growth and color perturbation on corresponding point cloud regions according to position information of an unnatural contour line, respectively mapping the point cloud regions into a submerged space, and obtaining a final potential representation by adopting LDM; and finally, the final potential representation passes through a decoder to obtain a final 3D representation, and 3D assets are reconstructed. According to the invention, the corresponding 3D assets can be generated according to a given illustration single view.

Description

Technical Field

[0001] The present invention relates to an image processing method and system, and in particular to a layer focusing illustration single view Figure 3 D reconstruction method and electronic device, storage medium, and product. Background Art

[0002] In recent years, AIGC's work has become a hot topic in the fields of entertainment, remote sensing and commerce. This trendy and complex technology has attracted more and more attention and investment with the construction of related communities and the development of hardware products. As a major direction in the 3D field, 3D generation and reconstruction has become simpler with the development of large-scale models and 3D representation methods. Even personal computers can deploy models to infer good 3D assets. The following are several popular technologies:

[0003] 1. Use 3D data (i.e. 3D diffusion model) to train a new diffusion model, directly generate 3D assets according to the conditions and maintain strong 3D consistency. The corresponding technical example is Get3D;

[0004] 2. Directly apply 2D diffusion models to 3D generation, which can handle various text prompts and produce highly detailed and complex geometry and appearance. The corresponding technical example is Dreamface;

[0005] 3. Obtain enough multi-views according to the new view generation idea, apply sparse view reconstruction method or score distillation sampling (SDS) optimization, and fuse these multi-view images into 3D shapes, which can produce high-quality 3D shape creation. The corresponding technical examples are Instant3d / Dreamfusion.

[0006] Although these technologies have developed impressive models and the generated 3D assets are indeed excellent, there are still some defects that need to be solved in these technologies during extensive experiments and applications.

[0007] First, for the first technical approach, there are difficulties in expanding to large generation domains because 3D data is usually difficult to obtain and costly, and the scale of current 3D datasets is much smaller than that of 2D datasets, which leads to the inadequacy of the generated 3D assets in processing complex text prompts and generating complex / delicate geometry and appearance.

[0008] Then for the second technical approach, since the 2D diffusion model cannot understand the camera view, it is difficult for the generated 3D assets to form geometric consistency, and the generated 3D assets often have misalignment problems, especially for instances with complex structures.

[0009] Finally, for the third technical method, the indirect generation process of multi-view images causes great inefficiency. In addition, the quality of the generated shape is heavily dependent on the fidelity and continuity of the multi-view images, which often leads to loss of details or reconstruction failure.

[0010] Moreover, the above technologies are mostly considered and optimized for problems related to natural images, and there are still defects in the optimization of 2D illustrations to 3D generation. Among them, the 3D reconstruction problems of 2D illustrations mainly include complex and extreme color matching and black outlines based on unnatural lines, but current research has problems such as color block stacking, large amount of calculation, lack of understanding of lighting layers, and insufficient retrieval of specific accessory information. Summary of the invention

[0011] In view of the above-mentioned deficiencies in the prior art, the present invention aims to provide a layer-focused illustration single-view Figure 3 The present invention relates to a 3D reconstruction method, which is applicable to generating 3D models from a single view of an illustration and can improve the quality of 3D reconstruction.

[0012] The technical solution provided by the present invention is:

[0013] In the first aspect, the present application provides a layer-focused illustration single view Figure 3 D reconstruction method, comprising the following steps:

[0014] Acquire an illustration single-view dataset T, wherein the illustration single-view dataset T includes a plurality of original illustration single views;

[0015] Perform a first segmentation on the original illustration single view in the data set T to obtain each feature area layer of the original illustration single view; perform a second segmentation on each feature area layer to obtain an effect area layer in each feature area layer; generate a region text pair including the original illustration single view and its layer (Layer), the weight of the corresponding layer when inputting the image encoder, the total number of layers corresponding to each feature area layer of the original illustration single view, and the two-dimensional position information of the effect area layer obtained by the second segmentation in the feature area layer obtained by the first segmentation; wherein the layer of the original illustration single view includes the feature area layer and the effect area layer; wherein each region text pair includes a layer obtained by the segmentation of the original illustration single view and the corresponding text description, the region is a layer in the original illustration single view, and the text content is a concise description of these layers;

[0016] Construct and train a layer-CLIP model, wherein the layer-CLIP model includes an image encoder and a text encoder; the image encoder includes an RGB image convolution layer and a layer convolution layer, which are respectively used to input the original illustration single view and its layer; the text encoder is used to input the text description of the corresponding layer; the text encoder is provided with an auxiliary channel for inputting the two-dimensional position information of the effect area layer obtained by the second segmentation in the feature area layer obtained by the first segmentation; the trained layer-CLIP model has the function of outputting the semantic information corresponding to the original illustration single view and the label c1 of the layer information according to the input original illustration single view and its layer and the corresponding text description;

[0017] For a single view of the original illustration, a token c1 containing corresponding semantic information and layer information is obtained through the layer-CLIP model;

[0018] The single view of the original illustration is input into the 3D diffusion model based on the single view. Combined with the marker c1, the corresponding point cloud is diffused using the latent diffusion model (LDM) according to the number of layers corresponding to each feature area to obtain the final 3D representation corresponding to the single view of the original illustration.

[0019] In a possible implementation, the single-view based 3D diffusion model includes an unnatural line extraction module, an initial point cloud generation module, a point cloud segmentation module, a point cloud-layer diffusion module and a latent-triplane module;

[0020] The single view of the original illustration is input into the 3D diffusion model based on the single view, and combined with the mark c1, the corresponding point cloud is diffused a corresponding number of times using the potential diffusion model according to the number of layers corresponding to each feature area, so as to obtain the final 3D representation corresponding to the single view of the original illustration, including:

[0021] The non-natural line extraction module extracts a first illustration single view from the original illustration single view; the first illustration single view is the illustration single view after the non-natural contour lines are removed from the original illustration single view;

[0022] The initial point cloud generation module selects a corresponding basic primitive model according to the character posture in the single view of the first illustration to generate an initial point cloud P;

[0023] The point cloud segmentation module divides the initial point cloud P into different point cloud parts, and performs basic coloring on each point cloud part according to the color of the area with the smallest weight value in the mark c1 corresponding to each point cloud part, to obtain the point cloud after basic coloring; wherein, the point cloud parts divided into by the initial point cloud P correspond one to one with the feature area layers obtained after the first segmentation; illustratively, for example, the first segmentation obtains four parts of head, body, hands and feet, and the subsequent point cloud segmentation is also divided according to these four parts.

[0024] The point cloud-layer diffusion module maps the basic colored point cloud to the latent representation z (Latent z); according to the latent representation z, the label c1 and the total number of layers corresponding to each feature area layer, the latent diffusion model is used to perform a corresponding number of diffusions each time through time t, and the final latent representation z is output m ;

[0025] Latent-three-plane module, based on the final latent representation z m The final 3D (Triplane) representation is obtained, that is, a three-plane representation.

[0026] In a possible implementation, extracting the first illustration single view from the original illustration single view includes:

[0027] S101, extracting the unnatural contour line x of the single view of the original illustration by using the difference Gaussian sketch extraction; during the extraction, using the existing facial landmark detector to create a convex hull around the key structure to prevent the lines of the key structure from being extracted into the unnatural contour line x, and to avoid the lines of the key structure from being filled when the pixel colors near the lines in the drawing are generated by subsequent linear interpolation;

[0028] S102, using a shallow convolutional neural network to extract preliminary features of a single view of the original illustration;

[0029] S103. Based on the preliminary features, multiple linear interpolations (Lerp) are performed on the area where the unnatural contour line x is located to regenerate the pixel color of the unnatural contour line x in the original illustration single view to obtain a first illustration single view.

[0030] In a possible implementation, selecting a corresponding basic primitive model according to the character posture in the single view of the first illustration to generate an initial point cloud P includes:

[0031] S201, selecting a basic model according to the character's posture in the single view of the first illustration as an initial 3D model;

[0032] S202, based on the initial 3D model, construct a triangular mesh M, use multi-layer perceptrons MLPs to predict the signed distance function SDF value and texture color of each vertex on the triangular mesh M, and convert the vertex SDF value and color of M into a point cloud, recorded as pt M (p M , c M ), where p M ∈R 3 Refers to the position of the point cloud, which is equal to the vertex coordinates of M, c M ∈R 3 Refers to the color of the point cloud, which is the same as the color of the vertices of M;

[0033] S203, in pt M Noise point cloud growth and color perturbation are performed on the surrounding point cloud, including:

[0034] First calculate pt M The bounding box on the surface, then according to the mapping area of ​​the non-natural contour line x on the point cloud, the noise point cloud pt is uniformly grown within the BBox covered by the mapping area r (p r , c r ), where p r and c r Represent the position and color of the noise point cloud respectively;

[0035] S204, filter the noise point cloud according to the position p M Construct a K-dimensional tree, based on the nearest point found in the K-dimensional tree and p r The distance between them is kept within the set distance threshold;

[0036] S205, pt M and pt r The position and color of the point cloud are combined to obtain the final initial point cloud P.

[0037] The quality of initialization can be improved by growing noise point clouds, color perturbations, and screening point clouds. During the screening of point clouds, fast search can be achieved by constructing a K-dimensional tree.

[0038] In a possible implementation, the potential diffusion model is used according to the potential representation z, the label c1 and the total number of layers corresponding to each feature region layer, and each time through time t, a corresponding number of diffusions are performed to output the final potential representation z m ,include:

[0039] According to the potential representation z, the label c1 and the total number of layers corresponding to each feature region layer, the potential representation z is noise diffused; in each noise diffusion, a cross attention layer is used to promote the corresponding layer information in the first label c1 and the final potential representation z m The interaction between them, where the order of noise diffusion is carried out in ascending order according to the weight value in the first tag c1, and the diffusion time t is appropriately increased when the weight decimal has only one digit; wherein, for the part of the feature area layer corresponding to the total number of layers m in the potential representation z, noise diffusion is performed m times.

[0040] In one possible implementation, based on the final latent representation z m Get the final 3D (Triplane) representation, including:

[0041] In order to obtain the final potential representation z m After that, it is reshaped into a three-plane representation z reshape , and vertically connect the three planes in the height dimension to obtain z concat Upsample to a high-resolution three-plane feature map; wherein the upsampling process is: using a convolutional decoder to gradually upsample the explicit final potential representation and obtain the final 3D representation.

[0042] In a second aspect, the present application provides an electronic device, including: a memory and a processor;

[0043] The memory is used to store computer programs;

[0044] The processor is used to call the computer program to execute the method as described above.

[0045] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed on an electronic device, the electronic device implements the method as described above.

[0046] In a fourth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed on an electronic device, the electronic device implements the method as described above.

[0047] The specific implementation methods of the second to fourth aspects of the present application can refer to the implementation methods of the first aspect, which will not be repeated here.

[0048] The technical solution provided by the present invention has the following beneficial technical effects:

[0049] 1. The layer-focused illustration single view used in the present invention Figure 33D reconstruction method is used to fully simulate the drawing process of 2D illustrations, that is, to understand the detailed information on the illustrations by layer, which greatly solves the problems of extreme color block stacking, highlight shadow layer interference, and loss of complex accessory information. At the same time, it uses mature 2D diffusion models to optimize 3D generation, reducing the demand for high-quality 3D assets, and accurately positions 2D details when mapping to 3D, reducing the problem of feature offset;

[0050] 2. The present invention extracts the unnatural contour line x, which not only reduces the artifact interference caused by the illustration lines, but also makes it convenient to use the area where the unnatural contour line x is located as the main target of the initial 3D point cloud noise point cloud growth and color disturbance, further improving the quality of the initialization point cloud;

[0051] 3. By segmenting the point cloud, the complex human body point cloud is divided into multiple components, and each component is operated separately in the low-dimensional latent space instead of directly in the high-dimensional 3D space. This greatly improves the scalability and computational efficiency of the model, while also maintaining the quality of the generated 3D content and avoiding the discontinuity problem in the synthesis of new perspectives.

[0052] 4. The construction of a weighted dataset both expands the generation range and reduces the risk of hallucinations when generating the required 3D assets. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the method of the embodiment of the present application;

[0054] Figure 2 This is a schematic diagram of the unnatural line extraction module in the embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of the initial point cloud generation module in the embodiment of the present application;

[0056] Figure 4 This is a schematic diagram of a point cloud segmentation module in an embodiment of the present application;

[0057] Figure 5 This is a schematic diagram of the point cloud-layer diffusion module in the embodiment of the present application;

[0058] Figure 6 This is a schematic diagram of a potential three-plane module in an embodiment of the present application;

[0059] Figure 7 This is an illustration of the single view segmentation, point cloud segmentation, and point cloud-layer diffusion correspondence diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0061] In order to better understand the embodiments of the present application, the following is a description of the relevant models or technologies involved in the embodiments of the present application:

[0062] 1. SAM model as the base model: SAM is the Segment Anything Model (SAM) proposed by Meta, which is a general model for image segmentation tasks. SAM consists of three parts: a powerful image encoder calculates image embedding, a prompt encoder embeds prompts, and then combines the two information sources in a lightweight mask decoder to predict the segmentation mask. Prompts can be used plug-and-play to solve various tasks involving object and image distribution, including edge detection, object proposal generation, instance segmentation, and text to mask prediction, etc., and can output segmentation masks in real time when prompted for interactive use.

[0063] 2. CLIP stands for Contrastive Language-Image Pre-training, which is a pre-training model based on contrastive text-image pairs. CLIP is a multimodal model based on contrastive learning. The training data of CLIP is text-image pairs: an image and its corresponding text description. Through contrastive learning, the model can learn the matching relationship between text-image pairs.

[0064] 2. kd-tree: A tree data structure that stores instance points in k-dimensional space for fast retrieval. It is mainly used for searching key data in multidimensional space (such as range search and nearest neighbor search).

[0065] 3. PointNet: It is a deep learning network architecture specifically designed for processing point cloud data. It can learn feature representations directly from point clouds without complex voxelization or shape assumptions, and segment point clouds into different objects or parts of objects.

[0066] The present application discloses a layer-focused illustration single-view Figure 3The invention discloses a 3D reconstruction method, comprising collecting a single-view dataset T of illustrations suitable for reconstruction; processing the original single-view of illustrations in T through SAM to generate a regional text pair training fine-tuning layer-CLIP model containing the original single-view of illustrations and its layer channels; S3: extracting and removing the unnatural contour lines from each original single-view of illustrations in T through a non-natural line extraction module; selecting a basic primitive model (the basic primitive model here refers to a basic, general, free, and easily available human body model) according to the body shape of each original single-view of illustrations in T, generating an initial point cloud P as an initial 3D model, and performing noise growth and color perturbation at the corresponding position of the initial point cloud according to the non-natural contour lines, and obtaining a basic colored point cloud N through point cloud segmentation and coloring of the initial point cloud P, and inputting the point cloud-layer diffusion module to map it to a latent space and diffuse it; obtaining a label c1 containing corresponding semantic information and layer information through a layer-CLIP model for each original single-view of illustrations in T, and inputting the label c1 into a point cloud-layer diffusion module in combination with the total number of corresponding layers, and outputting the final latent representation z after multiple diffusions through time t each time. m ; Based on the final potential representation z m The final 3D representation, ie, a three-plane representation, is obtained.

[0067] The following is combined with Figures 1 to 7 This application is described in detail.

[0068] like Figure 1 As shown, the embodiment of the present application discloses a layer focusing illustration single view Figure 3 D reconstruction method, the method comprising the following steps:

[0069] S1. Obtain an illustration single-view dataset T, wherein the illustration single-view dataset T includes a plurality of original illustration single views;

[0070] The image requirements of the dataset T include high resolution, front view, neutral expression, and uncropped. In some embodiments, the dataset T can use the Hololive character dataset, the Honkai Impact 3rd character dataset, or the Honkai: StarRail character dataset. The Hololive character dataset comes from the virtual idol company Hololive or VirtualYoutuberFandomWiki; the Honkai Impact 3rd character dataset comes from the official website database of miHoYo.

[0071] Preferably, the data set T is a Vroid3D data set, which comes from 11.2k 3D anime character data from VroidHub created by University of Maryland-College Park. High-resolution, neutral expression, uncropped images from the Vroid3D data set can be used as the main image data set for 3D generation. For the selected data set, a frontal face detection method can be used to filter out images from a frontal perspective.

[0072] S2, perform the first segmentation on the original illustration single view in the data set T to obtain each feature area layer of the original illustration single view; perform the second segmentation on each feature area layer to obtain the effect area layer in each feature area layer; generate a region text pair including the original illustration single view and its layer (Layer), the weight of the corresponding layer when inputting the image encoder, and the total number of layers corresponding to each feature area layer of the original illustration single view;

[0073] The layers of the original illustration single view include a feature area layer and an effect area layer;

[0074] The feature area layer corresponds to various features of the character, such as head, hands and feet, body, facial features, hair, accessories, etc.;

[0075] The effect area layer corresponds to the effect area on each feature of the character, such as highlights and shadows;

[0076] After the segmentation is completed, the weight set when the corresponding layer is input into the image encoder is set according to the segmentation situation, and the weight value is set between [0, 1]. The feature area layer obtained by the first segmentation takes the weight of the foreground area as 1 according to the masking relationship, and the weight values ​​of subsequent areas decrease with a gradient of 0.1; and the second segmentation decreases with a gradient of 0.01 based on the weight value of the feature area layer obtained by the first segmentation.

[0077] Record the total number of layers corresponding to each feature area layer;

[0078] For example, if a feature region layer obtained by the first segmentation is segmented for the second time and 4 effect region layers are obtained, the total number of layers corresponding to the feature region layer is 5;

[0079] Proportionally obtain the two-dimensional position information of the effect area layer obtained by the second segmentation in the feature area layer obtained by the first segmentation;

[0080] In the above steps, the segmentation of the original illustration single view of the dataset T is different from that of natural images. By performing secondary segmentation on the original illustration single view in the dataset, recording the illustration layer information such as highlights, shadows, etc., the interactive information is recorded in the form of multiple layers, and then passed to downstream tasks, which greatly reduces the problems of color block mixing, artifact interference, and accessory information loss. It is particularly suitable for generating 3D models from datasets with single views of illustrations, such as generating head models for game illustration characters.

[0081] In some embodiments, the original illustration single view in the dataset T may be segmented using a segmentation everything model SAM.

[0082] Each region text pair includes a layer obtained by segmenting the original illustration from a single view and the corresponding text description. The region is a layer in a single view of the original illustration, and the text content is a concise description of these layers (such as English);

[0083] S3, construct and train a layer-CLIP model, wherein the layer-CLIP model includes an image encoder and a text encoder; the image encoder includes an RGB image convolution layer and a layer convolution layer, which are respectively used to input the original illustration single view and its layer; the text encoder is used to input the text description of the corresponding layer; the trained layer-CLIP model has the function of outputting the semantic information corresponding to the original illustration single view and the tokens (tokens) c1 of the layer information according to the input original illustration single view and its layer and the corresponding text description;

[0084] Compared with the existing CLIP model, the above-mentioned layer-CLIP model adds an adaptive layer convolution layer in the VisionTransformer (ViT) structure of the image encoder, which is equivalent to adding a layer channel in the image encoder. This convolution layer is used to process the input data of the layer channel in parallel with the RGB convolution layer, allowing the CLIP image encoder to accept additional layer channels as input. The accepted weight value range is set between [0, 1], where 1 represents the foreground and 0 represents the background. When the input precision is 0.1, it is the picture obtained by the first segmentation, and when the precision is 0.01, it is the picture obtained by the second segmentation. For the picture of the first segmentation, the convolution kernel uses a 3*3 convolution kernel, and for the picture of the second segmentation, a 1*1 convolution kernel is used.

[0085] An auxiliary channel is added to the CLIP model text encoder setting to input the two-dimensional position information of the effect area layer obtained by the second segmentation in the feature area layer obtained by the first segmentation. This channel is open to accept input only when the layer image after the second segmentation is input to the image encoder. In other cases, the channel input can be set to (-2, -2) and regarded as invalid input.

[0086] Exemplarily, an auxiliary channel is added to the CLIP model text encoder setting to input the two-dimensional position information of the layer after the second segmentation in the first segmentation layer image, with a range of x to (-1, 1), y to (-1, 1). For example, for a hand, with the center of the back of the hand as the origin, the position information is (0, 0), while for the shadow on the middle finger of the hand, the position information may be (0.1, -0.9), and the position information is selected according to the specific situation.

[0087] When used in an example, the data of the original channel, i.e., the RGB channel, and the layer channel are simultaneously accepted to train and fine-tune the layer-CLIP model. That is, the original illustration single view and the layers obtained by segmentation are simultaneously input into the image encoder.

[0088] We can take image data from GRIT-20m and train the fine-tuning layer CLIP on RGBL region-text pairs.

[0089] S4, for the original illustration single view, obtain the tokens c1 containing the corresponding semantic information and layer information through the layer-CLIP model;

[0090] S5. Input the single view of the original illustration into the 3D diffusion model based on the single view. Combined with the marker c1, the corresponding point cloud is diffused using the latent diffusion model (LDM) according to the number of layers corresponding to each feature area to obtain the final 3D representation corresponding to the single view of the original illustration.

[0091] In some embodiments, the single-view based 3D diffusion model includes an unnatural line extraction module, an initial point cloud generation module, a point cloud segmentation module, a point cloud-layer diffusion module and a latent-triplane module;

[0092] The single view of the original illustration is input into the 3D diffusion model based on the single view, and combined with the mark c1, the corresponding point cloud is diffused a corresponding number of times using the potential diffusion model according to the number of layers corresponding to each feature area, so as to obtain the final 3D representation corresponding to the single view of the original illustration, including:

[0093] The non-natural line extraction module extracts a first illustration single view from the original illustration single view; the first illustration single view is the illustration single view after the non-natural contour lines are removed from the original illustration single view;

[0094] The initial point cloud generation module selects a corresponding basic primitive model according to the character posture in the single view of the first illustration to generate an initial point cloud P;

[0095] like Figure 4As shown in FIG. 1 , the point cloud segmentation module segments the initial point cloud P into different point cloud parts, and performs basic coloring on each point cloud part according to the color of the area with the smallest weight value in the marker c1 corresponding to each point cloud part, to obtain the point cloud after basic coloring; wherein, each point cloud part segmented from the initial point cloud P corresponds one to one to each feature area layer obtained after the first segmentation; for example, the first segmentation obtains four parts: head, body, hands, and feet, and the subsequent point cloud segmentation is also segmented according to these four parts, such as Figure 7 shown.

[0096] In some embodiments, the initial point cloud P can be segmented into different point cloud parts N using the machine learning model PointNet.

[0097] The point cloud-layer diffusion module maps the basic colored point cloud to the latent representation z (Latent z); according to the latent representation z, the label c1 and the total number of layers corresponding to each feature area layer, the latent diffusion model is used to perform a corresponding number of diffusions each time through time t, and the final latent representation z is output m ;

[0098] Latent-three-plane module, based on the final latent representation z m The final 3D (Triplane) representation is obtained, that is, a three-plane representation.

[0099] In some embodiments, Figure 2 As shown, the unnatural line extraction module extracts the first illustration single view from the original illustration single view, including:

[0100] S101, extracting the unnatural contour line x of the single view of the original illustration by difference Gaussian sketch extraction; during the extraction, using an existing facial landmark detector to create a convex hull around key structures (such as facial features, including eyes, nose, mouth, eyebrows, and ears) to prevent the lines of the key structures from being extracted into the unnatural contour line x, and to avoid the lines of the key structures from being filled when the pixel colors near the lines in the drawing are generated by subsequent linear interpolation;

[0101] S102, using a shallow convolutional neural network to extract preliminary features of a single view of the original illustration;

[0102] S103. Based on the preliminary features, multiple linear interpolations (Lerp) are performed on the area where the unnatural contour line x is located to regenerate the pixel color of the unnatural contour line x in the original illustration single view to obtain a first illustration single view.

[0103] In some embodiments, Figure 3 As shown, the initial point cloud generation module selects a corresponding basic primitive model according to the character posture in the single view of the first illustration to generate an initial point cloud P, including:

[0104] S201, selecting a basic model according to the character's posture in the single view of the first illustration as an initial 3D model;

[0105] S202: Based on the initial 3D model (asset), a triangular mesh M is constructed, and the signed distance function SDF value and texture color of each vertex on the triangular mesh M are predicted using multi-layer perceptrons (MLPs), and the vertex SDF value and color of M are converted into a point cloud, denoted as pt M (p M , c M ), where p M ∈R 3 Refers to the position of the point cloud, which is equal to the vertex coordinates of M, c M ∈R 3 Refers to the color of the point cloud, which is the same as the color of the vertices of M;

[0106] For example, the grid size is 128 3 , which means there are 128 points in each coordinate axis direction, with a total of 128×128×128 vertices. The signed distance function SDF(x, y, z) represents the distance from each point to the nearest surface of the initial 3D model, with a sign, where a negative value indicates that the point is on the inside of the surface and a positive value indicates that the point is on the outside of the surface. In MLPs, for each vertex (x, y, z), an SDF value is calculated to indicate its position relative to the surface of the initial 3D model. Each vertex (x, y, z) has not only position information, but also texture color information associated with it, which is usually represented as (r, g, b), representing the red, green, and blue color components.

[0107] S203, in pt M Noise point cloud growth and color perturbation are performed on the surrounding point cloud, including:

[0108] First calculate pt M The bounding box (BBox) on the surface, and then according to the mapping area of ​​the non-natural contour line x on the point cloud, the noise point cloud pt is uniformly grown within the BBox covered by the mapping area r (p r , c r ), where p r and c r Represent the position and color of the noise point cloud respectively;

[0109] Among them, BBox is to find the minimum circumscribed rectangle of the point cloud in three-dimensional space, traverse all points in the point cloud, and record the minimum and maximum values ​​of each point on the three coordinate axes of X, Y, and Z. The bounding box can be represented by any two diagonal points among its eight corner points, usually the minimum point and the maximum point;

[0110] Since the selected illustration images are all front views, the 2D image coordinates are mapped to the 3D space, ignoring the change of the Z axis. The 3D point cloud coordinates corresponding to each pixel point (u, v) in the unnatural contour line x of the 2D image are Where D is the distance of the 3D point in the depth direction (Z axis), and f is the focal length of the camera;

[0111] S204, filter the noise point cloud according to the position p M Construct a K-dimensional tree (KD-Tree), according to the nearest point found in the K-dimensional tree and p r The distance between them is kept within the set distance threshold; that is, each generated point is traversed, the nearest point is found and checked whether the distance condition is met.

[0112] By constructing a K-dimensional tree, fast searching can be achieved.

[0113] For example, the distance threshold may be a normalized distance of 0.01.

[0114] Among them, for the color of the noise point cloud, let c r With c M Similar, with some perturbations added:

[0115] c r =c M +a

[0116] The value of a is randomly sampled between 0 and 0.2.

[0117] S205, pt M and pt r The position and color of the point cloud are combined to obtain the final initial point cloud P.

[0118] The quality of initialization can be improved by growing noise point clouds, color perturbations, and screening point clouds. During the screening of point clouds, fast search can be achieved by constructing a K-dimensional tree.

[0119] In some embodiments, Figure 5 As shown, the point cloud-layer diffusion module maps the base colored point cloud to a latent representation z (Latent z), including:

[0120] Use Fourier features to represent the position structure of the point cloud after basic coloring, and introduce a series of learnable tokens The cross-attention layer is used to query the point cloud features so that the 3D information in the point cloud can be injected into the potential markers. Then multiple self-attention layers are used to enhance the representation of these markers to obtain the potential representation of the point cloud after base coloring. where r*r represents the resolution of the potential representation, d e represents the e channel dimension, d z Represents the z channel dimension.

[0121] Fourier features are used to represent the location information of point clouds. For each point position P i , it can be mapped to a high-dimensional space through Fourier feature mapping:

[0122] γ(p i )=[cos(2πB P i ), sin(2πB P i )]

[0123] where B∈R r×3 is the Fourier basis matrix, usually randomly sampled.

[0124] In some embodiments, the potential diffusion model is used according to the potential representation z, the label c1 and the total number of layers corresponding to each feature region layer, and each time through time t, a corresponding number of diffusions are performed to output the final potential representation z m ,include:

[0125] In each noise diffusion, a cross-attention layer is used to promote the corresponding layer information in the first mark c1 and the final potential representation z m The interaction between them, where the order of noise diffusion is carried out in ascending order according to the weight value in the first tag c1, and the diffusion time t can be appropriately increased when the weight decimal has only one digit; wherein, for the part of the feature area layer corresponding to the total number of layers m in the potential representation z, noise diffusion is performed m times.

[0126] In some embodiments, during each potential diffusion process, the model weight ε of the potential diffusion model is calculated using LoRA i Perform fine-tuning training; and add the semantic information in c1 to the data ε as interpolation i In , all LoRA matrices are connected to obtain the weight data representing a layer part; after training based on N different instances, the final model weight data set D = {ε1,ε2,…,ε N};

[0127] The model weights of the latent diffusion model are the parameters of the latent diffusion model, which are optimized during the training process to minimize the loss function. The model weights are the encoding of the knowledge learned by the model from the input data, which determines how the model maps the input to the output;

[0128] Because some layer attributes often appear together during layer training, these attributes may be entangled together during subsequent retrieval, such as reddish hair and warm highlights. This entanglement may limit the accuracy of the model when editing or generating specific attributes. Therefore, according to the information marked c1, when attribute entanglement is encountered, the result of fine-tuning the training with the smaller weight of the layer model marked c1 is used to cover the result of the other side.

[0129] Perform principal component analysis on the model weight dataset, retain the principal components of the model weight dataset for dimensionality reduction, compress the data points into fewer parameters, and retain only the most important information. After the dataset is reduced in dimensionality, use the model weight dataset as data to train the semantic retriever, and build a hypergraph based on the semantic information in c1 as labels.

[0130] The model weight dataset D can be sampled according to a piece of semantic information S to obtain a new model that generates a new layer part. A hypergraph can be defined with a layer part, and the semantic information of the tag c1 embedded in the model weight can be retrieved, and then the relevant model weights in the model can be edited. For example, if you want to obtain a female right hand with a red bracelet and a black tattoo in the sun, you can build a hypergraph that conforms to the semantics, connect the model weights of warm highlights, red bracelets, female right hands, and black tattoos in the model weight dataset with a hyperedge, and obtain the model weights that conform to the semantics according to the combination relationship of the hypergraph, and then continue to diffuse according to the model weights.

[0131] The above steps inject the semantic information and layer information in the image into the latent space, thereby accurately mapping the image features on the point cloud. This can ensure that the high-frequency details of the 3D assets generated by the diffusion model are aligned with the conditional image (the original illustration single view). At the same time, the model weights of each diffusion are used as new data to construct a model weight dataset for generating corresponding 3D asset features, which expands the range of generated objects and reduces the risk of hallucinations.

[0132] In some embodiments, Figure 6 As shown, the latent-three-plane module is based on the final latent representation z m Get the final 3D (Triplane) representation, including:

[0133] In order to obtain the final potential representation z m After that, it is reshaped into a three-plane representation z reshape , and vertically connect the three planes in the height dimension to obtain To prevent incorrect mixing of planes in the channel dimension, the latent-triplane module then concat Upsample to a high-resolution three-plane feature map with an upsampling factor of f.

[0134] The upsampling process is as follows: a convolutional decoder (convolutional network) is used to gradually upsample the explicit final potential representation to obtain the final 3D representation.

[0135] The reshaped three planes are expressed as:

[0136] z reshape = reshape(z,(3,r,r,d z ))

[0137] Connect the three planes vertically in the height dimension, and we get

[0138]

[0139] A convolutional network is used to gradually upsample the explicit final potential representation. In contrast, the Transformer decoder can effectively upsample through transposed convolution while providing a different feature extraction method from the encoder to achieve feature complementarity;

[0140] The method is used for generating head models of game illustration characters, generating head models of virtual anchors or illustrated 3D figures, and is preferably used for generating head models of game illustration characters.

[0141] Below is a 2D illustration of a character wearing an animal ear hair accessory. Figure 3 D reconstruction is used as an example.

[0142] 1. Input 2D illustrations

[0143] 2. Layer segmentation and weight setting

[0144] Feature area layer segmentation: The SAM model is used to perform the first segmentation of the two-dimensional illustration to obtain the basic feature layer, that is, the head, face, ears and hair accessories in the two-dimensional illustration are segmented into separate layers. The light and shadow effect layer is further segmented in the ear and hair accessory feature area.

[0145] Layer description and text generation: Based on the segmentation, each layer and its layer description are obtained, and regional text pairs (such as "head area", "ear hair accessories", "highlight effect") are generated. The layer-CLIP model is used to obtain tags containing corresponding semantic information and layer information for layer control during generation.

[0146] Weight setting: After the segmentation is completed, the weight set when the corresponding layer is input into the image encoder is set according to the segmentation situation.

[0147] 3. Initial point cloud generation

[0148] Select a basic model: According to the facial and body features of the character, select a basic model with a head and ear outline as the initial 3D model.

[0149] Triangular mesh and point cloud generation: A triangular mesh is constructed based on the selected primitive, and the signed distance function (SDF) and texture color of the mesh vertices are predicted through a multi-layer perceptron (MLP). The vertex SDF values ​​and colors of the mesh are converted into an initial point cloud to ensure that the contour and color information of features such as ears and hair accessories are completely preserved.

[0150] 4. Point cloud noise growth and color disturbance

[0151] Unnatural line extraction: Extract the unnatural contour lines of the illustration (such as the black edge lines of the ear accessories) through the difference Gaussian filtering method, and record the position of the contour lines in the illustration layer.

[0152] Noise point cloud growth: Generate a bounding box (BBox) in the mapping area of ​​the ear and hair accessories, and evenly grow the noise point cloud in the bounding box so that the density of the noise point cloud matches the ear and hair accessories area.

[0153] KD-Tree screening and color perturbation: Use the KD-Tree structure to screen the noise point cloud, retain the noise points within the distance threshold through the nearest neighbor distance, and add slight color perturbation to the ear area to make the color coordinate with the surrounding hair accessories and generate a natural hair accessory color transition effect.

[0154] 5. Point cloud-layer diffusion processing

[0155] Application of latent diffusion model: Map the point cloud to the latent representation space z, and diffuse the area a corresponding number of times according to the layer weight (for example, the weight of the ear and hair accessories is 0.9). The cross attention layer is used in the diffusion process to make the layer information of the ear and hair accessories interact with the point cloud features.

[0156] Diffusion order: Diffusion is performed in order from small to large layer weight values ​​to ensure that the structure of the background layer and foreground layers such as hair accessories are improved layer by layer. If the weight accuracy of the ear hair accessories is 0.1, increase the diffusion time t appropriately to improve the effect.

[0157] 6. Latent-Triplane Module Generates Triplane Representation

[0158] Three-plane construction: The final latent representation after diffusion is reshaped into a three-plane representation, and three planes are stacked vertically along the height dimension to ensure that the hair accessory layers are not confused when generated.

[0159] Upsampling to generate high-resolution 3D feature maps: The convolution decoder gradually upsamples the three-plane feature maps to generate high-resolution three-dimensional representations, making details such as ears and hair accessories clearly visible in the 3D model.

[0160] 7. Final 3D model output

[0161] Model output: The final generated 3D model contains the complete structure and texture of the head and ear hair accessories. The highlights, shadows and edge line details of the ear hair accessories are naturally presented in the 3D model, achieving a visual effect consistent with the original illustration.

[0162] This process makes full use of layer information, noise growth and multi-layer diffusion methods to ensure that the ears, hair accessories and other details are accurately presented in the generation process and conform to the characteristics of two-dimensional illustrations.

[0163] The embodiment of the present application also provides an electronic device, including: a memory and a processor;

[0164] The memory is used to store computer programs;

[0165] The processor is used to call the computer program to execute the method as described above.

[0166] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed on an electronic device, the electronic device implements the method as described above.

[0167] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed on an electronic device, the electronic device implements the method as described above.

[0168] The embodiments of the present application also provide a system, an electronic device, a computer-readable storage medium, and a computer program product. The specific implementation methods can refer to the specific embodiments of the above methods and will not be repeated here.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A single-view 3D reconstruction method for layer-focused illustrations, characterized in that: The following steps are involved: Acquire an illustration single-view dataset T, wherein the illustration single-view dataset T includes a plurality of original illustration single views; Perform a first segmentation on the original illustration single view in the data set T to obtain each feature area layer of the original illustration single view; perform a second segmentation on each feature area layer to obtain an effect area layer in each feature area layer; generate a region text pair including the original illustration single view and its layers, the weight of the corresponding layer when inputting the image encoder, the total number of layers corresponding to each feature area layer of the original illustration single view, and the two-dimensional position information of the effect area layer obtained by the second segmentation in the feature area layer obtained by the first segmentation; wherein the layers of the original illustration single view include feature area layers and effect area layers; wherein each region text pair includes a layer obtained by segmentation of the original illustration single view and a corresponding text description; Construct and train a layer-CLIP model, wherein the layer-CLIP model includes an image encoder and a text encoder; the image encoder includes an RGB image convolution layer and a layer convolution layer, which are respectively used to input the original illustration single view and its layer; the text encoder is used to input the text description of the corresponding layer; the text encoder is provided with an auxiliary channel for inputting the two-dimensional position information of the effect area layer obtained by the second segmentation in the feature area layer obtained by the first segmentation; the trained layer-CLIP model has the function of outputting the semantic information corresponding to the original illustration single view and the label c1 of the layer information according to the input original illustration single view and its layer and the corresponding text description; For a single view of the original illustration, a label c1 containing corresponding semantic information and layer information is obtained through the layer-CLIP model; The single view of the original illustration is input into the 3D diffusion model based on the single view. Combined with the marker c1, the corresponding point cloud is diffused a corresponding number of times using the potential diffusion model according to the number of layers corresponding to each feature area to obtain the final 3D representation corresponding to the single view of the original illustration.

2. The method according to claim 1, characterized in that The single-view based 3D diffusion model includes an unnatural line extraction module, an initial point cloud generation module, a point cloud segmentation module, a point cloud-layer diffusion module and a potential-three-plane module; The single view of the original illustration is input into the 3D diffusion model based on the single view, and combined with the mark c1, the corresponding point cloud is diffused a corresponding number of times using the potential diffusion model according to the number of layers corresponding to each feature area, so as to obtain the final 3D representation corresponding to the single view of the original illustration, including: The non-natural line extraction module extracts a first illustration single view from the original illustration single view; the first illustration single view is the illustration single view after the non-natural contour lines are removed from the original illustration single view; The initial point cloud generation module selects a corresponding basic primitive model according to the character posture in the single view of the first illustration to generate an initial point cloud P; The point cloud segmentation module segments the initial point cloud P into different point cloud parts, and performs basic coloring on each point cloud part according to the color of the area with the smallest weight value in the mark c1 corresponding to each point cloud part, to obtain a point cloud after basic coloring; wherein each point cloud part into which the initial point cloud P is segmented corresponds one-to-one to each feature area layer obtained after the first segmentation; The point cloud-layer diffusion module maps the point cloud after basic coloring to a potential representation z; according to the potential representation z, the label c1 and the total number of layers corresponding to each feature area layer, the potential diffusion model is used to perform a corresponding number of diffusions each time through time t, and the final potential representation z is output. m ; The latent-three-plane module is based on the final latent representation z m The final 3D representation, ie, a three-plane representation, is obtained.

3. The method according to claim 2, characterized in that The step of extracting the first illustration single view from the original illustration single view comprises: S101, extracting the unnatural contour line x of the single view of the original illustration by difference Gaussian sketch extraction; during the extraction, a convex hull around the key structure is created by using an existing facial landmark detector; S102, using a shallow convolutional neural network to extract preliminary features of a single view of the original illustration; S103. Based on the preliminary features, multiple linear interpolations are performed on the area where the unnatural contour line x is located to regenerate the pixel color of the unnatural contour line x in the original illustration single view to obtain a first illustration single view.

4. The method according to claim 3, characterized in that: The step of selecting a corresponding basic model according to the character posture in the single view of the first illustration to generate an initial point cloud P includes: S201, selecting a basic model according to the character's posture in the single view of the first illustration as an initial 3D model; S202, based on the initial 3D model, construct a triangular mesh M, use multi-layer perceptrons MLPs to predict the signed distance function SDF value and texture color of each vertex on the triangular mesh M, and convert the vertex SDF value and color of M into a point cloud, recorded as pt M (p M , c M ), where p M ∈R 3 Refers to the position of the point cloud, which is equal to the vertex coordinates of M, c M ∈R 3 Refers to the color of the point cloud, which is the same as the color of the vertices of M; S203, in pt M Noise point cloud growth and color perturbation are performed on the surrounding point cloud, including: First calculate pt M The bounding box on the surface, then according to the mapping area of ​​the non-natural contour line x on the point cloud, the noise point cloud pt is uniformly grown within the BBox covered by the mapping area r (p r , c r ), where p r and c r Represent the position and color of the noise point cloud respectively; S204, filter the noise point cloud according to the position p M Construct a K-dimensional tree, based on the nearest point found in the K-dimensional tree and p r The distance between them is kept within the set distance threshold; S205, pt M and pt r The position and color of the point cloud are combined to obtain the final initial point cloud P.

5. The method according to claim 4, characterized in that According to the potential representation z, the label c1 and the total number of layers corresponding to each feature region layer, the potential diffusion model is used to perform a corresponding number of diffusions each time through time t to output the final potential representation z m ,include: According to the potential representation z, the label c1 and the total number of layers corresponding to each feature region layer, the potential representation z is noise diffused; in each noise diffusion, a cross attention layer is used to promote the corresponding layer information in the first label c1 and the final potential representation z m The interaction between them, where the order of noise diffusion is carried out in ascending order according to the weight value in the first tag c1, and the diffusion time t is appropriately increased when the weight decimal has only one digit; wherein, for the part of the feature area layer corresponding to the total number of layers m in the potential representation z, noise diffusion is performed m times.

6. The method according to claim 5, characterized in that Based on the final potential representation z m Obtain the final 3D representation, including: In order to obtain the final potential representation z m After that, it is reshaped into a three-plane representation z reshape , and vertically connect the three planes in the height dimension to obtain z concat Upsample to a high-resolution three-plane feature map; wherein the upsampling process is: using a convolutional decoder to gradually upsample the explicit final potential representation and obtain the final 3D representation.

7. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is configured to call the computer program to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device implements the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed on an electronic device, the electronic device implements the method according to any one of claims 1 to 6.

Images