Three-dimensional model texture data generation method and related equipment

Abstract

The embodiment of the invention provides a three-dimensional model texture data generation method and related equipment, and the method comprises the steps: obtaining a text prompt, and obtaining a plurality of geometric condition images corresponding to a plurality of visual angles of a three-dimensional grid model; selecting a first view angle from the plurality of view angles, and inputting the initial noise image, the text prompt and a plurality of geometric condition images corresponding to the first view angle into a diffusion model for data processing to obtain a repaired image corresponding to the first view angle; an unprocessed second view angle is selected from the multiple view angles, projection and mask progressive generation are carried out at least based on the input image data and the geometric condition image corresponding to the second view angle, a repaired image corresponding to the second view angle is obtained, and the input image data are obtained according to the repaired images corresponding to all processed view angles; and back projection processing is carried out based on the restored images corresponding to all the visual angles to obtain the texture data of the three-dimensional grid model, so that the detail richness, the reality sense and the overall naturalness of the texture data of the three-dimensional model are improved.

Description

Technical Field

[0001] This application relates to the field of visual data processing technology, and in particular to methods and related equipment for generating texture data of three-dimensional models. Background Technology

[0002] With the rapid development of digital twins, animation games, and the metaverse industry, the market demand for high-quality and diverse 3D digital content is growing. In the field of AI-assisted content creation, generative AI has begun to be used to automatically generate 3D model textures. This involves using pre-trained text-to-image diffusion models to generate texture images that are aligned with the 3D model structure by rendering the model's geometric information (such as depth maps or normal maps) and combining it with control networks such as ControlNet.

[0003] However, when this method of generating 3D model textures using generative artificial intelligence relies on a pre-trained image diffusion model, the resulting image quality is limited due to a lack of comprehensive capture of the model's fine structure or difficulty in fully responding to complex text prompts. Furthermore, there are obvious inconsistencies between different views. This makes it easy for the final synthesized texture to exhibit blurring, artifacts, or obvious seams at the boundaries between different viewpoints during back projection (i.e., composite mapping). Consequently, the texture naturalness of the existing generated 3D model texture data is relatively low. Summary of the Invention

[0004] This application provides a method and related equipment for generating three-dimensional model texture data, which can improve the texture naturalness of the generated three-dimensional model texture data.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for generating three-dimensional model texture data, the method comprising: Get text prompts, and get multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model; A first perspective is selected from multiple perspectives. The initial noise image, the text prompt, and multiple geometric condition images corresponding to the first perspective are input into a diffusion model for data processing to obtain the repaired image corresponding to the first perspective. An unprocessed second viewpoint is selected from multiple viewpoints, and projection and mask progressive generation are performed based at least on the input image data and the geometric condition image corresponding to the second viewpoint to obtain the repaired image corresponding to the second viewpoint. The input image data is obtained based on the repaired images corresponding to all processed viewpoints. A multi-view image set is obtained based on the repaired images corresponding to all viewpoints. Back projection processing is performed on the multi-view image set to obtain the texture data of the three-dimensional mesh model.

[0006] In some embodiments, the geometric condition image includes a normal map, a position map, and a depth edge map. Acquiring multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model includes: Obtain multiple preset viewpoints for the three-dimensional mesh model, the viewpoints including a preset basic viewpoint and a preset oblique viewpoint; For each of the aforementioned viewpoints, the three-dimensional mesh model is rendered, and the normal vector direction of each pixel is calculated to obtain the normal map; The three-dimensional mesh model is rendered, and the three-dimensional spatial coordinate information of each pixel is calculated to obtain the position map; Edge detection and rendering are performed on the three-dimensional mesh model to obtain the depth edge map.

[0007] In some embodiments, the step of inputting the initial noisy image, the text prompt, and multiple geometric condition images corresponding to the first viewpoint into a diffusion model for data processing to obtain the repaired image corresponding to the first viewpoint includes: The normal map, the position map, and the depth edge map corresponding to the first viewpoint are stitched together to obtain geometric structure data; The geometric structure data is input into the control network for encoding to obtain geometric structure features, and the geometric structure features are then input into the diffusion model based on the control branch. The text prompts are encoded to obtain text prompt features, and the text prompt features are input into the diffusion model using cross-attention technology; In the diffusion model, the initial noisy image is denoised based on the geometric structure features and the text prompt features to obtain the repaired image corresponding to the first viewpoint.

[0008] In some embodiments, the step of projecting and masking progressively generating the repaired image corresponding to the second viewpoint based at least on the input image data and the geometric condition image corresponding to the second viewpoint includes: Based on the camera parameters corresponding to the viewpoint of the input image data, the input image data is projected onto the two-dimensional plane corresponding to the second viewpoint to obtain an incomplete projected image; A binary insulated mask is generated based on the region missing from the viewpoint corresponding to the input image data in the incomplete projection image. The initial noise image is initialized based on the binary repair mask to obtain an initial latent noise map. The initial potential noise map and multiple geometric condition images corresponding to the second viewpoint are input into the diffusion model for data processing to obtain the repaired image corresponding to the second viewpoint.

[0009] In some embodiments, the back-projection processing based on the multi-view image set to obtain the texture data of the three-dimensional mesh model includes: Based on the multi-view image set, illumination removal and super-resolution processing are performed to obtain an enhanced multi-view image set. For the image data corresponding to each viewpoint in the enhanced multi-view image set, the corresponding pixel confidence map is calculated; Based on a preset angle threshold, at least one of the pixel confidence maps is updated, and the updated pixel confidence map is used as the new pixel confidence map. The texture data of the three-dimensional mesh model is obtained by back-projection processing based on all the pixel confidence maps.

[0010] In some embodiments, calculating the corresponding pixel confidence map for the image data corresponding to each viewpoint in the enhanced multi-view image set includes: For each pixel in the image data corresponding to each viewpoint, the incident measurement angle is obtained based on the angle between the normal vector corresponding to the 3D mesh model and the camera direction vector corresponding to the viewpoint. Based on the incident measurement angle, the pixel confidence level corresponding to the pixel point is obtained; Based on the pixel confidence scores corresponding to all the pixels, a pixel confidence score map corresponding to the viewpoint is obtained.

[0011] In some embodiments, updating the confidence map of at least one pixel based on a preset angle threshold includes: For each viewpoint, the pixel confidence map is used to update the pixel points whose pixel confidence exceeds the preset angle threshold. Set the pixel confidence of the updated pixel to zero, and update the pixel confidence map based on the updated pixel confidence.

[0012] In some embodiments, the back-projection processing based on all the pixel confidence maps to obtain the texture data of the three-dimensional mesh model includes: For each viewpoint, the pixel confidence score corresponding to each pixel is multiplied by the color value to obtain the weighted color value corresponding to each pixel. Based on the mapping relationship between two-dimensional image pixels and three-dimensional texture space, for each texture element in the three-dimensional texture space, the weighted color values ​​corresponding to the texture element in multiple viewpoints are accumulated to obtain the cumulative color sum corresponding to the texture element, and the pixel confidence scores corresponding to the texture element in multiple viewpoints are accumulated to obtain the cumulative confidence score sum corresponding to the texture element. Based on the cumulative confidence, the cumulative color sum corresponding to each texture element is normalized to obtain the element texture data corresponding to the texture element; The texture data of the three-dimensional mesh model is obtained based on the element texture data of all the texture elements.

[0013] In some embodiments, obtaining the texture data of the 3D mesh model based on the element texture data of all the texture elements includes: Based on the element texture data of all the texture elements, a texture map is obtained; The texture map is input into a material estimation network for data processing to obtain a metallic map, a roughness map, and a bump map. The texture data of the three-dimensional mesh model is obtained based on the metallic texture, the roughness texture, and the bump map.

[0014] To achieve the above objectives, a second aspect of this application provides an apparatus for generating three-dimensional model texture data, the apparatus comprising: The acquisition module is used to acquire text prompts and multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model; The first-view processing module is used to select a first view from multiple views, input the initial noise image, the text prompt and multiple geometric condition images corresponding to the first view into the diffusion model for data processing, and obtain the repaired image corresponding to the first view. A multi-view processing module is used to select an unprocessed second view from multiple views, and perform projection and mask progressive generation based at least on the input image data and the geometric condition image corresponding to the second view to obtain the repaired image corresponding to the second view. The input image data is obtained based on the repaired images corresponding to all processed views. The texture data generation module is used to obtain a multi-view image set based on the repaired images corresponding to all viewpoints, and to perform back projection processing based on the multi-view image set to obtain the texture data of the three-dimensional mesh model.

[0015] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method for generating three-dimensional model texture data as described in the first aspect.

[0016] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for generating three-dimensional model texture data as described in the first aspect.

[0017] The method and related apparatus for generating texture data of a three-dimensional model proposed in this application include: First, acquiring a text prompt and acquiring multiple geometric condition images corresponding to multiple viewpoints of a three-dimensional mesh model; then, selecting a first viewpoint from the multiple viewpoints, inputting the initial noise image, the text prompt, and the multiple geometric condition images corresponding to the first viewpoint into a diffusion model for data processing to obtain a repaired image corresponding to the first viewpoint; next, selecting an unprocessed second viewpoint from the multiple viewpoints, and performing projection and mask progressive generation based at least on the input image data and the geometric condition images corresponding to the second viewpoint to obtain a repaired image corresponding to the second viewpoint, wherein the input image data is obtained based on the repaired images corresponding to all processed viewpoints; finally, obtaining a multi-view image set based on the repaired images corresponding to all viewpoints, and performing back projection processing based on the multi-view image set to obtain the texture data of the three-dimensional mesh model. This application embodiment acquires multiple geometric condition images, such as normal maps, position maps, and depth edge maps, corresponding to multiple viewpoints of a 3D mesh model. Combined with a text-based input diffusion model, it first ensures that the generated initial viewpoint repair images can fully fit the fine structure of the model and respond to text prompts. Then, when processing subsequent unprocessed viewpoints, a "projection and mask progressive generation" strategy is adopted. That is, the repair images corresponding to the processed viewpoints (as input image data) are first projected onto the new viewpoints to form incomplete images. Then, masks are used to repair and generate only the unknown areas under the new viewpoints. This progressive generation method from the known to the unknown ensures that the texture content of the repair images corresponding to all viewpoints has a high degree of continuity and global consistency. Thus, when performing back-projection processing based on the multi-view image set, it can effectively avoid and eliminate blurring, artifacts, and seam problems at the boundaries of different viewpoints, significantly improving the detail richness, realism, and overall naturalness of the 3D model texture data.

[0018] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for generating texture data of a three-dimensional model provided in an embodiment of this application.

[0020] Figure 2 yes Figure 1 The flowchart for step 101.

[0021] Figure 3 yes Figure 1 The flowchart for step 102.

[0022] Figure 4 yes Figure 1 The flowchart for step 103.

[0023] Figure 5 This is a schematic diagram illustrating the generation of multiple repaired images corresponding to second-viewpoints, provided in another embodiment of this application.

[0024] Figure 6 yes Figure 1 The flowchart for step 104.

[0025] Figure 7 yes Figure 6 The flowchart for step 602.

[0026] Figure 8 yes Figure 6 The flowchart for step 603.

[0027] Figure 9 yes Figure 6 The flowchart for step 604.

[0028] Figure 10 yes Figure 9 The flowchart for step 904.

[0029] Figure 11 This is a schematic diagram of generating texture data based on a multi-view image set, provided in another embodiment of this application.

[0030] Figure 12 This is a schematic flowchart illustrating the generation of three-dimensional model texture data, provided in another embodiment of this application.

[0031] Figure 13 This is a performance simulation diagram of the first method for generating texture data of a three-dimensional model provided in another embodiment of this application.

[0032] Figure 14 This is a performance simulation diagram of a second method for generating three-dimensional model texture data provided in another embodiment of this application.

[0033] Figure 15 This is a schematic diagram of the structure of a three-dimensional model texture data generation device provided in another embodiment of this application.

[0034] Figure 16 This is a schematic diagram of the hardware structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0038] With the rapid development of digital twins, animation games, and the metaverse industry, the market demand for high-quality and diverse 3D digital content is growing. In the field of AI-assisted content creation, generative AI has begun to be used to automatically generate 3D model textures. This involves using pre-trained text-to-image diffusion models to generate texture images that are aligned with the 3D model structure by rendering the model's geometric information (such as depth maps or normal maps) and combining it with control networks such as ControlNet.

[0039] However, when this method of generating 3D model textures using generative artificial intelligence relies on a pre-trained image diffusion model, the resulting image quality is limited due to a lack of comprehensive capture of the model's fine structure or difficulty in fully responding to complex text prompts. Furthermore, there are obvious inconsistencies between different views. This makes it easy for the final synthesized texture to exhibit blurring, artifacts, or obvious seams at the boundaries between different viewpoints during back projection (i.e., composite mapping). Consequently, the texture naturalness of the existing generated 3D model texture data is relatively low.

[0040] To improve the texture naturalness of the generated 3D model texture data, this application embodiment acquires multiple geometric condition images, such as normal maps, position maps, and depth edge maps, corresponding to multiple viewpoints of the 3D mesh model. Combined with a text prompt input diffusion model, it first ensures that the generated initial viewpoint repair images can fully fit the fine structure of the model and respond to the text prompts. Then, when processing subsequent unprocessed viewpoints, a "projection and mask progressive generation" strategy is adopted. That is, the repair images corresponding to the processed viewpoints (as input image data) are first projected onto the new viewpoints to form incomplete images. Then, masks are used to repair and generate only the unknown areas under the new viewpoints. This progressive generation method from the known to the unknown ensures that the texture content of the repair images corresponding to all viewpoints has a high degree of continuity and global consistency. Thus, when performing back projection processing based on the multi-view image set, it can effectively avoid and eliminate blurring, artifacts, and seam problems at the boundaries of different viewpoints, significantly improving the detail richness, realism, and overall naturalness of the 3D model texture data.

[0041] The method for generating three-dimensional model texture data and related equipment provided in the embodiments of this application will be further described below. The method for generating three-dimensional model texture data provided in the embodiments of this application can be applied to any processing system with computing resources (such as smart terminals, servers, computers, etc.).

[0042] Reference Figure 1 This is an optional flowchart of a method for generating 3D model texture data provided in an embodiment of this application. Figure 1 The method described may include, but is not limited to, steps 101 to 104. It is also understood that this embodiment... Figure 1 The order of steps 101 to 104 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0043] Step 101: Obtain text prompts and multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model.

[0044] Step 101 will be described in detail below.

[0045] In some embodiments, in response to generating texture data for a 3D mesh model, it is first necessary to obtain text prompts and multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model. This step is the data preparation stage of this application. "Text prompts" refer to natural language descriptions provided by the user to describe the desired texture semantic content, such as "European-style wooden dressing table." "3D mesh model" refers to the original 3D model data of the texture to be edited. "Multiple geometric condition images" differ from a single depth map or normal map in the prior art; instead, they are a multi-channel set of geometric constraints that provides comprehensive and accurate geometric constraints for the subsequent image generation process. Specifically, starting from multiple predefined camera "viewpoints" (e.g., including a basic viewpoint and oblique viewpoints), for each viewpoint, the normal map, position map, and depth edge map of the "3D mesh model" are rendered separately, and these two-dimensional information are stitched together into a multi-channel tensor. The input conditions are provided by these "geometric condition images". These images provide rich and precise geometric constraints on the model surface orientation, three-dimensional spatial coordinates and structural boundaries for the subsequent generation of diffusion models.

[0046] The following section will further describe how to obtain multiple geometrically conditional images corresponding to multiple viewpoints.

[0047] Reference Figure 2 The geometric condition images include normal maps, position maps, and depth edge maps. Multiple geometric condition images corresponding to multiple views of the 3D mesh model are obtained, including the following steps 201 to 204.

[0048] Step 201: Obtain multiple preset viewpoints for the 3D mesh model, including a preset basic viewpoint and a preset oblique viewpoint.

[0049] Step 202: For each viewpoint, render the 3D mesh model and calculate the normal vector direction of each pixel to obtain the normal map.

[0050] Step 203: Render the 3D mesh model and calculate the 3D spatial coordinates of each pixel to obtain the position map.

[0051] Step 204: Perform edge detection rendering on the 3D mesh model to obtain a depth edge map.

[0052] Steps 201 to 204 are described in detail below.

[0053] In some embodiments, multiple (e.g., N=10) preset viewpoints for the 3D mesh model are first obtained. The viewpoints include preset basic viewpoints (e.g., six basic viewpoints with weight W=1) and preset oblique viewpoints (e.g., elevation angles). azimuths Weight This step aims to establish a standardized, multi-directional set of observation points for subsequent geometric rendering. The "3D mesh model" is the target object from which textures need to be generated. The "viewpoint" refers to the simulated camera position and orientation. By "presetting" a fixed set of viewpoints, the consistency and reproducibility of the acquired geometric information can be ensured. Dividing the "viewpoint" into "preset basic viewpoints" (such as standard views such as front, back, left, right, top, and bottom) and "preset oblique viewpoints" (such as specific combinations of elevation and azimuth angles) is to comprehensively capture the geometric features of the model from different angles, laying the foundation for generating a fully covered geometric condition image.

[0054] Then, for each viewpoint, the 3D mesh model is rendered, and the normal vector direction of each pixel is calculated to obtain a normal map. This step is used to capture the local orientation and bump details of the model surface. Specifically, the system traverses each "viewpoint" and performs a rendering process on the "3D mesh model" from that viewpoint. The key in this process is calculating the 3D surface normal corresponding to each "pixel" in the 2D rendered image. The "normal vector direction" is the direction vector perpendicular to the tangent plane of the surface at that point. Finally, this vector information (usually its X, Y, and Z components are encoded into the RGB channels of the image) is stored, forming the "normal map." It provides strict constraints on the fine structure of the surface for subsequent texture generation. The normal map represents the normal direction of each point on the model surface, ensuring that texture details strictly correspond to the bump structure of the 3D surface.

[0055] Next, the 3D mesh model is rendered, and the 3D spatial coordinates of each pixel are calculated to obtain a position map. This step aims to provide the absolute position anchors of the model in 3D space. This step also requires rendering the "3D mesh model" (this process can be performed in the same rendering pipeline as the normal map). Its core is calculating the "3D spatial coordinates" (i.e., its X, Y, Z coordinates in the world coordinate system or camera coordinate system) of each "pixel" on the model surface in the rendered image. Storing this coordinate information as an image yields the "position map," which stores 3D points. Coordinates serve as spatial anchor points to facilitate the rational distribution of textures across structural partitions. The role of the "location map" is to help subsequent diffusion models understand the rational distribution of textures across different structural partitions, thereby promoting spatial consistency in generated textures.

[0056] Next, edge detection rendering is performed on the 3D mesh model to obtain a depth edge map. This step is used to clarify the structural boundaries and contours of the model. This "edge detection rendering" is a non-photorealistic rendering (NPR) technique. Its purpose is not to output color or normals, but to detect areas where the model has discontinuities in depth, or the boundaries between the model and the background (i.e., contour lines). The image generated after this rendering is the "depth edge map," which is usually a binary or grayscale image that clearly marks the boundaries of the model. This map is used to constrain the effective area for subsequent texture generation and can enhance the structural hierarchy of the generated texture at the edges.

[0057] Through steps 201 to 204 above, a multi-channel geometric condition image acquisition is achieved. By setting "preset basic viewpoint and preset oblique viewpoint", the comprehensiveness of the observation is ensured. The geometric features of the "3D mesh model" are extracted from three dimensions: surface orientation ("normal map"), spatial anchor point ("position map"), and structural boundary ("depth edge map"). This method of combining "normal map", "position map" and "depth edge map" as geometric conditions provides the subsequent diffusion model with richer, more comprehensive and accurate geometric constraints than a single condition map. This allows the model to fully "understand" the fine structure of the 3D model, thereby ensuring that the subsequently generated texture image can strictly correspond to the surface concavity and convexity in detail, be reasonably distributed in space, and be strictly limited within the model boundary.

[0058] Step 102: Select the first viewpoint from multiple viewpoints, input the initial noisy image, text prompt, and multiple geometric condition images corresponding to the first viewpoint into the diffusion model for data processing, and obtain the repaired image corresponding to the first viewpoint.

[0059] Step 102 is described in detail below.

[0060] Next, a first-person perspective is selected from multiple perspectives. The initial noise image, text prompt, and multiple geometric condition images corresponding to the first-person perspective are input into the diffusion model for data processing to obtain the first-person perspective. Corresponding repaired image This step aims to generate a high-quality reference view. First, a "first-person perspective" (e.g., a standard front view) is selected as the starting point for generation. Then, a randomly generated "initial noisy image," along with the acquired "textual cue" and the "multiple geometric condition images" corresponding to this "first-person perspective," are input into a pre-trained "diffusion model" (e.g., using a Kolors model). During the denoising process of this "diffusion model," the "textual cue" is injected with semantic information through a cross-attention mechanism, while the "geometric condition images" (e.g., through a parallel ControlNet branch) are injected with geometric constraints. Through "data processing" (i.e., iterative denoising of the diffusion model), a complete "restored image" (i.e., the RGB image of the "first-person perspective") that highly matches both the text semantics and the model geometry is finally generated, as described below.

[0061] Reference Figure 3 The initial noisy image, text prompt, and multiple geometric condition images corresponding to the first viewpoint are input into the diffusion model for data processing to obtain the repaired image corresponding to the first viewpoint, including the following steps 301 to 304.

[0062] Step 301: Stitch together the normal map, position map and depth edge map corresponding to the first view to obtain the geometric structure data.

[0063] Step 302: Input the geometric structure data into the control network for encoding processing to obtain geometric structure features, and input the geometric structure features into the diffusion model based on the control branch.

[0064] Step 303: Encode the text prompts to obtain text prompt features, and use cross-attention technology to input the text prompt features into the diffusion model.

[0065] Step 304: In the diffusion model, the initial noisy image is denoised based on geometric structure features and text prompt features to obtain the repaired image corresponding to the first viewpoint.

[0066] Steps 301 to 304 are described in detail below.

[0067] In some embodiments, the normal map, position map, and depth edge map corresponding to the first view are first stitched together to obtain geometric structure data. This step preprocesses and integrates various geometric constraint information from the "first-person perspective." The "normal map," "position map," and "depth edge map" are three different dimensions of geometric information rendered for this perspective in previous steps. "Concatenation" technically refers to merging these three independent images (or their channels) along the channel dimension to form a single, multi-channel tensor. This integrated tensor is defined as "geometric structure data," which, within a unified data structure, simultaneously carries the normal orientation of the model surface from this perspective, its 3D spatial coordinates, and structural boundary contour information, providing comprehensive input for the subsequent control network.

[0068] Then, the geometric structure data is input into the control network for encoding to obtain geometric structure features, and these features are then input into the diffusion model based on the control branches. This step is a specific technical path for achieving precise geometric constraints. The "control network" (e.g., ControlNet) is a specially designed neural network module whose core function is to receive the generated "geometric structure data" and "encode" it, converting it into an intermediate feature representation that the backbone network of the "diffusion model" can understand and utilize; this representation is the "geometric structure feature." "Based on control branches" describes the system architecture of this "control network," which is typically set up in parallel with the backbone denoising network of the "diffusion model" (e.g., U-Net), injecting its "geometric structure features" into the backbone network at multiple levels of the denoising process, thereby achieving precise layer-by-layer guidance of the generation process.

[0069] In addition, the text prompts are encoded to obtain text prompt features, and then input into the diffusion model using cross-attention technology. This step is the technical path to achieve semantic content constraints. "Text prompts" are natural language input by the user describing the desired texture. Before being input into the model, they first need to be "encoded" by a text encoder, converting them from a text sequence into a high-dimensional vector representation, i.e., "text prompt features" (Text Embeddings). "Cross-Attention Mechanism" is a standard technique in current mainstream diffusion models used to fuse information from different modalities (such as text and images). It allows the model to "pay attention" to relevant words and semantics in the "text prompt features" when generating each pixel of the image, thereby guiding the generated content to maintain a high degree of consistency with the description of the "text prompts."

[0070] Subsequently, in the diffusion model, the initial noisy image is denoised based on geometric structure features and text prompt features to obtain the repaired image corresponding to the first-person perspective. The core working principle of the "diffusion model" is "denoising." Starting from a completely random "initial noisy image," it iteratively predicts and removes noise to gradually restore a clear image, as shown in the learning objective in the following formula.

[0071]

[0072] in, This represents the "loss function" or "learning objective," which is the value that the model needs to minimize. The smaller the value, the more accurate the model's predictions. It is the expected value, meaning that the loss is calculated across all possible z (latent representation), c (geometric condition), and y (textual hint). The average value calculated over the combination of (real noise) and t (time step); During training, the system randomly generates and adds "real noise" to clear images. This is the noise "predicted" by the diffusion model (with parameter θ); Indicates time step The noise latent representation (i.e., the latent representation obtained by encoding the noisy image through a variational autoencoder (VAE), whose initial value is the latent representation of the initial noisy image). It is geometric structure data (i.e., a multi-channel tensor composed of normal maps, position maps, and depth edge maps). The text prompt features are used for identification. By repeatedly performing this process and minimizing the loss, the model eventually learns how to accurately separate noise from a noisy image given geometric conditions and text prompts. When generating images, this "denoising" model can start from a purely noisy image, progressively remove noise in reverse, and finally generate a clear image that meets the text and geometric conditions.

[0073] In the embodiments of this application, this "denoising process" is guided by dual conditions: it is simultaneously "based on geometric structural features" and "textual cue features". This dual guidance ensures that the model must follow both the precise geometric structure and the desired textual semantics at every step of noise removal, and the final generated "restored image corresponding to the first-person perspective" is a high-quality image that conforms to the fine structure of the 3D model and matches the user's textual description.

[0074] Through steps 301 to 304 above, a dual-constraint image generation pipeline is constructed. By "stitching" multiple geometric information ("normal map", "position map" and "depth edge map"), richer and more comprehensive "geometric structure data" is provided than a single condition map (such as only a depth map). Secondly, through "control network" and "cross-attention technology", independent geometric control channels and semantic control channels are opened for the "diffusion model". Finally, "denoising processing" is performed under the joint guidance of these two "features", which enables the generation process to respond to complex geometric structures and text prompts simultaneously and accurately. This effectively solves the problems of limited image quality, blurred details, or poor "fitting" of textures and geometric surfaces caused by insufficient geometric constraints or difficulty in responding to complex text prompts in existing technologies. It provides strong geometric fidelity and semantic controllability for generating "first-person view corresponding repaired images".

[0075] Step 103: Select an unprocessed second view from multiple viewpoints, and perform projection and mask progressive generation based at least on the input image data and the geometric condition image corresponding to the second viewpoint to obtain the repaired image corresponding to the second viewpoint. The input image data is obtained based on the repaired images corresponding to all processed viewpoints.

[0076] Step 103 will be described in detail below.

[0077] If the texture is generated solely based on the conditional image corresponding to the new viewpoint, it is often difficult to guarantee the texture consistency of the overlapping area between the new viewpoint and the existing viewpoint, which will have an adverse effect on the quality of subsequent texture synthesis.

[0078] Therefore, further, an unprocessed second perspective is selected from multiple perspectives. The process involves at least projecting and masking progressively based on the input image data and the geometric condition image corresponding to the second viewpoint to obtain the repaired image corresponding to the second viewpoint. The input image data is obtained from the repaired images corresponding to all processed viewpoints. This step is the core of achieving multi-view consistency and employs an iterative generation strategy "from known to unknown". First, an unprocessed "second viewpoint" is selected. The key lies in the "input image data", that is, the "repaired image" corresponding to the generated "first viewpoint" (or, in subsequent iterations, the set of all generated "repaired images"). The specific process of "projection and mask progressive generation" includes: 1) Projection: Projecting the "input image data" (known texture) onto the two-dimensional plane of the "second viewpoint" according to the camera parameters to form an "incomplete new viewpoint image"; 2) Masking: Generating a "binary inpainting mask" by identifying the missing regions in the incomplete image (i.e., regions visible in the "second viewpoint" but occluded in all processed viewpoints); 3) Generation: Initializing potential noise based on the mask, and then inputting the initialized noise map and the "geometric condition image" corresponding to the "second viewpoint" into the "diffusion model" to perform the inpainting process. The model will generate new textures only in the unknown regions specified by the mask, while ensuring smooth transition with known regions, thereby obtaining the "inpainted image" that is completely consistent with the previous viewpoint corresponding to the "second viewpoint". This step is executed iteratively until all "viewpoints" have been processed.

[0079] The process is described in detail below.

[0080] Reference Figure 4 The process involves at least projection and mask progressive generation based on the input image data and the geometric condition image corresponding to the second viewpoint, to obtain the repaired image corresponding to the second viewpoint, including the following steps 401 to 404.

[0081] Step 401: Based on the camera parameters of the viewpoint corresponding to the input image data, project the input image data onto the two-dimensional plane corresponding to the second viewpoint to obtain an incomplete projected image.

[0082] Step 402: Generate a binary inpainting mask based on the region missing from the viewpoint corresponding to the input image data in the incomplete projection image.

[0083] Step 403: Initialize the initial noisy image based on the binary repair mask to obtain the initial latent noise map.

[0084] Step 404: Input the initial latent noise map and multiple geometric condition images corresponding to the second viewpoint into the diffusion model for data processing to obtain the repaired image corresponding to the second viewpoint.

[0085] Steps 401 to 404 are described in detail below.

[0086] In some embodiments, for each second view First, based on the camera parameters corresponding to the viewpoint of the input image data, the input image data is projected onto a two-dimensional plane corresponding to the second viewpoint, resulting in an incomplete projected image. This step is fundamental to achieving the propagation of texture information between different viewpoints. "Input image data" refers to the collection of repaired images generated from all previously processed viewpoints (e.g., the first viewpoint). The "camera parameters" define the observation position and orientation of the "second viewpoint" (i.e., the current viewpoint to be generated). Through this "projection" operation, the system maps all known texture information ("input image data") to the "second viewpoint" according to their three-dimensional spatial relationships. On the corresponding two-dimensional plane, an "incomplete projection image" is formed that contains texture only in a portion of the area (i.e., the area that was previously covered by the viewpoint and is also visible in the current viewpoint).

[0087] Then, a binary inpainting mask is generated based on the regions missing from the corresponding viewpoint of the input image data in the incomplete projected image. The purpose of this step is to precisely define the range within which the diffusion model needs to generate new textures under the new viewpoint. "Regions missing from the corresponding viewpoint of the input image data" specifically refer to those model surface areas that are visible in the "second viewpoint" (current viewpoint) but were occluded or unobserved in all previous "corresponding viewpoints." "Binary inpainting mask" This is a binary image used to identify these "missing regions" (i.e. unknown regions). For example, unknown regions are set to 0, while known regions (regions whose textures have been obtained from the "incomplete projection image") are set to 1, thereby guiding the subsequent repair process.

[0088] Then, based on the binary repair mask The initial noisy image is initialized to obtain an initial latent noise map. This step employs a specific noise initialization strategy designed to combine known texture information with the random noise to be generated. The "binary inpainted mask" acts as a hybrid controller here: in the known regions identified by the mask (e.g., regions with a mask value of 1), the latent representation obtained from the projection of the "incomplete projected image" (i.e., the region with the mask value of 1) is used. The mask is used to fill in the unknown areas (e.g., areas with a mask value of 0); while in the unknown areas marked by the mask (e.g., areas with a mask value of 0), the "initial noise image" (i.e., randomly initialized noise) is used. Fill in the blanks as shown in the formula below.

[0089]

[0090] Through this "initialization process," the resulting "initial potential noise map" is a hybrid map that retains the texture information from the previous viewpoint in the known areas, while being ready for the generation of new content in the unknown areas.

[0091] Finally, the initial latent noise map and multiple geometric condition images corresponding to the second viewpoint are input into the diffusion model for data processing to obtain the repaired image corresponding to the second viewpoint. This step is the controlled "repair" process. The diffusion model receives the generated mixed initial latent noise map as the starting point for denoising. Simultaneously, the model also receives multiple geometric condition images corresponding to the second viewpoint (e.g., normal maps, position maps, and edge maps under this new viewpoint) as geometric constraints. Through data processing (i.e., the denoising process), the diffusion model generates new texture content only in the unknown regions (i.e., random noise regions) of the initial latent noise map, while ensuring that the new content smoothly connects with the textures of known regions and strictly adheres to the geometric constraints of the second viewpoint. Ultimately, the output "repaired image" is an image with complete and consistent content under this second viewpoint.

[0092] Reference Figure 5 This is a schematic diagram illustrating the generation of multiple repaired images corresponding to second viewpoints, provided in an embodiment of this application. For example... Figure 5 The diagram illustrates the process of "projection and mask progressive generation" of "restored images" corresponding to multiple "second-viewpoints." This process begins from a known initial viewpoint (such as...). Figure 5 The process begins with the "repaired image" corresponding to "Viewpoint 1" in the image. When an "unprocessed second viewpoint" (e.g., ...) is generated... Figure 5 When viewing "Viewpoint 2", the system first performs a "projection" operation on the "input image data" of the previous viewpoint ("Viewpoint 1") to obtain an "incomplete new viewpoint" image. Next, the system generates a corresponding "repair mask" based on the missing regions (i.e., unknown regions) in this incomplete image. Subsequently, this "repair mask" (used to initialize the latent noise map) and "multiple geometric condition images corresponding to the second viewpoint" (such as...) are then used... Figure 5 The "conditional image" is input into the "diffusion model" along with the data. After data processing, a complete "repaired image" corresponding to "viewpoint 2" and consistent with the content of "viewpoint 1" is obtained. This process continues to be executed in an iterative manner. For example, the image of "viewpoint 2" is then projected to generate "viewpoint 3" until all preset "viewpoints N" have been processed, and finally a set of multi-view images with high consistency is obtained.

[0093] This approach enables the model to generate semantically reasonable and visually coherent new textures in unknown regions specified by the mask, while maintaining smooth transitions with known regions and strictly adhering to the geometric constraints of the current viewpoint. By iteratively executing the "projection-mask construction-repair" process per viewpoint, the method provided in this application achieves the propagation and expansion of texture information between different views, ultimately resulting in a highly consistent multi-view image set. .

[0094] Through steps 401 to 404 above, a view-by-view iterative generation mechanism "from known to unknown" is constructed. By "projection" and "binary repair mask", it is ensured that the generated texture information ("input image data") can be accurately transmitted to the "second view". Then, the known information is further "implanted" into the noise map by "initialization processing", so that the "diffusion model" only needs to repair and generate in the "missing area" (i.e. unknown area) instead of completely regenerating. This realizes the propagation and expansion of texture information between different views, fundamentally ensuring that the final generated multi-view image set has high global consistency and texture continuity. It effectively avoids the defects of "multifaceted Janus problem" or texture inconsistency and obvious seams caused by independent generation of each view in the prior art.

[0095] Step 104: Obtain a multi-view image set based on the repaired images corresponding to all viewpoints, and perform back projection processing based on the multi-view image set to obtain the texture data of the three-dimensional mesh model.

[0096] Step 104 is described in detail below.

[0097] After obtaining the retouched images from all viewpoints, a multi-view image set is generated based on the retouched images corresponding to all viewpoints. Backprojection processing is then performed on this multi-view image set to obtain the texture data for the 3D mesh model. This step is the final texture synthesis stage. After iteration, a "multi-view image set" containing all N viewpoints with highly consistent content is obtained. Subsequently, "backprojection processing" is performed on this set. The purpose of this processing is to losslessly map these two-dimensional "retouched images" back to the UV space of the "3D mesh model" to synthesize the final texture map. In some preferred embodiments, this "backprojection processing" is not a simple pixel overlay but employs a "confidence-based fusion" strategy. Specifically, the system calculates a confidence score for each pixel in each view, which comprehensively considers viewpoint factors (such as the angle between the surface normal and the viewpoint to avoid grazing angles) and edge factors (to avoid seams). Finally, by performing confidence-weighted fusion and normalization on the pixel color values ​​of all views, seamless and high-quality "texture data" (e.g., albedo maps) is obtained, which can be further used to estimate PBR materials, as described below.

[0098] Reference Figure 6 The texture data of the three-dimensional mesh model is obtained by back-projection processing based on a multi-view image set, including the following steps 601 to 604.

[0099] Step 601: Perform illumination removal and super-resolution processing on the multi-view image set to obtain an enhanced multi-view image set.

[0100] Step 602: For the image data corresponding to each viewpoint in the enhanced multi-view image set, calculate the corresponding pixel confidence map.

[0101] Steps 601 to 602 are described in detail below.

[0102] In some embodiments, the multi-view image set is first subjected to delighting and super-resolution processing to obtain an enhanced multi-view image set. This step performs a crucial preprocessing operation on the "multi-view image set" generated in the previous step. The purpose of "delighting" is to eliminate the lighting inconsistencies between images from different viewpoints that may be caused by the native generation of the diffusion model, ensuring the uniformity of texture color to obtain pure albedo information, for example, using the hunyuan3d-delight-v2-0 delighting model. Subsequently, "super-resolution processing" upscales the resolution of each lighting-corrected image to a higher level (e.g., 2K) to significantly enhance texture detail and sharpness, for example, using the Real-ESRGAN super-resolution model. After this step, the resulting "enhanced multi-view image set" provides color-consistent and detail-rich "image data" for subsequent backprojection synthesis.

[0103] Then, for each viewpoint in the enhanced multi-view image set, a corresponding pixel confidence map is calculated. This step is performed for each image in the "enhanced multi-view image set" (i.e., "each viewpoint"). Each of the corresponding image data is equipped with a weighted map. "Pixel confidence map" The core function is to assign a quantified confidence score to each pixel in the "image data" to characterize the reliability or contribution value of the pixel's color information in the subsequent synthesis of the final texture, thereby quantifying the reliability of the view's contribution to the texture of each point on the model surface. In some specific embodiments, this calculation process evaluates geometric factors such as the pixel's viewing angle on the 3D model (i.e., the angle between the normal and the viewpoint), as described below.

[0104] Reference Figure 7 For each viewpoint in the enhanced multi-view image set, the corresponding pixel confidence map is calculated, including the following steps 701 to 703.

[0105] Step 701: For each pixel in the image data corresponding to each viewpoint, obtain the incident angle based on the angle between the normal vector corresponding to the 3D mesh model and the camera direction vector corresponding to the viewpoint.

[0106] Step 702: Based on the incident angle, obtain the pixel confidence level corresponding to the pixel.

[0107] Step 703: Based on the pixel confidence scores of all pixels, obtain the pixel confidence score map corresponding to the viewpoint.

[0108] Steps 701 to 703 are described in detail below.

[0109] First, the angle between the surface normal and the viewing direction is evaluated based on the normal map. Surfaces directly facing the camera are given a higher weight, while surfaces with angles exceeding a set threshold are weighted lower. The confidence level of each pixel is set to zero to avoid texture distortion and undersampling caused by excessive grazing angle. To further suppress seams that may be generated in areas of discontinuity in model contour or depth, the detected edge regions are also excluded from the mask, and their confidence level is also set to zero.

[0110] Based on this idea, for each pixel in the image data corresponding to each viewpoint, the incident angle is obtained based on the angle between the normal vector corresponding to the 3D mesh model and the camera direction vector corresponding to the viewpoint. This step is a pixel-by-pixel geometric calculation process. Specifically, for each pixel in the "image data," the system first determines its corresponding 3D spatial point on the "3D mesh model" and obtains the surface "normal vector" (i.e., the direction perpendicular to the surface of that point). At the same time, the system obtains the "camera direction vector corresponding to the viewpoint" (i.e., the direction from that 3D point to the camera position). Then, the system calculates the "angle" between these two vectors and defines this angle value as the "incident angle." This "incident angle" intuitively reflects the angle at which the camera observes the surface point: a small angle (close to 0 degrees) means that the line of sight is close to perpendicular to the surface, resulting in high observation quality; a large angle (close to 90 degrees) means that the line of sight is close to parallel to the surface (i.e., grazing angle), resulting in low observation quality and a tendency to produce texture stretching or distortion.

[0111] Then, based on the incident angle, the pixel confidence score is obtained. This step converts the obtained geometric angle value (“incident angle”) into a standardized reliability score. This score is the “pixel confidence score”. This conversion is usually an inverse relationship: a smaller “incident angle” (e.g., 0 degrees) maps to a higher “pixel confidence score” (e.g., 1.0), while a larger “incident angle” (e.g., close to or exceeding a certain threshold of 60 or 90 degrees) maps to a lower “pixel confidence score” (e.g., 0). In this way, the system assigns a quantified confidence score to the texture information of “each pixel” to characterize its contribution value in subsequent texture synthesis.

[0112] Finally, based on the pixel confidence scores of all pixels, a pixel confidence map corresponding to the viewpoint is obtained. This step aggregates these discrete confidence scores after calculating the pixel confidence scores of all pixels in the image data for the current viewpoint. The system generates a new map with the same size as the image data, where the value stored at each pixel location is the pixel confidence score corresponding to that location. This generated map is the "pixel confidence map corresponding to the viewpoint," which provides a complete, spatially aligned reliability weight map for all texture information under the current viewpoint.

[0113] Through steps 701 to 703 above, a precise pixel-level texture quality assessment mechanism based on physical observation principles is constructed. The observation quality of each pixel is quantified by calculating the "incident measurement angle" and converted into "pixel confidence". The resulting "pixel confidence map" provides a key weight map for each "view" in the "enhanced multi-view image set", providing a refined weighting basis for the subsequent back-projection fusion step. This enables the system to intelligently identify and prioritize high-quality (such as those facing the camera) pixel data, while effectively suppressing or eliminating texture distortion and undersampled pixel data caused by grazing angle. This provides a core guarantee for synthesizing a seamless and distortion-free final texture map.

[0114] Step 603: Based on a preset angle threshold, update at least one pixel confidence map and use the updated pixel confidence map as the new pixel confidence map.

[0115] Step 603 will be described in detail below.

[0116] In some embodiments, at least one pixel confidence map is updated based on a preset angle threshold, and the updated pixel confidence map is used as the new pixel confidence map. This step filters and updates the generated original "pixel confidence map" to remove low-quality pixel information. Specifically, the system uses a "preset angle threshold" (e.g., 60 degrees) to determine the observation angle. For pixels in the "pixel confidence map" whose corresponding observation angle exceeds the "preset angle threshold" (i.e., unreliable pixels at a "grazing angle"), the system forces their confidence scores to be updated to zero. In some embodiments, this "update process" may also include detecting edge regions with discontinuous model contours or depths and setting the confidence scores of these regions to zero as well, to avoid artifacts at seams. The resulting "new pixel confidence map" is a more accurate weight map that has removed all unreliable pixels (such as distortion and seam regions), as described below.

[0117] Reference Figure 8 Based on a preset angle threshold, the confidence map of at least one pixel is updated, including the following steps 801 to 802.

[0118] Step 801: For the pixel confidence map corresponding to each viewpoint, the pixels whose pixel confidence exceeds the preset angle threshold are used as the updated pixels.

[0119] Step 802: Set the pixel confidence of the updated pixel to zero, and update the pixel confidence map based on the updated pixel confidence.

[0120] Steps 801 to 802 are described in detail below.

[0121] In some embodiments, for each viewpoint's corresponding pixel confidence map, pixels whose pixel confidence exceeds a preset angle threshold are designated as update pixels. This step is a filtering process aimed at identifying pixels that need to be updated (typically suppressed). It iterates through the pixel confidence map for each viewpoint and performs a comparison operation on each pixel. This operation compares the pixel confidence (a numerical value representing reliability) with a preset angle threshold (a pre-defined baseline value). When a pixel's pixel confidence is determined to exceed the preset angle threshold, that pixel is marked as an update pixel and awaits further processing.

[0122] Next, the pixel confidence of the updated pixels is set to zero, and the pixel confidence map is updated based on the updated pixel confidence. This step is the specific operation performed on the identified "updated pixels". The system forces the "pixel confidence" value of all these "updated pixels" to "zero". This operation means that regardless of the original confidence of these pixels (i.e., higher values ​​exceeding the threshold), their reliability is now considered invalid. After all "updated pixels" have been "set to zero", the system obtains a set of "updated pixel confidence" and uses this to "update the pixel confidence map", generating a new weight map that removes these selected pixels.

[0123] By combining steps 801 and 802 above, a specific pixel confidence screening and suppression mechanism is provided. By comparing the "pixel confidence" with the "preset angle threshold", a specific set of "updated pixels" is accurately identified. By forcibly setting the "pixel confidence" of these "updated pixels" to zero, a "hard removal" of this part of the pixel data is achieved. This provides the system with the ability to actively and non-linearly filter out pixels whose "pixel confidence" is determined to "exceed" the threshold based on a preset benchmark (i.e., the "preset angle threshold"). This ensures that these specific pixel data contribute zero weight in the subsequent back-projection fusion process, thereby avoiding unnecessary impact on the final texture data quality and achieving precise control of the fusion weight.

[0124] Step 604: Perform back projection processing based on the confidence maps of all pixels to obtain the texture data of the 3D mesh model.

[0125] Step 604 will be described in detail below.

[0126] Further, backprojection processing is performed based on all pixel confidence maps to obtain the texture data of the 3D mesh model. This step is the final texture synthesis stage. The system iterates through all views in the "enhanced multi-view image set" and their corresponding updated "pixel confidence maps" for backprojection processing. The specific process of "backprojection processing" includes: weighting the color values ​​of the effective pixels (i.e., pixels with confidence greater than zero) in each view according to their corresponding "pixel confidence," and then combining them with the UV unfolding coordinates of the 3D model to "backproject" these weighted color values ​​and accumulate them to the corresponding positions in the UV texture space. After traversing all views, by performing confidence normalization processing on the accumulated color values ​​of each texture element (Texel) in the UV space, the final, seamlessly integrated "texture data of the 3D mesh model" (i.e., albedo map) is obtained, as described below.

[0127] Reference Figure 9 The texture data of the three-dimensional mesh model is obtained by back-projection based on the confidence maps of all pixels, including the following steps 901 to 904.

[0128] Step 901: For the pixel confidence map corresponding to each viewpoint, multiply the pixel confidence value of each pixel with the color value to obtain the weighted color value of each pixel.

[0129] Step 902: Based on the mapping relationship between two-dimensional image pixels and three-dimensional texture space, for each texture element in the three-dimensional texture space, the weighted color values ​​corresponding to the texture element from multiple perspectives are accumulated to obtain the cumulative color sum corresponding to the texture element, and the pixel confidence values ​​corresponding to the texture element from multiple perspectives are accumulated to obtain the cumulative confidence sum corresponding to the texture element.

[0130] Step 903: Based on the cumulative confidence, normalize the cumulative color sum corresponding to each texture element to obtain the element texture data corresponding to the texture element.

[0131] Step 904: Based on the element texture data of all texture elements, obtain the texture data of the 3D mesh model.

[0132] Steps 901 to 904 are described in detail below.

[0133] In some embodiments, for the pixel confidence map corresponding to each viewpoint, the pixel confidence score of each pixel is multiplied by its color value to obtain the weighted color value for each pixel. This step is a preprocessing step for backprojection weighted fusion. Specifically, the system iterates through the pixel confidence map of "each viewpoint" and obtains the pixel confidence score (a numerical value representing the reliability of the pixel, such as the score calculated and filtered based on the observation angle in previous steps) for "each pixel". Simultaneously, the system obtains the color value (e.g., RGB value, which is the prior weight) of the pixel in the corresponding image data. By multiplying the "pixel confidence" and the "color value," a "weighted color value" is obtained. This "weighted color value" represents the contribution of the pixel's color information to the final texture synthesis, after its reliability has been adjusted.

[0134] Then, based on the mapping relationship between 2D image pixels and 3D texture space, for each texture element in the 3D texture space, the weighted color values ​​corresponding to the texture element from multiple viewpoints are accumulated to obtain the cumulative color sum corresponding to the texture element. Simultaneously, the pixel confidence scores corresponding to the texture element from multiple viewpoints are accumulated to obtain the cumulative confidence sum corresponding to the texture element. This "mapping relationship between 2D image pixels and 3D texture space," i.e., the UV unwrapping coordinates of the 3D model, defines how "pixels" on the 2D image are mapped to "texture elements" (Texels) in the 3D texture space. The system iterates through "each texture element" in the 3D texture space and finds all "pixels" mapped to that "texture element" from "multiple viewpoints." Then, the weighted color values ​​of these "pixels" are accumulated to obtain the "cumulative color sum corresponding to the texture element"; at the same time, their original "pixel confidence scores" are also accumulated to obtain the "cumulative confidence sum corresponding to the texture element."

[0135] Next, based on the cumulative confidence score, the cumulative color sum corresponding to each texture element is normalized to obtain the element texture data corresponding to the texture element. This step is the final step of the weighted average, namely the "normalization process". Specifically, the system will perform a division operation on the "cumulative color sum corresponding to each texture element" (i.e., the weighted color sum, as the numerator) "based on the cumulative confidence score" (i.e., the obtained weighted sum, as the denominator). The result of this "normalization process" (i.e., cumulative color sum / cumulative confidence score sum) is the final average color value of the "texture element" that integrates the contributions of all viewpoints. This value is defined as the "element texture data corresponding to the texture element".

[0136] Finally, based on the elemental texture data of all texture elements, the texture data of the 3D mesh model is obtained. This step is the final texture map generation. After the elemental texture data (i.e., the final color) of all texture elements in the 3D texture space has been calculated, these elemental texture data are combined according to their corresponding positions in the 3D texture space to form the complete texture data of the 3D mesh model, such as an albedo map, which will be described further below.

[0137] Reference Figure 10 Based on the element texture data of all texture elements, the texture data of the three-dimensional mesh model is obtained, including the following steps 1001 to 1003.

[0138] Step 1001: Obtain the texture map based on the element texture data of all texture elements.

[0139] Step 1002: Input the texture map into the material estimation network for data processing to obtain the metallic map, roughness map and bump map.

[0140] Step 1003: Obtain the texture data of the 3D mesh model based on the metallic map, roughness map, and bump map.

[0141] Steps 1001 to 1003 are described in detail below.

[0142] In some embodiments, a texture map is obtained based on the element texture data of all texture elements. This step is the direct completion stage of texture synthesis. In the previous steps, the system has calculated the "elemental texture data" (i.e., the final blended color values) corresponding to each "texture element" in the "3D texture space". This step combines and arranges all these "elemental texture data" according to their predetermined coordinate positions in UV space to form a complete two-dimensional image, which is the "texture map". Technically, this "texture map" usually corresponds to the albedo map in physically based rendering (PBR), which defines the basic color information of the model's surface.

[0143] Next, the texture map is input into a material estimation network for data processing to obtain a metallic map, a roughness map, and a bump map. This step aims to infer the physical material properties of the model from the generated color information to enhance the realism of the rendering. The system takes the obtained texture map (albedo map) as input and feeds it into a dedicated material estimation network (such as the Hunyuan3D-Paint 2.1 model). This network is trained to analyze the visual features (such as texture, color, and brightness) in the texture map and "process the data" to generate a metallic map (defining the degree to which the surface "looks metallic"), a roughness map (defining the roughness of the surface's microstructure, affecting light scattering), and a bump map (or normal map, used to simulate the fine bump details of the surface during rendering).

[0144] Finally, based on the metallicity map, roughness map, and bump map, the texture data of the 3D mesh model is obtained. This step defines the final output. The "texture data of the 3D mesh model" defined in this invention is not merely the single "texture map" (albedo map) generated in step 1001, but a complete set including the albedo map and all the additional physical material maps (i.e., "metallicity map," "roughness map," and "bump map") generated in step 1002. This multi-channel, multi-map data set together constitutes all the texture information required to achieve high-quality rendering.

[0145] This invention, through a combination of steps 1001 to 1003, achieves a complete texture generation chain from base color to full PBR (Physically Based Rendering) materials. The advantage of this approach is that it doesn't stop at simply generating "texture maps" (albedo), but further generates "metallic maps," "roughness maps," and "bump maps," crucial for achieving photorealistic results, through the "material estimation network" in step 1002. The final "texture data of the 3D mesh model" obtained in step 1003 is a complete set of PBR materials. This allows the edited 3D model to simulate the interaction of light and surface in a physically accurate manner within the rendering engine, thereby greatly enhancing the physical realism, glossiness, and visual depth of the final rendering effect.

[0146] Through steps 901 to 904 above, a confidence-weighted "back-projection processing" scheme is implemented. Instead of simply superimposing images from multiple perspectives, it uses "pixel confidence" to weight "color values" to obtain "weighted color values". Then, the "weighted color values" and "pixel confidence" from "multiple perspectives" are "accumulated" to obtain "cumulative color sum" and "cumulative confidence sum". Finally, "normalization processing" (i.e. weighted average) is used to calculate the final "element texture data" of each "texture element". This can intelligently integrate information from different perspectives, giving higher weight to high-quality, high-confidence pixel data (e.g., facing the camera) while suppressing the influence of low-confidence pixel data (e.g., grazing angle or seams). This achieves a smooth and natural transition on "all texture elements", and the final "texture data of the 3D mesh model" has high seamless consistency and visual fidelity.

[0147] Reference Figure 11 This is a schematic diagram illustrating the generation of texture data based on a multi-view image set, as provided in an embodiment of this application. Figure 11The diagram illustrates the process of generating an albedo map (i.e., texture data of a 3D mesh model) through backprojection processing based on a multi-view image set. The process first acquires a normal map and calculates viewpoint weights (to obtain pixel confidence). Simultaneously, it acquires an edge map and generates a mask to exclude seam areas. Both are used to filter and generate a confidence map (i.e., a pixel confidence map). Next, this confidence map is weighted (represented by the ⊕ symbol in the diagram) with the corresponding image data (i.e., the repaired image) in the image space to obtain weighted color values ​​containing only valid pixels. Finally, these weighted color values ​​are backprojected from the image space to the UV space through UV mapping, and the contributions from all views are "mixed" (i.e., accumulated and normalized) in the UV space to obtain the final albedo map.

[0148] Through steps 601 to 604 above, a robust and high-quality back-projection texture synthesis scheme is provided. First, "light removal processing and super-resolution processing" provide a "set of enhanced multi-view images" with consistent colors and rich details for synthesis. Then, by calculating the "pixel confidence map" and performing "update processing" "based on a preset angle threshold", a precise pixel filtering mechanism is constructed. This mechanism can intelligently identify and remove (i.e., set the confidence to zero) those pixel data that are of low quality and prone to distortion due to grazing angles or model edges. Finally, through "back-projection processing", weighted fusion is performed only "based on all" of these filtered "pixel confidence maps", so that unreliable pixel areas can be actively avoided when fusing multi-view information. This fundamentally solves the problems of texture distortion, seam artifacts and insufficient sampling caused by simple superposition or poor fusion in the prior art, ensuring that the final "texture data of the 3D mesh model" has high clarity and seamless consistency.

[0149] Reference Figure 12 This is a schematic flowchart illustrating the generation of three-dimensional model texture data provided in an embodiment of this application. Figure 12 The diagram illustrates a process for generating texture data for a 3D model. First, the system acquires two initial inputs: an "input mesh" (i.e., a 3D mesh model) and a "text prompt" from the user. The "input mesh" first undergoes a "rendering" step to generate "conditional images" (i.e., multiple geometric conditional images, such as the normal map, position map, and edge map shown in the diagram).

[0150] These "conditional images" are then input into the "ControlNet" module for encoding to extract "feature embeddings"; meanwhile, "textual hints" are encoded through a "cross-attention" mechanism to provide semantic guidance to the model.

[0151] During the generation phase, the initial "noise" (i.e. the initial noisy image), the geometric "feature embedding" from "ControlNet", and the text prompt features from "cross attention" are all input into the "diffusion model" for data processing (i.e. denoising) to generate "view 1" (i.e. the repaired image corresponding to the first view).

[0152] After obtaining "viewpoint 1", the system enters the "mask-guided progressive generation" stage. Based on the image of "viewpoint 1" (as input image data) and "text prompts", this stage iteratively generates "viewpoint 2", "viewpoint 3" and up to "viewpoint N" (i.e., the repaired images corresponding to multiple second views) in the manner of "projection and mask progressive generation", and finally forms a "multi-view image set" with globally consistent content.

[0153] During the compositing stage, the "multi-view image set" first undergoes "de-illumination" and "super-resolution" processing to obtain an "enhanced multi-view image set." Subsequently, these images are fed into a "backprojection module" to perform "backprojection processing" to synthesize the final "albedo map." This "albedo map" can be further used to "estimate" and generate "PBR material maps" (such as metallicity and roughness maps). Finally, this complete set of "texture data" is "mapped" back to the original "input mesh" to obtain the final "texture mesh" with high-quality texture.

[0154] The technical solution provided in this embodiment of the invention (i.e., steps 101 to 104) ensures that the initially generated "repaired image" has high geometric fidelity and text alignment by using multiple "geometric condition images" such as normal maps, position maps, and edge maps to constrain the diffusion model. More importantly, it adopts a "projection and mask progressive generation" strategy, which forces the coherent propagation of texture information between different views by projecting known texture information ("input image data") onto a new perspective and repairing only unknown areas. This progressive generation method fundamentally guarantees the global consistency of the "multi-view image set". Therefore, when performing "backprojection processing", it can effectively overcome the problems of seams, artifacts, and blurring caused by inconsistent perspectives in the prior art. The final generated "texture data of the 3D mesh model" has significantly improved detail richness, visual realism, and overall naturalness.

[0155] Reference Figure 13 This is a performance simulation diagram of the first method for generating three-dimensional model texture data provided in the embodiments of this application. Figure 13 The diagram illustrates the three-dimensional model texture data generation method provided in this application (i.e., Figure 13The qualitative comparison results of our method and three existing technologies (MVPaint, MV-Adapter, EmbodiedGen) generating textures for three sets of input meshes (numbered 1, 2, and 3) under the same user-defined text prompts are presented. Figure 13 As shown, for complex text prompts (such as "pink frog with yellow star patterns" or "hand-knitted frog"), MVPaint struggles to fully respond to key semantics and exhibits inconsistencies across viewpoints; the output of MV-Adapter (such as a plush frog) shows obvious lighting artifacts; and EmbodiedGen fails to generate content highly consistent with the text in certain situations (such as a Disney-style pony). In contrast, the "edited texture mesh" generated by the 3D model texture data generation method provided in this application not only boasts high clarity and rich detail (such as plush textures or star patterns), accurately reflecting the visual description of the prompt, but also possesses excellent multi-view consistency.

[0156] Reference Figure 14 This is a performance simulation diagram of the second method for generating 3D model texture data provided in the embodiments of this application. Figure 14 The diagram illustrates a qualitative comparison of the texture editing results of the 3D model texture data generation method provided in this application with the three existing technologies mentioned above on a larger number of "input" mesh models. For each line of given text prompts (such as "Adusty faucet", "An Adidas shoe"), Figure 14 The results shown indicate that the textures generated by MVPaint exhibit significant multi-view inconsistencies; the MV-Adapter's generated results often retain residual lighting interference (e.g., "A Gatling gun"), leading to visible artifacts after texture mapping; and EmbodiedGen is prone to distortion when back-projected onto the mesh surface (e.g., "A school bus"). In contrast, the 3D model texture data generation method provided in this application generates textures (such as rust on a faucet, shoe logos, and train details) that consistently maintain high clarity and rich detail, accurately reflecting the semantic information in the text prompts, and, by incorporating PBR materials, visually presenting a superior sense of three-dimensionality and realism.

[0157] This application also provides a device for generating three-dimensional model texture data, which can implement the above-described method for generating three-dimensional model texture data. (Refer to...) Figure 15 The device 1500 includes: The acquisition module 1510 is used to acquire text prompts and multiple geometric condition images corresponding to multiple perspectives of the 3D mesh model. The first-view processing module 1520 is used to select a first view from multiple views, input the initial noise image, text prompt and multiple geometric condition images corresponding to the first view into the diffusion model for data processing, and obtain the repaired image corresponding to the first view. The multi-view processing module 1530 is used to select an unprocessed second view from multiple views, and perform projection and mask progressive generation based on at least the input image data and the geometric condition image corresponding to the second view to obtain the repaired image corresponding to the second view. The input image data is obtained based on the repaired images corresponding to all processed views. The texture data generation module 1540 is used to obtain a multi-view image set based on the repaired images corresponding to all viewpoints, and to perform back projection processing based on the multi-view image set to obtain the texture data of the three-dimensional mesh model.

[0158] In some embodiments, the acquisition module 1510 is further configured to: Obtain multiple preset viewpoints for the 3D mesh model, including a preset basic viewpoint and a preset oblique viewpoint; For each viewpoint, the 3D mesh model is rendered, and the direction of the normal vector of each pixel is calculated to obtain the normal map; The 3D mesh model is rendered, and the 3D spatial coordinates of each pixel are calculated to obtain a position map; Edge detection and rendering are performed on the 3D mesh model to obtain a depth edge map.

[0159] In some embodiments, the first-view processing module 1520 is further configured to: The normal map, position map, and depth edge map corresponding to the first-person perspective are stitched together to obtain the geometric structure data; The geometric structure data is input into the control network for encoding to obtain geometric structure features, and the geometric structure features are then input into the diffusion model based on the control branch. The text prompts are encoded to obtain text prompt features, and then the text prompt features are input into the diffusion model using cross-attention technology; In the diffusion model, the initial noisy image is denoised based on geometric structure features and text prompt features to obtain the repaired image corresponding to the first viewpoint.

[0160] In some embodiments, the multi-view processing module 1530 is further configured to: Based on the camera parameters corresponding to the viewpoint of the input image data, the input image data is projected onto the two-dimensional plane corresponding to the second viewpoint to obtain an incomplete projected image; A binary insulation mask is generated based on the region missing from the viewpoint corresponding to the input image data in the incomplete projection image. The initial noise image is initialized based on the binary repair mask to obtain the initial potential noise map; The initial potential noise map and multiple geometric condition images corresponding to the second viewpoint are input into the diffusion model for data processing to obtain the repaired image corresponding to the second viewpoint.

[0161] In some embodiments, the texture data generation module 1540 is further configured to: An enhanced multi-view image set is obtained by performing de-illumination and super-resolution processing on a multi-view image set. For each viewpoint in the enhanced multi-view image set, the corresponding pixel confidence map is calculated. Based on a preset angle threshold, the confidence map of at least one pixel is updated, and the updated pixel confidence map is used as the new pixel confidence map. Back-projection is performed based on the confidence maps of all pixels to obtain the texture data of the 3D mesh model.

[0162] In some embodiments, the texture data generation module 1540 is further configured to: For each pixel in the image data corresponding to each viewpoint, the incident angle is obtained based on the angle between the normal vector corresponding to the 3D mesh model and the camera direction vector corresponding to the viewpoint. Based on the incident angle, the pixel confidence level corresponding to the pixel is obtained; Based on the pixel confidence scores of all pixels, a pixel confidence score map corresponding to the viewpoint is obtained.

[0163] In some embodiments, the texture data generation module 1540 is further configured to: For each viewpoint, pixels with a confidence level exceeding a preset angle threshold are used as updated pixels. Set the pixel confidence of the updated pixel to zero, and update the pixel confidence map based on the updated pixel confidence.

[0164] In some embodiments, the texture data generation module 1540 is further configured to: For each viewpoint, the pixel confidence score is multiplied by the color value to obtain the weighted color value for each pixel. Based on the mapping relationship between two-dimensional image pixels and three-dimensional texture space, for each texture element in the three-dimensional texture space, the weighted color values ​​corresponding to the texture element from multiple perspectives are accumulated to obtain the cumulative color sum corresponding to the texture element, and the pixel confidence values ​​corresponding to the texture element from multiple perspectives are accumulated to obtain the cumulative confidence sum corresponding to the texture element. Based on the cumulative confidence, the cumulative color sum corresponding to each texture element is normalized to obtain the element texture data corresponding to the texture element. The texture data of the 3D mesh model is obtained based on the element texture data of all texture elements.

[0165] In some embodiments, the texture data generation module 1540 is further configured to: Based on the element texture data of all texture elements, a texture map is obtained; The texture map is input into the material estimation network for data processing to obtain the metallicity map, roughness map, and bump map; Texture data for a 3D mesh model is obtained based on metallic maps, roughness maps, and bump maps.

[0166] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the three-dimensional model texture data generation device is basically the same as the specific implementation of the three-dimensional model texture data generation method described above, and will not be repeated here.

[0167] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the method for generating three-dimensional model texture data as described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0168] Please see Figure 16 , Figure 16 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1602 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1602, and the processor 1601 calls and executes the method for generating three-dimensional model texture data according to the embodiments of this application. The input / output interface 1603 is used to implement information input and output; The communication interface 1604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1605 transmits information between various components of the device (e.g., processor 1601, memory 1602, input / output interface 1603, and communication interface 1604); The processor 1601, memory 1602, input / output interface 1603 and communication interface 1604 are connected to each other within the device via bus 1605.

[0169] This application also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described method for generating three-dimensional model texture data.

[0170] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0171] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0172] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0175] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0176] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0178] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating texture data for a three-dimensional model, characterized in that, The method includes: Get text prompts, and get multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model; A first perspective is selected from multiple perspectives. The initial noise image, the text prompt, and multiple geometric condition images corresponding to the first perspective are input into a diffusion model for data processing to obtain the repaired image corresponding to the first perspective. An unprocessed second viewpoint is selected from multiple viewpoints, and projection and mask progressive generation are performed based at least on the input image data and the geometric condition image corresponding to the second viewpoint to obtain the repaired image corresponding to the second viewpoint. The input image data is obtained based on the repaired images corresponding to all processed viewpoints. A multi-view image set is obtained based on the repaired images corresponding to all viewpoints. Back projection processing is performed on the multi-view image set to obtain the texture data of the three-dimensional mesh model.

2. The method for generating three-dimensional model texture data according to claim 1, characterized in that, The geometric condition images include normal maps, position maps, and depth edge maps. The acquisition of multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model includes: Obtain multiple preset viewpoints for the three-dimensional mesh model, the viewpoints including a preset basic viewpoint and a preset oblique viewpoint; For each of the aforementioned viewpoints, the three-dimensional mesh model is rendered, and the normal vector direction of each pixel is calculated to obtain the normal map; The three-dimensional mesh model is rendered, and the three-dimensional spatial coordinate information of each pixel is calculated to obtain the position map; Edge detection and rendering are performed on the three-dimensional mesh model to obtain the depth edge map.

3. The method for generating three-dimensional model texture data according to claim 2, characterized in that, The step of inputting the initial noisy image, the text prompt, and multiple geometric condition images corresponding to the first viewpoint into a diffusion model for data processing to obtain the repaired image corresponding to the first viewpoint includes: The normal map, the position map, and the depth edge map corresponding to the first viewpoint are stitched together to obtain geometric structure data; The geometric structure data is input into the control network for encoding to obtain geometric structure features, and the geometric structure features are then input into the diffusion model based on the control branch. The text prompts are encoded to obtain text prompt features, and the text prompt features are input into the diffusion model using cross-attention technology; In the diffusion model, the initial noisy image is denoised based on the geometric structure features and the text prompt features to obtain the repaired image corresponding to the first viewpoint.

4. The method for generating three-dimensional model texture data according to claim 1, characterized in that, The step of projecting and masking progressively generating the repaired image corresponding to the second viewpoint based at least on the input image data and the geometric condition image corresponding to the second viewpoint includes: Based on the camera parameters corresponding to the viewpoint of the input image data, the input image data is projected onto the two-dimensional plane corresponding to the second viewpoint to obtain an incomplete projected image; A binary insulated mask is generated based on the region missing from the viewpoint corresponding to the input image data in the incomplete projection image. The initial noise image is initialized based on the binary repair mask to obtain an initial latent noise map. The initial potential noise map and multiple geometric condition images corresponding to the second viewpoint are input into the diffusion model for data processing to obtain the repaired image corresponding to the second viewpoint.

5. The method for generating three-dimensional model texture data according to claim 1, characterized in that, The back-projection processing based on the multi-view image set to obtain the texture data of the 3D mesh model includes: Based on the multi-view image set, illumination removal and super-resolution processing are performed to obtain an enhanced multi-view image set. For the image data corresponding to each viewpoint in the enhanced multi-view image set, the corresponding pixel confidence map is calculated; Based on a preset angle threshold, at least one of the pixel confidence maps is updated, and the updated pixel confidence map is used as the new pixel confidence map. The texture data of the three-dimensional mesh model is obtained by back-projection processing based on all the pixel confidence maps.

6. The method for generating three-dimensional model texture data according to claim 5, characterized in that, The step of calculating the corresponding pixel confidence map for the image data corresponding to each viewpoint in the enhanced multi-view image set includes: For each pixel in the image data corresponding to each viewpoint, the incident measurement angle is obtained based on the angle between the normal vector corresponding to the 3D mesh model and the camera direction vector corresponding to the viewpoint. Based on the incident measurement angle, the pixel confidence level corresponding to the pixel point is obtained; Based on the pixel confidence scores corresponding to all the pixels, a pixel confidence score map corresponding to the viewpoint is obtained.

7. The method for generating three-dimensional model texture data according to claim 6, characterized in that, The step of updating the confidence map of at least one pixel based on a preset angle threshold includes: For each viewpoint, the pixel confidence map is used to update the pixel points whose pixel confidence exceeds the preset angle threshold. Set the pixel confidence of the updated pixel to zero, and update the pixel confidence map based on the updated pixel confidence.

8. The method for generating three-dimensional model texture data according to claim 5, characterized in that, The back-projection process based on all the pixel confidence maps to obtain the texture data of the 3D mesh model includes: For each viewpoint, the pixel confidence score corresponding to each pixel is multiplied by the color value to obtain the weighted color value corresponding to each pixel. Based on the mapping relationship between two-dimensional image pixels and three-dimensional texture space, for each texture element in the three-dimensional texture space, the weighted color values ​​corresponding to the texture element in multiple viewpoints are accumulated to obtain the cumulative color sum corresponding to the texture element, and the pixel confidence scores corresponding to the texture element in multiple viewpoints are accumulated to obtain the cumulative confidence score sum corresponding to the texture element. Based on the cumulative confidence, the cumulative color sum corresponding to each texture element is normalized to obtain the element texture data corresponding to the texture element; The texture data of the three-dimensional mesh model is obtained based on the element texture data of all the texture elements.

9. The method for generating three-dimensional model texture data according to claim 8, characterized in that, The process of obtaining the texture data of the 3D mesh model based on the element texture data of all the texture elements includes: Based on the element texture data of all the texture elements, a texture map is obtained; The texture map is input into a material estimation network for data processing to obtain a metallic map, a roughness map, and a bump map. The texture data of the three-dimensional mesh model is obtained based on the metallic texture, the roughness texture, and the bump map.

10. A device for generating texture data of a three-dimensional model, characterized in that, The device includes: The acquisition module is used to acquire text prompts and multiple geometric condition images corresponding to multiple viewpoints of the 3D mesh model; The first-view processing module is used to select a first view from multiple views, input the initial noise image, the text prompt and multiple geometric condition images corresponding to the first view into the diffusion model for data processing, and obtain the repaired image corresponding to the first view. A multi-view processing module is used to select an unprocessed second view from multiple views, and perform projection and mask progressive generation based at least on the input image data and the geometric condition image corresponding to the second view to obtain the repaired image corresponding to the second view. The input image data is obtained based on the repaired images corresponding to all processed views. The texture data generation module is used to obtain a multi-view image set based on the repaired images corresponding to all viewpoints, and to perform back projection processing based on the multi-view image set to obtain the texture data of the three-dimensional mesh model.

11. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method for generating three-dimensional model texture data according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for generating three-dimensional model texture data as described in any one of claims 1 to 9.

Images