Texture map processing method, server, storage medium and program product

Abstract

The present application provides a texture mapping processing method, server, storage medium and program product. The method of the present application generates 2D texture maps of multiple viewing angles through an image generation model according to the description information of the surface texture of a 3D model; maps the 2D texture maps of each viewing angle to a 3D space to obtain a local 3D texture map of the 3D model at each viewing angle; aggregates the local 3D texture maps of the 3D model at each viewing angle to obtain a complete 3D texture map of the 3D model. The texture mapping of the 3D model can be realized by providing only a 3D model (a blank model) and description information of the surface texture of the 3D model without additional data collection and annotation processes, thereby improving the efficiency of the 3D model texture mapping.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a texture mapping processing method, a server, a storage medium and a program product. Background Art

[0002] Texture mapping of three-dimensional (3D) models is a very important technology. It can add complex details and color information to the surface of 3D models, making the models look more realistic. It has a wide range of applications in games, animation, film and television production, virtual reality (VR) and augmented reality (AR). For example, skinning game characters and anime characters, updating 3D object materials, etc.

[0003] Current texture mapping solutions for 3D models usually use an optimization method of supervised pre-training and supervised fine-tuning training, which requires a large amount of labeled data as supervision data. The cost of data labeling is high, and the efficiency of texture mapping is low. Summary of the invention

[0004] The present application provides a texture mapping processing method, server, storage medium and program product to solve the problem that texture mapping of 3D models requires supervised training and data annotation is costly and inefficient.

[0005] In a first aspect, the present application provides a method for processing a texture map, comprising: generating 2D texture maps of multiple perspectives through an image generation model according to description information of the surface texture of a 3D model; mapping the 2D texture map of each perspective to a 3D space to obtain a local 3D texture map of the 3D model at each perspective; aggregating the local 3D texture maps of the 3D model at each perspective to obtain a complete 3D texture map of the 3D model.

[0006] In a second aspect, the present application provides a method for processing a texture map, including: receiving a request for obtaining a texture map of a 3D model sent by an end-side device, obtaining the 3D model for which the texture map is to be generated and description information of the surface texture of the 3D model; generating 2D texture maps of multiple perspectives through an image generation model according to the description information; mapping the 2D texture map of each perspective to a 3D space to obtain a local 3D texture map of the 3D model at each perspective; aggregating the local 3D texture maps of the 3D model at each perspective to obtain a complete 3D texture map of the 3D model; and returning the complete 3D texture map of the 3D model to the end-side device.

[0007] In a third aspect, the present application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the server executes a method provided in any of the preceding aspects.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method provided in any of the aforementioned aspects is implemented.

[0009] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method provided in any of the aforementioned aspects.

[0010] The texture mapping processing method, server, storage medium and program product provided by the present application generate 2D texture maps of multiple perspectives through an image generation model according to the description information of the surface texture of a 3D model; map the 2D texture map of each perspective to a 3D space to obtain a local 3D texture map of the 3D model at each perspective; aggregate the local 3D texture maps of the 3D model at each perspective to obtain a complete 3D texture map of the 3D model. Texture mapping of the 3D model can be achieved by providing only a 3D model (a white model) and description information of the surface texture of the 3D model, without the need for additional data collection and annotation processes, thereby improving the efficiency of texture mapping of the 3D model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 A schematic diagram of an exemplary system architecture applicable to the present application;

[0013] Figure 2 A flowchart of a texture mapping processing method provided for an exemplary embodiment of the present application;

[0014] Figure 3 A framework diagram for generating 2D texture maps of multiple perspectives based on description information provided by an exemplary embodiment of the present application;

[0015] Figure 4 A framework diagram of 2D-3D joint denoising using an SD model provided by an exemplary embodiment of the present application;

[0016] Figure 5A flowchart of 2D-3D joint denoising by SD model provided by an exemplary embodiment of the present application;

[0017] Figure 6 A flowchart of generating 2D texture maps of multiple perspectives through an image generation model provided by an exemplary embodiment of the present application;

[0018] Figure 7 A schematic diagram of an iterative denoising process provided by an exemplary embodiment of the present application;

[0019] Figure 8 A flowchart of texture repairing of a 3D texture map provided by an exemplary embodiment of the present application;

[0020] Fig. 9 An example diagram of texture restoration provided by an exemplary embodiment of the present application;

[0021] Fig.10 An overall framework diagram of texture mapping processing provided by an exemplary embodiment of the present application;

[0022] Fig.11 A flowchart of a texture mapping processing method provided by another exemplary embodiment of the present application;

[0023] Fig.12 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0024] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0026] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] First, the terms involved in this application are explained:

[0028] Diffusion Model (DM): It includes the forward process and the reverse denoising process. The forward process destroys the data x0 incrementally by adding noise ∈ into a noise sequence x1,…,x T , following the Markov chain. The inverse denoising process starts from x T Iterative denoising obtains clean data.

[0029] Stable Diffusion (SD): All features are projected into latent space to save diffusion calculations. SD also uses an encoder-decoder based U-Net for noise prediction. The multi-scale transformer block (Transformer Block) contained in U-Net consists of a self-attention model and a cross-attention module. The self-attention model allows the network to prioritize long-distance related features within a two-dimensional (2D) image, while the cross-attention aligns the denoising process with the description information to achieve controllable generation.

[0030] Mesh Rendering: Given a 3D mesh (denoted by M) with vertices V, triangles F, and attributes A, the mesh rendering process converts the 3D mesh into a 2D image by rasterization. Each triangle f∈F consists of three vertices and the corresponding attributes where f1, f2, and f3 represent the indices (identities) of the three vertices stored in face f. For each pixel I in the 2D image i , the rendering process consists of tracing a ray that passes through the triangle f k Point P i (called the center of gravity or mass) and intersects the 3D grid. i The property values ​​use the barycentric coordinates The interpolation calculation is obtained. Among them, Pixel value I i Calculated by linear combination:

[0031] Mapping function for mesh rendering: the process of rasterizing the mesh vertices of a 3D mesh onto a 2D image plane.

[0032] Inverse mapping function: refers to the inverse function of the mapping function of mesh rendering, which is the process of mapping the pixel values ​​of the 2D image to the mesh vertices of the 3D mesh.

[0033] Attention Mechanism (AM): It is a commonly used machine learning weighted query technology. The input of this mechanism consists of two parts: key-value pair features and query sets. For convenience, key-value pair features are often split into key sets and value sets for explanation.

[0034] Self-Attention Mechanism (SAM): It is a special attention mechanism whose input key set, value set and query set are all the same set.

[0035] Cross-Attention Mechanism (CAM): is a special attention mechanism whose input key set and query set are two different sets.

[0036] The present application provides a texture mapping processing method, which generates 2D texture maps of multiple viewing angles through an image generation model according to description information of the surface texture of a 3D model; maps the 2D texture map of each viewing angle to a 3D space to obtain a local 3D texture map of the 3D model at each viewing angle; aggregates the local 3D texture maps of the 3D model at each viewing angle to obtain a complete 3D texture map of the 3D model, and only needs to provide a 3D model (a blank model) and description information of the surface texture of the 3D model, without the need for additional data collection and annotation processes, to achieve texture mapping of the 3D model, thereby improving the efficiency of texture mapping of the 3D model.

[0037] The 3D model is a 3D model to be texture mapped, which can be a 3D blank model or a 3D model with an initial texture, which is not specifically limited in this embodiment. If the 3D model has an initial texture, the texture map is regenerated for the 3D model based on the description information by the method of the present application, and the initial texture map will not affect the effect of the regenerated texture map.

[0038] The image generation model can be a pre-trained deep learning model that generates 2D images based on input conditions, such as a conditional stable diffusion (SD) model, a text-based graph model (generating 2D images based on descriptive text), etc., which is not specifically limited in this embodiment.

[0039] The method of this embodiment can be applied to games, animations, film and television production, VR / AR and other fields to generate texture maps of 3D models based on description information. Specifically, it can be applied to 3D digital human mapping, game character skinning, anime character skinning, 3D object material updating, etc.

[0040] Figure 1 FIG. 1 is a schematic diagram of an exemplary system architecture applicable to the present application. Figure 1As shown, the system architecture includes a server and a terminal device. There is a communication link between the server and the terminal device, which can realize the communication connection between the server and the terminal device.

[0041] The server is a device with computing power deployed in the cloud or locally, such as a cloud cluster. The server stores a pre-trained image generation model. The server is responsible for generating 2D texture maps of multiple perspectives using the image generation model based on a given 3D model and the description information of the surface texture of the 3D model to be generated, mapping the 2D texture maps of each perspective to the 3D space, obtaining the local 3D texture maps of the 3D model at each perspective, and aggregating the local 3D texture maps of the 3D model at each perspective to obtain the complete 3D texture map of the 3D model.

[0042] The end-side device can be an electronic device running downstream application software / system, and specifically can be a hardware device with network communication function, computing function and information display function, including but not limited to smart phones, tablet computers, desktop computers, local servers, cloud servers, etc. For example, the end-side device can be an electronic device running various software applications / systems such as game design, film and television animation production, industrial design, architectural design, interior design, product design, etc. The downstream application software / system can generate a 3D texture map that conforms to the description information of the 3D model based on the given 3D model and description information, render the 3D model based on the 3D texture map, realize game character skinning, animation character skinning, 3D object material update, etc., and further complete the subsequent processing logic such as game design, film and television animation production, and 3D object dynamic display.

[0043] When texture mapping of a 3D model is required, the user sends the 3D model and description information of the 3D model surface texture to the server through the client device. The server generates 2D texture maps of multiple perspectives based on the description information of the 3D model surface texture through the image generation model, maps the 2D texture maps of each perspective to the 3D space, obtains the local 3D texture maps of the 3D model at each perspective, aggregates the local 3D texture maps of the 3D model at each perspective, and obtains the complete 3D texture map of the 3D model.

[0044] Furthermore, the server returns the complete 3D texture map of the 3D model to the end-side device. The end-side device renders the 3D model according to the complete 3D texture map of the 3D model, realizes game character skinning, animation character skinning, 3D object material update, etc., and further completes subsequent processing logic such as game design, film and television animation production, and 3D object dynamic display.

[0045] The texture mapping scheme provided in this application can generate 3D textures with high fidelity and show global and local consistency. In addition, by leveraging the capabilities of the pre-trained diffusion model, a 3D texture mapping method that does not require training is provided, which has good time efficiency and significant robustness and generalization capabilities when processing diverse 3D objects and complex description information (such as complex text descriptions).

[0046] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0047] Figure 2 The flowchart of the texture map processing method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0048] Step S201: Generate 2D texture maps of multiple viewing angles through an image generation model according to the description information of the surface texture of the 3D model.

[0049] The description information of the surface texture of the 3D model is used to describe the target texture map that the customer wants to generate for the 3D model. The description information includes but is not limited to text, images, videos, audio, etc., and may include at least one type of description information among text, images, videos, and audio.

[0050] The above description information is mainly used to describe the final presentation form of the 3D model, for example, the presentation style, texture, feel, state, color, logo, etc. of the 3D model. For example, the description information can be a text description, such as "brown bear", "baseball glove", etc., or a depth map of the 3D model.

[0051] In this embodiment, the image generation model can be a pre-trained deep learning model that generates 2D images based on input conditions, such as a conditional stable diffusion (SD) model, a text-based graph model (generating 2D images based on descriptive text), etc. This embodiment is not specifically limited here.

[0052] In this step, the description information of the surface texture of the 3D model is used as an input condition, and a pre-trained image generation model is used to generate multiple 2D images of different perspectives that match the description information based on the description information of the surface texture of the 3D model, thereby obtaining 2D texture maps of multiple perspectives.

[0053] The multiple perspectives include at least two different perspectives. For a simpler 3D model, 2D images from two perspectives can cover the entire surface of the 3D model. However, for a more complex 3D model, 2D images from more perspectives are required to cover the entire surface of the 3D model.

[0054] Exemplarily, the multiple viewing angles may include 8 viewing angles, such as 0°, 45°, 90°, 135°, ..., with adjacent viewing angles spaced 45° apart; or the multiple viewing angles may include 6 viewing angles, such as 0°, 60°, 120°, 180°, 240°, 300°, with adjacent viewing angles spaced 60° apart; or the multiple viewing angles may include 4 viewing angles, such as 0°, 80°, 160°, and 280°, with adjacent viewing angles spaced 90° apart. In this step, which viewing angles the multiple viewing angles specifically include may be configured according to the specific 3D model and actual application scenario, and is not specifically limited here.

[0055] Step S202: Map the 2D texture map of each viewing angle to the 3D space to obtain a local 3D texture map of the 3D model at each viewing angle.

[0056] After obtaining 2D texture maps of multiple viewing angles, the 2D texture map of any viewing angle is mapped to the 3D space where the 3D model is located to obtain the 3D texture map of the 3D model at the viewing angle.

[0057] For a 3D model, only part of the surface texture of the 3D model can be observed from a single perspective, that is, the 2D texture map from a single perspective only contains part of the surface texture of the 3D model. Therefore, the 3D texture map obtained by mapping the 2D texture map from a single perspective to the 3D space where the 3D model is located only covers part of the surface texture of the 3D model, which is called the local 3D texture map of the 3D model from a single perspective.

[0058] In this step, the 2D texture maps of multiple viewing angles are respectively mapped to the 3D space where the 3D model is located, so that the local 3D texture maps of the 3D model at each viewing angle can be obtained.

[0059] The process of mapping a 2D texture map of a single perspective to a 3D space is essentially the inverse process of rendering a 3D model as a 2D image at a corresponding perspective. In practical applications, the mapping function corresponding to the 3D model rendering and the inverse function of the mapping function can be determined according to the implementation principle of the 3D model rendering. In this step, the 2D texture map of a single perspective is mapped to the 3D space according to the inverse function of the predetermined and configured mapping function, and the local 3D texture map of the 3D model at the corresponding perspective can be obtained.

[0060] Step S203: aggregate the local 3D texture maps of the 3D model at each viewing angle to obtain a complete 3D texture map of the 3D model.

[0061] After obtaining the local 3D texture maps of the 3D model at multiple viewing angles, the local 3D texture maps of the 3D model at each viewing angle are aggregated to obtain a complete 3D texture map of the 3D model.

[0062] Specifically, for any point in the 3D space, the texture information at that point in the local 3D texture maps under multiple different perspectives is aggregated (such as averaging or weighted averaging) to obtain the 3D texture information at that point, thereby obtaining the 3D texture information at each point on the 3D model, that is, obtaining a complete 3D texture map of the 3D model.

[0063] Furthermore, after obtaining the 3D texture map of the 3D model, the server can also return the 3D texture map of the 3D model to the end-side device. After receiving the 3D texture map of the 3D model, the end-side device renders the 3D model based on the 3D texture map of the 3D model to achieve game character skinning, animation character skinning, 3D object material update, etc., and further complete subsequent processing logic such as game design, film and television animation production, and 3D object dynamic display.

[0064] Optionally, after obtaining the 3D texture map of the 3D model, the server can render the 3D model based on the 3D texture map of the 3D model to achieve game character skinning, animation character skinning, 3D object material update, etc. The server returns the rendered 3D model to the end-side device. The end-side device receives the rendered 3D model returned by the server.

[0065] The method of this embodiment generates 2D texture maps of multiple perspectives through an image generation model based on the description information of the surface texture of the 3D model; maps the 2D texture map of each perspective to a 3D space to obtain a local 3D texture map of the 3D model at each perspective; aggregates the local 3D texture maps of the 3D model at each perspective to obtain a complete 3D texture map of the 3D model. The method only needs to provide a 3D model (a blank model) and the description information of the surface texture of the 3D model, and does not require additional data collection and annotation processes to achieve texture mapping of the 3D model, thereby improving the efficiency of texture mapping of the 3D model.

[0066] In an optional embodiment, the image generation model used in step S201 can be a stable diffusion (SD) model based on input conditions. The stable diffusion model includes multiple transformer blocks (Transformer Block) of different scales, and the transformer block (Transformer Block) includes two modules: self-attention and cross-attention. There is a residual module (ResBlock) in front of each TransformerBlock, and the residual module is used to extract features and output hidden layer features (referred to as 2D intermediate features in this embodiment) as input features of the transformer block (Transformer Block).

[0067] Figure 3 A framework diagram for generating 2D texture maps of multiple perspectives based on description information provided by an exemplary embodiment of the present application. Figure 3 As shown in Figure 1, the process of generating 2D texture maps from multiple perspectives through a stable diffusion model is as follows:

[0068] The configured (initialized) 2D noise features of multiple views are used as 2D texture noise features of multiple views (such as Figure 3 The X shown in t ), the noise is predicted through the residual module (ResBlock) and the transformer block (Transformer Block) of the SD model to obtain the predicted noise features in the 2D texture noise features of each view (such as Figure 3 ∈ shown in t ), the 2D texture noise feature X t The predicted noise features ∈ t Remove, and obtain the denoised 2D features of each perspective after this iteration (current time step) (such as Figure 3 The X shown in t-1 ) as the initial 2D noise feature of the next iteration (next time step), and perform denoising in the next iteration (next time step). After the iteration is completed, the denoised 2D feature (such as Figure 3 X0) shown in , as 2D texture features of multiple viewing angles. The 2D texture features of each viewing angle are decoded by a decoder to obtain 2D texture maps of multiple viewing angles.

[0069] Using the SD model to t Perform an iterative denoising to obtain X t-1 As an example, the initial 2D texture noise features X tThe residual module is used to extract features and obtain the 2D intermediate features of each view. The 2D intermediate features of each view are input into the self-attention module for self-attention calculation to obtain the 2D fusion features of each view. Furthermore, the cross-attention module is used to perform cross-attention calculation on the 2D fusion features of each view and the features of the given description information (usually the embedding vector mapped to the description information) to obtain the 2D texture noise features X of each view. t The predicted noise features ∈ t The process of each iteration of iterative denoising using the SD model is similar, except that the scale of the transformer block used is different, which will not be repeated here.

[0070] Figure 3 The self-attention in the transformer block of the SD model shown is used to perform self-attention calculation on 2D features, which is referred to as a 2D self-attention module. In an optional embodiment, a module for performing self-attention calculation on 3D features that map 2D features to 3D space is added to the transformer block of the SD model, which is referred to as a 3D self-attention module.

[0071] Figure 4 This is a framework diagram for performing 2D-3D joint denoising based on description information provided in this embodiment. Figure 4 As shown in the figure, in each iterative denoising process, the 2D texture noise features of each perspective (such as Figure 4 The X shown in t ) to extract features and obtain 2D intermediate features of each view (such as Figure 4 F shown in t ), input transformer block (TransformerBlock).

[0072] The Transformer Block includes: Figure 4 The 3D self-attention and 2D self-attention modules shown in the figure, as well as the cross attention module. The 2D self-attention module is used to input the 2D intermediate features of multiple perspectives (such as Figure 4 F shown in t ) performs self-attention calculation to obtain the first 2D fusion features of multiple perspectives (such as Figure 4 As shown in ).

[0073] In this embodiment, the inverse function of the mapping function based on mesh rendering is used to convert the 2D intermediate features (such as Figure 4 F shown in t) are respectively de-rasterized into 3D intermediate features; the outer bounding box of the 3D intermediate feature is divided into multiple volume grids, and the 3D self-attention module is used to perform self-attention calculations on the 3D intermediate features in each volume grid to obtain the 3D fusion features of each volume grid (such as Figure 4 As shown in ), the 3D fusion features of each volume grid are spliced ​​to form a complete 3D fusion feature. By dividing the volume grid, the 3D receptive field is customized for the 3D fusion feature. The 3D fusion feature only focuses on the features within the same volume grid, while the same volume grid contains the cross-view information of the 2D intermediate features, and can fuse the feature information of multiple perspectives.

[0074] Specifically, when dividing the outer bounding box of the 3D intermediate feature into a plurality of volume grids, the outer bounding box of the 3D intermediate feature is calculated, and the outer bounding box of the 3D intermediate feature is divided into a plurality of volume grids according to the configured volume grid size. The volume grid may be a cubic grid or a grid of other shapes (such as a cuboid), and the shape, size, and division method of the volume grid may be configured according to the actual application scenario, and are not specifically limited here.

[0075] For example, the entire 3D intermediate feature can be divided into 4 equal parts in length, width, and height, respectively, to obtain 64 (4×4×4) volume grids; or, the entire 3D intermediate feature can be divided into 6 equal parts in length, width, and height, respectively, to obtain 216 (6×6×6) volume grids.

[0076] Optionally, since the volume grid has the same size and there is no overlap between the volume grids, only the features in the same volume grid interact during the self-attention calculation, and the volume grid is divided using the same volume grid size at different time steps, so that different time steps have repeated 3D receptive fields and feature interactions are limited. Therefore, in different time steps of the denoising process, two different volume grid sizes are alternately used to divide the volume grid, so that there are different 3D receptive fields in adjacent time steps, which can eliminate the isolation of using repeated 3D receptive fields.

[0077] Furthermore, based on the mapping function of the grid rendering, the complete 3D fusion feature is rasterized to obtain the second 2D fusion feature of each viewing angle.

[0078] Furthermore, the second 2D fusion feature of each viewing angle is fused with the first 2D fusion feature of the corresponding viewing angle to obtain a third 2D fusion feature of each viewing angle.

[0079] Furthermore, the cross-attention module performs cross-attention calculation on the third 2D fusion feature of each view and the feature of the given description information (usually the embedding vector mapped to the description information) to obtain the 2D texture noise feature X of each view t The predicted noise features ∈t . The 2D texture noise feature X t The predicted noise features ∈ t Remove, and obtain the denoised 2D features (X t-1 ), as the initial 2D noise feature of the next iteration (next time step), and perform denoising in the next iteration (next time step).

[0080] After the iteration is completed, the denoised 2D features (X0) obtained in the last iteration are used as the 2D texture features of multiple perspectives. The 2D texture features of each perspective are decoded by the decoder to obtain 2D texture maps of multiple perspectives.

[0081] Figure 5 The flowchart of 2D-3D joint denoising by SD model provided in this embodiment. Figure 5 As shown, according to the description information of the surface texture of the 3D model, the SD model is used to perform 2D-3D joint denoising on the 2D texture noise features of multiple perspectives to obtain the denoised 2D features of multiple perspectives. The specific implementation steps are as follows:

[0082] Step S501: extract 2D texture noise features of multiple viewing angles through the SD model to obtain 2D intermediate features of multiple viewing angles.

[0083] In this step, during an iterative denoising process using the SD model, the 2D texture noise features of each view are input into the residual module of the SD model for feature extraction to obtain the 2D intermediate features of each view (such as Figure 4 F shown in t ).

[0084] After obtaining the 2D intermediate features of each view (such as Figure 4 F shown in t ) After that, the self-attention calculation of the two branches is performed to realize 2D-3D joint denoising. Among them, one branch performs self-attention calculation on the 2D intermediate features of each perspective through the 2D self-attention module to obtain the first 2D fusion features of each perspective, see step S502. The other branch maps the 2D intermediate features of each perspective to the 3D intermediate features of the 3D space through the 3D self-attention module, performs self-attention calculation, and maps it back to the 2D space to obtain the second 2D fusion features, see steps S503-S505.

[0085] Step S502: perform self-attention calculation on the 2D intermediate features of multiple perspectives to obtain the first 2D fusion features of multiple perspectives.

[0086] In this step, the 2D intermediate features (such as Figure 4 F shown int ) is input into the 2D self-attention module, and the 2D self-attention module performs self-attention calculation on the 2D intermediate features of multiple perspectives to obtain the first 2D fusion features of multiple perspectives (such as Figure 4 As shown in ).

[0087] Step S503 : Based on the inverse function of the mapping function of the grid rendering, the 2D intermediate features of multiple perspectives are de-rasterized into 3D intermediate features.

[0088] In this step, the 2D intermediate features of a single perspective are mapped to the 3D space according to the inverse function of the mapping function of the predetermined and configured grid rendering, so that the local 3D intermediate features of each perspective can be obtained; the local 3D intermediate features of each perspective can be aggregated to obtain the complete 3D intermediate features.

[0089] Step S504: divide the outer bounding box of the 3D intermediate feature into multiple volume grids, perform self-attention calculation on the 3D intermediate features in each volume grid, and obtain a 3D fusion feature.

[0090] In this step, the 3D intermediate features are divided into multiple volume grids, and the 3D self-attention module performs self-attention calculations on the 3D intermediate features in each volume grid to obtain the 3D fusion features of each volume grid (such as Figure 4 As shown in ), the 3D fusion features of each volume grid are spliced ​​to form a complete 3D fusion feature. By dividing the volume grid, the 3D receptive field is customized for the 3D fusion feature. The 3D fusion feature only focuses on the features within the same volume grid, while the same volume grid contains the cross-view information of the 2D intermediate features, and can fuse the feature information of multiple perspectives.

[0091] Specifically, when the outer bounding box of the 3D intermediate feature is divided into multiple volume grids, the outer bounding box of the 3D intermediate feature is calculated, and according to the configured volume grid size, the outer bounding box of the 3D intermediate feature is divided into multiple volume grids, where the volume grid can be a cubic grid or a grid of other shapes (such as a cuboid). The shape, size and division method of the volume grid can be configured according to the actual application scenario and are not specifically limited here.

[0092] For example, the entire 3D intermediate feature can be divided into 4 equal parts in length, width, and height, respectively, to obtain 64 (4×4×4) volume grids; or, the entire 3D intermediate feature can be divided into 6 equal parts in length, width, and height, respectively, to obtain 216 (6×6×6) volume grids.

[0093] Optionally, since the volume grid has the same size and there is no overlap between the volume grids, only the features within the volume grid are allowed to interact during the self-attention calculation. The volume grid is divided using the same volume grid size at different time steps, so that different time steps have repeated 3D receptive fields and feature interactions are limited. Therefore, in different time steps of the denoising process, two different volume grid sizes are alternately used to divide the volume grid, so that there are different 3D receptive fields in adjacent time steps, which can eliminate the isolation of using repeated 3D receptive fields.

[0094] Step S505: rasterize the 3D fusion features into second 2D fusion features of each viewing angle.

[0095] In this step, based on a single viewing angle, the 3D fusion feature is rasterized into a corresponding second 2D fusion feature according to the mapping function of the grid rendering. Similarly, the 3D fusion feature is rasterized according to the mapping function of the grid rendering to obtain the second 2D fusion feature of each viewing angle.

[0096] Step S506: Fuse the second 2D fusion feature of each viewing angle with the first 2D fusion feature of the corresponding viewing angle to obtain a third 2D fusion feature of each viewing angle.

[0097] In this step, the second 2D fusion feature of each viewing angle is fused (summed, averaged, or weighted summed) with the first 2D fusion feature of the corresponding viewing angle to obtain the third 2D fusion feature of each viewing angle.

[0098] Step S507: Determine the denoised 2D features of each viewing angle according to the third 2D fusion features of each viewing angle.

[0099] In this step, the cross attention module performs cross attention calculation on the third 2D fusion feature of each view and the feature of the given description information (usually the embedding vector mapped to the description information) to obtain the 2D texture noise feature X of each view. t The predicted noise features ∈ t . The 2D texture noise feature X t The predicted noise features ∈ t Remove, and obtain the denoised 2D features (X 0|t ), and then added with noise again as the initial 2D noise feature (X) for the next iteration (next time step) t-1 ), and perform denoising in the next iteration (next time step).

[0100] After the iteration is completed, the denoised 2D features (X0) obtained in the last iteration are used as the 2D texture features of multiple perspectives. The 2D texture features of each perspective are decoded by the decoder to obtain 2D texture maps of multiple perspectives.

[0101] It should be noted that, in multiple iterations (multiple time steps) of denoising by the SD model, some iterations (partial time steps) can be sampled to use the method of this embodiment for 2D-3D joint denoising, while in other iterations only the 2D self-attention module is used (the 3D self-attention module is not used, and other modules are the same) for denoising. In addition, the method of this embodiment can also be used for 2D-3D joint denoising in each iteration (each time step) of denoising by the SD model.

[0102] The method of this embodiment proposes a method for combining a 3D model with 2D image features through 2D-3D joint denoising. Not only does the 2D self-attention module perform self-attention calculation on the 2D intermediate features of multiple perspectives to obtain a first 2D fused feature, but a 3D self-attention module is added to de-rasterize the 2D intermediate features of multiple perspectives into 3D intermediate features, and the outer bounding box of the 3D intermediate features is divided into multiple volume grids. The 3D self-attention module performs self-attention calculation on the 3D intermediate features in each volume grid to obtain a 3D fused feature, and then the 3D fused feature is rasterized into a second 2D fused feature of each perspective. The second 2D fused feature of each perspective is fused with the first 2D fused feature of the corresponding perspective for predicting noise. The 3D model is combined with the 2D image features in a self-attention manner, and the global consistency and local cross-perspective consistency of the 2D texture features of multiple perspectives are considered through a 2D-3D joint noise prediction method, which can improve the effect of generating 3D texture maps.

[0103] Figure 6 A flowchart of generating 2D texture maps of multiple perspectives through an image generation model is provided for an exemplary embodiment of the present application. In an optional embodiment, in the aforementioned step S201, 2D texture maps of multiple perspectives are generated through an image generation model according to the description information of the surface texture of the 3D model, which can be specifically implemented in the following manner:

[0104] Through the image generation model, according to the description information of the surface texture of the 3D model, the 2D texture noise features of the configured multiple perspectives are denoised and mapped to the 3D space to obtain multiple local 3D texture features of the 3D model; the multiple local 3D texture features are aggregated and rasterized to the 2D space to obtain 2D texture features of multiple perspectives; the 2D texture features of multiple perspectives are decoded to obtain 2D texture maps of multiple perspectives.

[0105] The following is an illustrative description of the image generation model using the SD model based on the input condition as an example. Figure 6 As shown, in the aforementioned step S201, 2D texture maps of multiple viewing angles are generated by an image generation model according to the description information of the surface texture of the 3D model, which can be specifically implemented by the following steps:

[0106] Step S601: De-noise the 2D texture noise features of the configured multiple viewing angles according to the description information of the surface texture of the 3D model through the image generation model to obtain the de-noised 2D features of the multiple viewing angles.

[0107] This step can be implemented by using the method of iteratively denoising the 2D texture noise features of multiple perspectives through the SD model in any of the aforementioned embodiments to obtain the denoised 2D features of multiple perspectives. Please refer to the aforementioned embodiments for details, which will not be repeated here.

[0108] Step S602 : Based on the inverse function of the mapping function of the grid rendering, the denoised 2D features of each viewing angle are de-rasterized to obtain the local 3D texture features of the 3D model at each viewing angle.

[0109] In this embodiment, the 3D model is represented by a 3D mesh, and the local 3D texture features of the 3D model at each viewing angle include 3D texture features of a plurality of mesh vertices in the 3D mesh.

[0110] In this step, for the denoised 2D features at any viewing angle, the denoised 2D features at that viewing angle are de-rasterized based on the inverse function of the mapping function of the grid rendering to obtain the local 3D texture features of the 3D model at the corresponding viewing angle.

[0111] Specifically, for the denoised 2D features at any viewing angle, the barycentric coordinates of the feature points in the denoised 2D features at that viewing angle projected onto the triangular facets in the 3D mesh are determined based on the inverse function of the mapping function of the mesh rendering; based on the barycentric coordinates of the feature points projected onto the triangular facets in the 3D mesh, the features of the feature points associated with the same mesh vertex are aggregated to obtain the 3D texture features of the mesh vertex at that viewing angle. The 3D texture features of multiple mesh vertices in the 3D mesh at that viewing angle constitute the local 3D texture features of the 3D model at that viewing angle.

[0112] The feature point associated with the mesh vertex refers to the feature point projected from the denoised 2D feature of the current view to the triangular face containing the mesh vertex. The centroid of the feature point refers to the projection point of the feature point on the triangular face.

[0113] For example, n represents any viewing angle, and X represents n,i Represents the 2D feature at any feature point i on the denoised 2D feature of view n. represents the 3D texture feature of any mesh vertex j in the 3D mesh under the viewing angle n. The inverse function of the mapping function based on mesh rendering can be expressed as the following formula (1):

[0114]

[0115] Where h and w represent the width and height of the denoised 2D feature (feature map) of view n, respectively. n,i Represents the centroid coordinates of the feature point i projected to the triangle where the mesh vertex j is located (the coordinates corresponding to the mesh vertex j). τ b is a configuration hyperparameter, which can be set to 2.0. It can be configured according to the actual application scenario. ψ(,) represents exponential operation.

[0116] Through the aforementioned steps S601-S602, the image generation model is used to denoise the 2D texture noise features of multiple configured perspectives according to the description information of the surface texture of the 3D model, and then map them to the 3D space to obtain multiple local 3D texture features of the 3D model.

[0117] In this embodiment, based on the denoised 2D features of a single viewing angle, the 3D texture features of the mesh vertices at the viewing angle are determined, and the aggregation weights of the 3D texture features of the mesh vertices at the viewing angle are determined.

[0118] Specifically, for the feature points in the denoised 2D features of any view, the view weight of the feature point associated with the mesh vertex is determined according to the cosine similarity between the normal vector of the mesh vertex and the direction of the view. The cosine similarity between the normal vector of mesh vertex j and the direction of view n is used as the view weight of each feature point i associated with mesh vertex j (which can be expressed as S n,i ).

[0119] The distance weight of the feature point associated with the mesh vertex is determined according to the distance between the feature point associated with the mesh vertex and the triangle patch to which the feature point is projected.

[0120] Exemplarily, the distance weight of feature point i associated with mesh vertex j can be determined as follows: n,i =1-d i / Z far , where d i Represents the distance between mesh vertex j and the associated feature point i, Z far Indicates the upper bound of the entire scene.

[0121] Furthermore, the aggregation weight of the 3D texture feature of the mesh vertex at the viewing angle is determined according to the barycentric coordinates, the viewing angle weight and the distance weight of the feature point associated with the mesh vertex.

[0122] For example, the barycentric coordinate diagram B can be derived from the mapping function of the grid rendering. n∈N , where N represents the number of viewing angles and n represents any viewing angle. The barycentric coordinates of the feature points associated with any mesh vertex constitute the barycentric coordinate graph B n∈N , barycentric coordinate diagram B n∈NThe dimension is h×w×3, including the centroid coordinates of h×w feature points, where the three channels are respectively The following is based on any view angle n∈N and any mesh vertex j, and the following formula (2) is used to define the aggregation weight W of the 3D texture feature of mesh vertex j under view angle n: n,j :

[0123]

[0124] Among them, S n,i Denotes the view weight of feature point i associated with mesh vertex j. n,i Represents the distance weight of feature point i associated with mesh vertex j. n,i Represents the centroid coordinates of feature point i associated with mesh vertex j (the coordinates corresponding to mesh vertex j). τ b and τ d are all configured hyperparameters, τ b The value can be 2.0, τ d The value can be 4.0, which can be configured according to the actual application scenario. ψ(,) represents exponential operation,

[0125] In this embodiment, increasing the distance weight helps to select more consistent pixels in the 3D texture feature aggregation process under multiple viewing angles, thereby improving the consistency of multiple viewing angles.

[0126] Step S603: Aggregate the local 3D texture features of the 3D model at each viewing angle to obtain the complete 3D texture features of the 3D model.

[0127] In this step, according to the aggregation weight of the 3D texture features of the mesh vertex at each viewing angle, the 3D texture features of the same mesh vertex at multiple viewing angles are aggregated to obtain the 3D texture features of the mesh vertex in the 3D model. The 3D texture features of all mesh vertices constitute the complete 3D texture features of the 3D model.

[0128] Optionally, the following formula (3) can be used to aggregate the 3D texture features of the same mesh vertex j under multiple perspectives n∈N to obtain the 3D texture features of the mesh vertex j in the 3D model:

[0129]

[0130] in, W represents the 3D texture feature of any mesh vertex j in the aggregated 3D model. n,j represents the aggregation weight of the 3D texture feature of mesh vertex j at view n. ψ(,) represents the exponential operation, τ w is the configured hyperparameter, τw The value can be 3.0, which can be configured according to the actual application scenario.

[0131] Step S604: rasterize the complete 3D texture features based on the mapping function of the grid rendering to obtain 2D texture features of multiple viewing angles.

[0132] After obtaining the complete 3D texture features of the 3D model by aggregating the local 3D texture features of the 3D model at each viewing angle, the complete 3D texture features of the 3D model are rasterized (grid rendering) based on each viewing angle to obtain the 2D texture features of each viewing angle.

[0133] Through the aforementioned steps S603-S604, multiple local 3D texture features are aggregated and then rasterized into a 2D space to obtain 2D texture features of multiple viewing angles.

[0134] In this embodiment, in at least one iteration of denoising the 2D texture noise features of multiple perspectives according to the description information of the surface texture of the 3D model through the SD model, the initial 2D texture noise features of multiple perspectives are denoised and mapped to the 3D space through steps S601-S602 to obtain multiple local 3D texture features of this iteration; further, through steps S603-S604, the multiple local 3D texture features of this iteration are aggregated (multi-perspective aggregation) and rasterized to the 2D space to obtain the 2D texture features of multiple perspectives after this iteration (using Indicates, including ). Since the obtained 2D texture features of multiple views are rendered from the 3D texture feature mesh, they naturally achieve better view consistency.

[0135] Then, noise is added to the 2D texture features of multiple perspectives after this iteration as the initial 2D texture noise features of the next iteration, and the next iteration is entered. After the iteration is completed, the 2D texture features of multiple perspectives obtained from the last iteration are used as the final 2D texture features of multiple perspectives.

[0136] It should be noted that, in the denoising process of multiple iterations (all time steps) of the SD model, some iterations (partial time steps) can be sampled and denoised by using the multi-view 2D texture feature aggregation and rasterization method of steps S601-S604, and other iterations can use the existing denoising process; or, in the denoising process of all iterations (all time steps) of the SD model, the multi-view aggregation and rasterization method of steps S601-S604 can be used for denoising. The specific configuration and adjustment can be made by relevant technical personnel according to actual application requirements, and no specific limitation is made here.

[0137] In the SD model, controlling the variance of each denoising iteration (time step) through the noise strategy is crucial to maintain stability and produce high-quality images. and unit constraints Observe that the mesh rendering (and rasterization) process can be represented as a convex combination. In addition, the variance Var(·) represents the expected value of a square function and is a convex function. Therefore, the variance satisfies the basic formula and conditions of convex functions and convex combinations of Jensen's inequality. Jensen's inequality states that if is a convex function, and z i ∈N z is a point in the interval Z, where N z is the number of sampling points, then for any condition satisfying The non-negative weight λ i , the following inequality (4) holds:

[0138]

[0139] Therefore, for a set of random variables x i and satisfy Any non-negative weight λ i , we can get the following inequality (5):

[0140]

[0141] From this we can see that the variance of a convex value combination will not be greater than the variance of a convex combination.

[0142] It can be concluded that the 3D texture features of the 3D model obtained by multi-view aggregation 2D texture features obtained by rasterization (mesh rendering) The variance is greater than the 3D texture features of the 3D model before rasterization. The variance of is smaller, that is, the following inequality (6) exists:

[0143]

[0144] in, Represents the three coordinate components of the barycentric coordinates, and u∈[1,2,3] represents the subscripts of the three different components of the barycentric coordinates.

[0145] In summary, through analysis, it can be found that the process of rasterization (grid rendering) during multiple iterative denoising will lead to a reduction in variance during the denoising process, resulting in a transitionally smooth result.

[0146] In an optional embodiment, multiple local 3D texture features are aggregated and rasterized into a 2D space to obtain 2D texture features of multiple perspectives, and then the variances of the 2D texture features of the multiple perspectives are aligned. The variances of the rasterized 2D texture features are aligned with the variances of the 3D texture features of the 3D grid. This allows the synthesis of a real texture with more high-frequency details, thereby solving the problem of variance reduction caused by rasterization (grid rendering) during the denoising process, and helps to generate high-fidelity texture maps.

[0147] Specifically, for any viewing angle, the variance alignment of the 2D texture features of the corresponding viewing angle is performed according to the 2D texture features of the viewing angle, the mean and standard deviation of the 2D texture features of the viewing angle, and the mean and standard deviation of the complete 3D texture features of the 3D model.

[0148] For example, the following formula (7) may be used as a variance alignment formula to perform variance alignment:

[0149]

[0150] in, Represents the 2D texture features after variance alignment. Represents the complete 3D texture features of the 3D model obtained by multi-view aggregation. Indicates the complete 3D texture features of the 3D model 2D texture features obtained by rasterization (grid rendering), including 2D texture features of multiple perspectives n∈N μ() represents the average operation, δ represents the standard deviation, δ 2D Representing 2D texture features The standard deviation of .

[0151] use Indicates that the 2D texture feature The 3D texture features of the 3D model obtained by de-rasterization to 3D space. 3D express The standard deviation of in, express The variance of is determined by the following formula (8):

[0152]

[0153] Among them, Cov(,) represents the covariance function, and the meanings of the remaining characters remain the same as above.

[0154] For example, Figure 7 FIG. 1 is a schematic diagram of an iterative denoising process provided in this embodiment. Figure 7 As shown, the left side of the dotted line is Figure 4Consistently, 2D-3D joint denoising is performed through the 2D self-attention module and the 3D self-attention module (prediction noise ∈ t ), for details, refer to the relevant contents of the above-mentioned embodiment. Using the 2D texture noise features X of the initial multiple perspectives of this iteration t , the 2D texture noise features X of the initial multiple perspectives of this iteration t The predicted noise features ∈ t By removing the noise from multiple perspectives after this iteration, we can obtain the denoised 2D features (using X 0|t The right side of the dotted line is the process of multi-view aggregation and rasterization. The multi-view aggregation process includes: 0|t De-rasterization obtains the local 3D texture features of the 3D model at each viewing angle, and aggregates the local 3D texture features of the 3D model at each viewing angle to obtain the 3D texture features of the 3D model (using Furthermore, the 3D texture features of the 3D model Rasterization (grid rendering) can obtain 2D texture features from multiple perspectives (using By multi-view aggregation and rasterization, the consistency of the 2D texture features of multiple viewpoints can be improved. After de-rasterization and aggregation, the 3D texture features with multi-view consistency of the 3D model can be obtained (using Represents), 2D texture features from multiple perspectives The variance of and the 3D texture features of the 3D model with multi-view consistency The variance is aligned to achieve 2D texture features from multiple perspectives Variance correction. Add new noise (indicated by ∈) to the 2D texture features of multiple perspectives after variance correction as the initial 2D texture noise features X of multiple perspectives for the next iteration t-1 , and enter the next iterative denoising process.

[0155] Step S605: Decode the 2D texture features of multiple viewing angles to obtain 2D texture maps of multiple viewing angles.

[0156] After the iterative denoising is completed to obtain the final 2D texture features of multiple perspectives, the 2D texture features of each perspective are decoded by a decoder to obtain a 2D texture map of each perspective.

[0157] The decoder may be implemented using a VAE (Variational Auto-Encoder) decoder or other decoders with similar decoding functions, which are not specifically limited here.

[0158] The method of this embodiment, in at least one iteration of the SD model denoising the 2D texture noise features of multiple perspectives, maps the initial 2D texture noise features of multiple perspectives to the 3D space after denoising, and obtains multiple local 3D texture features of this iteration; aggregates the multiple local 3D texture features of this iteration and rasterizes them to the 2D space, and obtains the 2D texture features of multiple perspectives after this iteration. Through the aggregation and rasterization of the 2D texture features of multiple perspectives, the 2D texture features of multiple perspectives obtained are obtained from the 3D texture feature grid rendering (rasterization), and they naturally achieve better perspective consistency. Subsequently, noise is added to the 2D texture features of multiple perspectives after this iteration, as the initial 2D texture noise features of the next iteration, and the next iteration is entered. After the iteration is completed, the 2D texture features of multiple perspectives obtained in the last iteration are used as the 2D texture features of multiple perspectives finally obtained, which can improve the consistency of the 2D texture features of multiple perspectives obtained.

[0159] In an optional embodiment, the 3D model is represented by a 3D mesh, and the local 3D texture map of the 3D model at each viewing angle includes a 3D texture map of a plurality of mesh vertices in the 3D mesh.

[0160] In the aforementioned step S202, the 2D texture maps of each viewing angle are mapped to the 3D space to obtain the local 3D texture maps of the 3D model at each viewing angle, which is specifically achieved by the following method:

[0161] Based on the inverse function of the mapping function of mesh rendering, the 2D texture maps of each viewing angle are de-rasterized to obtain the local 3D texture maps of the 3D model at each viewing angle.

[0162] Specifically, for a 2D texture map at any viewing angle, the barycentric coordinates of the pixel points in the 2D texture map at that viewing angle projected onto the triangular facets in the 3D mesh are determined based on the inverse function of the mapping function of the mesh rendering; according to the barycentric coordinates of the pixel points projected onto the triangular facets in the 3D mesh, the pixel values ​​of the pixel points associated with the same mesh vertex are aggregated to obtain the 3D texture map of the mesh vertex at that viewing angle. The 3D texture maps of multiple mesh vertices in the 3D mesh at that viewing angle constitute the local 3D texture map of the 3D model at that viewing angle.

[0163] The pixel point associated with the mesh vertex refers to the pixel point in the 2D texture map of the current viewing angle that is projected onto the triangular face containing the mesh vertex. The centroid of the pixel point refers to the projection point of the pixel point on the triangular face.

[0164] For example, n represents any viewing angle, and I represents n,i Represents the 2D texture at any pixel i on the 2D texture map of view angle n. represents the 3D texture of any mesh vertex j in the 3D mesh under the viewing angle n. The inverse function of the mapping function based on mesh rendering can be expressed as the following formula (9):

[0165]

[0166] Among them, h ′ and w ′ Respectively represent the width and height of the 2D texture map at viewing angle n. n,i Represents the centroid coordinates of the triangle where the pixel point i is projected to the mesh vertex j (the coordinates corresponding to the mesh vertex j). τ′ b is a configuration hyperparameter, which can be set to 2.0. It can be configured according to the actual application scenario. ψ(,) represents exponential operation.

[0167] Further, based on the 2D texture map of a single viewing angle, a 3D texture map of the mesh vertices at the viewing angle is determined, and an aggregation weight of the 3D texture map of the mesh vertices at the viewing angle is determined.

[0168] Specifically, for the pixel points in the 2D texture map at any viewing angle, the viewing weight of the pixel point associated with the mesh vertex is determined according to the cosine similarity between the normal vector of the mesh vertex and the direction of the viewing angle. The cosine similarity between the normal vector of mesh vertex j and the direction of viewing angle n is used as the viewing weight of each pixel point i associated with mesh vertex j (which can be expressed as S′ n,i ).

[0169] The distance weight of the pixel point associated with the mesh vertex is determined according to the distance between the pixel point associated with the mesh vertex and the triangle patch to which the pixel point is projected.

[0170] Exemplarily, the distance weight of pixel point i associated with mesh vertex j can be determined as follows: n,i =1-d′ i / Z far , where d′ i Represents the distance between mesh vertex j and the associated pixel i, Z far Indicates the upper bound of the entire scene.

[0171] For example, the barycentric coordinate diagram B′ can be derived from the mapping function of the grid rendering. n∈N , where N represents the number of viewing angles and n represents any viewing angle. The barycentric coordinates of the pixel points associated with any mesh vertex constitute the barycentric coordinate graph B′ n∈N , barycentric coordinate diagram B′ n∈N The dimension is h′×w′×3, including the centroid coordinates of h′×w′ pixels, where the three channels are respectively The following is based on any view angle n∈N and any mesh vertex j, and the following formula (10) is used to define the aggregation weight W′ of the 3D texture of mesh vertex j under view angle n: n,j :

[0172]

[0173] Among them, S′ n,i Represents the viewing weight of pixel i associated with mesh vertex j. D′ n,i Represents the distance weight of pixel i associated with mesh vertex j. n,i Represents the centroid coordinates of the pixel point i associated with the mesh vertex j (the coordinates corresponding to the mesh vertex j). τ′ b and τ′ d are all configured hyperparameters, τ′ b The value can be 2.0, τ′ d The value can be 4.0, which can be configured according to the actual application scenario. ψ(,) represents exponential operation,

[0174] Furthermore, the aggregation weight of the 3D texture map of the mesh vertex at the corresponding viewing angle is determined according to the centroid coordinates, the viewing angle weight and the distance weight of the pixel point associated with the mesh vertex.

[0175] In this embodiment, increasing the distance weight helps to select more consistent pixels in the 3D texture map aggregation process under multiple viewing angles, thereby improving the consistency of multiple viewing angles.

[0176] Furthermore, in the aforementioned step S203, according to the aggregation weight of the 3D texture map of the mesh vertex at each viewing angle, the 3D textures of the same mesh vertex at multiple viewing angles are aggregated to obtain the 3D texture of the mesh vertex in the 3D model. The 3D textures of all mesh vertices constitute a complete 3D texture map of the 3D model.

[0177] Optionally, the following formula (11) can be used to aggregate the 3D textures of the same mesh vertex j under multiple viewing angles n∈N to obtain the 3D texture of the mesh vertex j in the 3D model:

[0178]

[0179] in, Represents the 3D texture of any mesh vertex j in the aggregated 3D model. n,j represents the aggregation weight of the 3D texture map of mesh vertex j at view n. ψ(,) represents the exponential operation, τ′ w is the configured hyperparameter, τ′ wThe value can be 3.0, which can be configured according to the actual application scenario.

[0180] The method of this embodiment maps 2D texture maps of multiple perspectives to 3D space to obtain local 3D texture maps of the 3D model at each perspective; aggregates the local 3D texture maps of the 3D model at each perspective to obtain a complete 3D texture map of the 3D model, thereby improving the multi-perspective consistency and high fidelity of the 3D texture map.

[0181] In an optional embodiment, the 3D model is represented by a 3D mesh, and the local 3D texture map of the 3D model at any viewing angle includes the 3D texture map of a plurality of mesh vertices in the 3D mesh at a corresponding viewing angle.

[0182] Considering the inherent differences in resolution and number of channels between the latent domain and the image domain, consistent latent features cannot guarantee the consistency of the generated image. In this embodiment, a texture restoration step is introduced to further alleviate the inconsistency of pixel values ​​at different viewing angles in the image domain.

[0183] Figure 8 The flowchart of texture repairing of 3D texture map obtained by multi-view aggregation provided by the embodiment of the present application. Figure 8 As shown, after the local 3D texture maps of the 3D model under multiple (N) perspectives are aggregated to obtain the complete 3D texture map of the 3D model, the specific steps of texture repairing the 3D texture map of the 3D model are as follows:

[0184] Step S801: Calculate the variance of mesh vertices in the 3D model according to the local 3D texture map of the 3D model at each viewing angle.

[0185] In this embodiment, when performing texture restoration, N images may be sampled from N viewing angles. f perspectives, based on the 3D model at perspective n∈N f The local 3D texture map below is used for texture repair.

[0186] For example, assume that N=8, including 8 viewing angles of 0°, 45°, 90°, 135°, ... In this embodiment, 4 viewing angles of 0°, 90°, 180°, and 270° can be sampled for texture restoration, that is, N f =4. Here, for the sampling angle N f The number of viewing angles and sampling method are not specifically limited.

[0187] In this step, according to the local 3D texture map of the 3D model under the sampled multiple viewing angles, the variance of the 3D texture of the same mesh vertex in the 3D model under the multiple viewing angles is calculated as the variance of the mesh vertex.

[0188] For example, the variance of any mesh vertex j can be calculated by the following formula (12):

[0189]

[0190] Among them, Var j N represents the variance of mesh vertex j. f Indicates the number of sampled views, n f Represents any viewing angle of the sample. Indicates the viewing angle n f 2D texture map. Indicates the 3D model at view angle n f The local 3D texture map under the f 2D texture map De-rasterized. Indicates that the 3D model is at the viewing angle n f Local 3D texture map under The 3D texture of vertex j in the mesh. Indicates the 3D model in the sampling N f Local 3D texture map at each viewing angle The average value of the 3D texture of mesh vertex j in .

[0191] Step S802: Generate a 3D mask of the 3D model according to the variance and variance threshold of the mesh vertices in the 3D model, wherein the 3D mask includes a to-be-repaired area and a retained area, wherein the variance of the mesh vertices in the to-be-repaired area is greater than the variance threshold, and the variance of the mesh vertices in the retained area is less than or equal to the variance threshold.

[0192] After determining the variance of each mesh vertex in the 3D model, since the variance of each mesh vertex is obtained from the sampled N f The texture maps are accumulated from multiple viewing angles, and a pre-configured variance threshold can be used to identify mesh vertices that display inconsistencies.

[0193] In this step, the 3D mask of the 3D model is determined by the indicator function shown in formula (13):

[0194]

[0195] Among them, M j represents the value at mesh vertex j in the 3D mask, M j =1 means repair is needed, M j =0 means no need to repair. The 3D mask of the 3D model can be used to determine the repair area to be repaired and the reserved area that does not need to be repaired in the 3D model. λ represents a pre-configured variance threshold, which can be set to 0.005. It can be adjusted and configured according to actual needs and is not specifically limited here.

[0196] Step S803: Perform texture repair on the to-be-repaired area of ​​the complete 3D texture map of the 3D model according to the 3D mask of the 3D model, so as to obtain a repaired 3D texture map of the 3D model.

[0197] In this step, texture repair is performed on the portion of the complete 3D texture map of the 3D model that falls within the area to be repaired according to the 3D mask of the 3D model, so as to obtain a repaired 3D texture map. Through texture repair, the blurred area in the 3D texture map can be patched to obtain a 3D texture map with high visual consistency and quality, so that the rendered 3D object can maintain a realistic and coherent appearance when viewed from multiple perspectives.

[0198] In an optional embodiment, the specific process of texture repair in this step is as follows:

[0199] The complete 3D texture map of the 3D model is rasterized into 2D texture maps of multiple viewing angles, and the depth map of the complete 3D texture map of the 3D model at each viewing angle is rendered. The 3D mask is rasterized into 2D masks of multiple viewing angles, and the 2D mask includes a to-be-repaired area and a reserved area. The to-be-repaired area is the pixels that need to be repaired, and the reserved area is the pixels that do not need to be repaired.

[0200] Furthermore, according to the 2D mask of any viewing angle and the depth map of the 3D model at the corresponding viewing angle, the 2D texture map of the corresponding viewing angle is repaired to obtain the repaired 2D texture map of the corresponding viewing angle.

[0201] When repairing a 2D texture map at any viewing angle, any image repair algorithm / model that can repair a 2D image according to a given 2D mask and depth map can be used to repair the 2D texture map according to the 2D mask and depth map at the same viewing angle to obtain a repaired 2D texture map at the corresponding viewing angle. The image repair algorithm / model can be an SD model or other models or methods with similar image repair functions, which are not specifically limited in this embodiment.

[0202] Furthermore, the repaired 2D texture maps of multiple perspectives are mapped to the 3D space and aggregated to obtain the repaired 3D texture map of the 3D model. The specific implementation principle is referred to the relevant contents of steps S202-S203 in the above embodiment, which will not be repeated here.

[0203] For example, Fig. 9 The example diagram of texture restoration provided in this embodiment is as follows: Fig. 9 As shown, Fig. 9(a) is an example of a complete 3D texture map of a 3D model obtained by aggregation, (b) is the 3D mask corresponding to the 3D texture map shown in (a), and (c) is the depth map corresponding to the 3D texture map shown in (a). After texture restoration based on the 3D mask and depth map, the restored 3D texture map shown in (d) is obtained. By comparing the parts marked by the rectangular boxes in (a) and (d), it can be found that the 3D texture map has better visual consistency and higher quality after texture restoration.

[0204] In addition, when performing texture restoration in this step, image inpainting techniques such as partial differential equations, global variation methods, curvature-based diffusion models, and texture synthesis methods are also used, which are not specifically limited in this embodiment.

[0205] The method of this embodiment aggregates the local 3D texture maps of the 3D model at multiple perspectives to obtain the complete 3D texture map of the 3D model, then constructs a 3D mask of the 3D model by calculating the variance of the mesh vertices in the 3D model and taking the fuzzy area with a variance greater than a variance threshold as the area to be repaired. The 3D texture map of the 3D model is texture repaired according to the 3D mask of the 3D model, and the fuzzy area in the 3D texture map can be repaired, thereby improving the visual consistency and quality of the 3D texture map, so that the rendered 3D object can maintain a realistic and coherent appearance when viewed from multiple perspectives.

[0206] Fig.10 The overall framework diagram of texture mapping processing provided by an exemplary embodiment of the present application. Fig.10 As shown, the 2D texture noise features X of multiple viewpoints configured by the SD model t Perform 2D-3D joint denoising to obtain denoised 2D features X from multiple perspectives 0|t ; Furthermore, through the multi-view aggregation and rasterization method, the denoised 2D features X 0|t De-rasterize the local 3D texture features of the 3D model at each viewing angle, aggregate the local 3D texture features of the 3D model at each viewing angle, and obtain the 3D texture features of the 3D model Furthermore, the 3D texture features of the 3D model Rasterization (grid rendering) to obtain 2D texture features from multiple perspectives Through multi-view aggregation and rasterization processing, the consistency of the 2D texture features of multiple views can be improved. After variance alignment, new noise is added as the 2D texture noise feature X for the next iterative denoising t-1After multiple rounds of iterative denoising, the 2D texture features of multiple perspectives are finally obtained. Furthermore, the 2D texture features of multiple perspectives are decoded by the decoder to obtain 2D texture maps of multiple perspectives. Furthermore, 2D texture mapping for multiple viewing angles Perform multi-view aggregation to obtain a complete 3D texture map of the 3D model Furthermore, a complete 3D texture map of the 3D model Perform texture repair to obtain the repaired 3D texture map of the 3D model

[0207] Fig.11 This is a flowchart of a texture mapping processing method provided by another exemplary embodiment of the present application. The execution subject of this embodiment is the server in the above embodiment. Fig.11 As shown, the specific steps of this method are as follows:

[0208] Step S1101: A request for obtaining a texture map of a 3D model sent by a receiving terminal device is received to obtain the 3D model for which the texture map is to be generated and description information of the surface texture of the 3D model.

[0209] In this embodiment, the terminal device may be an electronic device running downstream application software / system, and may specifically be a hardware device with network communication function, computing function and information display function, including but not limited to smart phones, tablet computers, desktop computers, local servers, cloud servers, etc. For example, the terminal device may be an electronic device running various software applications / systems such as game design, film and television animation production, industrial design, architectural design, interior design, product design, etc. The downstream application software / system may generate a 3D texture map of the 3D model that conforms to the description information based on a given 3D model and description information, render the 3D model based on the 3D texture map, realize game character skinning, animation character skinning, 3D object material update, etc., and further complete subsequent processing logic such as game design, film and television animation production, and 3D object dynamic display.

[0210] When a 3D texture map that conforms to the description information needs to be generated based on a given 3D model and description information, the terminal device sends a request to obtain the texture map of the 3D model to the server, which includes the 3D model for which the texture map is to be generated and the description information of the surface texture of the 3D model.

[0211] The server receives a request for obtaining a texture map of a 3D model sent by a terminal device, and obtains description information of the 3D model for which the texture map is to be generated and the surface texture of the 3D model from the request.

[0212] The description information of the surface texture of the 3D model is used to describe the target texture map that the customer wants to generate for the 3D model. The description information includes but is not limited to text, images, videos, audio, etc., and may include at least one type of description information among text, images, videos, and audio.

[0213] The above description information is mainly used to describe the final presentation form of the 3D model, for example, the presentation style, texture, feel, state, color, logo, etc. of the 3D model. For example, the description information can be a text description, such as "brown bear", "baseball glove", etc., or a depth map of the 3D model.

[0214] Step S1102: Generate 2D texture maps of multiple viewing angles using an image generation model according to the description information.

[0215] Step S1103: Map the 2D texture map of each viewing angle to the 3D space to obtain a local 3D texture map of the 3D model at each viewing angle.

[0216] Step S1104: aggregate the local 3D texture maps of the 3D model at each viewing angle to obtain a complete 3D texture map of the 3D model.

[0217] In this embodiment, the implementation principle of generating a complete 3D texture map of a 3D model through steps S1102-S1104 can be found in the above embodiment and will not be repeated here.

[0218] Step S1105: Return the complete 3D texture map of the 3D model to the terminal device.

[0219] After generating the complete 3D texture map of the 3D model, the server returns the complete 3D texture map of the 3D model to the terminal device.

[0220] The end-side device renders the 3D model according to the complete 3D texture map of the 3D model, realizes game character skinning, animation character skinning, 3D object material update, etc., and further completes subsequent processing logic such as game design, film and television animation production, and 3D object dynamic display.

[0221] In the method of this embodiment, a server receives a request for obtaining a texture map of a 3D model sent by a terminal device, obtains the 3D model for which the texture map is to be generated and description information of the surface texture of the 3D model, generates 2D texture maps of multiple viewing angles through an image generation model according to the description information of the surface texture of the 3D model; maps the 2D texture map of each viewing angle to a 3D space to obtain a local 3D texture map of the 3D model at each viewing angle; aggregates the local 3D texture maps of the 3D model at each viewing angle to obtain a complete 3D texture map of the 3D model, and only needs to provide a 3D model (a white model) and description information of the surface texture of the 3D model, and no additional data collection and annotation process is required to achieve texture mapping of the 3D model, thereby improving the efficiency of texture mapping of the 3D model.

[0222] Fig.12 A schematic diagram of the structure of a server provided in an embodiment of the present application. Fig.12 As shown, the server includes: a memory 1201 and a processor 1202. The memory 1201 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1202 is connected to the memory 1201 in communication and is used to execute the computer-executable instructions stored in the memory 1201 to implement the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are similar and will not be repeated here.

[0223] Optional, such as Fig.12 As shown, the server also includes: a firewall 1203, a load balancer 1204, a communication component 1205, a power supply component 1206 and other components. Fig.12 Only some components are shown schematically, which does not mean that the server only includes Fig.12 Components shown. Fig.12 In the description, the server is only taken as a cloud server deployed in the cloud as an example for exemplary description. The server can also be deployed locally, and this embodiment is not specifically limited here.

[0224] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method of any of the aforementioned embodiments is implemented. The specific functions and technical effects that can be achieved are not repeated here.

[0225] The present application also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0226] The embodiment of the present application provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the server in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.

[0227] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0228] It should be understood that the above-mentioned processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in at least one processor. The memory may include a high-speed random access memory (RAM), and may also include non-volatile storage, such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0229] The above-mentioned memory may be an object storage service (OSS). The above-mentioned memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0230] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / Long Term Evolution (LongTerm Evolution, referred to as LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (Radio Frequency Identification, referred to as RFID) technology, infrared technology, ultra-wideband (Ultra Wide Band, referred to as UWB) technology, Bluetooth technology and other technologies.

[0231] The above-mentioned power supply component provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power for the device where the power supply component is located. The above-mentioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0232] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in a dedicated integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0233] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0234] The order of the above-mentioned embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above-mentioned embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed or executed in parallel in the order in which they appear in this article, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0235] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0236] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0237] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A texture mapping processing method, characterized in that: include: The image generation model is used to generate a plurality of local 3D texture features of the 3D model by denoising the 2D texture noise features of the configured multiple viewing angles according to the description information of the surface texture of the 3D model, and then mapping the 2D texture noise features to the 3D space to obtain multiple local 3D texture features of the 3D model; Aggregating the multiple local 3D texture features and rasterizing them into a 2D space to obtain 2D texture features of multiple viewing angles; Decoding the 2D texture features of the multiple viewing angles to obtain 2D texture maps of the multiple viewing angles; Mapping the 2D texture map of each viewing angle to the 3D space to obtain a local 3D texture map of the 3D model at each viewing angle; The local 3D texture maps of the 3D model at each viewing angle are aggregated to obtain a complete 3D texture map of the 3D model.

2. The method according to claim 1, characterized in that Also includes: In at least one iteration of denoising the 2D texture noise features of multiple perspectives according to the description information of the surface texture of the 3D model through the image generation model, the initial 2D texture noise features of the multiple perspectives are denoised and mapped to the 3D space to obtain multiple local 3D texture features of this iteration; Multiple local 3D texture features of this iteration are aggregated and rasterized into 2D space to obtain 2D texture features of multiple perspectives after this iteration. The 2D texture features of multiple perspectives after this iteration will be used as the initial 2D texture noise features of the next iteration.

3. The method according to claim 1, characterized in that: The image generation model removes noise from the 2D texture noise features of multiple configured perspectives according to the description information of the surface texture of the 3D model, and then maps them to the 3D space to obtain multiple local 3D texture features of the 3D model, including: Using the image generation model, according to the description information of the surface texture of the 3D model, denoise the 2D texture noise features of the configured multiple viewing angles to obtain the denoised 2D features of the multiple viewing angles; Based on the inverse function of the mapping function of the grid rendering, the denoised 2D features of each viewing angle are de-rasterized to obtain the local 3D texture features of the 3D model at each viewing angle.

4. The method according to claim 3, characterized in that The inverse function of the mapping function based on grid rendering de-rasterizes the denoised 2D features of each of the viewing angles to obtain local 3D texture features of the 3D model at each viewing angle, including: The 3D model is represented by a 3D mesh, and the local 3D texture features include 3D texture features of a plurality of mesh vertices in the 3D mesh. For the denoised 2D features of any viewing angle, determine the centroid coordinates of the feature points in the denoised 2D features of the viewing angle projected onto the triangular facets in the 3D mesh based on the inverse function of the mapping function of the mesh rendering; Aggregating features of feature points associated with the same mesh vertex according to the barycentric coordinates projected onto the triangular facets in the 3D mesh by the feature points to obtain 3D texture features of the mesh vertex at the viewing angle; The 3D texture features of a plurality of mesh vertices in the 3D mesh at the viewing angle constitute the local 3D texture features of the 3D model at the viewing angle.

5. The method according to claim 4, characterized in that Also includes: For a feature point in the denoised 2D feature of any viewing angle, determine a viewing angle weight of the feature point associated with the mesh vertex according to a cosine similarity between a normal vector of the mesh vertex and a direction of the viewing angle; Determine the distance weight of the feature point associated with the mesh vertex according to the distance between the feature point associated with the mesh vertex and the triangle patch to which the feature point is projected; The aggregation weight of the 3D texture feature of the mesh vertex at the viewing angle is determined according to the centroid coordinates, the viewing angle weight and the distance weight of the feature point associated with the mesh vertex.

6. The method according to claim 5, characterized in that The aggregating the local 3D texture features of the 3D model at each viewing angle to obtain the complete 3D texture features of the 3D model includes: According to the aggregation weights of the 3D texture features of the mesh vertices at each viewing angle, the 3D texture features of the same mesh vertex at multiple viewing angles are aggregated to obtain the complete 3D texture features of the 3D model.

7. The method according to claim 1, characterized in that After the plurality of local 3D texture features are aggregated and rasterized into a 2D space to obtain 2D texture features of multiple perspectives, the method further includes: For any viewing angle, the 2D texture features of the viewing angle are variance aligned according to the 2D texture features of the viewing angle, the mean and standard deviation of the 2D texture features of the viewing angle, and the mean and standard deviation of the complete 3D texture features of the 3D model.

8. The method according to claim 1, characterized in that The method of generating a model through an image and denoising the 2D texture noise features of multiple configured viewing angles according to the description information of the surface texture of the 3D model includes: The 2D texture noise features of multiple perspectives are extracted through the image generation model to obtain the 2D intermediate features of multiple perspectives; the self-attention calculation is performed on the 2D intermediate features of multiple perspectives to obtain the first 2D fusion features of multiple perspectives; The inverse function of the mapping function based on mesh rendering de-rasterizes the 2D intermediate features of multiple views into 3D intermediate features; Divide the outer bounding box of the 3D intermediate feature into multiple volume grids, perform self-attention calculation on the 3D intermediate features in each volume grid, and obtain a 3D fusion feature; rasterizing the 3D fused features into second 2D fused features for each of the viewing angles; The second 2D fusion feature of each viewing angle is fused with the first 2D fusion feature of the corresponding viewing angle to obtain a third 2D fusion feature of each viewing angle; According to the third 2D fusion features of each viewing angle, the denoised 2D features of each viewing angle are determined.

9. The method according to claim 1, characterized in that: After the local 3D texture maps of the 3D model at each viewing angle are aggregated to obtain a complete 3D texture map of the 3D model, the method further includes: The 3D model is represented by a 3D mesh, and the local 3D texture map of the 3D model at any viewing angle includes the 3D texture map of a plurality of mesh vertices in the 3D mesh at the viewing angle, Calculating the variance of mesh vertices in the 3D model according to the local 3D texture map of the 3D model at each viewing angle; Generate a 3D mask of the 3D model according to the variance and variance threshold of the mesh vertices in the 3D model, wherein the 3D mask includes a to-be-repaired area and a reserved area, wherein the variance of the mesh vertices in the to-be-repaired area is greater than the variance threshold, and the variance of the mesh vertices in the reserved area is less than or equal to the variance threshold; According to the 3D mask of the 3D model, texture repair is performed on the to-be-repaired area of ​​the complete 3D texture map of the 3D model to obtain a repaired 3D texture map of the 3D model.

10. The method according to claim 9, characterized in that The step of performing texture repair on a to-be-repaired area of ​​a complete 3D texture map of the 3D model according to the 3D mask of the 3D model to obtain a repaired 3D texture map of the 3D model includes: Rasterizing the complete 3D texture map of the 3D model into 2D texture maps of multiple viewing angles, and rendering depth maps of the complete 3D texture map of the 3D model at the multiple viewing angles; rasterizing the 3D mask into 2D masks of multiple viewing angles; Repairing the 2D texture map of the corresponding viewing angle according to the 2D mask of any viewing angle and the depth map of the 3D model at the corresponding viewing angle to obtain the repaired 2D texture map of the viewing angle; The repaired 2D texture maps of multiple perspectives are mapped to the 3D space and aggregated to obtain the repaired 3D texture map of the 3D model.

11. A texture mapping processing method, characterized in that: include: A request for obtaining a texture map of a 3D model sent by a receiving device on the terminal side is received to obtain the description information of the 3D model for which the texture map is to be generated and the surface texture of the 3D model; Denoising the 2D texture noise features of the configured multiple perspectives according to the description information through an image generation model and mapping them to a 3D space to obtain multiple local 3D texture features of the 3D model; Aggregating the multiple local 3D texture features and rasterizing them into a 2D space to obtain 2D texture features of multiple viewing angles; Decoding the 2D texture features of the multiple viewing angles to obtain 2D texture maps of the multiple viewing angles; Mapping the 2D texture map of each viewing angle to the 3D space to obtain a local 3D texture map of the 3D model at each viewing angle; Aggregating the local 3D texture maps of the 3D model at each viewing angle to obtain a complete 3D texture map of the 3D model; Return the complete 3D texture map of the 3D model to the client device.

12. A server, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the server to execute the method described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images