3D texture generation method based on multi-view pixel spatial diffusion model, electronic equipment and program product

Abstract

The invention provides a 3D texture generation method based on a multi-view pixel space diffusion model, electronic equipment and a program product. The 3D texture generation method based on the multi-view pixel spatial diffusion model comprises the following steps: acquiring geometric condition information and a reference image of a 3D model; splicing a multi-view noise map and the geometric condition information in a channel dimension to obtain a multi-view condition splicing feature; taking the reference image as an additional view angle, and splicing the reference image with the multi-view-angle condition splicing feature in a sequence dimension to obtain a sequence splicing feature; the pixel space multi-view-angle generation model obtains a multi-view-angle RGB image based on sequence splicing features; and projecting the multi-view RGB image to the surface of a 3D model to generate a complete texture. The purposes of high fidelity, pixel-level alignment and seamless 3D texture generation are achieved.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional image technology, and in particular relates to a 3D texture generation method, electronic device and program product based on a multi-view pixel spatial diffusion model. Background Technology

[0002] With the growing demand for 3D content creation (AIGC 3D), how to automatically generate high-quality 3D model textures from a single reference image has become a research hotspot in image reconstruction. Existing main technical approaches include optimization-based methods (such as ScoreDistillation Sampling).

[0003] Existing technologies mostly use VAEs. The encoding and decoding process of VAEs is a lossy compression process, which destroys the pixel-level geometric correspondence, resulting in blurring and splicing marks when the final generated texture is projected back to the 3D model.

[0004] Specifically, existing 3D texture generation technologies face the following problems when generating high-quality, multi-view consistent textures: (1) Loss of detail and blurring: Most existing mainstream methods (such as Hunyuan3D) are based on latent diffusion models, which rely on variational autoencoders (VAEs) to compress high-resolution images into a low-dimensional latent space. This lossy compression process filters out high-frequency texture details (such as text and fine patterns) in the reference image and introduces color deviation when decoding back to pixel space.

[0005] 2) Multi-view inconsistency: Consistency constraints applied in the latent space cannot guarantee precise pixel-level alignment after decoding. Because the VAE decoder decodes each view independently, different views often cannot perfectly overlap after reprojection, leading to texture misalignment. Pixel-level geometric alignment information also suffers from information loss after compression by the VAE encoder.

[0006] 3) Self-occlusion and texture loss of complex geometries: Traditional UV space generation methods are limited by the discontinuity of UV islands, which easily produces seams; while the inpainting method is prone to cumulative errors when dealing with self-occluded areas, resulting in incomplete textures or abrupt style changes.

[0007] In summary, the inventors discovered the following shortcomings of the prior art during the implementation of this embodiment: (1) VAE compression bottleneck: The latent space model sacrifices information accuracy for computational efficiency. The encoding and decoding process of VAE cannot guarantee the precise geometric correspondence between pixels, resulting in a lack of multi-view consistency in the generated results.

[0008] (2) Inaccurate geometric guidance: Existing 3D position coding (such as RoPE) is usually applied only at a coarse level of latent features and cannot achieve pixel-level precise geometric control.

[0009] (3) Incomplete texture: Generating images from a fixed perspective cannot cover all surfaces of complex geometries, and existing UV patching methods are prone to introducing seams at the edges of UV islands. Summary of the Invention

[0010] To address the aforementioned problems in the existing technology, this invention provides a 3D texture generation method, electronic device, and program product based on a multi-view pixel spatial diffusion model.

[0011] In a first aspect, embodiments of this disclosure provide a 3D texture generation method based on a multi-view pixel spatial diffusion model, including: Obtain the geometric condition information and reference image of the 3D model; The multi-view noise map and the geometric condition information are concatenated in the channel dimension to obtain the multi-view condition concatenation feature. The reference image is used as an additional viewpoint and concatenated with the multi-view conditional stitching features in the sequence dimension to obtain the sequence stitching features; A pixel-space multi-view generation model generates multi-view RGB images based on sequence splicing features; The multi-view RGB image is projected onto the surface of the 3D model to generate a complete texture.

[0012] Optionally, the pixel-space multi-view generation model obtains multi-view RGB images based on sequence splicing features, including: The sequence splicing features are extracted with coarse-grained features by the multi-view Transformer module, and 5D Rotational Position Encoding (RoPE) is fused into the attention mechanism of the multi-view Transformer module to ensure that the coarse-grained structure of cross-view images in multi-view RGB images remains consistent. The 5D RoPE is a fusion of 2D image patch index and 3D world space coordinates. The multi-view RGB image is obtained by decoding the features output by the multi-view Transformer module through a 3D position-aware pixel-level decoder; wherein, at the input end of the 3D position-aware pixel-level decoder, the pixel-level dense 3D global position embedding is spliced ​​with the features output by the multi-view Transformer module.

[0013] Optionally, the 3D world space coordinates in the 5D rotational position encoding RoPE are obtained by normalizing and discretizing the 3D surface point coordinates into voxel indices.

[0014] Optionally, the 3D position-aware pixel-level decoder is a lightweight U-Net structure, and utilizes skip connections to transmit high-frequency information of the input.

[0015] Optionally, after projecting the multi-view RGB image onto the surface of the 3D model, if there are uncolored texture missing areas, a multi-view repair process is performed, including: Identify texture-deficient regions based on the 3D model; Based on the areas with missing textures, select a set of repair perspectives; The partially colored texture map and the repair mask are used as conditions and input into the pixel space multi-view repair model to generate an RGB image of the repair view. The repaired RGB image is projected onto the surface of the 3D model to fill in the areas where the texture is missing.

[0016] Optionally, the selection of a set of repair perspectives employs a greedy algorithm to maximize the coverage of the texture-missing areas.

[0017] Optionally, the training of the pixel space multi-view generation model adopts a flow matching framework, and its loss function includes velocity matching loss and multi-view consistency loss. The multi-view consistency loss is achieved by calculating the photometric reprojection error between predicted images from different viewpoints based on geometric projection transformation.

[0018] Optionally, the pixel space multi-view generation model does not rely on variational autoencoder (VAE) for spatial compression and reconstruction.

[0019] Secondly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the 3D texture generation method based on the multi-view pixel spatial diffusion model as described in any of the first aspects.

[0020] Thirdly, embodiments of this disclosure also provide a computer program product, including a computer program / instruction, characterized in that, when the computer program / instruction is executed by a processor, it implements the 3D texture generation method based on the multi-view pixel spatial diffusion model described in any of the first aspects.

[0021] The present invention provides a 3D texture generation method, electronic device, and program product based on a multi-view pixel spatial diffusion model. The 3D texture generation method based on the multi-view pixel spatial diffusion model generates multiple views in a pixel space multi-view generation model by using splicing features obtained based on geometric condition information and reference images. No data compression is performed, thereby eliminating the loss of details and inconsistency of multiple views caused by lossy compression, thus achieving the purpose of high-fidelity, pixel-level alignment, and seamless 3D texture generation. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0023] Figure 1 A flowchart illustrating a 3D texture generation method based on a multi-view pixel spatial diffusion model provided in this embodiment of the disclosure; Figure 2 A schematic block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0024] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0025] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] It should be noted that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice method. Furthermore, this device and / or practice method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0027] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0028] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0029] CCM (Canonical Coordinate Map): refers to a canonical coordinate map that maps points on the surface of a 3D object to a normalized RGB color space, used to represent the spatial coordinates of the geometry.

[0030] RoPE (Rotary Positional Embedding): refers to rotational positional encoding, a relative positional encoding technique widely used in Transformer.

[0031] Rectified Flow: refers to a generative model training method that generates models faster and with a straighter trajectory than the standard diffusion model.

[0032] Latent Space: This refers to the low-dimensional feature space to which high-dimensional raw data (such as RGB image pixels) is mapped by an encoder (such as the Encoder in a VAE). In this space, the data is compressed and represented as abstract feature vectors. Existing techniques often generate data in this space to reduce computational cost, but this process is lossy compression, which usually leads to the loss of high-frequency details and artifacts after decoding.

[0033] Patchfy: refers to dividing a two-dimensional image into fixed-size segments (e.g., ...). The Transformer generates a set of pixels (patches) and flattens each patch into a one-dimensional vector sequence, which serves as the input token for the Transformer.

[0034] Unpachify: refers to the inverse process of Patchify, which rearranges and combines the one-dimensional feature vector sequence processed by the network according to its original spatial position, restoring it to the structure of a two-dimensional image. 5D Rotational Position Encoding (RoPE): This refers to a positional encoding strategy that modulates attention computation in the Transformer by encoding both 2D image coordinates and 3D world coordinates into a rotation angle. Its core objective is to establish a robust attention mechanism driven by geometric relationships at the level of multi-view image patches, thereby achieving coarse-grained structural consistency across viewpoints.

[0035] Pixel-level dense 3D global position embedding: This refers to assigning a high-dimensional vector encoded by the corresponding 3D world coordinates to each pixel in the generated image, and using this complete vector map as a conditional input to the decoding network, thereby forcing geometric consistency between multiple viewpoints at the finest level.

[0036] The pixel-space multi-view generative model is a generative artificial intelligence that does not rely on lossy compressors such as variational autoencoders (VAEs). It directly processes and generates RGB images from multiple perspectives simultaneously in high-resolution pixel space, and utilizes explicit geometric information to ensure that these generated perspectives have pixel-level consistency on the surface of the 3D model.

[0037] like Figure 1 As shown, this embodiment discloses a 3D texture generation method based on a multi-view pixel spatial diffusion model, including: Obtain the geometric condition information and reference image of the 3D model; The purpose of geometric condition information is to allow the model to understand its 3D shape and surface structure. This information is typically generated from the 3D model by renderers (such as Blender, Unity, and specialized rendering engines). It includes canonical coordinate maps, position maps, and depth maps, among others.

[0038] The purpose of reference images is to provide a reference for the texture style of the model. It tells the model what colors, patterns, and materials are desired. This includes style reference images, etc. Reference images do not need to be aligned with the 3D model's viewpoint, nor do they need to be the same size as the geometry map. They are simply a source of inspiration for style and content. For example, in generating a white 3D owl sculpture model, the reference image could be an image of blue and white porcelain downloaded from the internet.

[0039] The multi-view noise map and the geometric condition information are concatenated in the channel dimension to obtain the multi-view condition concatenation feature. The reference image is used as an additional viewpoint and concatenated with the multi-view conditional stitching features in the sequence dimension to obtain the sequence stitching features; A pixel-space multi-view generation model generates multi-view RGB images based on sequence splicing features; The multi-view RGB image is projected onto the surface of the 3D model to generate a complete texture.

[0040] Optionally, the pixel-space multi-view generation model obtains multi-view RGB images based on sequence splicing features, including: The sequence splicing features are extracted with coarse-grained features by the multi-view Transformer module, and 5D Rotational Position Encoding (RoPE) is fused into the attention mechanism of the multi-view Transformer module to ensure that the coarse-grained structure of cross-view images in multi-view RGB images remains consistent. The 5D RoPE is a fusion of 2D image patch index and 3D world space coordinates. The multi-view RGB image is obtained by decoding the features output by the multi-view Transformer module through a 3D position-aware pixel-level decoder; wherein, at the input end of the 3D position-aware pixel-level decoder, the pixel-level dense 3D global position embedding is spliced ​​with the features output by the multi-view Transformer module.

[0041] In a specific scenario, such as generating high-quality bronze textures for a 3D white model of the ancient Chinese mythical beast "stone lion", the texture details are required to be clear (including bronze rust, taotie patterns, etc.), and the textures should fit perfectly from any angle without seams or misalignments.

[0042] Input: 3D model: white model of a stone lion.

[0043] Geometric conditions: The multi-view normal map and standard coordinate map (CCM) of the stone lion are obtained through rendering.

[0044] Reference image: A photograph of a genuine bronze ding (tripod cauldron) from the Shang and Zhou dynasties, featuring exquisite green patina and taotie (a mythical beast) patterns.

[0045] The sequence splicing features are extracted with coarse-grained features using a multi-view Transformer module, and 5D Rotational Position Encoding (RoPE) is fused into the attention mechanism of the multi-view Transformer module to ensure the consistency of the coarse-grained structure of cross-view images in multi-view RGB images. The specific steps of the 5D RoPE, which integrates 2D image patch indexes and 3D world space coordinates, are as follows: The model begins processing the prepared input data (i.e., sequence splicing features, which include noise, geometric conditions, and reference graphs).

[0046] Input the noise map and CCM map of the stone lion from six perspectives (front, side, top, etc.) and the reference image of the bronze tripod into the multi-view Transformer module.

[0047] Patchify: The multi-view Transformer module first slices the entire high-resolution image into small blocks (e.g., 16x16 pixels), and each block is converted into a "Patch Token". At this point, the nose, paws, torso of the stone lion, and the patterns on the bronze tripod are all transformed into tokens.

[0048] The model is faced with hundreds of random tokens. 5D Rotational Position Encoding (RoPE) correctly associates these hundreds of random tokens, such as depicting the nose token from the front view of the stone lion and the nose token from the side view as the same 3D structure, instead of incorrectly associating the nose and paws.

[0049] For each token, the model not only knows its position in the 2D image (2D image patch index, such as "located in the 3rd row and 4th column of the front view"), but more importantly, through the CCM diagram, the model knows the precise position of the 3D point corresponding to this token in the world coordinate system of the stone lion (3D world space coordinates, such as "coordinates are (0.2, 0.8, 0.5), approximately on the lion's chest").

[0050] 5D RoPE encodes these two coordinate information into instructions and injects them into the attention calculation of the Transformer.

[0051] When the model calculates attention, it receives a geometric instruction like: "Pay attention to tokens that are close in 3D space, regardless of which viewpoint of the 2D image they appear in." Thus, the model (attention mechanism) successfully associates the tokens depicting the "lion's chest" region across all perspectives, exchanging and fusing information among them. In this way, it establishes a consistent semantic concept of the "lion's chest" across perspectives at the global structural level (coarse-grained). This ensures that the generated lion's chests from multiple perspectives are structurally unified.

[0052] The multi-view RGB image is obtained by decoding the features output by the multi-view Transformer module using a 3D position-aware pixel-level decoder. Specifically, at the input of the 3D position-aware pixel-level decoder, the dense 3D global position embedding at the pixel level is concatenated with the features output by the multi-view Transformer module as follows: The multi-view Transformer module outputs coarse-grained features with global consistency; these features now need to be reconstructed into high-resolution RGB images. This task is accomplished by a 3D position-aware pixel-level decoder.

[0053] The decoder receives coarsely aligned feature maps from the multi-view Transformer module. However, these feature maps have low resolution and blurred details.

[0054] Pixel-level dense 3D position embedding: To ensure perfect alignment from different viewing angles when generating the precise color of each pixel (such as a small spot in bronze rust), coarse-grained patch-level alignment is not precise enough.

[0055] By embedding pixel-level dense 3D global locations with features before the decoder starts working, coarse-grained patch-level alignment accuracy is ensured.

[0056] Generate a coordinate map. This map is exactly the same size as the image to be generated, but instead of storing color, each pixel location stores the coordinate code (a vector) of the 3D point corresponding to that pixel.

[0057] For example, when generating an image of the "left eye of a lion from the front", the 3D coordinate embedding vector of the pixel depicting the center of the pupil in the image points to the precise 3D position (x_eye, y_eye, z_eye) of the lion's left pupil.

[0058] Then, this 3D coordinate map is stitched together with the feature map from the multi-view Transformer module in the channel dimension, and then input into the decoder.

[0059] When the decoder decides what color a pixel should generate, it can directly and accurately see the absolute position of the pixel in 3D space.

[0060] When the model generates textures for a side view, although the 2D pixel positions change, the 3D coordinates (x_eye, y_eye, z_eye) of the corresponding point in the lion's left pupil remain unchanged. Because the decoder perceives this identical 3D coordinate, it generates a completely consistent bronze color and rust for this point. This achieves sub-pixel-level, precise texture alignment.

[0061] Transformer+5D RoPE ensures that the overall structure of the stone lion (such as the chest and head) is correct and consistent when viewed from different angles.

[0062] The 3D position-aware decoder uses the injected precise 3D coordinates to sculpt every fine bronze pattern, ensuring that these patterns are perfectly attached to the correct 3D position from any angle.

[0063] Optionally, the 3D world space coordinates in the 5D rotational position encoding RoPE are obtained by normalizing and discretizing the 3D surface point coordinates into voxel indices.

[0064] Optionally, the 3D position-aware pixel-level decoder is a lightweight U-Net structure, and utilizes skip connections to transmit high-frequency information of the input.

[0065] Optionally, after projecting the multi-view RGB image onto the surface of the 3D model, if there are uncolored texture missing areas, a multi-view repair process is performed, including: Identify texture-deficient regions based on the 3D model; Based on the areas with missing textures, select a set of repair perspectives; The partially colored texture map and the repair mask are used as conditions and input into the pixel space multi-view repair model to generate an RGB image of the repair view. The repaired RGB image is projected onto the surface of the 3D model to fill in the areas where the texture is missing.

[0066] Optionally, the selection of a set of repair perspectives employs a greedy algorithm to maximize the coverage of the texture-missing areas.

[0067] Optionally, the training of the pixel space multi-view generation model adopts a flow matching framework, and its loss function includes velocity matching loss and multi-view consistency loss. The multi-view consistency loss is achieved by calculating the photometric reprojection error between predicted images from different viewpoints based on geometric projection transformation.

[0068] Optionally, the pixel space multi-view generation model does not rely on variational autoencoder (VAE) for spatial compression and reconstruction.

[0069] This invention proposes a PixTex texture generation framework that operates directly in pixel space and includes two core stages: an initial multi-view generation stage and a multi-view inpainting stage.

[0070] Pixel-space multi-view diffusion model: The network architecture in this embodiment includes two main modules for processing multi-view noise maps, geometric conditions (normal maps, canonical coordinate maps CCMs), and reference images.

[0071] Input representation and concatenation: Unlike treating conditions as independent token sequences, this embodiment employs channel concatenation and a strategy: spatially aligned conditions (such as geometric conditions). ) and noise graph Stitching along the channel dimension; unaligned reference image. As an additional perspective, it is pieced together in the sequence dimension.

[0072] Formula expression: , , , This is a feature for multi-view conditional splicing. For multi-view geometric condition information, For the formatted reference image features, This is the total input to the model, i.e., the sequence concatenation features.

[0073] Multi-view Transformer with 5D RoPE (coarse-grained consistency), including: 1. Patchify: Divide high-resolution input into Patch Tokens (image patch tokens); 2. 5D Rotational Position Encoding (5D RoPE): To establish cross-view structural correspondences, this embodiment introduces 5D RoPE, which combines 2D Patch indexes (local coordinates) and 3D world space coordinates (global coordinates). Specifically, the coordinates of 3D surface points are normalized and discretized into voxel indices, which are then injected into the attention mechanism. This serves to establish coarse but globally consistent structural relationships at the Patch level.

[0074] 3D position-aware patch-to-pixel detailer includes: 1. Since the Transformer outputs coarse patch features, this embodiment designs a lightweight U-Net decoder (Detailer) to recover pixel details.

[0075] 2. Explicit 3D Position Injection: At the input of the Detailser, pixel-wise dense 3D position embeddings are directly spliced ​​with pixel features.

[0076] 3. Skip Connections: Skip connections are used to directly pass high-frequency information from the input to the output. Combined with global semantic features from the Transformer, the final high-definition image is decoded.

[0077] Training objective and loss function: This embodiment uses a flow matching framework for training and introduces a specific consistency loss: Velocity Matching Loss: Learns the transmission trajectory from the noise distribution to the data distribution.

[0078] Multi-view Consistency Loss ): Using geometric projection relationships to determine the viewpoint Predicted clean image Change to perspective Calculate the projected image and viewpoint. Predicted image The pixel difference in the overlapping area.

[0079] formula: in For transformation operations, To effectively mask out the code.

[0080] To address texture loss caused by self-occlusion, the multi-view inpainting process is as follows: 1. Greedy View Selection: Project the initially generated 6 orthogonal views back into the Mesh, identify unpainted areas, and select a set of new inpainting views that maximizes coverage of the remaining areas using a greedy algorithm.

[0081] 2. Pixel Space Inpainting Model: This model has the same structure as the generative model, but the input includes partial views and inpainting masks as conditions.

[0082] 3. Guided by the reference image and existing textures, the model fills in the blank areas to ensure that the newly generated textures are seamlessly connected with the existing textures.

[0083] In this embodiment, the Patch-to-Pixel decoder can be U-Net, Transformer, or a neural field-based decoding module.

[0084] In this embodiment, 5D RoPE can be replaced with learnable 3D positional embeddings or relative position biases, as long as the attention mechanism is guided by 3D coordinate information.

[0085] In this embodiment, the greedy view selection can be replaced by a fixed number of views. Although the computational load increases, it can still cover most blind spots.

[0086] Although this embodiment uses x0-prediction-based Flow Matching, this pixel-space architecture is also applicable to standard diffusion model frameworks such as DDPM and DDIM.

[0087] This implementation has the following effects: 1. Superior multi-view consistency: Existing technologies only perform alignment in Latent space, and errors still exist after decoding. This embodiment achieves State-of-the-Art (SOTA) cross-view consistency through a triple strategy of "Patch-level 5D RoPE + Pixel-level 3D position injection + explicit consistency loss".

[0088] 2. Seamless integrity: Compared with the method of repairing seams in UV space, the pixel space multi-view repair module of this embodiment can generate visually coherent and seamless complete textures, effectively handling the occlusion areas of complex geometries.

[0089] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0090] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the 3D texture generation method based on a multi-view pixel spatial diffusion model of the foregoing embodiments of this disclosure.

[0091] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0092] like Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0093] like Figure 2 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0094] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to exchange data wirelessly or via wired communication with other devices, such as edge computing devices. Although Figure 2 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0095] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the 3D texture generation method based on a multi-view pixel spatial diffusion model according to embodiments of this disclosure are performed.

[0096] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0097] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the 3D texture generation method based on a multi-view pixel spatial diffusion model according to the foregoing embodiments of the present disclosure are performed.

[0098] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0099] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0100] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0101] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0102] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0103] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0104] Various changes, substitutions, and modifications can be made to the techniques described herein without departing from the teachings defined in this embodiment. Furthermore, the scope of this embodiment is not limited to the specific aspects of the processes, machines, manufacturing processes, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufacturing processes, events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein can be utilized. Therefore, this embodiment includes such processes, machines, manufacturing processes, events, means, methods, or actions within its scope.

[0105] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0106] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A 3D texture generation method based on a multi-view pixel spatial diffusion model, characterized in that, include: Obtain the geometric condition information and reference image of the 3D model; The multi-view noise map and the geometric condition information are concatenated in the channel dimension to obtain the multi-view condition concatenation feature. The reference image is used as an additional viewpoint and concatenated with the multi-view conditional stitching features in the sequence dimension to obtain the sequence stitching features; A pixel-space multi-view generation model generates multi-view RGB images based on sequence splicing features; The multi-view RGB image is projected onto the surface of the 3D model to generate a complete texture.

2. The 3D texture generation method based on a multi-view pixel spatial diffusion model according to claim 1, characterized in that, The pixel-space multi-view generation model obtains multi-view RGB images based on sequence splicing features, including: The sequence splicing features are extracted with coarse-grained features by the multi-view Transformer module, and 5D Rotational Position Encoding (RoPE) is fused into the attention mechanism of the multi-view Transformer module to ensure that the coarse-grained structure of cross-view images in multi-view RGB images remains consistent. The 5D RoPE is a fusion of 2D image patch index and 3D world space coordinates. The multi-view RGB image is obtained by decoding the features output by the multi-view Transformer module through a 3D position-aware pixel-level decoder; wherein, at the input end of the 3D position-aware pixel-level decoder, the pixel-level dense 3D global position embedding is spliced ​​with the features output by the multi-view Transformer module.

3. The 3D texture generation method based on a multi-view pixel spatial diffusion model according to claim 2, characterized in that, The 3D world space coordinates in the 5D Rotational Position Encoding (RoPE) are obtained by normalizing and discretizing the 3D surface point coordinates into voxel indices.

4. The 3D texture generation method based on a multi-view pixel spatial diffusion model according to claim 2, characterized in that, The 3D position-aware pixel-level decoder is a lightweight U-Net structure and uses skip connections to transmit high-frequency input information.

5. The 3D texture generation method based on a multi-view pixel spatial diffusion model according to claim 1, characterized in that, After projecting the multi-view RGB image onto the surface of the 3D model, if there are uncolored texture missing areas, a multi-view repair process is performed, including: Identify texture-deficient regions based on the 3D model; Based on the areas with missing textures, select a set of repair perspectives; The partially colored texture map and the repair mask are used as conditions and input into the pixel space multi-view repair model to generate an RGB image of the repair view. The repaired RGB image is projected onto the surface of the 3D model to fill in the areas where the texture is missing.

6. The 3D texture generation method based on a multi-view pixel spatial diffusion model according to claim 5, characterized in that, The selection of a set of repair perspectives employs a greedy algorithm to maximize the coverage of the texture-missing areas.

7. The 3D texture generation method based on a multi-view pixel spatial diffusion model according to claim 1, characterized in that, The pixel space multi-view generation model is trained using a flow matching framework, and its loss function includes velocity matching loss and multi-view consistency loss. The multi-view consistency loss is achieved by calculating the photometric reprojection error between predicted images from different viewpoints based on geometric projection transformation.

8. The 3D texture generation method based on a multi-view pixel spatial diffusion model according to claim 1, characterized in that, The pixel space multi-view generation model does not rely on variational autoencoder (VAE) for spatial compression and reconstruction.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the 3D texture generation method based on the multi-view pixel spatial diffusion model as described in any one of claims 1-8.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the 3D texture generation method based on the multi-view pixel spatial diffusion model as described in any one of claims 1-9.

Images