Three-dimensional scene texture generation method and device, readable storage medium and computer program product

Abstract

The invention discloses a texture generation method and device of a three-dimensional scene, a readable storage medium and a computer program product. The method comprises the following steps: acquiring a scene grid and a prompt text of the three-dimensional scene containing a plurality of instances; performing instance segmentation on the scene grid to obtain a three-dimensional directional bounding box of each instance; based on the three-dimensional orientation bounding box of each instance, obtaining a two-dimensional instance layout and a two-dimensional orientation prompt graph of each instance in each view angle of the predetermined number of view angles; for any view angle in the predetermined number of view angles, executing the following processing: inputting a prompt text, a two-dimensional instance layout and a two-dimensional orientation prompt graph of the current view angle and an acquired target image into a diffusion model to obtain a target image of the three-dimensional scene at the current view angle; and generating a three-dimensional scene containing texture information based on all the target images.

Description

Technical Field

[0001] The present disclosure generally relates to the technical field of computer vision and computer graphics, and more specifically, to a texture generation method and apparatus for a three-dimensional scene, a readable storage medium, and a computer program product. Background Art

[0002] In the field of computer vision and graphics, automatically generating high-quality 3D scene textures is a fundamental task and is crucial for downstream applications such as games, movies, augmented reality, and digital twins. In recent years, with the significant progress made in text-to-texture synthesis using 2D diffusion models, it has become possible to generate realistic textures for a single instance. However, when applied to more complex scenes, related techniques have difficulty maintaining style consistency and semantic alignment, and require a lot of optimization time and memory resources.

[0003] Currently, related methods rely on generative models trained on specific datasets to synthesize textures, which limits the generalization ability on instances of other categories and the diversity of generated textures is limited. In addition, these methods usually require domain-specific expertise and tedious manual work, and creating high-quality textures remains a daunting and time-consuming task. Despite recent significant progress in mesh-based texture synthesis, related techniques still face challenges when dealing with large-scale scenes, such as texture seams and accumulated artifacts. Summary of the invention

[0004] The embodiments of the present disclosure provide a texture generation method and device for a three-dimensional scene, a readable storage medium, and a computer program product, which can effectively solve the problem that related technologies cannot generate high-quality textures.

[0005] In a general aspect, a texture generation method for a three-dimensional scene is provided, comprising: obtaining a scene grid and a prompt text of a three-dimensional scene containing multiple instances, wherein the prompt text contains global texture information of the three-dimensional scene and texture information of each instance in the multiple instances; performing instance segmentation on the scene grid to obtain a three-dimensional directional bounding box of each instance; based on the three-dimensional directional bounding box of each instance, obtaining a two-dimensional instance layout and a two-dimensional orientation prompt map of each instance in each of a predetermined number of viewing angles; for any of the predetermined number of viewing angles, performing the following processing: inputting the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current viewing angle, and an acquired target image into a diffusion model to obtain a target image of the three-dimensional scene in the current viewing angle, wherein the target image contains the texture of the three-dimensional scene, the acquired target image is a target image of other viewing angles acquired before the target image of the current viewing angle, and the acquired target image is empty when the target image of the first viewing angle is acquired; and generating a three-dimensional scene containing texture information based on all target images.

[0006] Optionally, after obtaining the two-dimensional instance layout and the two-dimensional orientation prompt map of each instance in each predetermined number of perspectives based on the three-dimensional directional bounding box of each instance, it also includes: obtaining a depth map and a line map of the three-dimensional scene in each predetermined number of perspectives; wherein the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current perspective, and the acquired target image are input into the diffusion model to obtain the target image of the three-dimensional scene in the current perspective, including: inputting the prompt text, the two-dimensional instance layout of the current perspective, the two-dimensional orientation prompt map, the line map and the depth map, and the acquired target image into the diffusion model to obtain the target image of the three-dimensional scene in the current perspective.

[0007] Optionally, before generating a three-dimensional scene containing texture information based on all target images, the method further includes: obtaining latent images corresponding to the target images under a predetermined number of viewing angles; adding noise to the latent images to obtain noisy latent images; and inputting the noisy latent images into a local multi-view diffusion model to obtain a refined target image.

[0008] Optionally, the local multi-view diffusion model performs a multi-step denoising process when processing a noisy latent image, wherein before each step of the denoising process, the following operations are performed: projecting the noisy latent image into a temporary texture space; determining the overlapping area of ​​latent images of different perspectives in the temporary texture space; weighting the parts of the latent images of different perspectives that fall in the overlapping area; and replacing the corresponding part of the latent image of the corresponding perspective with the weighted part.

[0009] Optionally, based on all target images, a three-dimensional scene containing texture information is generated, including: for each target image, after uniformly sampling in the target image, projecting the sampling points into the UV space to obtain UV coordinates corresponding to each sampling point; inputting the UV coordinates into a hash model to obtain texture features corresponding to each sampling point; averaging the texture features to obtain average features of the target image; and inputting the average features of all target images into a multilayer perceptron to obtain a three-dimensional scene containing texture information.

[0010] Optionally, the diffusion model performs a multi-step denoising process when processing the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current perspective, and the acquired target image, wherein in each step of the denoising process, the following operations are performed: when the current step of the denoising process is for the instance level, nearest neighbor sampling is performed in the latent space, wherein the latent space stores the texture information of the instance, and the texture information of the instance is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective; the first latent image of the image is obtained, and the first latent image is denoised; based on the instance-level mask of the current instance, the first latent image and the first latent image after denoising, the first image; when the current denoising process is for the scene level, bilinear sampling is performed through the RGB space, wherein the RGB space stores global texture information, and the global texture information is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective, wherein, when the current denoising process is the first processing for the scene level, the image of the current perspective is the first image; a second latent image of the image is obtained, and the second latent image is denoised; a second image is obtained based on the scene level mask, the second latent image and the denoised second latent image; wherein, after executing the denoising process for a predetermined number of steps, the second image is used as the target image of the current perspective.

[0011] Optionally, before each denoising process, it also includes: when the number of pixels contained in any instance of the three-dimensional scene after being projected to the current viewing angle exceeds a preset threshold, the texture information contained in the target image corresponding to any instance is only stored in the latent space.

[0012] Optionally, based on the three-dimensional directional bounding box of each instance, a two-dimensional instance layout and a two-dimensional orientation prompt map of each instance in each perspective in a predetermined number of perspectives are obtained, including: normalizing the coordinates of the three-dimensional directional bounding box of each instance and projecting them to a predetermined number of perspectives to obtain a two-dimensional bounding box of each instance at different perspectives; based on the two-dimensional bounding box and prompt text of each instance, obtaining a two-dimensional instance layout of each instance at different perspectives; normalizing the coordinates of the three-dimensional directional bounding box of each instance and converting them into RGB format, and projecting them to a predetermined number of perspectives to obtain a two-dimensional orientation prompt map of each instance at different perspectives.

[0013] In another general aspect, a texture generation device for a three-dimensional scene is provided, comprising: a first acquisition unit, configured to acquire a scene grid and a prompt text of a three-dimensional scene containing multiple instances, wherein the prompt text includes global texture information of the three-dimensional scene and texture information of each instance in the multiple instances; a second acquisition unit, configured to perform instance segmentation on the scene grid and acquire a three-dimensional directional bounding box of each instance; a third acquisition unit, configured to acquire a two-dimensional instance layout and a two-dimensional orientation prompt map of each instance in each of a predetermined number of viewing angles based on the three-dimensional directional bounding box of each instance; a processing unit, configured to perform the following processing for any one of the predetermined number of viewing angles: inputting the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current viewing angle, and the acquired target image into a diffusion model to obtain a target image of the three-dimensional scene in the current viewing angle, wherein the target image includes the texture of the three-dimensional scene, the acquired target image is a target image of other viewing angles acquired before the target image of the current viewing angle, and the acquired target image is empty when the target image of the first viewing angle is acquired; and a generation unit, configured to generate a three-dimensional scene containing texture information based on all target images.

[0014] Optionally, the third acquisition unit is further configured to obtain a depth map and a line map of the three-dimensional scene for each of a predetermined number of perspectives, after obtaining a two-dimensional instance layout and a two-dimensional orientation prompt map for each instance in each of a predetermined number of perspectives based on a three-dimensional directional bounding box for each instance; wherein the processing unit is further configured to input the prompt text, the two-dimensional instance layout of the current perspective, the two-dimensional orientation prompt map, the linear map and the depth map, and the acquired target image into the diffusion model to obtain the target image of the three-dimensional scene at the current perspective.

[0015] Optionally, the generation unit is further configured to obtain latent images corresponding to the target images under a predetermined number of viewing angles before generating a three-dimensional scene containing texture information based on all target images; add noise to the latent images to obtain noisy latent images; and input the noisy latent images into a local multi-view diffusion model to obtain a refined target image.

[0016] Optionally, the local multi-view diffusion model performs a multi-step denoising process when processing a noisy latent image, wherein before each step of the denoising process, the following operations are performed: projecting the noisy latent image into a temporary texture space; determining the overlapping area of ​​latent images of different perspectives in the temporary texture space; weighting the parts of the latent images of different perspectives that fall in the overlapping area; and replacing the corresponding part of the latent image of the corresponding perspective with the weighted part.

[0017] Optionally, the generation unit is further configured to, for each target image, after uniformly sampling in the target image, project the sampling points into the UV space to obtain UV coordinates corresponding to each sampling point; input the UV coordinates into a hash model to obtain texture features corresponding to each sampling point; average the texture features to obtain average features of the target image; and input the average features of all target images into a multilayer perceptron to obtain a three-dimensional scene containing texture information.

[0018] Optionally, the diffusion model performs a multi-step denoising process when processing the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current perspective, and the acquired target image, wherein in each step of the denoising process, the following operations are performed: when the current step of the denoising process is for the instance level, nearest neighbor sampling is performed in the latent space, wherein the latent space stores the texture information of the instance, and the texture information of the instance is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective; the first latent image of the image is obtained, and the first latent image is denoised; based on the instance-level mask of the current instance, the first latent image and the first latent image after denoising, the first image; when the current denoising process is for the scene level, bilinear sampling is performed through the RGB space, wherein the RGB space stores global texture information, and the global texture information is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective, wherein, when the current denoising process is the first processing for the scene level, the image of the current perspective is the first image; a second latent image of the image is obtained, and the second latent image is denoised; a second image is obtained based on the scene level mask, the second latent image and the denoised second latent image; wherein, after executing the denoising process for a predetermined number of steps, the second image is used as the target image of the current perspective.

[0019] Optionally, before each denoising process, it also includes: when the number of pixels contained in any instance of the three-dimensional scene after being projected to the current viewing angle exceeds a preset threshold, the texture information contained in the target image corresponding to any instance is only stored in the latent space.

[0020] Optionally, the third acquisition unit is further configured to normalize the coordinates of the three-dimensional directional bounding box of each instance and project them to a predetermined number of viewing angles to obtain a two-dimensional bounding box of each instance at different viewing angles; obtain a two-dimensional instance layout of each instance at different viewing angles based on the two-dimensional bounding box and prompt text of each instance; normalize the coordinates of the three-dimensional directional bounding box of each instance and convert them into RGB format, and project them to a predetermined number of viewing angles to obtain a two-dimensional orientation prompt map of each instance at different viewing angles.

[0021] In another general aspect, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform any of the above-mentioned methods for generating textures of a three-dimensional scene.

[0022] In another general aspect, a system is provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform any of the above-described methods for generating textures for a three-dimensional scene.

[0023] In another general aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the texture generation method for a three-dimensional scene as described above.

[0024] In another general aspect, an electronic device is provided, comprising at least one memory and at least one processor, wherein the at least one memory is used to store a set of computer executable instructions, which, when executed by the at least one processor, causes the at least one processor to execute any of the above-mentioned texture generation methods for a three-dimensional scene.

[0025] According to the texture generation method and device, readable storage medium, and computer program product of the three-dimensional scene of the embodiment of the present disclosure, the precise control of a single instance is achieved by means of the instance layout, that is, the instance layout is used as the control condition for texture generation, so that the generated texture is more consistent with the texture in the prompt text specified by the user; and the present disclosure projects the three-dimensional scene to two dimensions, and uses the two-dimensional instance layout and the two-dimensional orientation prompt map as the control conditions for texture generation, and generates the texture of the three-dimensional scene at the two-dimensional level, thereby avoiding the problem that the texture of good quality cannot be generated directly at the three-dimensional level due to the insufficient generation quality of the three-dimensional generation baseline model itself; further, the present disclosure also considers that the direction prompt plays a vital role in the texture generation of a single instance, and also introduces the orientation prompt map as the control condition for texture generation, so as to further improve the quality of the generated texture. Therefore, through the present disclosure, the problem that the related technology cannot generate high-quality textures can be effectively solved.

[0026] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects and features of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings showing the embodiments, in which:

[0028] Figure 1is a flowchart showing a texture generation method for a three-dimensional scene according to an embodiment of the present disclosure;

[0029] Figure 2 is a system flow diagram showing a texture generation method for a three-dimensional scene according to an embodiment of the present disclosure;

[0030] Figure 3 is a schematic diagram showing a denoising process of a diffusion model according to an embodiment of the present disclosure;

[0031] Figure 4 is a first effect comparison diagram showing an embodiment of the present disclosure and a related art;

[0032] Figure 5 is a diagram showing a second effect comparison between an embodiment of the present disclosure and a related art;

[0033] Figure 6 is a diagram showing a second effect comparison between an embodiment of the present disclosure and a related art;

[0034] Figure 7 is a fourth effect comparison diagram showing an embodiment of the present disclosure and a related art;

[0035] Figure 8 Detailed description is a block diagram showing a texture generating apparatus for a three-dimensional scene according to an embodiment of the present disclosure.

[0036] Fig. 9 Schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0038] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0039] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0040] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0041] In the specification, when an element (such as a layer, a region, or a substrate) is described as being “on”, “connected to”, or “coupled to” another element, the element may be directly “on”, “connected to”, or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may be no other elements present therebetween.

[0042] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0043] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0044] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0045] The present disclosure can be applied to computer graphics and computer vision technology scenarios, especially for texture generation under multiple perspectives, which significantly improves the realism and consistency of image and video generation.

[0046] Currently, automatically generating high-quality textures for complex scenes remains a major challenge in computer graphics. Although text-to-texture synthesis using a 2D diffusion model has produced good results for a single instance, it is still difficult to maintain style consistency and semantic alignment when applied to larger scenes, requiring a lot of optimization time and memory. In order to address these challenges, the texture synthesis method disclosed in the present invention can create realistic and stylistically consistent textures for large scenes with multiple instances. The core of the method is controllable texture synthesis at the instance level, that is, precise semantic control of a single instance is achieved with the help of instance layout, while ensuring the consistency of the overall style. Furthermore, the present disclosure also introduces a locally synchronized multi-view diffusion strategy to improve local texture consistency by sharing potential denoising content in small batches of images from adjacent perspectives. In addition, the present disclosure also introduces a neural MipTexture inspired by Mipmaps, which is specially designed for scene texture mapping to minimize aliasing.

[0047] The texture generation method and device for a three-dimensional scene, a readable storage medium, and a computer program product of the present disclosure are described in detail below with reference to the accompanying drawings.

[0048] This disclosure proposes a texture generation method for three-dimensional scenes, aiming to synthesize a highly detailed and semantically aligned texture that fits the large-scale scene geometry and is not affected by local or global discontinuities. Figure 1 Detailed description is a flowchart illustrating a method for generating texture of a three-dimensional scene according to an embodiment of the present disclosure.

[0049] Reference Figure 1 , the texture generation method of the three-dimensional scene comprises the following steps:

[0050] In step S101, a scene mesh and a prompt text of a three-dimensional scene including multiple instances are obtained, wherein the prompt text includes global texture information of the three-dimensional scene and texture information of each of the multiple instances.

[0051] As an example, the initial input of the method disclosed in the present invention can be understood as threefold, one input is a scene mesh of a three-dimensional scene containing multiple instances, the scene mesh displays a textureless 3D scene, one input is a scene-level prompt text of the global style (i.e., global texture information) of the three-dimensional scene specified by the user, and another input is an instance-level prompt text of the expected appearance of each instance specified by the user (i.e., the texture information of each instance mentioned above). It should be noted that the latter two texts are combined together to form the prompt text in step S101.

[0052] As an example, the prompt text in the above steps contains the texture information of each instance, which can specify the texture style of each instance to guide the diffusion model to generate a texture that meets the specific style and content requirements, thereby generating a texture that meets the instance description and is consistent in overall style.

[0053] According to an embodiment of the present disclosure, the description information of each instance in the three-dimensional scene (i.e., the texture information of each instance mentioned above) and the description information of the three-dimensional scene (i.e., the global texture information) can be obtained by manual setting or image guidance. For example, the image-guided acquisition method is to input images of multiple angles of the three-dimensional scene into the description information extraction unit to extract the corresponding description information, which is not limited by the present disclosure.

[0054] As an example, a texture unit (texel) is assigned to each vertex of the scene mesh to store the corresponding texture information, and the texture is mapped to the scene mesh through reverse mapping, thereby providing a more realistic appearance to the three-dimensional scene; if some vertices are not assigned to a texture unit, the Blender software smart UV mapping tool is used to map the texture to the scene mesh.

[0055] As an example, after obtaining the scene mesh of the three-dimensional scene, the scene mesh may be preprocessed, for example, normal vector calculation, point cloud simplification, etc., to ensure the efficiency and accuracy of subsequent processing.

[0056] In step S102, instance segmentation is performed on the scene mesh to obtain a three-dimensional oriented bounding box for each instance.

[0057] As an example, each instance in the scene network is segmented, and then the segmented scene network can be detected using an automatic detection method to generate a 3D oriented bounding box (OBB) for each instance in the scene network. When the 2D information is subsequently acquired, the 3D oriented bounding box can be used to control a powerful 2D diffusion model because it combines geometric and semantic information, thereby creating a consistent texture aligned with the scene layout.

[0058] In step S103, based on the three-dimensional directional bounding box of each instance, a two-dimensional instance layout (Instance Layout) and a two-dimensional position map (Position Map) of each instance in each of a predetermined number of viewing angles are obtained.

[0059] As an example, the predetermined number of viewing angles may be preset manually or in other ways, as long as all points on the scene grid can be observed at the preset viewing angles, so that a complete texture can be generated. The preset viewing angles may be represented as a sequence of camera poses, that is, a sequence consisting of a series of three-dimensional rotation matrices and translation vectors.

[0060] As an example, one of the key points of the present disclosure is to introduce an instance layout as a control condition for texture generation, where the instance layout contains the location of each instance and the corresponding prompt text, such as Figure 2 As shown in Figure 1, this enables precise and flexible instance-level control. Specifically, for each instance, the 3D OBB of the instance can be normalized to the range [0, 1], and then the 3D OBB can be rendered into a two-dimensional space according to a preset viewing angle, thereby obtaining a two-dimensional instance layout of the corresponding viewing angle.

[0061] As an example, directional cues play a crucial role in texture generation for a single instance, which motivates the introduction of an orientation cue map as a directional guide. Without the orientation cue map, the generated results lack consistency and sharp edges are produced during the restoration process. However, it is not easy to derive feasible directional cues for large scenes because different instances have different poses in a scene. To this end, a pose-aware position map can be defined to represent the relative pose between the current camera and the instance. Specifically, for each instance, the 3D OBB of the instance can be normalized to the [0, 1] range, then converted to RGB format, and the 3D OBB is rendered into a two-dimensional space according to a preset viewing angle, thereby obtaining a two-dimensional orientation cue map of the corresponding viewing angle.

[0062] It should be noted that, for any point in the orientation hint map, its value represents the relative position of the point in the 3D OBB of the corresponding instance, such as Figure 2 Therefore, for each view of the 3D scene, an orientation cue map containing multiple instances is generated as an additional control condition for texture generation.

[0063] According to an embodiment of the present disclosure, based on the three-dimensional directional bounding box of each instance, obtaining the two-dimensional instance layout and two-dimensional orientation prompt map of each instance in each perspective in a predetermined number of perspectives may include: normalizing the coordinates of the three-dimensional directional bounding box of each instance and projecting them to a predetermined number of perspectives to obtain the two-dimensional bounding box (2D Bounding Box) of each instance at different perspectives; based on the two-dimensional bounding box and prompt text of each instance, obtaining the two-dimensional instance layout of each instance at different perspectives; normalizing the coordinates of the three-dimensional directional bounding box of each instance and converting them into RGB format, and projecting them to a predetermined number of perspectives to obtain the two-dimensional orientation prompt map of each instance at different perspectives. Through this embodiment, the two-dimensional instance layout and two-dimensional orientation prompt map of each instance at different perspectives can be easily and quickly obtained.

[0064] return Figure 1 In step S104, for any of the predetermined number of perspectives, the following processing is performed: the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current perspective, and the acquired target image are input into the diffusion model to obtain the target image of the three-dimensional scene at the current perspective, wherein the target image includes the texture of the three-dimensional scene, the acquired target image is the target image of other perspectives acquired before the target image of the current perspective, and the acquired target image is empty when the target image of the first perspective is acquired.

[0065] As an example, in order to add the above instance-level control conditions, ie, two-dimensional instance layout and two-dimensional orientation prompt map, in the diffusion process, Instance Diffusion can be used as the basic model of the above diffusion model.

[0066] As an example, the orientation cue map can use the control network (ControlNet) to extract the corresponding features, that is, ControlNet is used as the encoder in the diffusion model to extract the features of the orientation cue map. However, ordinary ControlNet is pre-trained by copying the structure of Stable Diffusion, and directly applying it to Instance Diffusion will produce obvious artifacts. In order to better achieve geometric alignment, ControlNet can be fine-tuned by training an adapter for the orientation cue map in Instance Diffusion to avoid obvious artifacts. It should be noted that Stable Diffusion and Instance Diffusion are both potential diffusion models.

[0067] As an example, when the current perspective is the first perspective, the above-mentioned acquired target image may be an empty feature; when the current perspective is the second perspective, the above-mentioned acquired target image is the target image of the first perspective; when the current perspective is the third perspective, the above-mentioned acquired target image is the target image of the first perspective and the target image of the second perspective; and so on, when the current perspective is the nth perspective, the above-mentioned acquired target image is the target image of the first perspective to the target image of the n-1th perspective.

[0068] According to an embodiment of the present disclosure, after obtaining the two-dimensional instance layout and two-dimensional orientation prompt map of each instance in each of the predetermined number of viewing angles based on the three-dimensional directional bounding box of each instance, the depth map and line map of the three-dimensional scene in each of the predetermined number of viewing angles can also be obtained; wherein the prompt text, the two-dimensional instance layout and two-dimensional orientation prompt map of the current viewing angle, and the acquired target image are input into the diffusion model to obtain the target image of the three-dimensional scene in the current viewing angle, including: inputting the prompt text, the two-dimensional instance layout of the current viewing angle, the two-dimensional orientation prompt map, the line map and the depth map, and the acquired target image into the diffusion model to obtain the target image of the three-dimensional scene in the current viewing angle. Through this embodiment, the depth map and the line map are introduced so that the generated texture can be geometrically aligned with the three-dimensional scene.

[0069] As an example, the above 2D instance layout and hint text can be directly input into Instance Diffusion. However, the synthesized size texture must not only be constrained by the instance layout, but also aligned with the scene geometry. Therefore, for each view, a depth map and a line map can be rendered as geometric clues, such as Figure 2 In this embodiment, a set of viewing angles (i.e., the predetermined number of viewing angles mentioned above) may be defined, and for each viewing angle, a depth map, a linear map, and a two-dimensional orientation prompt map of the scene are rendered. These three conditions together with the two-dimensional instance layout are input into the diffusion model to generate a target image for each viewing angle.

[0070] As an example, the depth map and the linear map can be obtained in the following way: by rendering the geometric information of the scene mesh according to a predetermined number of viewing angles using the camera's pose and internal parameters in a standard rasterization method, a multi-channel geometric hint is obtained, including a multi-view depth map and a multi-view line map, that is, a depth map for each view and a line map for each view. The feature extraction of the depth map and the linear map can be performed using the orientation hint. Figure 1 This method will not be discussed here.

[0071] As an example, in order to ensure that the image generated by the diffusion model is consistent with the control conditions of the image domain, the present disclosure uses a large number of 2D / 3D data sets with text to train multiple control networks (ControlNet) for geometric conditions and camera pose conditions, which are used to extract the features of depth maps, linear maps, and orientation prompt maps. For instance layout, a model based on a cross-modal attention mechanism can be additionally trained on image data with instance-level semantics to extract the model of instance layout.

[0072] In summary, this embodiment generates image domain control conditions for scene mesh texture under multiple preset perspectives through a rendering method, which mainly include the following conditions: geometric conditions (depth map and line map), text conditions (instance layout), and camera pose conditions (orientation prompt map).

[0073] According to an embodiment of the present disclosure, the diffusion model performs a multi-step denoising process when processing prompt text, a two-dimensional instance layout and a two-dimensional orientation prompt map of the current perspective, and an acquired target image, wherein in each step of the denoising process, the following operations are performed: when the current step of the denoising process is for the instance level, nearest neighbor sampling is performed in the latent space, wherein the latent space stores texture information of the instance, and the texture information of the instance is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective; the first latent image of the image is acquired, and the first latent image is denoised; based on the instance-level mask of the current instance, the first latent image and the first latent image after denoising, to the first image; when the current denoising process is for the scene level, bilinear sampling is performed through the RGB space, wherein the RGB space stores global texture information, and the global texture information is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective, wherein when the current denoising process is the first processing for the scene level, the image of the current perspective is the first image; the second latent image of the image is obtained, and the second latent image is denoised; based on the scene level mask, the second latent image and the second latent image after denoising, the second image is obtained; wherein, after executing the denoising process for a predetermined number of steps, the second image is used as the target image of the current perspective. Through this embodiment, the consistency of the image generated by multiple perspectives can be preliminarily ensured.

[0074] As an example, after the prompt text, the two-dimensional instance layout of the current perspective, the two-dimensional orientation prompt map, the linear map and the depth map, and the acquired target image are input into the diffusion model, the two-dimensional instance layout can be pre-encoded using the specified UniFusion block, and the texture information and global texture information of the instance are determined based on the encoded two-dimensional instance layout, and the texture information and global texture information of the instance are mapped to the latent space and the RGB space respectively, wherein the latent space is used for instance-level denoising and the RGB space is used for scene-level denoising. It should be noted that after using the instance layout as input, this embodiment divides the denoising process of the image into two stages: instance-level denoising and scene-level denoising, such as Figure 2 Specifically, after receiving the 2D instance layout, first, the instances are individually denoised using a certain number of steps to achieve instance-level denoising, and then the scene-level latent representation is generated by merging the latent representations generated by the instance-level denoising, and then the scene-level denoising can be performed by denoising the remaining steps based on the generated scene-level latent representation.

[0075] As an example, in order to preliminarily ensure the consistency of images generated from multiple perspectives, and to integrate the diffusion prior generated from instance-level images into the image restoration architecture, this embodiment adopts a two-stage, iterative image restoration (Inpainting) generation strategy. In the instance-level denoising process, a latent space is maintained to store the texture information of the instance; in the scene-level denoising process, an RGB space is maintained to store the global texture information of the three-dimensional scene. The above two spaces continuously update the stored texture information during the multi-perspective iterative image restoration process. It should be noted that in each step of the denoising process, it can be determined whether the current step belongs to the instance-level denoising process or the scene-level denoising process.

[0076] As an example, after generating a texture of one perspective, that is, generating a target image of one perspective, you can continue to generate target images (also called texture images) of other perspectives with global and local style consistency based on the generated target image. If the generated target image is directly applied to the Instance Diffusion of a large scene for subsequent image generation, serious artifacts will be generated, because the introduction of the scene-level context of the previous frame in the instance-level denoising process will destroy the generation of images of different instances, resulting in information leakage between instances, that is: the prompt text of one instance will affect the appearance of another instance. In addition, due to the semantic complexity of the scene, using only scene-level repair increases the difficulty of the diffusion model to produce coordinated results, that is, it is challenging to draw the entire scene, which may cause large inconsistencies in the generated image texture, making it difficult to infer a reasonable texture.

[0077] To solve this problem, the present embodiment decomposes the denoising process of image generation of multiple instances into two stages: the denoising process at the instance level and the denoising process at the scene level, which largely solves the problems of information leakage and texture incoherence. In order to achieve two-stage image generation, a separate mask can be generated for each instance and the entire scene. In addition, the present embodiment also maintains two independent spaces, namely the latent space and the RGB space, which are used to record the texture information of the instance and the global texture information of the three-dimensional scene, respectively.

[0078] Specifically, when it is determined that the current step belongs to the instance-level denoising process, the nearest neighbor sampling is performed through the maintained latent space. After the existing texture information of the sample is projected to the image under the current perspective, the existing part of the sample is locally denoised, and the latent variables of the part to be generated are fused, thereby executing the denoising process of this stage, which can be expressed as follows:

[0079]

[0080] in, represents the inpainting mask corresponding to instance i, represents the latent variable generated by the denoising process of instance i at step t, and It represents the latent variable part generated by the instance i after the noise addition process. It should be noted that the iterative process of the denoising process can be regarded as a reverse processing, that is, the latent variable with T times of noise is denoised T times.

[0081] When judging that the current step belongs to the scene-level denoising process, first, the latent variables of the above instance-level denoising process are obtained. Then, bilinear sampling is performed in the maintained RGB space. After the sampled existing texture information is projected onto the image at the current viewing angle, local noise is added to the existing sampled part, and the potential variables of the part to be generated are fused to perform the denoising process at this stage, which can be expressed as follows:

[0082]

[0083] in, Refers to the scene-level inpainting mask, represents the latent variable generated by the denoising process of the 3D scene in step t, and Represents the potential variable part generated by the 3D scene after noise addition.

[0084] In summary, if Figure 3As shown in Fig. 1, for each instance at a viewing angle, the instance-level denoising process first projects the known context of the instance (i.e., the texture information sampled above) to the image at the current viewing angle. Then, in order to constrain the instance-level denoising process with the known context of the instance, noise can be added to the projected known context and the denoised latent image can be added using the corresponding inpainting mask. With the estimated denoising potential Mixed to obtain the updated latent variable of instance i. Then, an image of the 3D scene is generated by the remaining denoising steps, which are conditioned on the scene-level inpainting mask and the known context of the entire scene (i.e., the sampled texture information).

[0085] According to an embodiment of the present disclosure, before each denoising process of the diffusion model, when the number of pixels contained in any instance of the three-dimensional scene after being projected to the current viewing angle exceeds a preset threshold, the texture information contained in the target image corresponding to any instance is only stored in the latent space. Through this embodiment, the texture information is selectively stored, and unnecessary processing can be avoided to optimize computing resources, thereby improving generation efficiency.

[0086] As an example, multiple layers of latent variables help mitigate information leakage and achieve more consistent scene images with higher fidelity. Specifically, at a long distance, although a more complete texture is generated for the instance, the texture quality is lower due to its small size in the target image. In this embodiment, this low-quality texture can be retained as part of the initialization, that is, when viewing this example from a closer perspective, this initialization can serve as a consistency hint to convert the texture into a high-quality and stylistically consistent texture through image restoration (inpainting). This approach helps mitigate the complexity of the entire scene while ensuring the continuity of the generated texture through instance-level image generation operations.

[0087] As an example, this embodiment proposes the following strategy to select distant instances of a three-dimensional scene, and perform different instance-level and scene-level image generation operations on instances under different occlusion conditions. Specifically, a strict threshold is first established to identify instances that are far away from the current camera viewpoint. When the threshold is greater than the number of pixels contained in the instance after being projected to the current viewing angle, the instance can be considered to be far away from the current camera, and the texture contained in the image generated by the instance is only projected onto the latent space. This selective projection can avoid unnecessary processing to optimize computing resources, thereby improving generation efficiency. It should be noted that the number of pixels contained in the instance after being projected to the current viewing angle can be determined by the pixels contained in the projection of the triangle of the instance to the current viewing angle.

[0088] As an example, the above instances depend on how close they are to the current camera's viewpoint and occlusion status.

[0089] return Figure 1 In step S105, a three-dimensional scene containing texture information is generated based on all target images.

[0090] According to an embodiment of the present disclosure, before generating a three-dimensional scene containing texture information based on all target images, it is also possible to obtain potential images corresponding to the target images under a predetermined number of viewing angles; add noise to the potential images to obtain a noisy potential image; and input the noisy potential image into a local multi-view diffusion model to obtain a refined target image. Through this embodiment, the target image is refined using the local multi-view diffusion model, which can improve the texture consistency of the multi-view image.

[0091] As an example, for large-scale scene texture generation, it is usually difficult to maintain geometric and semantic consistency over a long trajectory. In order to improve the consistency of multi-view images, this embodiment introduces a local multi-view diffusion (LMVD) model to refine the target image generated previously.

[0092] Specifically, the LMVD model can be inserted into the image restoration process of the above embodiment. For example, after generating the first batch of target images containing images = {1, 2, ...,}, each target image is encoded into a latent space by an encoder, and noise is gradually added to obtain a noisy latent image, and the noisy latent image is input into the LMVD model to obtain a refined target image to refine the local details of each target image while maintaining its overall structure. Next, a denoising process can be performed through the latent space, which makes it possible to exchange information between different perspectives.

[0093] As an example, the above-mentioned LMVD model includes an encoder module, a diffusion module and a decoder module, wherein the encoder module extracts features from the input multi-perspective target image to generate a feature representation of the target image corresponding to the input; the diffusion model processes the feature representation output by the encoder module to generate high-quality textures that are consistent with multiple perspectives, that is, high-dimensional features; the decoder module maps the high-dimensional features generated by the diffusion module back to the original target image space to generate the final high-quality texture.

[0094] It should be noted that the encoder module can be composed of multiple parallel convolutional neural networks (CNNs) to process image data from different perspectives respectively; the diffusion module can gradually optimize the details and consistency of the texture through iterative calculation; the decoder module can use a deconvolution network to restore high-dimensional features to image texture layer by layer.

[0095] According to an embodiment of the present disclosure, the local multi-view diffusion model performs a multi-step denoising process when processing a noisy latent image, wherein before each step of the denoising process, the following operations are performed: projecting the noisy latent image into a temporary texture space; determining the overlapping area of ​​latent images of different perspectives in the temporary texture space; weighting the parts of the latent images of different perspectives that fall in the overlapping area; and replacing the corresponding parts of the latent images of the corresponding perspectives with the weighted parts. Through this embodiment, the potential denoising content is shared in the process of processing small batches of images, thereby further improving the consistency of local textures and the fidelity of details.

[0096] As an example, before each denoising process, the potential image can be shared between target images of different perspectives to synchronize the diffusion process. For example, in each denoising step, all the potential images after adding noise can be projected into a temporary texture space (such as a UV texture space), and then, in the temporary texture space, information sharing is achieved through the overlapping areas between target images of different perspectives. It should be noted that the temporary texture space may contain multiple overlapping areas, and the target images involved in each overlapping area can be regarded as a batch, that is, a small batch of images in the above embodiment. Specifically, the potential values ​​of different perspectives in the overlapping area (that is, the potential image parts falling in the overlapping area) are weighted mixed. Finally, the weighted mixed part is mapped back to the corresponding part of the potential image of the corresponding perspective, and an updated potential image is generated for denoising. It should be noted that the weight of the above weighted mixing can be calculated as the cosine value between the normal vector of the visible surface in the 3D mesh and the viewing direction, which is not limited by the present disclosure.

[0097] In summary, by sharing the denoised content between local views, better local consistency can be achieved. In addition, in order to ensure global consistency as much as possible, overlapping views can be set between two consecutive mini-batch processing processes.

[0098] It should be noted that UV space is a two-dimensional coordinate space related to the texture mapping of the 3D model surface. In UV space, the U axis and V axis represent two independent directions, usually U corresponds to the horizontal direction and V corresponds to the vertical direction. The coordinate range is usually between 0 and 1. (0, 0) represents the upper left corner of the texture and (1, 1) represents the lower right corner of the texture. This two-dimensional space is used to determine how each pixel on the texture image corresponds to the surface of the 3D model.

[0099] According to an embodiment of the present disclosure, generating a three-dimensional scene containing texture information based on all target images may include: for each target image, after uniformly sampling in the target image, projecting the sampling points into the UV space to obtain the UV coordinates corresponding to each sampling point; inputting the UV coordinates into a hash model to obtain the texture features corresponding to each sampling point; averaging the texture features to obtain the average features of the target image; and inputting the average features of all target images into a multi-layer perceptron to obtain a three-dimensional scene containing texture information. Through this embodiment, the use of a hash model and multi-sampling technology can reduce the aliasing effect and ensure the high fidelity and visual quality of texture synthesis.

[0100] As an example, with the primary generation of consistent multi-view images, a texture map needs to be unfolded for the entire scene, and directly projecting the generated 2D image onto the texture space of a large-scale 3D scene often faces aliasing problems, which will produce noise artifacts due to the different distances between different instances. Traditionally, mipmapping is a very effective anti-aliasing technique that builds a pre-filtered image pyramid by constructing texture images of gradually decreasing resolution. During rendering, the pixel color is determined by selecting the most appropriate texture map based on the pixel and UV size.

[0101] This embodiment proposes a Neural MipTexture network, which uses a multi-resolution hash model and mipmapping technology to expand textures in large scenes. Figure 2 As shown, for a given target image, uniform sampling is performed within the pixel, and then these sampling points are projected onto the UV space to obtain the corresponding UV coordinates. Based on these UV coordinates, the texture features corresponding to each sampling point can be extracted by a pre-trained multi-resolution hash model for efficient search, and these texture features can be arranged, each texture feature containing a high-dimensional feature vector. Then, these texture features are averaged to obtain an average feature (e.g., miptexture feature). The average feature is then fed into a compact multi-layer perceptron (MLP) to derive the final pixel color, that is, to obtain a three-dimensional scene containing texture information. It should be noted that the entire optimization process can be driven by the L2 loss of the true color and the predicted color, which is not limited in this disclosure.

[0102] The Neural MipTexture network proposed in this embodiment combines deep learning technology and traditional MipMapping technology, reduces the jagged effect through multi-level hash model and multi-sampling technology processing, and ensures high fidelity and visual quality of texture synthesis. Its implementation steps may include:

[0103] Multi-level detail processing: A multi-resolution hashing model is used to generate texture features at different levels. These feature levels can better preserve and synthesize texture details by learning multi-scale features of the image.

[0104] Multi-sampling technology: Each pixel of the target image is uniformly sampled multiple times, and these multi-sampled sub-pixels are input into a multi-resolution hash model to obtain the corresponding texture features. The original pixel features are replaced by the average features of the sampled sub-pixels to achieve the goal of pre-filtering. The average features are then input into a trainable multi-layer perceptron to predict the final texture color.

[0105] Anti-aliasing: Through the multi-resolution hash model and multi-sampling technology, the anti-aliasing effect in the texture can be identified and reduced, which makes the generated texture edges smoother and more natural, improving the realism and detail of the texture.

[0106] In summary, the present disclosure is aimed at three-dimensional scenes. Through instance-level controllable texture synthesis, it uses instance layout to achieve precise semantic control of individual instances while maintaining overall style consistency, significantly improving the quality and efficiency of 3D scene texture synthesis and solving the challenges encountered in the prior art. In addition, the present disclosure also significantly improves the local consistency of texture and reduces the aliasing effect through a local synchronized multi-view diffusion strategy and neural MipTexture technology, bringing innovative solutions to the field of 3D scene texture synthesis.

[0107] Specifically, the present disclosure performs instance segmentation on the input scene grid, and constructs an instance layout based on the segmentation results, which includes the position (i.e., two-dimensional bounding box) and appearance style of each instance. In view of the fact that texture processing for each instance alone may affect the consistency of the scene style, an instance conditional image generation method based on a flexible image rendering framework can be adopted. In order to solve the common discontinuity problem in the repair stage, the image denoising process can be decomposed into two stages: instance-level denoising process and scene-level denoising process. The former realizes precise control of each instance, and the latter ensures the consistency of the overall style. In addition, the present disclosure innovatively integrates the local multi-view diffusion (LMVD) model into the image repair framework, and develops a texture refinement strategy that unifies the diffusion process between adjacent view images and synergizes denoising to enhance local consistency. Finally, in order to efficiently project the generated multi-view image back to the UV atlas, a neural MipTexture is also introduced, which is a neural multi-scale texture mapping algorithm specifically used to create high-fidelity texture maps. The neural MipTexture disclosed in the present disclosure is specially designed to solve the aliasing problem often encountered in large scene textures.

[0108] Through all the above embodiments, the advantages of the present disclosure are summarized as follows:

[0109] 1. Improved the quality of texture generation: Through the synchronous diffusion of multi-view images and the integration of multi-view image information, high-quality texture generation is achieved. Compared with traditional methods, the present invention has significant advantages in terms of texture details, color consistency, geometric consistency, etc.

[0110] 2. Expanded application scope: The generated high-quality texture images can be applied to the texture coverage of various 3D models, and are widely used in game development, virtual reality, film and television production and other fields. At the same time, multi-view synchronous diffusion can also be extended to other fields such as medical image processing and remote sensing image analysis, and has a wide range of application value.

[0111] 3. Improved processing efficiency: By optimizing the algorithm structure and processing flow, the present disclosure improves the processing efficiency of texture generation, and can generate high-quality texture images in a shorter time to meet the needs of practical applications.

[0112] 4. Enhanced robustness of the system: Through the comprehensive application of feature matching, feature fusion, diffusion processing and other steps, the present disclosure enhances the robustness of the system and can achieve stable texture generation effects between images of different perspectives.

[0113] The texture generation method of the three-dimensional scene of the embodiment of the present disclosure has important innovation and practicality in the field of texture generation, can significantly improve the generation efficiency and quality of the three-dimensional scene texture, solve the problems of local texture inconsistency, missing details and jagged effect in the prior art, can significantly improve the visual effect of the three-dimensional model, and meet the needs of game development, virtual reality, film and television production, digital display and other fields for high-quality textures. At the same time, the method of the present disclosure has a scientific and reasonable structure, a rigorous and efficient processing flow, and has broad application prospects and commercial value. Through the above detailed implementation scheme, the present disclosure not only achieves high-quality texture generation, but also provides a solid foundation for further research and development of the generation and application of high-quality three-dimensional scenes in the future.

[0114] The present invention has conducted experiments on a large number of indoor and outdoor scenes, and all of them show that compared with related texture generation technologies, the method disclosed in the present invention can generate high-quality and consistent textures, which is better than existing texture generation methods. The experimental results are compared. Figures 4 to 7 shown.

[0115] Figure 8 is a block diagram showing a texture generating device for a three-dimensional scene according to an embodiment of the present disclosure, Figure 8 As shown, the device includes a first acquisition unit 80 , a second acquisition unit 82 , a third acquisition unit 84 , a processing unit 86 and a generating unit 88 .

[0116] The first acquisition unit 80 is configured to acquire a scene mesh and a prompt text of a three-dimensional scene containing multiple instances, wherein the prompt text includes global texture information of the three-dimensional scene and texture information of each instance in the multiple instances; the second acquisition unit 82 is configured to perform instance segmentation on the scene mesh and acquire a three-dimensional directional bounding box of each instance; the third acquisition unit 84 is configured to acquire a two-dimensional instance layout and a two-dimensional orientation prompt map of each instance in each of a predetermined number of viewing angles based on the three-dimensional directional bounding box of each instance; the processing unit 86 is configured to perform the following processing for any of the predetermined number of viewing angles: input the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current viewing angle, and the acquired target image into a diffusion model to obtain a target image of the three-dimensional scene in the current viewing angle, wherein the target image includes the texture of the three-dimensional scene, the acquired target image is a target image of other viewing angles acquired before the target image of the current viewing angle, and the acquired target image is empty when acquiring the target image of the first viewing angle; the generation unit 88 is configured to generate a three-dimensional scene containing texture information based on all target images.

[0117] According to an embodiment of the present disclosure, the third acquisition unit 84 is also configured to obtain a depth map and a line map of the three-dimensional scene for each of a predetermined number of perspectives, after obtaining a two-dimensional instance layout and a two-dimensional orientation prompt map for each instance in each of a predetermined number of perspectives based on a three-dimensional directional bounding box for each instance; wherein the processing unit is further configured to input the prompt text, the two-dimensional instance layout of the current perspective, the two-dimensional orientation prompt map, the linear map and the depth map, and the acquired target image into the diffusion model to obtain the target image of the three-dimensional scene at the current perspective.

[0118] According to an embodiment of the present disclosure, the generation unit 88 is also configured to obtain potential images corresponding to the target images under a predetermined number of viewing angles before generating a three-dimensional scene containing texture information based on all target images; add noise to the potential images to obtain noisy potential images; and input the noisy potential images into a local multi-view diffusion model to obtain a refined target image.

[0119] According to an embodiment of the present disclosure, a local multi-view diffusion model performs a multi-step denoising process when processing a noisy latent image, wherein before each step of the denoising process, the following operations are performed: projecting the noisy latent image into a temporary texture space; determining an overlapping area of ​​latent images of different perspectives in the temporary texture space; weighting the parts of the latent images of different perspectives that fall in the overlapping area; and replacing a corresponding part of the latent image of the corresponding perspective with the weighted part.

[0120] According to an embodiment of the present disclosure, the generation unit 88 is further configured to, for each target image, after uniformly sampling in the target image, project the sampling points into the UV space to obtain the UV coordinates corresponding to each sampling point; input the UV coordinates into the hash model to obtain the texture features corresponding to each sampling point; average the texture features to obtain the average features of the target image; and input the average features of all target images into a multilayer perceptron to obtain a three-dimensional scene containing texture information.

[0121] According to an embodiment of the present disclosure, the diffusion model performs a multi-step denoising process when processing prompt text, a two-dimensional instance layout and a two-dimensional orientation prompt map of the current perspective, and an acquired target image, wherein in each step of the denoising process, the following operations are performed: when the current step of the denoising process is for the instance level, nearest neighbor sampling is performed in the latent space, wherein the latent space stores texture information of the instance, and the texture information of the instance is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective; the first latent image of the image is acquired, and the first latent image is denoised; based on the instance-level mask of the current instance, the first latent image and the first latent image after denoising, to the first image; when the current denoising process is for the scene level, bilinear sampling is performed through the RGB space, wherein the RGB space stores global texture information, and the global texture information is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective, wherein when the current denoising process is the first processing for the scene level, the image of the current perspective is the first image; a second latent image of the image is obtained, and the second latent image is denoised; a second image is obtained based on the scene level mask, the second latent image and the denoised second latent image; wherein, after executing the denoising process for a predetermined number of steps, the second image is used as the target image of the current perspective.

[0122] According to an embodiment of the present disclosure, before each step of the denoising process, it also includes: when the number of pixels contained in any instance of the three-dimensional scene after being projected to the current viewing angle exceeds a preset threshold, the texture information contained in the target image corresponding to any instance is only stored in the latent space.

[0123] According to an embodiment of the present disclosure, the third acquisition unit 84 is further configured to normalize the coordinates of the three-dimensional directional bounding box of each instance and project them to a predetermined number of viewing angles to obtain a two-dimensional bounding box of each instance at different viewing angles; based on the two-dimensional bounding box and prompt text of each instance, obtain a two-dimensional instance layout of each instance at different viewing angles; normalize the coordinates of the three-dimensional directional bounding box of each instance and convert them into RGB format, and project them to a predetermined number of viewing angles to obtain a two-dimensional orientation prompt map of each instance at different viewing angles.

[0124] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the texture generation method of the three-dimensional scene as described above. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0125] According to an embodiment of the present disclosure, a system is provided including at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the texture generation method for the three-dimensional scene as described above.

[0126] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions. When the computer instructions are executed by a processor, the texture generation method of the three-dimensional scene as described above is implemented.

[0127] Fig. 9 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present disclosure, such as Fig. 9As shown, the electronic device includes at least one memory 930 and at least one processor 910, wherein the at least one memory 930 is used to store a computer executable instruction set, and when the computer executable instruction set is executed by the at least one processor 910, the at least one processor 910 is prompted to execute any of the above-mentioned texture generation methods for three-dimensional scenes. The electronic device transmits data internally through a communication bus 940 and interacts with other devices through a communication interface 920.

[0128] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A texture generation method for a three-dimensional scene, characterized in that: include: Acquire a scene mesh and a prompt text of a three-dimensional scene including a plurality of instances, wherein the prompt text includes global texture information of the three-dimensional scene and texture information of each of the plurality of instances; Performing instance segmentation on the scene grid to obtain a three-dimensional oriented bounding box for each instance; Based on the three-dimensional directional bounding box of each instance, obtaining a two-dimensional instance layout and a two-dimensional orientation prompt map of each instance in each of a predetermined number of viewing angles; For any of the predetermined number of viewing angles, the following processing is performed: the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current viewing angle, and the acquired target image are input into a diffusion model to obtain a target image of the three-dimensional scene at the current viewing angle, wherein the target image includes a texture of the three-dimensional scene, and the acquired target image is a target image of other viewing angles acquired before the target image of the current viewing angle, and the acquired target image is empty when the target image of the first viewing angle is acquired; Based on all target images, the three-dimensional scene including texture information is generated.

2. The texture generation method according to claim 1, characterized in that: After obtaining a two-dimensional instance layout and a two-dimensional orientation hint map of each instance in each of a predetermined number of viewing angles based on the three-dimensional directional bounding box of each instance, the method further includes: Acquire a depth map and a line map of the three-dimensional scene at each of the predetermined number of viewing angles; The step of inputting the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current viewing angle, and the acquired target image into the diffusion model to obtain the target image of the three-dimensional scene at the current viewing angle includes: The prompt text, the two-dimensional instance layout of the current viewing angle, the two-dimensional orientation prompt map, the linear map and the depth map, and the acquired target image are input into the diffusion model to obtain the target image of the three-dimensional scene at the current viewing angle.

3. The texture generation method according to claim 1, characterized in that: Before generating the three-dimensional scene containing texture information based on all target images, the method further includes: Acquire potential images corresponding to the target images under the predetermined number of viewing angles; adding noise to the latent image to obtain a latent image containing noise; The noisy latent image is input into a local multi-view diffusion model to obtain a refined target image.

4. The texture generation method according to claim 3, characterized in that: The local multi-view diffusion model performs a multi-step denoising process when processing a noisy latent image, wherein before each step of the denoising process, the following operations are performed: Projecting the noisy latent image into a temporary texture space; Determining overlapping areas of potential images at different perspectives in the temporary texture space; weighting the parts of the latent images of different perspectives that fall in the overlapping area; The weighted part is used to replace the corresponding part of the latent image of the corresponding perspective.

5. The texture generation method according to claim 1, characterized in that: The step of generating the three-dimensional scene containing texture information based on all target images comprises: For each target image, after uniform sampling in the target image, the sampling points are projected into the UV space to obtain the UV coordinates corresponding to each sampling point; the UV coordinates are input into the hash model to obtain the texture features corresponding to each sampling point; the texture features are averaged to obtain the average features of the target image; The average features of all target images are input into a multilayer perceptron to obtain the three-dimensional scene containing texture information.

6. The texture generation method according to claim 1, characterized in that: The diffusion model performs a multi-step denoising process when processing the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current viewing angle, and the acquired target image, wherein in each step of the denoising process, the following operations are performed: When the current denoising process is for the instance level, nearest neighbor sampling is performed in the latent space, wherein the latent space stores texture information of the instance, and the texture information of the instance is determined based on the two-dimensional instance layout; the sampled texture information is projected onto the image of the current perspective; a first latent image of the image is obtained, and the first latent image is denoised; a first image is obtained based on the instance level mask of the current instance, the first latent image, and the first latent image after denoising; When the current denoising process is for the scene level, bilinear sampling is performed through the RGB space, wherein the RGB space stores global texture information, and the global texture information is determined based on the two-dimensional instance layout; the sampled texture information is projected to the image of the current perspective, wherein when the current denoising process is the first processing for the scene level, the image of the current perspective is the first image; a second latent image of the image is obtained, and the second latent image is denoised; a second image is obtained based on the scene level mask, the second latent image and the denoised second latent image; After executing a denoising process for a predetermined number of steps, the second image is used as the target image of the current viewing angle.

7. The texture generation method according to claim 6, characterized in that: Before each denoising process, it also includes: When the number of pixels contained in any instance of the three-dimensional scene after being projected to the current viewing angle exceeds a preset threshold, texture information contained in the target image corresponding to any instance is only stored in the latent space.

8. The texture generation method according to claim 1, characterized in that: The step of obtaining a two-dimensional instance layout and a two-dimensional orientation prompt map of each instance in each of a predetermined number of viewing angles based on the three-dimensional directional bounding box of each instance includes: Normalizing the coordinates of the three-dimensional oriented bounding box of each instance and projecting them to the predetermined number of viewing angles to obtain a two-dimensional bounding box of each instance at different viewing angles; Based on the two-dimensional bounding box and prompt text of each instance, a two-dimensional instance layout of each instance at different viewing angles is obtained; The coordinates of the three-dimensional directional bounding box of each instance are normalized and converted into RGB format, and projected to the predetermined number of viewing angles to obtain a two-dimensional orientation prompt map of each instance at different viewing angles.

9. A texture generation device for a three-dimensional scene, characterized in that: include: A first acquisition unit is configured to acquire a scene mesh and a prompt text of a three-dimensional scene including a plurality of instances, wherein the prompt text includes global texture information of the three-dimensional scene and texture information of each of the plurality of instances; A second acquisition unit is configured to perform instance segmentation on the scene grid and acquire a three-dimensional oriented bounding box of each instance; A third acquisition unit is configured to acquire a two-dimensional instance layout and a two-dimensional orientation prompt map of each instance in each of a predetermined number of viewing angles based on the three-dimensional directional bounding box of each instance; The processing unit is configured to perform the following processing for any one of the predetermined number of viewing angles: input the prompt text, the two-dimensional instance layout and the two-dimensional orientation prompt map of the current viewing angle, and the acquired target image into a diffusion model to obtain a target image of the three-dimensional scene at the current viewing angle, wherein the target image includes a texture of the three-dimensional scene, and the acquired target image is a target image of other viewing angles acquired before the target image of the current viewing angle, and the acquired target image is empty when the target image of the first viewing angle is acquired; The generating unit is configured to generate the three-dimensional scene containing texture information based on all target images.

10. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the texture generation method for a three-dimensional scene as claimed in any one of claims 1 to 9.

11. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the texture generation method for a three-dimensional scene according to any one of claims 1 to 9.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the texture generation method for a three-dimensional scene according to any one of claims 1 to 9 is implemented.

13. An electronic device, characterized in that: It comprises at least one memory and at least one processor, wherein the at least one memory is used to store a set of computer executable instructions, and when the set of computer executable instructions is executed by the at least one processor, it prompts the at least one processor to execute the texture generation method for a three-dimensional scene as claimed in any one of claims 1 to 9.

Images