Text-driven hierarchical panoramic 3d gaussian scene generation method and system

Abstract

The application provides a layered panoramic three-dimensional Gaussian scene generation method and system based on text driving, applied to the field of artificial intelligence technology. It includes: obtaining the equirectangular projection (ERP) panorama corresponding to the text prompt, performing instance segmentation on the ERP panorama, and extracting the object instance mask of the ERP panorama; for each object instance mask, predicting the object thickness field of the object instance mask corresponding to the object instance along the panoramic ray direction and the background gap field between the back of the object instance and the next surface along the panoramic ray direction through the mask-by-mask perspective query network; obtaining the visible surface depth of the ERP panorama along the panoramic ray direction, determining the multi-layer radial geometry corresponding to each object instance according to the visible surface depth, the object thickness field and the background gap field; and generating a layered panoramic three-dimensional Gaussian scene according to the multi-layer radial geometry. The method realizes the generation of a roamable panoramic three-dimensional scene with geometric consistency from a text prompt.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text-driven method and system for generating layered panoramic 3D Gaussian scenes. Background Technology

[0002] Text-based image generation and 3D content generation technologies are developing rapidly. Text-to-Image (T2I) models can generate high-quality 2D images based on natural language descriptions; 3D Gaussian Splatting (3DGS), as a novel explicit neural rendering representation, has attracted widespread attention due to its fast training speed, strong real-time rendering capabilities, and good geometric interpretability. In panoramic scene generation, existing methods mainly adopt the following two technical approaches. One approach is end-to-end text-to-3D representation generation. This type of method typically relies on large-scale 3D datasets to train the generation model, directly outputting point clouds, meshes, neural radiation fields, or 3DGS. Its advantage lies in its simple generation process, but it is limited by the scale and diversity of 3D training data, making it difficult to guarantee the geometric consistency and multi-view coherence of the generated results, and its generalization ability to complex outdoor / indoor scenes is limited. Another approach involves first generating an ERP panoramic image and then reconstructing the 3D scene. This method first uses a text-to-ERP panoramic image diffusion model to generate an isometric cylindrical projection ERP panoramic image, and then restores the 3D structure through monocular depth estimation, multi-view reconstruction, or reverse rendering. The advantage of this approach is that it fully utilizes the strong appearance prior of the 2D generation model and does not rely on large-scale 3D labeled data, thus becoming the mainstream technical route for panoramic 3D scene generation.

[0003] However, most existing 3D reconstruction methods based on ERP panoramic images focus on the restoration of visible surfaces, which leads to problems such as voids, stretching, and disordered occlusion relationships when the viewpoint is translated, thus limiting the visual quality of free roaming. Summary of the Invention

[0004] This application provides a text-driven layered panoramic 3D Gaussian scene generation method and system to solve the defects in the prior art caused by the lack of layered geometric modeling, such as holes, stretching and disordered occlusion relationships when the viewpoint moves. It realizes the automatic generation of panoramic 3D Gaussian scenes with correct occlusion relationships, continuous seams, geometric consistency and free roaming, starting from text prompts.

[0005] This application provides a text-driven method for generating layered panoramic 3D Gaussian scenes, including: Obtain the equidistant cylindrical projection ERP panoramic image corresponding to the text prompt, perform instance segmentation on the ERP panoramic image, and extract the mask of each object instance in the ERP panoramic image; For each object instance mask, the object thickness field along the panoramic ray direction of the object instance corresponding to the object instance mask, and the background gap field from the back of the object instance to the next surface along the panoramic ray direction are predicted by the mask-by-mask perspective query network. Obtain the visible surface depth of the ERP panoramic image along the panoramic light direction, and determine the multi-layer radial geometry corresponding to each object instance based on the visible surface depth, the object thickness field, and the background gap field; Based on the multi-layered radial geometry, a layered panoramic 3D Gaussian scene is generated.

[0006] This application also provides a text-driven hierarchical panoramic 3D Gaussian scene generation system, including: The panoramic instance segmentation module is used to obtain the equidistant cylindrical projection ERP panoramic image corresponding to the text prompt, perform instance segmentation on the ERP panoramic image, and extract the mask of each object instance in the ERP panoramic image. The perspective query and thickness prediction module is used to predict, for each object instance mask, the object thickness field of the object instance corresponding to the object instance mask along the panoramic light direction, and the background gap field from the back of the object instance to the next surface along the panoramic light direction through a mask-by-mask perspective query network. A multi-layer geometry synthesis module is used to obtain the visible surface depth of the ERP panoramic image along the panoramic light direction, and determine the multi-layer radial geometry corresponding to each object instance based on the visible surface depth, the object thickness field and the background gap field. The 3D Gaussian scene generation module is used to generate a layered panoramic 3D Gaussian scene based on the multi-layer radial geometry.

[0007] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the text-driven layered panoramic 3D Gaussian scene generation method described above.

[0008] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the text-driven layered panoramic 3D Gaussian scene generation method as described above.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the text-driven layered panoramic 3D Gaussian scene generation method described above.

[0010] The text-driven layered panoramic 3D Gaussian scene generation method and system provided in this application embodiment obtains an isometric cylindrical projection ERP panoramic image corresponding to the text prompt, performs instance segmentation on the ERP panoramic image, and extracts the masks of each object instance in the ERP panoramic image; for each object instance mask, a mask-by-mask perspective query network is used to predict the object thickness field along the panoramic light direction of the object instance corresponding to the object instance mask, and the background gap field from the back of the object instance to the next surface along the panoramic light direction; the visible surface depth of the ERP panoramic image along the panoramic light direction is obtained, and multi-layer radial geometry corresponding to each object instance is determined based on the visible surface depth, the object thickness field, and the background gap field; and a layered panoramic 3D Gaussian scene is generated based on the multi-layer radial geometry. This method generates an ERP panoramic image from text and decomposes it into independent object instances. For each instance, it predicts its thickness field and background gap field, achieving explicit quantization of object depth and occlusion background. It expands the originally flat, single-layer ERP panoramic image into a multi-layered radial geometry with a complete radial layering structure, and transforms this layered geometry into a 3D Gaussian splash scene. This results in a 3D scene that retains correct occlusion layering relationships while possessing real-time rendering performance. Starting from text prompts, this method can automatically generate a geometrically consistent, roamable panoramic 3D scene, effectively solving problems such as voids, stretching, and occlusion relationship errors caused by the lack of layered geometric modeling in existing technologies. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the text-driven layered panoramic 3D Gaussian scene generation method provided in this application embodiment; Figure 2 This is a schematic diagram illustrating a multiple longitude rollover example provided in an embodiment of this application; Figure 3 This is a schematic diagram of the text-to-panorama generation and sky removal process provided in the embodiments of this application; Figure 4 This is a schematic diagram of the input and output of the mask-by-mask perspective query network provided in an embodiment of this application; Figure 5 This is a schematic diagram of ERP seam alignment and complementary cosine fusion provided in the embodiments of this application; Figure 6This is a schematic diagram of the three-layer radial geometry construction provided in the embodiments of this application; Figure 7 This is a schematic diagram of the adaptive Gaussian compaction principle provided in the embodiments of this application; Figure 8 These are comparison diagrams showing the effects of the embodiments provided in this application; Figure 9 This is a schematic diagram of the structure of the text-driven layered panoramic 3D Gaussian scene generation system provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] To better understand the embodiments of this application, the prior art will first be described in detail: Existing 3D reconstruction methods based on ERP panoramic images effectively utilize mature 2D generation technologies, but still face a series of inherent and unresolved key technical challenges: First, existing reconstruction methods essentially only model the visible first surface in the scene, lacking explicit modeling of the back of objects, occluded areas, and the background surface exposed after object removal. This leads to geometric holes, incorrect stretching, or content inconsistencies in newly exposed areas during viewpoint translation. This is essentially due to the lack of ability to decompose and reconstruct the complete radial geometry, such as object thickness, gaps between objects, and background layers.

[0015] Second, existing convolutional neural network or Transformer architectures treat the left and right boundaries as discontinuous edges of the image when performing depth estimation and semantic segmentation. This leads to drastic numerical jumps in the predicted depth map and instance mask at the seams. After being reconstructed into 3D geometry, this jump manifests as a noticeable and visually unacceptable geometric seam.

[0016] Third, in panoramic scenes, areas such as the sky, distant mountains, and atmosphere should be physically considered to be infinitely far from the viewpoint. Existing technologies often force the use of monocular depth estimation networks to predict a "finite but large" depth value for these areas and participate in 3D reconstruction. This improper handling introduces two problems: First, it generates a large amount of meaningless geometric noise for these large areas, wasting computational resources; second, when the viewpoint moves, these infinitely distant backgrounds that should remain stationary will generate parallax, causing serious visual distortion and loss of quality.

[0017] Fourth, existing methods often employ uniform or image gradient-based sampling strategies when initializing Gaussian primitives. However, for surfaces far from the viewpoint or surfaces close to the tangent direction of the viewing direction, the actual 3D area covered by a single pixel is larger. In these regions, sparse Gaussian sampling density is insufficient to provide adequate Gaussian primitive density, affecting rendering quality.

[0018] To address the aforementioned technical issues, this application proposes a text-driven, layered panoramic 3D Gaussian scene generation and system. By constructing multi-layered radial geometry of visible surfaces, object back surfaces, and occluded background surfaces, it correctly reveals newly exposed areas during viewpoint translation. Through multiple longitude roll-through fusion and complementary cosine weighting strategies, it eliminates depth, feature, and mask jumps at ERP boundaries. By combining semantic segmentation and depth confidence analysis, it removes infinitely distant background regions and completes a dynamic sky sphere around the scene, achieving a reasonable separation between the internal realistic geometry and the external infinitely distant background. Through an adaptive densification strategy based on projection sampling frequency, it dynamically supplements Gaussian primitives in areas with greater depth or surfaces close to the tangent direction, improving rendering quality. The combined effect of these techniques enables this application to automatically generate a 360° panoramic 3D Gaussian scene with correct occlusion relationships, continuous seams, consistent geometry, and free roaming capabilities, starting from text prompts.

[0019] It should be noted that the execution entity involved in the embodiments of this application can be a text-driven layered panoramic 3D Gaussian scene generation system or an electronic device. Optionally, the electronic device can be a server, computer, embedded device, robot controller, or a model training platform deployed in the cloud.

[0020] The following section uses an electronic device as an example to illustrate in detail the text-driven layered panoramic 3D Gaussian scene generation method provided in this application: Figure 1 This is a flowchart illustrating the text-driven, layered panoramic 3D Gaussian scene generation method provided in this application embodiment. Figure 1 As shown, the method includes the following steps 101-104.

[0021] Step 101: Obtain the equidistant cylindrical projection ERP panoramic image corresponding to the text prompt, perform instance segmentation on the ERP panoramic image, and extract the mask of each object instance in the ERP panoramic image.

[0022] Text prompts refer to natural language descriptions entered by the user, which are one or more paragraphs of text describing the expected 3D scene content to be generated.

[0023] Optionally, the text prompt may include descriptions of attributes such as the type, quantity, spatial layout, material, lighting, weather conditions, and time of objects in the scene, and its specific form and content do not constitute a limitation of the embodiments of this application. For example, a user may input "a modern living room with a city night view outside the floor-to-ceiling windows, a sofa, coffee table, and carpet" or "a busy city street with tall buildings on both sides and a clear sky," etc.

[0024] It should be noted that the text prompt can be a complete sentence or phrase entered by the user through the graphical user interface. The text prompt can also be selected and modified by the user from a preset prompt template library; this application does not impose specific limitations on this.

[0025] An equirectangular projection (ERP) panorama is a two-dimensional rectangular image formatted by unfolding a 360° spherical space using an equirectangular projection method. In ERP format, the width of the panorama corresponds to the 360° longitude range in the horizontal direction of the sphere, and the height corresponds to the 180° latitude range in the vertical direction of the sphere.

[0026] It should be noted that the default resolution of the ERP panoramic image is, for example, 1920×960 pixels, but the specific resolution can be adjusted according to the actual application scenario and computing resources, which does not constitute a limitation on the scope of protection of this application.

[0027] In this embodiment, the electronic device first receives a text prompt input by the user and generates a panoramic image in ERP format using a pre-trained text-to-ERP panoramic image generation model. Subsequently, the electronic device uses a pre-trained instance segmentation model to perform instance-level semantic segmentation on each independent object region in the ERP panoramic image, obtaining a mask set for each object instance in the scene. This step transforms the user-input natural language description into a two-dimensional panoramic image, providing rich appearance priors for subsequent 3D reconstruction; simultaneously, it decomposes the ERP panoramic image into independent object instance masks, providing semantic units for subsequent object-by-object layered geometric modeling, enabling each object to be processed independently, thereby supporting refined reconstruction of object thickness and occluded backgrounds.

[0028] It should be noted that the text-to-panorama generation models (such as HY-Pano, HY-World2.0, etc.) and instance segmentation models (such as OneFormer, etc.) used in this step are all existing publicly available pre-trained models. Those skilled in the art can replace them with other publicly available or self-developed models with equivalent functions according to actual deployment needs.

[0029] Step 102: For each object instance mask, predict the object thickness field of the object instance corresponding to the object instance mask along the panoramic ray direction, and the background gap field from the back of the object instance to the next surface along the panoramic ray direction using the mask-by-mask perspective query network.

[0030] Among them, the mask-by-mask perspective query network refers to a neural network model that performs independent perspective queries on a single object instance mask. It is used to predict the geometric structural parameters of the object instance in the depth direction by regressing pixel by pixel based on the image content, geometric features and light direction within the mask area.

[0031] The object thickness field refers to the spatial field formed by the radial distance along the panoramic ray direction corresponding to each pixel within the mask of each object instance, from the position where the ray first hits the surface of the object instance to the position where the ray exits from the back of the object instance.

[0032] Background gap field refers to the spatial field formed by the radial distance between the position where the light ray exits the back of the object instance and the first time it hits the surface of the next non-object instance along the panoramic light ray direction corresponding to that pixel within the mask of each object instance.

[0033] It is understood that "the next surface that is not an instance of this object" includes, but is not limited to: the visible surface of another object instance, the visible surface of a static background in the scene, and the surfaces of fixed structures such as the ground and walls in the scene. When there are no other surfaces in the same panoramic ray direction, the background gap field at that pixel location can be marked as an invalid value.

[0034] In this embodiment, after obtaining the masks of each object instance in the ERP panoramic image, the electronic device performs pixel-by-pixel dense prediction within the mask for each object instance mask using a pre-trained per-mask perspective query network, predicting the object thickness field and background gap field corresponding to that object instance. This step, through a per-mask perspective query method, achieves explicit modeling of the thickness of each independent object in the panoramic scene and the occluded background behind it for the first time, providing a core data foundation for the geometric consistency reconstruction of newly exposed areas when the viewpoint moves.

[0035] Step 103: Obtain the visible surface depth of the ERP panoramic image along the panoramic light direction. Based on the visible surface depth, object thickness field, and background gap field, determine the multi-layer radial geometry corresponding to each object instance.

[0036] The visible surface depth refers to the radial distance from the observation origin (i.e., the center of the viewpoint) to the scene surface where the light first hits, along the panoramic light direction corresponding to each pixel in the ERP panoramic image.

[0037] It should be noted that, in this embodiment, all depth values ​​related to the panoramic light direction are defined as the radial distance from the observation origin along the panoramic light direction to the corresponding surface intersection point, rather than the Z-axis depth under the perspective camera. This definition ensures consistency between the ERP format and spherical geometry, making the left and right boundaries of the panoramic image physically adjacent, and allowing direct algebraic comparison and superposition of depth values ​​at each layer.

[0038] Multi-layer radial geometry refers to the radial decomposition of a scene into multiple surface layers with different semantic meanings along the same panoramic light direction. The surfaces of each layer are arranged in order from near to far, and each layer is represented by a radial depth value to indicate its spatial position.

[0039] In this embodiment, the electronic device uses a pre-trained monocular depth estimation network to predict the depth of the ERP panoramic image, obtaining dense visible surface depth. After obtaining the visible surface depth, the object thickness field of each object instance, and the background gap field, the electronic device algebraically superimposes the visible surface depth, object thickness field, and background gap field along the same panoramic ray direction for each object instance, sequentially determining the spatial position of each layer surface in the multi-layer radial geometry corresponding to that object instance. This step, by superimposing the visible surface depth with the instance-by-instance predicted thickness field and gap field, expands the ERP panoramic image, which originally only contained the outermost surface, into a complete layered geometric structure including the front surface of the object, the back surface of the object, and the occluded background surface, laying the geometric foundation for the subsequent generation of a 3D scene with correct occlusion relationships.

[0040] Step 104: Generate a layered panoramic 3D Gaussian scene based on the multi-layered radial geometry.

[0041] In this embodiment, after determining the multi-layer radial geometry corresponding to each object instance, the electronic device converts each layer of radial geometry into a 3D Gaussian primitive, merges all 3D Gaussian primitives, and obtains a layered panoramic 3D Gaussian scene. This step transforms the layered geometric structure into a 3D Gaussian splash representation that can be rendered in real time, so that the generated 3D scene retains the correct occlusion layering relationship and has efficient rendering performance, supporting users to freely roam and browse.

[0042] In this embodiment, the technical solutions of steps 101-104 transform abstract natural language text into an ERP panoramic image containing rich appearance information, and use panoramic instance segmentation to decompose the scene from the whole image level into independent object semantic units, providing processing objects for subsequent object-by-object refined geometric modeling; for each object instance mask, the thickness field and background gap field of the object instance are independently predicted using a per-mask perspective query network, realizing for the first time explicit quantification of the depth size of each object in the panoramic scene and the background space it occludes, filling the gap of existing technology that only models the visible surface; by algebraically superimposing the visible surface depth with the thickness field and gap field predicted by the above per-instance along the light direction, a multi-layer radial geometry containing the visible surface of the object, the back surface of the object, and the background surface exposed after removing the object is constructed, expanding the originally flat single-layer ERP panoramic image into a geometric representation with a complete radial layered structure; by transforming this layered geometric structure into a three-dimensional Gaussian splash scene, the generated three-dimensional scene retains the correct occlusion layering relationship and has real-time rendering performance. This method can automatically generate a roamable panoramic 3D scene with geometric consistency based on text prompts, effectively solving problems such as spatial holes, stretching, and disordered occlusion relationships caused by the lack of layered geometric modeling in existing technologies.

[0043] In some embodiments, after the electronic device acquires the equidistant cylindrical projection ERP panoramic image corresponding to the text prompt, it further includes: performing a left-right RGB continuity check on the ERP panoramic image.

[0044] In this embodiment, the electronic device extracts the RGB color values ​​of the left and right edge regions of the ERP panoramic image and calculates the color difference metric between corresponding pixel rows on the left and right sides. When the color difference metric is less than or equal to a preset threshold, the ERP panoramic image is determined to have passed the continuity check and directly enters the subsequent processing flow. When the color difference metric is greater than the preset threshold, the ERP panoramic image is determined to have color discontinuity at the seams.

[0045] Optionally, for an ERP panorama that fails the continuity check, the electronic device may perform any of the following processing methods: Method 1: Repair the seam blending of the ERP panoramic image, for example, by using color interpolation or Poisson blending of the left and right boundary pixels to smooth the RGB transition of the seam area; Method 2: Re-invoke the text to the panoramic image generation model, change the random seed or adjust the generation parameters to regenerate the ERP panoramic image, until the newly generated ERP panoramic image passes the continuity check.

[0046] It is understood that the above processing method is merely an illustrative example, and those skilled in the art can use other methods to repair or redraw the seam area according to actual needs. This application does not make any specific limitations in this regard.

[0047] It should be noted that by verifying and repairing the seam quality of the input ERP panoramic image before 3D reconstruction, the RGB jump of the ERP boundary caused by the inherent defects of the generated model is eliminated. This provides consistent input data for subsequent depth prediction, instance segmentation and hierarchical geometry construction, further improving the visual coherence of the final 3D scene during 360° roaming.

[0048] In some embodiments, the step of the electronic device performing instance segmentation on the ERP panoramic image and extracting the object instance masks of each object in the ERP panoramic image may include: the electronic device rolling the ERP panoramic image along the horizontal longitude direction by a preset pixel distance to obtain multiple longitude-offset ERP panoramic images; the electronic device performing instance segmentation on each longitude-offset ERP panoramic image to obtain the initial object instance mask corresponding to each longitude-offset ERP panoramic image; the electronic device rolling the initial object instance mask corresponding to each longitude-offset ERP panoramic image in reverse to the original image coordinate system to obtain candidate instance masks in a unified coordinate system; the electronic device calculating the overlap between each candidate instance mask based on the intersection-union ratio, classifying candidate instance masks with an overlap greater than a preset instance matching threshold into the same object instance and performing mask fusion to obtain the object instance masks of each object in the ERP panoramic image.

[0049] The preset pixel distance refers to the number of offset pixels used to cyclically scroll the ERP panoramic image along the longitude direction.

[0050] Understandably, the value of this pixel distance should be such that the left and right boundary regions of the original ERP panoramic image are shifted into the image center region in at least one longitude-shifted panoramic image. For example, the preset pixel distance can be set to W / 3 or W / 2, where W is the width in pixels of the ERP panoramic image.

[0051] Multiple longitude offset ERP panoramic images refer to multiple panoramic images obtained by cyclically scrolling the original ERP panoramic image along the horizontal direction at different preset pixel distances. Each longitude offset panoramic image maintains the ERP format unchanged, but the image content undergoes an overall cyclic displacement in the horizontal direction.

[0052] The initial object instance mask refers to the object instance segmentation result obtained in the coordinate system of each offset image after performing instance segmentation on each longitude offset ERP panoramic image. This initial object instance mask records the object instance category and number to which each pixel belongs in the offset image.

[0053] Candidate instance mask refers to the segmentation prediction result of the same object instance in different longitude offset versions obtained under a unified coordinate system after the initial object instance mask corresponding to each longitude offset ERP panoramic image is rolled backward to the original image coordinate system.

[0054] The preset instance matching threshold is the intersection-union ratio threshold value used to determine whether two candidate instance masks belong to the same object instance.

[0055] For example, the instance matching threshold can be set to 0.5 or 0.6, and the specific value can be adjusted according to the actual segmentation accuracy and scene complexity. This application does not make any specific limitation in this regard.

[0056] In this embodiment, the electronic device rolls the ERP panoramic image along the horizontal longitude direction by different preset pixel distances to obtain multiple longitude-shifted ERP panoramic images. This moves objects located near the left and right boundaries in the original image to the center region of the image in each shifted image. Then, the electronic device uses a pre-trained instance segmentation model to perform instance segmentation on each longitude-shifted panoramic image, obtaining prediction results for multiple sets of initial object instance masks. Next, the electronic device rolls each set of initial object instance masks back to the original image coordinate system, so that all prediction results are in the same coordinate space, obtaining candidate instance masks in a unified coordinate system. Finally, the electronic device calculates the overlap between each candidate instance mask based on the intersection-union ratio (IU). Candidate instance masks with an overlap greater than the instance matching threshold are determined to be the same object instance, and multiple candidate masks belonging to the same object instance are merged to obtain the complete mask of each object instance in the ERP panoramic image. Conversely, candidate instance masks with an overlap less than or equal to the preset threshold are determined to belong to different object instances and are each retained as an independent instance mask.

[0057] For example, Figure 2 This is a schematic diagram illustrating a multiple longitude rollover example provided in an embodiment of this application. Combined with... Figure 2 The instance segmentation model is The scrolling operation is (·), for offsets Δ∈{0,W / 3,2W / 3}, the predicted mask is , This represents the original ERP overview.

[0058] Electronic devices can scroll the original ERP panoramic image horizontally by 0, W / 3, and 2W / 3 pixels respectively to obtain three longitude-shifted ERP panoramic images. For example... Figure 2As shown, the original ERP panoramic image contains a vehicle spanning the left and right boundaries. In the original panoramic image scrolled to 0 pixels, the vehicle is truncated into two separate segments by the left and right boundaries, and direct instance segmentation cannot identify them as the same object. After scrolling the original image horizontally by W / 3 pixels, the vehicle segment, originally located near the right boundary of the original image, is moved to the center of the image, and the segmentation model can accurately identify its complete outline. After scrolling the original image by 2W / 3 pixels, the vehicle segment, originally located near the left boundary of the original image, is also moved to the center of the image, and the segmentation model can accurately identify the complete outline of this segment.

[0059] Subsequently, the electronic device performs instance segmentation on each longitude-shifted panoramic image, obtaining three sets of initial object instance masks. These masks are then rolled back to the original image coordinate system to obtain candidate instance masks in a unified coordinate system. Finally, the electronic device performs matching and merging of the candidate instance masks based on intersection-union comparison (IoU) to obtain the final instance segmentation result. ,in, This represents the preset instance matching threshold. When the overlap ratio (cross-union ratio) of two candidate instance masks is greater than... At that time, they are grouped into the same object instance and masked for blending. For example... Figure 2 As shown, a vehicle that was originally truncated into two segments by the left and right boundaries of the ERP (Emergency Partition Image) is correctly identified as a complete vehicle instance after fusion. Simultaneously, objects such as buildings and trees located in the central area of ​​the ERP can be stably segmented in all offset maps, and their mask quality is further enhanced after fusion. Through this method, semantic continuity at the left and right boundaries is maintained, providing accurate and reliable mask input for subsequent object-by-object layered geometric modeling. This effectively solves the instance fragmentation problem at the ERP boundaries and improves the accuracy and completeness of panoramic instance segmentation.

[0060] It is understood that the above-mentioned scrolling pixel distances W / 3 and 2W / 3 are merely illustrative examples. Those skilled in the art can select other offsets and offset quantities based on the object size and boundary distribution in the actual scene, as long as the objects near the left and right boundaries of the original ERP are moved to the center region of the image in at least one offset image. This application does not impose any specific limitations on this.

[0061] It should be noted that by using the above-mentioned longitude roll-over fusion strategy, the same object at the left and right boundaries of the ERP can be completely segmented and accurately fused under different offset perspectives. This effectively eliminates the problem of instance breakage and identity confusion caused by spherical unfolding, improves the accuracy of panoramic instance segmentation, and provides reliable semantic units for subsequent object-by-object layered geometric modeling.

[0062] Optionally, after obtaining the complete masks of each object instance, for instance masks with low confidence or blurred boundaries in some segmentation results, interactive segmentation models can be used to refine the instance masks. For example, interactive segmentation models such as Segment Anything Model 3 (SAM3) can be used to refine the masks manually or automatically using bounding box hints or point hints to improve the precision of the mask boundaries.

[0063] Understandably, after obtaining all instance masks, the morphological operations, connected component analysis, bounding box calculations, and spherical centroid calculations performed on the masks all employ horizontal circular indexing for pixel addressing. When performing horizontal pixel indexing operations on the ERP panoramic image, horizontal circular indexing treats the left and right boundaries of the image as adjacent; that is, when the index exceeds the left boundary, it continues indexing from the right boundary; when the index exceeds the right boundary, it continues indexing from the left boundary. This horizontal circular indexing method ensures the topological correctness of adjacent pixels at the left and right boundaries of the ERP, avoiding errors in masking results caused by boundary truncation.

[0064] In some embodiments, before the step of performing instance segmentation on the ERP panoramic image, the electronic device may further include: performing semantic segmentation on the ERP panoramic image, identifying the sky and / or the infinitely distant background region, and generating a first mask; the electronic device marks the region corresponding to the first mask as an invalid region, and excludes the invalid region when predicting the visible surface depth, object thickness field and background gap field.

[0065] Among them, the sky and / or infinitely distant background area refers to the background area in the ERP panoramic image that is approximately infinitely distant or has unstable depth, such as the sky, distant clouds, fog, distant mountains, and atmospheric conditions.

[0066] In this embodiment of the application, before the electronic device performs instance segmentation and 3D reconstruction on the ERP panoramic image, it first identifies and removes invalid regions in the panoramic image. For example, Figure 3 This is a schematic diagram of the text-to-panorama generation and sky removal process provided in an embodiment of this application. Combined with... Figure 3 The electronic device first receives a text prompt from the user, namely "sunny city street, panoramic view, modern buildings, intersection, pedestrians and vehicles," and uses a pre-trained text-to-panorama generation model to generate a panoramic image in ERP format. Then, the electronic device uses a semantic segmentation model to perform semantic segmentation on the generated panoramic image, identifying the sky region, i.e., the sky and / or the infinitely distant background region. Next, the electronic device extracts the first mask corresponding to the sky and / or infinitely distant background region based on the semantic segmentation results, such as... Figure 3As shown in the "Extract Sky Mask" step, the sky area is marked in black, while non-sky areas are retained as valid regions. Finally, the electronic device uses the panoramic image after sky removal as input for subsequent instance segmentation and 3D reconstruction, as shown in the example. Figure 3 As shown in the "panoramic view after sky removal", the sky area is processed as transparent or marked as invalid. Subsequent depth prediction, object thickness prediction and background gap prediction automatically exclude this area, and only the valid mid-to-foreground areas are geometrically reconstructed.

[0067] Optionally, after marking the area corresponding to the first mask as an invalid area, the method may further include: setting the depth value at each pixel position in the ERP panorama corresponding to the first mask to a preset invalid value or retaining it as an infinite semantic label.

[0068] Among them, the infinity semantic tag is used to identify that the pixel location belongs to an infinitely distant background area such as the sky, distant mountains, or atmosphere, rather than the real three-dimensional geometric surface of the foreground or midground.

[0069] Understandably, in subsequent processing, for pixel locations marked with an infinity semantic label, the visible surface depth can be set to a preset invalid value or retained as an infinity semantic label and passed to subsequent steps. The invalid value or infinity semantic label is used to indicate that this location does not participate in the generation of the visible surface Gaussian primitive during the subsequent 3D Gaussian primitive generation process, while the background color in that direction is provided by the outer sky sphere during the rendering stage. This processing method ensures that the sky-culled area is clearly marked in the depth map, avoiding geometric noise caused by missing depth values ​​or default value filling, and providing semantic-level identification for the seamless integration of the final sky sphere rendering with the internal geometry.

[0070] It should be noted that by removing infinitely distant background areas such as the sky before 3D reconstruction, geometric noise and viewpoint distortion caused by the incorrect prediction of infinitely distant backgrounds as having a specific finite depth are avoided. This allows the subsequent object thickness prediction network to focus only on the near and mid-range areas with actual 3D structures, reducing the learning difficulty and inference burden of the network. At the same time, it leaves room for processing when finally completing the sky sphere externally, achieving a reasonable separation of "real internal geometry and infinitely distant external background", effectively improving the visual quality and geometric consistency of the final panoramic 3D scene.

[0071] In some embodiments, after marking the area corresponding to the first mask as an invalid area, the electronic device may further include: performing connectivity analysis on the valid areas in the ERP panorama, where the valid areas are the areas in the ERP panorama excluding the invalid areas; and when an isolated fragment area smaller than a preset area threshold is detected within the valid areas, merging or repairing the isolated fragment areas.

[0072] The effective region refers to all pixel regions in the ERP panoramic image after removing the invalid regions corresponding to the first mask. This effective region includes objects and surfaces with actual three-dimensional structures in the scene and is the set of pixels that need to participate in subsequent three-dimensional reconstruction.

[0073] Connectivity analysis refers to the identification and area statistics of each connected component in an effective region to determine whether there are isolated fragments with excessively small areas.

[0074] Isolated fragment regions refer to scattered pixel blocks within the valid area that are spatially disconnected from the main valid area or connected only by extremely narrow areas after large areas of sky have been removed. These fragment regions are typically located at the boundary between the sky boundary and the top of an object, such as the top of a tree canopy or the edge of a building roof.

[0075] The preset area threshold is the minimum area threshold used to determine whether a connected region is an isolated fragment.

[0076] For example, the preset area threshold can be set to 0.01% to 0.05% of the total number of pixels in the panoramic image, or it can be adaptively adjusted according to the actual scene resolution and object size. This application does not make any specific limitation on this.

[0077] In this embodiment, after the electronic device marks the sky and / or the infinitely distant background area as invalid regions, the effective region may become fragmented due to large-area removal. To address this issue, the electronic device performs connectivity analysis on the effective region after removing invalid regions, identifies all connected components in the effective region, and calculates the pixel area of ​​each connected component. When an isolated fragment region with an area smaller than a preset area threshold is detected, the electronic device merges the isolated fragment region into its adjacent largest connected component, or performs morphological closing operations to repair it, so that the pixels are reintegrated into the effective region.

[0078] It should be noted that by performing connectivity analysis and fragment repair on the effective region after removing the sky, the problem of instance edge fragmentation caused by large-scale deletion of the sky region can be avoided, ensuring the geometric integrity of the effective region.

[0079] Optionally, after merging or repairing isolated fragment regions, the electronic device also includes feathering the mask edges of the effective regions. By feathering the mask edges, the boundary between the effective and ineffective regions is smoothly transitioned, eliminating geometric abrupt changes and seam artifacts caused by hard edges, and ensuring the continuity and visual smoothness of boundary alignment and layered geometry construction.

[0080] In some embodiments, before marking the area corresponding to the first mask as an invalid area, the electronic device may further include: acquiring an initial depth map of the ERP panoramic image, identifying pixel areas with depth values ​​greater than a preset depth value threshold and / or depth confidence values ​​lower than a preset confidence threshold based on the initial depth map, and generating a second mask; the electronic device merging the first mask and the second mask to obtain a culling mask; and the electronic device marking the area corresponding to the culling mask as an invalid area.

[0081] The initial depth map refers to the preliminary depth map obtained after using a pre-trained multi-task feature extraction backbone network to predict the depth of the ERP panorama.

[0082] Optionally, the multi-task feature extraction backbone network can adopt publicly available pre-trained depth estimation networks such as UniSHARP and UniK3D, whose output is a depth map with the same resolution as the input ERP panoramic image. Each pixel value represents the radial distance of that location along the light direction. This application does not limit the specific multi-task feature extraction backbone network model used.

[0083] The preset depth threshold is used to determine whether a pixel belongs to the depth threshold of the infinite background.

[0084] For example, the depth threshold can be set as a multiple of the depth of the nearest object in the scene (e.g., 10 times) or dynamically determined based on the depth distribution statistics of the panoramic image. This application does not impose any specific limitations on this.

[0085] A preset confidence threshold is used to determine the confidence threshold for whether the depth prediction results are reliable.

[0086] For example, the confidence threshold can be set to 0.5, and the specific value can be adjusted according to the confidence distribution characteristics of the actual depth estimation model. This application does not impose any specific limitations on this.

[0087] In this embodiment, before directly marking the first mask (i.e., the sky / infinite background region obtained from semantic segmentation) as an invalid region, the electronic device can further identify and supplement the invalid region by combining depth information. Specifically, the electronic device uses a pre-trained monocular depth estimation model to predict the depth of the ERP panoramic image to obtain an initial depth map; then, according to a preset depth value threshold, the electronic device identifies pixel regions with depth values ​​greater than the preset depth value threshold as potential infinite background; at the same time, the electronic device can also identify pixel regions with depth confidence values ​​lower than the preset confidence threshold as potential rejection regions according to a preset confidence threshold; the electronic device merges the above two regions into a second mask; subsequently, the electronic device merges the first mask and the second mask to obtain the final rejection mask; finally, the electronic device marks the region corresponding to the rejection mask as an invalid region, so that the invalid region is automatically excluded in subsequent prediction steps.

[0088] It is understood that the above methods can identify based solely on depth thresholds, solely on confidence thresholds, or simultaneously on both depth and confidence thresholds. Those skilled in the art can flexibly choose any one or more of the above determination methods based on the distribution of the sky and the infinitely distant background in the actual scene; this application does not impose any specific limitations on this.

[0089] It should be noted that by combining semantic segmentation with deep confidence analysis, a dual verification method can be used to more comprehensively and accurately eliminate infinitely far and unreliable regions, improving the robustness and recall of invalid region identification and providing cleaner input data for subsequent 3D reconstruction.

[0090] In some embodiments, the step of the electronic device predicting, for each object instance mask, the object thickness field along the panoramic ray direction and the background gap field from the back of the object instance to the next surface along the panoramic ray direction using a mask-by-mask perspective query network may include: determining the centroid position of the object instance mask in the spherical coordinate system of the ERP panoramic image; horizontally rotating the ERP panoramic image according to the centroid position to obtain a rotated ERP panoramic image; acquiring the multi-scale visual features and ray features corresponding to the ERP panoramic image, and projecting the rotated ERP panoramic image, multi-scale visual features, ray features, object instance mask, and visible surface depth onto a pinhole cropped view to obtain an aligned multi-channel perspective view; and inputting the multi-channel perspective view into the mask-by-mask perspective query network to determine the object thickness field and background gap field corresponding to the object instance mask.

[0091] The centroid position refers to the weighted center position of the object instance mask in the ERP spherical coordinate system, which is used to represent the approximate orientation of the object instance on the sphere.

[0092] It should be noted that since the left and right boundaries of the ERP are adjacent in physical space, the centroid calculation needs to use a horizontal circular index to avoid the centroid position shifting to the image edge due to boundary truncation, which would affect the accuracy of subsequent rotation and alignment.

[0093] Multi-scale visual features refer to visual feature maps with different receptive field sizes extracted from ERP panoramic images. These multi-scale visual features include at least low-level detail features such as edges and textures, and high-level semantic features such as object categories and structural information, which are used to provide rich image content priors for the mask-by-mask perspective query network.

[0094] Optionally, the multi-scale visual features can be extracted by sampling the shared encoder of the pre-trained multi-task feature extraction backbone network. The shared encoder outputs multiple feature maps at different resolution levels. The spatial resolution of each feature map decreases step by step, and the number of channels increases step by step, forming a pyramid-shaped multi-scale feature representation. While outputting the above initial depth map, the multi-task feature extraction backbone network outputs its intermediate layer feature maps as the multi-scale visual features.

[0095] Light features refer to the three-dimensional spatial light direction information corresponding to each pixel in the ERP panoramic image, including the longitude and latitude angles of the light in the spherical coordinate system.

[0096] Alternatively, this light characteristic can be obtained through one of the following methods: Method 1: Calculated based on the mapping relationship between pixel coordinates and spherical coordinates of the ERP panoramic image. Let the pixel coordinates of the ERP panoramic image be... Its corresponding spherical latitude and longitude are Then the direction of the light ray corresponding to that pixel is Therefore, the ray direction vector corresponding to each pixel can be calculated. Method 2: The ray generation branch of the pre-trained multi-task feature extraction backbone network directly outputs the ray generation feature. The multi-task feature extraction backbone network has a built-in universal ray representation module applicable to any camera model, which can directly calculate and output the ray direction feature corresponding to each pixel based on the camera parameters of the input image (such as the inherent focal length and distortion parameters of the ERP panoramic image).

[0097] The pinhole cropping view refers to transforming the target area and its surrounding context in the ERP panoramic image into a pinhole camera perspective view centered on the target object through a spherical-to-plane projection transformation. This allows the target object to present its normal perspective geometry in the view, facilitating subsequent regression prediction of thickness and gaps by the network.

[0098] Multi-channel perspective view refers to the superposition of multiple information sources, namely the rotated ERP panoramic view, multi-scale visual features, lighting features, object instance masks, and visible surface depth, in the pinhole cropping view coordinate system, to form multi-channel tensor data, which is then used as the input of the mask-by-mask perspective query network.

[0099] In this embodiment, the electronic device first calculates the centroid position of the current object instance mask in the spherical coordinate system; then, based on the centroid position, the ERP panoramic image is rotated horizontally to obtain a rotated ERP panoramic image, so that the object instance is moved to the central area of ​​the panoramic image, away from the left and right seam boundaries of the ERP; next, the electronic device acquires the multi-scale visual features and lighting features of the ERP panoramic image, and projects the rotated ERP panoramic image, multi-scale visual features, lighting features, current object instance mask, and visible surface depth information onto a pinhole cropping view centered on the object through a spherical-to-plane projection transformation to obtain a multi-channel perspective view; finally, the multi-channel perspective view is input into a mask-by-mask perspective query network, so that the network performs pixel-by-pixel dense prediction under the normal perspective form of the target object, and outputs the object thickness field and background gap field corresponding to the object instance.

[0100] It should be noted that by rotating the target object to a central region away from the ERP seam and projecting it as a pinhole perspective view, the interference of ERP boundary discontinuity and spherical distortion on thickness prediction is eliminated. This allows the mask-by-mask perspective query network to learn and infer under standard perspective geometry, thereby improving the prediction accuracy of the object thickness field and background gap field.

[0101] In some embodiments, the step of an electronic device inputting a multi-channel perspective view into a mask-by-mask perspective query network and outputting the object thickness field and background gap field corresponding to the object instance mask may include: inputting the multi-channel perspective view into the mask-by-mask perspective query network, and performing pixel-by-pixel dense prediction on the multi-channel perspective view inside the object instance mask through a shared dense prediction head in the mask-by-mask perspective query network to obtain the object thickness field and background gap field corresponding to the object instance mask.

[0102] Among them, the shared dense prediction head refers to the neural network module in the mask-by-mask perspective query network used for pixel-by-pixel regression prediction. This module shares the same network weights among different object instances. For any input object instance mask, the same set of network parameters is used to perform dense prediction within the mask.

[0103] In this embodiment, after obtaining a multi-channel perspective view, the electronic device inputs the multi-channel perspective view into a mask-by-mask perspective query network. This network, through its internal shared dense prediction head, performs pixel-by-pixel dense regression prediction on each pixel location in the multi-channel perspective view within the effective area of ​​the object instance mask. For areas outside the mask, the network either does not perform prediction or marks its prediction results as invalid. The shared dense prediction head independently outputs two values ​​for each pixel location, corresponding to the object thickness value and background gap value along the panoramic ray direction at that pixel location, respectively. By combining the prediction results for all pixel locations, the object thickness field and background gap field corresponding to the object instance mask can be obtained.

[0104] For example, Figure 4 This is a schematic diagram of the input and output of the mask-by-mask perspective query network provided in an embodiment of this application. Combined with... Figure 4 The input to this mask-by-mask perspective query network is a perspective view centered on the target object, containing various information such as the object's appearance texture, edge contours, light direction, and visible surface depth. The mask-by-mask perspective query network uses a deep learning network to extract features and perform dense regression on this input, predicting two geometric fields pixel-by-pixel within the mask, such as... Figure 4 As shown, the object thickness field This represents the thickness of the object instance along each viewing direction; darker colors indicate greater thickness. Background gap field. This represents the gap distance from the back of the object instance to the background surface behind it that is occluded; the darker the color, the larger the gap.

[0105] It should be noted that by sharing a dense prediction head to perform pixel-by-pixel dense prediction inside the mask, we can make full use of the contextual information of every point inside the mask and output a dense object thickness field and background gap field that are aligned with the resolution of the input image, providing refined pixel-by-pixel geometric parameters for subsequent layered geometry construction.

[0106] In some embodiments, the mask-by-mask perspective query network is a pre-trained network. The training process of the electronic device for the mask-by-mask perspective query network may include: acquiring a multi-source panoramic 3D dataset, performing data augmentation processing on the ERP panoramic images in the multi-source panoramic 3D dataset to obtain training samples; constructing a supervision signal for the training samples, the supervision signal including the ground truth value of object thickness and the ground truth value of background gap; inputting the training samples into the network to be trained, and performing end-to-end optimization on the network to be trained based on the loss function to obtain the pre-trained mask-by-mask perspective query network; wherein, the loss function includes: a non-negativity loss for constraining the non-negativity of the predicted value, a label validity mask loss for calculating the loss only for pixels with ordered geometric intersections, and a boundary consistency loss for imposing cyclic constraints on the prediction results at the left and right boundaries of the ERP panoramic image.

[0107] The multi-source panoramic 3D dataset contains multiple ERP panoramic images and their corresponding ground truth annotations.

[0108] Optionally, the sources of the multi-source panoramic 3D dataset include, but are not limited to: synthetic panoramic datasets generated using a 3D rendering engine, and real panoramic scan data acquired using a 360° camera or a LiDAR-camera joint acquisition system.

[0109] It is understandable that in the multi-source panoramic 3D dataset, each ERP panoramic image has multiple layers of geometric ground truth annotations, such as visible surface depth, object instance ID, object back depth, and occluded background depth.

[0110] ERP panoramic images refer to panoramic image samples stored in an equidistant cylindrical projection format in a multi-source panoramic 3D dataset, with each sample image associated with a corresponding geometric ground truth label.

[0111] Data augmentation refers to the transformation operation performed on ERP panoramic images and their corresponding ground truth labels in a multi-source panoramic 3D dataset to expand the diversity and generalization ability of training samples.

[0112] Optionally, data augmentation processing includes, but is not limited to: randomly scrolling the ERP panoramic image along the horizontal longitude direction, cropping or resampling the ERP panoramic image along the vertical latitude direction, randomly adjusting the brightness / contrast / saturation of the image, simulating different weather and lighting conditions, and replacing the sky area in different scenes.

[0113] The supervision signal for training samples refers to the target ground truth value used to supervise the network output during training.

[0114] The target object refers to the object instance in the current training sample that is selected as the query object.

[0115] The true thickness of an object is calculated by the depth difference between the visible surface and the back surface within the instance mask region of the target object, which is the difference between the depth of the visible surface and the depth of the back surface.

[0116] The true value of the background gap is calculated by the difference between the depth of the exposed background surface after removing the target object and the depth of the back surface of the target object.

[0117] The loss function is a mathematical function used to measure the difference between the predicted output of the network to be trained and the supervision signal. During training, the network parameters are optimized by minimizing the value of the loss function.

[0118] Optionally, the loss function may include at least: nonnegativity loss, label validity masking loss, and boundary consistency loss.

[0119] Among them, the nonnegativity loss is used to constrain the object thickness value and background gap value predicted by the network to be trained to be nonnegative, and a penalty term is applied to pixels with negative predicted values.

[0120] Label validity mask loss refers to calculating the loss only for pixel positions with ordered geometric intersections. For pixels without ordered geometric intersections (i.e., there is no effective thickness or gap truth value at the pixel position), the loss is masked and does not participate in gradient backpropagation.

[0121] Boundary consistency loss refers to applying cyclic consistency constraints to the prediction results at the left and right boundaries of the ERP panoramic image, so that the prediction values ​​at corresponding positions at the left and right boundaries are consistent in the sense of physical adjacency, thus eliminating prediction jumps caused by the expansion of the ERP boundaries.

[0122] In this embodiment, the electronic device first constructs a multi-source panoramic 3D dataset containing a large number of ERP panoramic images and their corresponding multi-layer geometric ground truth annotations. Then, the electronic device performs data augmentation on the samples in the dataset to improve the generalization ability of the model. Next, the electronic device samples training samples from the augmented dataset to construct a supervision signal containing object thickness ground truth and background gap ground truth. Finally, the electronic device inputs the training samples into the network to be trained and performs end-to-end optimization training on the network to be trained by minimizing the total loss function composed of nonnegativity loss, label validity mask loss and boundary consistency loss until the network converges, thus obtaining a pre-trained mask-by-mask perspective query network.

[0123] Understandably, after training, the network weights of the mask-by-mask perspective query network are permanently stored. During the inference phase, the pre-trained network weights are directly loaded, and a single forward inference of the ERP panorama to be processed can output the prediction result, without the need for iterative optimization or online fine-tuning for the current input. Through the above-mentioned offline training and online inference method, the computational overhead of the inference phase is significantly reduced, enabling the embodiments of this application to achieve real-time or near-real-time processing efficiency in actual deployment.

[0124] It should be noted that by constructing multi-source datasets and employing diverse data augmentation strategies, the mask-by-mask perspective query network can adapt to panoramic inputs under different scenes and lighting conditions. Non-negativity loss and label validity mask loss ensure the physical plausibility of the network output, avoiding erroneous predictions of negative thickness or invalid regions. Boundary consistency loss ensures the smooth continuity of prediction results at the left and right boundaries of the ERP. These training strategies collectively guarantee the prediction accuracy and generalization ability of the mask-by-mask perspective query network during actual inference.

[0125] In some embodiments, before generating the layered panoramic 3D Gaussian scene, the electronic device may further include: performing ERP left and right seam and boundary alignment processing on the multi-layer radial geometry.

[0126] Among them, the ERP left and right seam and boundary alignment processing refers to the process of eliminating the jumps in depth value, feature value and mask value at the left and right boundaries of the ERP panoramic image caused by spherical unfolding, so that the two sides of the boundary maintain a continuous and consistent meaning in the sense of physical adjacency.

[0127] In this embodiment of the application, after obtaining the multi-layer radial geometry, the electronic device scrolls the ERP panoramic image horizontally by W / 2 pixels, and performs depth prediction, feature extraction and mask prediction on the scrolled ERP panoramic image to obtain the depth map, feature map and mask map in the scrolling coordinate system; at the same time, the corresponding prediction results of the original ERP panoramic image remain unchanged.

[0128] Subsequently, the electronic device will scroll the depth map, feature map, and mask map in the scroll coordinate system in the opposite direction by W / 2 pixels, so that they are in the same coordinate space as the corresponding data in the original coordinate system.

[0129] Next, the electronic device fits a shared metric alignment transformation in the overlapping area between the original data and the scrolled data after reverse scrolling, so that the depth values ​​and geometric quantities of the two data in the overlapping area are consistent on the metric scale.

[0130] For example, the shared metric alignment transformation can employ a linear transformation or a logarithmic domain transformation to fit the depth values ​​using the least squares method, minimizing the mean square error of the two depth maps in the overlapping region. Those skilled in the art can select an appropriate alignment transformation form based on the actual data distribution characteristics and accuracy requirements; this application does not impose specific limitations in this regard.

[0131] Finally, the electronic device uses complementary cosine weights to fuse the original data with the aligned scrolling data. For example, Figure 5 This is a schematic diagram of ERP seam alignment and complementary cosine fusion provided in an embodiment of this application. (Combined with...) Figure 5 The horizontal axis represents the horizontal pixel position of the ERP panorama. The left region corresponds to the left half of the original ERP panorama (i.e., image A), and the right region corresponds to the right half of the original ERP panorama (i.e., image B). The left and right sides are adjacent at the ERP boundaries. During the fusion process, the weights corresponding to the original prediction data... Weights corresponding to the rolling prediction data Within the overlapping region, they exhibit a complementary cosine distribution, increasing closer to the left and right boundaries of the ERP. Approaching 0 The closer the weight is to 1, the higher the weight is assigned to the scrolling prediction data; the closer to the center of the image, the higher the weight is assigned. Approaching 1 Approaching 0, that is, assigning higher weights to the original prediction data; in the middle of the overlapping region, and The weights are all approximately 0.5, achieving a balanced transition of weights, and finally obtaining a multi-layer radial geometry with smoothed seams, thus realizing the smoothing of the left and right boundaries of the ERP of each layer of the multi-layer radial geometry.

[0132] It is understandable that the depth map includes the radial depth corresponding to each layer of the radial geometry in multiple layers; the feature map includes multi-scale visual features and ray features; and the mask map includes the masks of each object instance. The depth map, feature map, and mask map mentioned above are all processed according to the same scroll-align-blend process in the seam alignment process to ensure the consistency of each type of data at the ERP boundary.

[0133] It should be noted that the above-mentioned ERP left and right seam and boundary alignment processing is also applicable to instance mask, object thickness field and background gap field. That is, instance mask prediction, thickness field prediction and gap field prediction are performed on the original ERP panoramic image and the scrolled ERP panoramic image respectively. After the prediction results are rolled back to the original coordinate system, they are fused using the same complementary cosine weights to obtain the instance mask, object thickness field and background gap field after seam smoothing.

[0134] The ERP left and right seam and boundary alignment processing only fuses the left and right boundaries of the ERP panoramic image, without affecting the original prediction results of the image's internal regions. Simultaneously, at instance boundaries, geometric information is not allowed to be blurred or interpolated between different object owners; any boundary feathering or smoothing operations are limited to the same surface owner to ensure clear geometric boundaries between different instances and avoid occlusion errors. Furthermore, in seam alignment processing, for hidden layer samples corresponding to the second and third radial geometry layers, if their instance owner attribution confidence near the ERP boundary is lower than a preset confidence threshold, these samples are marked as invalid or suppressed and do not participate in the subsequent generation of 3D Gaussian primitives. For the first layer of radial geometry, its ownership is uniquely determined by the instance mask, and no confidence judgment is performed, thus avoiding holes or missing parts on the visible surface due to excessive suppression.

[0135] Optionally, after completing the alignment of the left and right seams and boundaries of the ERP, the seam alignment quality can be evaluated using the following metrics: the difference in depth values ​​between the first and last columns of the ERP panorama, the statistical value of the depth gradient in the seam area, and the comparison value of the depth gradient with the non-seam control zone within the image. These metrics measure the seam alignment effect from two dimensions: numerical continuity and gradient smoothness, ensuring that geometric continuity meets the preset standards.

[0136] It should be noted that by using multiple roll-up predictions and complementary cosine fusion strategies, the numerical jumps in depth, features, and masks at the left and right boundaries of the ERP panoramic image are effectively eliminated, making the originally adjacent left and right boundaries in three-dimensional space truly continuous geometrically. This avoids cracks and visual artifacts in the three-dimensional scene caused by discontinuities in the seams, and improves the visual smoothness and immersion when crossing boundary areas during panoramic roaming.

[0137] In some embodiments, the step of determining the multi-layer radial geometry corresponding to each object instance based on the visible surface depth, the object thickness field, and the background gap field may include: for each object instance, the electronic device superimposes the visible surface depth, the object thickness field, and the background gap field along the same panoramic ray direction to determine the radial depth of each surface layer in the multi-layer radial geometry corresponding to the object instance; wherein, the multi-layer radial geometry includes at least a first layer radial geometry, a second layer radial geometry, and a third layer radial geometry; the first layer radial geometry is the visible surface represented by the visible surface depth; the second layer radial geometry is the back surface of the object instance, the depth of which is equal to the sum of the visible surface depth and the object thickness field; the third layer radial geometry is the background surface exposed after the object instance is removed, the depth of which is equal to the sum of the back surface depth and the background gap field; the first layer radial geometry, the second layer radial geometry, and the third layer radial geometry are arranged sequentially along the same panoramic ray direction.

[0138] In this embodiment, after obtaining the visible surface depth, the object thickness field of each object instance, and the background gap field, the electronic device, for each object instance, algebraically superimposes the visible surface depth, object thickness field, and background gap field along each effective panoramic ray direction inside the mask of that object instance, and sequentially calculates three layers of radial depth: the first layer of radial geometry. The visible surface depth; the second layer radial geometry The sum of the visible surface depth and the object thickness field, corresponding to the back surface of the object instance; third layer radial geometry. This is the sum of the depth of the back surface and the background gap field, corresponding to the background surface exposed after removing the object instance. The three surfaces are arranged along the same panoramic ray direction... The order is from nearest to farthest.

[0139] For example, Figure 6 This is a schematic diagram of the three-layer radial geometry construction provided in an embodiment of this application. Combined with... Figure 6 , with a certain object instance For example, let the visible surface depth at a certain pixel location within its mask region be... The predicted object thickness field value at this pixel location. Background gap field value [A], then the multi-layer radial geometry corresponding to this pixel location includes: the spatial location of the first layer radial geometry (i.e., the visible layer, corresponding to the front of the object) is... The spatial location of the second layer of radial geometry (i.e., the back surface layer of the object, corresponding to the back surface of object A) is: The spatial location of the third radial geometry layer (i.e., the background exposure layer, corresponding to the background wall exposed after removing object A) is: The three surfaces mentioned above are aligned along the same panoramic light direction. The items are arranged in order from nearest to farthest. Figure 6 It clearly demonstrates the complete layered geometry that, starting from the camera center, passes sequentially through the visible surface, the back of the object, and the obscured background wall, all along the same panoramic light direction.

[0140] Understandably, the second and third radial geometries are only valid at pixel locations where there are valid ordered geometric intersections within the object instance mask. That is, the second and third radial geometries are assigned valid depth values ​​and participate in the generation of subsequent 3D Gaussian primitives only when both the object thickness field and background gap field at that pixel location are valid prediction values. For pixel locations where the object thickness field or background gap field is marked as invalid, the corresponding second and third radial geometries are also marked as invalid and do not participate in subsequent processing. Furthermore, when multiple object instances overlap along the same panoramic ray direction, the multi-layer radial geometries corresponding to each instance are sorted in ascending order of radial depth, preserving the ownership information of each surface, rather than simply averaging or merging the depth values ​​of multiple surfaces along the same ray direction. This avoids occlusion errors and blurred object boundaries caused by depth confusion.

[0141] All the above calculations use the same unit of measurement for radial depth, i.e. , , , , All values ​​are expressed in radial distance units, ensuring that depth values ​​at each layer can be directly added or subtracted algebraically without unit conversion. Furthermore, each object instance corresponds to a valid mask. Effectiveness mask This identifies whether the object thickness field and background gap field at each pixel location within the object instance mask are both valid prediction values; for valid masks... Marked as valid pixel locations, the object thickness field and background gap field participate. and The calculation and subsequent generation of 3D Gaussian primitives; for the effectiveness mask. The pixel location marked as invalid, corresponding to and It is marked as invalid and will not be processed further.

[0142] It should be noted that by expanding the panoramic image, which originally only contained the outermost surface, into a complete layered geometry that includes the front surface of the object, the back surface of the object, and the occluded background surface, a geometric foundation is laid for the subsequent generation of a 3D scene with correct occlusion relationships. At the same time, by preserving the ownership of each layer of surface and sorting them by radial depth, depth confusion in multi-object overlapping scenes is avoided, ensuring the physical correctness of the occlusion relationships.

[0143] In some embodiments, the step of generating a layered panoramic 3D Gaussian scene based on multi-layer radial geometry by an electronic device may include: for the effective surface of each layer of geometry in the multi-layer radial geometry, the electronic device calculates the projection sampling frequency based on the radial depth, surface normal, and viewing direction; wherein the projection sampling frequency is proportional to the square of the radial depth and inversely proportional to the absolute value of the dot product of the surface normal and the viewing direction; the electronic device determines sampling points on the effective surface of each layer of geometry based on the projection sampling frequency, queries the light rays, depth, and shared features corresponding to each sampling point, and generates 3D Gaussian primitives at the sampling points; the electronic device merges the 3D Gaussian primitives corresponding to all geometric layers in the multi-layer radial geometry to obtain a layered panoramic 3D Gaussian scene.

[0144] Radial depth refers to the radial distance from the observation origin along the panoramic light direction to the intersection point of the surface.

[0145] A surface normal is the normal vector of an effective surface at a sampling point, used to describe the orientation of the surface at that point.

[0146] The line-of-sight direction refers to the directional vector from the sampling point to the observation origin, which is opposite to the direction of the panoramic light.

[0147] An effective surface refers to a surface region in a multi-layer radial geometry that has an effective radial depth value, meaning that the depth value at that location is not marked as invalid, and the corresponding object thickness field and / or background gap field are effective predicted values.

[0148] Projection sampling frequency refers to the sampling density corresponding to the projected area of ​​a unit image region on a three-dimensional surface, which is used to determine the number of Gaussian primitives that need to be generated for that surface region.

[0149] Light, depth, and shared features refer to the panoramic light direction, radial depth value, and multi-scale visual features extracted from the multi-task feature extraction backbone network corresponding to the sampling point.

[0150] In this embodiment, after determining the spatial positions of each surface layer in the multi-layer radial geometry, the electronic device needs to convert each surface layer into a three-dimensional Gaussian primitive. The electronic device employs an adaptive Gaussian compaction strategy, that is, for the effective surface of each layer geometry... First, based on the surface radial depth Surface normal and the direction of sight Calculate the projection sampling frequency ,in, , , Represents a proportionality constant. This represents a preset, extremely small positive number to prevent numerical overflow caused by the denominator approaching zero when the surface normal is nearly perpendicular to the viewing direction. The projection sampling frequency reflects the area projected by a unit image pixel onto the 3D surface; the greater the depth or the closer the surface is to the tangent direction (i.e., the surface normal is nearly perpendicular to the viewing direction), the higher the required sampling density. Subsequently, the electronic device determines sampling points on the effective surface based on this projection sampling frequency, and queries the ray direction, radial depth, and shared features corresponding to each sampling point using bilinear interpolation. Next, the queried ray direction, radial depth, and shared features are input into the Gaussian decoder to generate 3D Gaussian primitives for each sampling point. Each Gaussian primitive contains parameters such as position, color, opacity, scale, and rotation. Finally, the electronic device merges the 3D Gaussian primitives generated from all geometric layers in the multi-layer radial geometry to obtain a unified panoramic 3D Gaussian scene containing a complete layered structure.

[0151] For example, Figure 7 This is a schematic diagram illustrating the adaptive Gaussian compaction principle provided in an embodiment of this application. Combined with... Figure 7 Taking a panoramic ray emitted from the origin O as an example, this ray hits a sampling point on the effective surface. This is the radial distance from the observation origin O to this surface point. Let be the surface normal at that point on the surface. The direction of the line of sight (i.e., the direction from the surface point to the observation origin O). For surface normal With line of sight The angle between them. For example... Figure 7 As shown, when the surface is close to the observation origin and the line of sight is almost perpendicular to the surface, hour, When the surface distance is large and the projection sampling frequency is low, the coverage area of ​​a single Gaussian primitive is large, and the sampling density is relatively sparse. When the surface distance increases or the angle between the surface normal and the viewing direction increases, i.e., the surface tends to be tangential or glancing, When approaching 90°, As the sampling frequency decreases, the projection sampling frequency increases significantly. At this point, more Gaussian primitives need to be arranged per unit area. Since the coverage area of ​​a single Gaussian primitive is relatively small, the sampling density is increased accordingly. Figure 7The image shows three scenarios from left to right: a close-range vertical view (sparse sampling, large Gaussian ellipse), a medium-range oblique view (moderate sampling density, moderate Gaussian ellipse size), and a long-range or grazing angle view (dense sampling, small and densely arranged Gaussian ellipse). These scenarios intuitively reflect the variation of sampling frequency with depth and viewing direction.

[0152] Optionally, the layered panoramic 3D Gaussian scene is output as a 3DGS scene representation. The default output method is to merge all Gaussian primitives into a single Polygon File Format (PLY) file, which records the position, color, opacity, scale, rotation, and other attribute parameters of each Gaussian primitive. In debug mode or in scenes requiring layer-by-layer diagnostics, you can choose to output independent PLY files for each layer separately, so as to view the geometry of each layer individually, assess its quality, or analyze errors.

[0153] Understandably, during the merging process, each Gaussian primitive retains its layer identity and owner information. The layer identity is used to distinguish whether the Gaussian primitive belongs to the first, second, or third layer of radial geometry, and the owner information is used to distinguish which object instance mask the Gaussian primitive belongs to, so that each layer and instance can be differentiated during subsequent rendering, editing, or evaluation.

[0154] It should be noted that the above-mentioned method of dynamically adjusting the sampling density based on the projection sampling frequency only adds sampling points to the already predicted effective surfaces to supplement the sampling density of sparse regions and improve rendering quality. This method is not used to infer missing topological structures or generate new geometric surfaces; its scope is limited to the sampling optimization of existing geometric surfaces. Furthermore, the mask-by-mask perspective query network only outputs geometric information, namely the object thickness field and background gap field; while the non-geometric attributes of the 3D Gaussian primitives, such as color, opacity, scale, and rotation, are predicted by the Gaussian decoder, achieving decoupled prediction of geometric and non-geometric information.

[0155] In some embodiments, after generating a layered panoramic 3D Gaussian scene, the electronic device may further include: constructing an outer sky sphere surrounding the layered panoramic 3D Gaussian scene, the radius of which is greater than the maximum radial depth of all Gaussian primitives in the layered panoramic 3D Gaussian scene; generating an inner sky texture of the outer sky sphere based on a textual prompt and / or a sky region in the ERP panoramic map; setting the outer sky sphere as the outermost layer when rendering the layered panoramic 3D Gaussian scene; and returning the color of the outer sky sphere when rendering rays do not hit any 3D Gaussian primitive in the layered panoramic 3D Gaussian scene.

[0156] Among them, the outer sky sphere refers to a large-radius sphere that surrounds the entire layered panoramic 3D Gaussian scene. The center of the sphere is located at the observation origin, and its radius is much larger than the maximum radial depth of all Gaussian primitives in the scene. It is used to present the infinitely distant background such as the sky, distant mountains, and atmosphere outside the scene.

[0157] The inner sky texture refers to the image texture mapped onto the inner surface of the outer sky sphere. This texture covers the entire inner side of the sphere and is used to present a seamless 360° sky or environment background to the viewer during rendering.

[0158] In this embodiment, after generating a layered panoramic 3D Gaussian scene, to restore a complete panoramic immersive experience, the electronic device constructs a large-radius sphere surrounding the entire scene as an outer sky sphere, the radius of which is greater than the maximum radial depth of all Gaussian primitives in the scene. Then, based on the user-input text prompts and / or the sky region image extracted from the ERP panoramic image in the previous steps, the electronic device generates the inner sky texture of the outer sky sphere. Finally, during rendering, the electronic device sets the outer sky sphere at the outermost layer of all scene Gaussian primitives. The rendering light first determines whether it hits any 3D Gaussian primitive in the scene. If it hits, it returns the color of the Gaussian primitive. If it does not hit any Gaussian primitive (i.e., the light is directed towards infinity), it returns the inner texture color of the corresponding position of the outer sky sphere.

[0159] For example, the radius of the outer sky sphere is , The maximum radial depth is greater than the maximum radial depth of all 3D Gaussian primitives in the layered panoramic 3D Gaussian scene, and the inner sky texture of the outer sky sphere is... ,in Represents the longitude angle of a sphere. This represents the spherical latitude angle. During the rendering process, for any rendering ray emanating from the observation origin, the intersection of the ray and the scene's Gaussian primitives is first detected along the ray's direction. If at least one 3D Gaussian primitive is detected, that Gaussian primitive is rendered, and the color value of that Gaussian primitive at that location is returned; if the ray does not hit any 3D Gaussian primitive in the scene, that is, the ray is directed towards infinity. Then, the corresponding spherical latitude and longitude can be calculated based on the direction vector of the ray. It samples the color value at the corresponding position from the inner texture of the outer sky sphere. As the color of the returned light.

[0160] It should be noted that by using the separate representation of "internal real geometry + external sky sphere", the geometric noise and perspective distortion caused by the incorrect reconstruction of infinitely distant backgrounds such as the sky as finite depth geometry are avoided, while ensuring the visual integrity and immersion of the final panoramic tour.

[0161] Optionally, the step of generating the inner sky texture of the outer sky sphere can be performed using any one or more of the following methods in combination: Method 1: Directly extract color or texture information from the sky region of the original ERP panoramic image and map it to the inner side of the outer sky sphere. This method is suitable for scenarios where the sky region in the original ERP panoramic image is complete, of high quality, and does not require stylization transformation. It has the lowest computational cost and can preserve the sky appearance of the original generated panoramic image.

[0162] Method 2: Utilize a pre-trained text-to-image model to generate a high-resolution sky or environment texture based on text prompts, and then map it onto the inner side of the outer sky sphere. This method is suitable for scenarios where the sky area in the original ERP panoramic image is incomplete, has insufficient resolution, contains seam artifacts, or where the user expects to reset the sky style (e.g., changing "sunny day" to "starry sky" or "sunset"). It can generate high-quality sky textures that are consistent with the scene semantics.

[0163] Method 3: Utilizing scene semantic information and text prompts, the original sky area in the ERP panoramic image is repaired, expanded, or stylized to generate a seamless sky texture. This method combines the actual sky color distribution of the original panoramic image with the semantic constraints of the text prompts. It preserves the lighting harmony of the original scene while repairing seams, filling in missing areas, or stylizing the sky, balancing the realism and flexibility of the texture.

[0164] It is understood that, of the three methods described above, method one is suitable for scenarios where the original sky appearance remains unchanged, method two is suitable for scenarios where the sky texture is completely regenerated, and method three is suitable for scenarios where repair or stylization is performed on the basis of the original sky. Those skilled in the art can flexibly choose one of the above methods or a combination thereof according to actual application needs, computing resources, and desired sky effects, and this application does not impose specific limitations in this regard.

[0165] Optionally, after constructing the outer sky sphere and generating the inner sky texture, the method also includes: dynamically enhancing the outer sky sphere.

[0166] Dynamic enhancement processing refers to dynamically adjusting the visual appearance of the outer sky sphere based on preset scene parameters to enhance the immersion and realism of the scene.

[0167] The preset scene parameters include, but are not limited to: time parameters (such as early morning, noon, dusk, and night), weather parameters (such as sunny, cloudy, rainy, foggy, and snowy), and environmental description information implied in the text prompts.

[0168] Specifically, dynamic enhancement processing includes at least one of the following: Lighting Adjustment: Based on the time parameters or the description of the direction of light in the text prompts (such as "sunset" or "midday sun"), adjust the overall brightness and color temperature of the sky globe to make the sky reflect the lighting characteristics of the corresponding time of day. Cloud and fog simulation: Based on the description of weather conditions in the weather parameters or text prompts (such as "cloudy" or "foggy"), cloud and fog textures are superimposed on the surface of the sky sphere to change the transparency of the sky and the distribution of clouds. Adjusting the position of the sun or moon: Based on the time parameters or the description of the sun or moon's position in the text prompt, calculate the spatial azimuth and elevation angles of the sun or moon on the sky sphere, and render the halo of the sun or moon to the corresponding position on the sky sphere; Color style transfer: Based on the description of the environment atmosphere in the text prompts (such as "cyberpunk style" or "warm twilight tones"), perform overall color mapping or stylization transfer on the inner texture of the sky sphere to ensure that the sky appearance is consistent with the scene semantics.

[0169] Understandably, dynamic enhancement processing can be performed offline after the 3D scene is built, and the adjusted sky sphere texture can be stored in a fixed state; alternatively, it can be performed during real-time rendering, dynamically updating the display effect of the sky sphere based on user interaction commands or external environmental data.

[0170] It should be noted that by introducing a dynamic enhancement mechanism, the outer sky sphere can present diverse visual states based on time, weather, or user descriptions, greatly enhancing the immersiveness and realism of the panoramic 3D scene. This is especially suitable for application scenarios such as virtual tourism, games, or film and television pre-shows that require simulating day-night changes and weather changes.

[0171] To better understand the embodiments of this application, the text-driven layered panoramic 3D Gaussian scene generation method provided in the embodiments of this application will be illustrated below with specific application scenarios.

[0172] Example 1: Indoor Scene Generation The input text prompt is "A modern living room with a city night view through floor-to-ceiling windows, a sofa, a coffee table, and a rug."

[0173] The electronic device first generates an ERP panoramic image based on the text prompt, and then performs semantic segmentation on the ERP panoramic image, identifying the city night view area outside the floor-to-ceiling window as an infinitely distant background and removing it. Subsequently, the electronic device performs instance segmentation on the ERP panoramic image, extracting object instance masks such as sofas, coffee tables, and carpets, and reconstructing the visible surface depth of the ERP panoramic image. Next, the electronic device, for object instances such as sofas, coffee tables, and carpets, predicts their respective object thickness fields and background gap fields using a mask-by-mask perspective query network, and constructs multi-layer radial geometry for each instance accordingly. layer, Layers and The electronic device then performs ERP alignment processing on the left and right seams and boundaries of the multi-layer radial geometry, and generates 3D Gaussian primitives for each layer's surface through adaptive Gaussian densification. All Gaussian primitives are then merged to obtain a layered panoramic 3D Gaussian scene. Finally, the electronic device completes the city nightscape sky sphere around the scene, allowing users to see not only the front of the sofa and coffee table during free roaming, but also the back of objects and the interior background obscured by objects when the viewpoint moves to the side or back. Furthermore, the city nightscape outside the floor-to-ceiling windows is presented stably as an infinity background without distortion.

[0174] Example 2: Generation of outdoor urban scenes The input text prompt is "A busy city street with tall buildings on both sides and a clear sky".

[0175] The electronic device first generates an ERP panoramic image based on the text prompt. It then performs semantic segmentation on the ERP panoramic image, identifying the sky region as an infinitely distant background and removing it. Next, the device performs instance segmentation on the ERP panoramic image, extracting masks for object instances such as vehicles, pedestrians, and buildings, and reconstructing the visible surface depth of the ERP panoramic image. Then, the device predicts the thickness and gaps for vehicle and pedestrian instances respectively, constructing multi-layer radial geometry. It then performs ERP seam alignment and adaptive Gaussian compaction on the multi-layer radial geometry to generate a layered panoramic 3D Gaussian scene. Finally, the device completes the scene with a clear sky sphere around the perimeter. Through this method, the generated urban scene allows users to freely roam the streets. When the viewpoint is panned, building facades that were originally obscured by vehicles or pedestrians are correctly revealed, and the sky is stably presented as an infinitely distant background with no obvious abrupt changes at scene seams.

[0176] Example 3: Multi-object occlusion scene The input text prompt is "A person is sitting on a park bench with dense trees behind them".

[0177] The electronic device first generates an ERP panoramic image based on the text prompt. It then performs semantic segmentation on the ERP panoramic image, identifying the sky and distant areas as infinity backgrounds and removing them. Subsequently, the electronic device performs instance segmentation on the ERP panoramic image, extracting instance masks such as benches, people, and trees, and reconstructs the visible surface depth of the ERP panoramic image. Next, the electronic device predicts the thickness and gaps for a human instance, constructing the person's multi-layered radial geometry: The layer corresponds to the back surface of a person. The layers correspond to the exposed tree and grass surfaces after the person is removed. The electronic devices then correlate the multi-layered radial geometry of the person with the visible surfaces of the benches and trees. The layers are merged to generate a layered panoramic 3D Gaussian scene, and the skyline sky sphere is completed. In this way, the generated scene allows users to freely roam in the park. When the viewpoint moves from the front of the person to the side, the person's thickness can be correctly perceived; when the viewpoint moves behind the person, the surfaces of trees and grass that were originally obscured can be correctly revealed, and the occlusion relationship between objects remains physically correct throughout the change of viewpoint.

[0178] To better understand the embodiments of this application, the technical effects of the text-driven layered panoramic 3D Gaussian scene generation method provided in the embodiments of this application are explained below with specific comparative experiments.

[0179] For example, Figure 8 These are comparison diagrams showing the effects of the embodiments provided in this application. Figure 8 The left side shows the generation effect of the prior art, and the right side shows the generation effect provided by the embodiment of this application. The top and bottom rows correspond to the rendering results before and after the viewpoint is moved, respectively.

[0180] Depend on Figure 8 As can be seen, before the viewpoint moves, both the existing technology and the embodiments of this application can correctly present the initial appearance of the scene, and the rendering results of the two are basically the same under the reference viewpoint. However, when the viewing angle moves to the right and the background area that was originally obscured by the foreground vehicle is gradually exposed, the rendering quality of the two shows a significant difference.

[0181] After the viewpoint shifts, existing technologies, lacking explicit modeling of vehicle thickness and occluded background, rely solely on depth information from visible surfaces for new viewpoint synthesis. This results in large black holes appearing in the occluded background area upon exposure (e.g., ...). Figure 8 As indicated by the "background hole" in the image, the vehicle edges also exhibit severe radial stretching and adhesion artifacts due to abrupt changes in depth (as shown in the image). Figure 8 As indicated by the "severe stretching / adhesion" label, the geometry is clearly incorrect and cannot support effective free roaming browsing.

[0182] In contrast, the method provided in this application's embodiments predicts the vehicle's object thickness field and the background gap field exposed after the vehicle is removed using the mask-by-mask perspective query network described in steps 102 to 103, and constructs a field including the vehicle's rear surface based on this. and background exposed surface The multi-layered radial geometry. As the viewpoint moves to the right, the background structure that was originally obscured by the vehicle changes according to... The geometric information of the layer is correctly revealed (e.g.) Figure 8 As indicated by the annotation "correctly revealed background structure" (as shown in the image), the rear surface of the vehicle is based on... The layers are accurately rendered, the vehicle edges are clear with no stretching artifacts, the background area is complete without holes, and the geometric structure remains consistent.

[0183] In summary, the text-driven layered panoramic 3D Gaussian scene generation method proposed in this application effectively solves the problems of holes, stretching, and disordered occlusion relationships in newly exposed areas caused by the lack of layered geometric modeling in the prior art by explicitly modeling the object thickness and background gap and constructing multi-layered radial geometry. This significantly improves the rendering quality and visual realism during free roaming.

[0184] The following describes the text-driven layered panoramic 3D Gaussian scene generation system provided in the embodiments of this application. The text-driven layered panoramic 3D Gaussian scene generation system described below can be referred to in correspondence with the text-driven layered panoramic 3D Gaussian scene generation method described above.

[0185] Figure 9 This is a schematic diagram of the structure of a text-driven, layered panoramic 3D Gaussian scene generation system provided in an embodiment of this application. Figure 9 As shown, the system includes: a panoramic instance segmentation module 901, a perspective query and thickness prediction module 902, a multi-layer geometric synthesis module 903, and a 3D Gaussian scene generation module 904.

[0186] The panoramic instance segmentation module 901 is used to obtain the equidistant cylindrical projection ERP panoramic image corresponding to the text prompt, perform instance segmentation on the ERP panoramic image, and extract the mask of each object instance in the ERP panoramic image.

[0187] The perspective query and thickness prediction module 902 is used to predict, for each object instance mask, the object thickness field of the object instance corresponding to the object instance mask along the panoramic light direction, and the background gap field from the back of the object instance to the next surface along the panoramic light direction, through a mask-by-mask perspective query network.

[0188] The multi-layer geometry synthesis module 903 is used to obtain the visible surface depth of the ERP panoramic image along the panoramic light direction, and determine the multi-layer radial geometry corresponding to each object instance based on the visible surface depth, the object thickness field and the background gap field.

[0189] The 3D Gaussian scene generation module 904 is used to generate a layered panoramic 3D Gaussian scene based on the multi-layer radial geometry.

[0190] Optionally, the panoramic instance segmentation module 901 is specifically used to roll the ERP panoramic image along the horizontal longitude direction by a preset pixel distance to obtain multiple longitude-offset ERP panoramic images; to perform instance segmentation on each longitude-offset ERP panoramic image to obtain the initial object instance mask corresponding to each longitude-offset ERP panoramic image; to roll the initial object instance mask corresponding to each longitude-offset ERP panoramic image in reverse to the original image coordinate system to obtain candidate instance masks in a unified coordinate system; to calculate the overlap between each candidate instance mask based on the intersection-union ratio, and to classify the candidate instance masks with an overlap greater than a preset instance matching threshold into the same object instance and perform mask fusion to obtain the object instance masks of the ERP panoramic image.

[0191] Optionally, the panoramic instance segmentation module 901 is specifically used to perform semantic segmentation on the ERP panoramic image, identify the sky and / or the infinitely distant background region, and generate a first mask; the region corresponding to the first mask is marked as an invalid region, and the invalid region is excluded when predicting the depth of the visible surface, the thickness field of the object, and the gap field of the background.

[0192] Optionally, the panoramic instance segmentation module 901 is specifically used to perform connectivity analysis on the effective area of ​​the ERP panoramic image, where the effective area is the area in the ERP panoramic image other than the invalid area; when an isolated fragment area smaller than a preset area threshold is detected in the effective area, the isolated fragment area is merged or repaired.

[0193] Optionally, the panoramic instance segmentation module 901 is specifically used to obtain the initial depth map of the ERP panoramic image, identify pixel regions with depth values ​​greater than a preset depth value threshold and / or depth confidence lower than a preset confidence threshold based on the initial depth map, generate a second mask; merge the first mask and the second mask to obtain a culling mask; and mark the region corresponding to the culling mask as the invalid region.

[0194] Optionally, the perspective query and thickness prediction module 902 is specifically used to determine the centroid position of the object instance mask in the spherical coordinate system of the ERP panoramic image; to perform a horizontal rotation on the ERP panoramic image based on the centroid position to obtain a rotated ERP panoramic image; to obtain the multi-scale visual features and light features corresponding to the ERP panoramic image, and to project the rotated ERP panoramic image, the multi-scale visual features, the light features, the object instance mask, and the visible surface depth onto a pinhole cropping view to obtain an aligned multi-channel perspective view; and to input the multi-channel perspective view into the mask-by-mask perspective query network to determine the object thickness field and background gap field corresponding to the object instance mask.

[0195] Optionally, the perspective query and thickness prediction module 902 is specifically used to input the multi-channel perspective view into the mask-by-mask perspective query network, and through the shared dense prediction head in the mask-by-mask perspective query network, perform pixel-by-pixel dense prediction on the multi-channel perspective view inside the object instance mask to obtain the object thickness field and background gap field corresponding to the object instance mask.

[0196] Optionally, the perspective query and thickness prediction module 902 is specifically used to acquire a multi-source panoramic 3D dataset, perform data augmentation processing on the ERP panoramic images in the multi-source panoramic 3D dataset to obtain training samples; construct a supervision signal for the training samples, which includes the ground truth values ​​of object thickness and background gaps; input the training samples into the network to be trained, and perform end-to-end optimization on the network to be trained based on the loss function to obtain a pre-trained mask-by-mask perspective query network; wherein, the loss function includes: a non-negativity loss to constrain the non-negativity of the predicted values, a label validity mask loss to calculate the loss only for pixels with ordered geometric intersections, and a boundary consistency loss to apply cyclic constraints to the prediction results at the left and right boundaries of the ERP panoramic image.

[0197] Optionally, the multi-layer geometry synthesis module 903 is specifically used to superimpose the visible surface depth, the object thickness field, and the background gap field along the same panoramic ray direction for each object instance to determine the radial depth of each surface layer in the multi-layer radial geometry corresponding to the object instance; wherein, the multi-layer radial geometry includes at least a first layer radial geometry, a second layer radial geometry, and a third layer radial geometry; the first layer radial geometry is the visible surface represented by the visible surface depth; the second layer radial geometry is the back surface of the object instance, and the depth of the back surface is equal to the sum of the visible surface depth and the object thickness field; the third layer radial geometry is the background surface exposed after the object instance is removed, and the depth of the background surface is equal to the sum of the back surface depth and the background gap field; the first layer radial geometry, the second layer radial geometry, and the third layer radial geometry are arranged sequentially along the same panoramic ray direction.

[0198] Optionally, the 3D Gaussian scene generation module 904 is specifically used to calculate the projection sampling frequency for the effective surface of each layer of the multi-layer radial geometry, based on the radial depth, surface normal, and viewing direction; wherein, the projection sampling frequency is proportional to the square of the radial depth and inversely proportional to the absolute value of the dot product of the surface normal and the viewing direction; based on the projection sampling frequency, sampling points are determined on the effective surface of each layer of geometry, and the light, depth, and shared features corresponding to each sampling point are queried to generate the 3D Gaussian primitive at the sampling point; the 3D Gaussian primitives corresponding to all geometric layers in the multi-layer radial geometry are merged to obtain a layered panoramic 3D Gaussian scene.

[0199] Optionally, the system further includes: a sky sphere completion module for constructing an outer sky sphere surrounding the layered panoramic 3D Gaussian scene, the radius of which is greater than the maximum radial depth of all Gaussian primitives in the layered panoramic 3D Gaussian scene; generating an inner sky texture of the outer sky sphere based on the text prompt and / or the sky region in the ERP panoramic map; setting the outer sky sphere as the outermost layer when rendering the layered panoramic 3D Gaussian scene; and returning the color of the outer sky sphere when rendering rays do not hit any 3D Gaussian primitives in the layered panoramic 3D Gaussian scene.

[0200] Optionally, the system also includes a seam alignment module for performing ERP left and right seam and boundary alignment processing on the multi-layer radial geometry.

[0201] Figure 10 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, communications interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a text-driven, layered panoramic 3D Gaussian scene generation method.

[0202] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text-driven layered panoramic 3D Gaussian scene generation method provided by the above methods.

[0204] In another aspect, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the text-driven layered panoramic 3D Gaussian scene generation method provided by the methods described above.

[0205] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A text-driven, hierarchical panoramic 3D Gaussian scene generation method, characterized in that, include: Obtain the equidistant cylindrical projection ERP panoramic image corresponding to the text prompt, perform instance segmentation on the ERP panoramic image, and extract the mask of each object instance in the ERP panoramic image; For each object instance mask, the object thickness field along the panoramic light direction of the object instance corresponding to the object instance mask, and the background gap field from the back of the object instance to the next surface along the panoramic light direction are predicted by the mask-by-mask perspective query network. Obtain the visible surface depth of the ERP panoramic image along the panoramic light direction, and determine the multi-layer radial geometry corresponding to each object instance based on the visible surface depth, the object thickness field, and the background gap field; Based on the multi-layered radial geometry, a layered panoramic 3D Gaussian scene is generated.

2. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 1, characterized in that, The step of determining the multi-layer radial geometry corresponding to each object instance based on the visible surface depth, the object thickness field, and the background gap field includes: For each object instance, the visible surface depth, the object thickness field, and the background gap field are superimposed along the same panoramic light direction to determine the radial depth of each surface layer in the multi-layer radial geometry corresponding to the object instance; wherein, the multi-layer radial geometry includes at least a first layer radial geometry, a second layer radial geometry, and a third layer radial geometry; The first layer radial geometry is the visible surface represented by the visible surface depth; The second layer of radial geometry is the back surface of the object instance, and the depth of the back surface is equal to the sum of the depth of the visible surface and the object thickness field; The third layer of radial geometry is the background surface exposed after the object instance is removed, and the depth of the background surface is equal to the sum of the depth of the back surface and the background gap field. The first layer of radial geometry, the second layer of radial geometry, and the third layer of radial geometry are arranged sequentially along the same panoramic light direction.

3. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 1 or 2, characterized in that, The step of predicting, for each object instance mask, the object thickness field along the panoramic ray direction and the background gap field from the back of the object instance to the next surface along the panoramic ray direction using a mask-by-mask perspective query network includes: Determine the centroid position of the object instance mask in the spherical coordinate system of the ERP panoramic image; The ERP panoramic image is horizontally rotated based on the centroid position to obtain the rotated ERP panoramic image. The multi-scale visual features and light features corresponding to the ERP panoramic image are obtained, and the rotated ERP panoramic image, the multi-scale visual features, the light features, the object instance mask and the visible surface depth are projected into the pinhole cropping view to obtain an aligned multi-channel perspective view. The multi-channel perspective view is input into the mask-by-mask perspective query network to determine the object thickness field and background gap field corresponding to the object instance mask.

4. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 3, characterized in that, The step of inputting the multi-channel perspective view into the mask-by-mask perspective query network to determine the object thickness field and background gap field corresponding to the object instance mask includes: The multi-channel perspective view is input into the mask-by-mask perspective query network. Through the shared dense prediction head in the mask-by-mask perspective query network, the multi-channel perspective view is subjected to pixel-by-pixel dense prediction inside the object instance mask to obtain the object thickness field and background gap field corresponding to the object instance mask.

5. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 1, characterized in that, The step of segmenting the ERP panoramic image and extracting the masks of each object instance in the ERP panoramic image includes: The ERP panoramic image is scrolled along the horizontal longitude direction by a preset pixel distance to obtain multiple longitude-offset ERP panoramic images. Each of the longitude offset ERP panoramic images is segmented into instances to obtain the initial object instance mask corresponding to each of the longitude offset ERP panoramic images; The initial object instance mask corresponding to each of the longitude offset ERP panoramic images is scrolled in reverse to the original image coordinate system to obtain the candidate instance mask under a unified coordinate system. Based on the cross-union ratio, the overlap between the candidate instance masks is calculated. Candidate instance masks with an overlap greater than a preset instance matching threshold are grouped into the same object instance and the masks are fused to obtain the object instance masks of the ERP panoramic image.

6. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 1, characterized in that, Before the step of segmenting the ERP panorama into instances, the following steps are also included: Semantic segmentation is performed on the ERP panoramic image to identify the sky and / or the infinitely distant background region, and a first mask is generated. The region corresponding to the first mask is marked as an invalid region, and the invalid region is excluded when predicting the visible surface depth, the object thickness field, and the background gap field.

7. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 1, characterized in that, The step of generating a layered panoramic 3D Gaussian scene based on the multi-layered radial geometry includes: For the effective surface of each layer of the multi-layer radial geometry, the projection sampling frequency is calculated based on the radial depth, surface normal, and viewing direction; wherein, the projection sampling frequency is proportional to the square of the radial depth and inversely proportional to the absolute value of the dot product of the surface normal and the viewing direction. Based on the projection sampling frequency, sampling points are determined on the effective surface of each layer of geometry. The light rays, depth and shared features corresponding to each sampling point are queried, and the three-dimensional Gaussian primitives at the sampling points are generated. By merging the three-dimensional Gaussian primitives corresponding to all geometric layers in the multi-layer radial geometry, a layered panoramic three-dimensional Gaussian scene is obtained.

8. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 1 or 7, characterized in that, After generating the layered panoramic 3D Gaussian scene, the process also includes: Construct an outer sky sphere surrounding the layered panoramic 3D Gaussian scene, wherein the radius of the sky sphere is greater than the maximum radial depth of all Gaussian primitives in the layered panoramic 3D Gaussian scene; Generate the inner sky texture of the outer sky sphere based on the text prompts and / or the sky area in the ERP panoramic image; When rendering the layered panoramic 3D Gaussian scene, the outermost sky sphere is set to the outermost layer; when the rendering light does not hit any 3D Gaussian primitive in the layered panoramic 3D Gaussian scene, the color of the outermost sky sphere is returned.

9. The text-driven hierarchical panoramic 3D Gaussian scene generation method according to claim 1, characterized in that, Before generating the layered panoramic 3D Gaussian scene, the following steps are also included: The multi-layer radial geometry is then aligned with the left and right seams and boundaries using ERP.

10. A text-driven hierarchical panoramic 3D Gaussian scene generation system, characterized in that, The system is used to implement the text-driven hierarchical panoramic 3D Gaussian scene generation method as described in any one of claims 1-9, including: The panoramic instance segmentation module is used to obtain the equidistant cylindrical projection ERP panoramic image corresponding to the text prompt, perform instance segmentation on the ERP panoramic image, and extract the mask of each object instance in the ERP panoramic image. The perspective query and thickness prediction module is used to predict the object thickness field of the object instance corresponding to the object instance mask along the panoramic light direction, and the background gap field from the back of the object instance to the next surface along the panoramic light direction for each object instance mask through a mask-by-mask perspective query network. A multi-layer geometry synthesis module is used to obtain the visible surface depth of the ERP panoramic image along the panoramic light direction, and determine the multi-layer radial geometry corresponding to each object instance based on the visible surface depth, the object thickness field and the background gap field. The 3D Gaussian scene generation module is used to generate a layered panoramic 3D Gaussian scene based on the multi-layer radial geometry.

Images