A method for generating space constraint information based on real scene three-dimensional block plotting

Abstract

The application provides a space constraint information generation method based on real scene three-dimensional block plotting, which comprises the following steps: displaying a real scene three-dimensional scene based on WebGL, and defining a three-dimensional block; determining the view angle parameter of a current virtual camera corresponding to the three-dimensional block and the description of the block, determining three pixel-level aligned image channels to be synchronously output according to the view angle parameter of the current virtual camera and the description of the block; taking a geometry completion depth map as a geometry control condition, taking a binary mask image converted from a three-value mask image as a region control condition, and determining a generated region range according to the region control condition and a standard input format; and generating a complete geometry shape of the three-dimensional block according to a diffusion model and a real visual RGB image. At least the method can provide high-precision, pixel-level alignment, semantic completeness, complete geometry depth information containing a hidden region and a space constraint information distinguishing ability for an image generation model.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of surveying and mapping geographic information, real-scene 3D visualization and computer vision, and more specifically, to a method for generating spatial constraint information based on real-scene 3D block mapping. Background Technology

[0002] The mapping of three-dimensional scenes has important applications in many fields, such as land and space planning, beautification and mapping of real-scene three-dimensional platform results, emergency disaster assessment and simulation of damaged scene restoration, cultural tourism, and digital twin pre-drills of industrial parks.

[0003] Existing methods for generating spatial constraint information in real-world 3D block mapping suffer from at least the following problems:

[0004] 1. Existing depth maps only record scene depth (including foreground occlusion). The depth value of the occluded area is the distance to the foreground object, and the true geometric distance of the target object's surface is completely lost. Image generation models cannot obtain accurate depth constraints in the occluded area and can only "guess" to complete it, resulting in disordered depth hierarchy of the generated object and distortion of occlusion boundaries.

[0005] 2. Misalignment between depth map and RGB image: The separate screenshot methods have viewpoint deviation and resolution misalignment; the accuracy of monocular depth estimation is far lower than the true depth obtained by 3D rendering.

[0006] 3. Mask images cannot express occlusion relationships: Existing binary masks can only mark "whether it belongs to the target area", and cannot distinguish between the visible part and the occluded part in the target area.

[0007] 4. Loss of semantic information in surveying: The structured surveying attributes stored in the real-scene 3D database cannot be automatically transferred to the image generation model.

[0008] 5. Data compliance risk: Directly modifying the 3D model undermines the authority of the surveying and mapping results. Summary of the Invention

[0009] In view of this, the present invention discloses a method for generating spatial constraint information based on real-scene 3D block mapping, which can at least provide image generation models with high-precision, pixel-level aligned, semantically complete spatial constraint information containing complete geometric depth information of occluded areas and occlusion discrimination capabilities.

[0010] Specifically, the present invention is achieved through the following technical solutions:

[0011] This application proposes a method for generating spatial constraint information based on real-scene 3D block mapping, characterized in that the method includes:

[0012] To display a real-world 3D scene based on WebGL, 3D blocks are defined, including newly created blocks and existing blocks.

[0013] Determine the viewpoint parameters of the current virtual camera and the description of the volume block corresponding to the 3D volume block. Based on the viewpoint parameters of the current virtual camera and the description of the volume block, in the same rendering pipeline, using the same viewpoint parameters and projection matrix, determine the synchronous output of three pixel-level aligned image channels. The three pixel-level aligned image channels include: a ternary mask image, a geometrically completed depth image, and a true visual RGB image.

[0014] The geometrically completed depth map is used as the geometric control condition, and the binary mask map obtained by converting the ternary mask image is used as the region control condition. The range of the generated region is determined according to the region control condition and the standard input format. The standard input format is a standard single-channel depth map, which is compatible with the standard input format of the depth control branch of the image generation model.

[0015] Based on the diffusion model and the real visual RGB image, the complete geometry of the three-dimensional block is generated.

[0016] In some embodiments, a new block is generated through the following steps:

[0017] In response to the user's sequential clicks along the boundary of the proposed building's base, a base polygon is generated;

[0018] In response to the height value input by the user, a new solid is generated based on the base polygon and the height value;

[0019] The existing blocks are selected through the following steps:

[0020] In response to a user clicking on a selected block in the scene, its base polygon and height are extracted to generate the corresponding existing block.

[0021] In some embodiments, a ternary mask image is generated through the following steps:

[0022] With depth testing disabled and no writing to the depth buffer, the projected polygon of the 3D block is rasterized to obtain a complete mask M_full, and the pixel value in the projected area of ​​the 3D block is 1.

[0023] With depth testing enabled and depth buffer written, the 3D block is rendered normally to obtain a visible mask M_vis, and the pixel values ​​of pixels within the projection area of ​​the 3D block that pass the depth test are set to 1.

[0024] The complete mask M_full and the visible mask M_vis are merged into a ternary mask image M.

[0025] In some embodiments, a geometrically completed depth image is generated through the following steps:

[0026] Standard depth test rendering is performed on pre-stored real-scene 3D geometric data to generate a scene depth map D_scene. The scene depth map D_scene is a single-channel grayscale image, and each pixel value represents the real 3D spatial distance from the nearest surface in the scene to the optical center of the virtual camera.

[0027] Based on the geometric parameters of the 3D block, a block clipping operation is performed on the rendering pipeline to determine the complete surface depth map D_block of the 3D block. Each pixel value in the surface depth map D_block represents the true 3D distance of the surface of the 3D block in that pixel direction.

[0028] Based on the ternary mask image M, the scene depth map D_scene and the surface depth map D_block are fused at the pixel level to generate the geometrically completed depth image D_final.

[0029] In some embodiments, a volume pruning operation is performed on the rendering pipeline through the following steps:

[0030] Set a clipping plane or use the bounding box of the volume as a rendering range constraint in the rendering pipeline so that geometric fragments located inside or on the surface of the volume participate in depth testing and depth writing, while foreground occlusions do not participate in rendering.

[0031] In some embodiments, a realistic visual RGB image is generated through the following steps:

[0032] In the same rendering pipeline, conventional texture mapping and lighting rendering are performed on the stored real-world 3D geometric data to generate a realistic visual RGB image with the same resolution as the depth image; the realistic visual RGB image is the original scene base map.

[0033] In some embodiments, three pixel-aligned image channels are output through the following steps:

[0034] The ternary mask M, the geometrically completed depth map D_final, and the real visual RGB image are aligned to have identical camera parameters and projection matrices, and then output synchronously after being physically aligned at the pixel level.

[0035] In some embodiments, generating the complete geometry of the three-dimensional block based on the diffusion model and the real visual RGB image includes:

[0036] The original RGB image is encoded into a latent representation z_orig using a variational autoencoder (VAE);

[0037] For each region, if the binary mask represented by the binary mask image of that region is the region to be generated, then the latent variables of that region are updated during the denoising process, and new content is generated by using the depth value of the corresponding region in the standard single-channel depth map D_final as a geometric constraint.

[0038] For each region, if the binary mask represented by the binary mask image of that region is a non-generated region, then the original base image remains unchanged;

[0039] The original RGB image is a visual image rendered by the WebGL 3D engine, and the original RGB image is one that has not undergone the diffusion model.

[0040] In some embodiments, the method further includes:

[0041] The regions corresponding to non-generated regions in the potential representation z_orig are kept as their original values;

[0042] Initialize the region corresponding to the region to be generated in the potential representation z_orig to pure Gaussian noise;

[0043] Using a standard single-channel depth map as the depth control condition and a standard binary mask as the mask control condition, the latent variables in the region to be generated are updated through iterative denoising of the diffusion model, and the initial generated image I_gen is obtained after decoding.

[0044] Based on the original ternary mask M, the initially generated image I_gen is post-processed to generate the complete geometry of the three-dimensional block.

[0045] In some embodiments, the post-processing steps for the initially generated image I_gen include:

[0046] The initially generated image I_gen is post-processed based on the pixel value of each pixel in the ternary mask image.

[0047] The process involves post-processing the initially generated image I_gen based on the pixel value of each pixel in the ternary mask image, including:

[0048] If the pixel value indicates that the pixel is a non-projected area, then the original base image is retained;

[0049] If the pixel value indicates that the pixel is a visible area, then the generation result of I_gen is retained;

[0050] If the pixel value indicates that the pixel is an occluded area, then the generated result of I_gen is overwritten with the pixel value of the corresponding position in the real visual RGB image.

[0051] The spatial constraint information generation method based on real-scene 3D block mapping proposed in this application can at least provide image generation models with high-precision, pixel-level aligned, semantically complete spatial constraint information that includes complete geometric depth information of occluded areas and occlusion discrimination capabilities. Attached Figure Description

[0052] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0053] Figure 1 A flowchart illustrating the method for generating spatial constraint information based on real-scene 3D block mapping provided in this application embodiment;

[0054] Figure 2 A schematic diagram illustrating the failure of the existing binary masking method provided in this application to generate only the visible portion;

[0055] Figure 3 A schematic diagram illustrating the failed result of generating a complete projection using the existing binary masking method provided in this application embodiment;

[0056] Figure 4 This is a schematic diagram of the original white model building in the terrain-level realistic 3D scene provided in the embodiments of this application;

[0057] Figure 5 A schematic diagram of the interface in which a user selects a target white model by clicking in a terrain-level realistic 3D scene provided in this application embodiment;

[0058] Figure 6 A schematic diagram of the interface for creating a new block by clicking on the base corner points and inputting the height value in a city-level oblique photogrammetry real-world 3D scene provided in this application embodiment;

[0059] Figure 7 This is a schematic diagram of a ternary mask image provided in an embodiment of this application;

[0060] Figure 8 The scene depth map D_scene provided for the first depth rendering output in the embodiments of this application;

[0061] Figure 9 The volume clipping depth map provided for the second depth rendering output in this application embodiment;

[0062] Figure 10 The geometrically completed depth map output after pixel-level fusion of the depth map provided in the embodiments of this application;

[0063] Figure 11 This is a schematic diagram of a real visual RGB image provided for an embodiment of this application;

[0064] Figure 12 This is a terrain-level scene enhancement rendering result diagram provided in an embodiment of this application;

[0065] Figure 13 The final generated result in the city-level occlusion scene provided in the embodiments of this application.

[0066] Figure 14 A schematic diagram of the E-shaped complex self-shading substrate block provided in the embodiments of this application.

[0067] Figure 15 This is a schematic diagram of the E-shaped complex self-occluding substrate generation result provided in an embodiment of this application. Detailed Implementation

[0068] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of systems and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0069] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0070] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0071] This application relates to the interdisciplinary field of surveying and mapping geographic information, real-scene 3D visualization and computer vision. Specifically, it relates to a method for generating spatial constraint information based on real-scene 3D block mapping.

[0072] The spatial constraint information generation method based on real-scene 3D block mapping proposed in this application can be applied to the following fields:

[0073] 1. Territorial spatial planning and natural resource approval:

[0074] Application method: In terrain-level DOM+DEM+white model and city-level oblique photogrammetry real-world 3D scenes, planners can plot the 3D blocks of proposed buildings and automatically generate planning renderings that conform to land use planning attributes, have accurate geometric scales, and realistic occlusion relationships. The original statutory surveying results are not altered throughout the process, and the output drawings are used for planning publicity, scheme presentation, and land use pre-approval review, solving the problems of dimensional distortion, occlusion errors, and non-compliant modification of surveying data in traditional manual drawing.

[0075] 2. Real-world 3D platform result enhancement and rendering:

[0076] Application method: In response to the defects of existing real-scene 3D models such as hollow textures, simple white model materials, and model melting distortion, spatial constraint information is generated by 3D block plotting, which drives diffusion AI to generate realistic restoration effect pictures in batches and output standardized promotional pictures and display pictures in batches, replacing manual 3D model repair and PS hand-drawing, and significantly reducing the drawing cost.

[0077] 3. Visual simulation of urban renewal and renovation of old residential areas:

[0078] Application method: In the 3D scene of the city by oblique photography, the blocks of the buildings to be transformed and the supporting facilities to be added are plotted, and the real scene effect map after the transformation is automatically generated to support the review of transformation plan and public announcement.

[0079] 4. Emergency disaster assessment and damage scenario restoration simulation:

[0080] Application method: In the disaster scene 3D model, the buildings to be repaired and the blocks to be rebuilt are plotted, and the reconstruction simulation effect map is generated based on the complete depth constraints to assist in the visualization evaluation of the post-disaster reconstruction plan.

[0081] 5. Digital twin simulation of cultural tourism and industrial parks:

[0082] Application: Newly built venues, factories, and supporting buildings are plotted on a 3D base of scenic spots and industrial parks to generate a visually realistic pre-show map that fits the site scale and has natural occlusion relationships, which can be used for project investment promotion and scheme presentation.

[0083] 6. Visualized presentation of construction plans for housing and urban-rural development projects:

[0084] Application: 3D block mapping of municipal roads, bridges, and supporting buildings; automatic generation of project site selection and construction effect renderings for project approval and construction plan reporting.

[0085] 7. Automated map generation from the public surveying and mapping service platform:

[0086] Application method: A real-scene 3D query system for the public. After the user selects a plot of land, a realistic architectural rendering is automatically generated, enhancing the public's experience of visualizing geographic information.

[0087] 8. It can be embedded as an independent functional module into real-scene 3D management platforms, digital twin bases, planning auxiliary approval systems, etc., providing standardized spatial constraint generation interfaces and connecting to various open-source / commercial diffusion image generation models.

[0088] The related technologies have at least the following drawbacks:

[0089] (a) The technical level and inherent defects of real-scene 3D visualization.

[0090] Real-world 3D geographic scenes have been widely used in fields such as land spatial planning, natural resource management, and disaster assessment. Based on coverage and precision requirements, current mainstream construction schemes are divided into three levels:

[0091] Topographic level (national / provincial): This method uses a three-layer overlay of digital orthophoto (DOM), digital elevation model (DEM), and white model (extruded block) of buildings. The white model only contains outlines and heights, lacks textures and materials, and the rendering effect is severely disconnected from the style of the realistic image, failing to provide accurate visual reference for planning decisions.

[0092] City-level (key areas of cities and counties): Oblique photogrammetry is used to construct realistic 3D mesh models. This approach is superior to terrain-level in terms of geometric detail and texture realism, but it has obvious inherent defects: the texture of building facades is blurry and the detailed structure is difficult to express clearly; trees and ancillary facilities appear as a "creamy" melted form; the water surface is broken or presents incorrect triangular faces; the model edges are irregular; and problems such as texture stretching and voids caused by occlusion or matching errors are common.

[0093] Component-level (landmark buildings / model projects): This method uses detailed manual modeling or CG technology, resulting in high-quality images but extremely high costs and very limited coverage, making it unsuitable as a widely applicable approach.

[0094] (ii) Limitations of direct editing of 3D models.

[0095] To address the aforementioned visualization deficiencies, the industry urgently needs to enhance and edit scenes (such as replacing real materials in white models, fixing texture holes in tilted models, and inserting proposed buildings into planned land use). However, directly editing 3D models faces two constraints:

[0096] High cost: Manual modeling using software such as 3ds Max and Blender can cost tens of thousands to hundreds of thousands of yuan to repair a single building, with a cycle of several days to several weeks, making it difficult to cover the entire area.

[0097] Data authority constraint: Real-world 3D models (especially oblique photogrammetry mesh and DOM / DEM) are legally recognized surveying results, possessing authority and traceability. Arbitrary alteration of their geometric structure or texture will destroy their original evidentiary value.

[0098] (iii) On-demand drawing strategy.

[0099] Given the challenges of directly editing 3D models, the industry has developed an "on-demand mapping" strategy: extracting 2D images from a 3D scene, editing these images, and then using them for presentations, promotional materials, etc. This method does not modify the original 3D mapping results, but only changes their visualization, thus combining compliance with flexibility.

[0100] (iv) Cost and accuracy issues of traditional image editing methods.

[0101] Before generative AI technology matures, 2D image editing mainly relies on manual work using software such as Photoshop. This method has two shortcomings:

[0102] Costs remain high: the hourly rate for skilled graphic designers is still considerable for batch graphic production tasks.

[0103] Geometric accuracy cannot be guaranteed: manual outlining is difficult to accurately correspond to the real position and scale in three-dimensional space, especially in the case of perspective projection distortion. The deviation between the freehand drawn image and the boundary of the real object is obvious, resulting in positional offset and size distortion in the editing result.

[0104] (v) Technical bottlenecks of existing diffusion model editing methods.

[0105] In recent years, image generation techniques, represented by diffusion models (such as Stable Diffusion), have significantly reduced the cost and barrier to entry for image editing. Users can quickly generate stylized content using text prompts and hand-drawn masks. However, existing methods still cannot solve the problems of geometric accuracy and occlusion relationships in real-world 3D scenes.

[0106] Information Mixing: Real-world 3D platforms generally support 3D plotting, allowing users to plot blocks and take screenshots. However, the RGB channels in the screenshots contain a mixture of multiple information, including lighting, color, texture, shadow, normals, and depth, which cannot be separated. Diffusion models can only obtain the final pixel color and cannot acquire pure geometric depth information or accurately plot the boundaries of the region.

[0107] Depth information is missing—especially in occluded areas: Existing solutions acquire "scene depth maps"—that is, depth information including foreground occluders. The depth values ​​of the occluded area (the location of the target object) are covered by the foreground object, completely losing the true depth information of the target object's surface. Even with monocular depth estimation, its accuracy is far lower than that of depth images obtained directly from 3D model rendering, and there is a lack of strict spatial geometric consistency between the estimated depth map and the user mask. More importantly, even if the image generation model knows that "an object is occluded here," it cannot know the precise geometric distance of the occluded object's surface and can only rely on the model's own "imagination" to fill in the gaps, resulting in disordered depth levels and edge distortion of the generated object.

[0108] The dilemma of occlusion: Binary masking methods have a fundamental contradiction. If the mask only includes the visible part of the block (through depth testing), the diffusion model can only generate locally visible structures, and the continuous surfaces occluded by the foreground are completely missing, resulting in incomplete object shapes and phenomena such as "distortion" or "hanging," as shown in the example. Figure 2 As shown. If the mask contains the complete projected area of ​​the block (without depth testing), the diffusion model will generate new content for all pixels within the projected area (including parts that should be occluded by the foreground), causing the new object to incorrectly cover the foreground and disrupting the occlusion relationship, such as... Figure 3 As shown, existing methods cannot simultaneously provide the diffusion model with both "the complete geometry of the volume" and "the identification of the parts occluded by the foreground".

[0109] Lack of utilization of geographic attributes: Existing methods rely on manually edited free text prompts, which cannot automatically read the structured mapping attributes such as land use, building type, and structural parameters that come with the real-world 3D model. As a result, the generated content cannot match legal planning constraints and entity attributes, and its professional adaptability is insufficient.

[0110] In summary, the existing technology has at least the following problems:

[0111] 1. Complete loss of target depth in occluded areas: Existing depth maps only record scene depth (including foreground occlusion). The depth value of the occluded area is the distance to the foreground object, and the true geometric distance of the target object's surface is completely lost. Image generation models cannot obtain accurate depth constraints in occluded areas and can only "guess" to complete them, resulting in disordered depth hierarchy of generated objects and distortion of occlusion boundaries.

[0112] 2. Misalignment between depth map and RGB image: The separate screenshot method has viewpoint deviation and resolution misalignment; the accuracy of monocular depth estimation is much lower than the true depth obtained by 3D rendering.

[0113] 3. Masking images cannot express occlusion relationships: Existing binary masks can only mark "whether it belongs to the target area", and cannot distinguish between the visible part and the occluded part in the target area.

[0114] 4. Loss of semantic information in surveying: The structured surveying attributes stored in the real-scene 3D database cannot be automatically transferred to the image generation model.

[0115] 5. Data compliance risk: Directly modifying the 3D model undermines the authority of the surveying and mapping results.

[0116] Based on this, this application proposes a method for generating spatial constraint information based on real-scene 3D block mapping.

[0117] Please see Figure 1 This application discloses a flowchart illustrating a method for generating spatial constraint information based on real-world 3D block mapping. For example... Figure 1 As shown, the method for generating spatial constraint information based on real-scene 3D block mapping includes:

[0118] S101. Display a real-world 3D scene based on WebGL, and define 3D blocks, including newly created blocks and existing blocks.

[0119] Specifically, the new block is generated through the following steps:

[0120] In response to the user's sequential clicks along the boundary of the proposed building's base, a base polygon is generated;

[0121] In response to the height value input by the user, a new solid is generated based on the base polygon and the height value;

[0122] The existing blocks are selected through the following steps:

[0123] In response to a user clicking on a selected block in the scene, its base polygon and height are extracted to generate the corresponding existing block.

[0124] The real-world 3D scene includes: DOM + DEM + white model or city-level oblique photography Mesh.

[0125] In some embodiments, users can click sequentially along the boundary of the proposed building base to generate a base polygon (each vertex has three-dimensional spatial coordinates of longitude, latitude, and elevation), and then input the height value. The system will automatically generate a columnar block based on the base polygon and the height value.

[0126] In some embodiments, users can directly click to select an existing block (such as a white model) in the scene, and the system will automatically extract its base polygon and height (and simultaneously extract the semantics of the geographic entities associated with the white model to participate in the construction of prompt words) to generate the corresponding block.

[0127] For example, in a terrain-level scene constructed using DOM+DEM+white model, it is necessary to render a low-precision white model block of a certain plot of land as a skyscraper with realistic glass curtain wall materials for public geographic information services. Please refer to [link / reference]. Figures 4-5 , Figure 4 It is a raw white model building (white stretched blocks, lacking textures and materials). Figure 5 When a user directly clicks to select a target white model in a 3D scene, the system automatically obtains the base polygon and height of the white model and generates a corresponding semi-transparent red wireframe block (completely aligned with the position of the white model).

[0128] For example, in an oblique photogrammetry mesh model where a tall obstruction exists, a proposed high-rise building needs to be inserted behind the obstruction for planning assessment. See [link to relevant documentation]. Figure 6 , Figure 6 The system generates a semi-transparent green wireframe block (the block completely covers the expected building area) by having the user click on the corner points along the boundary of the proposed building's base on the plot behind the obstacle and enter the planned height limit value.

[0129] S102. Determine the viewpoint parameters of the current virtual camera and the description of the volume corresponding to the three-dimensional volume. Based on the viewpoint parameters of the current virtual camera and the description of the volume, in the same rendering pipeline, using the same viewpoint parameters and projection matrix, determine the synchronous output of three pixel-level aligned image channels. The three pixel-level aligned image channels include: a ternary mask image, a geometrically completed depth image, and a true visual RGB image.

[0130] In some optional embodiments, in a client-server architecture, the client sends the current virtual camera's viewpoint parameters and the volumetric description to the server. When the 3D volumetric design is a selected existing volumetric design (such as a white model), the system synchronously accesses the real-world 3D database attribute table and automatically extracts the semantic information of the geographic entities associated with the volumetric design, including structured fields such as land use type, building attributes, design height, and building structure. For example: in Figure 5 In the terrain-level scene shown, the selected white model corresponds to a high-rise building. The system extracts its structured data such as "entity type: high-rise building; number of floors: 30; structure: concrete; land use type: commercial and financial land" as constraints for image generation, ensuring that the generated content matches the land planning attributes and building positioning.

[0131] Here, this application can rely on structured semantic information to retrieve a pre-defined style reference library, rather than on user-inputted natural language prompts, to constrain the material and structural type of the generated content, ensuring that the generated content matches the land parcel's planning attributes and building positioning. The style reference library pre-stores typical building material and style samples, categorized by land use code, building type, and structural form. In a standalone architecture, the above process is completed within the same device.

[0132] The geometrically completed depth image is generated through the following steps:

[0133] Standard depth test rendering is performed on pre-stored real-scene 3D geometric data to generate a scene depth map D_scene. The scene depth map D_scene is a single-channel grayscale image, and each pixel value represents the real 3D spatial distance from the nearest surface in the scene to the optical center of the virtual camera.

[0134] Based on the geometric parameters of the 3D block, a block clipping operation is performed on the rendering pipeline to determine the complete surface depth map D_block of the 3D block. Each pixel value in the surface depth map D_block represents the true 3D distance of the surface of the 3D block in that pixel direction.

[0135] Based on the ternary mask image M, the scene depth map D_scene and the surface depth map D_block are fused at the pixel level to generate the geometrically completed depth image D_final.

[0136] The first step is to perform the first depth rendering—the scene depth map.

[0137] The stored real-world 3D geometric data (DEM + white model, oblique photogrammetry Mesh, etc.) is subjected to standard depth test rendering to generate a scene depth map D_scene. This image is a single-channel grayscale image, where each pixel value represents the real 3D spatial distance between the nearest surface and the optical center of the virtual camera in the scene corresponding to that pixel. This image contains depth information for all foreground occlusions.

[0138] An example of a city-level occlusion scene: scene depth map D_scene, see [link / reference] Figure 8 .

[0139] exist Figure 6 In the city-level occlusion scene shown, the scene depth map D_scene output by the first depth rendering is as follows: Figure 8 As shown, the depth value of the occluded area (the expected location of the building) is the distance of the foreground occluder, and the true depth of the building surface behind is completely lost.

[0140] Next, a second depth rendering—a volumetric clipping depth map—is performed.

[0141] Here, using the geometric parameters of the 3D block defined in step S101, a block clipping operation is performed on the rendering pipeline, that is: depth sampling is performed only on the 3D spatial surface covered by the block, rendering of all geometry outside the block (including foreground occluders) is turned off, and the complete surface depth map D_block of the block is output.

[0142] The specific implementation of this operation is as follows: clipping planes are set in the rendering pipeline, or the bounding box of the volume is used as the rendering range constraint, so that only geometric fragments located inside or on the surface of the volume participate in depth testing and depth writing. Foreground occluders are completely culled because they are located outside the volume and do not participate in rendering.

[0143] Each pixel value in D_block represents the true 3D distance of the block surface in that pixel direction, i.e., the "ideal depth" when the target object is not occluded. Pixel values ​​outside the block's projection area are set to invalid 0 values.

[0144] An example, city-level occlusion scenario: volume clipping depth map D_block, see [link / reference] Figure 9 .

[0145] exist Figure 6 In the scene shown, the second depth rendering uses volumetric clipping to truncate the rendering pipeline, completely removing foreground occlusions. The output volumetric clipping depth map D_block is as follows: Figure 9 As shown, each pixel within the block projection area accurately records the true three-dimensional distance of the building surface.

[0146] After that, pixel-level fusion of the depth map can be performed—geometric completion of the depth map.

[0147] Using the previously generated ternary mask image M, D_scene and D_block are fused pixel-level to generate the final geometrically constrained depth map D_final.

[0148] For M=0 (non-mass projection area): D_final=D_scene (preserve the original depth of the scene);

[0149] For M=1 (occluded area) or M=2 (visible area): D_final=D_block (use the full surface depth of the block to remove the influence of occlusion);

[0150] After merging, D_final presents the complete, unobstructed surface depth of the target object within the volume projection area, and the original depth of the scene outside the volume projection area.

[0151] An example, city-level occlusion scenario: geometric completion depth map D_final, see [link / reference] Figure 10 ,

[0152] exist Figure 6 In the scene shown, the D_scene and D_block are merged into D_final using the M generated in step C1. Figure 10As shown—M=1 (occluded area) and M=2 (visible area) both use the depth value of D_block, while M=0 uses the depth value of D_scene. The block projection area (including the occluded area and the visible area) presents the complete building surface depth.

[0153] Technical contribution of the depth dual-rendering scheme: Existing depth maps only record the distance to the scene surface (D_scene), and the target depth in occluded areas is completely lost. This invention achieves accurate acquisition of the surface depth of the target object in the occluded area through a second rendering (volume clipping), and generates a "geometric completion depth map" through pixel-level fusion, enabling the image generation model to obtain accurate depth constraints in the occluded area.

[0154] Specifically, a ternary mask image is generated through the following steps:

[0155] With depth testing disabled and no writing to the depth buffer, the projected polygon of the 3D block is rasterized to obtain a complete mask M_full, and the pixel value in the projected area of ​​the 3D block is 1.

[0156] With depth testing enabled and depth buffer written, the 3D block is rendered normally to obtain a visible mask M_vis, and the pixel values ​​of pixels within the projection area of ​​the 3D block that pass the depth test are set to 1.

[0157] The complete mask M_full and the visible mask M_vis are merged into a ternary mask image M.

[0158] The vertices (base corners and corresponding top corners) of the block defined in step S101 are transformed by perspective projection according to the current camera parameters and projection matrix to obtain the projected polygon in the screen coordinate system. A ternary mask image with the same resolution as the target is generated using the following method;

[0159] Perform two rasterization operations within the same rendering pipeline;

[0160] First rasterization (obtaining the complete projection area): Depth testing is disabled and no writing is done to the depth buffer. The projected polygons of the block are rasterized to obtain the complete mask M_full. Pixel values ​​within the block's projection area are set to 1. This operation is unaffected by any foreground occlusion and preserves the complete projected shape of the block.

[0161] Second rasterization (obtaining the visible area): Enable depth testing and write to the depth buffer, perform normal rendering on the block, obtain the visible mask M_vis, and set the pixel value of only the pixel within the block's projection area that passes the depth test (i.e. is not occluded by foreground objects) to 1.

[0162] Merge M_full and M_vis into a ternary mask image M, with pixel values ​​defined as follows:

[0163]

[0164] Here, the essential difference between the ternary mask and existing technologies lies in the fact that the ternary mask generation of this invention differs fundamentally from existing technologies that perform morphological dilation and edge expansion on two-dimensional images. Existing mask expansion is based on two-dimensional pixel-level image processing operations, which cannot obtain accurate three-dimensional spatial depth information, and the expansion range can only be set empirically. In contrast, this invention performs two rasterization operations within the same rendering pipeline using the same depth buffer (Z-buffer), directly defining the occlusion relationship from the depth comparison level of the three-dimensional geometric data, achieving accuracy completely consistent with the original three-dimensional mapping data.

[0165] An example, city-level occlusion scenario: ternary mask M, see Figure 7 .

[0166] exist Figure 6 In the scenario shown, the ternary mask image output by the server in step C1 is as follows: Figure 7 As shown: M_full marks the complete projection of the building (including the occluded parts); M_vis only marks the visible parts that are not occluded; after merging, M=1 is the occluded area (gray), and M=2 is the visible area (white).

[0167] Specifically, the following steps can be used to perform volume trimming on the rendering pipeline:

[0168] Set a clipping plane or use the bounding box of the volume as a rendering range constraint in the rendering pipeline so that geometric fragments located inside or on the surface of the volume participate in depth testing and depth writing, while foreground occlusions do not participate in rendering.

[0169] The following steps can be used to generate a realistic visual RGB image:

[0170] In the same rendering pipeline, conventional texture mapping and lighting rendering are performed on the stored real-world 3D geometric data to generate a realistic visual RGB image with the same resolution as the depth image; the realistic visual RGB image is the original scene base map.

[0171] Here, within the same rendering pipeline, the stored real-world 3D geometry data undergoes regular texture mapping and lighting rendering (for terrain-level rendering, the DOM is fitted to the DEM and a white model texture is overlaid; for city-level rendering, the oblique photogrammetric mesh is rendered for a realistic visual RGB image with the same resolution as the depth image). This image serves as the original scene base map, containing no block mapping or editing information. It is used to maintain the original appearance of non-edited areas during subsequent diffusion generation, and to perform content replacement or blending in occluded areas.

[0172] For example, the three-channel synchronous output includes: a ternary mask M generated by C1, a geometrically completed depth map D_final generated by C2c, and a real visual RGB image generated by C3. The three use the exact same camera parameters and projection matrix, and are output synchronously after being physically pixel-level strictly aligned.

[0173] Specifically, for city-level occlusion scenarios: see the original RGB base image. Figure 11 .

[0174] The original RGB image is a visual image rendered by the WebGL 3D engine, and the original RGB image is one that has not undergone the diffusion model.

[0175] exist Figure 6 In the scene shown, the synchronously output original oblique photogrammetric RGB image is as follows: Figure 11 As shown (including the complete texture of the occluded objects).

[0176] The process involves outputting three pixel-aligned image channels through the following steps:

[0177] The ternary mask M, the geometrically completed depth map D_final, and the real visual RGB image are aligned to have identical camera parameters and projection matrices, and then output synchronously after being physically aligned at the pixel level.

[0178] S103. The geometrically completed depth map is used as the geometric control condition, and the binary mask map obtained by converting the ternary mask image is used as the region control condition. The range of the generated region is determined according to the region control condition and the standard input format. The standard input format is a standard single-channel depth map, which is compatible with the standard input format of the depth control branch of the image generation model.

[0179] Here, the output geometric completion depth map D_final can be used as the geometric control condition, and the ternary mask image M output in step C1 can be converted into a binary mask image M_binary as the region control condition.

[0180] For example, the conversion rule of M_binary is as follows: for each pixel in M, if M=1 or M=2 (i.e. within the block projection area), then M_binary=1 (the area to be generated); if M=0 (the non-projection area), then M_binary=0 (keeping the original image).

[0181] M_binary is a standard binary mask, compatible with the standard input format of image generation models (including but not limited to the Inpainting / mask control branch of the diffusion model).

[0182] D_final is a standard single-channel depth map, compatible with the standard input format of depth control branches of image generation models (such as ControlNet's Depth mode).

[0183] D_final and M_binary serve as two independent control conditions input to the image generation model. D_final transmits the complete three-dimensional geometric structure of the target region to the model, while M_binary transmits the range of the region to be generated to the model.

[0184] S104. Based on the diffusion model and the real visual RGB image, generate the complete geometry of the three-dimensional block.

[0185] The process of generating the complete geometry of the three-dimensional block based on the diffusion model and the real visual RGB image includes:

[0186] The original RGB image is encoded into a latent representation z_orig using a variational autoencoder (VAE);

[0187] For each region, if the binary mask represented by the binary mask image of that region is the region to be generated, then the latent variables of that region are updated during the denoising process, and new content is generated by using the depth value of the corresponding region in the standard single-channel depth map D_final as a geometric constraint.

[0188] For each region, if the binary mask represented by the binary mask image of that region is a non-generated region, then the original base image remains unchanged.

[0189] The method further includes:

[0190] The regions corresponding to non-generated regions in the potential representation z_orig are kept as their original values;

[0191] Initialize the region corresponding to the region to be generated in the potential representation z_orig to pure Gaussian noise;

[0192] Using a standard single-channel depth map as the depth control condition and a standard binary mask as the mask control condition, the latent variables in the region to be generated are updated through iterative denoising of the diffusion model, and the initial generated image I_gen is obtained after decoding.

[0193] Based on the original ternary mask M, the initially generated image I_gen is post-processed to generate the complete geometry of the three-dimensional block.

[0194] The post-processing steps for the initially generated image I_gen include:

[0195] The initially generated image I_gen is post-processed based on the pixel value of each pixel in the ternary mask image.

[0196] The process involves post-processing the initially generated image I_gen based on the pixel value of each pixel in the ternary mask image, including:

[0197] If the pixel value indicates that the pixel is a non-projected area, then the original base image is retained;

[0198] If the pixel value indicates that the pixel is a visible area, then the generation result of I_gen is retained;

[0199] If the pixel value indicates that the pixel is an occluded area, then the generated result of I_gen is overwritten with the pixel value of the corresponding position in the real visual RGB image.

[0200] Here, this application employs an image generation architecture based on a diffusion model in this step. The original base image is derived from the real visual RGB image output in step C3.

[0201] The original RGB image is encoded into a latent representation z_orig using a variational autoencoder (VAE). Based on the binary mask M_binary:

[0202] Region M_binary=1 (region to be generated): During the denoising process, update the latent variables of this region, and generate new content using the depth value of the corresponding region in D_final as a geometric constraint.

[0203] Region M_binary=0 (non-generated region): Keep the original base map unchanged.

[0204] In the specific denoising process, the regions in z_orig corresponding to M_binary=0 are kept at their original values, while the regions corresponding to M_binary=1 are initialized with pure Gaussian noise. Then, using D_final as the depth control condition and M_binary as the mask control condition, the latent variables in the regions of M_binary=1 are updated through the iterative denoising process of the diffusion model. After decoding, the preliminary generated image I_gen is obtained.

[0205] For example, post-processing—occlusion restoration includes: post-processing the initially generated image I_gen based on the original ternary mask M:

[0206] For M=0 (non-projection area): retain the original base map;

[0207] For M=2 (visible region): retain the generation result of I_gen;

[0208] For M=1 (occluded area): overwrite the generated result of I_gen with the pixel value of the corresponding position in the original RGB base image.

[0209] Here, the technical motivation for post-processing is as follows: the purpose of using the M=1 region (the occluded area) for denoising updates is not to generate the final visible content, but to construct a continuous geometric manifold in the latent space of the diffusion model. Because the U-Net network of the diffusion model has a global receptive field and a self-attention mechanism, the latent variable updates of the occluded region provide the correct boundary conditions and geometric constraints for the generation of the visible region, ensuring smooth edges and continuous normals in the visible region, and avoiding problems such as shape anomalies and edge distortions. In the post-processing stage, the M=1 region is covered with the original base map, restoring the correct occlusion relationship.

[0210] In this way, the generated object has a complete geometry (because the model is generated based on the complete depth information of D_final within the projection region), while strictly adhering to the foreground and background occlusion relationships in the real scene (the occluded parts are covered by the original foreground in the final output). This solves the dilemma of "shape aberration" or "covering the foreground" in existing binary masks.

[0211] The essential difference between this invention and existing methods lies in the following: Existing methods use masks derived solely from segmentation or hand-drawing of 2D RGB images, failing to obtain accurate 3D spatial information; while the ternary mask of this invention originates from two rasterization operations within the same rendering pipeline, directly generated based on the depth buffer of 3D mapping data. Existing methods generate depth maps that are merely scene depth maps, completely losing the depth of occluded areas; this invention's D_final, through dual-depth rendering and pixel-level fusion, accurately preserves the true depth information of the target object's surface in occluded areas. In existing methods, occluded areas are either not involved in generation or are incorrectly visible; in this invention, occluded areas participate in the complete denoising process to ensure geometric continuity, and the occlusion relationship is ultimately restored through post-processing.

[0212] Example, terrain-level scene: generated results, see Figure 12 .

[0213] exist Figure 4 In the terrain-level scene shown (no foreground occlusion, M=1 does not exist, M_binary corresponds to the M=2 region), the diffusion model generates architectural images with realistic glass curtain wall materials and conforming to spatial lighting relationships based on D_final and M_binary, such as... Figure 12 As shown. Since the M=1 region does not exist in post-processing, the generated result is output directly.

[0214] For an example, a city-level scenario: see the generated results and comparisons. Figure 13 , combined Figure 2 , Figure 3 contrast.

[0215] exist Figure 6 In the occlusion scenario shown, the diffusion model performs denoising generation based on D_final (containing the complete depth of the occluded region) and M_binary (labeling the entire volume projection area). D_final provides an accurate depth prior for the occluded region, and the model generates a geometrically continuous complete building (avoiding...). Figure 2 The image shows the shape distortion problem of only generating the visible portion. In the post-processing stage, based on the original ternary mask M, the generated result for the M=1 region is replaced with the occluding pixels from the original base image. In the final image, only the visible portion of the building is displayed (unlike...). Figure 3 (As shown in the example of the problem of incorrect foreground coverage), the occluded part is hidden behind the occluder, but the model has already constructed a continuous geometry based on complete depth information. The result is as follows... Figure 13 As shown.

[0216] For example, see self-occluding blocks with complex base morphology. Figure 14 , Figure 15 .

[0217] The user needs to generate a building with an E-shaped plan. Figure 14 After the user clicks on 12 vertices along the E-shaped outline and specifies the height, the system generates an E-shaped cross-section 3D block. In step C2, a depth-dual rendering scheme generates a complete surface depth map D_block for the E-shaped block; in step C1, the E-shaped block forms a self-occluding complex polygon mask. Subsequent generation processes are the same as in the terrain-level implementation. Figure 15 To generate the results, it was verified that the present invention is not limited to rectangular or simple polygonal bases, but can support any complex polygonal base with a self-occluding state.

[0218] The core of this invention lies in providing image generation models with high-precision, pixel-level aligned, semantically complete spatial constraint information that includes complete geometric depth information of the occluded region and occlusion discrimination capability.

[0219] To achieve the above objectives, this invention makes key improvements to the generation of depth maps: dual depth rendering and pixel-level fusion—through two depth renderings (the first rendering of the scene depth including occluders, and the second using block clipping to remove occluders and obtain the complete surface depth of the block), and then pixel-level fusion based on a ternary mask, a "geometrically complete depth map" is generated. In this depth map, the pixel values ​​of the occluded areas accurately record the true 3D distance of the target object's surface, rather than the distance of the foreground occluders.

[0220] The ternary mask image described in this invention refers to a mask image that can simultaneously mark the non-volume projection region, the part of the volume projection region that is occluded by the foreground, and the part that is not occluded. Its typical implementation is to use three pixel values ​​of 0, 1, and 2 to represent the above three regions respectively.

[0221] This invention employs the same rendering pipeline to synchronously output three pixel-level aligned image channels. The generated spatial constraint information is input into the depth control channel and mask control channel of the image generation model in the form of a depth map and a mask map, respectively. The occlusion recovery of the occluded area is completed through post-processing.

[0222] Finally, it should be noted that although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe the features of specific embodiments of a particular invention. Certain features described in the various embodiments of this specification may also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0223] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0224] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0225] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating spatial constraint information based on real-scene 3D block mapping, characterized in that, The method includes: To display a real-world 3D scene based on WebGL, 3D blocks are defined, including newly created blocks and existing blocks. Determine the viewpoint parameters of the current virtual camera and the description of the volume block corresponding to the 3D volume block. Based on the viewpoint parameters of the current virtual camera and the description of the volume block, in the same rendering pipeline, using the same viewpoint parameters and projection matrix, determine the synchronous output of three pixel-level aligned image channels. The three pixel-level aligned image channels include: a ternary mask image, a geometrically completed depth image, and a true visual RGB image. The geometrically completed depth map is used as the geometric control condition, and the binary mask map obtained by converting the ternary mask image is used as the region control condition. The range of the generated region is determined according to the region control condition and the standard input format. The standard input format is a standard single-channel depth map, which is compatible with the standard input format of the depth control branch of the image generation model. Based on the diffusion model and the real visual RGB image, the complete geometry of the three-dimensional block is generated.

2. The method according to claim 1, characterized in that, Generate a new block using the following steps: In response to the user's sequential clicks along the boundary of the proposed building's base, a base polygon is generated; In response to the height value input by the user, a new solid is generated based on the base polygon and the height value; The existing block is selected through the following steps: In response to a user clicking on a selected block in the scene, its base polygon and height are extracted to generate the corresponding existing block.

3. The method according to claim 1, characterized in that, Generate a ternary mask image using the following steps: With depth testing disabled and no writing to the depth buffer, the projected polygon of the 3D block is rasterized to obtain a complete mask M_full, and the pixel value in the projected area of ​​the 3D block is 1. With depth testing enabled and depth buffer written, the 3D block is rendered normally to obtain a visible mask M_vis, and the pixel values ​​of pixels within the projection area of ​​the 3D block that pass the depth test are set to 1. The complete mask M_full and the visible mask M_vis are merged into a ternary mask image M.

4. The method according to claim 1, characterized in that, Generate a geometrically complete depth image using the following steps: Standard depth test rendering is performed on pre-stored real-scene 3D geometric data to generate a scene depth map D_scene. The scene depth map D_scene is a single-channel grayscale image, and each pixel value represents the real 3D spatial distance from the nearest surface in the scene to the optical center of the virtual camera. Based on the geometric parameters of the 3D block, a block clipping operation is performed on the rendering pipeline to determine the complete surface depth map D_block of the 3D block. Each pixel value in the surface depth map D_block represents the true 3D distance of the surface of the 3D block in that pixel direction. Based on the ternary mask image M, the scene depth map D_scene and the surface depth map D_block are fused at the pixel level to generate the geometrically completed depth image D_final.

5. The method according to claim 1, characterized in that, Perform volume trimming on the rendering pipeline using the following steps: Set a clipping plane or use the bounding box of the volume as a rendering range constraint in the rendering pipeline so that geometric fragments located inside or on the surface of the volume participate in depth testing and depth writing, while foreground occlusions do not participate in rendering.

6. The method according to claim 1, characterized in that, Generate a realistic visual RGB image using the following steps: In the same rendering pipeline, the stored real-world 3D geometric data is subjected to regular texture mapping and lighting rendering to generate a realistic visual RGB image with the same resolution as the depth image. The real visual RGB image is the original scene base map.

7. The method according to claim 1, characterized in that, Output three pixel-aligned image channels using the following steps: The ternary mask M, the geometrically completed depth map D_final, and the real visual RGB image are aligned to have identical camera parameters and projection matrices, and then output synchronously after being physically aligned at the pixel level.

8. The method according to claim 1, characterized in that, Based on the diffusion model and the real visual RGB image, the complete geometry of the three-dimensional block is generated, including: The original RGB image is encoded into a latent representation z_orig using a variational autoencoder (VAE); For each region, if the binary mask represented by the binary mask image of that region is the region to be generated, then the latent variables of that region are updated during the denoising process, and new content is generated by using the depth value of the corresponding region in the standard single-channel depth map D_final as a geometric constraint. For each region, if the binary mask represented by the binary mask image of that region is a non-generated region, then the original base image remains unchanged; The original RGB image is a visual image rendered by the WebGL 3D engine, and the original RGB image is one that has not undergone the diffusion model.

9. The method according to claim 8, characterized in that, The method further includes: The regions corresponding to non-generated regions in the potential representation z_orig are kept as their original values; Initialize the region corresponding to the region to be generated in the potential representation z_orig to pure Gaussian noise; Using a standard single-channel depth map as the depth control condition and a standard binary mask as the mask control condition, the latent variables in the region to be generated are updated through iterative denoising of the diffusion model, and the initial generated image I_gen is obtained after decoding. Based on the original ternary mask M, the initially generated image I_gen is post-processed to generate the complete geometry of the three-dimensional block.

10. The method according to claim 9, characterized in that, The post-processing steps for the initially generated image I_gen include: The initially generated image I_gen is post-processed based on the pixel value of each pixel in the ternary mask image. The process involves post-processing the initially generated image I_gen based on the pixel value of each pixel in the ternary mask image, including: If the pixel value indicates that the pixel is a non-projected area, then the original base image is retained; If the pixel value indicates that the pixel is a visible area, then the generation result of I_gen is retained; If the pixel value indicates that the pixel is an occluded area, then the generated result of I_gen is overwritten with the pixel value of the corresponding position in the real visual RGB image.

Images