A three-dimensional building model texture repairing method, system, terminal and storage medium based on a bimodal collaborative repairing network

Abstract

This invention belongs to the field of geographic information system technology and discloses a method, system, terminal, and storage medium for texture restoration of 3D building models based on a bimodal collaborative restoration network. The method includes: acquiring an image of the building to be restored; performing occlusion detection and segmentation on the image of the building to be restored based on a semantically guided visual model to obtain an occlusion region mask; inputting the image of the building to be restored and the occlusion region mask into a bimodal collaborative restoration network for texture restoration; wherein the bimodal collaborative restoration network is an image restoration network employing a symmetrical U-shaped encoding-decoding hierarchical structure fused state space model; and outputting a texture-restored building image. This invention significantly reduces the computational complexity of the model while ensuring restoration quality, is suitable for high-resolution, large-scale urban scenes, and can effectively improve the integrity, structural rationality, and visual realism of 3D building model textures.

Description

Technical Field

[0001] This invention relates to the field of geographic information system technology, and in particular to a method, system, terminal, and storage medium for texture restoration of three-dimensional building models based on a dual-modal collaborative repair network. Background Technology

[0002] In applications such as digital twin cities, smart city refined management, and 3D visualization analysis, 3D building models serve as key spatial carriers, demanding high levels of texture integrity and realism. Currently, oblique photogrammetry technology can efficiently acquire city-level 3D reality models; however, due to factors such as shooting angle, distribution of obstructions, and lighting conditions, building facades are easily obscured by trees, vegetation, vehicles, scaffolding, and temporary structures, leading to missing, fragmented, or severely distorted texture information.

[0003] The aforementioned issues not only affect the visual realism of 3D building models but also further reduce their reliability in applications such as planning analysis, digital asset management, and simulation. Therefore, achieving automated and highly consistent texture restoration of occluded areas of buildings without extensive manual intervention or additional reference images has become a core technical challenge that urgently needs to be addressed in the field of 3D reality modeling.

[0004] Existing methods for repairing 3D building models still face the challenge of balancing the fidelity of the repair results with efficiency. They either suffer from poor semantic consistency and blurred textures, or have high computational complexity, making them unsuitable for the efficient repair needs of large-scale urban 3D models.

[0005] Therefore, existing technologies still need improvement. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a method, system, terminal and storage medium for texture repair of three-dimensional building models based on a dual-modal collaborative repair network, so as to solve the problem that existing three-dimensional building model repair methods cannot simultaneously achieve both the fidelity of repair results and efficiency.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention provides a method for texture restoration of three-dimensional architectural models based on a dual-modal collaborative repair network, comprising: Obtain images of the building to be repaired; The image of the building to be repaired is occlusion detected and segmented based on a semantically guided visual model to obtain an occlusion mask. The image of the building to be repaired and the mask of the occluded area are input into a bimodal collaborative inpainting network; wherein, the bimodal collaborative inpainting network is an image inpainting network that uses a symmetrical U-shaped encoder-decoder hierarchical structure and a fused state space model; the bimodal collaborative inpainting network includes: a context compensation hybrid module; the context compensation hybrid module consists of a deep learning model branch and a structured state space model branch set in parallel; The encoder of the dual-modal collaborative repair network performs overlapping convolutions to decompose the image of the building to be repaired into features of multiple scales, and performs stepwise downsampling to obtain global structure and local texture fusion features. Based on the global structure and local texture fusion features, the decoder of the dual-modal collaborative repair network is used for upsampling, and global average pooling and activation function mapping are performed in the pixel-level channels using the dimensional reciprocal attention mechanism to obtain the texture-repaired building image. Output the texture-restored building image.

[0008] In one implementation, the semantically guided visual model performs occlusion detection and segmentation on the image of the building to be repaired to obtain an occlusion region mask, including: Based on the semantically guided visual model, the building and the target object causing the occlusion in the image of the building to be repaired are detected and located at the semantic level using preset text prompts, so as to obtain the initial detection results of the occlusion area. The initial detection results of the occluded region are segmented at the pixel level based on the image segmentation model to obtain complete boundary information of the occluded region, and the occluded region mask is obtained based on the boundary information of the occluded region.

[0009] In one implementation, the encoder of the dual-modal collaborative repair network performs overlapping convolutions to decompose the image of the building to be repaired into features at multiple scales, and performs stepwise downsampling to obtain global structure and local texture fusion features, including: Channel stitching is performed on the image of the building to be repaired and the mask of the occluded area to obtain the stitched image features; Overlapping convolutions are performed on the stitched image features to generate initial mapping features; The initial mapping features are downsampled step by step, and the semantic understanding and spatial detail fusion of the features are optimized through the context compensation hybrid module and the hierarchical reconstruction attention module to obtain the global structure and local texture fusion features.

[0010] In one implementation, the fusion of semantic understanding and spatial details of the optimized features through the context-compensated hybrid module and the hierarchical reconstruction attention module to obtain the global structure and local texture fusion features includes: Global context dependency modeling is performed based on the deep learning model branch of the context compensation hybrid module to capture scene semantic structure information, and temporal and spatial joint optimization is performed based on the structured state space model branch of the context compensation hybrid module. The hierarchical reconstruction attention module performs weighted fusion of feature maps at different scales to obtain the global structure and local texture fusion features.

[0011] Secondly, the present invention provides a texture restoration system for three-dimensional architectural models based on a dual-modal collaborative repair network, comprising: The image acquisition module is used to acquire images of the building to be repaired. The occlusion detection and segmentation module is used to perform occlusion detection and segmentation on the image of the building to be repaired based on a semantically guided visual model, and obtain an occlusion region mask. A bimodal inpainting module is used to input the image of the building to be repaired and the mask of the occluded area into a bimodal collaborative inpainting network. The bimodal collaborative inpainting network is an image inpainting network employing a symmetrical U-shaped encoder-decoder hierarchical structure and a fusion state space model. The bimodal collaborative inpainting network includes a context-compensated hybrid module, which consists of parallel deep learning model branches and structured state space model branches. The encoder of the bimodal collaborative inpainting network performs overlapping convolutions to decompose the image of the building to be repaired into features at multiple scales, and performs progressive downsampling to obtain global structure and local texture fusion features. Based on the global structure and local texture fusion features, the decoder of the bimodal collaborative inpainting network performs upsampling, and global average pooling and activation function mapping are applied in the pixel-level channels using a dimensional reciprocal attention mechanism to obtain the texture-repaired building image. The output module is used to output the texture-repaired building image.

[0012] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a three-dimensional building model texture repair program based on a bimodal collaborative repair network, and the three-dimensional building model texture repair program based on the bimodal collaborative repair network is executed by the processor to implement the operation of the three-dimensional building model texture repair method based on the bimodal collaborative repair network as described in the first aspect.

[0013] Fourthly, the present invention also provides a computer-readable storage medium storing a three-dimensional architectural model texture restoration program based on a bimodal collaborative repair network. When executed by a processor, the three-dimensional architectural model texture restoration program based on a bimodal collaborative repair network is used to implement the operation of the three-dimensional architectural model texture restoration method based on a bimodal collaborative repair network as described in the first aspect.

[0014] The present invention, by employing the above technical solution, has the following effects: This invention constructs a semantically guided occlusion detection mechanism to accurately identify building occlusion areas. Using the original image and occlusion mask as joint inputs, it introduces a dual-modal collaborative repair network that integrates a deep learning model and a structured state space model. This network performs context-aware and structurally constrained texture reconstruction of the occluded areas. Finally, through multi-scale feature modeling and a reciprocal dimensional attention mechanism, it achieves collaborative optimization of global semantic consistency and local texture details of the building facade. This invention significantly reduces model computational complexity while ensuring repair quality, making it suitable for high-resolution, large-scale urban scenes. It can effectively improve the integrity, structural rationality, and visual realism of 3D building model textures. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the three-dimensional building model texture restoration method based on a dual-modal collaborative repair network in this invention.

[0017] Figure 2 This is a schematic diagram of the overall structure of the dual-modal collaborative repair network in this invention.

[0018] Figure 3 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0019] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0021] Exemplary methods Oblique photogrammetry can efficiently acquire city-level 3D reality models, but due to factors such as shooting angle, distribution of obstructions and lighting conditions, building facades are easily obscured by trees, vegetation, vehicles, scaffolding and temporary structures, resulting in missing, fragmented or severely distorted texture information.

[0022] The aforementioned issues not only affect the visual realism of 3D building models but also further reduce their reliability in applications such as planning analysis, digital asset management, and simulation. Therefore, achieving automated and highly consistent texture restoration of occluded areas of buildings without extensive manual intervention or additional reference images has become a core technical challenge that urgently needs to be addressed in the field of 3D reality modeling.

[0023] Existing methods for repairing 3D building models still face the challenge of balancing the fidelity of the repair results with efficiency. They either suffer from poor semantic consistency and blurred textures, or have high computational complexity, making them unsuitable for the efficient repair needs of large-scale urban 3D models.

[0024] To address the above technical problems, this invention provides a method for texture restoration of 3D building models based on a bimodal collaborative restoration network. The method includes: acquiring an image of the building to be restored; performing occlusion detection and segmentation on the image of the building to be restored based on a semantically guided visual model to obtain an occlusion region mask; inputting the image of the building to be restored and the occlusion region mask into a bimodal collaborative restoration network for texture restoration; wherein the bimodal collaborative restoration network is an image restoration network employing a symmetrical U-shaped encoding-decoding hierarchical structure fused state space model; and outputting a texture-restored building image. This invention significantly reduces the computational complexity of the model while ensuring restoration quality, is suitable for high-resolution, large-scale urban scenes, and can effectively improve the integrity, structural rationality, and visual realism of the texture of 3D building models.

[0025] like Figure 1 As shown, this embodiment of the invention provides a method for texture restoration of a 3D architectural model based on a dual-modal collaborative repair network, comprising the following steps: Step S100: Obtain an image of the building to be repaired; Step S200: Based on a semantically guided visual model, perform occlusion detection and segmentation on the image of the building to be repaired to obtain an occlusion area mask.

[0026] This embodiment proposes a 3D building model texture restoration method based on a bimodal collaborative restoration network. This method constructs a semantically guided occlusion detection mechanism to accurately identify occluded areas of the building. Using the original image (the image of the building to be restored) and an occluded area mask as joint input, a bimodal collaborative restoration network integrating Transformer (a deep learning model) and Mamba (a structured state-space model) mechanisms is introduced to perform context-aware and structurally constrained texture reconstruction of the occluded areas. This embodiment achieves collaborative optimization of global semantic consistency and local texture details of the building facade through multi-scale feature modeling and a reciprocal dimensional attention mechanism.

[0027] Specifically, in one implementation of this embodiment, step S200 includes the following steps: Step S201: Based on the semantically guided visual model, use preset text prompts to perform semantic-level detection and localization of buildings and occluded target objects in the image of the building to be repaired, and obtain the initial detection results of the occluded area. Step S202: Based on the image segmentation model, the initial detection results of the occluded region are segmented at the pixel level to obtain complete boundary information of the occluded region, and the occluded region mask is obtained based on the boundary information of the occluded region.

[0028] In this embodiment, the image of the building to be repaired is first acquired. The image of the building to be repaired can be a building image to be textured and repaired based on oblique photogrammetry technology, or it can be a building image to be textured and repaired based on video shooting and frame segmentation technology.

[0029] For the acquired images of the building to be repaired, a text-guided target detection method is used to perform semantic-level recognition and analysis of key targets in the images. The specific target detection method is as follows: First, a semantically guided visual model is used to detect targets in the image of the building to be repaired. Specifically, the GroundingDINO model, combined with text prompts (TextPrompt), is used to perform semantic-level detection and localization of buildings and potentially occluding objects (e.g., trees, billboards, streetlights, utility poles) in the image. By introducing text prompts with clear semantic information, the model is guided to accurately identify different categories of target objects, thereby effectively distinguishing and labeling the relationship between buildings and various occluding target objects. During the detection process, the GroundingDINO model can establish a correspondence between visual and linguistic information based on the text prompts, thus more accurately identifying the spatial occlusion relationship between buildings and target objects such as trees and billboards, providing reliable target detection results for subsequent processing.

[0030] After obtaining the approximate detection area of ​​the occluded object (i.e., the initial detection result of the occluded area), the initial detection result is further refined by combining SegmentAnythingModel (SAM, an image segmentation model). Utilizing the powerful general segmentation capabilities of the SAM model, and guided by the detection boxes or cue points output by the GroundingDINO model, pixel-level precise segmentation of the occluded area is performed, thereby obtaining more detailed and complete boundary information of the occluded area. This precise segmentation process effectively improves the accuracy of occluded object recognition and avoids the inaccuracy problem caused by relying solely on bounding boxes in traditional detection methods.

[0031] This embodiment, through the collaborative work of the GroundingDINO model and the SAM model, enables a complete occlusion recognition process from semantic-level detection to pixel-level segmentation. This ensures both the accuracy of target recognition and the refinement of occluded region extraction. Using open-source semantic algorithms, mask creation can be completed, ultimately generating high-quality occluded region mask data. This provides a reliable data foundation and technical support for subsequent tasks such as building facade reconstruction, image restoration, or occlusion removal.

[0032] like Figure 1 As shown, this embodiment of the invention provides a method for texture restoration of a 3D architectural model based on a dual-modal collaborative repair network, comprising the following steps: Step S300: Input the image of the building to be repaired and the mask of the occluded area into the dual-modal collaborative repair network for texture repair; Step S400: Output the texture-repaired building image.

[0033] In this embodiment, the image of the building to be repaired and the mask of the occluded area are input into a bimodal collaborative repair network for texture repair. The bimodal collaborative repair network is an end-to-end processing model for building occlusion texture recovery, which integrates an image repair network based on the Mamba state space model.

[0034] As an example, such as Figure 2 As shown, in this embodiment, the bimodal collaborative repair network adopts a symmetrical U-shaped encoder-decoder hierarchical structure, which incorporates the Mamba state space model; specifically, the bimodal collaborative repair network includes a context compensation hybrid module; the context compensation hybrid module consists of a deep learning model branch and a structured state space model branch set in parallel.

[0035] The proposed dual-modal collaborative inpainting network adopts a symmetrical U-shaped encoder-decoder hierarchical structure, balancing feature compression and detail restoration. The encoder consists of three feature extraction layers, corresponding to multi-scale downsampling rates of 4×, 8×, and 16×. Each layer incorporates several context-compensated hybrid modules (CCHM) to achieve multi-layer coupled learning of cross-modal semantics.

[0036] Specifically, the feature transformation process in the encoding stage can be formally represented as: ; in, This indicates a downsampling operation in spatial dimensions.

[0037] The decoding stage uses deconvolution and pixel-shuffle for upsampling reconstruction: ; The final restored image is output by convolutional tail layers and Tanh (an activation function): ; This top-down multi-scale feature fusion structure effectively maintains the continuity of information between spatial levels, ensuring the synergistic effect of deep semantic features and shallow geometric textures, thereby avoiding the detail loss problem commonly found in high-level semantic representations of traditional U-Net networks (an end-to-end image segmentation model based on convolutional neural networks).

[0038] Based on the above model structure and repair principle, this embodiment effectively balances the global structural stability of the building facade with the realism of local texture details through the dual-modal collaborative modeling strategy of the Transformer module and the Mamba module of the dual-modal collaborative repair network, thereby reducing the risk of semantic illusion.

[0039] Specifically, in one implementation of this embodiment, step S300 includes the following steps: Step S301: Input the image of the building to be repaired and the mask of the occluded area into the dual-modal collaborative repair network; Step S302: The encoder of the dual-modal collaborative repair network performs overlapping convolution to decompose the image of the building to be repaired into features of multiple scales, and performs downsampling step by step to obtain global structure and local texture fusion features.

[0040] In one implementation of this embodiment, step S302 includes the following steps: Step S302a: Perform channel stitching on the image of the building to be repaired and the mask of the occluded area to obtain stitched image features; Step S302b: Perform overlapping convolution on the spliced ​​image features to generate initial mapping features; Step S302c: The initial mapping features are downsampled step by step, and the semantic understanding and spatial details of the features are optimized by the context compensation hybrid module and the hierarchical reconstruction attention module to obtain the global structure and local texture fusion features.

[0041] In this embodiment, the process of optimizing the semantic understanding and spatial details of features through the context-compensated hybrid module and the hierarchical reconstruction attention module to obtain the global structure and local texture fusion features includes: performing global context dependency modeling based on the deep learning model branch of the context-compensated hybrid module to capture scene semantic structure information, and performing temporal and spatial joint optimization based on the structured state space model branch of the context-compensated hybrid module; and performing weighted fusion of feature maps at different scales through the hierarchical reconstruction attention module to obtain the global structure and local texture fusion features.

[0042] In this embodiment, the dual-modal collaborative repair network employs a hierarchical multi-scale feature pyramid, combining contextual information from multiple feature layers. The encoder part decomposes the input image of the building to be repaired into features at multiple scales through overlapping convolution and performs stepwise downsampling. After multiple cross-modal interactive learning processes, these features achieve efficient fusion of global and local information.

[0043] Each convolutional layer undergoes downsampling, and the semantic understanding and spatial detail fusion of features are further optimized through the CCHM module and the Hierarchical Reconstruction Attention module (HRAM). The decoder part gradually restores the texture details of the image through upsampling and pixel rearrangement strategies, from low resolution to high resolution, ensuring the continuity and naturalness of the restoration results.

[0044] The input consists of an image of the building to be repaired (i.e., the occluded image). Compared to the mask of the occluded area (i.e., the reference image). The two components are combined into a single input through a channel splicing operation: ; Initial feature maps are generated through overlapping convolutions (kernel, stride 2, padding 3): ; Subsequently, during the encoding stage, a hierarchical subsampling strategy is used to achieve multi-resolution extraction of the feature space: ; The decoding stage employs a symmetrical upsampling and pixel rearrangement mechanism: ; This strategy preserves local texture features while enhancing the expressive power of high-dimensional semantic features, thereby ensuring the structural integrity and detail continuity of the reconstructed image.

[0045] In this embodiment, the CCHM module is one of the core components of the entire bimodal collaborative repair network. It consists of parallel branches from the Transformer module and the Mamba module, constructing a cross-modal collaborative inference mechanism. Its mathematical form can be described as follows: ; in, For feature aggregation operators, This indicates a feature-level fusion operation.

[0046] The Transformer module branch uses a self-attention mechanism to model global contextual dependencies, thereby capturing scene semantic structure information; the Mamba module branch performs joint temporal and spatial optimization based on a state-space model (SSM) to improve the dynamic adaptability of feature responses. The core update equation of the Mamba module optimizer is: ; in, The SiLU activation function is used. This module is responsible for dynamic state propagation. The results from the two branches are aggregated to achieve dual compensation in terms of semantics and spatial structure, providing a high-resolution contextual representation for subsequent repair stages.

[0047] In this embodiment, to achieve cross-scale context compensation and feature reconstruction, a Hierarchical Reconstruction Attention Module (HRAM) is also set up. Its function is to perform weighted fusion of feature maps at different scales, thereby forming a hierarchical global-local feature representation. Its calculation expression is: ; in, These are learnable scale weight coefficients. For the first Upsampling operator for layer features, This is the corresponding scale feature mapping.

[0048] In this way, the HRAM module effectively integrates global structural features with local texture information, making the repaired area exhibit higher consistency and naturalness in terms of building outlines, shadow transitions, and material details.

[0049] In this embodiment, after encoding the global structure and local texture fusion features, the decoder of the dual-modal collaborative repair network is used to perform upsampling and pixel rearrangement processing to obtain a finely repaired building image.

[0050] Specifically, in one implementation of this embodiment, step S300 further includes the following steps: Step S303: Based on the global structure and local texture fusion features, the decoder of the dual-modal collaborative repair network is used to perform upsampling and pixel rearrangement processing to preserve local texture features and obtain the texture-repaired building image.

[0051] In one implementation of this embodiment, step S303 includes the following steps: Step S303a: Based on the global structure and local texture fusion features, the decoder of the dual-modal collaborative repair network is used for upsampling, and global average pooling and activation function mapping are performed in the pixel-level channels using the dimensional reciprocal attention mechanism to obtain the texture-repaired building image.

[0052] In this embodiment, image reconstruction is performed based on the global structure and local texture fusion features. During the image reconstruction process, to ensure the continuity and naturalness of the restoration result, a reciprocal dimension attention mechanism is used for global average pooling and activation function mapping in the pixel-level channels. Specifically, the reciprocal dimension attention (RDA) mechanism aims to establish bidirectional information interaction between the pixel dimension and the patch-level dimension. This reciprocal dimension attention mechanism enhances feature saliency through global average pooling and sigmoid (an activation function) activation mapping. Its calculation process is as follows: ; in, For global average pooling operators, This is a channel weighting matrix. The mutual attention mechanism in this dimension can dynamically adjust the channel strength of the feature response, enabling the bimodal collaborative repair network to form more distinctive feature representations in key areas such as building facades and window frames.

[0053] Specifically, in the dimensional reciprocal attention mechanism, cross-dimensional feature fusion is achieved through a learnable gating function: ; in, This is an adaptive balancing factor used to dynamically control the weight allocation between pixel-level and block-level features. Block-level attention is achieved through a multi-scale window mask matrix. Modeling spatial dependencies: ; Finally, the fusion result is mapped in the spatial domain through patching / unfolding operations to achieve pixel-to-block semantic alignment, and finally outputs a texture-restored building image.

[0054] This embodiment introduces a semantically guided occlusion detection mechanism to accurately distinguish between occluded objects and the real surface of the building, providing reliable constraints for subsequent texture restoration. Furthermore, by using the Transformer module and the Mamba module for bimodal collaborative modeling, it effectively balances the global structural stability of the building facade with the realism of local texture details, reducing the risk of semantic illusion. In addition, by utilizing the linear complexity characteristics of the Mamba state space model, it significantly reduces the consumption of computational resources in high-resolution urban scenes while maintaining the ability to model long-range dependencies.

[0055] In summary, this embodiment seamlessly integrates semantic-level occlusion detection based on language models with texture restoration based on deep learning to form an end-to-end solution. Furthermore, it deeply integrates the Transformer module and the Mamba module in the image restoration task, achieves complementary advantages through the CCHM module, and utilizes a two-dimensional attention mechanism of pixel channels and image block space to achieve fine-grained calibration and fusion of features.

[0056] Based on the technical solution provided in this embodiment, improvements or modifications can be made according to the above description, as follows: 1. Replacement of occlusion detection module: Other advanced open vocabulary detection or panoramic segmentation models (such as OV-DETR model, OpenSeg model, etc.) can be used to replace the GroundingSAM model to complete the initial occlusion detection and segmentation task.

[0057] 2. Repairing Network Architecture Deformations: The Transformer module branch in the bimodal collaborative repair network can be replaced with other global context modelers (such as SwingTransformer, Longformer, etc.); the Mamba branch can also adopt variants of other efficient sequence models (such as LinearTransformer model, Perceiver model, etc.). The fusion method of CCHM modules can be changed from simple addition or concatenation to learnable dynamic gating or attention-weighted fusion.

[0058] 3. Input and Output Expansion: This embodiment can be used not only for single image restoration but also extended to video sequence restoration by introducing a temporal Mamba module to handle temporal continuity. In addition to the restored RGB image, the output can simultaneously output depth information or normal maps of the restored area to assist in 3D geometric optimization.

[0059] In addition to the improvements to the above technical solutions, based on the technical solutions provided in this embodiment, the dual-modal collaborative repair network of this embodiment can be used as a preprocessing module and seamlessly integrated into existing commercial 3D modeling software (such as ContextCapture software, Metashape software) or game / simulation engines (such as Unreal Engine, Unity Engine) to enhance their automated texture processing capabilities.

[0060] Furthermore, the method provided in this embodiment can be combined with emerging 3D representation methods such as Neural Radiation Field (NeRF) to directly perform occlusion inference and texture completion in the 3D radiation field space to obtain better multi-view consistency.

[0061] This embodiment achieves the following technical effects through the above technical solution: This embodiment constructs a semantically guided occlusion detection mechanism to accurately identify building occlusion areas. Using the original image and occlusion mask as joint inputs, it introduces a bimodal collaborative repair network that integrates a deep learning model and a structured state space model. This network performs context-aware and structurally constrained texture reconstruction of the occluded areas. Finally, through multi-scale feature modeling and a reciprocal dimensional attention mechanism, it achieves collaborative optimization of global semantic consistency and local texture details of the building facade. This embodiment significantly reduces model computational complexity while ensuring repair quality, making it suitable for high-resolution, large-scale urban scenes. It can effectively improve the integrity, structural rationality, and visual realism of 3D building model textures.

[0062] Exemplary device Based on the above embodiments, the present invention also provides a texture restoration system for three-dimensional architectural models based on a dual-modal collaborative repair network, comprising: The image acquisition module is used to acquire images of the building to be repaired. The occlusion detection and segmentation module is used to perform occlusion detection and segmentation on the image of the building to be repaired based on a semantically guided visual model, and obtain an occlusion region mask. A bimodal inpainting module is used to input the image of the building to be repaired and the mask of the occluded area into a bimodal collaborative inpainting network for texture inpainting; wherein, the bimodal collaborative inpainting network is an image inpainting network that adopts a symmetrical U-shaped encoder-decoder hierarchical structure and a fused state space model; The output module is used to output the texture-repaired building image.

[0063] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 3 As shown.

[0064] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0065] When executed by a processor, this computer program is used to implement a method for texture restoration of 3D building models based on a bimodal collaborative repair network.

[0066] It will be understood by those skilled in the art that Figure 3 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0067] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a 3D building model texture restoration program based on a bimodal collaborative repair network, the 3D building model texture restoration program based on the bimodal collaborative repair network being executed by the processor to implement the above-described 3D building model texture restoration method based on the bimodal collaborative repair network.

[0068] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a 3D building model texture restoration program based on a bimodal collaborative repair network, the 3D building model texture restoration program based on a bimodal collaborative repair network being executed by a processor to implement the above-described 3D building model texture restoration method based on a bimodal collaborative repair network.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0070] In summary, this invention provides a method, system, terminal, and storage medium for texture restoration of 3D building models based on a bimodal collaborative restoration network. The method includes: acquiring an image of the building to be restored; performing occlusion detection and segmentation on the image based on a semantically guided visual model to obtain an occlusion region mask; inputting the image of the building to be restored and the occlusion region mask into a bimodal collaborative restoration network for texture restoration; wherein the bimodal collaborative restoration network is an image restoration network employing a symmetrical U-shaped encoder-decoder hierarchical structure fused state space model; and outputting a texture-restored building image. This invention significantly reduces the computational complexity of the model while ensuring restoration quality, is suitable for high-resolution, large-scale urban scenes, and can effectively improve the integrity, structural rationality, and visual realism of 3D building model textures.

[0071] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for texture restoration of a 3D architectural model based on a dual-modal collaborative repair network, characterized in that, include: Obtain images of the building to be repaired; The image of the building to be repaired is occlusion detected and segmented based on a semantically guided visual model to obtain an occlusion mask. The image of the building to be repaired and the mask of the occluded area are input into a bimodal collaborative inpainting network; wherein, the bimodal collaborative inpainting network is an image inpainting network that uses a symmetrical U-shaped encoder-decoder hierarchical structure and a fused state space model; the bimodal collaborative inpainting network includes: a context compensation hybrid module; the context compensation hybrid module consists of a deep learning model branch and a structured state space model branch set in parallel; The encoder of the dual-modal collaborative repair network performs overlapping convolutions to decompose the image of the building to be repaired into features of multiple scales, and performs stepwise downsampling to obtain global structure and local texture fusion features. Based on the global structure and local texture fusion features, the decoder of the dual-modal collaborative repair network is used for upsampling, and global average pooling and activation function mapping are performed in the pixel-level channels using the dimensional reciprocal attention mechanism to obtain the texture-repaired building image. Output the texture-restored building image.

2. The method for texture restoration of three-dimensional architectural models based on a dual-modal collaborative repair network according to claim 1, characterized in that, The semantically guided visual model performs occlusion detection and segmentation on the image of the building to be repaired, and obtains an occlusion region mask, including: Based on the semantically guided visual model, the building and the target object causing the occlusion in the image of the building to be repaired are detected and located at the semantic level using preset text prompts, so as to obtain the initial detection results of the occlusion area. The initial detection results of the occluded region are segmented at the pixel level based on the image segmentation model to obtain complete boundary information of the occluded region, and the occluded region mask is obtained based on the boundary information of the occluded region.

3. The method for texture restoration of three-dimensional architectural models based on a dual-modal collaborative repair network according to claim 1, characterized in that, The encoder of the dual-modal collaborative repair network performs overlapping convolutions to decompose the image of the building to be repaired into features at multiple scales, and performs stepwise downsampling to obtain global structure and local texture fusion features, including: Channel stitching is performed on the image of the building to be repaired and the mask of the occluded area to obtain the stitched image features; Overlapping convolutions are performed on the stitched image features to generate initial mapping features; The initial mapping features are downsampled step by step, and the semantic understanding and spatial detail fusion of the features are optimized through the context compensation hybrid module and the hierarchical reconstruction attention module to obtain the global structure and local texture fusion features.

4. The method for texture restoration of three-dimensional architectural models based on a dual-modal collaborative repair network according to claim 3, characterized in that, The semantic understanding and spatial detail fusion of features optimized by the context-compensated hybrid module and the hierarchical reconstruction attention module are used to obtain the global structure and local texture fusion features, including: Global context dependency modeling is performed based on the deep learning model branch of the context compensation hybrid module to capture scene semantic structure information, and temporal and spatial joint optimization is performed based on the structured state space model branch of the context compensation hybrid module. The hierarchical reconstruction attention module performs weighted fusion of feature maps at different scales to obtain the global structure and local texture fusion features.

5. A texture restoration system for three-dimensional architectural models based on a dual-modal collaborative repair network, characterized in that, include: The image acquisition module is used to acquire images of the building to be repaired. The occlusion detection and segmentation module is used to perform occlusion detection and segmentation on the image of the building to be repaired based on a semantically guided visual model, and obtain an occlusion region mask. A bimodal inpainting module is used to input the image of the building to be repaired and the mask of the occluded area into a bimodal collaborative inpainting network. The bimodal collaborative inpainting network is an image inpainting network employing a symmetrical U-shaped encoder-decoder hierarchical structure and a fusion state space model. The bimodal collaborative inpainting network includes a context-compensated hybrid module, which consists of parallel deep learning model branches and structured state space model branches. The encoder of the bimodal collaborative inpainting network performs overlapping convolutions to decompose the image of the building to be repaired into features at multiple scales, and performs progressive downsampling to obtain global structure and local texture fusion features. Based on the global structure and local texture fusion features, the decoder of the bimodal collaborative inpainting network performs upsampling, and global average pooling and activation function mapping are applied in the pixel-level channels using a dimensional reciprocal attention mechanism to obtain the texture-repaired building image. The output module is used to output the texture-restored building image.

6. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a 3D building model texture restoration program based on a bimodal collaborative repair network, and the 3D building model texture restoration program based on a bimodal collaborative repair network is executed by the processor to implement the operation of the 3D building model texture restoration method based on a bimodal collaborative repair network as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a three-dimensional building model texture restoration program based on a bimodal collaborative repair network. When executed by a processor, the three-dimensional building model texture restoration program based on a bimodal collaborative repair network is used to implement the operation of the three-dimensional building model texture restoration method based on a bimodal collaborative repair network as described in any one of claims 1-4.

Images