Single building image reflection artifact elimination method and system based on multi-scale space attention mechanism

Abstract

The invention discloses a single building image reflection artifact elimination method and system based on a multi-scale space attention mechanism, and the method mainly comprises the steps: constructing a single building image reflection artifact elimination network, taking a mixed image as a network input end, and carrying out the reflection artifact elimination of a single building image through the multi-characteristic module coupled single building image reflection artifact elimination network. The output end is a building entity and reflection artifact image; and designing a combined graph perception loss function, constraining the training process of the two-dimensional reflection artifact elimination network of the single building image by utilizing combined convergence of multiple graph perception losses, and guiding the decoupling of the mixed graph layer. According to the method, the reflection artifact elimination problem of the complex building image is solved, and the method is of great significance to image-based intelligent perception and urban ecological inversion.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and graphics imaging, and in particular to a method and system for eliminating reflection artifacts in a single building image based on a multi-scale spatial attention mechanism. Background Art

[0002] With the acceleration of urbanization, traditional urban planning and management methods are no longer able to meet the complex demands of modern urban development. Urban management is shifting towards refinement, digitization, and intelligence to achieve efficient utilization of urban resources and facilities, as well as safe and stable operation. Against this backdrop, the concepts of smart cities and digital twin cities have emerged. Smart cities originated in Europe in the late 1990s, initially achieving efficient urban management through automated means, such as intelligent traffic management systems. With the rapid development of information technology, the internet, and mobile communications, the concept of smart cities has continued to evolve, encompassing diverse aspects such as data sharing, energy management, and intelligent public services. Smart cities have expanded from physical to digital spaces, gradually developing into "digital cities" that contain vast amounts of data, enable data sharing, and interact with each other, and further evolving into "digital twin cities" based on digital technology and intelligent software. In the context of modern urban development, smart cities and digital twin cities not only promote intelligent, sustainable, and innovative urban development, but also improve the efficiency and quality of urban management, with significant implications for urban ecological, social, and economic development. Therefore, the development of smart cities and digital twin cities has become an essential requirement for modern urban governance and development.

[0003] Currently, reverse digital modeling of urban buildings is a key requirement for the development of smart cities and digital twin cities. Drone technology, due to its maneuverability, flexibility, timeliness, and accessibility in complex and hazardous environments, offers a new approach to capturing architectural information. Therefore, using drones to approach buildings for aerial photography is a crucial means of acquiring architectural information and enabling detailed digital modeling. However, when acquiring data from drones close to urban buildings, reflections from window panes inevitably alias the captured indoor building data with reflection artifacts from the exterior scene (such as the sky and trees), resulting in distorted or obscured architectural details, severely impacting the quality of the generated digital building models.

[0004] Cruise photography from discrete viewpoints is a common method used by drones to capture architectural images, resulting in unstructured image collections. These images lack clear spatial continuity, feature correlation is weak, and the shooting postures vary significantly. Under this shooting method, reflection artifacts exhibit several complex characteristics: their irregularity and diversity cause them to appear in unpredictable forms and distributions within the image; their positional instability and dynamic variability further complicate their removal; and their illumination dependence causes significant hue variations under varying lighting conditions. Furthermore, reflection artifacts interfere with the true features of entities, obscuring or distorting their information. Due to their intermingling with the physical features, they are difficult to effectively separate using traditional global methods. Reflection artifacts can also cause localized image contrast shifts, affecting the overall consistency of the architectural scene. Existing reflection artifact removal methods address these scenarios by setting manual constraints or integrating neural networks. However, these methods suffer from the limited applicability of the constraint functions, the lack of feature focus within the network, and their limited ability to describe the distribution of real-world features, resulting in a lack of generalization for complex architectural images.

[0005] Therefore, at this stage, there is an urgent need for a reflection artifact elimination method for complex architectural images to remove the interference of reflection artifacts in architectural images, enhance the accuracy and usability of digital body generation, and thus provide key technical guarantees for the construction of smart cities and digital twin cities. Summary of the Invention

[0006] The purpose of the present invention is to provide a method and system for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism, to solve the problems of low dissociation, poor generalization of feature description, and insufficient restoration of underlying texture in removing reflection artifacts from building images, as well as to develop an adaptive reinforcement network module for focusing on and diverting important features of complex building images, and to design multiple graphics perception loss functions to constrain the optimal solution of the network.

[0007] The technical solutions for achieving the purpose of the present invention are:

[0008] A method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism, comprising:

[0009] Collect building images and construct a dataset including mixed images, building entities and reflection artifact images;

[0010] A single building image reflection artifact removal network is constructed, with the mixed image as the network input. The single building image reflection artifact removal network coupled with multiple feature modules outputs the building entity and reflection artifact image.

[0011] Design a combined image-aware loss function and use the combined convergence of multiple image-aware losses to constrain the training process of the 2D reflection artifact removal network for a single building image and guide the decoupling of mixed layers.

[0012] A single building image reflection artifact removal network is trained using the dataset, and a combined image-aware loss function is used to optimize the training process.

[0013] Through the trained single building image reflection artifact removal network, the building entity and reflection artifact images are output in real time.

[0014] A single-frame building image reflection artifact removal system, comprising:

[0015] A data set construction unit, configured to collect building images and construct a data set including mixed images, building entities, and reflection artifact images;

[0016] A single building image reflection artifact elimination network construction unit is used to construct a single building image reflection artifact elimination network, which takes the mixed image as the network input and is coupled with a multi-feature module to form a single building image reflection artifact elimination network. The output is the building entity and the reflection artifact image.

[0017] A combined image perception loss function design unit is used to design a combined image perception loss function. It uses the combined convergence of multiple image perception losses to constrain the training process of the 2D reflection artifact removal network for a single building image and guide the decoupling of mixed layers.

[0018] The training unit trains a single building image reflection artifact removal network using a dataset and uses a combined image-aware loss function to optimize the training process;

[0019] The output unit outputs the building entity and reflection artifact images in real time through the trained single building image reflection artifact elimination network.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] (1) The present invention proposes a method for eliminating reflection artifacts from a single building image based on a multi-scale spatial attention mechanism. By analyzing the unstructured data characteristics and complex semantics of urban building images obtained by low-altitude cruising of drones, based on the concept of feature focusing and diversion, a single building image reflection artifact elimination network based on a multi-scale spatial attention mechanism and feature enhancement is created. Combined with the combined convergence of multiple graphic perception loss functions, the reflection artifact elimination of building images is performed. For the construction of high-quality platforms for smart cities and digital twin cities, a guarantee method that integrates important feature focusing, adaptive enhancement of diversion features and collaborative convergence of multiple losses is provided, breaking the limitations of existing reflection artifact elimination methods that lack feature focusing and insufficient ability to describe complex building scenes, and enriching the modeling theory of building digital twins.

[0022] (2) The present invention proposes a method for eliminating reflection artifacts from a single building image based on a multi-scale spatial attention mechanism, which breaks through the existing limitations of fitting the complex nonlinear relationship between building entities and reflection artifacts, enhances the consistent representation of virtual building ecology and real buildings, and provides a supporting tool for high-precision modeling of building digital twins.

[0023] (3) This method has strong scalability and can provide a solid technical foundation for the digital transformation of related industries in areas such as urban design, virtual reality, and augmented reality; BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0025] Figure 1 The single building image reflection artifact removal network based on multi-scale spatial attention mechanism and feature enhancement described in the present invention;

[0026] Figure 2 The multi-scale spatial attention module structure of the present invention;

[0027] Figure 3 This is the enhanced characteristic diversion module structure of the present invention;

[0028] Figure 4 This is the overall framework of the combined image perception loss described in the present invention;

[0029] Figure 5 Qualitative analysis of ablation experiments on multiple image perception loss functions described in this invention;

[0030] Figure 6 Qualitative analysis of the ablation experiment of the multi-feature network module described in the present invention;

[0031] Figure 7 This is a multi-model qualitative analysis of the single building image reflection artifact removal network described in the present invention. DETAILED DESCRIPTION

[0032] This embodiment provides a method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism. The method mainly includes two parts: constructing a reflection artifact removal network for a single building image based on a multi-scale spatial attention mechanism and feature enhancement, and designing a combined graphic perception loss function.

[0033] A single building image reflection artifact elimination network based on multi-scale spatial attention mechanism and feature enhancement is constructed. By coupling multi-feature network modules, a single building image reflection artifact elimination network with the functions of important feature focusing and diversion feature adaptive enhancement is built, and the forward flow and reverse iteration of the network are constructed based on image domain optical physics modeling; the network takes the mixed image as the network input, and the single building image reflection artifact elimination network coupled by multi-feature modules has the output of the building entity and reflection artifact image.

[0034] A combined image-aware loss function is designed, and the combined convergence of multiple image-aware losses is utilized to constrain the training process of the two-dimensional reflection artifact removal network for a single building image, guiding the decoupling of mixed layers.

[0035] The present invention is used to build a multi-feature coupled single-frame building image reflection artifact elimination network. It uses the physical light field of the two-dimensional graphic image domain as the basis and combines a combined graphic perception loss function to achieve the elimination of reflection artifacts in building images, providing important technical support for the efficient construction of smart cities, digital twin cities, etc. This method is based on the concepts and ideas of feature focusing and diversion, embeds a multi-scale spatial attention module and an enhanced feature diversion module into the reflection artifact elimination network, and adaptively selects important areas of the flow feature descriptor, thereby enhancing the network's ability to describe the nonlinear interaction between real-world building entities and reflection artifacts. In addition, a number of graphic perception loss functions are designed to guide layer dissociation and improve the network's reflection artifact elimination efficiency. The present invention solves the problem of eliminating reflection artifacts in complex building images, and is of great significance to image-based intelligent perception and urban ecological inversion.

[0036] Figure 1 It is a single building image reflection artifact removal network based on multi-scale spatial attention mechanism and feature enhancement. Figure 1 The single building image reflection artifact removal network is based on the development concept of feature focusing and diversion, and includes a feature preprocessing module, a multi-scale spatial attention module, an enhanced feature diversion module, and an image reconstruction module connected in series. Among them:

[0037] The feature preprocessing module captures the hypercolumn features of the mixed image through a pre-trained classification network, and connects them in series with the mixed image to expand the multi-dimensional semantic information.

[0038] The multi-scale spatial attention module introduces multi-scale global context information into the single image reflection artifact removal network by embedding a multi-scale spatial attention mechanism, and weights important feature blocks.

[0039] The enhanced feature diversion module improves the quality of layer separation by performing adaptive selection and attention reinforcement on favorable items of diversion features.

[0040] The image reconstruction modeling group generates building entity and reflection artifact images through the systematic combination of components such as residual structure, transposed convolution and pyramid pooling.

[0041] The image processing process of the single building image reflection artifact removal network includes: concatenating the hypercolumn features extracted by the pre-trained VGG-19 classification network with the mixed image through a feature preprocessing module, with a total of 1475 (1472+3) feature channels as input; then, the mixed features pass through a multi-scale spatial attention module composed of multi-layer convolution and multi-scale spatial attention mechanism to perform multi-scale spatial sampling on the flow features to obtain global average pooling and global maximum pooling feature descriptors of different scales; through aggregated convolution fusion, multi-scale global context information is introduced into the network to focus on the important feature distribution that affects reflection removal; and the output features are injected into the enhanced feature diversion module, which improves the accurate separation of layer features through efficient control of dual-thread diversion interaction; finally, the diversion features are transmitted to the corresponding image reconstruction group to realize image generation of building entities and reflection artifacts.

[0042] Figure 2 To construct a multi-scale spatial attention module, the data processing process includes: first, calculating .in , Represents the source feature implement Downsampling obtains the spatial global average pooling and global maximum pooling of feature bodies of each scale, thereby obtaining the global important feature distribution of multiple scales Then, aggregate coding is used to interleave the multi-scale spatial information and concatenate them into Finally, Inject a single layer of convolution to calculate the weighted distribution of spatial features ,in is the attachment matrix of the trainable weights, Represents the Sigmoid function. It is related to Multiply pixel by pixel to get ,Depend on Calibrate the map to achieve feature focus.

[0043] Figure 3 To enhance the feature splitting module architecture, features processed by the multi-scale spatial attention module are received by the feature splitting component, where they undergo multi-block interactive forward propagation. Due to the need for dual-threaded splitting and transmission of mixed features, previous research has focused on directly concatenating inactive and discarded information from the source stream to the other stream without processing or selecting it, severely impacting model robustness. Therefore, focusing on the representation of favorable terms in interactive features during feature splitting is a key approach to achieving high-quality decoupling of the entity and reflection artifact layers. The enhanced feature splitting module introduces a trainable adaptive convolution matrix in the transmission of interactive features to preserve favorable terms. An attention mechanism is also embedded to enhance selection and improve the efficacy of feature splitting. This improves the quality of layer decoupling.

[0044] Figure 4 This paper presents an overall framework for a combined image-perceptual loss. The combined convergence of multiple image-perceptual loss functions is achieved through the coordinated convergence and mutual feedback of a pixel-and-gradient dual semantic supervision loss, a multi-layer feature-perceptual loss, a generative adversarial loss, and a gradient-domain mutual exclusion loss. In the two-dimensional image domain, the essence of architectural image decomposition lies in the decoupling of the underlying geometric structure. Guided by the image-perceptual loss, the model can identify and distinguish the morphological differences between architectural entities and reflection artifacts in a manner that mimics human visual perception, thereby achieving layer dissociation. The dual semantic supervision of pixels and gradients serves as the backbone loss, applying cross-modal semantic supervision to the generated image in both the color and gradient domains to ensure the fundamental dissociation of aliased regions in the architectural image. The multi-layer feature-perceptual loss uses semantic reasoning to extract the differences between the generated image and the supervised image in multi-channel feature spaces, further exploring feature distinctions beyond color. The generative adversarial loss enhances the realism of the generated image. The gradient-domain mutual exclusion loss develops a gradient normalization method with an explosion-proof constant to effectively eliminate the effects of pixel aliasing. The Lp norm is utilized to ensure the preservation of texture details during the layer separation process. To enhance the constraint effectiveness of the graph-perceptual loss, this paper employs a combined convergence approach using multiple losses. Based on the overall framework of the combined graph-perceptual loss, the weights of the aforementioned losses are set to 0.5, 0.1, 0.001, and 1, respectively, according to the distribution of generated data obtained during training and the results of multiple sets of visual verification.

[0045] The pixel and gradient dual semantic supervision loss function applies semantic supervision to the generated image in the color domain and gradient domain to ensure the basic dissociation of the aliased areas of the building image, including:

[0046]

[0047] in 、 represent the gradient operators along the X and Y directions respectively, 、 、 、 are weighting coefficients, is the training feature of the network, , The generated building entity and reflection artifact images are respectively , is the supervision image of the building entity and the reflection artifact image, Dual semantic supervision loss functions for the building entity layer and the reflection artifact layer respectively.

[0048] The multi-layer feature perception loss function uses semantic reasoning to obtain the differences between the generated image and the supervised image in the multi-channel feature space, and further explores the differences in features other than color information, including:

[0049]

[0050] in is the number of feature layers, 、 are the characteristic distributions of building entities and reflection artifact images T and R, respectively. 、 are the feature distributions of the supervision images of the building entity and the reflection artifact images T and R respectively; 、 are the generated building entities and reflection artifact images, is the training feature of the network, is the mixed image at the input, A collection of images.

[0051] The generative adversarial perceptual loss function enhances the realism of the generated images through the mutual constraints between the generator and the discriminator, including:

[0052]

[0053] in For mixed images, represents the discriminator, , represent the discriminator loss and generator loss respectively.

[0054] The gradient domain mutual exclusion loss function creates a gradient normalization method with an explosion-proof constant to eliminate pixel aliasing in the layer dissociation stage and retain basic details using the Lp norm, including:

[0055]

[0056]

[0057]

[0058] in , represents the normalization factor, Refers to the number of multi-scale sampling layers, is the Lp norm, is element-wise multiplication, represents the average, It is the explosion-proof constant.

[0059] Figure 5 This paper provides a qualitative analysis of ablation experiments on multiple image-perceptual loss functions. This experiment demonstrates the effectiveness of visualizing a single perceptual loss, where (a) it serves as the input, (b) the pixel gradient dual semantic supervision loss maintains the underlying representation of pixel intensity and region contours, (c) the multi-layer feature-perceptual loss enables high-level semantic reasoning, (d) the generative adversarial perceptual loss ensures the realism of the reconstructed image, and (e) the gradient domain mutual exclusion loss enhances the separation between layers. Finally, the synergistic effect (f) demonstrates that a network model based on the shared constraints of multiple perceptual losses can produce accurate and clear predictions.

[0060] Figure 6 Qualitative analysis of ablation experiments on multi-feature network modules, which demonstrates the visualization efficacy of a single network module, including (a) a fully convolutional baseline network, (b) a coupled multi-scale spatial attention module network, (c) a coupled enhanced feature split module network, and (d) a coupled full module network. Figure 4 As shown in (d), the network of the present invention has the best reflection artifact elimination effect.

[0061] Figure 7 This is a qualitative analysis of a multi-model network for single-building image reflection artifact removal. This experiment demonstrates the advantages of our proposed method over existing methods for removing reflection artifacts, including Zhang et al., YTMT, ERRNet, IBCLN, Kim, DSRNet, LANet, Zhu et al., and Zheng et al., which are representative multi-class reflection artifact removal methods.

[0062] The present invention provides a method for eliminating reflection artifacts from single building images based on a multi-scale spatial attention mechanism. This method is primarily used to eliminate the interference caused by reflection artifacts when drones capture urban building information at low altitude and perform inverse inversion. This method ensures the accuracy and consistency of digital body generation for urban buildings. This is of great significance for achieving high-quality platforms for smart cities and digital twin cities, enhancing the reusability and platform independence of digital bodies, and improving the modeling accuracy and development efficiency of urban digital twins. The research presented in this invention has important theoretical significance and practical application value.

[0063] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism, characterized in that: include: Collect building images and construct a dataset including mixed images, building entities and reflection artifact images; A single building image reflection artifact removal network is constructed, with the mixed image as the network input. The single building image reflection artifact removal network coupled with multiple feature modules outputs the building entity and reflection artifact image. Design a combined image-aware loss function and use the combined convergence of multiple image-aware losses to constrain the training process of the 2D reflection artifact removal network for a single building image and guide the decoupling of mixed layers. A single building image reflection artifact removal network is trained using the dataset, and a combined image-aware loss function is used to optimize the training process. Through the trained single building image reflection artifact removal network, the building entity and reflection artifact images are output in real time.

2. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 1, characterized in that: The single building image reflection artifact removal network includes a feature preprocessing module, a multi-scale spatial attention module, an enhanced feature diversion module and an image reconstruction module connected in series; wherein: The feature preprocessing module captures the hypercolumn features of the mixed image through a pre-trained classification network and connects them in series with the mixed image to expand the multi-dimensional semantic information; The multi-scale spatial attention module introduces multi-scale global context information into the single image reflection artifact removal network by embedding a multi-scale spatial attention mechanism, and weights important feature blocks; The enhanced feature diversion module improves the quality of layer separation by performing adaptive selection and attention reinforcement on favorable items of diversion features; The image reconstruction modeling group generates building entity and reflection artifact images through a systematic combination of residual structure, transposed convolution and pyramid pooling.

3. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 2, characterized in that: The process of image processing by the multi-scale spatial attention module specifically includes: First, calculate the global important feature scores at multiple scales ,in 、 Represents the source feature implement Downsampling obtains spatial global average pooling and global maximum pooling of feature bodies of each scale; Then, aggregate coding is used to interleave the multi-scale spatial information and concatenate them into ; Finally, Inject a single layer of convolution to calculate the weighted distribution of spatial features ,in is the attachment matrix of the trainable weights, Represents the Sigmoid function, weighted distribution and source feature body Multiply pixel by pixel to get ,Depend on Calibrate the map to achieve feature focus.

4. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 1, characterized in that: The combined graphics perception loss function includes a pixel and gradient dual semantic supervision loss function, a multi-layer feature perception loss function, a generative adversarial perception loss function and a gradient domain mutual exclusion loss function.

5. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 4, characterized in that: The pixel and gradient dual semantic supervision loss function includes: ； in 、 represent the gradient operators along the X and Y directions respectively, 、 、 、 are weighting coefficients, is the training feature of the network, , The generated building entity and reflection artifact images are respectively , is the supervision image of the building entity and the reflection artifact image, Dual semantic supervision loss functions for the building entity layer and the reflection artifact layer respectively.

6. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 4, characterized in that: The multi-layer feature perception loss function includes: ； in is the number of feature layers, 、 are the characteristic distributions of building entities and reflection artifact images T and R, respectively. 、 are the feature distributions of the supervision images of the building entity and the reflection artifact images T and R respectively; 、 are the generated building entities and reflection artifact images, is the training feature of the network, is the mixed image at the input, A collection of images.

7. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 6, characterized in that: The generated adversarial perception loss function includes: ； in For mixed images, represents the discriminator, , represent the discriminator loss and generator loss respectively.

8. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 4, characterized in that: The gradient domain mutually exclusive loss function includes: ； ； ； in 、 represents the normalization factor, Refers to the number of multi-scale sampling layers, is the Lp norm, is element-wise multiplication, Represents the average, For explosion-proof constants, Represents the gradient normalized dot product of the building entity and the image generated by the reflection artifact.

9. The method for removing reflection artifacts from a single building image based on a multi-scale spatial attention mechanism according to claim 4, characterized in that: The weights of the pixel and gradient dual semantic supervision loss function, the multi-layer feature perception loss function, the generative adversarial perception loss function, and the gradient domain mutual exclusion loss function are 0.5, 0.1, 0.001, and 1, respectively.

10. A system for eliminating reflection artifacts from a single building image, implementing the method according to any one of claims 1 to 9, characterized in that: include: A data set construction unit, configured to collect building images and construct a data set including mixed images, building entities, and reflection artifact images; A single building image reflection artifact elimination network construction unit is used to construct a single building image reflection artifact elimination network, which takes the mixed image as the network input and is coupled with a multi-feature module to form a single building image reflection artifact elimination network. The output is the building entity and the reflection artifact image. A combined image perception loss function design unit is used to design a combined image perception loss function. It uses the combined convergence of multiple image perception losses to constrain the training process of the 2D reflection artifact removal network for a single building image and guide the decoupling of mixed layers. The training unit trains a single building image reflection artifact removal network using a dataset and uses a combined image-aware loss function to optimize the training process; The output unit outputs the building entity and reflection artifact images in real time through the trained single building image reflection artifact elimination network.

Images