Building outer wall image processing method, electronic device and storage medium

Abstract

The application relates to the technical field of image processing, and discloses a building outer wall image processing method, an electronic device and a storage medium. The method comprises the following steps: acquiring a plurality of building outer wall images and corresponding EXIF metadata; selecting a plurality of matching image pairs from the building outer wall images according to the EXIF metadata; determining matching point pairs between the images in the matching image pairs according to the spatial domain features and the frequency domain features of the building outer wall images; performing fine processing on the camera poses corresponding to the building outer wall images according to the matching point pairs and the EXIF metadata, to obtain fine camera poses and a sparse three-dimensional point cloud; performing multi-view stereo matching processing on the building outer wall images according to the fine camera poses and the sparse three-dimensional point cloud, to generate a dense point cloud; generating a three-dimensional mesh model of the building outer wall according to the dense point cloud, and generating structured measurement data of the building outer wall. The embodiment of the application can improve the matching precision of the building outer wall images and the reliability of the structured measurement data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for processing images of building exterior walls, electronic devices, and storage media. Background Technology

[0002] In the field of building facade image processing, existing image matching methods, especially deep learning-based matching algorithms such as Efficient LoFTR, face significant challenges when processing building facade images. Building facades commonly exhibit repetitive texture structures, such as tile cladding, glass curtain wall grilles, and brickwork. These structures present highly similar local regions in the spatial domain, making them difficult for traditional matching methods to effectively distinguish. Existing technologies typically employ multi-layer depthwise separable convolutional stacks as the backbone network. Each layer has an extremely limited receptive field, covering only a 3×3 or 5×5 pixel area. This results in the global receptive field growing linearly with network depth, failing to fully capture the global periodic frequency distribution characteristics of the repetitive textures on building facades. This periodic texture manifests as significant spectral peaks at specific spatial frequencies in the frequency domain. However, existing backbone networks completely lack the ability to perceive and process frequency domain information. Therefore, feature descriptors cannot utilize the global frequency distribution to identify and distinguish highly similar local regions. As a result, the point pair extraction process frequently fails in repetitive texture scenes, and it is difficult to obtain effective matching points in uniform regions. This leads to inaccurate camera pose estimation, low quality of 3D point cloud reconstruction, and ultimately affects the generation accuracy and reliability of structured measurement data for building exterior walls. Summary of the Invention

[0003] The purpose of this application is to provide a method for processing building exterior wall images, an electronic device, and a storage medium that can improve the matching accuracy of building exterior wall images, the quality of three-dimensional reconstruction, and the reliability of structured measurement data.

[0004] This application provides a method for processing building exterior wall images, including: Acquire multiple images of building exterior walls and their corresponding EXIF ​​metadata; Based on the EXIF ​​metadata, select several matching image pairs from the building exterior wall images; Based on the spatial domain and frequency domain features of the building exterior wall image, the matching point pairs between the images in the matching image pair are determined; Based on the matching point pairs and the EXIF ​​metadata, the camera pose corresponding to the building exterior wall image is refined to obtain the corresponding refined camera pose and sparse 3D point cloud. Based on the refined camera pose and the sparse 3D point cloud, multi-view stereo matching processing is performed on the building exterior wall image to generate a dense point cloud. A three-dimensional mesh model of the building's exterior wall is generated based on the dense point cloud, and structured measurement data of the building's exterior wall is generated based on the three-dimensional mesh model.

[0005] In some embodiments, selecting a plurality of matching image pairs from the building exterior image based on the EXIF ​​metadata includes: Based on the location information, pose information, and imaging parameter information in the EXIF ​​metadata, the coverage area of ​​the building exterior wall image on the building exterior wall is determined; The matching image pair is selected from the building exterior wall images based on the degree of overlap between the covered areas of any two building exterior wall images.

[0006] In some embodiments, determining the matching point pairs between images in the matching image pair based on the spatial domain features and frequency domain features of the building exterior wall image includes: Multi-scale spatial domain feature extraction and frequency domain feature extraction are performed on the images in the matched image pair to obtain coarse-grained feature maps and fine-grained feature maps; Using the frequency domain features in the coarse-grained feature map as the query basis and the spatial domain features in the coarse-grained feature map as the matching basis, region matching is performed on the images in the matching image pair to obtain matching region pairs between the images in the matching image pair. Based on the matching region pair, sub-pixel precision matching is performed on the fine-grained feature map to obtain the matching point pair.

[0007] In some embodiments, the multi-scale spatial domain feature extraction and frequency domain feature extraction of the images in the matched image pair includes: Extract spatial domain features of the images in the matched image pairs at multiple scales; Fourier transforms are performed on the images in the matched image pair at multiple scales, and the transform results are processed by learnable channel mixing in the frequency domain. Then, the images are restored to the spatial domain by inverse Fourier transform to obtain the frequency domain features of the images in the matched image pair. According to the preset fusion weights, the spatial domain features and frequency domain features of the images in the matched image pair at the same scale are weighted and fused to obtain the coarse-grained feature map and the fine-grained feature map.

[0008] In some embodiments, the step of using the frequency domain features in the coarse-grained feature map as the query basis and the spatial domain features in the coarse-grained feature map as the matching basis to perform region matching on the images in the matched image pair includes: The frequency domain features in the coarse-grained feature map are subjected to frequency domain enhancement processing, and a query vector and a key vector are generated based on the frequency domain features after frequency domain enhancement. A value vector is generated based on the spatial domain features in the coarse-grained feature map. Attention is calculated based on the query vector, the key vector, and the value vector to obtain a similarity matrix between images in the matched image pair. Based on the similarity matrix, several local region pairs with the highest confidence are selected from the images in the matched image pairs as the matched region pairs.

[0009] In some embodiments, before using the frequency domain features in the coarse-grained feature map as the query basis, the method further includes: Based on the spectral distribution characteristics of the frequency domain features in the coarse-grained feature map, a content-adaptive positional bias feature is generated and superimposed on the frequency domain features in the coarse-grained feature map.

[0010] In some embodiments, refining the camera pose corresponding to the building exterior image based on the matching point pairs and the EXIF ​​metadata includes: Based on the position and pose information in the EXIF ​​metadata, the camera pose corresponding to each of the building exterior wall images is initialized to obtain the initial camera pose. The building exterior wall images are sorted according to the inlier rate of the matched point pairs, and the building exterior wall images are sequentially registered to the 3D point cloud according to the sorting result and the initial camera pose. During the registration process, geometric consistency screening is performed on the matching point pairs participating in the registration, and periodic global optimization is performed on the registered building exterior wall images until all building exterior wall images are registered, thereby obtaining the refined camera pose and the sparse 3D point cloud.

[0011] In some embodiments, the step of performing multi-view stereo matching processing on the building exterior wall image based on the refined camera pose and the sparse 3D point cloud includes: The depth search range is determined based on the refined camera pose and the sparse 3D point cloud; Based on the depth search range and the refined camera pose, the building exterior wall image is subjected to multi-view stereo matching processing to generate the dense point cloud.

[0012] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described building exterior wall image processing method.

[0013] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described building exterior wall image processing method.

[0014] The beneficial effects of this application are as follows: By simultaneously utilizing the spatial and frequency domain features of building exterior wall images to determine matching point pairs, and by fusing local detail information in the spatial domain with global periodicity information in the frequency domain, the matching accuracy and inlier rate in repetitive texture scenes can be significantly improved. During camera pose refinement and dense point cloud generation, the initial information provided by EXIF ​​metadata and the refined sparse point cloud are fully utilized, avoiding blind searches from scratch. Iterative optimization is performed through matching point pairs, combined with geometric consistency filtering and periodic global optimization, ensuring the accuracy of the camera pose. Based on the refined camera pose and sparse point cloud, the depth search range can be determined more accurately in the multi-view stereo matching stage, improving the efficiency and accuracy of dense point cloud generation, and enhancing the matching accuracy of building exterior wall images, the quality of 3D reconstruction, and the reliability of structured measurement data. Attached Figure Description

[0015] Figure 1 This is a flowchart of the building exterior wall image processing method provided in the embodiments of this application.

[0016] Figure 2 This is a flowchart of a method for selecting several matching image pairs from building exterior wall images provided in an embodiment of this application.

[0017] Figure 3 This is a flowchart of a method for determining matching point pairs between images in a matching image pair, provided in an embodiment of this application.

[0018] Figure 4 This is a flowchart of a method for refining the camera pose corresponding to an image of a building exterior wall, provided in an embodiment of this application.

[0019] Figure 5 This is a flowchart of a method for performing multi-view stereo matching processing on building exterior wall images provided in an embodiment of this application.

[0020] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application. Furthermore, the information, data, and signals involved in the embodiments of this application are all authorized by relevant parties or have been fully authorized by all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0024] In existing methods for matching building facade images, the receptive field of the backbone network has limitations. Current methods employ multiple layers of depthwise separable convolutional stacks as the backbone network, with each layer having a receptive field of only 3×3 or 5×5. The global receptive field grows linearly with depth, failing to perceive the global periodic frequency distribution of repetitive textures on building facades. These repetitive textures in the frequency domain appear as spectral peaks concentrated at specific spatial frequencies. However, existing backbone networks lack frequency domain awareness, causing feature descriptors to be unable to distinguish highly similar local regions using the global periodic structure. This results in a reduced match point ratio in repetitive texture scenes, and the inability to extract effective match points in uniform regions. For example, in the detection of glass curtain walls in high-rise office buildings, the curtain wall forms a repetitive texture pattern with regularly arranged glass grilles. Existing methods, unable to capture periodic spectral peaks in the frequency domain, result in sparse and unstable distribution of match point pairs at grille boundaries. Large areas of uniform glass lack effective match point generation, affecting the accurate restoration of the curtain wall's geometry in subsequent 3D reconstruction.

[0025] If the above problems are not solved, the building exterior wall image matching process will produce incomplete matching point pairs, resulting in sparse 3D point clouds lacking key structural information, dense point cloud generation failing, 3D mesh models becoming distorted, and structured measurement data failing to accurately reflect the actual geometric features of the building exterior walls, thus hindering the implementation of automated exterior wall inspection processes.

[0026] Based on this, embodiments of this application provide a method for processing building exterior wall images, an electronic device, and a storage medium. By combining spatial and frequency domain features for image matching and camera pose refinement, the accuracy of building exterior wall image matching, the quality of 3D reconstruction, and the reliability of structured measurement data can be improved.

[0027] See Figure 1In one embodiment, a method for processing images of building exterior walls is provided, including but not limited to steps S101 to S106.

[0028] Step S101: Obtain multiple images of the building's exterior walls and their corresponding EXIF ​​metadata.

[0029] Building facade images are digital images taken with cameras or other imaging devices to record the details of a building's external structure and surface. These images typically contain information such as the building's facade, windows, doors, decorations, and material textures, serving as the data foundation for building structural analysis and measurement.

[0030] EXIF metadata is additional information embedded in image files (such as JPEG format) that records various parameters at the time the image was captured. These parameters typically include camera model, shooting time, exposure settings, focal length, GPS location information, and camera attitude information (such as pitch, yaw, and roll angles). EXIF ​​metadata provides important contextual information for the image, which is helpful for subsequent image processing and 3D reconstruction.

[0031] Building facade images can be acquired in various ways. For example, a handheld camera or drone can be used to photograph the building facade from different angles and heights, generating a series of images with overlapping areas. EXIF ​​metadata is typically recorded automatically by the camera and embedded into the image file during image capture. In some cases, if the EXIF ​​metadata is incomplete or missing, it can also be manually entered or recorded using external sensors (such as GPS modules or IMU inertial measurement units) and associated with the building facade images.

[0032] Step S102: Select several matching image pairs from the building exterior wall images based on EXIF ​​metadata.

[0033] Matched image pairs refer to two images selected from multiple images of building facades that have sufficient overlap and similar features. These image pairs are the basis for feature matching and 3D reconstruction, ensuring that the correspondence between points in the same scene from different viewpoints can be established.

[0034] Several matching image pairs can be selected from building exterior wall images. Initial screening can be performed based on shooting time, approximate location information, etc., recorded in EXIF ​​metadata. Images with similar shooting time or location can be used as potential matching image pairs. Then, several matching image pairs can be selected based on the overlapping areas between building exterior wall images.

[0035] Step S103: Based on the spatial domain features and frequency domain features of the building exterior wall image, determine the matching point pairs between the images in the matching image pair.

[0036] Spatial domain features refer to the local or global characteristics of an image in pixel space, such as corners, edges, textures, and color distribution. These features are typically extracted using various image processing algorithms (such as SIFT, SURF, and ORB) to describe the local structural information of the image content.

[0037] Frequency domain features refer to the characteristics of an image in the frequency space after Fourier transform. They can reveal periodic structures, directional textures, and the overall smoothness or detail richness of an image. For example, repeating textures appear as specific spectral peaks in the frequency domain, which can be used to identify regions with periodic structures.

[0038] Based on the spatial and frequency domain features of building exterior wall images, matching point pairs between images in a matching image pair can be determined. This can be achieved by extracting spatial and frequency domain features of the images in parallel, such as using SIFT or SURF algorithms to detect and describe key points in the images, analyzing the image spectrum through Fourier transform to identify regions with similar frequency distributions, and then matching them based on the similarity between feature descriptors and the extracted frequency domain features to obtain matching point pairs.

[0039] Step S104: Based on the matching point pairs and EXIF ​​metadata, refine the camera pose corresponding to the building exterior wall image to obtain the corresponding refined camera pose and sparse 3D point cloud.

[0040] Refining the camera pose corresponding to the building exterior image can be achieved by using the GPS position and attitude information provided in the EXIF ​​metadata as an initial estimate, and then adjusting these initial poses through iterative optimization algorithms (such as Bundle Adjustment). During the optimization process, the geometric constraints provided by the matching point pairs can be used to continuously reduce the reprojection error, thereby obtaining a more accurate camera pose and the corresponding sparse 3D point cloud.

[0041] Step S105: Based on the refined camera pose and sparse 3D point cloud, perform multi-view stereo matching processing on the building exterior wall image to generate a dense point cloud.

[0042] Multi-view stereo matching processing of building exterior images can be performed by defining a depth search range for each image based on the refined camera pose and sparse point cloud. Within this depth search range, the depth value of each pixel is estimated by comparing the similarity of corresponding pixels in different images, for example, using photometric consistency or geometric consistency criteria, thereby generating dense point cloud data.

[0043] Step S106: Generate a three-dimensional mesh model of the building exterior wall based on the dense point cloud, and generate structured measurement data of the building exterior wall based on the three-dimensional mesh model.

[0044] Structured measurement data includes the actual area of ​​building exterior walls, statistically analyzed according to at least one of the following partitioning methods: facade orientation, floor level, and material type. In essence, structured measurement data refers to measurement information extracted from a 3D mesh model, exhibiting a specific organization and classification. This data can include the actual area, volume, and dimensions of building exterior walls, and can be partitioned and statistically analyzed according to facade orientation, floor level, material type, etc., providing quantitative data for building maintenance, renovation, and evaluation.

[0045] Generating a 3D mesh model of the building's exterior walls from a dense point cloud can be achieved using algorithms such as Poisson reconstruction, Delaunay triangulation, or Marching Cubes, converting the dense point cloud into a continuous 3D mesh model. Based on this, structured measurement data for the building's exterior walls is generated from the 3D mesh model. This can be done manually or semi-automatically by dividing the 3D mesh model into regions, for example, according to the building's structural lines or user-specified zoning rules, dividing the exterior walls into different facade, floor, or material regions, and then calculating the actual area of ​​each region.

[0046] Based on the above examples, the technical concept of this embodiment demonstrates a significant technical contribution in solving the problems of existing technologies. Existing building exterior wall image matching methods, especially those based on Efficient LoFTR, suffer from low inlier rates and the inability to extract effective matching points when processing building exterior wall images with repetitive textures (such as tile cladding) and uniform regions (such as glass curtain walls). This is mainly due to the limited receptive field of the backbone network in existing methods, which lacks the ability to perceive global periodic frequency distributions, making it difficult for feature descriptors to distinguish highly similar local regions. This embodiment effectively overcomes the limitations of existing technologies by simultaneously utilizing the spatial domain features and frequency domain features of building exterior wall images to determine matching point pairs. By fusing local detail information in the spatial domain with global periodicity information in the frequency domain, this embodiment can significantly improve the matching accuracy and inlier rate in repetitive texture scenes, extracting more effective and accurate matching points compared to existing methods that rely solely on spatial domain features. Furthermore, this embodiment fully utilizes the initial information provided by EXIF ​​metadata and the refined sparse point cloud during camera pose refinement and dense point cloud generation, avoiding blind searches from scratch. Iterative optimization through matching point pairs, combined with geometric consistency filtering and periodic global optimization, ensures the accuracy of camera pose. Based on these refined poses and sparse point clouds, the depth search range can be determined more accurately during the multi-view stereo matching stage, improving the efficiency and accuracy of dense point cloud generation. The resulting structured measurement data can calculate the actual area of ​​building exterior walls according to various partitioning methods such as facade orientation, floor, and material type, providing quantitative and actionable data support for the refined management and maintenance of buildings—something that existing technologies cannot directly provide.

[0047] See Figure 2 In one embodiment, the method for selecting several matching image pairs from building exterior wall images includes, but is not limited to, steps S201 to S202.

[0048] Step S201: Determine the coverage area of ​​the building exterior wall image on the building exterior wall based on the location information, pose information and imaging parameter information in the EXIF ​​metadata.

[0049] Step S202: Based on the degree of overlap between the coverage areas of any two building exterior wall images, select matching image pairs from the building exterior wall images.

[0050] Based on the location, attitude, and imaging parameters in the EXIF ​​metadata, the coverage area of ​​the building facade image can be determined. This can be achieved by constructing a camera projection model and projecting the image's field of view onto the geometric model of the building facade, thus obtaining its two-dimensional projection area. For example, GPS data and camera attitude angles from the EXIF ​​metadata can be used in conjunction with a rough 3D model of the building facade to calculate the precise coverage area of ​​the image on the building facade through ray tracing or geometric projection. Another approach is to project the four corner points of the image or preset grid points onto the building facade plane in the world coordinate system using camera intrinsic and extrinsic parameter matrices, and then calculate the area and shape of the region enclosed by these projection points to determine the coverage area.

[0051] To select matching image pairs from any two building facade images, the degree of overlap between their covered areas can be calculated. This can be done by dividing the area of ​​the intersection of the covered areas of the two building facade images by the area of ​​each image. If this ratio is higher than a certain threshold (e.g., 25%), the pair of images is considered a matching image pair. Alternatively, the Jaccard similarity coefficient (intersection area divided by union area) can be used to measure the degree of overlap, and the selection can be based on this coefficient.

[0052] This application's solution utilizes the rich information contained in EXIF ​​metadata to first accurately determine the actual coverage area of ​​each building exterior wall image in 3D space. Specifically, using positional, pose, and imaging parameter information, a camera model for each image can be constructed, and its view frustum can be projected onto the building exterior wall surface, thus obtaining its 2D coverage area on the building exterior wall. Based on this, the overlap between the coverage areas of any two building exterior wall images is systematically calculated. By setting a reasonable overlap threshold, image pairs with sufficient spatial correlation and suitable for subsequent feature matching can be effectively selected, thus avoiding unnecessary calculations for image pairs with insufficient or no overlap, significantly reducing the computational burden of subsequent matching point pair determination, and improving the accuracy and efficiency of matching. This selection mechanism based on spatial coverage area overlap ensures the effectiveness and high quality of the selected matching image pairs, laying a solid foundation for subsequent camera pose refinement and 3D reconstruction.

[0053] See Figure 3 In one embodiment, the method for determining matching point pairs between images in a matching image pair includes, but is not limited to, steps S301 to S303.

[0054] Step S301: Perform multi-scale spatial domain feature extraction and frequency domain feature extraction on the images in the matched image pair to obtain coarse-grained feature maps and fine-grained feature maps.

[0055] Step S302: Using the frequency domain features in the coarse-grained feature map as the query basis and the spatial domain features in the coarse-grained feature map as the matching basis, perform region matching on the images in the matching image pair to obtain the matching region pairs between the images in the matching image pair.

[0056] Step S303: Based on the matching region pairs, perform sub-pixel precision matching on the fine-grained feature map to obtain matching point pairs.

[0057] Multi-scale spatial domain feature extraction can be performed on images in a matching image pair using algorithms such as Scale Invariant Feature Transform (SIFT) and Speed-Up Robust Feature Transform (SURF), or deep learning models (such as convolutional neural networks) can be used to output feature maps of different scales at different levels.

[0058] Multi-scale frequency domain feature extraction can be performed on images in a matched image pair by performing multi-scale Fourier transforms, wavelet transforms, or Gabor filtering to obtain frequency domain information. Within a deep learning framework, dedicated frequency domain convolutional layers or attention mechanisms can also be designed to extract frequency domain features.

[0059] Coarse-grained feature maps contain macroscopic structure and low-frequency information of an image, have a large receptive field and a small resolution, and are used to capture the overall layout and salient regions of an image.

[0060] Fine-grained feature maps contain detailed textures and high-frequency information of an image, have a smaller receptive field and higher resolution, and are used to accurately describe local features.

[0061] This application's solution effectively addresses the challenge of accurately and robustly determining matching point pairs in complex building facade images by introducing multi-scale feature extraction and a coarse-to-fine-grained matching strategy. Specifically, it first performs multi-scale spatial and frequency domain feature extraction on the images in the matching image pair. This simultaneously captures local geometric details and global texture patterns at different scales, generating coarse-grained and fine-grained feature maps. The coarse-grained feature map is used for preliminary region matching. Frequency domain features serve as the query criterion, quickly identifying candidate regions with similar textures or structural patterns, while spatial domain features, as the matching criterion, further refine the similarity judgment of regions, thus obtaining matching region pairs between the images in the matching image pair. This coarse-grained region matching strategy effectively reduces the search space for subsequent fine-grained matching and improves the robustness of the matching. Based on this, sub-pixel precision matching is performed on the fine-grained feature map according to the determined matching region pairs. Fine-grained feature maps provide rich image details, and combined with sub-pixel matching technology, the coordinate precision of matching points can be improved to below the pixel level, thereby obtaining high-precision matching point pairs. Through this layered and progressive matching process, the proposed solution not only improves matching efficiency but, more importantly, significantly enhances matching accuracy and robustness, laying a solid foundation for subsequent camera pose refinement and sparse 3D point cloud generation.

[0062] The following is a concrete example. When performing multi-scale spatial and frequency domain feature extraction on images in a matching image pair, a deep convolutional neural network (CNN) can be used as the feature extractor. The shallow layer output of this CNN can serve as a fine-grained feature map, capturing the edges and texture details of the image; while the deep layer output serves as a coarse-grained feature map, capturing the semantic information and macroscopic structure of the image. Simultaneously, a Fourier transform module can be introduced after each layer of the CNN to perform frequency domain analysis on the feature map and extract its frequency domain features. For region matching, an attention-based matching network can be constructed. This network encodes the frequency domain features in the coarse-grained feature map into query and key vectors, and the spatial domain features into value vectors. By calculating the attention weights between the query vector, key vector, and value vector, a similarity matrix can be obtained, indicating the matching probability between different regions in the image pair. Then, the region pairs with the highest confidence are selected from the similarity matrix as the matching region pairs. Finally, based on these matching region pairs, a dedicated subpixel regression network can be used on the fine-grained feature map to perform refined matching of feature points within the matching regions, outputting matching point pairs with subpixel accuracy.

[0063] In some embodiments, multi-scale spatial domain feature extraction and frequency domain feature extraction are performed on the images in the matched image pair, including: extracting spatial domain features of the images in the matched image pair at multiple scales; performing Fourier transform on the images in the matched image pair at multiple scales, and performing learnable channel mixing processing on the transform result in the frequency domain, and then restoring it to the spatial domain through inverse Fourier transform to obtain the frequency domain features of the images in the matched image pair; and performing weighted fusion of the spatial domain features and frequency domain features of the images in the matched image pair at the same scale according to preset fusion weights to obtain coarse-grained feature maps and fine-grained feature maps.

[0064] Extracting spatial domain features from matched image pairs at multiple scales can be achieved in various ways. For example, classic image processing algorithms such as Scale Invariant Feature Transform (SIFT) or Speed-Up Robust Feature Transform (SURF) can be used, as these algorithms inherently support the detection and description of multi-scale features. Alternatively, deep learning methods can be employed, utilizing different layers of convolutional neural networks (CNNs) or feature pyramid networks (FPNs) to acquire spatial features of images at different levels of abstraction and resolutions.

[0065] Performing a Fourier transform typically employs the Fast Fourier Transform (FFT) algorithm, converting the pixel value distribution of an image into a frequency spectrum. Learnable channel blending in the frequency domain can be achieved through a lightweight neural network module, such as a module consisting of 1x1 convolutional layers, fully connected layers, or attention mechanisms. Its role is to adaptively adjust and combine information from different frequency channels according to the requirements of the matching task, highlighting frequency components beneficial to matching while suppressing noise or irrelevant information. Subsequently, the processed frequency domain data is transformed back into the spatial domain using an Inverse Fourier Transform (IFFT), resulting in a spatial domain representation that incorporates the characteristics of the frequency domain.

[0066] According to preset fusion weights, the spatial domain features and frequency domain features of images in matching image pairs at the same scale are weighted and fused. A linear weighted summation method can be used, adding the spatial and frequency domain features according to their weights. Coarse-grained feature maps typically refer to feature representations with lower resolution, larger receptive fields, and more global information, while fine-grained feature maps refer to feature representations with higher resolution, smaller receptive fields, and more local detail information. By fusing at different scales, coarse-grained or fine-grained feature maps at the corresponding scales can be obtained respectively.

[0067] This application's solution extracts spatial and frequency domain features from building facade images in parallel at multiple scales and then performs weighted fusion to generate coarse-grained and fine-grained feature maps for subsequent matching. Specifically, firstly, spatial domain features are extracted from the image at different scales to capture local details and structural information at different resolutions. Simultaneously, at the same scale, a Fourier transform is performed on the image to enter the frequency domain, where learnable channel mixing is performed. This process adaptively enhances frequency components beneficial to the matching task, thereby improving the discriminative power of the frequency domain features. Subsequently, the processed frequency domain features are restored to the spatial domain through an inverse Fourier transform. Finally, the spatial domain features extracted at the same scale and the processed frequency domain features are weighted and fused according to preset fusion weights. This fusion mechanism enables the generated feature map to possess both the local accuracy of spatial domain features and the global robustness of frequency domain features. Especially when facing complex situations such as lighting and texture changes, the learnable frequency domain channel mixing can more effectively extract stable and discriminative features. In this way, this application can generate high-quality coarse-grained feature maps and fine-grained feature maps, providing richer and more reliable basis for subsequent region matching and sub-pixel precision matching, and improving the accuracy and robustness of matching point pair determination.

[0068] The following is a concrete example to illustrate this. When extracting spatial domain features from a matched image pair, a pre-trained deep convolutional neural network (e.g., ResNet-50 or VGG-16) can be used as a feature extractor. By acquiring the outputs of its different convolutional layers, spatial domain feature maps at multiple scales can be obtained. For example, features at different scales can be extracted from the `res2`, `res3`, `res4`, and `res5` layers of ResNet-50. When extracting frequency domain features, for each scale of image features, a two-dimensional Fast Fourier Transform (FFT) is first performed to transform it to the frequency domain. In the frequency domain, a lightweight neural network module can be designed, such as a channel attention module consisting of a 1x1 convolutional layer and activation functions, to weight and mix different frequency channels of the Fourier transform result to enhance specific frequency information. Subsequently, an Inverse Fast Fourier Transform (IFFT) is performed on the processed frequency domain data to restore it to the spatial domain, thus obtaining spatial domain features with frequency domain characteristics. Finally, at the same scale, the extracted spatial domain features and frequency domain features are weighted and fused. For example, the fusion weight of spatial domain features can be set to 0.7 and the fusion weight of frequency domain features to 0.3, and the fused features are obtained by linear weighted summation. Among them, the fused features with lower resolution (e.g., corresponding to the `res4` or `res5` layers of ResNet-50) can be used as coarse-grained feature maps, while the fused features with higher resolution (e.g., corresponding to the `res2` or `res3` layers) can be used as fine-grained feature maps.

[0069] In some embodiments, frequency domain features in the coarse-grained feature map are used as the query basis, and spatial domain features in the coarse-grained feature map are used as the matching basis to perform region matching on the images in the matching image pair. This includes: performing frequency domain enhancement processing on the frequency domain features in the coarse-grained feature map; generating a query vector and a key vector based on the frequency domain features after frequency domain enhancement; generating a value vector based on the spatial domain features in the coarse-grained feature map; performing attention calculation based on the query vector, key vector, and value vector to obtain a similarity matrix between the images in the matching image pair; and selecting several local region pairs with the highest confidence in the images in the matching image pair based on the similarity matrix as matching region pairs.

[0070] Frequency domain enhancement processing can be performed on the frequency domain features in coarse-grained feature maps by designing specific filters (such as high-pass filters and band-pass filters) to perform convolution operations on the features in the frequency domain to enhance high-frequency details or information within a specific frequency range; or by adopting adaptive enhancement strategies, such as dynamically adjusting the gain of different frequency components based on the statistical properties of the feature map.

[0071] Query vectors and key vectors can be generated based on the frequency domain features after frequency domain enhancement. This can be achieved by mapping each feature point or region in the frequency domain enhanced feature map to the corresponding query vector and key vector through one or more fully connected layers (or 1x1 convolutional layers). These layers can learn how to convert the original features into a representation suitable for similarity calculation. Alternatively, a linear projection layer in a multi-head attention mechanism can be used to project the frequency domain features into different subspaces, thereby generating multiple query vectors and key vectors.

[0072] Value vectors can be generated from spatial domain features in a coarse-grained feature map. This can be achieved by mapping the spatial domain features in the coarse-grained feature map to value vectors through a fully connected layer (or a 1x1 convolutional layer); or by directly using the spatial domain features themselves as value vectors, or by performing a linear transformation on them.

[0073] Attention is calculated based on the query vector, key vector, and value vector to generate an attention weight distribution. Then, these weights are used to sum the value vectors in a weighted manner, which can focus on the most relevant part of the input. The attention weights obtained through attention calculation determine the similarity scores between different regions or feature points in the matched image pair. The similarity scores of each region are summarized to obtain the similarity matrix between the images in the matched image pair.

[0074] Based on the similarity matrix, select several local region pairs with the highest confidence in the images of the matched image pairs. A preset similarity threshold can be set to filter out all region pairs with similarity higher than the threshold, and then select the N region pairs with the highest similarity from these region pairs; or non-maximum suppression and other techniques can be used to find local maxima in the similarity matrix and suppress the non-maxima around them.

[0075] The following is a concrete example. When performing frequency domain enhancement on the frequency domain features in the coarse-grained feature map, a Gaussian filter can be used to smooth the features in the frequency domain to reduce the impact of high-frequency noise. When generating the query vector, key vector, and value vector, three independent 1x1 convolutional layers can be used to linearly transform the frequency-enhanced frequency domain features and spatial domain features to obtain vector representations of different dimensions. Attention calculation can employ a Scaled Dot-Product Attention mechanism, which calculates the dot product of the query vector and the key vector, divides it by a scaling factor, and then uses a Softmax function to obtain the attention weights. These weights are then weighted and summed with the value vector to form the final similarity matrix. When selecting the most confident local region pairs from the similarity matrix, the similarity matrix can first be thresholded. For example, a dynamic threshold can be set that is adaptively adjusted according to the feature distribution of the current image pair. Then, region pairs with the highest similarity scores in the local range that are higher than the threshold can be selected. For example, by traversing the similarity matrix, for each local maximum value, the non-maximum values ​​in a certain range around it can be suppressed to ensure that the selected region pairs are representative local optimal matches.

[0076] In some embodiments, before using the frequency domain features in the coarse-grained feature map as the basis for querying, the method further includes: generating content-adaptive positional bias features based on the spectral distribution characteristics of the frequency domain features in the coarse-grained feature map, and superimposing them onto the frequency domain features in the coarse-grained feature map.

[0077] Spectral distribution characteristics refer to the energy distribution of different frequency components in the frequency domain. For the frequency domain features of an image, its spectral distribution characteristics can reflect structural information such as texture, periodicity, and edge direction. For example, high-frequency components usually correspond to image details and edges, while low-frequency components correspond to the overall outline and brightness variations of the image. By analyzing these distribution characteristics, we can understand the local structure and patterns of the image at different spatial locations.

[0078] Content-adaptive positional bias features are features that dynamically generate and encode spatial positional information based on image content. Unlike fixed positional encoding, they adjust their representation of positional information according to the visual content (such as texture, structure, and semantic information) of different regions in the image, thus making the positional information more closely integrated with the image content. This type of feature can help understand the relative positional relationships between different elements in an image and provide richer spatial context for frequency domain features. Its generation methods can include, but are not limited to: extracting and encoding positional information from the original frequency domain features through learned transformations (e.g., using convolutional neural networks); or using self-attention mechanisms to enable features to perceive and integrate their relative spatial positions.

[0079] After obtaining the frequency domain features from the coarse-grained feature map, their spectral distribution characteristics are analyzed in depth. This analysis aims to understand the structural, textural, and directional information contained in the frequency domain features in different regions, thereby enabling the dynamic generation of positional bias features that match the image content based on these content characteristics. These positional bias features are not preset fixed values, but rather adaptively reflect the relative spatial relationships and contextual information of different parts of the image. Subsequently, these content-adaptive positional bias features are superimposed on the original coarse-grained frequency domain features. Through this superposition, the original frequency domain features are endowed with rich spatial contextual information, enabling them to retain their frequency domain expressive power while also gaining the ability to perceive the local spatial structure and position of the image. When this enhanced frequency domain feature is used as the query basis for subsequent region matching, it can more accurately and robustly identify matching region pairs, effectively overcoming the potential problem of missing spatial information when dealing with complex scenes using only frequency domain features, thus laying a solid foundation for subsequent sub-pixel precision matching and the determination of the final matching point pairs.

[0080] The following is a concrete example. After performing multi-scale spatial and frequency domain feature extraction on the images in the matching image pair to obtain a coarse-grained feature map, the frequency domain features in the coarse-grained feature map can be analyzed first. For example, a series of learnable convolutional kernels can be used to process the frequency domain feature map. These kernels are trained to capture texture patterns and edge information at different directions and scales, thereby revealing its spectral distribution characteristics. Based on these analysis results, a lightweight neural network module can be designed. This module takes the frequency domain feature map as input and outputs a positional bias map of the same size as the frequency domain feature map. Each pixel value in this positional bias map encodes the spatial context information of the feature at that location relative to the overall image content, and is dynamically generated according to the current image content, i.e., a content-adaptive positional bias feature. For example, this module can contain several residual blocks, each containing convolutional layers, normalization layers, and activation functions to learn complex nonlinear mapping relationships. Finally, the generated positional bias map is added element-wise to the original coarse-grained frequency domain feature map to obtain the enhanced frequency domain feature map. This enhanced frequency domain feature map can then be used as a query basis for subsequent region matching processes.

[0081] See Figure 4 In one embodiment, the method for refining the camera pose corresponding to the building exterior wall image includes, but is not limited to, steps S401 to S403.

[0082] Step S401: Based on the position and pose information in the EXIF ​​metadata, initialize the camera pose corresponding to each building exterior wall image to obtain the initial camera pose.

[0083] Step S402: Sort the building exterior wall images according to the inlier rate of the matched point pairs, and register the building exterior wall images to the 3D point cloud in sequence according to the sorting results and the initial camera pose.

[0084] In step S403, during the registration process, geometric consistency screening is performed on the matching point pairs participating in the registration, and periodic global optimization is performed on the registered building exterior wall images until all building exterior wall images are registered, resulting in refined camera pose and sparse 3D point cloud.

[0085] This application's approach first initializes the camera pose using EXIF ​​metadata, providing a good starting point for subsequent refinement. Then, it sorts the building exterior wall images by calculating the inlier rate of matched point pairs, prioritizing high-quality, reliable images, which helps construct a more robust initial 3D structure. During image registration to the 3D point cloud, a geometric consistency screening mechanism is introduced to effectively eliminate mismatched points, ensuring the accuracy of each registration. Furthermore, periodic global optimization, such as using bundle adjustment techniques, corrects accumulated errors, achieving optimal consistency of all camera poses and 3D point clouds globally. This phased, iterative refinement strategy with quality control makes camera pose estimation and sparse 3D point cloud generation more robust and accurate, laying a solid foundation for subsequent dense point cloud generation and 3D mesh model construction.

[0086] The following is a concrete example to illustrate this. After acquiring multiple images of building exterior walls and their corresponding EXIF ​​metadata, the GPS location information and initial attitude angles provided by the attitude sensor (such as an IMU) for each image are first extracted from the EXIF ​​metadata. This information is then converted into the camera's initial pose in the world coordinate system. Next, for any two building exterior wall images, the fundamental matrix is ​​calculated using the RANSAC algorithm based on the matching point pairs determined in the previous steps, and the number of inliers is counted. This number of inliers is used as a criterion for evaluating the matching quality of the image pairs. Then, all building exterior wall images are sorted according to these inlier counts; for example, images with a high inlier rate matching more images are prioritized as starting images. Subsequently, according to the sorting results, the building exterior wall images are registered to the 3D point cloud sequentially. During the registration process, whenever a new image is added and matches with an already registered image, geometric consistency filtering is performed; for example, matching point pairs with a distance greater than 2 pixels are discarded by calculating the epipolar distance between the matching point pairs. Furthermore, after every five images are registered, a global bundle adjustment optimization is triggered to make overall adjustments to the poses of all currently registered camera images and sparse 3D points to minimize reprojection errors. This process continues until all building exterior images are registered, ultimately resulting in a set of high-precision refined camera poses and corresponding sparse 3D point clouds.

[0087] See Figure 5 In one embodiment, the method for performing multi-view stereo matching processing on building exterior wall images includes, but is not limited to, steps S501 to S502.

[0088] Step S501: Determine the depth search range based on the refined camera pose and sparse 3D point cloud.

[0089] Step S502: Based on the depth search range and refined camera pose, perform multi-view stereo matching processing on the building exterior wall image to generate a dense point cloud.

[0090] This application's solution optimizes the efficiency and accuracy of multi-view stereo matching processing by introducing a defined depth search range. Based on the refined camera pose and sparse 3D point cloud, this information is first used to estimate the possible depth of each image pixel or region in 3D space, and a limited depth search range is set accordingly. This preprocessing step effectively reduces the search space of the subsequent stereo matching algorithm, eliminating the need for blind searching across the entire possible depth range and significantly reducing computational load. Subsequently, within the limited depth search range, multi-view stereo matching processing is performed on the building exterior image in conjunction with the refined camera pose. Specifically, the stereo matching algorithm searches for the best matching point for each pixel within its preset depth search range and calculates its 3D coordinates. In this way, the algorithm can focus more on the effective area, improving the accuracy and robustness of matching, and ultimately generating a high-quality dense point cloud. This strategy of determining the depth search range before matching makes the entire dense point cloud generation process more efficient and accurate, avoiding the computational burden and potential mismatches caused by an excessively large search space in complex scenes.

[0091] The following is a concrete example to illustrate this. After refining the camera pose and obtaining a sparse 3D point cloud, depth analysis is first performed on each point in the sparse 3D point cloud. Combining its projection position in different images with the corresponding refined camera pose, an initial depth estimate is calculated for each image pixel or image patch. For example, the sparse point cloud can be triangulated or a surface reconstructed, and then the image pixels can be projected onto this surface to obtain its approximate depth. Based on this, a depth fluctuation range based on statistical or empirical values ​​can be set. For example, based on the estimated depth value, the depth value can be expanded upwards and downwards by 5% each, forming a preliminary depth search range. Subsequently, this depth search range is used as input to the multi-view stereo matching module. This module can employ a deep learning-based MVS method, for example, constructing a cost volume containing multiple depth hypothesis planes, each plane corresponding to a certain depth value within the depth search range. The cost volume is processed by a convolutional neural network to predict the depth map of each pixel, and finally these depth maps are fused to generate a dense point cloud covering the surface of the building's exterior wall.

[0092] This application also provides an electronic device. (See appendix.) Figure 6 , Figure 6 This is a schematic diagram of the main structure of an electronic device according to an embodiment of this application. Figure 6As shown, the electronic device in this embodiment mainly includes a processor 601 and a memory 602. The memory 602 can be configured to store a program for executing the methods of the above-described method embodiments, and the processor 601 can be configured to execute the program in the memory 602. This program includes, but is not limited to, a program for executing the methods of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application.

[0093] In some possible embodiments of this application, the electronic device may include multiple processors 601 and multiple memories 602. The program executing the methods of the above method embodiments may be divided into multiple subroutines, each of which may be loaded and run by a processor 601 to perform different steps of the methods of the above method embodiments. Specifically, each subroutine may be stored in a different memory 602, and each processor 601 may be configured to execute programs in one or more memories 602 to jointly implement the methods of the above method embodiments; that is, each processor 601 executes different steps of the methods of the above method embodiments to jointly implement the methods of the above method embodiments.

[0094] The aforementioned multiple processors 601 can be processors deployed on the same device. For example, the aforementioned electronic device can be a high-performance device composed of multiple processors, and the aforementioned multiple processors 601 can be processors configured on that high-performance device. Alternatively, the aforementioned multiple processors 601 can also be processors deployed on different devices. For example, the aforementioned electronic device can be a server cluster, and the aforementioned multiple processors 601 can be processors on different servers within the server cluster.

[0095] This application also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the methods of the above-described method embodiments, the program of which can be loaded and run by a processor to implement the above-described methods. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a memory formed by various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0096] The building exterior wall image processing method, electronic device, and storage medium provided in this application determine matching point pairs by simultaneously utilizing the spatial and frequency domain features of the building exterior wall image. By fusing local detail information in the spatial domain with global periodicity information in the frequency domain, the matching accuracy and inlier rate in repetitive texture scenes can be significantly improved. During camera pose refinement and dense point cloud generation, the initial information provided by EXIF ​​metadata and the refined sparse point cloud are fully utilized to avoid blind searching from scratch. Iterative optimization is performed through matching point pairs, combined with geometric consistency filtering and periodic global optimization to ensure the accuracy of the camera pose. Based on the refined camera pose and sparse point cloud, the depth search range can be determined more accurately in the multi-view stereo matching stage, improving the efficiency and accuracy of dense point cloud generation, and enhancing the matching accuracy of building exterior wall images, the quality of 3D reconstruction, and the reliability of structured measurement data.

[0097] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for processing images of building exterior walls, characterized in that, include: Acquire multiple images of building exterior walls and their corresponding EXIF ​​metadata; Based on the EXIF ​​metadata, select several matching image pairs from the building exterior wall images; Based on the spatial domain and frequency domain features of the building exterior wall image, the matching point pairs between the images in the matching image pair are determined; Based on the matching point pairs and the EXIF ​​metadata, the camera pose corresponding to the building exterior wall image is refined to obtain the corresponding refined camera pose and sparse 3D point cloud. Based on the refined camera pose and the sparse 3D point cloud, multi-view stereo matching processing is performed on the building exterior wall image to generate a dense point cloud. A three-dimensional mesh model of the building's exterior wall is generated based on the dense point cloud, and structured measurement data of the building's exterior wall is generated based on the three-dimensional mesh model. The step of selecting several matching image pairs from the building exterior wall image based on the EXIF ​​metadata includes: Based on the location information, pose information, and imaging parameter information in the EXIF ​​metadata, the coverage area of ​​the building exterior wall image on the building exterior wall is determined; The matching image pair is selected from the building exterior wall images based on the degree of overlap between the covered areas of any two building exterior wall images.

2. The building exterior wall image processing method according to claim 1, characterized in that, The step of determining the matching point pairs between images in the matching image pair based on the spatial domain features and frequency domain features of the building exterior wall image includes: Multi-scale spatial domain feature extraction and frequency domain feature extraction are performed on the images in the matched image pair to obtain coarse-grained feature maps and fine-grained feature maps; Using the frequency domain features in the coarse-grained feature map as the query basis and the spatial domain features in the coarse-grained feature map as the matching basis, region matching is performed on the images in the matching image pair to obtain matching region pairs between the images in the matching image pair. Based on the matching region pair, sub-pixel precision matching is performed on the fine-grained feature map to obtain the matching point pair.

3. The building exterior wall image processing method according to claim 2, characterized in that, The step of performing multi-scale spatial domain feature extraction and frequency domain feature extraction on the images in the matched image pair includes: Extract spatial domain features of the images in the matched image pairs at multiple scales; Fourier transforms are performed on the images in the matched image pair at multiple scales, and the transform results are processed by learnable channel mixing in the frequency domain. Then, the images are restored to the spatial domain by inverse Fourier transform to obtain the frequency domain features of the images in the matched image pair. According to the preset fusion weights, the spatial domain features and frequency domain features of the images in the matched image pair at the same scale are weighted and fused to obtain the coarse-grained feature map and the fine-grained feature map.

4. The building exterior wall image processing method according to claim 2, characterized in that, The step of using the frequency domain features in the coarse-grained feature map as the query basis and the spatial domain features in the coarse-grained feature map as the matching basis to perform region matching on the images in the matched image pair includes: The frequency domain features in the coarse-grained feature map are subjected to frequency domain enhancement processing, and a query vector and a key vector are generated based on the frequency domain features after frequency domain enhancement. A value vector is generated based on the spatial domain features in the coarse-grained feature map. Attention is calculated based on the query vector, the key vector, and the value vector to obtain a similarity matrix between images in the matched image pair. Based on the similarity matrix, several local region pairs with the highest confidence are selected from the images in the matched image pairs as the matched region pairs.

5. The building exterior wall image processing method according to claim 2, characterized in that, Before using the frequency domain features in the coarse-grained feature map as the query basis, the method further includes: Based on the spectral distribution characteristics of the frequency domain features in the coarse-grained feature map, a content-adaptive positional bias feature is generated and superimposed on the frequency domain features in the coarse-grained feature map.

6. The building exterior wall image processing method according to claim 1, characterized in that, The step of refining the camera pose corresponding to the building exterior wall image based on the matching point pairs and the EXIF ​​metadata includes: Based on the position and pose information in the EXIF ​​metadata, the camera pose corresponding to each of the building exterior wall images is initialized to obtain the initial camera pose. The building exterior wall images are sorted according to the inlier rate of the matched point pairs, and the building exterior wall images are sequentially registered to the 3D point cloud according to the sorting result and the initial camera pose. During the registration process, geometric consistency screening is performed on the matching point pairs participating in the registration, and periodic global optimization is performed on the registered building exterior wall images until all building exterior wall images are registered, thereby obtaining the refined camera pose and the sparse 3D point cloud.

7. The building exterior wall image processing method according to claim 1, characterized in that, The step of performing multi-view stereo matching processing on the building exterior wall image based on the refined camera pose and the sparse 3D point cloud includes: The depth search range is determined based on the refined camera pose and the sparse 3D point cloud; Based on the depth search range and the refined camera pose, the building exterior wall image is subjected to multi-view stereo matching processing to generate the dense point cloud.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the building exterior wall image processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the building exterior wall image processing method according to any one of claims 1 to 7.

Images