A method for enhancing a three-dimensional map of a residential building

Abstract

The application provides a residential building three-dimensional image enhancement processing method, relates to the technical field of three-dimensional image enhancement processing methods, and comprises the following steps: obtaining a residential building image and three-dimensional point cloud data, constructing a cross-modal feature encoder, constructing a multi-modal feature bridging module, constructing a cross-modal feature decoder, and constructing a three-dimensional image enhancement model; in the cross-modal feature encoder, a method combining a convolutional neural network and a Transformer is used to extract features of the residential building image, and a method combining random point sampling and spherical neighborhood search is used in combination with a modular network based on the Transformer to extract features of the three-dimensional point cloud data; the multi-modal feature bridging module fully fuses the residential building image and the point cloud information by using a cross-modal attention mechanism, extracts complementary features, and enhances the consistency of the reconstructed three-dimensional space; and in the cross-modal feature decoder, deconvolution, up-sampling and a multi-layer perception machine are used to restore the residential building image features to the original image space.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional image enhancement processing methods, and more particularly relates to a three-dimensional image enhancement processing method for residential buildings. BACKGROUND

[0002] With the continuous development of three-dimensional vision technology, three-dimensional data processing and analysis have been widely applied in many fields, especially in the fields of architecture, urban planning and autonomous driving. The acquisition of three-dimensional data usually relies on technologies such as laser radar, structured light, RGB-D sensor, stereo vision system and deep learning model. These data provide more rich spatial information than traditional two-dimensional images, and can better express the geometric shape and spatial structure of objects.

[0003] In the field of three-dimensional image enhancement and reconstruction, point cloud data, as a common three-dimensional data representation method, provides detailed information about the surface and spatial position of objects. However, the sparsity and irregular shape of point cloud data make its processing and analysis more complex, especially in three-dimensional enhancement and reconstruction tasks. Traditional image data performs well in object detection and recognition tasks, but lacks depth information when dealing with three-dimensional spatial structures, which limits its accuracy in modeling and object recognition.

[0004] In order to realize high-quality three-dimensional image enhancement, a method based on multi-modal learning is proposed, which combines image data and three-dimensional point cloud data. Through the deep fusion of the two modal information, the problem of insufficient expression ability of single modal information is solved, and three-dimensional images with rich details and significant overall effect are generated. SUMMARY

[0005] In order to solve the above technical problems, the application provides a three-dimensional image enhancement processing method for residential buildings to solve the above problems.

[0006] A three-dimensional image enhancement processing method for residential buildings, comprising the following steps:

[0007] S1, obtaining residential building images and three-dimensional point cloud data, and preprocessing the residential building images and three-dimensional point cloud data respectively;

[0008] S2, constructing a cross-modal feature encoder, the cross-modal feature encoder being used for extracting features of the residential building images and the three-dimensional point cloud data;

[0009] S3, constructing a multi-modal feature bridging module, the multi-modal feature bridging module being used for fusing features of the residential building images and the three-dimensional point cloud data;

[0010] S4, a cross-modal feature decoder is constructed, which is used to reconstruct the fused features of the multi-modal feature bridge module, and generate an enhanced three-dimensional map;

[0011] S5, a three-dimensional map enhancement model is constructed, which is composed of input, cross-modal feature encoder, multi-modal feature bridge module, cross-modal feature decoder and output.

[0012] Preferably, in the S2 step, for the cross-modal feature encoder, for the residential building image, the input residential building image to the three-dimensional map enhancement model, , , and are the height, width and channel number of the residential building image respectively, the residential building image is divided into non-overlapping image blocks, and each image block is embedded into a feature vector space using linear projection, , is the linear projection, is the position embedding, is the modal embedding, a feature extractor combining ResNet50 and Transformer is used to extract the features of the residential building image, and the residential building image feature is obtained, which contains image tokens, and the feature dimension of each token is , for the point cloud data, the input point cloud data to the three-dimensional map enhancement model, , is a three-dimensional real space, the point cloud data is down-sampled using random point sampling, and representative points are selected, , for each representative point , the domain of is determined using spherical neighborhood search, and the domain is feature-extracted using a modular network based on Transformer to obtain the center feature of each neighborhood , for each , linear projection is used to embed into a feature vector space, , to obtain the point cloud feature , which contains point cloud tokens, and the feature dimension of each token is , in the S3 step, for the multi-modal feature bridge module, linear transformation matrices and are used to map the residential building image feature and the point cloud feature to a shared embedding space, wherein, , , For the input feature dimension, It is the dimension of the embedded space that yields the image features of the residential building. and point cloud features ,satisfy , , to compute an enhanced representation of point clouds.

[0013] Preferably, point cloud features are used. As a query Residential building image features As a key Sum Calculate the attention weights for point cloud features. , Using the values ​​of residential building image features Weights of point cloud features Computational point cloud augmentation representation, Calculate an enhanced representation of residential building images using residential building image features. As a query Point cloud features As a key Sum Calculate the attention weights of residential building image features. , Using point cloud feature values Weights of residential building image features Computational image enhancement representation of residential buildings Residual connections are used to combine the attention-enhanced features with the original features respectively. , , to obtain the image features of the residential building and point cloud features .

[0014] Preferably, in step S4, for the cross-modal feature decoder, for the residential building image features generated in step S3... Using deconvolution and upsampling operations to extract features from residential building images The image space is restored to a higher dimension, and then a multilayer perceptron is used to restore it to the original image space, resulting in the decoded features of the residential building image. For the point cloud features generated in step S3 The geometric structure of the point cloud is restored using back projection, and the structure of the point cloud is reconstructed using a dynamic graph convolutional neural network to obtain the decoded point cloud features. The decoded residential building image features and point cloud features are stitched and fused together. The characteristics after fusion are obtained. Features after fusion Reconstructing 3D images using 3D convolution. Generate enhanced 3D images of residential buildings .

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The technical solution provided by this invention proposes a 3D graph enhancement model. In the cross-modal feature encoder, a combination of convolutional neural networks and Transformers is used to extract features from residential building images. Random point sampling and sphere neighborhood search methods are combined with a Transformer-based modular network to extract features from 3D point cloud data. The multimodal feature bridging module uses a cross-modal attention mechanism to fully integrate residential building images and point cloud information, extracting complementary features and enhancing the consistency of the reconstructed 3D space. In the cross-modal feature decoder, deconvolution, upsampling, and a multilayer perceptron are used to restore the features of the residential building images to the original image space. Back-projection operations and a dynamic graph convolutional neural network are used to reconstruct the structure of the point cloud. 3D convolution operations are used to produce the enhanced 3D image of the residential building.

[0017] Advantages of Multimodal Fusion: This invention successfully achieves deep fusion of residential building images and 3D point cloud data through the collaborative work of a cross-modal feature encoder, a multimodal feature bridging module, and a cross-modal feature decoder. This fusion method fully utilizes the texture and semantic information of the images as well as the geometric structure information of the point clouds, making up for the shortcomings of single-modal data in representing the 3D structure of residential buildings, so that the generated 3D map can more comprehensively and accurately present the true appearance of the residential buildings.

[0018] Feature Extraction Effectiveness: In the cross-modal feature encoder, a combination of convolutional neural networks and Transformers is used to extract features from residential building images. This method captures both local texture and semantic information of the images and obtains global contextual relationships through a self-attention mechanism, thus enhancing feature representation capabilities. For 3D point cloud data, a combination of random point sampling and sphere neighborhood search methods with a Transformer-based modular network is used for feature extraction. This effectively captures the local geometric features of the point cloud, providing a high-quality feature foundation for subsequent fusion and reconstruction.

[0019] Enhanced Spatial Consistency: The cross-modal attention mechanism of the multimodal feature bridging module enhances the interaction between the image and point cloud during the fusion process by calculating attention weights between features of different modalities. This results in better consistency of the fused features when reconstructing 3D space. This helps improve the accuracy and stability of the 3D map and reduces errors and distortions during the reconstruction process.

[0020] High-quality image reconstruction: The cross-modal feature decoder processes the features of residential building images through operations such as deconvolution, upsampling, and multilayer perceptrons, effectively restoring image details and the original spatial structure, ensuring accurate matching between the decoded image features and the original image space. Simultaneously, for point cloud features, backprojection and dynamic graph convolutional neural networks are used for reconstruction, accurately restoring the geometric shape and spatial positioning of the point cloud, further improving the quality of the 3D image. Attached Figure Description

[0021] Figure 1 This is a flowchart of the three-dimensional image enhancement processing for residential buildings provided by the present invention.

[0022] Figure 2 This is a structural diagram of the cross-modal feature encoder module provided by the present invention.

[0023] Figure 3 This is a structural diagram of the cross-modal feature decoder module provided by the present invention. Detailed Implementation

[0024] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0025] Please see Figures 1-3 This invention provides a method for enhancing 3D images of residential buildings. The method proposes a 3D image enhancement model. In the cross-modal feature encoder, a combination of convolutional neural networks and Transformers is used to extract features from the residential building image. Random point sampling and sphere neighborhood search methods are combined with a Transformer-based modular network to extract features from the 3D point cloud data. The multimodal feature bridging module uses a cross-modal attention mechanism to fully integrate the residential building image and point cloud information, extracting complementary features and enhancing the consistency of the reconstructed 3D space. In the cross-modal feature decoder, deconvolution, upsampling, and a multilayer perceptron are used to restore the features of the residential building image to the original image space. Back-projection operations and a dynamic graph convolutional neural network are used to reconstruct the structure of the point cloud. 3D convolution operations are used to produce the enhanced 3D image of the residential building.

[0026] Please see Figure 1 As shown in the embodiment of this application, a method for enhancing a 3D image of a residential building is presented.

[0027] S1. Acquire images and 3D point cloud data of the residential building, and preprocess the images and 3D point cloud data of the residential building respectively.

[0028] Furthermore, in step S1, the preprocessing of the residential building image involves using a Gaussian filtering algorithm to denoise the image, normalizing the image, and then scaling the image to... To improve the size of the residential building image dataset, a radiometric transformation method was used to enhance the image dataset. For the preprocessing of the 3D point cloud data, a statistical outlier removal algorithm was used to remove noisy point cloud data. A voxel grid downsampling method was used to reduce the number of point clouds while retaining sufficient structural information. Surface reconstruction was performed on the sparse point cloud to fill the gaps between point clouds. After surface reconstruction, a smoothing algorithm was used to remove irregular or abnormal points to improve the quality of the point cloud.

[0029] S2. Construct a cross-modal feature encoder, which is used to extract features from residential building images and 3D point cloud data.

[0030] Furthermore, in step S2, for the cross-modal feature encoder, its structure is as follows: Figure 2 As shown, for a residential building image, input a residential building image. To 3D graph augmentation model, , , and These are the height, width, and number of channels of the residential building image, respectively. The residential building image is divided into non-overlapping image blocks, and each image block is embedded into the feature vector space using linear projection. , It is a linear projection. It is position embedding. It is a modal embedding, using a feature extractor combining ResNet50 and Transformer to extract features from residential building images, thus obtaining residential building image features. ,Include There are image tokens, and the feature dimension of each token is . For point cloud data, input point cloud data. To 3D graph augmentation model, , It is a three-dimensional real space, using random point sampling to downsample the point cloud data, selecting... A representative point, , For each representative point Use sphere neighborhood search to determine In the domain, a modular network based on Transformer is used to extract features from the domain, obtaining the central features of each neighborhood. For each Linear projection embedding into the feature vector space. Point cloud features were obtained. ,Include There are point cloud tokens, and the feature dimension of each token is... .

[0031] S3. Construct a multimodal feature bridging module, which is used to fuse features from residential building images and 3D point cloud data.

[0032] Furthermore, in step S3, for the multimodal feature bridging module, a linear transformation matrix is ​​used. and The image features and point cloud features of the residential building are mapped to a shared embedding space, where, , , For the input feature dimension, It is the dimension of the embedded space that yields the image features of the residential building. and point cloud features ,satisfy , Calculate an enhanced representation of the point cloud using point cloud features. As a query Residential building image features As a key Sum Calculate the attention weights for point cloud features. , Using the values ​​of residential building image features Weights of point cloud features Computational point cloud augmentation representation, ; Calculate an enhanced representation of the residential building image using residential building image features As a query Point cloud features As a key Sum Calculate the attention weights of residential building image features. , Using point cloud feature values Weights of residential building image features Computational image enhancement representation of residential buildings Residual connections are used to combine the attention-enhanced features with the original features respectively. , , to obtain the image features of the residential building and point cloud features .

[0033] S4. Construct a cross-modal feature decoder, which is used to reconstruct the features fused by the multimodal feature bridging module and generate an enhanced 3D map.

[0034] Furthermore, in step S4, the structure of the cross-modal feature decoder is as follows: Figure 3 As shown, for the residential building image features generated in step S3 Using deconvolution and upsampling operations to extract features from residential building images The image space is restored to a higher dimension, and then a multilayer perceptron is used to restore it to the original image space, resulting in the decoded features of the residential building image. For the point cloud features generated in step S3 The geometric structure of the point cloud is restored using back projection, and the structure of the point cloud is reconstructed using a dynamic graph convolutional neural network to obtain the decoded point cloud features. The decoded residential building image features and point cloud features are stitched and fused together. The characteristics after fusion are obtained. Features after fusion Reconstructing 3D images using 3D convolution. Generate enhanced 3D images of residential buildings .

[0035] S5. Construct a three-dimensional graph augmentation model, which consists of an input, a cross-modal feature encoder, a multimodal feature bridging module, a cross-modal feature decoder, and an output.

[0036] Furthermore, in step S5, the 3D graph augmentation model is written based on the PyTorch framework, using the Adam optimizer, mean squared error loss function, learning rate of 0.001, and training batch size of 300.

[0037] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for enhancing 3D images of residential buildings, characterized in that, Includes the following steps: S1. Acquire images and 3D point cloud data of the residential building, and preprocess the images and 3D point cloud data of the residential building respectively; S2. Construct a cross-modal feature encoder, which is used to extract features from residential building images and 3D point cloud data; S3. Construct a multimodal feature bridging module, which is used to fuse features from residential building images and 3D point cloud data; S4. Construct a cross-modal feature decoder, which is used to reconstruct the features fused by the multimodal feature bridging module and generate an enhanced 3D map; S5. Construct a three-dimensional graph augmentation model, which consists of an input, a cross-modal feature encoder, a multimodal feature bridging module, a cross-modal feature decoder, and an output. For cross-modal feature encoders, the input is a residential building image. To 3D graph augmentation model, , , and These are the height, width, and number of channels of the residential building image, respectively. The residential building image is divided into non-overlapping image blocks, and each image block is embedded into the feature vector space using linear projection. , It is a linear projection. It is position embedding. It is a modal embedding, using a feature extractor combining ResNet50 and Transformer to extract features from residential building images, thus obtaining residential building image features. ,Include There are image tokens, and the feature dimension of each token is . ; For point cloud data, input the point cloud data. To 3D graph augmentation model, , It is a three-dimensional real space, using random point sampling to downsample the point cloud data, selecting... A representative point, , For each representative point Use sphere neighborhood search to determine The neighborhood of a given region is used to extract features from the neighborhood using a modular network based on Transformer, obtaining the central features of each neighborhood. For each feature, a linear projection is used to embed it into the feature vector space. Point cloud features were obtained. ,Include There are point cloud tokens, and the feature dimension of each token is... ; For the multimodal feature bridging module, a linear transformation matrix is ​​used. and The image features and point cloud features of the residential building are mapped to a shared embedding space, where, , , For the input feature dimension, It is the dimension of the embedded space that yields the image features of the residential building. and point cloud features ,satisfy , To compute an enhanced representation of the point cloud; Computing augmented representations of residential building images: using residential building image features As a query Point cloud features As a key Sum Calculate the attention weights of residential building image features. , Using point cloud feature values Weights of residential building image features Computational image enhancement representation of residential buildings ; Residual connections are used to combine the attention-enhanced features with the original features respectively. , , obtain the image features of the residential building and point cloud features ; Decoded residential building image features and point cloud features splicing and fusion The characteristics after fusion are obtained. Features after fusion Reconstructing 3D images using 3D convolution. Generate enhanced 3D images of residential buildings .

2. The method for enhancing 3D images of residential buildings as described in claim 1, characterized in that, Using point cloud features As a query Residential building image features As a key Sum Calculate the attention weights for point cloud features. , Using the values ​​of residential building image features Weights of point cloud features Computational point cloud augmentation representation, .

3. The method for enhancing a 3D image of a residential building as described in claim 2, characterized in that, In step S4, for the cross-modal feature decoder, for the residential building image features generated in step S3... Using deconvolution and upsampling operations to extract features from residential building images The image space is restored to a higher dimension, and then a multilayer perceptron is used to restore it to the original image space, resulting in the decoded features of the residential building image. .

4. The method for enhancing 3D images of residential buildings as described in claim 3, characterized in that, For the point cloud features generated in step S3 The geometric structure of the point cloud is restored using back projection, and the structure of the point cloud is reconstructed using a dynamic graph convolutional neural network to obtain the decoded point cloud features. .

Images