Method, device and equipment for extracting three-dimensional building information based on multimodal data

Abstract

The present invention provides a method, device, and apparatus for extracting three-dimensional building information based on multimodal data. The method comprises: obtaining heterogeneous multimodal data corresponding to a study area and fusing the heterogeneous multimodal data to obtain multimodal fused data corresponding to the study area; predicting the planar information of buildings corresponding to the study area based on the multimodal fused data using a pre-trained multimodal semantic segmentation network; predicting the height information of buildings corresponding to the study area based on the multimodal fused data and the building planar information using a pre-trained height inversion network; and generating three-dimensional building information corresponding to the study area based on the building height information and the building planar information. This invention significantly improves existing technologies, such as slow speed, inability to detect large areas, and data redundancy.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing monitoring technology, and in particular to a method, device and equipment for extracting three-dimensional information of a building based on multimodal data. Background Art

[0002] Building stereo information has a wide range of applications in urban planning, architectural design, military simulation, and map navigation. At present, the methods for extracting building stereo information can be roughly divided into three categories: laser point cloud, oblique photogrammetry, and multi-source data fusion. The laser point cloud method uses LiDAR technology to obtain high-precision point cloud data and uses the geometric features and spatial distribution of the point cloud to extract the stereo information of the building. However, this method has slow processing speed and high cost; the multi-angle oblique photogrammetry method uses drones to obtain stereo image pairs to construct the stereo information of the building. However, this method has complex data processing and is easily restricted by airspace, making it impossible to achieve large-scale monitoring; the method based on multi-source data fusion is to comprehensively utilize the advantages of LiDAR, satellite images, drones, and other data to achieve the purpose of obtaining building stereo information. However, this method has the characteristics of diverse data sources, multiple latitudes, and easy redundancy, and cannot extract information quickly and effectively.

[0003] In summary, the above methods have problems such as slow speed, inability to monitor large areas, and data redundancy. Therefore, there is an urgent need for a method that can quickly and large-scale apply three-dimensional information of building objects. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device and equipment for extracting three-dimensional information of buildings based on multimodal data, which can significantly improve the problems of slow speed, inability to detect large areas, data redundancy and other problems existing in the existing technology.

[0005] In a first aspect, the present invention provides a method for extracting three-dimensional building information based on multimodal data, comprising:

[0006] Obtain heterogeneous multimodal data corresponding to the study area and perform data fusion processing on the heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area. The heterogeneous multimodal data include high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data and surface temperature data;

[0007] The pre-trained multimodal semantic segmentation network is used to predict the building plan information corresponding to the study area based on multimodal fusion data;

[0008] The pre-trained height inversion network is used to predict the building height information corresponding to the study area based on multimodal fusion data and building plan information;

[0009] Based on the building height information and building plane information, generate the three-dimensional building information corresponding to the study area.

[0010] In one embodiment, data fusion processing is performed on heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area, including:

[0011] Preprocessing, geometric registration and resampling of high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data and land surface temperature data are performed to obtain processed surface reflectivity, SAR backscatter coefficient data, digital surface model data, digital elevation model data and land surface temperature data;

[0012] The processed surface reflectivity, the SAR backscatter coefficient data and the surface temperature data are subjected to band combination processing, standardization processing and principal component analysis processing to obtain multimodal fusion data.

[0013] In one embodiment, a multimodal semantic segmentation network includes a block layer, multiple Mix Transform encoding layers, a feature extraction layer, and a decoding layer. The pre-trained multimodal semantic segmentation network predicts the building plan information corresponding to the study area based on multimodal fusion data, including:

[0014] Through the block layer, the multimodal fusion data is divided into multiple multimodal fusion sub-data, and the multimodal fusion sub-data is converted into a one-dimensional feature vector;

[0015] Through multiple Mix Transform encoding layers, the hierarchical feature vector corresponding to the one-dimensional feature vector is extracted;

[0016] Through the feature extraction layer, the hierarchical feature vector is subjected to down-sampling feature extraction processing and up-sampling feature fusion processing to obtain the target feature vector;

[0017] The target feature vector is decoded through the decoding layer to predict the building plan information corresponding to the study area.

[0018] In one embodiment, the height inversion network obtained by pre-training is used to predict the building height information corresponding to the study area based on multimodal fusion data and building plan information, including:

[0019] Using the building plane information as a mask, the multimodal fusion data corresponding to each building unit in the study area is extracted from the multimodal fusion data;

[0020] For any building unit, the multimodal fusion data corresponding to the building unit is processed by band mean to obtain the multimodal value corresponding to the building unit;

[0021] The pre-trained height inversion network is used to predict the building height information corresponding to each building unit in the study area based on the multimodal values ​​corresponding to each building unit.

[0022] In one embodiment, generating three-dimensional building information corresponding to a study area based on building height information and building plan information includes:

[0023] For any building unit in the study area, the building height information corresponding to the building unit is written into the building plane information to generate the building three-dimensional information corresponding to each building unit in the study area.

[0024] In one embodiment, the method further comprises:

[0025] A multimodal semantic segmentation sample set is constructed based on the multimodal fusion data and building plan samples corresponding to the specified sub-areas in the study area, and the multimodal semantic segmentation network is trained using the multimodal semantic segmentation sample set.

[0026] In addition, a multimodal height inversion dataset is constructed based on the digital surface model data, digital elevation model data, multimodal fusion data and building plane samples corresponding to the specified sub-areas in the study area, and the height inversion network is trained using the multimodal height inversion dataset.

[0027] In one embodiment, a multimodal height inversion dataset is constructed based on digital surface model data, digital elevation model data, multimodal fusion data, and building plane samples corresponding to a specified sub-area within a study area, including:

[0028] Determining relative height information corresponding to the designated sub-region based on the digital surface model data and the digital elevation model data corresponding to the designated sub-region;

[0029] Using the building plane sample as a mask, the multimodal value and building height value corresponding to each building unit in the specified sub-region are extracted from the multimodal fusion data and relative height information corresponding to the specified sub-region;

[0030] The multimodal values ​​are used as model input and the building height values ​​are used as labels to obtain a multimodal height inversion dataset.

[0031] In a second aspect, the present invention further provides a device for extracting three-dimensional building information based on multimodal data, comprising:

[0032] The data processing module is used to obtain heterogeneous multimodal data corresponding to the study area and perform data fusion processing on the heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area. The heterogeneous multimodal data includes high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data and surface temperature data;

[0033] The plane information prediction module is used to predict the plane information of buildings corresponding to the study area based on multimodal fusion data using a pre-trained multimodal semantic segmentation network;

[0034] The height information prediction module is used to predict the building height information corresponding to the study area based on multimodal fusion data and building plan information through the pre-trained height inversion network;

[0035] The three-dimensional information generation module is used to generate three-dimensional building information corresponding to the study area based on building height information and building plane information.

[0036] In a third aspect, the present invention further provides an electronic device comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0037] In a fourth aspect, the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0038] Embodiments of the present invention provide a method, apparatus, and device for extracting three-dimensional building information based on multimodal data. The method first acquires heterogeneous multimodal data corresponding to a study area and performs data fusion processing on the heterogeneous multimodal data to obtain multimodal fused data corresponding to the study area. The heterogeneous multimodal data includes high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data, and ground surface temperature data. A pre-trained multimodal semantic segmentation network is then used to predict the planar information of buildings corresponding to the study area based on the multimodal fused data. A pre-trained height inversion network is then used to predict the height information of buildings corresponding to the study area based on the multimodal fused data and the building planar information. Finally, based on the building height information and the building planar information, three-dimensional building information corresponding to the study area is generated. The method fuses heterogeneous multimodal data, effectively improving data validity and addressing data redundancy. Furthermore, a multimodal semantic segmentation model and a height inversion model suitable for the multimodal fused data are constructed to further improve data utilization, thereby obtaining building planar information and building height information, respectively. This allows efficient three-dimensional building information to be extracted even in large-area detection scenarios.

[0039] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A schematic flow chart of a method for extracting three-dimensional building information based on multimodal data provided by an embodiment of the present invention;

[0043] Figure 2 A technical framework diagram of a method for extracting three-dimensional building information based on multimodal data provided by an embodiment of the present invention;

[0044] Figure 3A schematic diagram of a process for processing heterogeneous multimodal data provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a multimodal data fusion process provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of a process for plane feature extraction provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the structure of a multimodal semantic segmentation network provided by an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of a process for height information inversion provided by an embodiment of the present invention;

[0049] Figure 8 The embodiment of the present invention provides an overall structure of an inversion deep learning network constructed using Transformer;

[0050] Figure 9 A schematic structural diagram of a device for extracting three-dimensional building information based on multimodal data provided by an embodiment of the present invention;

[0051] Figure 10 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] At present, the existing technology has problems such as slow speed, inability to monitor large areas, and data redundancy. Therefore, there is an urgent need for a method that can quickly and large-scale apply the three-dimensional information of building objects. With the development of the Internet of Things and multi-sensor technology, multi-source data fusion is one of the methods with the most development potential. In order to solve the difficulties of multi-source data fusion and improve the extraction performance at the same time, a building three-dimensional information extraction method based on heterogeneous multimodal data is designed. This method can stably and efficiently realize large-scale building three-dimensional information extraction.

[0054] Based on this, the present invention provides a method, device and equipment for extracting three-dimensional information of buildings based on multimodal data, which can significantly improve the problems of slow speed, inability to detect large areas, data redundancy and other problems existing in the existing technology, and at the same time effectively improve the problem of difficulty in multi-source data fusion existing in the existing technology.

[0055] To facilitate understanding of this embodiment, firstly, a method for extracting three-dimensional building information based on multimodal data disclosed in an embodiment of the present invention is described in detail. Figure 1 FIG. 1 is a flow chart of a method for extracting three-dimensional building information based on multimodal data, wherein the method mainly includes the following steps S102 to S108:

[0056] Step S102: Obtain heterogeneous multimodal data corresponding to the study area, and perform data fusion processing on the heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area.

[0057] Among them, heterogeneous multimodal data include high-resolution multispectral data, synthetic aperture radar data (SAR), digital surface model data (DSM), digital elevation model data (DEM) and surface temperature data. Multimodal fusion data are obtained based on the processed surface reflectivity, SAR backscatter coefficient data and surface temperature data corresponding to the study area.

[0058] In one example, high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data, and surface temperature data were preprocessed respectively, and geometric correction and resampling were performed on the preprocessed data to obtain the processed data corresponding to the study area. On this basis, multimodal fusion data were determined through band combination processing, standardization processing, and principal component analysis.

[0059] Step S104: using the pre-trained multimodal semantic segmentation network, predict the building plan information corresponding to the study area based on the multimodal fusion data.

[0060] The multimodal semantic segmentation network takes multimodal fusion data as input and outputs building plan information corresponding to the study area. This building plan information describes the distribution of each building unit within the study area. In one example, the multimodal fusion data is input to the multimodal semantic segmentation network, and the building plan information output by the multimodal semantic segmentation network is obtained.

[0061] Step S106: using the pre-trained height inversion network, the building height information corresponding to the study area is predicted based on the multimodal fusion data and the building plane information.

[0062] The input to the height inversion network is the multimodal value corresponding to each building unit within the study area, and the output is the building height information corresponding to each building unit within the study area. In one example, the building plan information can be used as a mask to extract the multimodal fusion data corresponding to each building unit. The multimodal value corresponding to each building unit is then determined and input into the height inversion network to obtain the building height information output by the height inversion network.

[0063] Step S108: Generate three-dimensional building information corresponding to the study area based on the building height information and the building plane information.

[0064] In one example, building height information can be written into building plane information to obtain building three-dimensional information corresponding to each building unit in the study area. The building three-dimensional information is used to describe the plane coordinates of the building unit (that is, the building plane information) and the building height information.

[0065] The method for extracting three-dimensional building information based on multimodal data provided by the embodiment of the present invention performs data fusion on heterogeneous multimodal data, effectively improving data validity and alleviating data redundancy. In addition, a multimodal semantic segmentation model and height inversion model suitable for multimodal fusion data are constructed to further improve data utilization, thereby obtaining building plan information and building height information respectively, and achieving efficient three-dimensional building information even in large-area detection scenarios.

[0066] Furthermore, an embodiment of the present invention also provides an implementation method for training a multimodal semantic segmentation network and a high-level inversion network, including:

[0067] (1) A multimodal semantic segmentation sample set is constructed based on the multimodal fusion data and building plane samples corresponding to the specified sub-areas in the study area, and the multimodal semantic segmentation network is trained using the multimodal semantic segmentation sample set.

[0068] In one example, a multimodal semantic segmentation sample set of size (s, m, m, 3) was constructed using multimodal fusion data and building plan samples corresponding to a specified subregion, combined with inversion and rotation light sample augmentation techniques. S represents the number of samples, and the multimodal semantic segmentation sample set was trained, validated, and tested in a 6:2:2 ratio. Furthermore, key hyperparameters of the multimodal semantic segmentation network, such as the training batch size, learning rate, and c1, were set to train and obtain the optimal multimodal semantic segmentation network.

[0069] (2) A multimodal height inversion dataset is constructed based on the digital surface model data, digital elevation model data, multimodal fusion data and building plane samples corresponding to the specified sub-areas in the study area, and the height inversion network is trained using the multimodal height inversion dataset.

[0070] In one example, the process of constructing a multimodal height inversion dataset is as follows:

[0071] (2.1) Determine the relative height information corresponding to the designated sub-area based on the digital surface model data and digital elevation model data corresponding to the designated sub-area. The relative elevation data is obtained based on the processed digital surface model data and digital elevation model data corresponding to the designated sub-area within the study area.

[0072] Specifically, the digital surface model data corresponding to the specified sub-area is used and digital elevation model data , calculate the relative height data H:

[0073] ;

[0074] (2.2) Using the building plan samples as masks, extract the multimodal values ​​and building height values ​​corresponding to each building unit within the specified sub-region from the multimodal fusion data and relative height information corresponding to the specified sub-region. Specifically, based on the spatial information of the building plan samples, the multimodal fusion data and relative height information are superimposed to obtain the multimodal fusion data and relative height information within each building unit. The mean of each band of the multimodal fusion data and relative height information within each building unit is calculated to obtain the multimodal value and building height value for each building unit.

[0075] (2.3) Using the multimodal values ​​as model input and the building height values ​​as labels, a multimodal height inversion dataset is obtained. Specifically, all building units are processed one by one according to (2.2) to obtain a multimodal height inversion dataset, where the multimodal height inversion dataset size is (3, s) and the height data size is (1, s), where s is the number of samples.

[0076] The multimodal height inversion dataset is trained, validated, and tested in a 6:2:2 ratio. Key hyperparameters of the inversion deep learning network, such as training batch size, learning rate, and d, are set to train and obtain the optimal model.

[0077] On this basis, the embodiment of the present invention provides the following Figure 2The technical framework diagram of a building stereo information extraction method based on multimodal data shown in the figure includes: firstly processing high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data and surface temperature data to generate heterogeneous multimodal data to be fused; then using standardization, principal component analysis and other technologies to generate multimodal fusion data, reducing the noise of heterogeneous multimodal data and improving the availability of data; using existing building plane samples and multimodal fusion data to generate a multimodal semantic segmentation sample set, and constructing a semantic segmentation network suitable for multimodal fusion data, and obtaining the optimal multimodal semantic segmentation network through model training and verification. A network is built to predict multimodal fusion data over a larger area and obtain building plane information over a large area; relative heights are calculated for digital surface model data and digital elevation model data to generate relative height data, which are then superimposed on multimodal fusion data and building plane samples to generate a multimodal height inversion dataset. At the same time, a height inversion network suitable for multimodal data is constructed, and the optimal height inversion model is obtained through model training and verification; the predicted building plane information data is superimposed on a larger area of ​​multimodal fusion data and predicted through the height inversion model to obtain building height information; finally, the building height information is written into the building plane information to obtain the building three-dimensional information.

[0078] exist Figure 2 On the basis of this, an embodiment of the present invention provides a specific implementation of a method for extracting three-dimensional information of a building based on multimodal data.

[0079] (1) Heterogeneous multimodal data processing: Preprocessing, geometric registration, and resampling of high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data, and land surface temperature data are performed to obtain processed surface reflectivity, SAR backscatter coefficient data, digital surface model data, digital elevation model data, and land surface temperature data.

[0080] In the specific implementation, see Figure 3The flowchart of a heterogeneous multimodal data processing shown in the figure includes: firstly, data preprocessing is performed on the original high-resolution multispectral data and synthetic aperture radar (SAR) data, wherein the high-resolution multispectral data is preprocessed by orthorectification, atmospheric correction, fusion and other preprocessing to generate surface reflectance data; the SAR data is processed by radiation correction, filtering, terrain correction and other processing to generate SAR backscatter coefficient data; the DSM is filtered to eliminate the influence of small noise; since the spatial reference and spatial position of the heterogeneous multimodal data are not unified, the optical surface reflectance data is used as a reference to geometrically align the SAR backscatter coefficient data, DSM, and surface temperature data, and since the optical data is orthorectified based on the digital elevation model (DEM) data, the geometric position of the DEM data and the surface reflectance data is the same. Consistent, after geometric registration, the spatial positions and spatial references of all heterogeneous multimodal data are consistent; then the resampling technology is used to process other data into a spatial resolution consistent with the surface reflectivity data, and the surface reflectivity data, SAR backscatter coefficient data, and surface temperature data at this time are recorded as SR, BC, and BT; since DSM data are usually data of a smaller range obtained by drones, and other data are basically obtained by satellites, and the collection range is much larger than the DSM range, it is necessary to perform intersection processing on the data to obtain the area where all data overlap, and the surface reflectivity data, SAR backscatter coefficient, DEM, DSM, and surface temperature data at this time are recorded as M1, M2, M3, M4, and M5. The number of bands after these data is N1, N2, N3, N4, and N5, and the length and width are both (H, W).

[0081] It should be noted that during the training phase of the network, the data needs to be intersected to obtain areas where all data overlap in order to construct a multimodal semantic segmentation sample set and a multimodal height inversion dataset. DSM is mainly used to determine the building height values ​​(as labels) in the multimodal height inversion dataset. During the application phase of the network, there is no need to intersect the data. It only needs to undergo preprocessing, geometric calibration, and resampling to obtain the surface reflectivity data, SAR backscatter coefficient, and surface temperature data of the entire study area, so as to facilitate the subsequent determination of multimodal fusion data as the input of the multimodal semantic segmentation network and the height inversion network.

[0082] (2) Multimodal data fusion: performing band combination processing, normalization processing and principal component analysis processing on the processed surface reflectivity, the SAR backscatter coefficient data and the surface temperature data to obtain multimodal fusion data.

[0083] Since heterogeneous multimodal data are diverse, multi-dimensional, and prone to redundancy, and the radiation value ranges and physical meaning representations of optical, SAR, and temperature data are inconsistent, and various types of noise may appear between the data, they need to be standardized and processed using PCA (principal component analysis) to achieve the purpose of multimodal data fusion and effective information extraction.

[0084] See also Figure 4 The following is a flow chart of multimodal data fusion, which includes: first, band combination of pre-processed optical, SAR, and temperature data. Since semantic segmentation deep learning networks are usually oriented towards natural images, the data needs to be quantized to 0-255. Multimodal data includes not only image data but also non-standardized data such as SAR and temperature, and the value range is not uniform, so the combined data is standardized. Then, PCA is used to extract the main effective information from the combined data, reduce the noise interference of different data sources, and generate multimodal fusion data. The specific process is as follows:

[0085] First, combine the M1, M2, and M5 bands to generate a multidimensional array F1 of size (N, H, W). Then transform the three-dimensional array into a two-dimensional array F2 of size (N, H*W), where N=N1+N2+N5. Then perform normalization to generate F3:

[0086] ;

[0087] ;

[0088] ;

[0089] Where i is The i-th band, is the mean value of band i, is the standard deviation of band i, is the standardized data of the i band.

[0090] Then use PCA to reduce the dimension of F3 as follows:

[0091] ;

[0092] ;

[0093] in, is the mean of the i-th dimension array, C is The covariance of .

[0094] Perform eigendecomposition on the covariance C to obtain eigenvalues ​​and eigenvectors:

[0095] ;

[0096] in is a feature vector of size (H*W, 1), is the characteristic value.

[0097] Select the eigenvectors v1, v2, and v3 corresponding to the three largest eigenvalues, and generate the PCA dimensionality reduction array P of size (3, H*W):

[0098] ;

[0099] Perform array size transformation on P to generate fused data F, whose size is (3, H, W).

[0100] (3) Plane feature extraction: see Figure 5 The schematic diagram of a plane feature extraction process shown in the figure includes: first, building a deep learning network suitable for multimodal fusion data, combining existing building plane samples to construct a multimodal semantic segmentation sample set, then training the model, and using the verified multimodal semantic segmentation network to predict the multimodal fusion data, and finally generating a large range of building plane information. Specifically, it includes:

[0101] Multimodal semantic segmentation network: see Figure 6 The diagram shows the structure of a multimodal semantic segmentation network, which includes a block layer, multiple Mix Transform encoding layers, a feature extraction layer, and a decoding layer. In this embodiment of the present invention, a building semantic segmentation network is constructed using the Mix Transformer, which is suitable for multimodal data, as the basic backbone. The core of this network is the Mix Transformer Block, which combines the advantages of convolutional neural networks (CNNs) and Transformers to effectively extract hierarchical features from multimodal data and efficiently obtain building plan information.

[0102] Based on this, the process of using the network to predict the building plan information is as follows: Through the blocking layer, the multimodal fusion data is divided into multiple fixed-size multimodal fusion sub-data, and the multimodal fusion sub-data is converted into a one-dimensional feature vector; through multiple Mix Transform encoding layers, the hierarchical feature vector corresponding to the one-dimensional feature vector is effectively extracted; through the feature extraction layer, the hierarchical feature vector is downsampled and processed by feature extraction and upsampled by fusion to obtain the target feature vector, which is used to gradually reduce the spatial resolution of the feature map, increase the abstract level of the feature, and fuse features at different levels to enhance the model's perception of global and local information; through the decoding layer, the feature map is restored to the original input resolution to achieve decoding of the target feature vector to predict the building plan information corresponding to the study area.

[0103] In the specific implementation, please refer to the structural information of the multimodal semantic segmentation network shown in Table 1:

[0104] Table 1 Structural information of multimodal semantic segmentation network

[0105]

[0106] Based on this, the specific process is as follows: First, the size of the multimodal sample data is set to (m, m, 3) and input into the block layer. The sample data is divided into sub-data blocks of (4, 4). Each sub-database block is transformed into a one-dimensional feature vector after linear projection, and the output data size is (m / 4, m / 4, c1); then the data enters the encoding layer with Mix Transformer as the basic network. The encoding layer contains 3 Mix Transformer Blocks. Each Mix Transformer Block includes a normalization layer, a Multi-Head Self-Attention (MHSA) layer, a Residual Connection layer, a LayerNormalization layer, a Mixer layer, and a Residual Connection layer, the output data size is (m / 4, m / 4, c1); the data passing through the encoding layer flows into the feature extraction layer, which includes the downsampling feature extraction module and the upsampling feature fusion module. The output data size of this layer is (m / 4, m / 4, c1); then the data flows into the decoding layer, which includes the upsampling module, convolution layer and activation function, and the output data size is (m, m, 1); the loss function uses cross-entropy loss (Cross-Entropy Loss).

[0107] The training process of the multimodal semantic segmentation network can be found in the above (1), and will not be described in detail in the embodiment of the present invention. After the multimodal semantic segmentation network training is completed, multimodal data fusion is performed on SR, BC, and BT to generate multimodal fusion data of a larger range. The optimal multimodal semantic segmentation network is then used to predict the large-scale multimodal fusion data to obtain the building plan information D1.

[0108] (IV) Height information inversion: see Figure 7The schematic diagram of the process of height information inversion shown includes: first, using M3 and M4 data to calculate relative height data, superimposing relative height data and multimodal fusion data by building plane samples to generate a multimodal height inversion data set, constructing a height inversion network suitable for multimodal data, and performing model training and verification to obtain the optimal height inversion network, then superimposing multimodal fusion data by building plane information to generate data to be inverted for large-scale inversion of building height information, and then using the optimal height inversion data to predict the data to be inverted, and finally obtaining a large range of building height information.

[0109] High-level inversion networks: Generally, high-level inversion networks include random forests, support vector machines, recurrent neural networks, etc. However, these networks often fail to meet the performance requirements when adapting to multimodal data. Transformer-based deep learning networks can be applied to complex multimodal data, such as Figure 8 The figure shows an overall structure of an inversion deep learning network constructed using Transformer, including an input layer, an embedding layer, a position encoding layer, a Transformer encoder, a global pooling layer, and an output layer.

[0110] The data to be inverted is sent from the input layer to the embedding layer, mapped to a high-dimensional space, and then the position encoding layer is used to assign position information to the features so that they can be effectively used by the Transformer encoder; the Transformer encoder uses a multi-head self-attention mechanism and a feedforward neural network to learn the relationship between data features and extract key information; finally, the global pooling layer integrates the encoded information, and the output layer completes the linear transformation to generate building height information.

[0111] In the specific implementation, please refer to the structural information of the height inversion network shown in Table 2:

[0112] Table 2 Structural information of the height inversion network

[0113]

[0114] Based on this, the specific process is as follows: Let the training sample data be X(3, s), where s is the number of samples and d is the number of fused data features. First, an embedding layer is used to map the training data consisting of three features into a high-dimensional space to better capture complex feature relationships. The output data size becomes (1, s, d), where d is the hidden dimension. Then, a positional encoding layer adds positional information to the data. Positional encoding helps the algorithm identify the locations of different features. The output positional encoding data size is (1, s, d). The Transformer encoder is the core component of the network. It consists of multiple layers of Transformer encoders. Through a self-attention mechanism and a feedforward neural network, it learns the interdependencies between features and deeply encodes the input feature sequence. The output encoded data retains important feature information and has an output data size of (1, s, d). The global pooling layer performs a pooling operation on the encoded feature sequence to extract the key features of each sample. The output data size is (s, d). Finally, these features are integrated in the output layer and then subjected to a linear transformation to obtain the final regression prediction value. The output size is (1, s). The network uses mean squared error (MSE) as the loss function and the Adam optimizer for training.

[0115] The training process of the height inversion network can be found in the above (2), and the embodiment of the present invention will not be described in detail. After the height inversion network training is completed, the building height information can be predicted according to the following process: using the building plane information as a mask, the multimodal fusion data corresponding to each building unit in the study area is extracted from the multimodal fusion data; for any building unit, the multimodal fusion data corresponding to the building unit is processed according to the band mean to obtain the multimodal value corresponding to the building unit; through the height inversion network obtained in advance, based on the multimodal value corresponding to each building unit, the building height information corresponding to each building unit in the study area is predicted. In the specific implementation, multimodal data fusion is performed on SR, BC, and BT to generate a larger range of multimodal fusion data, and then the building plane information D1 is superimposed to obtain the multimodal value mean of each D1 building element. Based on the multimodal mean, a large range of multimodal data to be inverted is constructed, and the optimal height inversion network is used to predict the large range of multimodal fusion data to be inverted to obtain a large range of building height information D2.

[0116] (V) 3D Information Acquisition: For any building unit within the study area, the building height information corresponding to that building unit is written into the building plan information to generate 3D building information corresponding to each building unit within the study area. In specific implementation, 3D building information D is obtained by integrating D1 and D2 and writing D2's height information into D1 data.

[0117] Furthermore, an embodiment of the present invention also provides an application example of extracting three-dimensional building information based on multimodal data, including:

[0118] (a) Heterogeneous Multimodal Data Processing: The study area for this embodiment of the present invention is Hebi City, Henan Province. The heterogeneous data sources include multispectral imagery, SAR data, temperature products, DSM, and DEM. The multispectral imagery is acquired by the GF1 PMS sensor and, after preprocessing, is surface reflectance data with a resolution of 2 meters and four bands. SAR data is acquired by Sentinel1 and, after preprocessing, is VV and VH backscatter coefficients with a resolution of 20 meters and two bands. The temperature product is acquired by the Landsat9 TIRS sensor with a resolution of 30 meters. The DSM is acquired by drone with a resolution of 0.2 meters and one band. The DEM has a resolution of 12 meters and one band. Except for the DEM, all other data were acquired in May 2024.

[0119] Based on the multispectral surface reflectance data and DEM, SAR, temperature, and DSM are registered respectively. Then all data are sampled to a resolution of 2m to generate SR, BC, and BT. Then, intersection extraction is performed on all data to obtain M1, M2, M3, M4, and M5. The number of bands is 4, 2, 1, 1, and 1, respectively, and the size is (8671, 10829).

[0120] (b) Multimodal data fusion: First, the preprocessed M1, M2, and M5 data are combined by bands. Then, the combined data is standardized. Finally, PCA is used to extract effective information from the standardized data to generate multimodal fusion data F with a size of (3, 8671, 10829).

[0121] (c) Plane Information Extraction: The sample size m of the multimodal semantic segmentation network was set to 512, and c1 to 128. Using F and the building plan samples, 2150 samples were constructed, of which 1290 were used for training, 428 for validation, and 428 for testing. The training batch size was 200, and the learning rate was 0.1. The optimal model was generated and then used to predict a larger range of multimodal fusion data, obtaining the building plan data D1.

[0122] (d) Height Inversion: The training batch size, learning rate, and number of hidden layers d for the height inversion network were set to 100, 0.1, and 64, respectively. The sample size m was set to 512, and c1 was set to 128. Using the multimodal fusion data F, building plan samples, and building height data, 2150 inversion samples were constructed, of which 1290 were used for training, 428 for validation, and 428 for testing. Through training and validation, an optimal inversion model was generated. This model was then used to predict building height data D2 using a larger range of multimodal fusion data.

[0123] (e) Acquisition of stereoscopic information:

[0124] By integrating D1 and D2, we can obtain the 3D information corresponding to each building unit in Hebi City, Henan Province.

[0125] In summary, the method for extracting three-dimensional building information based on multimodal data provided by the embodiments of the present invention utilizes techniques such as principal component analysis and standardization to improve data validity and achieve heterogeneous multimodal data fusion. By constructing a semantic segmentation network and an inversion network suitable for multimodal fusion data, data utilization efficiency can be further improved, and building plan and height information can be obtained separately. This method is applicable to a variety of heterogeneous data sources such as optical, SAR, temperature, DSM, DEM, etc., and has the advantages of strong robustness, high efficiency, and convenience for large-scale monitoring, providing basic technical support for urban monitoring and supervision.

[0126] On the basis of the above embodiments, the present invention provides a device for extracting three-dimensional building information based on multimodal data. Figure 9 The structure diagram of a device for extracting three-dimensional building information based on multimodal data is shown in FIG. The device mainly includes the following parts:

[0127] The data processing module 902 is used to obtain heterogeneous multimodal data corresponding to the study area and perform data fusion processing on the heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area. The heterogeneous multimodal data includes high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data, and surface temperature data.

[0128] Plane information prediction module 904, used to predict the building plane information corresponding to the study area based on the multimodal fusion data using a pre-trained multimodal semantic segmentation network;

[0129] A height information prediction module 906 is configured to predict building height information corresponding to a study area based on multimodal fusion data and building plan information using a pre-trained height inversion network;

[0130] The 3D information generating module 908 is configured to generate 3D building information corresponding to the study area based on the building height information and the building plane information.

[0131] The device for extracting three-dimensional building information based on multimodal data provided by the embodiment of the present invention performs data fusion on heterogeneous multimodal data, effectively improving data validity and alleviating data redundancy. In addition, a multimodal semantic segmentation model and a height inversion model suitable for multimodal fusion data are constructed to further improve data utilization, thereby obtaining building plane information and building height information respectively, and realizing efficient three-dimensional building information even in large-area detection scenarios.

[0132] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0133] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0134] Figure 10 A structural diagram of an electronic device provided in an embodiment of the present invention, wherein the electronic device 100 includes: a processor 10, a memory 11, a bus 12 and a communication interface 13, wherein the processor 10, the communication interface 13 and the memory 11 are connected via the bus 12; the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.

[0135] The memory 11 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 13 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0136] The bus 12 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0137] Among them, the memory 11 is used to store programs, and the processor 10 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 10 or implemented by the processor 10.

[0138] The processor 10 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 10. The processor 10 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 11 , and the processor 10 reads the information in the memory 11 and completes the steps of the above method in combination with its hardware.

[0139] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the previous method embodiment and will not be repeated here.

[0140] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0141] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for extracting three-dimensional building information based on multimodal data, characterized in that: include: Acquiring heterogeneous multimodal data corresponding to a study area, and performing data fusion processing on the heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area, wherein the heterogeneous multimodal data includes high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data, and land surface temperature data; Predicting the building plan information corresponding to the study area based on the multimodal fusion data using a pre-trained multimodal semantic segmentation network, wherein the multimodal semantic segmentation network includes a block layer, multiple Mix Transform encoding layers, a feature extraction layer, and a decoding layer; Predicting building height information corresponding to the study area based on the multimodal fusion data and the building plan information using a pre-trained height inversion network; Generating three-dimensional building information corresponding to the study area based on the building height information and the building plane information; Predicting the building plan information corresponding to the study area based on the multimodal fusion data using a pre-trained multimodal semantic segmentation network, including: segmenting the multimodal fusion data into a plurality of multimodal fusion sub-data through the block layer, and converting the multimodal fusion sub-data into a one-dimensional feature vector; The hierarchical feature vector corresponding to the one-dimensional feature vector is extracted through multiple MixTransform encoding layers; the hierarchical feature vector is subjected to downsampling feature extraction processing and upsampling feature fusion processing through the feature extraction layer to obtain a target feature vector; the target feature vector is decoded through the decoding layer to predict the building plan information corresponding to the study area.

2. The method for extracting three-dimensional building information based on multimodal data according to claim 1, wherein: Performing data fusion processing on the heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area includes: performing preprocessing, geometric registration processing, and resampling processing on the high-resolution multispectral data, the synthetic aperture radar data, the digital surface model data, the digital elevation model data, and the ground surface temperature data to obtain processed ground surface reflectivity, SAR backscatter coefficient data, the digital surface model data, the digital elevation model data, and the ground surface temperature data; The processed surface reflectivity, the SAR backscatter coefficient data and the surface temperature data are subjected to band combination processing, standardization processing and principal component analysis processing to obtain multimodal fusion data.

3. The method for extracting three-dimensional building information based on multimodal data according to claim 1, wherein: Predicting building height information corresponding to the study area based on the multimodal fusion data and the building plan information using a pre-trained height inversion network, including: Using the building plane information as a mask, extracting the multimodal fusion data corresponding to each building unit in the study area from the multimodal fusion data; For any of the building units, performing band mean processing on the multimodal fusion data corresponding to the building unit to obtain a multimodal value corresponding to the building unit; The height inversion network obtained through pre-training is used to predict the building height information corresponding to each building unit in the study area based on the multimodal value corresponding to each building unit.

4. The method for extracting three-dimensional building information based on multimodal data according to claim 1, wherein: Generating three-dimensional building information corresponding to the study area based on the building height information and the building plane information, including: For any building unit in the study area, the building height information corresponding to the building unit is written into the building plane information to generate building three-dimensional information corresponding to each building unit in the study area.

5. The method for extracting three-dimensional building information based on multimodal data according to claim 1, wherein: The method further comprises: constructing a multimodal semantic segmentation sample set based on the multimodal fusion data and building plane samples corresponding to the designated sub-area within the study area, and training the multimodal semantic segmentation network using the multimodal semantic segmentation sample set; Furthermore, a multimodal height inversion dataset is constructed based on the digital surface model data, the digital elevation model data, the multimodal fusion data and building plane samples corresponding to a designated sub-area within the study area, and the height inversion network is trained using the multimodal height inversion dataset.

6. The method for extracting three-dimensional building information based on multimodal data according to claim 5, characterized in that: Constructing a multimodal height inversion dataset based on the digital surface model data, the digital elevation model data, the multimodal fusion data, and building plane samples corresponding to a designated sub-area within the study area, including: Determining relative height information corresponding to the designated sub-area based on the digital surface model data and the digital elevation model data corresponding to the designated sub-area; Using the building plane sample as a mask, extracting the multimodal value and building height value corresponding to each building unit in the designated sub-area from the multimodal fusion data and the relative height information corresponding to the designated sub-area; The multimodal values ​​are used as model inputs and the building height values ​​are used as labels to obtain a multimodal height inversion dataset.

7. A building three-dimensional information extraction device based on multimodal data, characterized in that: include: a data processing module, configured to acquire heterogeneous multimodal data corresponding to a study area and perform data fusion processing on the heterogeneous multimodal data to obtain multimodal fusion data corresponding to the study area, wherein the heterogeneous multimodal data includes high-resolution multispectral data, synthetic aperture radar data, digital surface model data, digital elevation model data, and land surface temperature data; A plane information prediction module is configured to predict the plane information of buildings corresponding to the study area based on the multimodal fusion data using a pre-trained multimodal semantic segmentation network, wherein the multimodal semantic segmentation network includes a block layer, multiple Mix Transform encoding layers, a feature extraction layer, and a decoding layer; A height information prediction module is configured to predict the building height information corresponding to the study area based on the multimodal fusion data and the building plane information using a pre-trained height inversion network; A three-dimensional information generating module, configured to generate three-dimensional building information corresponding to the study area based on the building height information and the building plane information; The plane information prediction module is specifically used to: divide the multimodal fusion data into multiple multimodal fusion sub-data through the blocking layer, and convert the multimodal fusion sub-data into a one-dimensional feature vector; extract the hierarchical feature vector corresponding to the one-dimensional feature vector through multiple MixTransform encoding layers; perform downsampling feature extraction processing and upsampling feature fusion processing on the hierarchical feature vector through the feature extraction layer to obtain a target feature vector; and decode the target feature vector through the decoding layer to predict the building plane information corresponding to the study area.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 6.

Images