Multi-modal data fused building mixed purpose classification method and system

Abstract

The invention discloses a building mixed purpose classification method and system fused with multi-modal data, and belongs to the technical field of remote sensing image data processing, and the method comprises the steps: obtaining the multi-modal data of a research region, and carrying out the preprocessing of the multi-modal data; constructing a hybrid network model based on the convolutional neural network, wherein the hybrid network model comprises a parallel branch network, a buffer area module and a predictor which are connected in sequence; and inputting the preprocessed multi-modal data into the hybrid network model, extracting a multi-scale feature map of the multi-modal data through the parallel branch network, and extracting semantic information of a main environment feature region around a building in the multi-scale feature map through the buffer region module, and outputting the probability that the building belongs to each category through the predictor. According to the invention, the method achieves the recognition of the mixed purpose of the building based on the multi-modal data, can make up for the defects existing in the classification research of urban buildings only depending on single-modal data, and improves the recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a method and system for classifying mixed-use buildings by integrating multimodal data. Background Art

[0002] With the accelerated urbanization process in various regions, the number, scale and spatial density of urban buildings continue to grow. Accurately identifying the functions of urban buildings and detailed urban building distribution have become key links in urban planning. Existing studies on the functions of urban buildings usually assign a dominant category to each building, or even only classify buildings with a single function, for example, dividing buildings into five categories: residential, retail, office, industrial, and commercial, ignoring the actual mixed use, which limits the application value of such methods in modern urban planning.

[0003] However, mixed use is becoming increasingly important as a key factor in promoting smart urban growth. Mixed use refers to the coexistence of multiple functions in the same space, which is widely seen in single buildings. Obtaining mixed use information is important for evaluating the current state of urban planning and design and formulating future urban development strategies.

[0004] At present, there are few studies involving the classification of mixed-use buildings, and some studies introduce ground image data from street view images accessed through online map services (such as Open Street Map or Google Street View) when identifying the functions of individual buildings. These images provide a reference by including detailed features of building footprints and building fronts. However, it is difficult to obtain and process street view data. If the amount of street view data is to be guaranteed, it is necessary to use special equipment such as panoramic cameras to shoot in the target area. This data acquisition method is time-consuming and labor-intensive. However, the street view data publicly available on the Internet can only provide images of buildings near the road, and the amount of existing data is small, which leads to challenges in using street view data to process buildings far from the road. Remote sensing images have a wide imaging range, which can ensure the full capture of buildings and their contextual information. Therefore, sampling remote sensing images as a data source can not only ensure that the entire city is used as a study area, but also have a good temporal resolution.

[0005] However, remote sensing images may be confused when identifying buildings that are similar in appearance but different in function, so it is a trend to introduce multimodal data sources with distinguishing capabilities. For example, POI data can well reflect the spatial characteristics of the city's socio-economic and functional structure, so binding POI data with building footprints can determine the category of the building; night light data reflects the intensity and pattern of human activities, especially in urban environments. This data reflects the lighting pattern and brightness distribution caused by human activities at night, which not only provides information about the level of urban activities, but also provides the possibility of using night light data to classify urban buildings; synthetic aperture radar (SAR) data can provide unique echo characteristics, and can also reflect information such as building surface materials, roughness, and building height, which is also useful for distinguishing buildings of different uses; the geographic spatial feature data of building vectors, the geometric features of building vectors, and the distance from building vectors to road networks and building groups can also reflect the structure of the building itself and the relationship between the building and the spatial group to a certain extent, and thus reflect the category of the building itself.

[0006] Therefore, a new mixed-use classification method for buildings is needed, which can not only identify the mixed-use of buildings, but also make up for the defects of relying solely on single-modal data in the study of urban building classification, and can determine urban planning activities more comprehensively and accurately. Summary of the invention

[0007] In view of this, the present invention proposes a mixed-use building classification method and system integrating multimodal data, which is used to solve the problem that the existing urban building classification technology cannot accurately identify the mixed-use of buildings.

[0008] In a first aspect, the present invention discloses a method for classifying mixed-use buildings by integrating multimodal data, the method comprising: Acquire multimodal data of the study area respectively and perform preprocessing; the multimodal data includes remote sensing image data, POI data, SAR image data, night light image data and building vector data; Constructing a hybrid network model based on a convolutional neural network, wherein the hybrid network model includes a parallel branch network, a buffer module, and a predictor connected in sequence; The preprocessed multimodal data is input into the hybrid network model, a multiscale feature map of the multimodal data is extracted through the parallel branch network, semantic information of the main environmental feature area around the building in the multiscale feature map is extracted through the buffer module, and the probability of the building belonging to each category is output through the predictor.

[0009] On the basis of the above technical solution, preferably, the pretreatment includes: For remote sensing image data, the panchromatic band and multispectral band are fused and projected and transformed; For POI data, the kernel density distribution of various types of POI data is calculated by kernel density estimation method; For SAR image data and night light image data, projection transformation and resampling processing are performed; For building vector data, road network data is used as reference data to calculate the Euclidean distance of each building to the nearest traffic route, waterway and nearest neighboring building as the environmental attribute data of the building vector, and the maximum side length of the minimum enclosing rectangle of the building, the width of the minimum enclosing rectangle and the area of ​​the minimum enclosing rectangle are calculated as the shape attribute data of the building vector.

[0010] On the basis of the above technical solution, preferably, the parallel branch network includes a first branch network, a second branch network and a feature splicing module; The first branch network includes a residual network and a feature pyramid network connected in sequence, and is used to extract multi-scale semantic features of pre-processed remote sensing image data, SAR image data, and night light image data; The second branch network includes a CNN convolutional layer and an SE module connected in sequence, which is used to extract the environmental and geometric features of the preprocessed POI data and building vector data; the SE module is a self-attention mechanism, which multiplies the learned weights by the original feature map element by element, thereby adjusting the activation function value of each channel; The feature splicing module is used to splice the multi-scale semantic features with the environment and geometric features to generate a multi-scale feature map.

[0011] On the basis of the above technical solution, preferably, the buffer module is implemented based on the Mask R-CNN model, which is used to extract the environmental semantic features around the building based on the main environmental feature area, generate the bounding box of each object, and perform feature alignment.

[0012] On the basis of the above technical solution, preferably, the extraction of environmental semantic features around the building based on the main environmental feature area specifically includes: Determine the size of the main environmental feature areas in each direction of each building based on the size of the building; Move the main environmental feature area, take the center of the building as the middle, upper left corner, upper right corner, lower left corner and lower right corner of the main environmental feature area, perform multi-directional cropping on the periphery of the building according to the size of the main environmental feature area, and extract the environmental semantic features of each direction of the building according to the cropping results.

[0013] On the basis of the above technical solution, preferably, the coordinates of the main environmental feature areas in each direction are: Upper left corner position: (min x, min y), (min x+128(1.5×h), min y+128(1.5×h)); Upper right corner position: (max x- 128(1.5×h),min y), (max x, min y+ 128(1.5×h)); Lower left corner orientation: (min x, max y - 128 (1.5 × h)), (min x + 128 (1.5 × h), max y); Lower right corner position: (max x- 128(1.5×h), max y- 128(1.5×h)), (max x, max y); Among them, (x, y) is the coordinate of any point on the building, and h is the maximum side length of the minimum circumscribed rectangle of the building.

[0014] Based on the above technical solution, preferably, the predictor includes 3 fully connected layers for outputting the probability that each building belongs to each category.

[0015] In a second aspect of the present invention, a mixed-use building classification system integrating multimodal data is disclosed, the system comprising: Preprocessing module: used to obtain and preprocess the multimodal data of the study area respectively; the multimodal data includes remote sensing image data, POI data, SAR image data, night light image data and building vector data; Model building module: used to build a hybrid network model based on a convolutional neural network, wherein the hybrid network model includes a parallel branch network, a buffer module and a predictor connected in sequence; Building classification module: used to input the preprocessed multimodal data into the hybrid network model, extract the multi-scale feature map of the multimodal data through the parallel branch network, extract the semantic information of the main environmental feature area around the building in the multi-scale feature map through the buffer module, and output the probability of the building belonging to each category through the predictor.

[0016] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor, at least one memory, a communication interface and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to the first aspect of the present invention.

[0017] According to a fourth aspect of the present invention, a computer-readable storage medium is disclosed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method described in the first aspect of the present invention.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1) The present invention obtains multimodal data of the study area and constructs a hybrid network model including a buffer module and a predictor. The embedded semantic features of three modal data, namely, remote sensing image data, night light image data and SAR image data, are mined through one branch of the parallel branch network. At the same time, the POI data and the building vector data are extracted by another branch of the parallel branch network to extract the contextual environmental features of the building. Finally, the semantic information of the main environmental feature area around the building is obtained from the two types of features obtained above through the buffer module. Finally, the features of the main environmental feature area are aligned and put into the classifier to generate the category probability of each building, thereby realizing the recognition of mixed-use buildings. This can make up for the defects in the urban building classification research relying only on single-modal data and improve the recognition accuracy.

[0019] 2) The present invention designs a first branch network to deal with the difference problem of multimodal image data. By integrating the residual network model and the feature pyramid network module, the generalization ability of the model is enhanced, which can highlight important features and suppress irrelevant features, thereby improving the sensitivity to key information. It can also adaptively adjust the weights between channels to achieve attention allocation to different channel features.

[0020] 3) According to the characteristics of two types of frequency band data, building vector data and POI data, the present invention designs a simple and efficient second branch network. Through the second branch network with a simple structure but strong pertinence, the features of numerical frequency band data can be quickly extracted, which can avoid learning unnecessary features, reduce model complexity, and reduce the risk of overfitting.

[0021] 4) The present invention designs a buffer zone module, which extends to the four sides of the building to form the main environmental feature area around the building, and performs multi-directional cropping and feature alignment. It can not only fully consider the semantic information around different modal data, but also better consider the problem of different sizes of buildings of different types. At the same time, it can also solve the problem of imbalance in the number of buildings with different uses and the imbalance in the number of mixed and single category buildings to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flow chart of a method for classifying mixed-use urban buildings based on multimodal data fusion according to the present invention; Figure 2 It is a schematic diagram of the principle of the hybrid network model of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The present invention proposes a mixed-use classification method and system for urban buildings based on multimodal data fusion, aiming to more accurately quantify and analyze the characteristics of mixed use, thereby providing more scientific data support for urban planning.

[0026] See also Figure 1 The present invention discloses a mixed-use classification method for buildings by integrating multimodal data, the method comprising: S1. Obtain multimodal data of the study area and perform preprocessing.

[0027] In an embodiment of the present invention, remote sensing image data, POI data, SAR image data, night light image data and building vector data are respectively used as a modality and fused to form multimodal data. Multimodal data is used to make up for the defects of relying solely on single modal data in urban building classification research, so as to determine urban planning activities more comprehensively and accurately.

[0028] Specifically, the present invention obtains remote sensing image data, POI data, SAR image data, night light image data and building vector data respectively, and preprocesses the data of each modality respectively to facilitate input into a neural network model for feature extraction and classification.

[0029] (1) Remote sensing image data acquisition and preprocessing The GF-2 remote sensing image data of the study area was obtained on the National Remote Sensing Data and Application Service Platform. The images were preprocessed and stitched using ENVI to ensure that the final image could cover the study area. The preprocessing included radiometric calibration, atmospheric correction, orthorectification of multispectral images, radiometric calibration, orthorectification of panchromatic images, image fusion of multispectral images and panchromatic images, and projection transformation. Then, ENVI was used to stitch the multi-view images to cover the entire study area.

[0030] (2) POI data acquisition and preprocessing As an important information carrier reflecting the internal spatial structure, socio-economic dynamics and population distribution of a city, the POI (Point of Interest) dataset contains extensive and accurate geospatial information. POI data is obtained from existing map software such as Amap. Since the geographic coordinate system of the original Amap POI data is the Mars coordinate system, it is uniformly converted to the WGS-84 coordinate system. After that, the POI data after coordinate conversion is cleaned. Then the POI data is reclassified into three categories: residential land, commercial service facility land, and public management and public service land. Finally, the POI data containing three attributes of name, function and longitude and latitude are retained.

[0031] In order to overcome the problem of uneven distribution of POIs in different areas of the city, especially the limitations of high-density aggregation of POIs in commercial centers and urban core areas and sparse distribution of POIs in marginal areas, the present invention introduces an analysis method based on kernel density estimation (KDE). With a search radius of 500 meters and 1000 meters respectively, the kernel density distribution of various types of POI data is calculated by the kernel density analysis tool of ArcGIS Pro. This method considers both the impact of POIs on the building itself and its spatial expansion effect on the surrounding area. The dual-scale analysis of the kernel density map generated with a search radius of 500 meters and 1000 meters can enhance the grasp of the influence range and spatial distribution law of POIs, thereby providing richer contextual information support for the classification of building functions.

[0032] (3) SAR image data acquisition and preprocessing The SAR image data of the study area corresponding to the global mosaic 25m resolution SAR image dataset (PALSAR-2 / PALSAR global mosaic dataset) released by JAXA (Japan Aerospace Exploration Agency) were selected, and the SAR image data were projected and resampled on the ArcGIS Pro platform.

[0033] (4) Night light image data acquisition and preprocessing Download the low-light data released by the Sustainable Development Science Satellite-1 (SDGSAT-1). The low-light sensor of SDGSAT-1 can capture smaller ground features, provide more detailed information, and has a higher radiation resolution. Compared with other night light data, it can capture weak light signal changes. Perform projection transformation, resampling and other preprocessing on the ArcGIS Pro platform to obtain the night light image data of the corresponding study area.

[0034] (5) Building vector data acquisition and preprocessing In the embodiment of the present invention, the building vector data requires the use of road network (OSM, Open Street Map) data and building footprint data, so it is necessary to obtain the OSM data and the building footprint data.

[0035] According to the scope of the study area, OSM data related to the area are cut out. OSM data provides rich geographic data, including road networks, land use maps, waterways, and buildings. Among them, road networks and waterways are selected for subsequent building function classification. After obtaining OSM data, the internal roads and invalid roads of the plots are removed to determine the primary, secondary, and tertiary road networks. The roads are topologically checked. For roads with hanging points, the roads that were not initially connected are connected, and there is no overlap of plots. Finally, the road network layer is superimposed on the remote sensing image, which is subsequently used to obtain the vector spatial attribute data of the building based on the OSM data.

[0036] Select an open source building footprint dataset (e.g. https: / / doi.org / 10.5281 / zenodo.8174931), which provides the geographic location and boundary information of each building, laying a solid data foundation for subsequent processing. First, fine-tune the building footprint to better include information such as roads and environment around the building. Then, verify the data through Google Maps and Street View data to ensure correctness.

[0037] The data provided by OSM is used as reference data, and the Euclidean distance from each building to the nearest traffic route, waterway and nearest neighboring building is calculated through the ArcGIS Pro platform as the environmental attribute data of the building vector; the geometric feature data of the Building footprint data is calculated on the ArcGIS Pro platform, including the maximum side length of the minimum enclosing rectangle of the building, the width of the minimum enclosing rectangle and the area of ​​the minimum enclosing rectangle. The maximum side length, the width and the area of ​​the minimum enclosing rectangle of the building are used as the shape attribute data of the corresponding building vector and converted into raster data to form the corresponding geometric feature data raster map.

[0038] In the embodiment of the present invention, the functional category of the building is delineated according to the urban and rural land classification and planning and construction land standard (GB50137), and the sample annotation principle also adopts this standard. First, in the functional delineation, POI data is used as a reference, and the selection of POI is based on the building boundary (Building footprint) and the POI within the extension of 5 meters. If the number of commercial type POIs within the range exceeds 3, it is defined as a commercial building. For residential and public types with less POI distribution, satellite images of Gaode Map and a small number of POI points are combined for auxiliary judgment. According to the standard, the main functions of the building can be divided into residential, commercial, and public, and the building footprint is assigned by the geographical location and use information of the building. In addition, for mixed-use buildings, the present invention defines four types of mixed buildings: residential and commercial, residential and public, residential and commercial and public. If the Building footprint of a residential building has commercial POI points (such as retail stores) and the number of POIs exceeds 3, but the upper floor is residential, the building is defined as a commercial and residential mixed building, and the same applies to other mixed types.

[0039] Taking some urban areas of a city as an example, the number of buildings is as follows: 595 residential buildings, 466 commercial buildings, 430 public buildings, 273 residential and commercial mixed buildings, 118 residential and public mixed buildings, 35 residential, commercial and public mixed buildings, 69 commercial and public mixed buildings, and the total number of buildings is 1952. Due to the significant differences in the distribution of residential, commercial, public and mixed buildings, it shows that the sample data is obviously unbalanced, and this data imbalance is obviously present in the urban building group.

[0040] In order to ensure the diversity of samples, the present invention performs data clipping and data enhancement operations on all the above data. In terms of data clipping, according to the design concept of the main environmental feature area (PEFA) around the building, the size of the main environmental feature area in the four directions of each building is determined according to the size of the building, and the optimal environmental coverage range of each building is generated. The various types of data are clipped according to this range. For categories with a small amount of data, the main environmental feature area is moved, and the multi-source data is clipped from multiple angles with the building's material center in the middle, upper left corner, upper right corner, lower left corner and lower right corner of the main environmental feature area, respectively, to ensure that the environmental semantic features of the building are extracted from all directions. For categories with a large amount of data, in order to balance the number of samples, only the building is clipped in the middle of the main environmental feature area. During the clipping process, it is ensured that most of the generated images are 128×128 in size. For large buildings, the image size is set to 1.5h×1.5h. In terms of data enhancement, in order to avoid premature overfitting of the model and ensure sufficient data volume, a variety of enhancement methods were used, including horizontal flipping, vertical flipping, diagonal mirroring, 90° rotation, and 270° rotation.

[0041] After specific processing, the data volume of various types of buildings is as follows: 595 images were obtained after cropping of residential buildings, and a total of 4165 images were obtained after data enhancement; 1398 images were obtained after cropping of commercial buildings, and the data was enhanced to 9786 images; 860 images were obtained after cropping of public buildings, and the data was enhanced to 6020 images; 819 images were obtained after cropping of residential and commercial mixed buildings, and the data was enhanced to 5733 images; 590 images were obtained after cropping of residential and public mixed buildings, and the data was enhanced to 4130 images; 315 images were obtained after cropping of residential, commercial and public mixed buildings, and the data was enhanced to 2205 images; 621 images were obtained after cropping of commercial and public mixed buildings, and the data was enhanced to 4347 images. Through these data cropping and enhancement methods, the balance and sufficiency of various types of data are ensured, providing sufficient and diverse samples for model training.

[0042] It can be understood that, in addition to providing samples for model training based on multimodal data, multimodal data to be tested are also obtained and preprocessed the same as step S1.

[0043] S2. Construct a hybrid network model based on convolutional neural network.

[0044] In the embodiment of the present invention, a hybrid network model (MultiMixedNet) based on a convolutional neural network is constructed, such as Figure 2 As shown, the hybrid network model includes a parallel branch network, a buffer module and a predictor connected in sequence, and the parallel branch network includes a first branch network, a second branch network and a feature splicing module.

[0045] In an embodiment of the present invention, the first branch network includes a residual network (ResNet 50) and a feature pyramid network (FPN) connected in sequence, which are used to extract multi-scale semantic features of pre-processed remote sensing image data, SAR image data and night light image data.

[0046] Among them, ResNet 50 works as an encoder to obtain a feature map containing useful information, and the processing result of ResNet 50 is placed in FPN to help generate feature maps by considering objects of different scales. In an embodiment of the present invention, the first branch network generates feature maps of 4 scales. Its height, width and number of channels are 200×200×256, 100×100×512, 50×50×1024 and 25×25×2048 respectively.

[0047] In an embodiment of the present invention, the second branch network includes a CNN convolutional layer and a SE module connected in sequence, and the activation function adopts a Relu function to extract the environment and geometric features of the preprocessed POI data and building vector data.

[0048] The SE module is a self-attention mechanism that multiplies the learned weights by the original feature map element by element, thereby adjusting the activation function value of each channel so that the network can automatically learn how to reweight the importance of each channel. This mechanism enables the network to allocate more attention between different channels and enhances the network's attention to useful features.

[0049] In an embodiment of the present invention, the feature splicing module is used to splice the multi-scale semantic features output by the first branch network with the environment and geometric features output by the second branch network to generate a multi-scale feature map. That is, the tensor data output by the first branch network and the second branch network are spliced, and the splicing result is input into the buffer module.

[0050] In an embodiment of the present invention, the buffer zone module (BZ) is implemented based on the Mask R-CNN model, which is used to extract the environmental semantic features around the building based on the main environmental feature area, generate the bounding box of each object, and perform feature alignment.

[0051] The buffer module of the present invention can specifically obtain the semantic information of the main environmental feature area (PEFA) around the building from the large-scale semantic feature map. In the PEFA theory, the main environmental feature areas are opened with the center of mass of the building as the center and the size of the building as the reference. Generally speaking, the size of the main environmental feature area can be set to 128×128, but since the range of factors surrounding the building that need to be considered is related to the size of the building itself, such as the large size of a large shopping mall, a larger range of surrounding factors needs to be considered in the size processing. Therefore, 128 cannot be directly used as the side length range of the main environmental feature area. If the side length of the building itself is greater than 128, consider sampling 1.5×h as the side length of the main environmental feature area, where h represents the maximum side length of the minimum circumscribed rectangle of the building. In addition, 128 is used as the side length of the main environmental feature area to ensure the stability of the model.

[0052] Specifically, let (x, y) be the coordinates of any point on the building shp, take the upper left corner of the building (min x, min y) as the base point, and determine the size of the main environmental feature area in each direction of each building in combination with the size of the building. The coordinates of the main environmental feature area in each direction are: Upper left corner position: (min x, min y), (min x+128(1.5×h), min y+128(1.5×h)); Upper right corner position: (max x- 128(1.5×h),min y), (max x, min y+ 128(1.5×h)); Lower left corner orientation: (min x, max y - 128 (1.5 × h)), (min x + 128 (1.5 × h), max y); Lower right corner position: (max x- 128(1.5×h), max y- 128(1.5×h)), (max x, max y); Where h is the maximum side length of the minimum circumscribed rectangle of the building.

[0053] Then, the main environmental feature area is moved, with the center of the building as the middle, upper left corner, upper right corner, lower left corner and lower right corner of the main environmental feature area respectively, and the surrounding area of ​​the building is cropped in multiple directions according to the size of the main environmental feature area, and the environmental semantic features of each direction of the building are extracted according to the cropping results.

[0054] The present invention specifically obtains environmental semantic features of a principal environmental feature area (PEFA) around a building from a large-scale semantic feature map, performs feature alignment on the environmental semantic features of the principal environmental feature area, and inputs the features into a predictor.

[0055] In an embodiment of the present invention, the predictor includes 3 fully connected layers for outputting the probability that each building belongs to each category.

[0056] Specifically, the object-level features of each individual building extracted from the buffer module block are used to estimate the category of the building, that is, after obtaining the features of each building and passing through 3 fully connected layers, the estimated probability of each building category is output.

[0057] In the embodiment of the present invention, a cross entropy loss function (CrossEntropyLoss) is used, and the estimated probability of each building category is output through the LogSoftmax operation of the CrossEntropyLoss function.

[0058] In an embodiment of the present invention, the hybrid network model is trained based on the multi-source data in step S1.

[0059] Specifically, data fusion is performed on GF-2 remote sensing image data, SAR image data, and night light image data to generate cube data with a dimension of 8, and data fusion is performed on POI data and building vector data to generate cube data with a dimension of 9. All cube data are input into the MultiMixedNet model for model training. In a certain embodiment of the present invention, the height, width, and number of channels of the input data are 128, 128, and 17, respectively, and the input data is cube data with 17 channels. Among them, the cube data with a dimension of 8 is input into the first branch network of the MultiMixedNet model to generate feature maps of four sizes, and obtain feature maps with heights and widths of 200×200, 100×100, 50×50, and 25×25, respectively. The cube data with a dimension of 9 is input into the second branch network of the MultiMixedNet model, and feature maps of four sizes are also generated, and feature maps with heights and widths of 200×200, 100×100, 50×50, and 25×25, respectively, are obtained.

[0060] For the feature maps output by the first branch network and the second branch network, the feature maps of the same size are grouped together for data splicing and input into the buffer module. The buffer module obtains the x and y coordinates of the buildings with assigned labels, and obtains the corresponding optimal environmental coverage range of the buildings based on each pair of x and y coordinates. By recording the x and y coordinates of the optimal environmental coverage range, the model focuses on each building and its surroundings. Therefore, the buffer module will collect the optimal range of buildings cropped with different buildings as the center. For example, the data set size cropped with building A as the center is 128×128. If building B is included in this size, according to the BZ module, the feature maps of buildings A and B will be obtained at the same time. Finally, the feature maps corresponding to the optimal environmental range coverage of each building were obtained and aligned, resulting in 31,581 feature maps of residential buildings, 29,878 feature maps of commercial buildings, 19,147 feature maps of public buildings, 17,116 feature maps of residential and commercial mixed type buildings, and 21,527 feature maps of residential and public mixed type buildings. There are 8,371 feature maps of residential, commercial and public mixed type buildings and 13,684 feature maps of commercial and public mixed type buildings.

[0061] Finally, the features extracted by the buffer module are aligned, and the aligned tensor values ​​are input to the predictor. Before input, the feature map is processed to obtain a one-dimensional tensor with a shape of 1×144256. The shape of the input tensor is then converted from 1×144256 to 1×1024 through the first fully connected layer. The second fully connected layer does not change the shape of the tensor, but only deepens the network. The ReLU layer is followed by the two fully connected layers. The last fully connected layer will output a value representing the predicted category of the selected building. For each building feature map, the predictor outputs the probability of each category as the prediction result. The output set of the predictor contains the prediction results for each category of each individual building, and finally the probability of the building belonging to each category is obtained.

[0062] S3. Input the preprocessed multimodal data into the hybrid network model, and output the probability that the building belongs to each category.

[0063] The multimodal data to be tested in step S1 is obtained and preprocessed, and the mixed-use identification of the building is performed through the trained hybrid network model.

[0064] In an embodiment of the present invention, a multi-scale feature map of multimodal data is extracted through the parallel branch network of the hybrid network model, semantic information of the main environmental feature area around the building in the multi-scale feature map is extracted through the buffer module, and the probability of the building belonging to each category is output by the predictor to obtain a mixed-use recognition result of the building.

[0065] The mixed-use classification method of urban buildings by integrating multimodal data proposed in the present invention makes full use of GF-2 remote sensing image data, POI data, SAR image data, SDGSAT-1 night light image data, and building vector geographic spatial feature data. Through the MultiMixedNet model, it can learn features at multiple scales and in multiple directions, and can also weight the importance of each channel to ensure that different modal data can be used for certain learning.

[0066] Among them, the use of POI kernel density map not only considers the impact of points of interest on individual buildings, but also takes into account the impact of social perception data around the buildings; taking night light data as one of the multimodal data sources can provide information about the level of urban activities, and can also reveal the brightness distribution of different types of buildings at night. These differences provide more information for using night light data to classify urban buildings; taking SAR data as one of the multimodal data sources can provide the echo characteristics of buildings, which can be used to distinguish the surface materials and roughness of buildings, and then distinguish different types of buildings; taking the geospatial feature data of building vectors as one of the multimodal data sources can reflect the coverage density of buildings, the distance of buildings from roads and waters, and indirectly reflect the information of buildings and the surrounding environment. At the same time, it can deepen the extraction of building information and provide a reference for building classification.

[0067] Therefore, selecting multiple modal data for urban building classification can improve the accuracy of building classification, deepen the understanding of building information, optimize urban resource allocation, and achieve further development of urban planning.

[0068] In addition, the buffer module of the present invention can fully obtain the semantic information of the surrounding environment of the building and solve the problem of imbalance of building categories, so that when the model faces building categories with multiple differences in number, it can ensure that the model can pay attention to a small number of building categories, thereby further improving the recognition accuracy.

[0069] Corresponding to the above method embodiment, the present invention also discloses a building mixed-use classification system integrating multimodal data, the system comprising: Preprocessing module: used to obtain and preprocess the multimodal data of the study area respectively; the multimodal data includes remote sensing image data, POI data, SAR image data, night light image data and building vector data; Model building module: used to build a hybrid network model based on a convolutional neural network, wherein the hybrid network model includes a parallel branch network, a buffer module and a predictor connected in sequence; Building classification module: used to input the preprocessed multimodal data into the hybrid network model, extract the multi-scale feature map of the multimodal data through the parallel branch network, extract the semantic information of the main environmental feature area around the building in the multi-scale feature map through the buffer module, and output the probability of the building belonging to each category through the predictor.

[0070] The above system embodiments and method embodiments correspond one to one, and for a brief description of the system embodiments, please refer to the method embodiments.

[0071] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the aforementioned method of the present invention.

[0072] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to implement all or part of the steps of the method described in the embodiment of the present invention. The storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0073] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be distributed to multiple network units. A person skilled in the art may select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment without creative effort.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A mixed-use building classification method integrating multimodal data, characterized in that: The method comprises: Acquire multimodal data of the study area respectively and perform preprocessing; the multimodal data includes remote sensing image data, POI data, SAR image data, night light image data and building vector data; Constructing a hybrid network model based on a convolutional neural network, wherein the hybrid network model includes a parallel branch network, a buffer module, and a predictor connected in sequence; The preprocessed multimodal data is input into the hybrid network model, a multiscale feature map of the multimodal data is extracted through the parallel branch network, semantic information of the main environmental feature area around the building in the multiscale feature map is extracted through the buffer module, and the probability of the building belonging to each category is output through the predictor.

2. The mixed-use classification method of buildings based on multimodal data fusion according to claim 1, characterized in that: The pre-processing comprises: For remote sensing image data, the panchromatic band and multispectral band are fused and projected and transformed; For POI data, the kernel density distribution of various types of POI data is calculated by kernel density estimation method; For SAR image data and night light image data, projection transformation and resampling processing are performed; For building vector data, road network data is used as reference data to calculate the Euclidean distance of each building to the nearest traffic route, waterway and nearest neighboring building as the environmental attribute data of the building vector, and the maximum side length of the minimum enclosing rectangle of the building, the width of the minimum enclosing rectangle and the area of ​​the minimum enclosing rectangle are calculated as the shape attribute data of the building vector.

3. The mixed-use classification method of buildings based on multimodal data fusion according to claim 1, characterized in that: The parallel branch network includes a first branch network, a second branch network and a feature splicing module; The first branch network includes a residual network and a feature pyramid network connected in sequence, and is used to extract multi-scale semantic features of pre-processed remote sensing image data, SAR image data, and night light image data; The second branch network includes a CNN convolutional layer and a SE module connected in sequence, and is used to extract the environment and geometric features of the preprocessed POI data and building vector data; The SE module is a self-attention mechanism that multiplies the learned weights by the original feature map element by element to adjust the activation function value of each channel; The feature splicing module is used to splice the multi-scale semantic features with the environment and geometric features to generate a multi-scale feature map.

4. The mixed-use classification method of buildings based on multimodal data fusion according to claim 1, characterized in that: The buffer module is implemented based on the Mask R-CNN model, which is used to extract the environmental semantic features around the building based on the main environmental feature areas, generate the bounding box of each object, and perform feature alignment.

5. The mixed-use classification method of buildings based on multimodal data fusion according to claim 4, characterized in that: The extraction of environmental semantic features around the building based on the main environmental feature area specifically includes: Determine the size of the main environmental feature areas in each direction of each building based on the size of the building; Move the main environmental feature area, take the center of the building as the middle, upper left corner, upper right corner, lower left corner and lower right corner of the main environmental feature area, perform multi-directional cropping on the periphery of the building according to the size of the main environmental feature area, and extract the environmental semantic features of each direction of the building according to the cropping results.

6. The mixed-use classification method of buildings based on multimodal data fusion according to claim 5, characterized in that: The coordinates of the main environmental feature areas in each direction are: Upper left corner position: (min x, min y), (min x+128(1.5×h), min y+128(1.5×h)); Upper right corner position: (max x- 128(1.5×h),min y), (max x, min y+ 128(1.5×h)); Lower left corner orientation: (min x, max y - 128 (1.5 × h)), (min x + 128 (1.5 × h), max y); Lower right corner position: (max x- 128(1.5×h), max y- 128(1.5×h)), (max x, max y); Among them, (x, y) is the coordinate of any point on the building, and h is the maximum side length of the minimum circumscribed rectangle of the building.

7. The mixed-use classification method of buildings by fusing multimodal data according to claim 1, characterized in that: The predictor consists of 3 fully connected layers that output the probability of each building belonging to each category.

8. A mixed-use building classification system integrating multimodal data, characterized in that: The system comprises: Preprocessing module: used to obtain and preprocess the multimodal data of the study area respectively; the multimodal data includes remote sensing image data, POI data, SAR image data, night light image data and building vector data; Model building module: used to build a hybrid network model based on a convolutional neural network, wherein the hybrid network model includes a parallel branch network, a buffer module and a predictor connected in sequence; Building classification module: used to input the preprocessed multimodal data into the hybrid network model, extract the multi-scale feature map of the multimodal data through the parallel branch network, extract the semantic information of the main environmental feature area around the building in the multi-scale feature map through the buffer module, and output the probability of the building belonging to each category through the predictor.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method according to any one of claims 1 to 7.

Images