Building function identification method based on foreground guidance and dynamic fusion

Abstract

The invention discloses a building function identification method and device based on foreground guidance and dynamic fusion, a medium and equipment, and the method comprises the steps: inputting an enhanced satellite image and an enhanced street view image into a deep convolutional network for feature extraction, and obtaining a satellite view feature and a street view feature; cascading the satellite view features and the streetscape view features to generate enhancement factors; the enhancement factor acts on the streetscape view feature to obtain a streetscape enhancement feature, the similarity between the streetscape pooling feature vector and the satellite view feature is calculated based on dynamic learnable linear mapping to obtain a similarity mapping graph, and after the satellite view feature, the similarity mapping graph and the streetscape enhancement feature are cascaded, the streetscape enhancement feature is obtained. Generating a cross-view fusion feature by applying a variable convolution operation; and inputting the cross-view fusion features into a decoder, and outputting a prediction result of the social function of the building. Compared with the prior art, the method has the advantages that the recognition precision is obviously improved, and the recognition precision of the social function of the building can be improved by 0.7%-3.3%.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pattern recognition, and in particular to a building function recognition method and device based on foreground guidance and dynamic fusion, a medium and equipment. BACKGROUND

[0002] The social use of a building usually includes commercial, residential, industrial, public and other types, which is an important embodiment of urban functional zoning and social activities. Compared with traditional remote sensing interpretation focusing on the geometric features of buildings, function recognition expands the application range of interpretation results, and has important value in urban planning, disaster emergency management and smart city construction, which can provide dynamic and real-time data support for functional zoning, resource optimization and intelligent city operation. Therefore, how to accurately identify the social function of a building is a hot issue in the field of image analysis and application.

[0003] Satellite images mainly provide macro information such as shape, size and location of buildings, and lack of facade details related to social functions. Therefore, it is difficult to accurately distinguish the specific use of a building only by relying on satellite images. In order to make up for this deficiency, existing methods usually use street view images as a supplementary data source to use the detailed information such as building texture and facade identification provided by street view images to improve the accuracy of building function recognition. Existing technology one proposes a projection pooling layer capable of using image metadata based on the geometric relationship specific to the building scene. The pooling layer unifies the representation of satellite images and street view images in a high-dimensional space, and combines with a standard convolutional network to realize the prediction of building social functions. Existing technology two discloses an attention module capable of perceiving geographic spatial information, which is used to fuse satellite image features and street view image features. The module maps the latitude and longitude information into a direction distance feature map, and compresses it together with the street view image features into a feature vector containing rich geographic information. Finally, the feature vector with geographic spatial information is input into the decoder together with the original satellite image feature vector to complete the building function recognition.

[0004] There is a lot of non-building target information in street view images, and direct information interaction between street view image features and satellite image features is easy to introduce noise interference, affecting building function recognition. In addition, there is a big difference between the imaging angle and target scale of bird's eye view images and street view images, which is easy to cause mismatch of feature distribution in the cross-view fusion process, reducing the model accuracy. Therefore, the above methods still face certain limitations in practical application. SUMMARY

[0005] The main purpose of the present application is to provide a building function recognition method and device based on foreground guidance and dynamic fusion, a medium and equipment, which aims to effectively suppress street view image noise interference and promote the consistency of semantic representation under different angles.

[0006] To achieve the above object, the application provides a building function recognition method based on foreground guidance and dynamic fusion, comprising: performing data enhancement operation on input satellite images and street view images respectively; inputting the enhanced satellite images and the enhanced street view images into a parameter-independent deep convolutional network respectively for feature extraction to obtain satellite view features and street view features; concatenating the satellite view features and the street view features, and generating an enhancement factor through convolution operation and an activation function; applying the enhancement factor to the street view features to obtain street view enhanced features; performing average pooling operation on the street view enhanced features to obtain a street view pooled feature vector, calculating the similarity between the street view pooled feature vector and the satellite view features based on a dynamic learnable linear mapping to obtain a similarity mapping; concatenating the satellite view features, the similarity mapping and the street view enhanced features, and applying a variability convolution operation to generate a cross-view fusion feature; inputting the cross-view fusion feature into a decoder containing a convolution layer and an up-sampling layer to output a prediction result of building social function.

[0007] Optionally, when the network is in a training phase: a building social function label image is obtained; a cross-entropy loss between the building social function label image and the prediction result is calculated; the building social function label image is binarized to obtain a binarized label image; a foreground guidance loss function is calculated based on the binarized label image, and the foreground guidance loss function is used to constrain the learning process of the satellite view features; a total loss function is calculated according to the foreground guidance loss function and the weighted cross-entropy loss; and the parameters of the deep convolutional network are updated according to the total loss function.

[0008] Optionally, the foreground guidance loss function is calculated by the following formula:

[0009] L F =L BCE (g1(z1),Y F )

[0010] Wherein, L BCE is a binary cross-entropy function, g1 is a decoder composed of a convolution layer and an up-sampling layer, Y F is a foreground and background label image obtained by binarizing the building social function label image.

[0011] Optionally, the similarity mapping is calculated by the following formula:

[0012]

[0013] Wherein, A and B are matrices to be learned, b1 and b2 are offset to be learned, is a Hadamard product, w is a dynamic weight based on , and is the pooled street view enhanced features.

[0014] Optionally, the deep convolutional network is a pre-trained MSCAN-B network.

[0015] Optionally, the method further comprises: The obtained dynamic weight w comprises: processing the pooled street view enhanced feature by using a transformation network based on a three-layer convolutional network layer to obtain the dynamic weight w.

[0016] Optionally, the application of the variable convolution operation to generate the cross-view fusion feature comprises: learning a sampling point offset by using deformable convolution to dynamically adapt to shape range changes of the building target under different viewing angles to obtain the cross-view fusion feature.

[0017] To achieve the above object, the application further provides a building function recognition device combining satellite images and street view images, comprising: a data enhancement module configured to perform a data enhancement operation on input satellite images and street view images respectively; a feature extraction module configured to input the enhanced satellite images and the enhanced street view images into a parameter-independent deep convolutional network respectively to extract satellite view features and street view features; a feature enhancement module configured to cascade the satellite view features and the street view features, generate an enhancement factor by using a convolution operation and an activation function, and apply the enhancement factor to the street view features to obtain street view enhanced features; a similarity mapping module configured to perform an average pooling operation on the street view enhanced features to obtain a street view pooled feature vector, calculate a similarity between the street view pooled feature vector and the satellite view features based on a dynamically learnable linear mapping, and obtain a similarity mapping graph; a feature fusion module configured to cascade the satellite view features, the similarity mapping graph and the street view enhanced features, and apply a variable convolution operation to generate a cross-view fusion feature; and a prediction module configured to input the cross-view fusion feature into a decoder comprising a convolutional layer and an up-sampling layer, and output a prediction result of a building social function.

[0018] To achieve the above object, the application further provides a computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the building function recognition method based on foreground guidance and dynamic fusion provided in the above embodiments.

[0019] To achieve the above object, the application further provides an electronic device, comprising: at least one processor, a memory and an input-output unit; wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program stored in the memory to execute the building function recognition method based on foreground guidance and dynamic fusion provided in any of the preceding embodiments.

[0020] This application proposes a method, apparatus, medium, and device for building function recognition based on foreground guidance and dynamic fusion. The method involves performing data augmentation operations on input satellite images and street view images respectively; inputting the augmented satellite images and street view images into parameter-independent deep convolutional networks for feature extraction to obtain satellite view features and street view features; concatenating the satellite view features and street view features, and generating an enhancement factor through convolution operations and activation functions; applying the enhancement factor to the street view features to obtain enhanced street view features. This application utilizes foreground features of the satellite images to guide the learning of street view features, which helps suppress noise interference in the street view images; the street view enhancement... The features are subjected to average pooling to obtain street view pooled feature vectors. The similarity between the street view pooled feature vectors and satellite view features is calculated based on a dynamically learnable linear mapping to obtain a similarity map. By introducing dynamic similarity and spatial scale measures, it is beneficial to promote the consistency of semantic representations across different viewpoints. The satellite view features, similarity map, and street view enhanced features are concatenated, and a variable convolution operation is applied to generate cross-view fusion features. The cross-view fusion features are input into a decoder containing convolutional layers and upsampling layers to output the prediction results of the social functions of buildings. This application significantly improves the recognition accuracy, which can improve the recognition accuracy of the social functions of buildings by 0.7% to 3.3%. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an embodiment of the building function identification method based on foreground guidance and dynamic fusion provided in this application;

[0022] Figure 2 This is a visualization of the building social function prediction results of the present invention, based on an embodiment of the building function identification method based on prospect guidance and dynamic fusion.

[0023] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0025] Reference Figure 1 The building function identification method based on foreground guidance and dynamic fusion provided in the first embodiment of this application may include...

[0026] S10. Perform data augmentation operations on the input satellite image and street view image respectively.

[0027] Specifically, consistent data augmentation operations are applied to the input satellite image and the corresponding street view image to expand the data pattern, including scaling the image along random edges, scaling the image along specified edges, randomly flipping the image horizontally or vertically, and adjusting the image brightness.

[0028] S20. Input the enhanced satellite image and the enhanced street view image into a parameter-independent deep convolutional network for feature extraction to obtain satellite view features and street view features.

[0029] The enhanced satellite street view image pairs are used as input, and pre-trained deep convolutional networks are used to obtain satellite view features and street view features, respectively.

[0030] For example, the enhanced satellite-street view image pair (X1, X2) is used as input, and pre-trained deep convolutional networks f1(*; θ1) and f2(*; θ2) are used to extract features, respectively, to obtain independent view features (satellite view feature z1 and street view feature z2), i.e.:

[0031] z1=f1(X1;θ1),z2=f2(X2;θ2)

[0032] Where θ1 and θ2 represent network parameters. For the experiments involved in this application, the MSCAN-B network was selected as f1(*; θ1) and f2(*; θ2).

[0033] S30. After concatenating the satellite view features and street view features, an enhancement factor is generated through convolution operations and activation functions.

[0034] Specifically, satellite view features and street view features are concatenated to obtain cross-view concatenated features. Then, a street view feature enhancement factor is calculated using a sigmoid activation function and a convolutional layer with a kernel size of 1. This, combined with foreground-guided loss-optimized dynamic calculation of satellite view features and street view features, effectively guides the street view features to highlight the foreground region and suppresses noise interference. The enhancement factor is then applied to the street view features to obtain enhanced street view features.

[0035] It should be noted that if the network is in the training phase, the foreground guidance loss L is calculated as follows: F :

[0036] L F =L BCE (g1(z1),Y F )

[0037] Among them, L BCE It is a binary cross-entropy function, g1 is a decoder composed of convolutional layers and upsampling layers, Y FThis is a foreground and background map obtained by binarizing the social function map of the building. In L F Under the constraints, z1 can better model the differences between buildings and non-buildings in satellite images, thus providing accurate building foreground guidance for the optimization of z2. The deep convolutional network mentioned is a pre-trained MSCAN-B network.

[0038] Specifically, the processor calculates the street view feature enhancement factor α using the following expression:

[0039] α=σ(Φ(z c1 ))

[0040] Here, σ is the Sigmoid activation function, and Φ is a convolutional layer with a kernel size of 1. α is dynamically calculated by combining satellite view features optimized by foreground guidance loss with street view features, thereby effectively guiding street view features to highlight the foreground region and suppressing noise interference.

[0041] Next, the processor calculates the street view enhancement features according to the following formula.

[0042]

[0043] Reference Figure 1 (c) The content within the dashed box: The execution process of the similarity mapping module and feature fusion module in this application is shown in S40-S50:

[0044] S40. Perform average pooling on the street view enhanced features to obtain a street view pooled feature vector. Calculate the similarity between the street view pooled feature vector and the satellite view features based on a dynamically learnable linear mapping to obtain a similarity mapping map.

[0045] S50. After concatenating the satellite view features, similarity map, and street view enhancement features, apply variable convolution operation to generate cross-view fusion features.

[0046] For example, the processor enhances street view features The average pooling operation is applied to obtain the street view pooling features.

[0047]

[0048] The processor calculates the similarity map as follows:

[0049]

[0050] Where A and B are learnable matrices, b1 and b2 are learnable offsets, and ⊙ is the Hadamard product. w is based on The resulting dynamic weights are calculated as follows:

[0051]

[0052] Here, h is a transform network based on a three-layer convolutional network.

[0053] The processor concatenates satellite view features, computed similarity maps, and street view augmented features to obtain cross-view similarity guided features z. c2 .

[0054] The processor computes the cross-view fusion feature z:

[0055] z=τ(z c2 )

[0056] Here, τ represents the deformable convolution operation. Deformable convolution learns the sampling point offset and dynamically adapts to changes in the shape range of building targets from different viewpoints, thus effectively addressing the spatial scale differences between satellite imagery and street view imagery. Understandably, this method effectively addresses the spatial scale differences between satellite imagery and street view imagery.

[0057] Reference Figure 1 (d) The content within the dashed box: The implementation steps of the prediction module in this application are as follows:

[0058] S60. Input the cross-view fusion features into a decoder containing convolutional layers and upsampling layers, and output the prediction results of the social functions of the building.

[0059] Feature decoding is performed on the cross-view fusion feature z to obtain the prediction result of the building's social function.

[0060]

[0061] The decoder consists of convolutional layers and upsampling layers (independent of the g1 parameters).

[0062] In one embodiment of this application, when the network is in the training phase: a building social function annotation map is acquired; the cross-entropy loss between the building social function annotation map and the prediction result is calculated; the building social function annotation map is binarized to obtain a binarized annotation map; a foreground guidance loss function is calculated based on the binarized annotation map, and the learning process of satellite view features is constrained by the foreground guidance loss function; a total loss function is calculated based on the foreground guidance loss function and the weighted cross-entropy loss; and the parameters of the deep convolutional network are updated based on the total loss function.

[0063] For example, if the network is in the training phase, the total loss function L is calculated using the following formula, and the network parameters are updated:

[0064]

[0065] Among them, L Ce Y is the cross-entropy function, Y is the social function annotation map of buildings, and γ is the loss weighting factor.

[0066] To achieve the above objectives, this application also provides a building function recognition device combining satellite imagery and street view imagery, comprising: a data augmentation module for performing data augmentation operations on the input satellite imagery and street view imagery respectively; a feature extraction module for inputting the augmented satellite imagery and the augmented street view imagery into a parameter-independent deep convolutional network for feature extraction, obtaining satellite view features and street view features; a feature enhancement module for concatenating the satellite view features and street view features, generating an enhancement factor through convolution operations and activation functions; and applying the enhancement factor to the street view imagery. The system uses a graph feature module to obtain street view enhanced features; a similarity mapping module performs average pooling on the street view enhanced features to obtain a street view pooled feature vector, and calculates the similarity between the street view pooled feature vector and the satellite view features based on a dynamically learnable linear mapping to obtain a similarity mapping map; a feature fusion module concatenates the satellite view features, the similarity mapping map, and the street view enhanced features, and applies a variable convolution operation to generate cross-view fused features; a prediction module inputs the cross-view fused features into a decoder containing convolutional layers and upsampling layers, and outputs the prediction results of the social functions of buildings.

[0067] The data augmentation operations include scaling the image along random edges, scaling the image along specified edges, randomly flipping the image horizontally or vertically, and adjusting the image brightness. Feature extraction uses a pre-trained MSCAN-B network. In the feature augmentation module, street view feature enhancement factors are calculated using a sigmoid activation function and a convolutional layer with a kernel size of 1. The similarity mapping module calculates the similarity mapping map according to the formula, and obtains dynamic weights by processing the pooled street view enhancement features based on a transform network of three convolutional layers. The feature fusion module generates cross-view fusion features by learning the sampling point offset through deformable convolution.

[0068] The effectiveness of this application can be further illustrated by the following experimental results.

[0069] 1. Experimental Environment and Setup

[0070] The experimental hardware environment for this application is as follows: Intel(R) Xeon(R) CPU E5-2680 v4@2.40GHz, 128GB RAM, and RTX 3090 GPU.

[0071] The experimental software environment for this application is: Ubuntu operating system (18.04.4) and MMSegmentation algorithm framework (v1.2.0).

[0072] The data used in the experiment was expanded from a publicly available dataset. The street view images were returned by Baidu Maps' panoramic static image API, and the geographic location parameters required by the API corresponded to those of the satellite images. As shown in Table 1, due to access restrictions of the Baidu Maps API, this experiment ultimately constructed a dataset of social functions for individual buildings consisting of 356 pairs of satellite-street view image pairs, which was divided into training and test sets at an approximately 4:1 ratio. The building social function annotation map provides five categories of annotation information for individual buildings: residential buildings, commercial buildings, industrial buildings, public buildings, and other buildings.

[0073] Table 1. Building Function Identification Dataset

[0074]

[0075] 2. Evaluation Indicators

[0076] This experiment uses the mean Intersection over Union (mIoU), a commonly used evaluation metric in image segmentation tasks. mIoU represents the degree of overlap between the network predictions and the ground truth annotations; a higher mIoU value indicates higher accuracy in the segmentation results. It can be calculated using the following formula:

[0077]

[0078]

[0079] Where C is the number of categories, TP i It is the number of pixels correctly classified into that category, FP i It is the number of pixels misclassified into this category, FN i It represents the number of pixels that were missed in this category.

[0080] 3. Experiment Content

[0081] First, the network is trained using the total loss function L on the training set. Then, the trained model is used to make predictions on the test set, and the mIoU is calculated. To demonstrate the effectiveness of this application, we selected the geometric projection method proposed in the prior art as a comparison method, and the results are shown in Table 2. It can be seen that the mIoU value of this application is higher than all the comparison methods. This indicates that this application can achieve better building function recognition results, confirming the effectiveness of this application.

[0082] Table 2 Comparison of Experimental Results

[0083]

[0084]

[0085] Secondly, to visually verify the effectiveness of this application, we visualized the prediction results of each method, as shown in the following figures. Figure 2 As shown in the figure, the prediction results given in this application show that individual buildings have higher integrity and fewer misclassifications.

[0086] Finally, to evaluate the overall design rationality of this application, we conducted ablation experiments on the foreground guidance strategy used in the street view image feature enhancement module and the dynamic fusion strategy used in the cross-view feature fusion module. The experimental results are shown in Table 3. It can be seen that the mIoU value is improved after applying either strategy alone, which verifies the effectiveness of the strategies adopted in this application. After applying both strategies in combination, the mIoU value is further improved, indicating that the strategies in this application are mutually reinforcing, thus confirming the rationality of the design concept.

[0087] Table 3 Comparison of ablation test results

[0088] Method mloU (%) No other strategy 31.13 Only foreground guidance strategy 32.25 Only dynamic fusion strategy 33.03 The present application 35.83

[0089] To achieve the above objectives, this application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the building function recognition method based on foreground guidance and dynamic fusion provided in the above embodiments.

[0090] To achieve the above objectives, this application also provides an electronic device, the electronic device comprising: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to invoke the computer program stored in the memory to execute the building function recognition method based on foreground guidance and dynamic fusion provided in any of the foregoing embodiments.

[0091] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for building function recognition based on foreground guidance and dynamic fusion, characterized in that, include: Perform data augmentation operations on the input satellite images and street view images respectively; The enhanced satellite image and the enhanced street view image are respectively input into a parameter-independent deep convolutional network for feature extraction to obtain satellite view features and street view features; The satellite view features and street view features are concatenated and then an enhancement factor is generated through convolution operations and activation functions. The enhancement factor is applied to the street view features to obtain the street view enhancement features; The street view enhancement features are subjected to average pooling to obtain street view pooled feature vectors. The similarity between the street view pooled feature vectors and satellite view features is calculated based on a dynamically learnable linear mapping to obtain a similarity mapping map. After concatenating the satellite view features, similarity map, and street view enhancement features, a variable convolution operation is applied to generate cross-view fusion features; The cross-view fusion features are input into a decoder containing convolutional and upsampling layers, and the output is a prediction of the social function of the building.

2. The building function recognition method based on foreground guidance and dynamic fusion as described in claim 1, characterized in that, When the network is in the training phase: Obtain a map showing the social functions of buildings; Calculate the cross-entropy loss between the building social function annotation map and the prediction results; The social function annotation map of the building is binarized to obtain a binarized annotation map; The foreground guidance loss function is calculated based on the binarized labeled map, and the learning process of satellite view features is constrained by the foreground guidance loss function. The total loss function is calculated based on the prospect-guided loss function and the weighted cross-entropy loss. The parameters of the deep convolutional network are updated based on the total loss function.

3. The building function recognition method based on foreground guidance and dynamic fusion as described in claim 2, characterized in that, The foreground guidance loss function is calculated using the following formula: L F =L BCE (g1(z1),Y F ) Among them, L BCE It is a binary cross-entropy function, g1 is a decoder composed of convolutional layers and upsampling layers, Y F It is a foreground and background map obtained by binarizing the social function map of the building.

4. The building function recognition method based on foreground guidance and dynamic fusion as described in claim 1, characterized in that, The similarity mapping is calculated using the following formula: Where A and B are matrices to be learned, b1 and b2 are offsets to be learned, ⊙ is the Hadamard product, and w is based on... The obtained dynamic weights, It is a street view enhancement feature after pooling.

5. The building function recognition method based on foreground guidance and dynamic fusion as described in claim 1, characterized in that, The deep convolutional network is a pre-trained MSCAN-B network.

6. The building function recognition method based on foreground guidance and dynamic fusion as described in claim 1, characterized in that, based on The obtained dynamic weights w include: A transform network based on a three-layer convolutional network is used to process the pooled street scene enhancement features. The dynamic weight w is obtained.

7. The building function recognition method based on foreground guidance and dynamic fusion as described in claim 1, characterized in that, The application of variable convolution operations to generate cross-view fusion features includes: By learning the sampling point offset through deformable convolution, the shape range of the building target changes under different viewpoints, and cross-view fusion features are obtained.

8. A building function recognition device combining satellite imagery and street view imagery, characterized in that, include: The data augmentation module is used to perform data augmentation operations on the input satellite images and street view images respectively; The feature extraction module is used to input the enhanced satellite image and the enhanced street view image into a parameter-independent deep convolutional network for feature extraction, so as to obtain satellite view features and street view features. The feature enhancement module is used to concatenate the satellite view features and street view features, and then generate enhancement factors through convolution operations and activation functions; The enhancement factor is applied to the street view features to obtain the street view enhancement features; The similarity mapping module is used to perform average pooling on the street view enhanced features to obtain street view pooled feature vectors, and calculate the similarity between the street view pooled feature vectors and satellite view features based on a dynamically learnable linear mapping to obtain a similarity mapping map. The feature fusion module is used to concatenate the satellite view features, similarity map, and street view enhancement features, and then apply variable convolution operation to generate cross-view fused features. The prediction module is used to input the cross-view fusion features into a decoder containing convolutional layers and upsampling layers, and output the prediction results of the social functions of buildings.

9. A computer-readable storage medium, characterized in that, It includes instructions that, when executed on a computer, cause the computer to perform the building function recognition method based on foreground guidance and dynamic fusion as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes: At least one processor, memory, and input / output unit; The memory is used to store computer programs, and the processor is used to call the computer programs stored in the memory to execute the building function recognition method based on foreground guidance and dynamic fusion according to any one of claims 1 to 7.

Images