Building semantic change detection method for unmanned aerial vehicle image

Abstract

The invention relates to the improvement of an unmanned aerial vehicle image detection technology, in particular to a building semantic change detection method aiming at an unmanned aerial vehicle image, which can simultaneously capture two types of key information of position change and semantic change of a building on a multi-temporal image. Reliable technical support can be provided for urban illegal building monitoring, urban update drawing and the like; comprising three parts: sample data acquisition, multi-task collaborative learning building change detection network design, and model training and reasoning deployment. Simultaneous output of semantic information such as building addition and reduction can be realized, and the business requirements of urban illegal building monitoring can be better met; the change detection model designed by the invention can better support a domestic hardware platform, and can realize autonomous controllability.

Description

Technical Field

[0001] The present invention relates to an improvement in drone image detection technology, and in particular to a method for detecting semantic changes of buildings using drone images. Background Art

[0002] The use of multi-temporal satellite remote sensing images to detect changes in building locations and structures is widely used in the fields of national land space planning and illegal building monitoring. The main advantage of using satellite remote sensing images to detect building changes is its wide bandwidth and low cost. However, satellite remote sensing images have objective factors such as insufficient resolution and uncontrollable monitoring cycles in urban illegal building monitoring. Therefore, the use of drone images (hereinafter referred to as UAV images) to detect building changes has become a common means of business.

[0003] At present, the detection of building changes in business areas based on multi-temporal drone images is still highly dependent on manual visual interpretation. In the overall technical process, the UAV images collected with spatial location information are firstly spliced ​​into a large picture through color unification, image mosaicking and other means through special software; then the visual interpreters, with the assistance of spatial software such as ArcGIS or QGIS, complete the detection of building position and structural changes in two or more phases through rolling curtain comparison; finally, the outlined vector results are output. The above method has a good effect in small areas or short time scales. However, for large-scale application scenarios or long monitoring cycles, there are disadvantages of huge costs and long time.

[0004] To solve the above problems, automatic building change detection methods have become a hot research topic in the field of engineering applications; UAV image-based building change detection can be divided into two categories: traditional machine learning methods and deep learning methods.

[0005] Traditional machine learning methods usually use support vector machines, random forests and other methods to identify building changes in multi-temporal UAV images by modeling on spectral features and texture features. This type of method is relatively simple, but it has problems of low accuracy and poor generalization ability for complex scenes, especially when the trained model is migrated to different areas, the model accuracy drops sharply, which has great limitations for engineering applications.

[0006] Deep learning method is the most widely used method in recent years. This method can obtain a more robust general building change detection model by designing a reasonable deep learning model with the assistance of a large number of labeled samples and combining model parameter training and tuning. At present, there have been many studies on building change detection based on deep learning methods. Most of the studies have made a lot of improvements in the model structure, and a series of excellent building change detection models have been born, such as Unet++, DeepLabV3+, ChangeNet, ChangeFormer, SiamHRNet, etc.

[0007] However, most existing deep learning building change detection methods can only detect the location changes of buildings in two-phase images, but rarely take into account the semantic changes of buildings, such as the inability to detect the addition and removal of buildings. The semantic information of the addition and removal of buildings is of great value for building violation detection, and can help regulatory agencies to understand the changes of buildings in a timely manner.

[0008] Therefore, improving the method of developing semantic changes in buildings is a technical problem that needs to be solved. Summary of the invention

[0009] In response to the problems mentioned in the technical background, the purpose of this patent is to provide a method for detecting semantic changes in buildings based on drone images. This method can simultaneously capture two types of key information: position changes and semantic changes of buildings in multi-temporal images, and can provide reliable technical support for monitoring illegal buildings in cities.

[0010] The technical solution of the present invention is as follows: a method for detecting building semantic changes in drone images, comprising three parts: sample data collection, multi-task collaborative learning building change detection network design, model training and inference deployment;

[0011] Sample data collection uses real-time shooting and historical image collection methods to construct sample data to assist in subsequent model training;

[0012] The design of the multi-task collaborative learning building change detection network is divided into three parts, including: input part, semantic feature extraction part and multi-task learning head;

[0013] Input part: The semantic combination of two phase images is realized by channel superposition for subsequent model input;

[0014] Semantic feature extraction part: Based on the encoding-decoding framework, multi-scale semantic feature extraction of buildings in UAV images is realized;

[0015] Multi-task learning head: Three outputs are used to realize multi-task semantic learning of buildings, namely, building new output detection head, building reduced output detection head and building change detection head;

[0016] Model training and inference deployment: During the training phase, any two phases of UAV images are input through a multi-task collaborative learning change detection network to generate three output heads, which are then cross-entropy loss calculated with the true value of the label, and automatic gradient backpropagation is completed to achieve model parameter update.

[0017] As a preference, in the inference deployment phase, open source model conversion technology is used to convert the model into the general format of ONNX.

[0018] The beneficial effects of the seat of the present invention are:

[0019] 1) The present invention designs a multi-task learning building semantic change model, which can realize the simultaneous output of semantic information such as new additions and reductions of buildings, and can better meet the business needs of urban illegal building monitoring.

[0020] 2) The change detection model designed by the present invention can better support domestic hardware platforms and can achieve independent control. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the multi-task collaborative learning building semantic change detection model structure;

[0022] Figure 2 This is a structural diagram of the semantic feature extraction part. Specific implementation methods

[0024] The present invention provides a method for detecting building semantic changes in drone images, and a specific technical solution includes three parts: sample data collection, multi-task collaborative learning building change detection network design, and model training and inference deployment.

[0025] Sample data collection: Whether the sample data is accurately labeled and whether the quantity is sufficient will have a significant impact on the accuracy of the model. In order to collect more high-precision multi-temporal UAV image building change samples, real-time shooting, historical image collection and other methods are used to construct a set of high-quality sample data to assist in subsequent model training. Specifically, semi-automatic human-computer interaction is used to quickly construct sample data, such as using professional software such as ArcGIS and QGIS, combined with the Segment Anything Model to quickly label the sample set. After the labeling is completed, quality inspection and positive and negative sample ratio calculation are performed uniformly. In order to achieve the optimal accuracy of the model, the ratio of positive and negative samples is as close to 1:1 as possible.

[0026] Design of multi-task collaborative learning building change detection network: Based on the open source deep learning framework and referring to the existing change detection network, a high-precision multi-task collaborative learning building monitoring network is designed. The network is divided into three parts, including: input part, semantic feature extraction part, and multi-task learning head. In the input part, the semantic combination of two phase images is realized by channel superposition to facilitate the subsequent model input; in the semantic feature extraction part, the focus is on the encoding-decoding framework to realize the multi-scale semantic feature extraction of buildings on UAV images; in the multi-task learning head part, three outputs are used to realize the multi-task semantic learning of buildings, namely the building new output detection head, the building reduction output detection head, and the building change detection head; for the convenience of training, the image sizes of the above three output detection heads are all upsampled to the original size. Figure 1 The size of the sample is used to facilitate the unification of loss calculation and model output.

[0027] Model training and inference deployment: In the training phase, any two phases of UAV images are input through a multi-task collaborative learning change detection network to generate three output heads, which are then cross-entropy loss calculated with the true value of the label, and automatic gradient return is completed to achieve model parameter update. In the inference deployment phase, open source model conversion technology is used to convert the model into a general format such as ONNX to meet the deployment support of different devices, especially compatible with domestic GPU devices, to achieve independent control.

[0028] In the specific implementation, the designed multi-task collaborative learning building semantic change detection model structure is as follows Figure 1 As shown; in the input part, the information combination of the two-phase UAV images is completed by band superposition. Generally speaking, drone images are usually three-band visible light, mainly RGB three-color channels. After band superposition, the two-phase UAV images are combined into a 6-channel image as the input part of the model; it is worth mentioning that the present invention is not limited to three channels of visible light, and it can also be very convenient to adapt on the drone multispectral camera. For example, if the input image contains a near-infrared channel, then you only need to set the number of input bands to 4, and combine the two phase images into 8 channels to complete the adaptation. Images of other different channels are similarly completed to unify the format of the input end.

[0029] In the semantic feature extraction part, this module mainly completes the multi-scale high-order semantic feature extraction of the input two-phase UAV image. Its specific structure can be found in Figure 2As shown in the figure, the two input phase images are respectively passed through multiple superimposed convolution modules (A and B in the figure represent different levels of convolution module groups), and then through dense connection, the learning of cross-scale semantic features is realized. In the output part of the semantic feature module, a channel attention module (CAM) is connected. The main function of this module is to learn different features of high-order semantic features and realize more effective semantic feature output; the specific structure of the channel attention module is shown in Figure 2 As shown in the figure, after the input two-phase images pass through the semantic feature extraction module, high-order semantic information of different scales will be obtained. This feature information can be used as the input of the channel attention module. The channel attention uses maximum pooling and average pooling to extract semantic information at different levels, and then uses a multi-layer perceptron to realize fine-grained semantic feature extraction. Finally, the multi-level semantic information is aggregated to obtain more effective channel semantic information to improve the subsequent segmentation accuracy.

[0030] In the multi-task learning head part, it is divided into three outputs, namely, building reduction detection head, building change detection head, and building addition detection head. Among them, the building change detection head is the sum of the building reduction and building addition heads, and its main function is to realize the cross-semantic learning of two semantic features. The three output heads perform cross entropy Loss calculation with the true value of the label respectively to realize the gradient update of the model parameters and complete the final model training; in the specific Loss calculation, considering the imbalance problem of positive and negative samples, the present invention adopts weighted cross entropy to realize Loss calculation, and the Loss function of the three output heads can adaptively adjust the weights according to the positive and negative sample ratio of the input samples, so as to realize dynamic weighted cross entropy calculation.

[0031] In the final model deployment stage, the universal ONNX format conversion method is used to achieve unified model format conversion. In this format, the model can well support the friendly support and efficient reasoning of domestic GPU cards; it is worth mentioning that the ONNX format currently provides good support for computing hardware of different architectures such as GPU, CPU, and NPU. For example, the ONNX format model has been fully supported on the domestic Cambrian computing card, and can seamlessly migrate related software algorithms to domestic devices, achieving independent control of software and hardware to avoid problems.

[0032] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting building semantic changes based on drone images, characterized in that: It includes three parts: sample data collection, multi-task collaborative learning building change detection network design, model training and inference deployment; Sample data collection uses real-time shooting and historical image collection methods to construct sample data to assist in subsequent model training; The design of the multi-task collaborative learning building change detection network is divided into three parts, including: input part, semantic feature extraction part and multi-task learning head; Input part: The semantic combination of two phase images is realized by channel superposition for subsequent model input; Semantic feature extraction part: Based on the encoding-decoding framework, multi-scale semantic feature extraction of buildings in UAV images is realized; Multi-task learning head: Three outputs are used to realize multi-task semantic learning of buildings, namely, building new output detection head, building reduced output detection head and building change detection head; Model training and inference deployment: In the training phase, any two phase UAV images input are transformed into a multi-task collaborative learning change detection network to generate three output heads, which are then cross-entropy loss calculated with the true label value, and automatic gradient backpropagation is completed to update the model parameters. In the deployment phase, it supports general format conversion and domestic equipment deployment.

2. The method for detecting building semantic changes based on drone images as claimed in claim 1, characterized in that: During the inference deployment phase, open source model conversion technology is used to convert the model into the general format of ONNX.

3. The building semantic change detection method for drone images as claimed in claim 1, characterized in that: In the input part, the information combination of the two-phase UAV images is completed by band superposition; UAV images are usually three-band visible light, mainly RGB three color channels. After band superposition, the two-phase UAV images are combined into a 6-channel image as the input part of the model; for other multispectral images, it is also only necessary to simply modify the input channel settings to complete the adaptation of the multi-channel image.

4. The building semantic change detection method for drone images according to claim 1, characterized in that: In the semantic feature extraction part, multi-scale high-order semantic feature extraction is completed for the input two-phase UAV images; the two input phase images are respectively passed through multiple superimposed convolution modules, and then through dense connection to achieve cross-scale semantic feature learning.

5. The building semantic change detection method for drone images according to claim 1, characterized in that: The output part of the semantic feature module is connected to a channel attention module, which learns different features of high-order semantic features to achieve effective semantic feature output.

6. The building semantic change detection method for drone images according to claim 1, characterized in that: In the multi-task learning head part, it is divided into three outputs, namely building reduction detection head, building change detection head, and building addition detection head; among them, the building change detection head is the addition of the building reduction and building addition heads, realizing cross-semantic learning of two semantic features.

7. The building semantic change detection method for drone images according to claim 1, characterized in that: The three output heads perform cross entropy loss calculations with the true label values ​​to achieve gradient updates of the model parameters and complete the final model training.

Images