Multi-source Remote Sensing Image Scene Recognition Method Based on Deformable Convolution Selection and Allocation Network

Abstract

The present invention discloses a method for multi-source remote sensing image scene recognition based on a deformable convolutional selection and allocation network. Deformable convolutions are introduced into the feature extraction basic network of the YOLOv3 network to extract and retain the spatial structure information of scene representation with multiple scales and shapes. A selective allocation and fusion module is introduced on the multi-source feature pyramid to fuse multi-source remote sensing images, and different fusion weights are assigned to different image features. The present invention can better extract the spatial information of various scenes in the image through deformable convolutions, and combine the advantages of multi-source remote sensing data through the fusion module to achieve a relatively high-precision scene recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of multi-source remote sensing image scene recognition, and in particular to a multi-source remote sensing image scene recognition method based on a deformable convolutional selection allocation network. Background Art

[0002] With the continuous development of remote sensing in my country, more and more remote sensing data has become publicly available. Using remote sensing data to perform scene recognition within a specific area has become an important research direction in the field. Its purpose is to effectively identify specific scenes in a given image, accurately locate them, and effectively classify them into the scene category they belong to. This work is of great significance for guiding production and life, preventing illegal expansion or mineral mining, and providing decision-making support for relevant units.

[0003] In remote sensing image scene recognition tasks, due to intra-class variations between different scenes in complex environments and the observation limitations of specific sensors under extreme conditions, using a single type of remote sensing imagery, such as hyperspectral imagery, multispectral imagery, or optical imagery, has proven difficult to meet the growing accuracy requirements in practical applications. Therefore, the use of multi-source remote sensing imagery for scene recognition tasks has gradually attracted the attention of more researchers. In recent years, the emergence and development of deep learning has provided new development models and methods for various industries. Convolutional neural networks, due to their powerful learning capabilities and generalization, have been widely used in various research fields. However, due to the fixed shape and size of the convolution kernel, their performance in handling various complex and changing scenes in practical tasks is somewhat insufficient.

[0004] In addition, due to the differences in the observation advantages of multi-source remote sensing images, how to more effectively utilize the respective advantages of multi-source remote sensing images and achieve complementary advantages has also become an issue that must be considered in the multi-source remote sensing image scene recognition task. Summary of the Invention

[0005] Aiming at the problems of existing methods, the present invention fully considers the characteristics of different scenes in the multi-source remote sensing scene recognition task, the model's processing ability for scene representations of different scales and shapes, and the different levels of attention that the model should pay to heterogeneous remote sensing images. Based on the YOLOv3 model, a multi-source remote sensing image scene recognition method based on a deformable convolutional selective allocation network is proposed. The method uses deformable convolution to process multi-scale and multi-shape scene representations, and effectively fuses multi-source remote sensing images through a selective allocation module.

[0006] In order to achieve the above object of the invention, the technical solution adopted by the present invention is as follows:

[0007] A multi-source remote sensing image scene recognition method based on a deformable convolutional selective allocation network includes the following steps:

[0008] Step 1: Construct the basic feature extraction network. Since the basic feature extraction network in YOLOv3 only contains ordinary convolutions and lacks feature extraction ability when dealing with scenes with multi-scales and multi-shapes, the present invention introduces deformable convolutions to replace ordinary convolutions in the convolutional neural network of YOLOv3, so as to obtain the spatial structure information of scenes with multi-scales and multi-shapes. In addition, due to the feature differences of multi-source remote sensing images, the present invention provides a unique basic feature extraction network for each type of remote sensing data.

[0009] Step 2: Fuse the target features of the corresponding scale branches on the feature pyramid of heterologous remote sensing images. A selective assignment fusion module is introduced on each scale branch of the feature pyramid of YOLOv3 to fuse multi-source remote sensing images, and different fusion weights are assigned to the features of different images.

[0010] Step 3: The fused features reconstitute new feature representations of different scales. Using the loss function of the YOLOv3 model, with minimizing the loss as the optimization goal, the model parameters are optimized using backpropagation to train the scene recognition model.

[0011] Step 4: After the model training is completed, save it, collect and input the remote sensing images for the scene recognition task to be performed, load the scene recognition model trained in Step 3, use the above model to identify different scenes in the images, obtain the positions and types of the target scenes, and output them visually.

[0012] Furthermore, Step 1 includes the following sub-steps:

[0013] Step 1.1: Construct deformable convolutions. For ordinary convolution operations, for the input feature map U, its normal convolution output contains N channels. While deformable convolutions use 2N ordinary convolutions on U to obtain a feature map containing 2N channels. Then, the feature map with 2N channels (in the shape of B, H, W, 2N) is added to the input feature map (in the shape of B, H, W, N) to obtain the actual sampling positions.

[0014] Step 1.2: Based on the darknet53 feature extraction network of YOLOv3, replace the ordinary convolutions in the darknet53 network with deformable convolutions to obtain multi-scale and multi-shape spatial features without changing the size of the feature map, and improve the scene positioning accuracy.

[0015] Furthermore, Step 2 includes the following sub-steps:

[0016] Step 2.1: The feature extraction base network in Step 1 will obtain feature maps of different scales through convolution-downsampling operations. Here, reverse upsampling operations are performed on these feature maps, and they are concatenated with the feature maps of the same scale obtained in the base network to form the structure of the feature pyramid.

[0017] Step 2.2: Construct a selective assignment fusion module. For the heterogeneous feature inputs X A and X B , first add the two to obtain X. Then, use global average pooling and multiple fully connected layers to obtain two feature-weighted attention vectors α A and α B . Multiply the attention vectors by the original feature inputs to obtain the weighted feature outputs X' A and X' B , so that different fusion weights can be assigned to different remote sensing data. Then, add the two to obtain the fused and enhanced feature output.

[0018] Step 2.3: Introduce the selective assignment fusion module on the multi-scale feature pyramid constructed in Step 1. Use the features of the corresponding scales of different categories of remote sensing data as inputs to obtain the fused multi-source multi-scale feature pyramid.

[0019] Furthermore, Step 3 includes the following sub-steps:

[0020] Step 3.1: Input multi-source remote sensing training images to train the scene recognition model. The output of the model is a feature of S×S×(A*(4 + Class)) dimensions, and the specific meaning is the position, size of the scene bounding box predicted in the input image, and the probability of the included scene category. Among them, S×S is the number of networks predicted by the scene recognition model, A is the number of prior boxes corresponding to each network in each scale branch of the feature pyramid, and Class represents the number of scene categories.

[0021] Step 3.2: Compare the scene recognition results output by the model with the prior labels, calculate the loss between them, use the loss function of TOLOv3, aim to minimize the loss, optimize the network parameters, and save the trained scene recognition model.

[0022] Furthermore, Step 4 includes the following sub-steps: s

[0023] Step 4.1: Collect and input the multi-source remote sensing images of the target area, read the trained scene recognition model saved in Step 3, predict the position coordinates and the category to which the scene belongs, and output and display the scene recognition results of the multi-source remote sensing images.

[0024] Compared with the existing methods, the advantages of the present invention are as follows:

[0025] Fully considering the characteristics of multi-source remote sensing images, the model has better feature extraction ability for scenes of different scales and shapes, and effectively fuses multi-source remote sensing images by making full use of the complementary advantages of multi-source data; based on the YOLOv3 model, deformable convolution is used to obtain the spatial structure information of scenes with multiple scales and shapes, and a selective assignment fusion module is introduced at the corresponding scale of the feature pyramid constructed by multi-source data, enabling the model to selectively assign different fusion weights to the features of heterogeneous images, thus better realizing the complementary advantages of multi-source remote sensing images; a method for scene recognition of multi-source remote sensing images based on a deformable convolution selective assignment network is proposed, achieving a high-precision scene recognition task within the target area. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flowchart of the method for scene recognition of multi-source remote sensing images based on a deformable convolution selective assignment network in the present invention;

[0027] Figure 2 is a structural diagram of the basic network for feature extraction of YOLOv3 improved by deformable convolution constructed in the present invention;

[0028] Figure 3 is a selective assignment fusion module constructed in the present invention for fusing the features of heterogeneous remote sensing images;

[0029] Figure 4 is a flowchart of scene recognition of the multi-source remote sensing image scene recognition based on a deformable convolution selective assignment network in the embodiment of the present invention on a multi-source remote sensing image dataset. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the drawings and by way of examples.

[0031] As Figure 1 shown, a method for scene recognition of multi-source remote sensing images based on a deformable convolution selective assignment network includes the following steps:

[0032] Step 1: Construct a basic network for feature extraction. Deformable convolution is introduced into the convolutional neural network of YOLOv3 to replace ordinary convolution, thereby obtaining the spatial structure information of scenes with multiple scales and shapes. For each type of multi-source remote sensing image received, an independent basic network for feature extraction is available.

[0033] Step 2: Fuse the target features of the corresponding scale branches on the feature pyramid of multi-source remote sensing images. A selective assignment fusion module is introduced on each scale branch of the feature pyramid of YOLOv3 to fuse multi-source remote sensing images, and different fusion weights are assigned to the features of different images;

[0034] Step 3: The fused features are recombined into new feature representations of different scales. Using the loss function of the YOLOv3 model, with minimizing the loss as the optimization goal, the model parameters are optimized using backpropagation to train the scene recognition model.

[0035] Step 4: After the model training is completed, it is saved. Remote sensing images for the scene recognition task are collected and input. The scene recognition model trained in Step 3 is loaded, and different scenes in the images are recognized using the above model to obtain the positions and types of the target scenes, and the results are visualized and output.

[0036] Specifically, Step 1 includes the following steps:

[0037] Step 1.1: Construct deformable convolution. For ordinary convolution operations, for the input feature map U, its normal convolution output contains N channels. While deformable convolution uses 2N ordinary convolutions on U to obtain a feature map containing 2N channels. Then, the feature map with 2N channels (in the shape of B, H, W, 2N) is added to the input feature map (in the shape of B, H, W, N) to obtain the actual sampling positions.

[0038] Step 1.2: Based on the darknet53 feature extraction network of YOLOv3, replace the ordinary convolutions in the darknet53 network with deformable convolutions to obtain multi-scale and multi-shaped spatial features without changing the size of the feature map, and improve the scene localization accuracy.

[0039] In the newly constructed basic feature extraction network, the residual module structure with deformable convolution is as Figure 2 shown.

[0040] Step 2 includes the following steps:

[0041] Step 2.1: The feature extraction basic network in Step 1 will obtain feature maps of different scales through convolution-downsampling operations. Here, an inverse upsampling operation is performed on these feature maps, and they are concatenated with the feature maps of the same scale obtained in the basic network to form the structure of the feature pyramid.

[0042] Step 2.2: Construct a selective assignment fusion module. For the heterogeneous feature inputs X A and X B , first add the two to get X. Then, using global average pooling and multiple fully connected layers, two feature-weighted attention vectors α A and α B are obtained. Multiply the attention vectors by the original feature inputs to obtain the weighted feature outputs X' A and X' B, different fusion weights can be assigned to unused remote sensing data. Then, the two are added together to obtain the fused and enhanced feature output.

[0043] Step 2.3: Introduce a selective assignment fusion module on the multi-scale feature pyramid constructed in Step 2.1. Take the features of corresponding scales of different categories of remote sensing data as inputs to obtain a fused multi-source and multi-scale feature pyramid.

[0044] The structure of the constructed selective assignment fusion module is as Figure 3 shown.

[0045] Step 3 includes the following steps:

[0046] Step 3.1: Input multi-source remote sensing training images to train a scene recognition model. The output of the model is a feature of S×S×(A*(4 + Class)) dimensions, and the specific meaning is the position, size of the scene bounding box predicted in the input image, and the probability of the included scene category. Among them, S×S is the number of networks predicted by the scene recognition model, A is the number of prior boxes corresponding to each network in each scale branch of the feature pyramid, and Class represents the number of scene categories.

[0047] Step 3.2: Compare the scene recognition results output by the model with the prior labels, calculate the loss between them, use the loss function of YOLOv3, aim to minimize the loss, optimize the network parameters, and save the trained scene recognition model.

[0048] Step 4 includes the following steps:

[0049] Step 4.1: Collect and input multi-source remote sensing images of the target area, read the trained scene recognition model saved in Step 3, predict the position coordinates and the category to which the scene belongs, and output and display the scene recognition results of the multi-source remote sensing images.

[0050] Example 1

[0051] The example realizes the recognition of multiple scenes in multi-source remote sensing images within the target area, and its specific implementation process is as Figure 4 shown.

[0052] Step 1: Collect multi-source remote sensing images, label the target scenes in the images and construct a multi-source remote sensing image scene recognition dataset, and preprocess the samples in the dataset, including a series of operations such as radiometric correction, spectral correction, and multi-source registration. In addition, perform operations such as random selection, translation, and adding random noise to the training images for data augmentation to enhance the diversity of training samples. For the obtained remote sensing images, label the scene positions and categories in the images to obtain the corresponding labels of the samples suitable for training the scene recognition model.

[0053] Step 2: Introduce deformable convolution into the feature extraction basic network of YOLOv3 to extract and retain the spatial structure information of multi-scale and multi-shape scene representations. The output feature map of ordinary convolution contains N channels, while deformable convolution obtains a feature map with 2N channels by using 2N convolutions. Here, the feature map represents the offsets in the x and y directions at the corresponding positions. By adding these offsets to the original feature map, the actual sampling positions are obtained.

[0054] Step 3: Construct a three-layer feature pyramid structure with features of different scales. According to the respective advantages of multi-source remote sensing images, construct a unique feature pyramid for each type of feature. Send the features of the corresponding scales of the multi-source remote sensing feature pyramids into the selective assignment fusion module to fuse the multi-source data, so as to obtain the fused and enhanced multi-scale feature output.

[0055] Step 4: Load the dataset, send the training samples into the scene recognition model, compare the obtained scene recognition results with the prior labels, and use the loss function of YOLOv3 to continuously iterate until the loss function converges, so as to obtain a trained scene recognition model.

[0056] Step 5: Collect and input the multi-source remote sensing images of the area to be measured, load the model trained according to Step 4 for scene recognition, obtain the position coordinates and the belonging categories of each scene in the images, and input the scene recognition results of the multi-source remote sensing images.

[0057] Those of ordinary skill in the art will realize that the embodiments described herein are for helping readers understand the implementation methods of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. A multi-source remote sensing image scene recognition method based on a deformable convolutional selection and distribution network, characterized in that It includes the following steps: Step 1: Construct a feature extraction basic network; In the convolutional neural network of YOLOv3, deformable convolutions are introduced to replace ordinary convolutions to obtain multi-scale and multi-shaped scene space structure information; a feature extraction basic network is provided for each type of remote sensing data; Step 2: Fuse the target features of the corresponding scale branches on the feature pyramid of heterogeneous remote sensing images; On each scale branch of the feature pyramid of YOLOv3, a selective assignment fusion module is introduced to fuse multi-source remote sensing images, and different fusion weights are assigned to the features of different images; It includes the following sub-steps: Step 2.1: The feature extraction basic network in Step 1 will obtain feature maps of different scales through convolutional-downsampling operations, perform reverse upsampling operations on these feature maps, and splice them with the feature maps of the same scale obtained in the basic network to form the structure of the feature pyramid; Step 2.2: Construct a selective assignment fusion module; for the heterogeneous feature inputs and , first add the two to obtain ; then use global average pooling and multiple fully connected layers to obtain two feature-weighted attention vectors and ; multiply the attention vectors by the original feature inputs to obtain the weighted feature outputs and , and assign different fusion weights to different remote sensing data; then, add the two to obtain the fused and enhanced feature output; Step 2.3: A selective assignment fusion module is introduced on the multi-scale feature pyramid constructed in Step 1; the features of the corresponding scales of different types of remote sensing data are used as inputs to obtain a fused multi-source multi-scale feature pyramid; Step 3: The fused features are reconstituted into new feature representations of different scales. Using the loss function of the YOLOv3 model, with minimizing the loss as the optimization goal, the model parameters are optimized using backpropagation to train the scene recognition model; Step 4: After the model training is completed, it is saved. The remote sensing images for the scene recognition task are collected and input, the scene recognition model trained in Step 3 is loaded, and the model is used to recognize different scenes in the images to obtain the position and type of the target scene, and the output is visualized.

2. The multi-source remote sensing image scene recognition method based on the deformable convolution selection and distribution network according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 1.1: Construct deformable convolutions; The deformable convolution is used on to use ordinary convolutions to obtain a feature map containing channels; Add the resulting feature map with the number of channels being to the input feature map to obtain the actual sampling positions; The feature map, with a shape of ; Input feature map, with a shape of ; Step 1.2: Based on the darknet53 feature extraction network of YOLOv3, the ordinary convolutions in the darknet53 network are replaced with deformable convolutions to obtain multi-scale and multi-shaped spatial features without changing the size of the feature maps.

3. The multi-source remote sensing image scene recognition method based on the deformable convolution selection and distribution network according to claim 1, characterized in that: Step 3 includes the following sub-steps: Step 3.1: Input multi-source remote sensing training images to train the scene recognition model; Step 3.2: Compare the scene recognition results output by the model with the prior labels, calculate the loss between them, use the loss function of TOLOv3, with minimizing the loss as the goal, optimize the network parameters, and save the trained scene recognition model.

4. The multi-source remote sensing image scene recognition method based on the deformable convolutional selection and allocation network according to claim 3, characterized in that: The output of the model in Step 3.1 is -dimensional features, specifically meaning the position, size of the scene bounding box in the predicted input image, and the probability of the included scene category; where is the number of networks predicted by the scene recognition model, is the number of prior boxes corresponding to each network in each scale branch of the feature pyramid, represents the number of scene categories.

5. The multi-source remote sensing image scene recognition method based on the deformable convolution selection and distribution network according to claim 1, characterized in that: Step 4 includes the following sub-steps: Step 4.1: Collect and input the multi-source remote sensing images of the target area, read the trained scene recognition model saved in Step 3, predict the position coordinates and the category to which the scene belongs, and output and display the scene recognition results of the multi-source remote sensing images.

Images