Remote sensing image road extraction method based on CATransFuse model

Abstract

The invention relates to the field of computer vision, in particular to an improved high-resolution remote sensing image road extraction method based on a CATransFuse model. The method comprises the following steps of: 1, acquiring a remote sensing image, and slicing the remote sensing image according to a fixed size to obtain remote sensing image slices; and a second step, inputting the remote sensing image slices into a trained network CATransFuse for forward propagation operation, obtaining road features in the network CATransFuse, and outputting the road features. According to the method provided by the invention, by optimizing the structure of the network CATransFuse and training the parameters, the network CATransFuse capable of accurately identifying the road features is obtained. In the training process, the ability of identifying various road features is achieved by optimizing the sample quality and the number. Therefore, the identified road features are closer to the actual situation, and the accuracy is higher. According to the method, the CBAM and ASPP modules are introduced, so that feature relevance, context information and road target modeling can be better processed, and better performance can be obtained in a road extraction task.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an improved high-resolution remote sensing image road extraction method based on a CATransFuse model. Background Art

[0002] With the development of aerospace technology and computer technology, various application technologies of remote sensing images have developed rapidly. Including but not limited to road navigation, land resource analysis, urban intelligence and various military application technologies, it can be seen that high-resolution remote sensing image analysis has become one of the key technologies in various fields. Various high-resolution remote sensing image information extraction technologies have created a fast, economical and efficient method for obtaining key and useful geographic information, which can greatly improve production efficiency.

[0003] The recognition and interpretation of remote sensing images are mainly used for road navigation and disaster emergency response. If the road segmentation and extraction of remote sensing images are performed by manual interpretation, the disadvantages are very obvious: high labor costs and long investment cycles. However, with the development of artificial intelligence in recent years, especially the major breakthroughs in the field of computer vision, combined with the high resolution and fast update of remote sensing images, we have the ability to quickly extract roads from remote sensing images.

[0004] The characteristics of roads are varied and can be summarized into the following four characteristics:

[0005] Spectral characteristics: Spectral characteristics refer to the color and spectral information of roads in remote sensing images. Different types of roads have different spectral reflectance characteristics in remote sensing images, which can be distinguished and extracted by analyzing pixel values ​​in different bands. For example, roads usually have higher brightness values ​​and appear gray or white in the visible spectrum, while the surrounding background objects appear in other colors.

[0006] Texture features: Texture features refer to the texture information of roads in remote sensing images, including the texture of the road surface, the distribution of textures, and changes in textures. Texture features can be used to identify the surface texture of roads, such as road stripes, gravel, and other texture features. These features usually have certain regularity in remote sensing images.

[0007] Geometric features: Geometric features refer to the shape and geometric properties of roads in remote sensing images. Roads can have different geometric shapes, such as straight lines, curves, bends, etc. Analyzing the geometric features of roads can help with road extraction and recognition.

[0008] Size feature: Size feature refers to the size and length of the road in the remote sensing image. The size feature of the road can be used to determine the width, length and shape of the road, which is important for road extraction and road network analysis.

[0009] With the development of technology, the resolution of remote sensing images has gradually increased, and the feature information that can be extracted has become richer and richer, but it is also accompanied by increasingly complex road feature information and more and more interference information. In the past twenty years, domestic and foreign scholars have conducted a lot of research on traditional road extraction methods and road extraction methods based on deep learning, and many methods and strategies have achieved good results. These extraction methods target different road feature planning strategies.

[0010] According to the use of different knowledge-based road extraction methods, various road extraction methods are divided into two categories: one is the traditional road extraction method; the other is the road extraction method based on deep learning.

[0011] According to the feature information of road extraction, traditional road extraction methods can be divided into the following two categories: one is the pixel-based road extraction method; the other is the object-based road extraction method.

[0012] Pixel-based road extraction methods rely on the spectral and spatial characteristics of pixels to construct the geometry of the road. This method can only perform road extraction tasks on simple backgrounds, but for road extraction tasks with complex backgrounds, it is easy to generate salt noise and cannot distinguish the shadows caused by vehicles, trees and buildings on the road well. At the same time, the road intersection extraction effect is not ideal, and post-processing is often required to repair the road.

[0013] The object-based road extraction method is used to segment the remote sensing image into small blocks and extract road information in small blocks. This method has good noise resistance, but it is easy to cause confusion for adjacent spaces, similar objects and various shadows, causing adhesion and fracture in the extraction results.

[0014] The road extraction method based on deep learning in remote sensing images is still a problem worth studying. This paper reviews the relevant literature in the field of road extraction in remote sensing images in the past five years, classifies and distinguishes the model structures, and lists the structures, advantages and limitations of various model methods. Compared with traditional road extraction methods, the road extraction method based on deep learning undoubtedly has better performance and higher efficiency. However, due to the particularity of remote sensing images, road images in remote sensing are often interfered by high resolution, such as:

[0015] 1) Shadows from plants, buildings and vehicles;

[0016] 2) Overlap and intersection of roads with different characteristics;

[0017] 3) Some roads are not dispersed, resulting in insufficient differentiation between them and the surrounding environment.

[0018] This makes the road extraction results in remote sensing images fail to achieve the expected results. Therefore, in future research, this type of problem remains the top priority of road extraction in remote sensing images.

[0019] I believe that in future research, methods based on deep learning will also be a key link. Further research may include the following:

[0020] First, unsupervised learning. At present, the accuracy of fully supervised learning is already quite high in the road extraction method based on deep learning, but unsupervised learning is the focus of future road extraction methods. Unsupervised learning can perform self-cognition, self-learning and summarization.

[0021] Second, universality. Among the various methods mentioned in this article, many methods have very different extraction results under different data sets. If a specific method can only be applied to one data set but not to other data sets, it will increase the R&D cost. The structure of the model needs to be constantly adjusted, and the final performance cannot be guaranteed. Therefore, it is also very important to develop a model with good generalization ability.

[0022] Third, the integration of knowledge models. Extracting roads from remote sensing images requires the integration of multiple knowledge, such as background knowledge, auxiliary knowledge, etc. Integrating multiple knowledge methods and more reasonable reasoning for road extraction will improve the accuracy of the extraction results. Summary of the invention

[0023] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a remote sensing image road extraction method based on the CATransFuse model, which integrates the Transformer module and combines multi-scale features to achieve more accurate recognition effect and can adapt to a variety of road scenarios.

[0024] First, in order to achieve the above purpose, a remote sensing image road extraction method based on the CATransFuse model is proposed, and the steps include:

[0025] The first step is to obtain the remote sensing image and slice it according to a fixed size to obtain remote sensing image slices.

[0026] In the second step, the remote sensing image slices are input into the trained network CATransFuse for forward propagation operation to obtain the road features therein and output the road features.

[0027] Among them, in the second step, the network CATransFuse is trained according to the following steps:

[0028] Step s1, slicing the remote sensing image and its road annotation image according to a fixed size to obtain remote sensing image slice samples and its road annotation slice samples.

[0029] Step s2, performing data preprocessing on the samples in step s1, including operations such as denoising and image enhancement.

[0030] Step s3, construct the network CATransFuse, whose network structure includes two branches, the Transformer branch and the CNN branch. The Transformer branch is composed of 8 Transformers in series. The CNN branch uses ResNet as the backbone and adds the CBAM+ASPP module before its first layer. Finally, the BiFusion module fuses the features of the two branches to obtain the final output. The loss function is constructed as a structured loss function.

[0031] Structure Loss=BCELoss+Weighted IOU Loss

[0032] It includes the binary cross entropy loss function

[0033]

[0034] With the weighted intersection-over-union loss function

[0035]

[0036] Where m is the value of the true label (0 or 1), and p is the prediction result of the model (a probability value between 0 and 1). By minimizing the Structure Loss, the prediction result of the model will be close to the true label, thereby improving the classification accuracy of the model.

[0037] In step s4, the labeled remote sensing image slices are input into the network CATransFuse and the loss is calculated.

[0038] Step s5, performing back propagation operation on the loss obtained by the forward propagation operation in step s4 to optimize the network parameters.

[0039] Step s6, repeating steps s4 to s5, training the network CATransFuse, optimizing its network parameters, until the loss of the network G converges, and outputting the network CATransFuse at this time as the trained network CATransFuse.

[0040] Optionally, in the above extraction method, in step s1, the ratio of positive samples to negative samples in the image slice samples is close.

[0041] Optionally, in the above extraction method, the image sample slices are binary images of equal size.

[0042] Beneficial effects of the present invention:

[0043] The present invention obtains a network CATransFuse that can accurately identify road features by optimizing the network CATransFuse structure and training parameters. During the training process, the quality and quantity of samples are optimized to achieve the ability to identify multiple road features. The road features thus identified are closer to the actual situation and have higher accuracy.

[0044] Introducing the CBAM module in the CNN branch of the network CATransFuse: CATransFuse adds the CBAM (Convolutional Block Attention Module) module based on TransFuse. The CBAM module can adaptively weight the channels in the feature map and weight the spatial position to better capture the correlation and spatial importance between features. This enables CATransFuse to more accurately understand the important features in the image and improve the performance and generalization ability of the model.

[0045] Add ASPP module: CATransFuse also introduces the ASPP (Atrous Spatial Pyramid Pooling) module to effectively capture contextual information of different scales. The ASPP module performs multi-scale hole convolution operations on the input feature map using different dilation rates to obtain feature representations of different receptive fields. This helps the CATransFuse model better handle road targets of different sizes and scales and improves the ability to model road structures.

[0046] Improve road extraction performance: Since the CATransFuse model combines the CBAM and ASPP modules, it can better utilize channel attention and spatial attention while effectively capturing contextual information. This gives CATransFuse stronger modeling and perception capabilities in road extraction tasks, and can more accurately locate and extract road targets, improving the accuracy and robustness of road extraction.

[0047] In summary, by introducing the CBAM and ASPP modules, the present invention can better handle the modeling of feature relevance, context information and road targets, thereby achieving better performance in the road extraction task.

[0048] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practicing the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 It is a network structure diagram of the road extraction method CATransFuse according to the present invention.

[0051] Figure 2 FIG. 4 is a structural diagram of a CBAM+ASPP module according to the present invention.

[0052] Figure 3 It is the CBAM+ASPP module layer structure according to the present invention.

[0053] Figure 4 This is an example of the predicted sample annotation of CATransFuse according to the present invention.

[0054] Figure 5 The road features are extracted from the Massachusetts Road Dataset by CATransFuse according to the present invention.

[0055] Figure 6 The road detail features extracted by CATransFuse according to the present invention from the Massachusetts Road Dataset.

[0056] Figure 7 It is the evaluation indicator of CATransFuse according to the present invention on the Massachusetts Road Dataset validation set.

[0057] Figure 8 It is an evaluation index of road detail features extracted from two remote sensing image samples selected by CATransFuse in the Massachusetts Road Dataset according to the present invention.

[0058] Fig. 9 The road features are extracted from the DeepGlobe Road Dataset using CATransFuse according to the present invention.

[0059] Fig.10 The road detail features extracted by CATransFuse in the DeepGlobe Road Dataset according to the present invention.

[0060] Fig.11It is the evaluation indicator of CATransFuse according to the present invention on the DeepGlobe Road Dataset validation set.

[0061] Fig.12 It is an evaluation index of road detail features extracted from two remote sensing image samples selected by CATransFuse in the DeepGlobe Road Dataset according to the present invention.

[0062] Fig.13 The road features extracted by CATransFuse in the GF-2Road Dataset according to the present invention.

[0063] Fig.14 It is the road detail features extracted by CATransFuse in the GF-2Road Dataset according to the present invention.

[0064] Fig.15 It is the evaluation indicator of CATransFuse according to the present invention on the GF-2Road Dataset validation set.

[0065] Fig.16 It is an evaluation index of road detail features extracted from two remote sensing image samples selected by CATransFuse in the GF-2Road Dataset according to the present invention. DETAILED DESCRIPTION

[0066] The preferred examples of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0067] Figure 1 It is a network structure diagram of the road extraction method CATransFuse according to the present invention.

[0068] The first step is to obtain the remote sensing image and slice it according to a fixed size to obtain remote sensing image slices.

[0069] In the second step, the remote sensing image slices are input into the trained network CATransFuse for forward propagation operation to obtain the road features therein and output the road features.

[0070] The above extraction method can extract road features from remote sensing images. This method is obtained by optimizing and improving related links based on the TransFuse model. The method mainly includes the following innovations:

[0071] 1: Introducing the CBAM module: CATransFuse adds the CBAM (Convolutional Block Attention Module) module based on TransFuse. The CBAM module can adaptively weight the channels in the feature map and weight the spatial position to better capture the correlation and spatial importance between features. This enables CATransFuse to more accurately understand the important features in the image and improve the performance and generalization ability of the model.

[0072] 2: Add ASPP module: CATransFuse also introduces the ASPP (Atrous Spatial PyramidPooling) module to effectively capture contextual information of different scales. The ASPP module performs multi-scale hole convolution operations on the input feature map using different dilation rates to obtain feature representations of different receptive fields. This helps the CATransFuse model better handle road targets of different sizes and scales and improves the ability to model road structures.

[0073] The above innovations are analyzed one by one below.

[0074] 1: The CBAM module focuses on the feature representation in the channel dimension. It contains two key components: channel attention mechanism and spatial attention mechanism.

[0075] Channel attention mechanism: By learning the weights between channels, the feature representation ability between different channels is enhanced. The channel attention mechanism models the features of each channel by using global average pooling and fully connected layers, and generates a channel weight vector. This weight vector is multiplied by the original features to weight the information of different channels. This can increase the network's attention to different feature channels and improve the ability to represent features.

[0076] Spatial attention mechanism: By learning the weights of spatial positions, it focuses on the feature representations at different positions. The spatial attention mechanism models the features of each spatial position by using global maximum pooling and fully connected layers, and generates a spatial weight vector. This weight vector is multiplied with the original features to weight the information at different spatial positions. This can increase the network's attention to different spatial positions and enhance the spatial perception of feature representation.

[0077] 2: The ASPP module focuses on multi-scale spatial features to capture contextual information at different scales.

[0078] Multi-scale receptive field: The ASPP module increases the receptive field of the convolution kernel by using dilated convolutions with different dilation rates, thereby capturing contextual information at different scales. By using multiple convolutional layers with different dilation rates in parallel, the network's perception range of features at different scales can be effectively expanded.

[0079] Contextual information fusion: The ASPP module aggregates the results of multiple dilated convolutions and fuses feature information of different scales by using global pooling or average pooling operations. This can preserve contextual information at different scales and enhance the multi-scale capability of feature representation.

[0080] The above two innovative points are referenced Figure 2 and Figure 3 As shown in the figure, the present invention trains the same sample multiple times through models with different structures to obtain several groups of training results, selects the results with advantages and fuses and reorganizes them according to their original parameters and structural settings, and finally obtains the fused model. The road generation accuracy is improved as much as possible, and its effect is as follows Figure 5-16 shown.

[0081] A specific implementation of the present invention is as follows:

[0082] The Massachusetts Road Dataset is used as the sample data source to extract road features.

[0083] According to the model requirements, 10,000 256×256 resolution remote sensing image slices of various terrains and road elements are selected as training data. In order to enable the model to obtain better training results, the proportion of positive and negative samples in the slice area should not differ too much when selecting samples. Finally, the sample data set is obtained as follows Figure 4 shown.

[0084] Therefore, the present invention is trained according to the above samples, and through dual-branch feature fusion, a network CATransFuse that can accurately identify road features is obtained. The purpose of sample training is to enable the model to optimize its own structure during a large number of learning processes, and finally obtain a network model with good adaptability and accuracy to extract road features based on remote sensing images. The specific training process is as follows:

[0085] Step s1, slicing the remote sensing image and its road annotation image according to a fixed size to obtain remote sensing image slice samples and its road annotation slice samples.

[0086] Step s2, performing data preprocessing on the samples in step s1, including operations such as denoising and image enhancement.

[0087] Step s3, construct the network CATransFuse, whose network structure includes two branches, the Transformer branch and the CNN branch. The Transformer branch is composed of 8 Transformers in series. The CNN branch uses ResNet as the backbone and adds the CBAM+ASPP module before its first layer. Finally, the BiFusion module fuses the features of the two branches to obtain the final output. The loss function is constructed as a structured loss function.

[0088] Structure Loss=BCELoss+Weighted IOU Loss

[0089] It includes the binary cross entropy loss function

[0090]

[0091] With the weighted intersection-over-union loss function

[0092]

[0093] Where m is the value of the true label (0 or 1), and p is the prediction result of the model (a probability value between 0 and 1). By minimizing the Structure Loss, the prediction result of the model will be close to the true label, thereby improving the classification accuracy of the model.

[0094] In step s4, the labeled remote sensing image slices are input into the network CATransFuse and the loss is calculated.

[0095] Step s5, performing back propagation operation on the loss obtained by the forward propagation operation in step s4 to optimize the network parameters.

[0096] Step s6, repeating steps s4 to s5, training the network CATransFuse, optimizing its network parameters, until the loss of the network G converges, and outputting the network CATransFuse at this time as the trained network CATransFuse.

[0097] The present invention proposes a new method for extracting roads from high-resolution remote sensing images of urban and rural areas. The CATransFuse model is based on a fusion structure, adopts an ASPP spatial pyramid structure and a convolutional block attention module, and improves the fusion structure to more accurately detect road areas. The method was evaluated using three challenging datasets and compared with other methods. Experiments show that the present invention achieves good results in the task of road extraction in complex backgrounds (urban and rural areas), and under the premise of having sufficient sample number support, the road feature extraction model provided by the present invention can well cope with the shortcomings of traditional methods in image recognition accuracy.

[0098] Those skilled in the art can understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image road extraction method based on the CATransFuse model, characterized in that the steps include: The first step is to obtain a remote sensing image and slice it according to a fixed size to obtain a remote sensing image slice; The second step is to input the remote sensing image slices into the trained network CATransFuse for forward propagation operation, obtain the road features therein and output the road features; Among them, in the second step, the network CATransFuse is trained according to the following steps: Step s1, slicing the remote sensing image and its road annotation image according to a fixed size to obtain remote sensing image slice samples and its road annotation slice samples; Step s2, performing data preprocessing on the samples in step s1, including operations such as denoising and image enhancement; Step s3, construct the network CATransFuse, whose network structure includes two branches, the Transformer branch and the CNN branch. The Transformer branch is a series network composed of 8 Transformers. The CNN branch takes ResNet as the backbone and adds the CBAM+ASPP module before its first layer. Finally, the BiFusion module fuses the features of the two branches to obtain the final output. The loss function is constructed as a structured loss function. Structure Loss=BCELoss+Weighted IOU Loss It includes the binary cross entropy loss function With the weighted intersection-over-union loss function Where m is the value of the true label (0 or 1), and p is the model's prediction result (a probability value between 0 and 1). By minimizing Structure Loss, the model's prediction result will be closer to the true label, thereby improving the model's classification accuracy. Step s4, input the labeled remote sensing image slices into the network CATransFuse and calculate the loss; Step s5, performing a back propagation operation on the loss obtained by performing a forward propagation operation in step s4 to optimize the network parameters; Step s6, repeating steps s4 to s5, training the network CATransFuse, optimizing its network parameters, until the loss of the network CATransFuse converges, and outputting the network CATransFuse at this time as the trained network CATransFuse.

2. The method for extracting roads from remote sensing images based on the CATransFuse model according to claim 1, characterized in that: In step s3, a CBAM+ASPP module is added to the CNN branch of the network CATransFuse.

3. The remote sensing image road extraction method based on the CATransFuse model according to claim 1 is characterized in that: In the step s1, the ratio of positive samples to negative samples in the image slice samples is close.

4. The remote sensing image road extraction method based on the CATransFuse model according to claim 1 is characterized in that: The image slices or the image slices are binary images of equal size.

Images