A remote sensing image road extraction method based on a multi-task neural network

Abstract

A kind of remote sensing image road extraction method based on multi-task neural network, first based on remote sensing image, the remote sensing image obtained is preprocessed and artificial road edge and road surface are marked to obtain two kinds of data sets, then the road edge detection and road surface segmentation of remote sensing image are trained simultaneously by multi-task multi-scale neural network, and the training process is divided into two steps, the first step is to train the road edge data set of remote sensing image to obtain the optimal parameter of secondary task, the second step is to train the road semantic segmentation extraction of remote sensing image based on the optimal parameter, the whole training process is a cycle of the first two steps, the introduction of road edge detection data can realize more accurate remote sensing road segmentation extraction, then for the new remote sensing image road segmentation extraction task.The present application realizes fast and accurate road extraction by the road edge detection and road surface segmentation of remote sensing image based on multi-task neural network.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing technology, and specifically relates to a method for road extraction from remote sensing images based on a multi-task neural network. Background Technology

[0002] With the rapid development of remote sensing technology, it has been widely applied in fields such as civil aviation, urban planning, and autonomous driving. Emerging remote sensing technologies utilize various active or passive sensors to indirectly detect target objects. By analyzing information from different wavebands collected from satellites and drones, the characteristics of the target objects can be obtained, making it a highly practical detection technology. Due to its advantages of acquiring abundant data, high spatiotemporal resolution, and wide-area detection capabilities, remote sensing technology is the optimal method and means for timely obtaining road conditions. This is particularly significant for areas where large-scale road construction and map updates are frequent. Remote sensing technology can achieve large-scale road extraction from remote sensing images over a wide time series, providing road map creation within a specific period, which plays a crucial role in the rational planning of urban traffic and the safety of civil aviation airport traffic.

[0003] In recent years, with the rapid development of computing power and deep learning, Convolutional Neural Networks (CNNs) have achieved remarkable results in computer vision and other fields. Artificial intelligence neural networks based on supercomputing power are an emerging research field. Their basic principle is to construct many neurons, arrange them in a regular pattern, and then train them using a large existing dataset to learn key features, ultimately improving the accuracy of classification or prediction. Because convolutional layers are very effective for image feature extraction, many researchers have attempted to apply deep convolutional neural networks to semantic segmentation for pixel-level classification. In recent years, deep learning algorithms have become increasingly advantageous in image processing, which is of great significance to research in remote sensing image processing. The successful application of deep learning for semantic segmentation in general everyday scene images has made it possible to apply deep learning to the field of remote sensing image processing. Because road extraction from remote sensing images is often obstructed by vegetation such as trees, conventional road extraction algorithms often fail to extract obscured roads or achieve low accuracy. Road extraction from such remote sensing images remains a challenge. Therefore, more research and exploration are needed to successfully apply deep learning to remote sensing images. In other words, how to use deep learning to segment and extract roads in obscured remote sensing images more effectively and accurately is a necessary and urgent issue in the study of semantic segmentation of remote sensing images. Summary of the Invention

[0004] To address the above issues, this application proposes a road extraction method for remote sensing images based on a multi-task neural network. This method can be used in applications such as airport autonomous driving route planning and urban traffic management planning. It can solve the problem of road extraction failure or low extraction accuracy caused by occlusion and other reasons in high-resolution remote sensing images.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] This invention provides a method for road extraction from remote sensing images based on a multi-task neural network, the specific steps of which are as follows:

[0007] S11 uses remote sensing images acquired by remote sensing satellites or aerial remote sensing drones to manually annotate the remote sensing images to be processed to obtain road edges and road surfaces, and performs image preprocessing, including noise reduction, correction and segmentation.

[0008] The above-mentioned methods for processing remote sensing images include:

[0009] The Labelme program was used to annotate and draw the labeled map data. The process of manually annotating the dataset is as follows:

[0010] First, the remote sensing image is resampled to high resolution. Then, the road boundary is determined. Next, it is manually delineated, with a buffer of one pixel inward from the road boundary during image annotation.

[0011] S12 constructs a multi-task neural network model, which includes a main classification task and a sub-classification task;

[0012] The primary classification task is used to perform semantic segmentation and extraction of roads in the image, while the secondary classification task is used to detect and extract the edges of the roads.

[0013] The aforementioned multi-task neural network model includes a module where semantic segmentation for the main task and edge detection for the secondary task are performed on the same network, but the specific training and inference process is divided into two steps.

[0014] The first step is training the secondary task t for road edge detection. The entire multi-task neural network model is f. θ The secondary task t performs initial training on the neural network with randomized initial parameters θ. The optimal parameters are found by calculating the loss, using gradient descent to minimize the loss, and finally, the optimal parameters that minimize the loss are identified.

[0015]

[0016] In the formula, θ i The optimal parameters for subtask t are: road edge detection, θ is the initialization parameter, and α is the learning rate hyperparameter during subtask training. The gradient of task t;

[0017] The preliminary training of road edge detection provides the optimal initial parameters θ' for the subsequent optimization of the main task. The training of the main task T, road semantic segmentation, can begin under the optimal initial parameters θ', obtaining the optimal parameters for the target main task T. With the help of the optimal initial parameters, the randomly initialized parameters θ' can find an initial parameter that is more similar to the target task. When training the main task T, it is not necessary to take many gradient steps. The whole process is described as follows:

[0018]

[0019] In the formula, θ' is the optimal initialization parameter, and β is the learning rate hyperparameter during the main task training. It is the gradient of each main task T with respect to the parameter θ';

[0020] S13 uses the trained model for prediction, fine-tuning it on a dataset with simple road edge annotations on a dataset that requires road extraction. It extracts the location and type information of roads from the input remote sensing image and performs post-processing on the extracted roads, including noise removal, missing data filling, and connection breakage processing.

[0021] As a further improvement of the present invention, in step S12, two learning rate hyperparameters α and β are designed in the gradient update of the main task T and the secondary task t. The two learning rates are optimized using the particle swarm optimization algorithm. The formulas for the velocity and position after the iterative update by the particle swarm optimization algorithm are as follows:

[0022] V id =ωV id +C1random(0,1)(P id -X id )+C2random(0,1)(P gd -X id )X id =X id +V id

[0023] X id =X id +V id

[0024] In the formula, ω is the inertia coefficient, C1 and C2 are called acceleration constants, and C1 = C2 ∈ [0,4] represents a random number in the interval; random(0,1) represents a random number in the interval [0,1]; P id Represents the global optimal solution in dimension d; the termination condition is the maximum number of iterations.

[0025] The fitness function for optimizing the learning rate in the particle swarm optimization algorithm is as follows:

[0026] minf(r1,r2)=minL(f θ )

[0027] In the formula, r1 and r2 represent two learning rate hyperparameters designed in the gradient update of the main task T and the secondary task t, respectively.

[0028] As a further improvement of the present invention, the multi-task neural network model used in step S12 employs a convolutional neural network, comprising two parts: a downsampling path for extracting low-level features of the image, and an upsampling path for combining these low-level features into high-level features. The two paths are connected by skip connections.

[0029] The modules for extracting features in the downsampling and upsampling paths of the neural network model are implemented using a multi-scale convolutional module with a visual attention mechanism. This module mainly consists of three parts.

[0030] First, there is a deep convolution module for aggregating local and global information, followed by a strip convolution module for obtaining multi-scale information of remote sensing images, and then a 1*1 convolution module for aggregating different channel scales.

[0031] The multi-scale convolutional module with visual attention mechanism mentioned above is preceded by a batch normalization layer and a 1*1 convolutional layer, followed by a linear rectified function as the activation function f(x) = max(0, x).

[0032] To address the problem of segmentation failure or low accuracy in remote sensing road segmentation under occlusion conditions, this invention proposes a road extraction method based on a multi-task neural network. This method divides the neural network training process into two steps by establishing a multi-task neural network model. The first step involves inputting manually annotated road edge data into the secondary task training of the multi-task model, learning optimal parameters through gradient descent. The second step involves inputting the extracted road segmentation data into the main task of the multi-task model, iteratively learning based on the optimal parameters obtained in the first step, ultimately yielding a high-precision road extraction model. When extracting new airport or urban roads, only simple road edge annotation and fine-tuning of the trained model are needed for fast and accurate road extraction.

[0033] This invention proposes a method for joint training of a multi-task neural network model to address the semantic segmentation problem of remote sensing images. This method can detect road edges and perform road semantic segmentation in remote sensing images, effectively solving the problem of road occlusion in remote sensing images being unable to be segmented or having low segmentation accuracy. It can be widely applied to airport or urban road extraction. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the overall process of a remote sensing image road extraction method based on a multi-task neural network according to the present invention.

[0035] Figure 2 This is a schematic diagram of the attention multi-scale module in a multi-task neural network.

[0036] Figure 3 This is a schematic diagram of the remote sensing image training images of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0038] This invention provides a method for road extraction from remote sensing images based on a multi-task neural network, such as... Figure 1 As shown in the diagram, the training images for remote sensing are as follows: Figure 3 As shown, this invention is a semantic segmentation method based on multi-task remote sensing image data, comprising the following steps:

[0039] S11 uses high-resolution remote sensing images acquired by remote sensing satellites or aerial remote sensing drones. The images are manually annotated to obtain road edges and surfaces, and then preprocessed, including denoising, correction, and segmentation.

[0040] The above-mentioned methods for processing remote sensing images include:

[0041] The Labelme program was used to annotate and draw the labeled map data. The drawing result is as follows: Figure 1 As shown, the process of manually annotating the dataset involves first resampling the remote sensing image to high resolution, and then determining the road boundaries. Secondly, the roads studied in this invention are manually delineated. To reduce errors during the annotation process, a buffer of one pixel is applied inwards from the road boundary during image annotation.

[0042] S12 constructs a multi-task neural network model, which includes a main classification task and a sub-classification task. The main classification task is used to perform semantic segmentation and extraction of roads in the image, and the sub-classification task is used to detect and extract the edges of the roads.

[0043] First, we have a road semantic segmentation problem with two categories. The input image size is 521×521. After a series of convolution and pooling operations, the feature-extracted image is fed into the final fully connected layer. The fully connected layer processes the data to produce a vector of the same length as the task categories, where each vector represents the probability of the image belonging to each category. Generally, the category with the highest probability value is chosen. Typically, for training semantic segmentation of remote sensing images, the images are too large to be directly used as training data. Therefore, it is necessary to crop the large remote sensing images into smaller images for training. Finally, the semantic segmentation results of all the smaller images are aggregated to obtain the final result.

[0044] The feature extraction modules in the downsampling and upsampling paths of the neural network model are implemented using multi-scale convolutional modules with visual attention mechanisms. A schematic diagram of the attention multi-scale module in the multi-task neural network is shown below. Figure 2 As shown, this module mainly consists of three parts: first, a deep convolution module for aggregating local and global information; second, a strip convolution module for obtaining multi-scale information of remote sensing images; and third, a 1*1 convolution module for aggregating different channel scales.

[0045] The aforementioned multi-scale convolutional module with visual attention mechanism is preceded by a batch normalization layer and a 1*1 convolutional layer. Convolutional neural networks employ batch normalization after each convolutional operation, ensuring that the input and output data of each layer lie on the same Gaussian distribution. This leads to smoother convergence during training. A certain layer has d-dimensional input: X = (x... (1) ...x (d) Therefore, each dimension after batch normalization can be represented as:

[0046]

[0047] After processing using the above batch normalization operation, the distribution of the input data to this layer may sometimes change due to the relatively coarse processing method. Therefore, learnable parameters γ and β are introduced, with the following formula:

[0048] y (k) =γ (k) x (k) +β (k)

[0049]

[0050] β (k) =E[x (k) ]

[0051] Introducing reconstruction parameters γ and β allows the neural network to recover the data feature distribution when the original neural network parameters were used. Therefore, the forward propagation formula for the batch normalization network layer is:

[0052]

[0053]

[0054]

[0055] y i ←γx i +β≡BN γβ (x i )

[0056] In the formula, m is the minimum batch size for batch uniformization.

[0057] Introducing batch uniformization can prevent gradient explosion caused by gradient vanishing during training. On the other hand, it can reduce overfitting due to excessive training iterations, allowing us to use a larger learning rate during training, thereby improving training speed and model accuracy. A linear rectified function is then used as the activation function, specifically:

[0058] f(x) = max(0,x)

[0059] The aforementioned multi-task neural network model includes a module where semantic segmentation for the main task and edge detection for the secondary task are performed on the same network, but the actual training and inference process is divided into two steps.

[0060] The first step is training the secondary task t for road edge detection. The entire multi-task neural network model is f. θ The secondary task t performs initial training on the neural network with randomized initial parameters θ. We find the optimal parameters by calculating the loss. We use gradient descent to minimize the loss and find the optimal parameters that minimize the loss:

[0061]

[0062] In the formula, θ i The optimal parameters for subtask t are: road edge detection, θ is the initialization parameter, and α is the learning rate hyperparameter during subtask training. The gradient of task t.

[0063] The preliminary training for road edge detection described above can provide optimal initial parameters θ' for the subsequent optimization of the main task. Training for the main task T, road semantic segmentation, can begin with these optimal initial parameters θ', yielding the optimal parameters for the target main task T. These optimal initial parameters help find an initial parameter θ' that more closely approximates the target parameter, thus eliminating the need for numerous gradient steps when training the main task T. The entire process can be described as follows:

[0064]

[0065] In the formula, θ' is the optimal initialization parameter, and β is the learning rate hyperparameter during the main task training. It is the gradient of each main task T with respect to the parameter θ'.

[0066] In the gradient update of the main task T and the secondary task t, two learning rate hyperparameters α and β are designed. The selection of these two hyperparameters is related to the learning speed and performance of the neural network model. This invention utilizes the particle swarm optimization algorithm to optimize the two learning rates. By optimizing, different learning rates are obtained, thereby accelerating the learning speed and improving the performance of the network model. The formulas for the velocity and position after iterative update by the particle swarm optimization algorithm are shown below:

[0067] V id =ωV id +C1random(0,1)(P id -X id )

[0068] +C2random(0,1)(P gd -X id )X id =X id +V id

[0069] X id =X id +V id

[0070] In the formula, ω is the inertia coefficient, C1 and C2 are called acceleration constants, and C1 is generally taken as C2∈[0,4] to represent a random number in the interval; random(0,1) represents a random number in the interval [0,1]; P id Represents the global optimal solution in dimension d; the termination condition is the maximum number of iterations.

[0071] The fitness function for optimizing the learning rate in the particle swarm optimization algorithm is as follows:

[0072] minf(r1,r2)=minL(f θ )

[0073] In the formula, r1 and r2 represent two learning rate hyperparameters designed in the gradient update of the main task T and the secondary task t, respectively.

[0074] The two-step training framework effectively utilizes the parameters learned from the secondary task for the primary task, enabling faster learning of the primary task. Simultaneously, it leverages the road edges learned from the secondary task to improve the final performance and accuracy of the primary task's road semantic segmentation.

[0075] S13 uses the trained model for prediction, fine-tuning it on a dataset with simple road edge annotations to extract the location and type information of roads from the input remote sensing image. Post-processing is then applied to the extracted roads, including noise removal, missing data filling, and connection breakage correction, to make the extraction results more accurate and complete.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A method for road extraction from remote sensing images based on a multi-task neural network, comprising the following steps, characterized in that: S11 uses remote sensing images acquired by remote sensing satellites or aerial remote sensing drones to manually annotate the remote sensing images to be processed to obtain road edges and road surfaces, and performs image preprocessing, including noise reduction, correction and segmentation. The above-mentioned methods for processing remote sensing images include: The Labelme program was used to annotate and draw the labeled map data. The process of manually annotating the dataset is as follows: First, the remote sensing image is resampled to high resolution. Then, the road boundary is determined. Next, it is manually delineated, with a buffer of one pixel inward from the road boundary during image annotation. S12 Constructs a multi-task neural network model, which includes a main classification task and a sub-classification task; The primary classification task is used to perform semantic segmentation and extraction of roads in the image, while the secondary classification task is used to detect and extract the edges of the roads. The aforementioned multi-task neural network model includes a module where semantic segmentation for the main task and edge detection for the secondary task are performed on the same network, but the specific training and inference process is divided into two steps. The first step is the secondary task. The training of the road edge detection multi-task neural network model is as follows: Sub-task Initial training is performed on the neural network, with randomized initial parameters as follows: The optimal parameters are found by calculating the loss, using gradient descent to minimize the loss, and finally finding the optimal parameters that minimize the loss. ； In the formula, This mission Optimal parameters for road edge detection These are initialization parameters. This is the learning rate hyperparameter during training for this task. Task The gradient; The preliminary training for road edge detection provides optimal initial parameters for the subsequent optimization of the main task. Main task Road semantic segmentation can be trained with optimal initial parameters. Training begins now, and the target main task is obtained. The optimal parameters are obtained by using the optimal initial parameters. Make random initialization parameters Find initial parameters that more closely approximate the target task when training the main task. It does not require many gradient steps; the entire process is described as follows: ； In the formula, These are the optimal initialization parameters. It is the learning rate hyperparameter during training on the main task. For each main task For parameters The gradient; S13 uses the trained model for prediction, fine-tuning it on a dataset with simple road edge annotations on the dataset requiring road extraction. It extracts the location and type information of roads from the input remote sensing image and performs post-processing on the extracted roads, including noise removal, missing data filling, and connection breakage processing.

2. The method for road extraction from remote sensing images based on a multi-task neural network according to claim 1, characterized in that: Main task in step S12 and sub-task In gradient updates, two learning rate hyperparameters are designed. and The particle swarm optimization algorithm is used to optimize the two learning rates. The formulas for velocity and position after iterative updates by the particle swarm optimization algorithm are shown below: ； ； In the formula, The inertia coefficient, and This is called the acceleration constant, and is taken as... Represents a random number within a given interval; Representative interval Random numbers; represent The global optimal solution for the dimension; the termination condition is the maximum number of iterations. The fitness function for optimizing the learning rate in the particle swarm optimization algorithm is as follows: ； In the formula, and Representing the main task and sub-task Two learning rate hyperparameters are designed in the gradient update.

3. The method for road extraction from remote sensing images based on a multi-task neural network according to claim 1, characterized in that: The multi-task neural network model used in step S12 uses a convolutional neural network, which includes two parts: a downsampling path for extracting low-level features of the image, and an upsampling path for combining these low-level features into high-level features. The two paths are connected by a skip connection. The modules for extracting features in the downsampling and upsampling paths of the neural network model are implemented using a multi-scale convolutional module with a visual attention mechanism. This module mainly consists of three parts. First, there is a deep convolution module for aggregating local and global information, followed by a strip convolution module for obtaining multi-scale information of remote sensing images, and then a 1*1 convolution module for aggregating different channel scales. The aforementioned multi-scale convolutional module with visual attention mechanism is preceded by a batch normalization layer and a 1x1 convolutional layer, followed by a linear rectified function as the activation function. .

Images