Road extraction method and system based on fully convolutional neural network

Abstract

This invention discloses a road extraction method and system based on a fully convolutional neural network. The method includes improving a network model based on an FCN to obtain a deep convolutional neural network; training the neural network on a training set with the goal of minimizing the loss function to obtain a road extraction network model; and inputting a SAR image for testing into the road extraction model to obtain the road network in the SAR image. The invention can achieve road extraction.

Description

Technical Field

[0001] The present invention relates to the field of road extraction, and in particular to a road extraction method and system based on a fully convolutional neural network. Background Art

[0002] Synthetic Aperture Radar (SAR) is a modern, high-resolution microwave imaging radar that actively transmits electromagnetic waves toward a target and generates an image by receiving and analyzing the reflected echo signals. It operates in all weather conditions and around the clock. SAR operates over a wide wavelength range and utilizes multiple polarization modes. It has a certain degree of penetration through obstructions such as vegetation and clouds, making it a significant advantage in observing targets on land and at sea. Consequently, SAR is widely used in both military and civilian fields.

[0003] SAR systems are capable of acquiring large quantities of high-resolution images, in which roads are a key target. Roads are a crucial component of modern transportation systems and are the primary objects identified and recorded in maps and information systems. Their detection holds significant geographical, economic, and military significance, with applications in traffic management, road monitoring, urban planning, and map updates. Consequently, the precise extraction of roads from SAR images has attracted considerable research interest.

[0004] Semantic segmentation networks are widely used in SAR image road extraction tasks due to their ability to automatically extract features at all levels of the target and their simplicity. Fully Convolutional Neural Networks (FCNs) are the most common algorithm in semantic segmentation networks. Their simple and practical structure often serves as the basis for designing higher-performance semantic segmentation networks. Summary of the Invention

[0005] The purpose of the present invention is to provide a road extraction method and system based on a fully convolutional neural network, aiming to solve the road extraction problem.

[0006] The present invention provides a road extraction method based on a fully convolutional neural network, comprising:

[0007] S1. Build a deep convolutional neural network;

[0008] S2. Train a deep convolutional neural network on the SAR image training set with the goal of minimizing the loss function to obtain a road extraction network model;

[0009] S3. Input the SAR image used for testing into the road extraction model to obtain the road network in the SAR image.

[0010] The present invention also provides a road extraction system of a fully convolutional neural network, comprising:

[0011] A road extraction system based on a fully convolutional neural network, characterized by including:

[0012] Building module: used to build deep convolutional neural network;

[0013] Training module: used to train a deep convolutional neural network on the SAR image training set with the goal of minimizing the loss function to obtain a road extraction network model;

[0014] Testing module: Input the SAR image used for testing into the road extraction model to obtain the road network in the SAR image.

[0015] By adopting the embodiment of the present invention, road extraction from SAR images can be achieved.

[0016] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flow chart of a road extraction method using a fully convolutional neural network according to an embodiment of the present invention;

[0019] Figure 2 1 is a schematic diagram of the structure of a fully convolutional neural network of a road extraction method using a fully convolutional neural network according to an embodiment of the present invention;

[0020] Figure 3 2 is a schematic diagram of the structure of a multi-scale feature extraction module of a road extraction method using a fully convolutional neural network according to an embodiment of the present invention;

[0021] Figure 4 Schematic diagram of the dilated convolution module structure of the road extraction method of the fully convolutional neural network according to an embodiment of the present invention

[0022] Figure 5 2 is a schematic diagram of the deconvolution layer structure of the road extraction method of the fully convolutional neural network according to an embodiment of the present invention;

[0023] Figure 6 2 is a schematic diagram of a road connectivity enhancement network structure of a road extraction method using a fully convolutional neural network according to an embodiment of the present invention;

[0024] Figure 7 2 is a schematic diagram of a road extraction result of a road extraction method using a fully convolutional neural network according to an embodiment of the present invention;

[0025] Figure 8 2 is a schematic diagram of a road extraction system using a fully convolutional neural network according to an embodiment of the present invention.

[0026] Description of reference numerals:

[0027] 810: Establishment module; 820: Training module; 830: Testing module. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0029] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0030] In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a communication between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0031] Method Example

[0032] According to an embodiment of the present invention, a road extraction method using a fully convolutional neural network is provided. Figure 1 Flowchart of the road extraction method of the fully convolutional neural network according to an embodiment of the present invention. Figure 1 As shown, specifically including:

[0033] (1) Improve the network model based on FCN to obtain a deep convolutional neural network;

[0034] (2) Train the neural network on the training set with the goal of minimizing the loss function to obtain a road extraction network model;

[0035] (3) Input the SAR image used for testing into the road extraction model to obtain the road network in the SAR image.

[0036] Figure 2 : is a schematic diagram of the fully convolutional neural network structure of the road extraction method of the fully convolutional neural network according to an embodiment of the present invention, such as Figure 2 As shown, the deep convolutional neural network constructed by improving FCN in step (1) includes: an input layer, a multi-scale feature extraction layer, a hole convolution connection layer, a deconvolution layer, a road connectivity enhancement network layer and an output layer.

[0037] The input layer includes a convolution layer with a convolution kernel size of 7*7, a stride of 2, and a padding of 3, a batch normalization (BN) layer, a ReLU activation function layer, and a maximum pooling layer with a size of 3*3, a stride of 2, and a padding of 1, which is used to reduce the input image size.

[0038] The multi-scale feature extraction layer includes four units, and the four units are respectively composed of 3, 4, 6, and 3 multi-scale feature extraction modules. Figure 3 : is a schematic diagram of the multi-scale feature extraction module structure of the road extraction method of the full convolutional neural network according to an embodiment of the present invention. Figure 3 As shown, the multi-scale feature extraction module first uses a 3*3 convolution to extract features from the input. The feature map is then divided into three equal parts by channel. 3*3, 5*5, and 7*7 convolutions are applied to these three feature maps, respectively, to obtain features at different scales. The three resulting feature maps are then concatenated by channel and fused using a 1*1 convolution to combine the features of each channel. Finally, the input features are added to the fused feature map to prevent gradient vanishing. Each convolution layer is followed by a batch normalization (BN) layer and a ReLU activation function layer. In the feature extraction module, the 5*5 and 7*7 convolution kernels focus on regional features at different road scales, while the 3*3 convolution kernel primarily learns detailed road features. Comprehensive road features are obtained by fusing multiple features.

[0039] Figure 4: is a schematic diagram of the structure of the dilated convolution module of the road extraction method of the fully convolutional neural network according to an embodiment of the present invention. Figure 4 As shown in the figure, the dilated convolutional connection layer contains four consecutive dilated convolutions, with dilated kernels having dilation ratios d of 1, 2, 4, and 8, respectively. Each dilated convolution is followed by a ReLU activation function layer. The output feature map is obtained by summing the outputs of each dilated convolution, which is used to expand the receptive field, fuse multi-scale features, and enhance feature extraction.

[0040] Figure 5 : is a schematic diagram of the deconvolution layer structure of the road extraction method of the fully convolutional neural network according to an embodiment of the present invention, such as Figure 5 As shown in the figure, the deconvolution layer includes five deconvolution units. Among them, the first four deconvolution units are composed of a 3*3 deconvolution layer, a batch normalization layer (BN) and a ReLU activation function layer, and the last deconvolution unit contains a 4*4 deconvolution, two 3*3 regular convolutions and a Sigmoid activation function in sequence. After the fifth, fourth and third deconvolution units, the feature maps output by the corresponding units in the multi-scale feature extraction layer are processed by the attention mechanism module and then added to the feature maps at the deconvolution unit for feature fusion. The deconvolution layer is used to upsample and restore the spatial structure of the road image, and the attention mechanism module is used to perform secondary attention enhancement in the channel domain and spatial domain of the feature map.

[0041] Figure 6 : is a schematic diagram of a road connectivity enhancement network structure of a road extraction method of a fully convolutional neural network according to an embodiment of the present invention. Figure 6 As shown, the road connectivity enhancement network consists of two parts: an encoder and a decoder. The encoder uses four 3*3 convolutions to extract information from the road prediction probability map, each followed by a batch normalization (BN) layer and a ReLU activation function. After the second, third, and fourth 3*3 convolutions, a 2*2 max pooling layer with a stride of 2 is applied to downsample the current feature map by a factor of 2, resulting in feature maps with spatial sizes equal to 1 / 2, 1 / 4, and 1 / 8 of the input, respectively. The decoder uses four 3*3 convolutions to learn the correction value for each pixel in the road prediction probability map. Before the first, second, and third convolutions, a 2*upsampling with bilinear interpolation is used to gradually restore the spatial structure of the road prediction probability map. After each upsampling, the encoder feature map of the corresponding size is channel-wise concatenated with the current feature map to fuse features from different levels. After the fourth 3*3 convolution, the convolution output is added to the input road prediction probability map, correcting the prediction probability map while avoiding loss of original image information. Finally, the Sigmoid activation function is used to obtain the road prediction probability map after connectivity enhancement.

[0042] The output layer performs a threshold segmentation operation on the connectivity-enhanced road prediction probability map, setting pixel values ​​above the threshold to 1 and pixel values ​​below the threshold to 0. This results in a binary road prediction map, which serves as the final road extraction result. After testing, the optimal segmentation threshold was selected as 0.24.

[0043] Figure 7 : is a schematic diagram of the road extraction result of the road extraction method of the full convolutional neural network according to an embodiment of the present invention, such as Figure 7 As shown in the figure, the SAR images used in the test, the true road values, and the results of road extraction from SAR images using the neural network model of the present invention are schematically illustrated. The figure selects the road extraction results in three different scenarios: town, suburb, and village to illustrate the performance of the model in different situations. The first column is the SAR image to be extracted, the second column is the manually annotated road network true value, the third column is the road extraction result before the connectivity enhancement network correction, and the fourth column is the road extraction result after the connectivity enhancement network correction. It can be seen that the neural network model can extract most of the roads in the SAR image before the connectivity enhancement network correction, but the extracted roads are relatively fragmented, and there are many breakpoints between the roads; after the connectivity enhancement network correction, the road breakpoints are significantly reduced, the connectivity of the extracted roads is enhanced, and the road network is more complete. Recall, precision, and F1-score are introduced to measure the final road extraction results. ResUnet, which is currently commonly used in road extraction, achieved a recall rate of 72.0%, a precision rate of 73.8%, and an F1-score of 72.9% on the SAR dataset used. The neural network model in the present invention achieved a recall rate of 73.8%, a precision rate of 79.5%, and an F1-score of 76.5% on the SAR dataset used. All three indicators are higher than ResUnet, showing better road extraction effect.

[0044] The principles of this invention are as follows: extracting and fusing multi-scale features using multi-size convolution kernels can achieve stronger feature extraction capabilities; using dilated convolution can expand the receptive field and preserve spatial information; and inputting the road prediction probability map into the connectivity enhancement network model can further enhance road network extraction and reduce discontinuities in the road extraction results. The combination of these three methods can achieve excellent road extraction results in SAR images.

[0045] The present invention relates to a SAR image road extraction method based on a fully convolutional neural network with a modified prediction probability map, and an improved FCN model is constructed. Convolution kernels of different sizes are used to extract features of different scales from the input road image, and these features are fused through convolution operations to obtain a feature map, thereby strengthening the model feature expression; a dilated convolution operation is applied to the obtained feature map to obtain a larger receptive field and retain spatial information; upsampling is performed through deconvolution to restore image details, and features of different levels processed by the attention module are fused during the deconvolution process to obtain a road prediction probability map; the road prediction probability map is input into a connectivity enhancement network to correct the probability prediction value of each pixel point and reduce the discontinuities of the extracted road network; finally, the connectivity-enhanced road prediction probability map is converted into a road prediction binary map through threshold segmentation to obtain a road extraction result; the present invention achieves good accuracy in SAR image road extraction by introducing multi-scale feature extraction, feature fusion, and a connectivity enhancement network, and has certain generalizability.

[0046] The beneficial effects of the present invention are: using multi-scale feature extraction to enhance the feature extraction capability of the network model; using the connectivity enhancement network to correct the road prediction probability map and reduce the discontinuities in the extracted roads.

[0047] System Example

[0048] According to an embodiment of the present invention, a road extraction system based on a fully convolutional neural network is provided. Figure 8 Schematic diagram of a road extraction system of a fully convolutional neural network according to an embodiment of the present invention. Figure 8 As shown, specifically including:

[0049] Establishing module 810: used to establish a deep convolutional neural network;

[0050] Training module 820: used to train a deep convolutional neural network on the SAR image training set with the goal of minimizing the loss function to obtain a road extraction network model;

[0051] Testing module 830: Inputting the SAR image for testing into the road extraction model to obtain the road network in the SAR image.

[0052] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0053] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements of the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of this solution.

Claims

1. A road extraction method based on a fully convolutional neural network, characterized in that: include: S1. Establish a deep convolutional neural network, including: Improve the network model based on the fully convolutional neural network FCN to obtain a deep convolutional neural network; The deep convolutional neural network includes, in sequence: an input layer, a multi-scale feature extraction layer, a dilated convolution connection layer, a deconvolution layer, a road connectivity enhancement network layer, and an output layer; The road connectivity enhancement network layer includes an encoder and a decoder; The encoder includes four 3*3 convolutions, each of which is connected to a batch normalization layer and a ReLU activation function. A maximum pooling function is connected after the second, third, and fourth 3*3 encoder convolutions. The first 3*3 encoder convolution inputs the road prediction probability map, and the four 3*3 convolutions are used to extract information from the road prediction probability map. The decoder includes four 3*3 convolutions, which are used to learn the correction value of each pixel in the road prediction probability map. Before the first, second, and third decoder convolutions, bilinear interpolation and 2x upsampling are used to gradually restore the spatial structure of the road prediction probability map. After each upsampling, the feature map of the corresponding size in the encoder is channel-concatenated with the current feature map to fuse features at different levels. After the fourth 3*3 convolution, the output result is added to the input road prediction probability map. Finally, a sigmoid activation function is used to obtain a road prediction probability map with enhanced connectivity. S2. Train a deep convolutional neural network on the SAR image training set with the goal of minimizing the loss function to obtain a road extraction network model, specifically including: The multi-scale feature extraction layer uses convolution kernels of different sizes to extract features of different scales from the road images in the SAR image training set, and these features are fused through convolution operations to obtain feature maps. Using a dilated convolution operation on the feature map through a dilated convolution connection layer to obtain a larger receptive field and retain spatial information; The deconvolution layer upsamples the feature map after the convolution operation, and fuses the features of different levels processed by the attention module during the deconvolution process to obtain the road prediction probability map; Inputting the road prediction probability map into the road connectivity enhancement network layer to correct the probability prediction value of each pixel point and reduce the discontinuities of the extracted road network to obtain a road prediction probability map with enhanced connectivity; using the road prediction probability map with enhanced connectivity and the true road value to calculate the loss function value; and updating the neural network parameters through the back propagation algorithm; The road prediction probability map after connectivity enhancement is converted into a road prediction binary map through threshold segmentation; S3. Input the SAR image used for testing into the road extraction network model to obtain the road network in the SAR image.

2. The method according to claim 1, characterized in that The input layer includes a convolutional layer, a BN layer, a ReLU activation function layer and a maximum pooling layer connected in sequence, and the input layer is used to reduce the size of the input image.

3. The method according to claim 1, characterized in that The multi-scale feature extraction layer specifically includes 4 units connected in sequence, and the first unit, the second unit, the third unit and the fourth unit are respectively composed of 3, 4, 6 and 3 multi-scale feature extraction modules, wherein each multi-scale feature extraction module is used to use a 3*3 convolution to extract the input features to obtain a feature map, and then the feature map is evenly divided into three parts according to the channel, and the three feature maps are respectively subjected to 3*3, 5*5 and 7*7 convolutions to obtain features at different scales. Each convolution is followed by a batch normalization layer and a ReLU activation function layer. The three feature maps of different scales are spliced ​​according to the channel and then a 1*1 convolution is used to fuse the features of each channel to obtain a fused feature map. Finally, the input feature is added to the fused feature map. The first unit inputs the reduced-size input image, and the fourth unit outputs the fused road feature.

4. The method according to claim 1, wherein The dilated convolution connection layer specifically includes four dilated convolutions connected in sequence. There is a ReLU activation function layer after the first dilated convolution, the second dilated convolution, the third dilated convolution and the fourth dilated convolution. The first dilated convolution is the input, and the outputs of the first dilated convolution, the second dilated convolution, the third dilated convolution and the fourth dilated convolution are added together to obtain enhanced road features.

5. The method according to claim 3, characterized in that The deconvolution layer specifically includes five deconvolution units connected in sequence; Among them, the first four deconvolution units are composed of a 3*3 deconvolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence. The first four deconvolution units are the fifth deconvolution unit, the fourth deconvolution unit, the third deconvolution unit and the second deconvolution unit. The input of the fifth deconvolution unit is the enhanced road feature. The last first deconvolution unit includes a 4*4 deconvolution, two 3*3 regular convolutions and a Sigmoid activation function connected in sequence. The feature maps output by the first, second and third units are processed by the attention mechanism module and added to the feature maps output by the fifth, fourth and third deconvolution units respectively for feature fusion, and the road prediction probability map is output from the first deconvolution unit.

6. The method according to claim 1, characterized in that The output layer is used to set pixel values ​​greater than a threshold to 1 and pixel values ​​less than a threshold to 0.

7. The method according to claim 1, characterized in that The S2 specifically includes: obtaining a road prediction probability map with enhanced connectivity through training, calculating the loss function value in combination with the true road value, updating the neural network parameters through the backpropagation algorithm, comparing the current loss function value with the loss function value of the previous epoch at the end of each training epoch, saving the current network model if the loss function value decreases, and updating the learning rate to 0.2 times the original value when the loss function value does not decrease for four consecutive epochs; and determining that the model has converged and terminating the training when the loss function value does not decrease for seven consecutive epochs.

8. The method according to claim 1, characterized in that The S3 specifically includes: The test SAR image is input into the trained road extraction network model to obtain a road prediction binary map. By changing the output layer segmentation threshold, comparing the regression rate and accuracy, the optimal segmentation threshold is determined and the optimal road extraction binary map is obtained.

9. A road extraction system based on a fully convolutional neural network, used in the method according to any one of claims 1 to 8, characterized in that: include: Building module: used to build deep convolutional neural networks, specifically for: Improve the network model based on the fully convolutional neural network FCN to obtain a deep convolutional neural network; The deep convolutional neural network includes, in sequence: an input layer, a multi-scale feature extraction layer, a dilated convolution connection layer, a deconvolution layer, a road connectivity enhancement network layer, and an output layer; The road connectivity enhancement network layer includes an encoder and a decoder; The encoder includes four 3*3 convolutions, each of which is connected to a batch normalization layer and a ReLU activation function. A maximum pooling function is connected after the second, third, and fourth 3*3 encoder convolutions. The first 3*3 encoder convolution inputs the road prediction probability map, and the four 3*3 convolutions are used to extract information from the road prediction probability map. The decoder includes four 3*3 convolutions, which are used to learn the correction value of each pixel in the road prediction probability map. Before the first, second, and third decoder convolutions, bilinear interpolation and 2x upsampling are used to gradually restore the spatial structure of the road prediction probability map. After each upsampling, the feature map of the corresponding size in the encoder is channel-concatenated with the current feature map to fuse features at different levels. After the fourth 3*3 convolution, the output result is added to the input road prediction probability map. Finally, a sigmoid activation function is used to obtain a road prediction probability map with enhanced connectivity. Training module: used to train a deep convolutional neural network on a SAR image training set with the goal of minimizing the loss function to obtain a road extraction network model. Specifically used for: The multi-scale feature extraction layer uses convolution kernels of different sizes to extract features of different scales from the road images in the SAR image training set, and these features are fused through convolution operations to obtain feature maps. Using a dilated convolution operation on the feature map through a dilated convolution connection layer to obtain a larger receptive field and retain spatial information; The deconvolution layer upsamples the feature map after the convolution operation, and fuses the features of different levels processed by the attention module during the deconvolution process to obtain the road prediction probability map; Inputting the road prediction probability map into the road connectivity enhancement network layer to correct the probability prediction value of each pixel point and reduce the discontinuities of the extracted road network to obtain a road prediction probability map with enhanced connectivity; using the road prediction probability map with enhanced connectivity and the true road value to calculate the loss function value; and updating the neural network parameters through the back propagation algorithm; The road prediction probability map after connectivity enhancement is converted into a road prediction binary map through threshold segmentation; Testing module: Input the SAR image used for testing into the road extraction network model to obtain the road network in the SAR image.

Images