Multi-category road automatic extraction method and system combining direction and semantic features

Abstract

The present invention discloses a method and system for automatically extracting multi-category roads by combining directional and semantic features. First, remote sensing image data is acquired and data enhancement is performed. Then, the enhanced remote sensing image data is input into a multi-category road extraction network, and an accurate multi-category road extraction result map is output. The multi-category road extraction network includes a dense feature sharing encoder, a directional guided feature stacking module, a semantic feature enhancement decoder, and a directional feature decoder. The present invention can significantly alleviate the fragmentation of road extraction and category confusion, and can be effectively applied to large-scale, multi-category road extraction tasks, thereby significantly improving the update efficiency of various road networks, and can extract railways, highways, paths, and bridges from remote sensing images with high quality.

Description

Technical Field

[0001] The present invention relates to the fields of remote sensing image information processing and machine learning technology, and relates to a multi-category road automatic extraction method and system, and specifically to a multi-category road automatic extraction method and system combining direction and semantic features, which can be applied to extract and update multi-category road network information in real time from high spatial resolution remote sensing images. Background Art

[0002] Road information extraction and updating is an important task in the field of surveying and mapping. It is of great significance to the construction of smart cities, unmanned autonomous driving, urban construction planning and other fields. Traditional road data is mainly realized through manual visual identification and annotation, which is time-consuming, inefficient, and has large subjective errors. It is far from meeting the requirements of real-time updating of road network information in today's society. With the development of high-resolution earth observation technology and machine learning, extracting road information from high-spatial-resolution remote sensing images can greatly reduce labor costs and have higher extraction accuracy.

[0003] Existing road extraction methods can be generally divided into expert knowledge-based methods and deep learning-based methods. Expert knowledge-based methods mainly extract road information by using texture information, morphological features, etc. This type of method can process remote sensing images with simple backgrounds and clear road networks, but has great limitations in processing road extraction tasks in complex scenes; deep learning-based methods have been a research hotspot in recent years. They can automatically obtain the texture features of the image surface and the deep abstract features through deep neural networks, and often have higher extraction accuracy.

[0004] As the research deepens, the above methods have certain limitations in practical applications: first, road extraction is often regarded as a pixel-level classification task, which is easy to lose detailed morphology in common encoder structures, making it poorly represented in terms of connectivity; in addition, existing studies only roughly divide road extraction into background and road in the image, and no research can handle the real-time extraction of multiple different types of roads at the same time. Existing methods have serious category confusion problems when dealing with multi-class road extraction tasks. Therefore, how to improve the morphological mining and semantic understanding capabilities of existing models when extracting road targets is still a difficult problem facing many challenges. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes a multi-category road automatic extraction method and system that combines direction and semantic features, which can realize the simultaneous extraction of road network information such as railways, highways, paths and bridges in complex scenarios, and ensure the category accuracy and road connectivity of the extracted results.

[0006] The technical solution adopted by the method of the present invention is: a method for automatically extracting multi-category roads by combining direction and semantic features, comprising the following steps:

[0007] Step 1: Obtain remote sensing image data and perform data enhancement operations;

[0008] Step 2: Input the enhanced remote sensing image data into the multi-category road extraction network and output an accurate multi-category road extraction result map;

[0009] The multi-category road extraction network includes a dense feature sharing encoder, a direction-guided feature stacking module, a semantic feature enhancement decoder and a directional feature decoder;

[0010] The dense feature sharing encoder enables users to obtain a multi-scale refined road feature map; the direction-guided feature stacking module is used to introduce road direction features and fully integrate them with road semantic features to obtain a feature map after fusion of direction information; the feature map after fusion of direction information is subjected to dual-branch decoding using a semantic feature enhancement decoder and a directional feature decoder respectively.

[0011] Preferably, in step 1, the data enhancement operation includes random flipping, random rotation, random erasing and other operations to obtain enhanced data.

[0012] Preferably, in step 2, the dense feature shared encoder is composed of 1 initialization convolution layer, 2 dense connection blocks and 1 maximum pooling layer; the dense connection block is composed of several intermediate layers, and the intermediate layers are composed of normalization layers, nonlinear activation layers, and convolution layers. The input of each intermediate layer comes from the output of all the previous intermediate layers, and the relationship with the previous feature map is obtained by formula (1):

[0013] X N =H N ([X 0 ,X 1 ,…,X N-1 ]) (1)

[0014] In the formula, [X 0 ,X 1 ,…,X N-1 ] indicates that the output feature maps from layer 0 to layer N-1 are merged into channels, H N Indicates the Nth intermediate layer.

[0015] Preferably, in step 2, the direction-guided feature stacking module is constructed by two dual-branch attention modules and one fusion module;

[0016] The dual-branch attention module includes two network branches with the same structure, namely a semantic branch and a direction branch, which are used to learn road semantic information and direction information respectively; the two branches share the downsampling part and the stride connection part of the entire module, and have separate network propagation paths in the upsampling part;

[0017] The downsampling part includes three groups of maximum pooling layers and residual blocks arranged in series; the maximum pooling layer of each group uses a 2×2 kernel to reduce redundant information of the feature map, and then uses the residual block to extract road information at different scales;

[0018] The upsampling part, whose input is the high-dimensional road feature information finally obtained by the downsampling part, then gradually restores the road features to the original size by using a series of residual modules and nearest neighbor interpolation upsampling operation feature maps;

[0019] The stride connection part includes three sets of parallel coordinate attention modules and residual blocks; the coordinate attention module takes as input a feature map with a channel number of C, a width of W, and a height of H, performs average pooling in the length and width dimensions respectively, and then performs channel splicing operation, and passes the spliced ​​feature map through a convolution layer, a batch normalization layer, and a sigmoid layer in sequence to achieve feature dimensionality reduction; then the feature map is separated again, and the two feature maps with embedded specific direction information are respectively encoded into attention maps in two directions through a convolution layer and a sigmoid layer, each of which captures the long-range dependency of the feature map in the horizontal or vertical direction; then the attention maps in the two directions are applied to the input feature map by multiplication to emphasize the spatial feature representation of the road area; finally, a residual block and a convolution layer are passed to finally restore the feature map to the original resolution;

[0020] The fusion module includes an intermediate supervision part and a branch fusion part; after the feature maps of the semantic branch and the direction branch are restored to the original resolution in the upsampling part, a rough road category prediction map or a road direction prediction map is obtained through a classification convolution layer in the intermediate supervision part, and the loss value is directly calculated with the true value map of the corresponding scale to achieve intermediate supervision; in the branch fusion part, based on the output of the semantic branch and the direction branch, a convolution layer is used to restore the rough road category prediction map to the initial number of channels, and the feature is preliminarily fused with the output feature of the convolution layer; in the first dual-branch attention module, the directional feature will be fused with the semantic branch feature, and finally realize the information flow sharing between tasks.

[0021] Preferably, in step 2, the semantic feature enhancement decoder comprises a deconvolution-based decoder, a deep supervision module and a category attention module;

[0022] The deconvolution-based decoder gradually restores the feature map to the input size through a series of deconvolutions and transition convolutions;

[0023] The category attention module first passes the input feature map through a convolution layer with a convolution kernel of 1×1, and adjusts the number of channels to C'; then the feature map is subjected to matrix transposition and deformation operations to obtain a two-dimensional feature map with a size of HW×C'; then the input rough prediction map is matrix-transformed to obtain a category attention matrix P with a size of N×HW, and the category attention matrix P is matrix-multiplied with the above two-dimensional feature map to obtain a class center of N×C'; then the class center is transposed into a two-dimensional matrix of C'×N and multiplied with the category attention matrix of N×HW, and a feature map with a size of C'×H×W is obtained after matrix deformation; finally, the feature map is passed through another convolution layer with a convolution kernel of 1×1, and finally a category optimization feature with a size of C×H×W is output;

[0024] The category-optimized features obtained through the category attention module are fused with the features in the deconvolution-based decoder, and finally a refined road category extraction result is obtained through the classification layer.

[0025] Preferably, in step 2, the directional feature decoder is a deconvolution-based decoder, which gradually restores the feature map to the input size through a series of deconvolutions and transition convolutions.

[0026] Preferably, the multi-category road extraction network is a trained network;

[0027] The training process consists of the following sub-steps:

[0028] Step 2.1: Create a multi-category road dataset;

[0029] The high-resolution remote sensing image data containing roads is randomly divided into training set, validation set and test set according to the preset ratio. At the same time, the categories of road pixels in the corresponding area are manually calibrated into railway, highway, road and bridge to obtain road category labels. Then, the road direction calculation method is used to generate road direction labels using the binary road category labels. Finally, data enhancement operations are performed on the remote sensing images, road category labels and road direction labels in the training set data set to complete the data set establishment.

[0030] Step 2.2: input the training set data into the multi-category road extraction network to perform network training;

[0031] During the training process, the learning rate is adjusted by using the SGD optimizer to achieve parameter update optimization; given the road category labels and road direction labels in the training set, the network is trained using the joint loss function; after the training, the optimal network parameters are saved.

[0032] Preferably, in step 2.1, the road direction label is generated by using the binary road category label, and the specific process is as follows:

[0033] (1) The road centerline and its key points are extracted using the skeleton thinning algorithm, and the key point coordinates are arranged from top to bottom and from left to right;

[0034] (2) Connect the key points (p 1 ,p 2 ,…,p i ), calculate the unit direction vector formed by connecting adjacent key points Then the direction vector is converted into the polar coordinate system to calculate its corresponding road direction angle value;

[0035] (3) Then, a pixel point M is connected to the nearest key point p i Connect to form a line vector and calculate the unit direction vector If the projection length is within the automatically calculated road radial width threshold, the pixel at that point is assigned the same direction value as the road unit direction vector;

[0036] (4) Traverse all pixels in the image and finally obtain the corresponding road direction label.

[0037] Preferably, in step 2.2, the joint loss function consists of two parts: road direction loss and road semantic loss, where the road direction loss Cross entropy loss is used to calculate the loss value; for road semantic loss Calculated using SoftIOU loss based on category balance;

[0038]

[0039] Where s represents the scale, and its value is {(H, W), (H / 2, W / 2), (H / 4, W / 4)}, H is the height of the feature map, and W is the width of the feature map; and They are the predicted road category results and the actual road category results respectively; is the number of valid samples of class i, E all is the total number of all samples;

[0040] Finally, the total loss function L total The definition is as follows:

[0041]

[0042] The technical solution adopted by the system of the present invention is: a multi-category road automatic extraction system combining direction and semantic features, comprising:

[0043] one or more processors;

[0044] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method for automatically extracting multiple categories of roads by combining directions and semantic features.

[0045] The advantages of the present invention compared with the prior art are:

[0046] ① Unlike other road network extraction methods that can only extract a single type of road, the present invention can simultaneously extract multiple types of road networks including railways, highways, paths, bridges, etc. with high precision. It has strong fine-grained classification capabilities and can adapt to the extraction of various types of roads in various complex scenes such as cities, villages, waters, and mountainous areas.

[0047] ② This paper adopts a multi-task joint learning mode and introduces prior information of road direction. Compared with other semantic segmentation methods, it effectively improves the connectivity of road extraction results.

[0048] ③The present invention constructs a multi-category road dataset, which can provide reliable training data for other road extraction methods.

[0049] ④ The present invention adopts a joint loss function, including a directional loss function and a class-balanced SoftIoU loss function, which can effectively alleviate the problem of unbalanced number of pixels of different road categories in training data in real scenarios, and further improve the road classification ability of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The technical solution of this invention is further described below using embodiments and specific implementation methods. In addition, some drawings are also used in the process of describing the technical solution. For those skilled in the art, other drawings and the intention of the invention can also be obtained based on these drawings without paying creative labor.

[0051] Figure 1 is an overall structural diagram of a network model according to an embodiment of the present invention;

[0052] Figure 2 is a structural diagram of a direction-guiding stacking module according to an embodiment of the present invention;

[0053] Figure 3 is a semantic enhancement branch structure diagram of an embodiment of the present invention;

[0054] Figure 4 is a flow chart of network training according to an embodiment of the present invention;

[0055] Figure 5It is a sample image of a remote sensing image and a road category label in a data set in an embodiment of the present invention;

[0056] Figure 6 is a flow chart of generating road direction labels in a data set according to an embodiment of the present invention;

[0057] Figure 7 A visualization comparison of the results of the road extraction task in remote sensing images using the method of the present invention and other methods. DETAILED DESCRIPTION

[0058] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0059] This embodiment provides a method for automatically extracting multi-category roads by combining direction and semantic features, comprising the following steps:

[0060] Step 1: Obtain remote sensing image data and perform data enhancement operations;

[0061] In one implementation, the data enhancement operation includes random flipping, random rotation, random erasing and other operations to obtain enhanced data.

[0062] Step 2: Input the enhanced remote sensing image data into the multi-category road extraction network and output an accurate multi-category road extraction result map;

[0063] In one embodiment, see Figure 1 , the multi-category road extraction network includes a dense feature sharing encoder, a direction-guided feature stacking module, a semantic feature enhancement decoder and a directional feature decoder;

[0064] The dense feature sharing encoder enables users to obtain a multi-scale refined road feature map; the direction-guided feature stacking module is used to introduce road direction features and fully integrate them with road semantic features to obtain a feature map after fusion of direction information; the feature map after fusion of direction information is subjected to dual-branch decoding using a semantic feature enhancement decoder and a directional feature decoder respectively.

[0065] In one embodiment, see Figure 1 , the dense feature shared encoder consists of 1 initialization convolution layer, 2 dense connection blocks and 1 maximum pooling layer; the dense connection block consists of several intermediate layers, and the intermediate layers consist of normalization layers, nonlinear activation layers, and convolution layers. The input of each intermediate layer comes from the output of all the previous intermediate layers, and the relationship with the previous feature map is obtained by formula (1):

[0066] X N =H N ([X 0 ,X 1 ,…,X N-1 ]) (1)

[0067] In the formula, [X 0 ,X 1 ,…,X N-1 ] indicates that the output feature maps from layer 0 to layer N-1 are merged into channels, H N Represents the Nth intermediate layer (normalization-linear correction-convolutional layer). This architecture can effectively increase the information interaction between layers, improve the information flow and gradient flow of feature extraction, and fully mine the fine features of the road.

[0068] In one embodiment, see Figure 2 ,The direction-guided feature stacking module is constructed by two dual-branch attention modules and a fusion module;

[0069] The dual-branch attention module includes two network branches with the same structure, namely a semantic branch and a directional branch, which are used to learn road semantic information and directional information respectively. The two branches share the downsampling part and the stride connection part of the entire module, and have separate network propagation paths in the upsampling part.

[0070] The downsampling part includes three groups of maximum pooling layers and residual blocks arranged in series. The maximum pooling layer of each group uses a 2×2 kernel to reduce redundant information of the feature map, and then uses the residual block to extract road information at different scales.

[0071] The upsampling part, whose input is the high-dimensional road feature information finally obtained by the downsampling part, then uses a series of residual modules and nearest neighbor interpolation upsampling operation feature maps to gradually restore the road features to their original size.

[0072] The stride connection part is used to retain the detailed spatial information lost in the original features during the downsampling process, and includes 3 sets of parallel coordinate attention modules and residual blocks. The coordinate attention module is used to further highlight important spatial features. Its input is a feature map with a channel number of C, a width of W, and a height of H. Average pooling is performed in the length and width dimensions respectively. The purpose of this operation is to avoid the loss of position information caused by traditional two-dimensional global pooling. Then the channel splicing operation is performed, and the spliced ​​feature map is passed through a convolution layer, a batch normalization layer, and a sigmoid layer in sequence to achieve feature dimensionality reduction. Then the feature map is separated again, and the two feature maps with embedded specific direction information are respectively encoded into two-directional attention maps through a convolution layer and a sigmoid layer. Each attention map captures the long-range dependency of the feature map along the horizontal or vertical direction. Then the attention maps in both directions are applied to the input feature map by multiplication to emphasize the spatial feature representation of the road area. The multi-scale features passed through each residual module are added to the feature map of the same size corresponding to each task branch in the upsampling part by using stride connection. Finally, through a residual block and a convolution layer, the feature map is restored to the original resolution.

[0073] The fusion module includes an intermediate supervision part and a branch fusion part. The purpose of intermediate supervision is to enable the intermediate layer of the road extraction network to directly obtain the real semantic or directional information from the truth map during training, so as to train the network more stably. After the feature maps of the semantic branch and the directional branch are restored to the original resolution in the upsampling part, a rough road category prediction map or road direction prediction map is obtained through a classification convolution layer in the intermediate supervision part, and the loss value is directly calculated with the truth map of the corresponding scale to achieve intermediate supervision. In the branch fusion part, taking the semantic branch as an example, a convolution layer is used to restore the rough road category prediction map to the initial number of channels, and the feature is preliminarily fused with the output feature of the convolution layer. The same operation is performed on the feature of the directional branch. In the first dual-branch attention module, the directional feature will be fused with the semantic branch feature, and finally the information flow sharing between tasks will be realized, which will help improve the connectivity of the road.

[0074] In one embodiment, see Figure 3 The semantic feature enhancement decoder includes a deconvolution-based decoder, a deep supervision module and a category attention module; the deconvolution-based decoder gradually restores the feature map to the input size through a series of deconvolutions and transition convolutions; the deep supervision module uses the feature maps of different scales in the deconvolution-based decoder and the true values ​​of the category labels of the corresponding scales to supervise and generate a rough classification map;

[0075] The category attention module is used to enhance the road category segmentation capability. By checking the consistency of each pixel with the calculated class center, the network can understand the overall presentation of each class from a global perspective. The module first passes the input feature map through a convolution layer with a convolution kernel of 1×1, and adjusts the number of channels to C'. The feature map is then subjected to matrix transposition and deformation operations to obtain a two-dimensional feature map of size HW×C'. The input rough prediction map is matrix-transformed to obtain a category attention matrix P of size N×HW, and the category attention matrix P is matrix-multiplied with the above two-dimensional feature map to obtain a class center of N×C'. The class center is then transposed to a two-dimensional matrix of C'×N and multiplied with the category attention matrix of N×HW. After matrix deformation, a feature map of size C'×H×W is obtained. Finally, the feature map is passed through another convolution layer with a convolution kernel of 1×1, and finally a category optimized feature of size C×H×W is output. The category-optimized features obtained through the category attention module are fused with the features in the deconvolution-based decoder, and finally a refined road category extraction result is obtained through the classification layer.

[0076] In one embodiment, the directional feature decoder is a deconvolution-based decoder, which gradually restores the feature map to the input size through a series of deconvolutions and transition convolutions.

[0077] In one embodiment, see Figure 4 , the multi-category road extraction network is a trained network; the training process includes the following sub-steps:

[0078] Step 2.1: Create a dataset;

[0079] We obtained high-resolution remote sensing image data containing roads. After preprocessing such as cropping, we divided the data into 80% training set, 10% validation set, and 10% test set. Then we manually divided the road pixel categories of each image block into four categories: railway, highway, path, and bridge. Figure 5 The corresponding road category label is shown in . Then, the road direction label is generated using the binary road category label, i.e., the road mask. The calculation steps are as follows: Figure 6 As shown in the figure: ① Use the skeleton thinning algorithm to obtain the road centerline and its turning points, endpoints and other key points. ② Connect the key points (p 1 ,p 2 ,…,p i ), calculate the unit direction vector formed by connecting adjacent key points Then the direction vector is converted to the polar coordinate system to calculate its corresponding road direction angle value ∠θ; ③ Then a certain pixel point M is compared with the nearest key point P 1 Connect to form a connection vector Formula (2) is used to calculate the projection length on the unit road direction vector. If the projection length is within the given road radial width threshold λ Road If the pixel at this point is within the road, the pixel is assigned the same direction value O as the road unit direction vector. r For all other pixels, assign the non-road direction angle O b ④Then traverse all the pixels in the image to complete the above operations and finally get the corresponding road direction label.

[0080]

[0081]

[0082] Finally, synchronous data enhancement operations are performed on the remote sensing images, road category labels, and road direction labels in the training set to complete the establishment of the dataset.

[0083] Step 2.2: Train the network model and save the optimal parameters;

[0084] The remote sensing road image training data in step 1 is input into the multi-category road extraction network model, and then the network model is trained. In the process, the joint loss function is used for loss calculation, and the SGD optimizer is used to achieve parameter update optimization. During the training, the average intersection-over-union accuracy is calculated once every two epochs using the validation set. If the accuracy is better, the current network parameters are saved. The training process continues until the network converges, and the optimal network model parameters can be obtained.

[0085] The joint loss function consists of two parts: road direction loss and road semantic loss. Cross entropy loss is used to calculate the loss value. For road semantic loss The SoftIOU loss based on category balance is used for calculation, as shown in formula (3), which can effectively solve the problem of imbalanced number of categories in the data set:

[0086]

[0087] Where s represents the scale, and its value is {(H,W), (H / 2,W / 2), (H / 4,W / 4)}, H is the height of the feature map, and W is the width of the feature map. and They are the predicted road category results and the actual road category results respectively. is the number of valid samples of class i, E all is the total number of all samples.

[0088] Finally, the total loss function L total The definition is as follows:

[0089]

[0090] In this embodiment, for a given remote sensing image containing a road, the network uses an end-to-end method to simultaneously extract the road area and the corresponding direction angle. The shared encoder uses densely connected convolution blocks as the main feature extractor, and strengthens feature reuse through dense connections, thereby facilitating the mining and extraction of deep road features; the direction-guided stacking module uses a dual-branch attention module as the main component, which is similar to a mini codec structure. While strengthening the multi-scale capture of spatial context information, it uses the coordinate attention module to calculate the horizontal and vertical spatial attention weights of the input features of each scale, and then inputs it into the residual block to generate a refined feature map to achieve road feature correction and rough prediction. The fusion layer in the two dual-branch attention modules can realize the feature interaction between direction and semantics; the semantic enhancement decoding branch not only restores the output image size of the semantic branch to the original image size through deconvolution, but also uses a deep supervision module and a category attention module.

[0091] The deep supervision module consists of a transposed convolution layer, a convolution layer, and a classification layer. It uses feature maps of different scales in the original decoder to generate rough predictions and fused features required by the category attention module. In addition, the deep supervision method allows intermediate layers of different scales to learn road features directly from the true category labels, reducing the adverse effects of unstable gradients. The category attention module can effectively increase the feature distinction of different road classes, and by calculating the category center, the network can optimize the features from a global perspective of the category. In actual calculations, given a feature map with a channel number of C, a height of H, and a width of W Finally, a convolution layer with a convolution kernel of 1×1 is applied to reduce the number of channels to C'. Then, by transposing and deforming, it is combined with the coarse prediction map Features after deformation Inspired by the attention mechanism, this embodiment regards the deformed coarse prediction map as the attention map and performs matrix multiplication with the transposed category center. The specific category attention feature of each pixel can be obtained by calculation The calculation formula is as follows:

[0092]

[0093] The calculated category attention features are further resized to C×H×W through a convolution layer with a convolution kernel of 1×1, and then fused with the features of the semantic decoder. Finally, a refined road segmentation result can be obtained through classification.

[0094] In order to further verify the effectiveness and feasibility of this method, the following experiment was conducted in this embodiment.

[0095] This experiment uses Pytorch1.10 as the deep learning network building framework, and the development environment is Python3.6. The dataset uses a self-built multi-category road dataset. In order to fully collect various types of road data, the remote sensing images of the dataset come from Google Earth and three public road datasets: Deepglobe dataset, RoadNet dataset, and CHN6-CUG dataset. For the evaluation of extraction results, the commonly used indicators of semantic segmentation accuracy, intersection-over-union ratio, average intersection-over-union ratio, and frequency-weighted intersection-over-union ratio, are used for accuracy evaluation.

[0096] Table 1 shows the quantitative comparison results of the method of the present invention and other methods in the task of road extraction from remote sensing images. Figure 7 Visual comparison results are given.

[0097] Table 1 Comparison of road extraction accuracy between the method of the present invention and other methods

[0098]

[0099] From Table 1 and Figure 7 From the results shown, the method described in the present invention is effective in extracting multiple types of roads. Not only does it achieve the best overall accuracy in extracting four types of roads, but it also has significant advantages over other methods in terms of road connectivity and classification accuracy.

[0100] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A multi-category road automatic extraction method combining direction and semantic features, characterized in that: The following steps are involved: Step 1: Obtain remote sensing image data and perform data enhancement operations; Step 2: Input the enhanced remote sensing image data into the multi-category road extraction network and output an accurate multi-category road extraction result map; The multi-category road extraction network includes a dense feature sharing encoder, a direction-guided feature stacking module, a semantic feature enhancement decoder and a directional feature decoder; The dense feature sharing encoder allows users to obtain a multi-scale refined road feature map; the direction-guided feature stacking module is used to introduce road direction features and fully integrate them with road semantic features to obtain a feature map after fusion of direction information; the feature map after fusion of direction information is subjected to dual-branch decoding using a semantic feature enhancement decoder and a directional feature decoder respectively; The direction-guided feature stacking module is constructed by two dual-branch attention modules and a fusion module; The fusion module is arranged between the two dual-branch attention modules, and is used to superimpose and fuse the two feature maps output by the first dual-branch attention module to form a fused feature map, which is then input into the second dual-branch attention module; The dual-branch attention module includes two network branches with the same structure, namely a semantic branch and a direction branch, which are used to learn road semantic information and direction information respectively; the two branches share the downsampling part and the stride connection part of the entire module, and have separate network propagation paths in the upsampling part; The downsampling part includes three groups of maximum pooling layers and residual blocks arranged in series; the maximum pooling layer of each group uses a 2×2 kernel to reduce redundant information of the feature map, and then uses the residual block to extract road information at different scales; The upsampling part inputs the high-dimensional road feature information finally obtained by the downsampling part and the feature information output by the stride connection part, and then uses a series of residual modules and nearest neighbor interpolation upsampling operations to gradually process the feature information and restore the feature information to the original size; the stride connection part is provided with N groups, which are respectively provided between the input and the output, between the first downsampling output and the N-1th upsampling output, ..., between the N-1th downsampling output and the first upsampling output, wherein N-1 is the number of downsampling times; the high-dimensional road feature information finally obtained by the downsampling part is used as the initial feature map, and a group of residual blocks and nearest neighbor interpolation upsampling layers are input to obtain an upsampling feature map, and the upsampling feature map is added to the feature map having the same width and height as the upsampling feature map obtained by the stride connection part, and then input into the subsequent residual blocks and nearest neighbor interpolation upsampling layers; The stride connection part includes 3 sets of parallel coordinate attention modules and residual blocks; the coordinate attention module takes as input a feature map with a channel number of C, a width of W, and a height of H, performs average pooling in the length and width dimensions respectively, and then performs a channel splicing operation, and passes the spliced ​​feature map through a convolution layer, a batch normalization layer, and a sigmoid layer in sequence to achieve feature dimensionality reduction; then the feature map is separated again, and the two feature maps with embedded specific direction information are respectively encoded into attention maps in two directions through a convolution layer and a sigmoid layer, and each attention map captures the long-range dependency of the feature map in the horizontal or vertical direction; finally, the horizontal attention map and the vertical attention map are respectively passed through a residual block and a convolution layer, so that the feature map is finally restored to the original resolution; The fusion module includes an intermediate supervision part and a branch fusion part; after the feature maps of the semantic branch and the direction branch are restored to the original resolution in the upsampling part, a rough road category prediction map or a road direction prediction map is obtained through a classification convolution layer in the intermediate supervision part, and the loss value is directly calculated with the true value map of the corresponding scale to achieve intermediate supervision; in the branch fusion part, based on the output of the semantic branch and the direction branch, a convolution layer is used to restore the rough road category prediction map to the initial number of channels, and the output features of the dense feature sharing encoder are preliminarily fused with the output features after the convolution layer; in the first dual-branch attention module, the direction branch features will be fused with the semantic branch features, and finally the information flow sharing between tasks is realized; The semantic feature enhancement decoder includes a deconvolution-based decoder, a deep supervision module and a category attention module; The deconvolution-based decoder gradually restores the feature map to the input size through a series of deconvolutions and transition convolutions; The deep supervision module uses feature maps of different scales in the deconvolution-based decoder and the true values ​​of the category labels of the corresponding scales to supervise and generate a rough classification map; The category attention module first passes the feature map output by the deep supervision module through a convolution layer with a convolution kernel of 1×1, and adjusts the number of channels to C'; then the feature map is subjected to matrix transposition and deformation operations to obtain a two-dimensional feature map with a size of H·W×C'; then the input coarse prediction map is matrix-transformed to obtain a category attention matrix P with a size of N×H·W, and the category attention matrix P is matrix-multiplied with the above two-dimensional feature map to obtain a class center of N×C'; then the class center is transposed into a two-dimensional matrix of C'×N and multiplied with the category attention matrix of N×H·W, and a feature map with a size of C'×H×W is obtained after matrix deformation; finally, the feature map is passed through another convolution layer with a convolution kernel of 1×1, and finally a category optimization feature with a size of C×H×W is output; The category-optimized features from the category attention module are fused with the features from the deconvolution-based decoder, and finally a refined road category extraction result is obtained through the classification layer. The directional feature decoder is a deconvolution-based decoder that gradually restores the feature map to the input size through a series of deconvolutions and transition convolutions.

2. The method for automatically extracting multi-category roads by combining direction and semantic features according to claim 1, characterized in that: In step 1, the data enhancement operation includes random flipping, random rotation, random erasing and other operations to obtain enhanced data.

3. The method for automatically extracting multi-category roads by combining direction and semantic features according to claim 1, characterized in that: In step 2, the dense feature shared encoder consists of 1 initialization convolution layer, 2 dense connection blocks and 1 maximum pooling layer; the dense connection block consists of several intermediate layers, and the intermediate layers are composed of normalization layers, nonlinear activation layers, and convolution layers. The input of each intermediate layer comes from the output of all the previous intermediate layers, and the relationship with the previous feature map is obtained by formula (1): X N =H N ([X0,X1,…,X N-1 ]) (1) In the formula, [X0,X1,…,X N-1 ] indicates that the output feature maps from layer 0 to layer N-1 are merged into channels, H N Indicates the Nth intermediate layer.

4. The method for automatically extracting multi-category roads by combining direction and semantic features according to any one of claims 1 to 3, characterized in that: The multi-category road extraction network is a trained network; The training process consists of the following sub-steps: Step 2.1: Create a multi-category road dataset; The high-resolution remote sensing image data containing roads is randomly divided into training set, validation set and test set according to the preset ratio. At the same time, the categories of road pixels in the corresponding area are manually calibrated into railway, highway, road and bridge to obtain road category labels. Then, the road direction calculation method is used to generate road direction labels using the binary road category labels. Finally, data enhancement operations are performed on the remote sensing images, road category labels and road direction labels in the training set data set to complete the data set establishment. Step 2.2: input the training set data into the multi-category road extraction network to perform network training; During the training process, the learning rate is adjusted by using the SGD optimizer to achieve parameter update optimization. Given the road category labels and road direction labels in the training set, the network is trained using the joint loss function. After the training is completed, the optimal network parameters are saved. Among them, the joint loss function consists of two parts: road direction loss and road semantic loss. Cross entropy loss is used to calculate the loss value; for road semantic loss Calculated using SoftIOU loss based on category balance; Where s represents the scale, and its value is {(H, W), (H / 2, W / 2), (H / 4, W / 4)}, H is the height of the feature map, and W is the width of the feature map; and They are the predicted road category results and the actual road category results respectively; is the number of valid samples of class i, E all is the total number of all samples; Finally, the total loss function L total The definition is as follows:

5. The method for automatically extracting multi-category roads by combining direction and semantic features according to claim 4, characterized in that: In step 2.1, the road direction label is generated by using the binary road category label. The specific process is as follows: (1) The road centerline and its key points are extracted using the skeleton thinning algorithm, and the key point coordinates are arranged from top to bottom and from left to right; (2) Connect the key points (p1, p2, ..., p i ), calculate the unit direction vector formed by connecting adjacent key points Then the direction vector is converted into the polar coordinate system to calculate its corresponding road direction angle value; (3) Then, the pixel point M is connected to the nearest key point to form a line vector, and the unit direction vector is calculated. The projection length on the road, where M is any pixel point; if the projection length is within the road radial width threshold, the pixel of the pixel point M is assigned the same direction value as the road unit direction vector; (4) Traverse all pixels in the image and finally obtain the corresponding road direction label.

6. A multi-category road automatic extraction system combining direction and semantic features, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the automatic extraction method of multi-category roads combining direction and semantic features as described in any one of claims 1 to 5.

Images