Road segmentation and path automatic extraction method based on air-ground cooperation

Abstract

The invention discloses a road segmentation and path automatic extraction method based on air-ground cooperation, and the method comprises the steps: 1, carrying out the road network extraction through a remote sensing image, controlling an unmanned plane to hover or fly to a certain road point, and collecting a ground image from an airspace perspective; step 2, using a target detection algorithm based on a convolutional neural network to carry out vehicle obstacle identification according to the collected ground image; and step 3, fusing the recognition result of the obstacle into the road network topological graph so as to realize autonomous obstacle avoidance of the ground unmanned platform. The method can assist in improving the autonomous obstacle avoidance capability of the ground unmanned platform, and supports intelligent decision making and flexible operation of the ground unmanned platform in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic driving of ground unmanned platforms, and specifically relates to a road segmentation and path automatic extraction method based on air-ground collaboration. Background Art

[0002] In the field of ground unmanned platform autonomous driving technology, the accuracy and real-time performance of the path of the autonomous navigation map are key factors in determining the autonomous maneuverability of the ground unmanned platform. Accurate and real-time paths can ensure the efficiency of the autonomous maneuverability of the ground unmanned platform. At present, the main methods of path acquisition include: on-site road network collection (collecting road networks in areas where autonomous maneuvers are required), road network extraction based on remote sensing images / maps, and road network extraction based on sensors carried by ground unmanned platforms (using on-board sensors such as lidar and cameras to perceive the surrounding environment in real time, identify road, lane line and other ground features, and thus extract drivable areas). Field road network collection is a technology that collects road data in field scenes to obtain road networks and related information. Common field road network collection technologies include GPS mapping (using global positioning system receivers to measure road networks and mark geographic locations. GPS devices can be installed on vehicles or carried by personnel to record their movement paths and location information, thereby mapping the shape and layout of the road network), laser rangefinders (using laser rangefinders to scan the surrounding environment to obtain three-dimensional coordinate information of roads, buildings and other objects. This technology can generate high-precision maps and road network data), ground surveys (through manual ground surveys and data collection, directly marking road networks and map information to form road network data), etc. Road network extraction technology based on remote sensing images / maps is a technology that uses satellite images, aerial images or other remote sensing data and map data in geographic information systems (GIS) to extract road network information through image processing, feature extraction, pattern recognition and other methods. Road network extraction technology based on sensors carried by ground unmanned platforms is a technology that uses vehicle-mounted sensors to perceive and collect data in the displayed road environment to extract road information. The main sensors used include lidar, millimeter-wave radar, and visual camera systems.

[0003] Existing technologies for on-site road network collection, sensor-based road extraction, and road network extraction based on remote sensing images / maps have some shortcomings in practical applications. For example, on-site road network collection requires professional measuring equipment and sensors, which are expensive, and on-site collection takes a lot of time and labor, especially in large-scale and complex terrain environments, where the manpower and time costs are even higher. In addition, for areas with complex terrain or difficult access, such as mountainous areas and forests, on-site road network collection may not cover all road information, resulting in incomplete collection results. Road network extraction based on remote sensing images or maps is limited by conditions such as image and map resolution and data update frequency, and its real-time and accuracy cannot be effectively guaranteed. When using vehicle-mounted sensors to extract road networks, the perception range of vehicle-mounted sensors is limited, they are easily blocked by obstructions, and the performance of sensors is affected by weather, light and other conditions, resulting in incomplete or reduced quality paths. In addition, a large amount of data usually requires complex processing and analysis to extract useful road network information, which consumes a lot of computing resources and time, and its accuracy and robustness in complex environments are also challenged. Summary of the invention

[0004] In view of this, the present invention provides a road segmentation and automatic path extraction method based on air-ground collaboration, which can realize the autonomous obstacle avoidance and maneuverability of ground unmanned platforms and support intelligent decision-making and flexible operation of ground unmanned platforms in complex environments.

[0005] The technical solution for implementing the present invention is as follows:

[0006] A road segmentation and path automatic extraction method based on air-ground collaboration includes the following steps:

[0007] Step 1: Use remote sensing images to extract road networks, control the drone to hover or fly to a certain waypoint to collect ground images from an airspace perspective;

[0008] Step 2: Use the target detection algorithm based on convolutional neural network to identify vehicle obstacles based on the collected ground images;

[0009] Step 3: Integrate the obstacle recognition results into the road network topology map to achieve autonomous obstacle avoidance for the ground unmanned platform.

[0010] Furthermore, the road network extraction adopts a multi-scale attention remote sensing image road extraction model with a high-level semantic information fusion module and a multi-scale skip connection attention module. The model includes an encoder, a high-level semantic information fusion module, a multi-scale skip connection attention module and a decoder.

[0011] Furthermore, the encoder adopts the encoding structure of ResNet34 and is initialized using encoder pre-training parameters.

[0012] Furthermore, the high-level semantic information fusion module further processes the feature map output by the encoder to obtain high-level semantic features with better quality, thereby performing pixel-level semantic segmentation of the remote sensing image.

[0013] Furthermore, a multi-scale skip connection attention module connects the encoder and the decoder, compresses the feature maps of different scales at different stages of the decoder into a fixed number of channels through convolution operations, and adjusts them to a consistent resolution after maximum pooling or upsampling to assist in generating decoding feature maps; the multi-scale feature maps in the encoder are fused and the weights of the fused feature maps are further optimized using the CAM attention module, and the final module output is fed into the corresponding position in the decoding network, helping the extraction model to restore the scale while ensuring that narrow roads are not lost and local detail information such as sharp road edges is not lost, thereby achieving accurate road prediction.

[0014] Furthermore, in step 2, a target detection algorithm based on a convolutional neural network is used to implement a single-stage target detection algorithm through data preprocessing and enhancement, a feature extraction backbone network, a multi-scale feature enhancement network, and a prediction result output network. The algorithm model is pre-trained using a drone aerial photography dataset, and then the network model parameters are fine-tuned using small samples of sample images collected on-site at the test site. This overcomes the multi-scale air-to-ground image target detection technology for small sample data, and ultimately trains a target detection model suitable for the test site.

[0015] Beneficial effects:

[0016] 1. The present invention combines the road network information extracted from remote sensing images with the obstacle targets extracted by the sensors carried by drones, breaks through the high-resolution road extraction technology of remote sensing images, designs a road extraction model with a high-level semantic information fusion module and a multi-scale skip connection attention module, and realizes pixel-level road prediction for the original remote sensing images.

[0017] 2. The present invention adopts the UAV airspace perspective to detect target obstacles, adopts a target detection algorithm based on a convolutional neural network, overcomes the multi-scale target detection algorithm fine-tuning technology of small sample data, and ultimately realizes the target detection requirements of actual ground-air collaborative tasks, ensuring the accuracy and real-time performance of obstacle detection.

[0018] 3. The present invention utilizes the sensors carried by the drone to detect obstacles on the ground from an airspace perspective, which can make up for the limited field of view of the ground unmanned platform sensors to a certain extent.

[0019] 4. The obstacle longitude and latitude calculation method designed by the present invention calculates the real coordinates of the obstacle, completes the real-time update of the road network status, and forms the ability of active air-to-ground obstacle detection and synchronous road network update.

[0020] 5. The present invention uses remote sensing images for path extraction, and takes advantage of the wide range, high resolution, and weather-insensitiveness of remote sensing images. It can extract paths in a large area. The selected remote sensing images have high resolution, the extracted roads also have high resolution, and the remote sensing images are updated frequently, so there are many extraction base maps available. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the road extraction process of the present invention.

[0022] Figure 2 This is the overall architecture of the remote sensing image road extraction model based on multi-scale attention of the present invention.

[0023] Figure 3 It is a schematic diagram of the road extraction result of the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0025] The present invention provides a road segmentation and automatic path extraction method based on air-ground collaboration. In order to solve the problems faced by a single ground unmanned platform in a complex environment, such as limited perception and unclear traversable areas in a large-scale environment, the method extracts the path by combining remote sensing images with the sensor information carried by the drone, and provides waypoint information for the ground unmanned platform. Figure 1 As shown in the figure, the overall technical solution includes three steps: road network extraction, target detection, and path update. By controlling the drone to hover or fly to a certain waypoint, the internal perception data of the drone is collected, and the designed algorithm is used to first segment and extract the roads in the top view, obtain the road pixel map at the location, and build the road network topology map based on it; then use the target detection algorithm to identify the obstacle information in the road network, and solve and publish the real position coordinates of the obstacles based on their pixel coordinates; finally, integrate the perception information obtained above, update the road network topology map, and send waypoint information to the ground unmanned platform, thereby realizing the autonomous obstacle avoidance maneuverability of the ground unmanned platform, and supporting the intelligent decision-making and flexible operation of the ground unmanned platform in complex environments.

[0026] (1) Road network extraction

[0027] First, we study the road extraction model of remote sensing images. We use the multi-scale information and attention mechanism as the starting point, use deep learning technology to autonomously learn the semantic features of the road network, and build a multi-scale attention remote sensing image road extraction model with high-level semantic information fusion module and multi-scale skip connection attention module. Figure 2As shown in the figure. The network model mainly consists of four parts, namely encoder, high-level semantic information fusion module, multi-scale skip connection attention module and decoder. The high-level semantic information fusion module adopts an encoder based on the encoding structure of ResNet34, initializes it with its pre-trained parameters, and further processes the feature map output by the encoder to obtain high-level semantic features with better quality, so as to perform pixel-level semantic segmentation of remote sensing images. In order to ensure the accuracy and connectivity of road prediction, the receptive field is effectively expanded through the dilated convolution without adding additional parameters or calculation amount, and the global context information is mastered under the premise of ensuring spatial detail information. The attention mechanism is introduced to improve the road extraction quality of remote sensing images. The multi-scale skip connection attention module connects the encoder and the decoder, and the feature maps of different scales at different stages of the decoder are uniformly compressed into a fixed number of channels through convolution operations, and adjusted to a consistent resolution after maximum pooling or upsampling to assist in generating decoding feature maps. The multi-scale feature maps in the encoder are fused and the CAM attention module is used to further optimize the weights of the fused feature maps. The final module output is fed into the corresponding position in the decoding network, helping the extraction model to restore the scale while ensuring that narrow roads are not lost and local detail information such as sharp road edges is not lost, thereby achieving accurate road prediction.

[0028] The encoder stage mainly uses convolutional layers to deepen the number of network layers and learn the deep semantic features of the image. However, if the number of network layers is simply deepened through convolutional layers, it may cause problems such as gradient disappearance or gradient explosion. ResNet proposes to use residual block structures (Res-Blocks) to reduce the difficulty of training deep networks and solve the problem of network degradation. The residual block structure uses shortcut connections to achieve identity mapping and cooperates with the forward network to form residual blocks. The encoder part uses the ResNet34 pre-trained model. Since the low-level features of the image have certain similarities, training based on the parameters of the pre-trained model will be conducive to model convergence and obtain better performance when the training data is limited. Specifically, for an input image of size 1024×1024×3, it will first pass through a convolutional layer with a stride of 2, a convolution kernel of 7×7, and an output channel number of 64 to obtain a 512×512×64 feature map E1, and then pass through a maximum pooling layer with a stride of 2 to reduce the width and height of the feature map by half. Afterwards, the network forms the remaining four-layer encoding network by stacking 3, 4, 6, and 3 residual blocks in sequence, and obtains feature maps E2, E3, E4, and E5 of different sizes. At this point, the encoder stage will output a high-level semantic feature map of size 32×32×512. In addition, since the road network accounts for a small proportion of the entire remote sensing image, and the distribution span in the image is large and the connectivity of the road needs to be maintained, it is difficult to obtain high-quality high-level semantic feature maps by only extracting features from the image through the encoder. Therefore, the present invention proposes a high-level semantic information fusion module (HSIFM), which is placed between the encoder and the decoder to further enhance the network's ability to process long-distance contextual information, and help the network better obtain high-level semantic features that conform to the characteristics of the road network.

[0029] The decoder part mainly draws on the decoder structure of LinkNet, but there are slight differences. After the layer-by-layer downsampling operation in the encoder stage, spatial information will inevitably be lost. It is very difficult to restore the image size and detail information only by using the feature map finally output by the encoder and the high-level semantic information fusion module. Therefore, LinkNet helps the network to maintain detail information in the process of restoring the scale by skip-connecting the feature maps of each level in the encoder stage to the corresponding position of the decoder. It should be noted that although U-net also adopts the skip connection mechanism, U-net is implemented by splicing dimensions. This method will double the channel dimension. When restoring to the original number of channels, additional convolution operations are required to achieve dimensionality reduction, and the amount of calculation is large. LinkNet adopts the element-by-element addition method, so it can effectively reduce the calculation and parameter amount of the decoding network, which is a more efficient implementation method. The present invention improves the skip connection mechanism and designs a multi-scale skip connection attention module, which will be specifically introduced in Section 2.4. The decoder uses the information of deconvolution and skip connection to gradually restore the feature map size from the D5 to the D2 stage of the decoder. That is, the decoder blocks (Decoder Blocks) composed of two conventional convolutions with a convolution kernel of 1×1 and a deconvolution with a convolution kernel of 3×3 and a stride of 2 are upsampled and added to the corresponding skip connection feature maps, so that the D5 feature map of size 32×32×512 is gradually restored to the D2 feature map of size 256×256×64. D2 directly obtains the D1 feature map of size 512×512×64 through a decoding block. Finally, the D1 feature map is upsampled to a feature map of 1024×1024×32 through a deconvolution with a convolution kernel of 4×4 and a stride of 2, and then reduced in dimension through two conventional convolutions with a convolution kernel of 3×3. Finally, the Sigmoid function is used to map the value of each pixel to the (0,1) interval. At this point, the network model will output a predicted image of size 1024×1024×1, and the probability of each pixel belonging to the road category is expressed in the form of (0,1). In actual use, different thresholds can be set, such as taking a threshold of 0.5, to obtain the final binary prediction map, that is, when the corresponding pixel probability is greater than 0.5, the pixel point is judged as a road type.

[0030] As mentioned above, the encoder of the present invention adopts the encoding structure of ResNet34 and is initialized using its pre-trained parameters. It should be noted that ResNet34 was originally trained on the Imagnet-1K object classification dataset, and the size of the cropped training image is 224×224. In the remote sensing image road extraction task of the present invention, it is necessary to perform pixel-level semantic segmentation on images of size 1024×1024, which places higher requirements on the network's ability to extract feature information. Therefore, it is necessary to further process the feature map output by the encoder to obtain high-level semantic features with better quality. Considering the uneven distribution of roads in remote sensing images, the winding shape of some roads, the complex background of non-road areas, and the tree canopies, buildings, etc. that easily cause obstruction to the roads, to ensure the accuracy and connectivity of road prediction, it is necessary to grasp the global contextual information through a larger receptive field (Receptive Field) while ensuring the spatial detail information. The receptive field refers to the size of the area mapped by a unit in the output feature map of a certain layer of the convolutional neural network on the input image. The receptive field calculation formula can be expressed by the formula:

[0031]

[0032] Among them, RF i+1 represents the receptive field of the i+1th layer, RF i represents the receptive field of the i-th layer, k i+1 represents the convolution kernel size of the i+1th layer, S n Represents the step size of the nth layer. Downsampling by pooling and convolution can also expand the receptive field, but it will reduce the resolution of the image and lose some spatial information.

[0033] In semantic segmentation, the use of dilated convolution can effectively expand the receptive field without adding additional parameters or computational complexity, and can also avoid reducing the image resolution. The so-called dilated convolution (or atrous convolution) is formed by adding holes to the standard convolution. The dilated convolution with a dilation rate of r is to insert r-1 holes between the elements of the standard convolution kernel k to form a new convolution kernel.

[0034] In order to improve the skip connection mechanism of LinkNet, a multi-scale skip connection attention module is designed. This module first fuses the multi-scale feature maps in the encoder and then uses the CAM attention module to further optimize the weights of the fused feature maps. Finally, the module output is fed into the corresponding position in the decoding network, helping the extraction model to restore the scale while ensuring that narrow roads are not lost and local details such as sharp road edges are not lost, so as to achieve accurate road prediction, such as Figure 3shown.

[0035] (2) Object Detection

[0036] After road extraction through remote sensing images, obstacle detection is required. For the problem of air-to-ground obstacle target detection, a target detection algorithm based on convolutional neural network is used. Through data preprocessing and enhancement, feature extraction backbone network, multi-scale feature enhancement network and prediction result output network, a single-stage target detection algorithm is implemented. The algorithm model is pre-trained using the drone aerial photography data set, and then the network model parameters are fine-tuned on a small sample basis using sample images collected on-site at the test site. The multi-scale air-to-ground image target detection technology with small sample data is mastered, and finally a target detection model suitable for the test site is trained to realize vehicle obstacle recognition at the test site, ensuring the accuracy and real-time performance of obstacle detection.

[0037] Aiming at the target detection problem in obstacle detection tasks, a small sample target detection scheme is designed based on the YOLO algorithm to achieve efficient and accurate multi-scale obstacle target detection. The single-stage YOLO algorithm can improve real-time performance while ensuring system accuracy. The complete target detection algorithm mainly includes four parts: input data preprocessing and enhancement, feature extraction backbone network, multi-scale feature enhancement network and prediction result output network.

[0038] The input data pre-training and enhancement mainly includes the following steps: first, randomly select the coordinates (xc, yc) of the reference point for image splicing; then, resize and scale the four randomly selected images according to the reference point coordinates, and place them at the upper left, upper right, lower left, and lower right positions of the specified size image; then, according to the size transformation method of each image, map the mapping relationship to the image label; finally, splice the large image according to the specified horizontal and vertical coordinates, and process the coordinates of the detection box that exceeds the boundary, finally realizing the enhancement and expansion of the training data.

[0039] The main function of the feature extraction backbone network is to extract the features of the picture. For the input image data, the deep features of the image are aggregated through convolutional neural networks of different structures. In the process of downsampling, the dimension of the features is deepened, and finally the extraction of multi-scale image features is realized. The feature extraction backbone network of this method mainly includes an image slice (Focus) module, a standard convolution (CBL) module, a CSP module and an SPP module. The multi-scale feature enhancement network adopted by the present invention mainly includes an FPN (Feature Pyramid Network) network and a PAN (Path Aggregation Network) network to realize the fusion of shallow graphic features and deep semantic features. In the shallow layer of the convolutional neural network, the network extracts some simple features such as color, contour, texture and shape, which belong to graphic features; as the network continues to deepen, the neural network will continuously fuse and upgrade these features to generate new features, which belong to semantic features. In short, the graphic features come from the shallow layer of the convolutional neural network, which are just simple graphics with less semantic information; while the semantic features come from the deep layer of the convolutional neural network, which are rich in semantic information, but lack local detail feature information.

[0040] The multi-scale feature enhancement network includes a top-down FPN module and a bottom-up PAN module. FPN is top-down, transferring deep semantic features to shallow layers, enhancing semantic information, but does not transfer shallow positioning information; PAN is a supplement to FPN, adding a bottom-up pyramid structure behind FPN to transfer strong shallow positioning features to deep layers. The combined use of FPN and PAN can achieve a richer feature extraction process.

[0041] In order to further improve the network's ability to recognize multi-scale targets, the prediction result output network contains three feature maps of different sizes, namely 255*H*W, 255*2H*2W and 255*4H*4W. The smallest size 255*H*W is responsible for detecting large targets, 255*2H*2W is responsible for detecting medium targets, and the largest size 255*4H*4W is responsible for detecting small targets. The number of channels is 255=3*(5+80), where 3 represents 3 anchors, 5 represents four position information (x, y, w, h) and a confidence level (indicating the probability that an object may exist in the current grid).

[0042] (3) Coordinate calculation and path update

[0043] After obstacle identification is completed, the longitude and latitude of the obstacle needs to be solved. The camera longitude and latitude, posture and obstacle image information provided by the drone platform are used. The camera's intrinsic parameter matrix and distortion parameters are used to convert the pixel coordinates of the obstacle into normalized coordinates in the camera coordinate system. The drone's posture information is then used to convert the coordinates into real coordinates in the earth coordinate system. Finally, the real coordinates of the obstacle are solved and the road section where the obstacle is located is deleted. The real-time update of the road network status is completed, forming the ability to actively detect air-to-ground obstacles and synchronize road network updates.

[0044] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A road segmentation and path automatic extraction method based on air-ground collaboration, characterized in that: The following steps are involved: Step 1: Use remote sensing images to extract road networks, control the drone to hover or fly to a certain waypoint to collect ground images from an airspace perspective; Step 2: Use the target detection algorithm based on convolutional neural network to identify vehicle obstacles based on the collected ground images; Step 3: Integrate the obstacle recognition results into the road network topology map to achieve autonomous obstacle avoidance for the ground unmanned platform.

2. The road segmentation and path automatic extraction method based on air-ground collaboration as claimed in claim 1 is characterized in that: The road network extraction adopts a multi-scale attention remote sensing image road extraction model with a high-level semantic information fusion module and a multi-scale skip connection attention module. The model includes an encoder, a high-level semantic information fusion module, a multi-scale skip connection attention module and a decoder.

3. The road segmentation and path automatic extraction method based on air-ground collaboration as claimed in claim 2 is characterized in that: The encoder adopts the encoding structure of ResNet34 and is initialized using encoder pre-training parameters.

4. The road segmentation and path automatic extraction method based on air-ground collaboration according to claim 2 or 3 is characterized in that: The high-level semantic information fusion module further processes the feature map output by the encoder to obtain high-level semantic features with better quality, thereby performing pixel-level semantic segmentation of remote sensing images.

5. The method for road segmentation and automatic path extraction based on air-ground collaboration as claimed in claim 4, characterized in that: The multi-scale skip connection attention module connects the encoder and the decoder, compresses the feature maps of different scales at different stages of the decoder into a fixed number of channels through convolution operations, and adjusts them to a consistent resolution after maximum pooling or upsampling to assist in generating decoding feature maps; the multi-scale feature maps in the encoder are fused and the weights of the fused feature maps are further optimized using the CAM attention module. The final module output is fed into the corresponding position in the decoding network, helping the extraction model to restore the scale while ensuring that narrow roads are not lost and local detail information such as sharp road edges is not lost, thereby achieving accurate road prediction.

6. The road segmentation and path automatic extraction method based on air-ground collaboration as claimed in claim 1 is characterized in that: In step two, a target detection algorithm based on convolutional neural networks is used to implement a single-stage target detection algorithm through data preprocessing and enhancement, feature extraction backbone network, multi-scale feature enhancement network and prediction result output network. The algorithm model is pre-trained using a drone aerial photography dataset, and then the network model parameters are fine-tuned using sample images collected on-site at the test site. This overcomes the multi-scale air-to-ground image target detection technology for small sample data, and ultimately trains a target detection model suitable for the test site.

Images