Fine road extraction method and system based on multi-task learning

Abstract

The invention discloses a fine road extraction method based on multi-task learning, and the method comprises the following operation steps: obtaining a road data set, carrying out the road prediction of a padleseg-lite model for the road data set, and extracting a predicted road image; an RGBM-LinKnet model is constructed, road mask optimization is carried out on the predicted road image through the RGBM-LinKnet model, and a final road image is obtained; performing further morphological operation optimization on the final road image of the road mask of the optimized predicted road image, and extracting a road boundary to obtain a road boundary image; according to the method, road extraction and related task learning are carried out at the same time by adopting a padleseg-lite model and an RGBM-LinKnet model of a multi-task learning strategy, so that the models can learn richer features, and the accuracy and precision of road extraction are improved.

Description

Technical Field

[0001] The present invention relates to the field of road extraction, and in particular to a fine road extraction method and system based on multi-task learning. Background Art

[0002] With the acceleration of urbanization, the structure of urban road networks is becoming increasingly complex, and the demand for road extraction technology is also increasing. In the field of road extraction, existing deep learning methods have shown their potential in improving road recognition accuracy. These methods usually use convolutional neural networks to automatically extract road features and optimize model performance by training a large amount of labeled data. However, although deep learning technology has made significant progress in the field of image recognition, existing technologies still face challenges in terms of road integrity, connectivity, and boundary accuracy. In particular, in complex urban environments, the problems of road breaks and blurred boundaries are particularly prominent.

[0003] In recent years, the rapid development of deep learning technology has provided new solutions for road extraction; however, the existing technical defects are also particularly obvious, such as the limitations of pixel segmentation: the existing road pixel segmentation model may not be able to accurately identify road boundaries, especially when the road width changes or is similar to other landform types. The need for boundary correction: the existing technology often lacks flexibility and accuracy when dealing with subtle adjustments to road boundaries. These are the technical problems that need to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide a fine road extraction method and system based on multi-task learning, which solves the above-mentioned technical problems pointed out in the prior art.

[0005] The present invention provides a fine road extraction method based on multi-task learning, comprising the following steps:

[0006] Acquire a road data set, perform road prediction of the paddleseg-lite model on the road data set, and extract a predicted road image;

[0007] Constructing an RGBM-LinKnet model, and optimizing a road mask of a predicted road image by using the RGBM-LinKnet model to obtain a final road image;

[0008] The final road image of the road mask of the optimized predicted road image is further optimized by morphological operation, and the road boundary is extracted to obtain a road boundary image.

[0009] Preferably, a road data set is obtained, and the road prediction of the paddleseg-lite model is performed on the road data set to extract the predicted road image. The specific operation steps are as follows:

[0010] A plurality of road images are collected by a recording device of a drone, and the plurality of road images are annotated to obtain a road data set;

[0011] Preprocessing the road data set to obtain a preprocessed road data set;

[0012] Dividing the preprocessed road data set into a road training set and a road test set;

[0013] Using the PaddleSeg-Lite model to train the road training set to obtain a PaddleSeg-Lite model;

[0014] Using the road test set to test the PaddleSeg-Lite model to obtain a tested PaddleSeg-Lite model;

[0015] The PaddleSeg-Lite model is used to predict the road in the road image to obtain a predicted road image.

[0016] Preferably, the road training set is trained using a PaddleSeg-Lite model to obtain a PaddleSeg-Lite model. The specific operation steps are as follows:

[0017] Add an encoder and a decoder to the PaddleSeg-Lite model according to the DEM structure;

[0018] Inputting the preprocessed road image in the road training set into the convolution layer of the encoder to extract and output a feature map; re-inputting the feature map into the convolution layer of the encoder to perform a downsampling operation to adjust the number of channels to adjust the spatial resolution of the feature map, thereby obtaining an adjusted feature map;

[0019] Using the convolutional layer of the decoder to perform an upsampling operation to adjust the number of channels, the spatial resolution of the feature map is adjusted;

[0020] The PaddleSeg-Lite model is trained for recognition using the feature map after adjusting the spatial resolution to obtain the PaddleSeg-Lite model.

[0021] Preferably, the encoder comprises four convolutional layers and two skip connections;

[0022] The decoder consists of three convolutional layers.

[0023] Preferably, the DEM structure further comprises:

[0024] The DEM structure utilizes channel attention mechanism and spatial attention mechanism to enhance the feature map.

[0025] Preferably, an RGBM-LinKnet model is constructed, and the predicted road image is optimized by the RGBM-LinKnet model to obtain a final road image. The specific operation steps are as follows:

[0026] All predicted road images are used as a predicted road image dataset, and road mask labels are annotated on the predicted road image dataset;

[0027] Input the missing road segments of the road mask label and the road images in the road data set into the RGBM-LinKnet model for prediction task training to obtain a preliminary RGBM-LinKnet model;

[0028] Re-inputting the road mask label and the road image in the road data set into the preliminary RGBM-LinKnet model for adjustment to obtain an adjusted RGBM-LinKnet model;

[0029] The RGBM-LinKnet model after input adjustment of the predicted road image and the road image is optimized to obtain a final road image of the road mask of the optimized predicted road image, and a road mask label corresponding to the final road image is output.

[0030] Preferably, the road mask tag includes a complete road mask and road missing segments.

[0031] Preferably, the prediction task training includes a road direction prediction task and a road pixel prediction task.

[0032] Accordingly, the present invention also proposes a fine road extraction system based on multi-task learning, comprising: a prediction module; an optimization module; a final module;

[0033] The prediction module is used to obtain a road data set, perform road prediction of the paddleseg-lite model on the road data set, and extract a predicted road image;

[0034] The optimization module is used to construct an RGBM-LinKnet model, and optimize the road mask of the predicted road image through the RGBM-LinKnet model to obtain a final road image;

[0035] The final module is used to further optimize the final road image of the road mask of the optimized predicted road image by morphological operation, and extract the road boundary to obtain a road boundary image.

[0036] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0037] From the analysis of the above-mentioned fine road extraction method and system based on multi-task learning provided by the present invention, it can be known that in specific applications, a number of road images are collected by the recording equipment of the drone, and different types of roads (i.e., roads or non-roads) are annotated to form a road data set; the road data set is preprocessed and divided into a road training set and a road test set, so that the PaddleSeg-Lite model learns how to identify and extract road pixels;

[0038] The road training set is trained by the PaddleSeg-Lite model, so that it can learn to recognize and segment road pixels from the image, and the parameters of the PaddleSeg-Lite model are optimized to reduce the gap between the predicted road image and the actual road annotation; the PaddleSeg-Lite model is tested by the road test set to obtain the tested PaddleSeg-Lite model; the road in the road image is predicted by the PaddleSeg-Lite model to obtain the predicted road image, so as to realize the extraction of the road to verify the performance of the PaddleSeg-Lite model and obtain the final road prediction result;

[0039] Furthermore, the predicted road image dataset is a dataset after road prediction, and annotated with road mask labels including complete road masks and missing road segments; the missing road segments of the road mask labels and the road images in the road dataset are input into the RGBM-LinKnet model for prediction task training to obtain a preliminary RGBM-LinKnet model; the road mask labels and the road images in the road dataset are re-input into the preliminary RGBM-LinKnet model to adjust the model and optimize the preliminary RGBM-LinKnet model. The input data at this stage is more accurate than that in the pre-training stage, thereby making the fine-tuning process more effective and improving the recognition ability of the RGBM-LinKnet model; the adjusted RGBM-LinKnet model is used to optimize the predicted road image to obtain a road mask, and the PaddleSeg-Lite model is used to generate the road to optimize the predicted road image, so that the PaddleSeg-Lite model can extract road information more accurately when processing real scenes;

[0040] Furthermore, the road mask of the optimized predicted road image is further optimized by morphological operations, and the road mask of the optimized predicted road image is "expanded", which helps to eliminate small black holes and enhance the continuity of the road area; after corrosion, small noise points are removed; and the final largest area is used as the final road mask to remove small, erroneous prediction areas, so that the road mask of the predicted road image is optimized twice, making the predicted road recognition more accurate; the road boundary of the road mask of the secondarily optimized predicted road image is extracted through edge detection, and a clear road boundary can be obtained, thereby identifying and detecting the road. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 An overall flow chart of a fine road extraction method based on multi-task learning provided in Example 1 of the present invention;

[0043] Figure 2 A schematic diagram of a paddleseg-lite model of a fine road extraction method based on multi-task learning provided in Example 1 of the present invention;

[0044] Figure 3 A schematic diagram of a convolution process of an encoder of a fine road extraction method based on multi-task learning provided in Embodiment 1 of the present invention;

[0045] Figure 4 A convolution diagram of a decoder of a fine road extraction method based on multi-task learning provided in the first embodiment of the present invention;

[0046] Figure 5 A schematic diagram of an attention mechanism of a DEM structure of a fine road extraction method based on multi-task learning provided in Example 1 of the present invention;

[0047] Figure 6 A schematic diagram of an RGBM-LinKnet model of a fine road extraction method based on multi-task learning provided in Example 1 of the present invention;

[0048] Figure 7 A flowchart of a fine road extraction system based on multi-task learning provided in Example 2 of the present invention.

[0049] Label: prediction module 10; optimization module 20; final module 30. DETAILED DESCRIPTION

[0050] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0052] Embodiment 1

[0053] like Figure 1 As shown, the present invention proposes a fine road extraction method based on multi-task learning, which includes the following steps:

[0054] S10: Acquire a road data set, perform road prediction of the paddleseg-lite model on the road data set, and extract a predicted road image;

[0055] It should be noted that the acquisition of datasets usually involves collecting image data containing road information. Common sources include public image databases or images obtained by taking pictures. The collected road image datasets should contain diverse scenes to ensure the generalization ability of the model in different environments and conditions. The collected image data are processed using the PaddleSeg-Lite framework. First, data annotation is required to mark the road area in the image. The PaddleSeg-Lite model is trained, and the trained model is used to predict new images, output the classification results of each pixel, and generate a binary mask of the road (that is, the separation of road and non-road areas, that is, complete roads and incomplete roads with missing roads; according to the prediction results of the model, a binary mask image of the road part is extracted, usually the white area represents the predicted road, and the black area represents the background; the road mask is used to extract the road part from the input image, and help identify and analyze the road location by comparing with the original image);

[0056] S20: constructing an RGBM-LinKnet model, and optimizing a road mask of the predicted road image by using the RGBM-LinKnet model to obtain a final road image;

[0057] It should be noted that RGBM-LinKnet is a deep learning model based on the LinkNet architecture, which combines RGB image and depth image information. When constructing the RGBM-LinKnet model, it is necessary to design the network structure, including the encoder and decoder parts, to ensure that multi-scale features can be effectively extracted; training the RGBM-LinKnet model requires the use of the road mask of the previous predicted road image as a label (that is, the road mask is represented as a binary image (it can also be a multi-channel segmented image), which indicates which pixels in the image belong to the road area and which do not belong to the road area (that is, incomplete roads), where the value of each pixel indicates whether the pixel is part of the road; when predicting through the model, a probability map is usually obtained, indicating the probability of each pixel belonging to the road, such as: by setting a threshold (for example, 0.5), the probability map is converted into a binary road mask, 1 represents road, 0 represents non-road (and incomplete road), and combined with the original image data (that is, when training the RGBM-LinKnet model, the predicted road image obtained is The row is edited into a dataset of predicted road images, which belongs to the predicted road images obtained after prediction by the paddleseg-lite model in step S10, so the RGBM-LinKnet model is trained by the newly collected dataset of predicted road images) to build the RGBM-LinKnet model; after being processed by the RGBM-LinKnet model, the optimized road mask of the predicted road image is finally obtained (that is, the road mask is a binary image output by the model, in which the value of each pixel indicates whether the pixel belongs to a part of the road, and the final road image is obtained by optimizing the values ​​of the pixels belonging to the road. The final road image is the result obtained after optimizing the road mask, which represents the refined extraction of the RGBM-LinKne model in the actual road scene). This mask should be more accurate than the initial prediction and better conform to the shape and position of the actual road;

[0058] S30: further optimizing the final road image of the road mask of the optimized predicted road image through morphological operations, and extracting the road boundary to obtain a road boundary image.

[0059] It should be noted that morphological operation is an image processing technique used to process binary images. In the process of processing the road mask of the predicted road image, the objects in the road mask of the optimized predicted road image are "expanded" to fill small holes and connect adjacent pixel areas. During the expansion process, each foreground pixel in the image (for example, the value is 1 or 255) will affect the surrounding neighborhood, so that the pixels in the neighborhood also become foreground. This helps to eliminate small black holes and enhance the continuity of the road area. Subsequently, erosion is performed in the opposite way to the expansion. Erosion can remove small noise points and reduce the area of ​​the foreground area. During the erosion process, the central pixel will remain as the foreground only when all pixels in the neighborhood are foreground pixels, which is very effective for removing isolated noise points. Finally, the incorrectly predicted roads are filtered out by retaining the largest connected area. The connected component analysis algorithm is used to identify all connected areas in the image and calculate their sizes. Then, the largest area is selected as the final road mask, which can remove small, incorrectly predicted areas, thereby performing a second optimization on the final road image of the road mask of the optimized predicted road image.

[0060] The road boundary is extracted from the final road image of the secondary optimization, and the road boundary can be extracted from the road mask of the predicted road image of the secondary optimization using the edge detection operator; through edge detection, a clear road boundary can be obtained, so as to identify and detect the road;

[0061] The real road image is collected and input into the paddleseg-lite model to identify the real predicted road image; then the real predicted road image is continuously optimized through the RGBM-LinKnet model and morphological operations, and finally the final real road boundary image is obtained.

[0062] Specifically, in step S10, a road data set is obtained, and the road prediction of the paddleseg-lite model is performed on the road data set to extract a predicted road image. The specific operation steps are as follows:

[0063] S11: collecting a number of road images of the road through a video recording device of a drone, and annotating the number of road images to obtain a road data set;

[0064] It should be noted that the road is photographed and collected by using the recording equipment equipped by the drone. Each road image includes different types of roads (i.e. roads or non-roads), and these collected road images are annotated (i.e., the annotated road is the different types of roads included in each road image) to form a road dataset, which can play an outstanding training role and the final accuracy of the subsequent paddleseg-lite model training;

[0065] S12: preprocessing the road data set to obtain a preprocessed road data set;

[0066] Dividing the preprocessed road data set into a road training set and a road test set;

[0067] It should be noted that the PaddleSeg-Lite model in the PaddlePaddle framework is used for training. PaddleSeg-Lite is a lightweight and efficient semantic segmentation model that is particularly suitable for mobile devices and edge computing scenarios. By inputting the preprocessed dataset, the model learns how to identify and extract road pixels.

[0068] S13: Using the PaddleSeg-Lite model to train the road training set to obtain a PaddleSeg-Lite model;

[0069] It should be noted that at this stage, the PaddleSeg-Lite model provided by the PaddlePaddle framework (an open source deep learning platform) is used for training the PaddleSeg-Lite model; PaddleSeg is a library specifically for semantic segmentation, and the Lite version is designed to be more efficient when running on mobile devices or edge devices;

[0070] The training process involves using the preprocessed data from the previous step to train the PaddleSeg-Lite model so that it can learn to identify and segment road pixels from images. This usually involves multiple iterations of optimizing the parameters of the PaddleSeg-Lite model through the back-propagation algorithm to reduce the gap between the prediction and the actual annotation.

[0071] S14: Testing the PaddleSeg-Lite model using the road test set to obtain a tested PaddleSeg-Lite model;

[0072] Predicting a road in the road image using the PaddleSeg-Lite model to obtain a predicted road image;

[0073] It should be noted that after the PaddleSeg-Lite model training is completed, the trained PaddleSeg-Lite model is used to predict the road test set, which is the road image that is not used in the process of road data set division, to evaluate the generalization ability of the preliminary PaddleSeg-Lite model. Figure 2 As shown;

[0074] By inputting the road test set into the trained PaddleSeg-Lite model, the PaddleSeg-Lite model outputs the classification result of each pixel (for example, marked as road or non-road), thereby realizing the extraction of roads. This process is also called inference and is a key step in verifying the performance of the PaddleSeg-Lite model.

[0075] Ultimately, the extracted road pixels will form a new image showing the PaddleSeg-Lite model's prediction results for the road, which can be further used for analysis, visualization, or other applications.

[0076] Specifically, in step S13, the road training set is trained using the PaddleSeg-Lite model to obtain the PaddleSeg-Lite model. The specific operation steps are as follows:

[0077] S131: adding an encoder and a decoder to the PaddleSeg-Lite model according to the DEM structure;

[0078] The encoder includes four convolutional layers and two skip connections;

[0079] The decoder includes three convolutional layers;

[0080] It should be noted that the convolution layer is the basic building block of the convolutional neural network, which is mainly used for feature extraction and usually adopts a 3x3 convolution kernel. Through the convolution operation, the model can automatically learn important features in the image, such as edges, textures, and shapes. In the PaddleSeg-Lite model, convolution layers are widely used in the encoder part to help extract features from the input image and reduce the spatial resolution of the feature map; the encoder is the part of the model responsible for extracting and encoding features from the input image;

[0081] The decoder is a part of the neural network architecture and is usually used in conjunction with the encoder. Its main task is to convert the context information generated by the encoder into the target output sequence.

[0082] The DEM structure usually refers to the depthwise separable convolution, which plays an important role in building efficient models. It includes depthwise convolution and pointwise convolution, and each input channel applies a convolution kernel independently. If the input has C channels, the depthwise convolution will use C convolution kernels, each of which is k×k in size, and perform convolution operations on each channel separately. The output of the depthwise convolution is combined and transformed using a 1×1 convolution kernel. This step is actually a linear transformation in the channel dimension, so that the spatial resolution of the channel can be adjusted for the subsequent feature map transformation.

[0083] S132: inputting the road image in the preprocessed road training set into the convolution layer of the encoder to extract and output a feature map; re-inputting the feature map into the convolution layer of the encoder to perform a downsampling operation to adjust the number of channels to adjust the spatial resolution of the feature map, thereby obtaining an adjusted feature map;

[0084] It should be noted that in PaddleSeg-Lite, the encoder usually consists of multiple convolutional layers and jump connections, such as Figure 3 As shown in the figure; the first convolution layer uses a 3x3 convolution kernel to convert the number of channels from m to n (that is, m here represents the number of channels of the input feature map, and n represents the number of channels of the output feature map after convolution; therefore, the first convolution layer (that is, the first convolution layer converts the number of channels from m to n and reduces the spatial resolution of the feature map) converts the number of channels of the input feature map from m to n), and reduces the spatial resolution of the feature map through a downsampling operation with a stride of 2 (that is, downsampling refers to the process of reducing the spatial resolution of the feature map after the convolution operation. In this description, downsampling is achieved through a convolution operation with a stride of 2, which means that in each convolution, every other pixel is sampled once, so that the width and height of the feature map are halved (that is, the spatial resolution is reduced to half)). The remaining three convolution layers also use a 3x3 convolution kernel (that is, the remaining three convolution layers refer to the number of input and output channels are n, and these convolution layers can further extract features to obtain a more optimized model), and the number of input and output channels is n. Specifically, the output of the first layer of convolution is added to the output of the second layer of convolution through the first skip connection (i.e., the first skip connection adds the output of the first layer of convolution to the output of the second layer of convolution; this means that on the basis of the features of the second layer of convolution, the features extracted by the first layer of convolution are added, thereby retaining the information of the early features), and the output of the third layer of convolution is added to the output of the fourth layer of convolution through the second skip connection (i.e., the second skip connection adds the output of the third layer of convolution to the output of the fourth layer of convolution, and similarly, the features extracted by the third layer of convolution are added to enhance the output of the fourth layer);

[0085] Through this residual learning mechanism, the network can effectively alleviate the gradient vanishing problem, improve the training efficiency and feature expression ability of deep networks, improve the subsequent recognition ability of PaddleSeg-Lite models, and reduce the spatial resolution of feature maps; Residual Learning: By introducing jump connections, the network not only learns outputs (such as the outputs of the second or fourth layers), but also learns how to adjust from inputs (such as the outputs of the first and third layers). This method can help the network learn better, especially in very deep networks, significantly alleviate the gradient vanishing problem, and improve training efficiency and feature expression ability.

[0086] The reason why downsampling reduces the spatial resolution of feature maps is that as the number of network layers increases, the size of feature maps will remain unchanged or gradually increase if they are not downsampled, which will lead to a sharp increase in the amount of calculation. Reducing the size of feature maps by downsampling can effectively reduce the computational complexity and the number of parameters, thereby reducing the computational burden of the model and thus reducing the amount of calculation; at the same time, downsampling helps the model capture a wider range of contextual information, because the model can identify global or large-scale features through a larger receptive field; and, because the model needs to learn fewer parameters, reducing the size of feature maps and the number of parameters also helps reduce the risk of overfitting;

[0087] S133: using the convolutional layer of the decoder to perform an upsampling operation to adjust the number of channels and adjust the spatial resolution of the feature map;

[0088] It should be noted that the three convolutional layers included in the DEM structure are: the first and third convolutional layers are 1x1 convolutions, the second convolution layer is 3x3 convolution; the first convolution layer (1x1 convolution), such as Figure 4 As shown in the figure, by using the 1x1 convolution kernel of the first convolution layer, the number of input channels is reduced from m to m / 4 (that is, the 1x1 convolution kernel of the first convolution layer is mainly used to adjust the number of channels. This dimension reduction operation of the feature map helps to reduce the computational complexity while retaining important features); and the second convolution layer (3x3 convolution): use the second convolution 3x3 convolution kernel for full convolution operation (that is, full-conv; the second convolution layer keeps the number of channels at m / 4, and doubles the spatial resolution of the feature map through upsampling operation, which means that the width and height of the feature map will increase, so that the details in the road image can be better captured); and then through the third convolution layer (1x1 convolution): once again use the third convolution 1x1 convolution kernel to restore the number of channels of the feature map from m / 4 to m. The purpose of this step is to restore the number of channels so that the subsequent layers can use the complete feature representation;

[0089] S134: Perform recognition training on the PaddleSeg-Lite model using the feature map after adjusting the spatial resolution to obtain the PaddleSeg-Lite model.

[0090] Specifically, in step S131, the DEM structure further includes:

[0091] S1311: The DEM structure uses a channel attention mechanism and a spatial attention mechanism to enhance the feature map;

[0092] It should be noted that, first of all, the channel attention mechanism of the DEM structure is used, such as Figure 5 As shown in the figure, the input feature map H×W×C (i.e., length is H, width is W, and number of channels is C) is globally average pooled to generate a feature vector; the feature vector is processed through three fully connected layers to obtain a channel weight vector (i.e., through the weight vector, the obtained PaddleSeg-Lite model can effectively extract features at different levels and gradually learn more complex features from the original feature map), and the weight vector is multiplied channel by channel with the original feature map (i.e., the original feature map is the input feature map, and a matrix is ​​formed after passing through the input layer, and the feature map H×W×C changes) (i.e., in the convolution layer, the multiplication of the weight vector and the original feature map actually refers to the convolution operation, and the convolution kernel (or filter) slides on the original feature map and is calculated based on the local area; at each window position, the weight of the convolution kernel is multiplied element by element with the corresponding area of ​​the original feature map, and then these products are summed to form the pixels of the new feature map, thereby achieving the effect of enhancing the feature map), achieving the feature enhancement effect in the channel dimension;

[0093] Spatial attention mechanism: The channel-enhanced feature map enters the spatial attention mechanism; first, global maximum pooling and global average pooling are performed to generate two feature maps; after splicing these two feature maps, they are processed through a 3x3 convolution layer to generate a spatial attention map; finally, the spatial attention map is multiplied with the channel-enhanced feature map to output an enhanced feature map with a size of H×W×C; the spatial attention mechanism can focus on the key spatial positions in the input feature map, thereby better capturing the important areas in the image;

[0094] Through the combined effect of channel attention and spatial attention mechanisms, the subsequent PaddleSeg-Lite model can adaptively capture important features while suppressing unnecessary interference and improving overall performance.

[0095] Specifically, in step S20, an RGBM-LinKnet model is constructed, and the road mask of the predicted road image is optimized by the RGBM-LinKnet model to obtain a final road image. The specific operation steps are as follows:

[0096] S21: taking all predicted road images as a predicted road image dataset, and annotating the predicted road image dataset with road mask labels;

[0097] The road mask tag contains the complete road mask and the missing road segments;

[0098] It should be noted that the RGBM-LinKnet model usually requires a large annotated training data set so that the model can learn the details in complex scenes. Since it combines RGB images (that is, RGB images are the original road images of the sample prediction images in the predicted road image data set) and depth information, the cooperation of these two types of data is required; the design of the RGBM-LinKnet model emphasizes the fusion of features of different scales and is suitable for segmentation tasks in complex scenes. It performs well when processing objects with clear boundaries such as roads and buildings;

[0099] The predicted road image dataset is a dataset of predicted roads, and the road mask labels are annotated to include complete road masks and road missing fragments. During the training process, the missing road masks are generated by randomly missing fragments, which simulates the incomplete information in the real world and allows the RGBM-LinKnet model to learn to make predictions under imperfect data.

[0100] S22: Inputting the missing road segments of the road mask label and the road images in the road data set into the RGBM-LinKnet model for prediction task training to obtain a preliminary RGBM-LinKnet model;

[0101] The prediction task training includes a road direction prediction task and a road pixel prediction task;

[0102] It should be noted that the generated Mask with missing information is combined with the corresponding RGB image (i.e., the road image of the road dataset) as the input data of the model, so that the RGBM-LinKnet model can learn how to infer the complete road situation from partial information;

[0103] According to the tasks in the pre-training phase, the road pixel prediction task: This task aims to predict whether each pixel in the image belongs to the road. This process enables the model to classify each pixel in the image into two categories, thereby generating a prediction mask for the road; at the same time, according to the road direction prediction task: The goal of this task is to learn the directional relationship between the pixels of the same road and adjacent roads. By calculating the direction vector between two pixels, the model can obtain the connectivity information of the road. Such constraints help ensure the continuity and connectivity of the road when predicting road images. These two tasks work together to promote the model to learn more comprehensive features, so that it performs better when dealing with complex scenes;

[0104] S23: re-inputting the road mask label and the road image in the road data set into the preliminary RGBM-LinKnet model for adjustment to obtain an adjusted RGBM-LinKnet model;

[0105] It should be noted that after completing the pre-training task, the model is fine-tuned; the Mask result obtained by PaddleSeg-Lite and the original road RGB image of the predicted road image (i.e. the road image of the road dataset) are input into the model. The input data at this stage is more accurate than that in the pre-training stage, making the fine-tuning process more effective; the road direction and road pixel prediction tasks are still performed during the fine-tuning process. This ensures the adaptability and accuracy of the model on new data, and further optimizes the road extraction results, such as Figure 6 shown.

[0106] S24: Optimizing the RGBM-LinKnet model after input adjustment of the predicted road image and the road image to obtain a final road image of the road mask of the optimized predicted road image, and outputting a road mask label corresponding to the final road image;

[0107] It should be noted that, first, the PaddleSeg-Lite model is used to generate a preliminary mask of the road. This step provides the model with basic predictions when processing real scenes; optimized prediction results: the obtained mask is input into the RGBM-LinkNet model together with the RGB image. By combining this information, the RGBM-LinkNet model can generate more accurate road extraction results; after optimization, the model will output an improved "road mask", which represents the road area marked in the predicted image. The final road image is obtained after these optimization steps, with a more accurate road area extraction result; the model not only outputs the final optimized road image, but also generates corresponding road mask labels, which refer to the specific identification of the road area, such as: road boundaries, etc.

[0108] Embodiment 2

[0109] like Figure 7 As shown, accordingly, the present invention also proposes a fine road extraction system based on multi-task learning, including: a prediction module 10; an optimization module 20; a final module 30;

[0110] The prediction module 10 is used to obtain a road data set, perform road prediction of the paddleseg-lite model on the road data set, and extract a predicted road image;

[0111] The optimization module 20 is used to construct an RGBM-LinKnet model, and optimize the road mask of the predicted road image through the RGBM-LinKnet model to obtain a final road image;

[0112] The final module 30 is used to further optimize the final road image of the road mask of the optimized predicted road image by morphological operation, and extract the road boundary to obtain a road boundary image.

[0113] In summary, the present invention proposes a method and system for fine road extraction based on multi-task learning. The method collects a number of road images through the video recording equipment of the drone, and labels different types of roads (i.e., roads or non-roads) to form a road dataset; the road dataset is preprocessed and divided into a road training set and a road test set, so that the PaddleSeg-Lite model learns how to identify and extract road pixels;

[0114] The road training set is trained by the PaddleSeg-Lite model, so that it can learn to recognize and segment road pixels from the image, and the parameters of the PaddleSeg-Lite model are optimized to reduce the gap between the predicted road image and the actual road annotation; the PaddleSeg-Lite model is tested by the road test set to obtain the tested PaddleSeg-Lite model; the road in the road image is predicted by the PaddleSeg-Lite model to obtain the predicted road image, so as to realize the extraction of the road to verify the performance of the PaddleSeg-Lite model and obtain the final road prediction result;

[0115] Furthermore, the predicted road image dataset is a dataset after road prediction, and annotated road mask labels include complete road masks and missing road segments; the missing road segments of the road mask labels and the road images in the road dataset are input into the RGBM-LinKnet model for prediction task training to obtain a preliminary RGBM-LinKnet model; the road mask labels and the road images in the road dataset are re-input into the preliminary RGBM-LinKnet model to adjust the model and optimize the preliminary RGBM-LinKnet model. The input data at this stage is more accurate than that in the pre-training stage, thereby making the fine-tuning process more effective and improving the recognition ability of the RGBM-LinKnet model; the adjusted RGBM-LinKnet model is used to optimize the predicted road image to obtain a road mask, and the PaddleSeg-Lite model is used to generate the road to optimize the predicted road image, so that the PaddleSeg-Lite model can extract road information more accurately when processing real scenes;

[0116] Furthermore, the road mask of the optimized predicted road image is further optimized by morphological operations, and the road mask of the optimized predicted road image is "expanded", which helps to eliminate small black holes and enhance the continuity of the road area; after corrosion, small noise points are removed; and the final largest area is used as the final road mask to remove small, erroneous prediction areas, so that the road mask of the predicted road image is optimized twice, making the predicted road recognition more accurate; the road boundary of the road mask of the secondarily optimized predicted road image is extracted through edge detection, and a clear road boundary can be obtained, thereby identifying and detecting the road.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace part or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fine road extraction method based on multi-task learning, characterized in that: The steps are as follows: Acquire a road data set, perform road prediction of the paddleseg-lite model on the road data set, and extract a predicted road image; Constructing an RGBM-LinKnet model, and optimizing a road mask of a predicted road image by using the RGBM-LinKnet model to obtain a final road image; The final road image of the road mask of the optimized predicted road image is further optimized by morphological operation, and the road boundary is extracted to obtain a road boundary image.

2. According to claim 1, a fine road extraction method based on multi-task learning is characterized in that: Obtain a road dataset, perform road prediction of the paddleseg-lite model on the road dataset, and extract the predicted road image. The specific steps are as follows: A plurality of road images are collected by a recording device of a drone, and the plurality of road images are annotated to obtain a road data set; Preprocessing the road data set to obtain a preprocessed road data set; Dividing the preprocessed road data set into a road training set and a road test set; Using the PaddleSeg-Lite model to train the road training set to obtain a PaddleSeg-Lite model; Using the road test set to test the PaddleSeg-Lite model to obtain a tested PaddleSeg-Lite model; The PaddleSeg-Lite model is used to predict the road in the road image, and a predicted road image is extracted.

3. According to claim 2, a fine road extraction method based on multi-task learning is characterized in that: The road training set is trained using the PaddleSeg-Lite model to obtain the PaddleSeg-Lite model. The specific steps are as follows: Add an encoder and a decoder to the PaddleSeg-Lite model according to the DEM structure; The preprocessed road images in the road training set are input into the convolutional layer in the encoder to extract the output feature map; Re-inputting the feature map into the convolution layer of the encoder to perform a downsampling operation to adjust the number of channels to adjust the spatial resolution of the feature map, thereby obtaining an adjusted feature map; Using the convolutional layer of the decoder to perform an upsampling operation to adjust the number of channels, the spatial resolution of the feature map is adjusted; The PaddleSeg-Lite model is trained for recognition using the feature map after adjusting the spatial resolution to obtain the PaddleSeg-Lite model.

4. According to claim 3, a fine road extraction method based on multi-task learning is characterized in that: The encoder includes four convolutional layers and two skip connections; The decoder consists of three convolutional layers.

5. According to claim 4, a fine road extraction method based on multi-task learning is characterized in that: The DEM structure further includes: The DEM structure utilizes channel attention mechanism and spatial attention mechanism to enhance the feature map.

6. According to claim 5, a fine road extraction method based on multi-task learning is characterized in that: Construct an RGBM-LinKnet model, and optimize the predicted road image through the RGBM-LinKnet model to obtain the final road image. The specific operation steps are as follows: All predicted road images are used as a predicted road image dataset, and road mask labels are annotated on the predicted road image dataset; Input the missing road segments of the road mask label and the road images in the road data set into the RGBM-LinKnet model for prediction task training to obtain a preliminary RGBM-LinKnet model; Re-inputting the road mask label and the road image in the road data set into the preliminary RGBM-LinKnet model for adjustment to obtain an adjusted RGBM-LinKnet model; The RGBM-LinKnet model after input adjustment of the predicted road image and the road image is optimized to obtain a final road image of the road mask of the optimized predicted road image, and a road mask label corresponding to the final road image is output.

7. The method for extracting fine roads based on multi-task learning according to claim 6, characterized in that: The road mask tag contains a complete road mask and road missing segments.

8. The method for extracting fine roads based on multi-task learning according to claim 7, characterized in that: The prediction task training includes a road direction prediction task and a road pixel prediction task.

9. A fine road extraction system based on multi-task learning, comprising: Prediction module; Optimization module; Final module; The prediction module is used to obtain a road data set, perform road prediction of the paddleseg-lite model on the road data set, and extract a predicted road image; The optimization module is used to construct an RGBM-LinKnet model, and optimize the road mask of the predicted road image through the RGBM-LinKnet model to obtain a final road image; The final module is used to further optimize the final road image of the road mask of the optimized predicted road image by morphological operation, and extract the road boundary to obtain a road boundary image.

Images