Method and program for identifying drivable area of unstructured road based on deep learning

Abstract

According to the deep learning-based unstructured road travelable area identification method and program, a road image is input into a travelable area identification model for image identification, and a road identification result is obtained. A backbone network of the driving area identification model is provided with a C2f-iRMB module, a neck network is provided with a feature fusion module, and the segmentation detection head is fused and integrated with a SeaFormer module. Compared with an original YOLOv8n-seg model, the driving area identification model integrates position information in a feature image into channel attention through a C2f-iRMB module, semantic segmentation is performed on a road image through a neck network, and target detection and target image segmentation are performed through head segmentation detection. The accuracy, the recall rate, the mAP, the FPS and the FLOPS of the travelable area identification model are greatly improved, and the method is suitable for travelable area identification and segmentation of unstructured roads.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more particularly to a method and program for identifying drivable areas on unstructured roads based on deep learning. Background Art

[0002] In the field of autonomous driving and intelligent transportation systems, semantic segmentation of road scenes is an important research task, which aims to classify pixels in road images according to their semantic labels, such as segmenting roads, vehicles, pedestrians, etc. In unstructured road environments, there are various complex objects and backgrounds in road traffic scenes, such as trees, buildings, pedestrians, etc., making the semantic segmentation task of road scenes more challenging.

[0003] Data preprocessing is a key preliminary step, which includes techniques such as denoising, enhancement, filtering and segmentation, aiming to improve image quality and lay the foundation for subsequent analysis. Feature extraction is a key follow-up link, among which deep learning networks such as convolutional residual networks, long short-term memory networks, recurrent neural networks and convolutional neural networks (CNN) are widely used. CNN is particularly favored because it can automatically learn features from images. It effectively captures the complex features of images through a multi-layer structure, and then uses it to identify and classify targets. The semantic segmentation algorithm is currently widely used in the U-Net series, which has high accuracy but poor real-time performance. The YOLO (You Only Look Once) series is widely used in the industrial field, combining feature extraction and detection, and quickly detecting targets through gridded images. It is fast but has relatively low accuracy.

[0004] With the continuous development of autonomous driving technology, detecting drivable areas is an important link in the driving process. In particular, compared with structured roads, unstructured roads have a series of complex problems such as blurred road lines, overlapping obstacles and drivable areas. Compared with the lane line detection technology commonly used in the field of autonomous driving, to find drivable areas on unstructured roads, the application of instance segmentation or semantic segmentation algorithms will be more effective, and the drivable areas found will be more feasible. It is worth mentioning that the application of semantic segmentation technology in the field of autonomous driving requires not only high accuracy, but also real-time performance. In particular, in terms of real-time performance, the current on-board recorded video stream is about 40FPS, while the current semantic segmentation algorithm emphasizes accuracy. Therefore, the development of a semantic segmentation algorithm that takes into account both real-time and accuracy is crucial to the development of drivable area detection on unstructured roads. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present application provides a method for identifying drivable areas of unstructured roads based on deep learning, comprising:

[0006] S1, obtaining a road image;

[0007] S2, preprocessing the road image;

[0008] S3, constructing a drivable area recognition model, and pre-training the drivable area recognition model according to a road image training set;

[0009] The network structure of the drivable area recognition model includes: an input end, a backbone network, a neck network, and a segmentation detection head;

[0010] The backbone network is provided with a C2f-iRMB module, which is used for feature extraction and integrates the position information in the feature image into the channel attention;

[0011] The neck network is used to perform semantic segmentation on the road image;

[0012] The segmentation detection head is used for target detection and target image segmentation;

[0013] S4. Input the road image into the drivable area recognition model to perform image recognition to obtain a road recognition result.

[0014] In one implementation, the C2f-iRMB module includes a CBL submodule and an iRMB submodule. The input vector is first subjected to feature extraction by the first CBL submodule, and then the output of the first CBL submodule is segmented, and the segmented vectors are output to the iRMB submodule for feature extraction, and finally the outputs of the iRMB submodules are spliced.

[0015] In one implementation, the C2f-iRMB module is disposed at the p2 layer, the p4 layer, the p6 layer, and the p8 layer.

[0016] In one embodiment, the neck network includes: an upsampling module, a feature fusion module and a C2f2_3d module.

[0017] In one embodiment, the feature fusion module outputs to the C2f2_3d module through a spatial attention mechanism and a channel attention mechanism.

[0018] In one embodiment, the feature fusion module is located in the p11 layer, the p14 layer, the p17 layer and the p20 layer.

[0019] In one embodiment, the segmentation detection head includes: a SeaFormer module, a CBL module and a Conv2d module.

[0020] The SeaFormer module is a Squeeze-enhanced Axial attention mechanism.

[0021] In one embodiment, before constructing the drivable area recognition model and pre-training the drivable area recognition model according to the road image training set, the method further includes:

[0022] Obtain road image dataset;

[0023] The road image data set is divided into a road image training set, a road image test set and a road image verification set; wherein the road image is a structured or unstructured road image with annotated obstacle categories.

[0024] In one embodiment, after constructing the drivable area recognition model and pre-training the drivable area recognition model according to the road image training set, the method further includes:

[0025] The drivable area recognition model is tested according to the road image test set.

[0026] A second aspect of the present application provides a deep learning-based unstructured road drivable area identification program, and when the program product is executed by a processor, the unstructured road drivable area identification method as described in the present application is implemented.

[0027] The technical solution provided by this application may have the following beneficial effects:

[0028] Since traditional convolution may lose the spatial information of the image, iRMB absorbs the efficiency of CNN-like to simulate short-distance dependencies and the dynamic modeling ability of Transfommer-like to learn long-distance interactions, and can integrate the position information in the feature map into the channel attention. Therefore, iRMB can better adapt to the complex and changeable unstructured road environment, thereby enhancing the segmentation performance.

[0029] In this application, the drivable area recognition method based on the iss-YOLO-seg algorithm can more effectively learn image pixel-level features and improve the accuracy of target detection.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a flow chart of a method for identifying a drivable area on an unstructured road as shown in an embodiment of the present application;

[0032] Figure 2A network structure diagram of a drivable area identification model shown in an embodiment of the present application;

[0033] Figure 3 A network structure diagram of a C2f-iRMB module shown in an embodiment of the present application;

[0034] Figure 4 A network structure diagram of a feature fusion module shown in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of a program product shown in an embodiment of the present application;

[0036] Figure 6 A PR curve comparison diagram of the drivable area identification model shown in the embodiment of the present application;

[0037] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application. DETAILED DESCRIPTION

[0038] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0039] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0041] Embodiment 1

[0042] In the automobile's autonomous driving technology, the target detection and segmentation algorithm is the most important part for the autonomous driving system.

[0043] The autonomous driving system needs to collect road scene images through sensors such as cameras and radars, detect pedestrians, lanes, vehicles or road signs in the current road scene images, and obtain information such as the type and position of the target. The autonomous driving system determines the drivable area of ​​the car based on the type and position of the target.

[0044] Road scene images include structured road scenes and unstructured road scenes. Structured road scenes refer to roads with clear lane divisions and accurate road markings, such as expressways. Unstructured roads refer to roads with unclear lane lines and missing road markings, such as rural roads.

[0045] Therefore, the accuracy of target detection and target segmentation in road scene images directly affects whether the autonomous driving system can cope with various road scenes.

[0046] At present, the drivable area recognition method based on the YOLOv8n-seg algorithm has the characteristics of small parameters and fast detection speed. The YOLOv8n-seg algorithm is suitable for the real-time requirements of autonomous driving, but the accuracy of the YOLOv8n-seg algorithm needs to be improved.

[0047] In order to further improve the YOLOv8n-seg algorithm and improve the accuracy of drivable area recognition, the present application embodiment provides a method for identifying drivable areas on unstructured roads based on deep learning, such as Figure 1 As shown, the following steps are included:

[0048] S1, obtaining a road image;

[0049] S2, preprocessing the road image;

[0050] S3, constructing a drivable area recognition model, and pre-training the drivable area recognition model according to a road image training set;

[0051] S4. Input the road image into the drivable area recognition model to perform image recognition to obtain a road recognition result.

[0052] The drivable area recognition model of the embodiment of the present application is an improvement made on the basis of the YOLOv8n-seg algorithm.

[0053] Figure 2 It is a network structure diagram of the drivable area identification model shown in an embodiment of the present application.

[0054] like Figure 2 As shown, the network structure of the drivable area recognition model includes: an input end, a backbone network, a neck network, and a segmentation detection head.

[0055] Based on the YOLOv8n-seg algorithm, it can be understood that the backbone network is used to extract features from the road image, the neck network is used to perform semantic segmentation on the road image, and the segmentation detection head is used for target detection and target image segmentation.

[0056] Furthermore, the drivable area recognition model of the embodiment of the present application is constructed and trained based on the iss-YOLO-seg algorithm.

[0057] Different from the existing YOLOv8n-seg algorithm, the backbone network of the iss-YOLO-seg algorithm is equipped with a C2f-iRMB module to replace the original C2f module.

[0058] Further, such as Figure 2 As shown, the C2f-iRMB modules are arranged at the p2 layer, the p4 layer, the p6 layer and the p8 layer.

[0059] like Figure 3 As shown in the network structure diagram of the C2f-iRMB module, the C2f-iRMB module includes a CBL submodule and an iRMB submodule. The input vector first passes through the first CBL submodule for feature extraction, and then the output of the first CBL submodule is segmented, and the segmented vector is output to the iRMB submodule for feature extraction, and finally the output of the iRMB submodule is spliced.

[0060] Since traditional C2f convolution may ignore some details when processing image features, it has the disadvantages of information loss, low computational efficiency, and unsuitability for complex tasks, making it difficult to accurately capture the subtle features of the image.

[0061] Therefore, in the embodiment of the present application, the C2f-iRMB module is provided with an inverted residual structure (ie, Inverted Residual Mobile Block, iRMB for short).

[0062] Since traditional convolution may lose the spatial information of the image, iRMB absorbs the efficiency of CNN-like to simulate short-range dependencies and the dynamic modeling ability of Transformer-like to learn long-range interactions, and is able to integrate the position information in the feature map into the channel attention.

[0063] Therefore, iRMB can better adapt to complex and changeable unstructured road environments, thereby enhancing segmentation performance. The model mAP and speed are nearly 1 / 3 higher than the original YOLOv8n-seg model in terms of map and processing frame rate.

[0064] In an embodiment of the present application, the drivable area recognition method based on the iss-YOLO-seg algorithm can more effectively learn image pixel-level features and improve the accuracy of target detection.

[0065] Embodiment 2

[0066] Based on the unstructured road drivable area identification method shown in Example 1, the iss-YOLO-seg algorithm used in the embodiment of the present application has other differences compared with the existing YOLOv8n-seg algorithm.

[0067] like Figure 2 As shown, in the drivable area recognition model shown in the embodiment of the present application, its neck network includes: an upsampling module, a feature fusion module and a C2f2_3d module.

[0068] Compared with the existing YOLO algorithm, the embodiment of the present application adds a feature fusion module to the neck network.

[0069] The feature fusion module outputs to the C2f2_3d module through the spatial attention mechanism and the channel attention mechanism. The structure of the feature fusion model (i.e., SDI) is as follows: Figure 4 shown.

[0070] In the embodiment of the present application, the feature fusion module is located in the p11 layer, the p14 layer, the p17 layer and the p20 layer. The feature fusion module is linked to different layers of the backbone network, and semantic segmentation is performed on the feature maps output at different layers. Compared with the Concat module of the original model, more underlying semantic features can be retained, and the map and processing frame rate are increased by nearly 1 / 4.

[0071] Furthermore, the original YOLOv8n-seg algorithm directly passes the information to the head for processing after obtaining the features in the convolutional layer, which makes the information large and difficult to process.

[0072] In the embodiment of the present application, the segmentation detection head includes: a SeaFormer module, a CBL module and a Conv2d module.

[0073] The SeaFormer module is the Squeeze-enhanced Axial attention module in the SeaFormer algorithm.

[0074] In the embodiment of the present application, the SeaFormer module can better extract global semantic information and enhance details.

[0075] Embodiment 3

[0076] Based on the unstructured road drivable area detection method based on deep learning in Example 1 and Example 2, the present application embodiment provides a drivable area recognition model training method, including the following steps:

[0077] S301, obtaining a road image dataset;

[0078] In the embodiments of the present application, two public semantic segmentation image datasets are used: the COCO dataset and the IDD dataset.

[0079] The COCO dataset contains 91 categories, 118,287 training images, 5,000 validation images, and more than 40,670 test images. This embodiment of the application extracts 800 of them as a training set, 200 as a test set, and 800 as a validation set. It has multi-mode and multi-resolution characteristics and is suitable for testing model robustness.

[0080] The IDD dataset consists of images obtained by the front camera installed on the car. The car is driven in cities such as Hyderabad, Bangalore and their suburbs. Most of the images are in 1080p resolution, but some images are in 720p and other resolutions. The IDD dataset extracts the boundary coordinates of each mask in the form of coordinates, where the drivable area is marked in purple. In addition to the drivable area, it also includes 38 other obstacle categories, including drivable, non-drivable, biological, driving tools, roadside objects, distant objects and sky. S302, divide the road image dataset into a road image training set, a road image test set and a road image verification set;

[0081] In the embodiment of the present application, the two data sets are respectively divided into a training set, a test set and a validation set.

[0082] Before training begins, since the label format of the COCO dataset and IDD dataset is JSON, which is incompatible with the YOLO format, it needs to be converted into labels files suitable for YOLO.

[0083] Specifically, the road image is a structured or unstructured road image marked with obstacle categories.

[0084] S303, constructing a drivable area recognition model;

[0085] S304, inputting the road image training set into the drivable area recognition model for training;

[0086] In the embodiment of the present application, the model training is based on the 1.12.1 version of the PyTorch framework, and the programming language used is Python, whose version number is 3.8.5. The training and testing of all models are performed in the PyCharm integrated development environment (IDE).

[0087] Specifically, the hardware and software configuration parameters of the model are shown in Table 1.

[0088] equipment parameter system Windows 11 language Python 3.8.5 CPU AMD Ryzen 7 5800H RAM 16G GPU NVIDIA GeForce RTX 3060 Laptop GPU memory 6G YOLO Version YOLOv8.1.9 frame pytoch-1.12.1

[0089] Table 1

[0090] S305: Testing the drivable area recognition model according to the road image test set.

[0091] Specifically, the training parameters of the model are shown in Table 2.

[0092] The embodiment of the present application uses SGD as the optimizer, and adjusts the learning rate through the cosine annealing learning rate scheduler. The initial learning rate is set to 0.01, and the IoU matching threshold between the label and the anchor is set to 0.7. The final learning rate (lrf) is also set to 0.01.

[0093] During the training process, each batch (Batch size) contains 4 samples, and the number of epochs is adjusted according to the difficulty of training different models.

[0094] On the road image dataset, the size of the input image is set to 640 pixels.

[0095]

[0096]

[0097] Table 2

[0098] In step S305, testing the drivable area recognition model requires calculating the model's accuracy (i.e., Precision), recall (i.e., Recall), mean average precision (i.e., mAP), floating-point operations (i.e., FLOPS), and real-time processing frames (i.e., FPS).

[0099] Specifically, the calculation formulas for each indicator of the model are:

[0100]

[0101] like Figure 6As shown, in the COCO dataset, each point on the PR curve of the model shown in the embodiment of the present application and the original model represents the precision and recall rate under a certain threshold. By changing the classification threshold, a series of combinations of precision and recall rate can be obtained, and then these points are plotted on the chart to form the PR curve comparison chart.

[0102] like Figure 6 As shown, the PR curve area of ​​the recognition result of the drivable area recognition model of the present application for the COCO dataset is larger.

[0103] Compared with the existing YOLOv8n-seg algorithm, the test results of the drivable area recognition model shown in the embodiment of the present application in the IDD data set are shown in Table 3.

[0104]

[0105] Table 3

[0106] As can be seen from Table 3, in the IDD dataset, the drivable area recognition model shows good properties on the IDD dataset. Its mAP is improved by 56% compared with the original network structure, the speed is improved by 8%, and the segmentation effect is improved by nearly 50%.

[0107] Furthermore, the embodiment of the present application also tests the module ablation experiment of the drivable area recognition model on the COCO dataset, and the experimental results are shown in Table 4.

[0108]

[0109] Table 4

[0110] In Table 4, the models are numbered 1-6, and each group retains a different combination of modules. Among them, the model numbered 0 eliminates all 5 modules. The module numbers and module names are shown in Table 5. The modules marked with √ in Table 4 are retained.

[0111] Module Number Module Name (1) Feature fusion module (SDI) (2) Inverse residual moving block (iRMB) (3) Cross-modal Transformer Attention Mechanism (CoTAttention) (4) Compressed Enhanced Axial Transformer Attention Mechanism (SeaAttention) (5) Lightweight backbone network (LargeMobilev3backbone)

[0112] Table 5

[0113] In the embodiment of the present application, (1) is an improved module for the neck network, (2) and (5) are improved modules for the backbone network, (3) and (4) are two attention mechanism improvement modules for the segmentation detection head, (3) is the CoTAttention attention mechanism, and (4) is the SeaAttention attention mechanism adopted in the embodiment of the present application.

[0114] As shown in Table 4, from the comparison results of numbers 0 and 1, it can be seen that the combination of the cross-modal Transformer attention mechanism and the multi-level feature fusion module can pay attention to the high-level information well and is more sensitive to feature capture. At the same time, the application of the attention mechanism greatly improves the real-time performance of the model.

[0115] From the comparison results of No. 0 and No. 5, it can be seen that the combination of multi-level feature fusion and the C2f module with inverted residual can greatly improve the average accuracy of the model. From the comparison results of No. 1 and No. 5, it can be seen that the feature fusion module based on iRMB is more advantageous than the combination of cross-modal Transformer and multi-level feature fusion.

[0116] From the comparison results of No. 0 and 6, it can be seen that although the lightweight backbone can reduce the size of the model, it has little effect on the average accuracy.

[0117] From the comparison results of No. 2 and 4, it can be seen that for the improved attention mechanism based on Transformer, the SeaAttention mechanism is easier to integrate with the model than the CoTAttention mechanism, and is more suitable for visual tasks of image processing.

[0118] In the embodiment of the present application, the SeaAttention mechanism is integrated into the Head layer, which improves the scores of mAPbox5 and mAPmask5 to 81.6 and 67.4 respectively.

[0119] A deep learning-based procedure for identifying drivable areas on unstructured roads, such as Figure 5 As shown, when the program product is executed by a processor, the method for identifying a drivable area of ​​an unstructured road in Embodiment 1 or 2 is implemented.

[0120] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0121] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying drivable areas on unstructured roads based on deep learning, characterized in that: include: S1, obtaining a road image; S2, preprocessing the road image; S3, constructing a drivable area recognition model, and pre-training the drivable area recognition model according to a road image training set; The network structure of the drivable area recognition model includes: an input end, a backbone network, a neck network, and a segmentation detection head; The backbone network is provided with a C2f-iRMB module, which is used for feature extraction and integrates the position information in the feature image into the channel attention; The neck network is used to perform semantic segmentation on the road image; The segmentation detection head is used for target detection and target image segmentation; S4. Input the road image into the drivable area recognition model to perform image recognition to obtain a road recognition result.

2. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 1, characterized in that: The C2f-iRMB module includes a CBL submodule and an iRMB submodule. The input vector is first subjected to feature extraction by the first CBL submodule, and then the output of the first CBL submodule is segmented, and the segmented vectors are output to the iRMB submodule for feature extraction, and finally the outputs of the iRMB submodules are spliced.

3. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 2, characterized in that: The C2f-iRMB modules are arranged on the p2 layer, the p4 layer, the p6 layer and the p8 layer.

4. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 1, characterized in that: The neck network includes: an upsampling module, a feature fusion module and a C2f2_3d module.

5. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 4, characterized in that: The feature fusion module outputs to the C2f2_3d module through the spatial attention mechanism and the channel attention mechanism.

6. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 5, characterized in that: The feature fusion module is located at the p11 layer, the p14 layer, the p17 layer and the p20 layer.

7. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 1, characterized in that: The segmentation detection head includes: SeaFormer module, CBL module and Conv2d module, The SeaFormer module is a Squeeze-enhanced Axialattention attention mechanism.

8. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 1, characterized in that: Before constructing the drivable area recognition model and pre-training the drivable area recognition model according to the road image training set, the method further includes: Obtain road image dataset; The road image data set is divided into a road image training set, a road image test set and a road image verification set; wherein the road image is a structured or unstructured road image with annotated obstacle categories.

9. The method for identifying drivable areas on unstructured roads based on deep learning according to claim 8, characterized in that: After constructing the drivable area recognition model and pre-training the drivable area recognition model according to the road image training set, the method further includes: The drivable area recognition model is tested according to the road image test set.

10. A deep learning-based unstructured road drivable area recognition program, characterized in that: When the program product is executed by a processor, the method for identifying a drivable area on an unstructured road as claimed in any one of claims 1 to 9 is implemented.

Images