Unstructured road travelable area identification method based on improved UNet3 + network model

Abstract

The invention belongs to the field of road recognition, and discloses an unstructured road travelable area recognition method based on an improved UNet3 + network model, which comprises the following steps: S1, acquiring unstructured road data, and acquiring a training set and a verification set based on the unstructured road data; s2, performing data enhancement on the training set; s3, performing normalization processing on the verification set and the training set after data enhancement to obtain the verification set and the training set after normalization processing; s4, training the unstructured road travelable area identification model by using the training set obtained through normalization processing in the step S3; and S5, using the trained unstructured road travelable area identification model to predict unstructured road data needing to be identified. According to the method, starting from three aspects of a data set, model improvement and a loss function, the recognition precision of the model for the drivable area of the unstructured road is improved, and the adaptability of the model to various different driving scenes is improved.

Description

Technical Field

[0001] The present invention relates to the field of road recognition, and in particular to a method for identifying drivable areas of unstructured roads based on an improved UNet3+ network model. Background Art

[0002] With the improvement of hardware performance, the rapid development of computer technology and machine learning, Advanced Driving Assistant System (ADAS) has also been rapidly updated and iterated. Through on-board sensors, controllers and other devices, vehicles can perceive the surrounding environment information, make intelligent decisions, and plan the vehicle's driving path through various algorithms, laying the foundation for autonomous driving. The key to achieving a safe and stable autonomous driving system lies in whether it can correctly perceive the surrounding environment, identify roads, vehicles, pedestrians, etc., and make correct decisions. At present, vision-based road drivable area recognition can be mainly divided into two categories: structured road recognition and unstructured road recognition.

[0003] Structured roads such as city streets and highways are usually smooth, well-maintained, with distinct road boundaries, clear lane divisions, and clear traffic signs, making it relatively easy to identify driving areas and navigate vehicles. The current autonomous driving technology is already able to effectively identify drivable areas.

[0004] On unstructured roads, such as rural dirt roads, there are often problems such as blurred road boundaries, strong randomness of traffic participants, and lack of clear lane lines and various traffic signs. The changing road environment and complex traffic dynamics place higher demands on the perception system of autonomous driving vehicles. The detection algorithm for the drivable area of ​​structured roads is difficult to capture the characteristic information of unstructured roads, resulting in a decrease in the accuracy of its recognition results or even failure. Therefore, the recognition of drivable areas on unstructured roads has become a technical problem in autonomous driving systems.

[0005] Realizing unstructured road recognition is a key link in achieving autonomous driving in all road conditions. In addition to realizing the vehicle's autonomous driving capabilities in rural areas and the wild, it can also enhance environmental perception under special weather and lighting conditions, which is of great significance for the practical application of autonomous driving technology.

[0006] Among the existing technologies, methods based on road features, preset road models, etc. cannot adapt to the changeable road shapes of unstructured roads. Methods based on traditional machine learning have the problem of low recognition accuracy. Some road recognition methods based on deep learning only target simpler road scenes or specific road scenes such as field roads, and are difficult to adapt to the complex and changeable driving scenes of unstructured roads. Summary of the invention

[0007] The purpose of the present invention is to disclose a method for identifying drivable areas of unstructured roads based on an improved UNet3+ network model to solve the problems pointed out in the background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention provides a method for identifying a drivable area of ​​an unstructured road based on an improved UNet3+ network model, comprising:

[0010] S1, obtaining unstructured road data, and obtaining a training set and a validation set based on the unstructured road data;

[0011] S2, data augmentation for the training set;

[0012] S3, normalizing the validation set and the training set after data augmentation to obtain a normalized validation set and training set;

[0013] S4, using the training set obtained by the normalization process in S3 to train the unstructured road drivable area recognition model;

[0014] S5, using the trained unstructured road drivable area recognition model to predict the unstructured road data that needs to be recognized.

[0015] Preferably, S1 includes:

[0016] Unstructured road data is obtained from a public dataset and divided into a training set and a validation set.

[0017] Preferably, the public datasets include Indian Driving Dataset, Mapillary Vistas and Bilbao Raw.

[0018] Preferably, S2 includes:

[0019] Data enhancement is performed by horizontally flipping, randomly Gaussian blurring, randomly adjusting brightness, and randomly adding noise to the images in the training set.

[0020] Preferably, S3 includes:

[0021] The training set and validation set are normalized separately.

[0022] Preferably, the unstructured road drivable area recognition model includes a Stem layer, 5 encoders, 5 Att layers and 4 decoders;

[0023] The 5 encoders are Enc1, Enc2, Enc3, Enc4 and Enc5;

[0024] The five Att layers are Att1, Att2, Att3, Att4, and Att5;

[0025] The 4 decoders are Dec1, Dec2, Dec3 and Dec4;

[0026] The Stem layer is connected to Enc1, which is also connected to Att1;

[0027] Enc2 is connected to Enc1, Att2, and Enc3 respectively;

[0028] Enc3 is connected to Enc2, Att3, and Enc4 respectively;

[0029] Enc4 is connected to Enc3, Att4, and Enc5 respectively;

[0030] Enc5 is connected to Enc4 and Att5 respectively;

[0031] Att1 is connected to Dec1, Dec2, Dec3, and Dec4 respectively;

[0032] Att2 is connected to Dec2, Dec3, and Dec4 respectively;

[0033] Att3 is connected to Dec3 and Dec4 respectively;

[0034] Att4 is connected to Dec4;

[0035] Att5 is connected to Dec1, Dec2, Dec3, and Dec4 respectively;

[0036] Dec4 is connected to Dec1, Dec2, and Dec3 respectively;

[0037] Dec3 is connected to Dec1 and Dec2 respectively;

[0038] Dec2 is concatenated with Dec1.

[0039] Preferably, the Stem layer includes Conv 7×7, LayerNorm, GELU and ConvNext v2Block connected in sequence.

[0040] Preferably, the ConvNext v2 Block includes Conv 7×7, LayerNorm, Conv 1×1, GELU, GRN, Conv 1×1 and DropPath connected in sequence.

[0041] Preferably, the encoder comprises LayerNorm, Conv 3×3, ConvNext v2 Block, LayerNorm and GELU connected in sequence.

[0042] Preferably, during the training process, a hybrid loss function is used, where the hybrid loss function is a combination of Lovász-SoftmaxLoss, FocalLoss and MS-SSIM Loss.

[0043] Beneficial effects:

[0044] The present invention improves the recognition accuracy of the model for the drivable area of ​​unstructured roads and improves the adaptability of the model to a variety of different driving scenarios by starting with three aspects: data set, model improvement and loss function. The data set integrated by the present invention includes different road scenes in different weather and time from all over the world, and adopts a random data enhancement method to alleviate the problems of insufficient number and diversity of unstructured road samples, so that the model can adapt to the complex and changeable driving scenarios of unstructured roads. The present invention improves the model by adding a Stem layer, using a more advanced feature extraction network and adding an attention mechanism, thereby improving the accuracy of the model. The present invention uses a mixed loss function composed of Lovász-Softmax Loss, Focal Loss and MS-SSIM Loss for training to alleviate the problems caused by difficult and easy samples, avoid the model from paying too much attention to common samples and simple samples, and improve the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of the unstructured road drivable area identification method based on the improved UNet3+ network model of the present invention.

[0047] Figure 2 It is a schematic diagram of the unstructured road drivable area identification model of the present invention.

[0048] Figure 3 Schematic diagram of the structure of the Stem layer of the present invention.

[0049] Figure 4 Schematic diagram of the ConvNeXt v2 block structure of the present invention.

[0050] Figure 5It is a schematic diagram of the structure of the encoder of the present invention.

[0051] Figure 6 It is a schematic diagram of the structure of the decoder of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The invention content of the present invention is to propose a technology for identifying drivable areas on unstructured roads, aiming to improve the accuracy of identifying drivable areas on unstructured roads in complex and changeable driving environments. The present invention improves the existing UNet3+ network model by adding a Stem layer, using a more advanced backbone network for feature extraction, and adding a CBAM attention mechanism to make the model focus on key features in space and channels to enhance the accuracy of the model, and uses a mixed loss function composed of Lovász-Softmax Loss, Focal Loss and MS-SSIM Loss for training to alleviate problems caused by difficult and easy samples and improve the semantic segmentation performance of the model.

[0054] like Figure 1 As shown, the present invention provides a method for identifying a drivable area of ​​an unstructured road based on an improved UNet3+ network model, comprising:

[0055] S1, obtaining unstructured road data, and obtaining a training set and a validation set based on the unstructured road data;

[0056] S2, perform data augmentation on the training set;

[0057] S3, normalizing the validation set and the training set after data enhancement to obtain a normalized validation set and training set;

[0058] S4, using the training set obtained by the normalization process in S3 to train the unstructured road drivable area recognition model;

[0059] S5, using the trained unstructured road drivable area recognition model to predict the unstructured road data that needs to be recognized.

[0060] Preferably, S1 includes:

[0061] Unstructured road data is obtained from a public dataset and divided into a training set and a validation set.

[0062] Unstructured road data is obtained for deep learning model training and verification. The unstructured road data used in the present invention are all obtained from public data sets, including the Indian Driving Dataset (IDD), Mapillary Vistas data set, WildDash data set and Bilbao Raw data set.

[0063] The IDD-20k dataset includes 20,101 images from around Indian cities and suburbs, most of which are in 1080p resolution; the Mapillary Vistas dataset is a large street-level image dataset containing 25,000 high-resolution street view images from all over the world; the WildDash 2 dataset contains 4,256 images from all over the world, including many difficult scenes such as rainy days, low-light environments, overexposure, and camera characteristics such as distortion and noise; the Bilbao Raw dataset contains 1,000 unstructured, non-asphalt road images taken in the Basque region.

[0064] The present invention integrates the adopted data sets and filters out the unstructured road data therein, obtaining a total of 20,527 pictures, which are divided into a training set of 14,368 pictures for model training and a verification set of 6,159 pictures for verifying the effect of model training.

[0065] Preferably, the public datasets include Indian Driving Dataset, Mapillary Vistas and Bilbao Raw.

[0066] Preferably, S2 includes:

[0067] Data enhancement is performed by horizontally flipping, randomly Gaussian blurring, randomly adjusting brightness, and randomly adding noise to the images in the training set.

[0068] The unstructured roads obtained in S1 are enhanced to increase data diversity, alleviate the problem of insufficient data samples, and help prevent overfitting of the model, so that the model can better cope with the complex and changeable driving environment of unstructured roads.

[0069] Data augmentation can eliminate the model's dependence on the horizontal direction and simulate various situations during vehicle driving, such as blur caused by bumps, lighting environments under different weather conditions, and distortion caused by electrical signal interference.

[0070] The present invention uses an online data enhancement method to perform real-time data enhancement on input images during the training process, so that the data used in each training process is different, which increases the diversity of the data, helps to reduce the model's dependence on specific samples, and increases the model's generalization ability.

[0071] Preferably, S3 includes:

[0072] The training set and validation set are normalized separately.

[0073] The present invention uses Z-Score normalization to normalize data features, reduce the scale differences between features, improve the stability of model training, and prevent the large numerical differences between features from causing the model training to fail to converge. The Z-Score normalization formula is as follows:

[0074]

[0075] Wherein μ is the mean value, σ corresponds to the standard deviation. The mean values ​​of the data set used in the present invention on the three color channels of R, G, and B are 0.485, 0.456, and 0.406, respectively, and the corresponding standard deviations are 0.229, 0.224, and 0.225, respectively.

[0076] Specifically, the unstructured road drivable area recognition model is trained:

[0077] The normalized unstructured road data in S3 is sent to the unstructured road drivable area recognition model for training. The structure of the unstructured road drivable area recognition model is shown in Figure 2 First, the input passes through the Stem layer for preliminary feature extraction. Its structure is shown in Figure 3 The ConvNeXt v2 block structure used is shown in Figure 4 The model then enters the downsampling phase, where each layer of encoder is responsible for extracting feature information and outputting it to the attention module of the same layer. At the same time, it performs a pooling operation and reduces the dimension of the feature map to pass it to the next layer of encoder. The structure of the encoder is shown in Figure 5 The attention module learns the importance of each feature and dynamically adjusts the weight of each feature to amplify important features and suppress redundant information and noise, and submits the results to multiple decoders of the same scale and higher scale. The decoder is responsible for fusing feature inputs from different scales, submitting the obtained feature map to a lower scale decoder, decoding the feature map at the same time, and then obtaining the probability distribution of each classification through the Softmax function to obtain the semantic segmentation result. The decoder structure taking the decoder of the third layer as an example is shown in Figure 6 The final model outputs the results of each layer of the decoder through deep supervision, and uses real values ​​to supervise the training of each layer.

[0078] Preferably, the unstructured road drivable area recognition model includes a Stem layer, 5 encoders, 5 Att layers and 4 decoders;

[0079] The 5 encoders are Enc1, Enc2, Enc3, Enc4 and Enc5;

[0080] The five Att layers are Att1, Att2, Att3, Att4, and Att5;

[0081] The 4 decoders are Dec1, Dec2, Dec3 and Dec4;

[0082] The Stem layer is connected to Enc1, which is also connected to Att1;

[0083] Enc2 is connected to Enc1, Att2, and Enc3 respectively;

[0084] Enc3 is connected to Enc2, Att3, and Enc4 respectively;

[0085] Enc4 is connected to Enc3, Att4, and Enc5 respectively;

[0086] Enc5 is connected to Enc4 and Att5 respectively;

[0087] Att1 is connected to Dec1, Dec2, Dec3, and Dec4 respectively;

[0088] Att2 is connected to Dec2, Dec3, and Dec4 respectively;

[0089] Att3 is connected to Dec3 and Dec4 respectively;

[0090] Att4 is connected to Dec4;

[0091] Att5 is connected to Dec1, Dec2, Dec3, and Dec4 respectively;

[0092] Dec4 is connected to Dec1, Dec2, and Dec3 respectively;

[0093] Dec3 is connected to Dec1 and Dec2 respectively;

[0094] Dec2 is concatenated with Dec1.

[0095] Preferably, the Stem layer includes Conv 7×7, LayerNorm, GELU and ConvNext v2Block connected in sequence.

[0096] Preferably, the ConvNext v2 Block includes Conv 7×7, LayerNorm, Conv 1×1, GELU, GRN, Conv 1×1 and DropPath connected in sequence.

[0097] Preferably, the encoder comprises LayerNorm, Conv 3×3, ConvNext v2 Block, LayerNorm and GELU connected in sequence.

[0098] Preferably, during the training process, a hybrid loss function is used, where the hybrid loss function is a combination of Lovász-SoftmaxLoss, Focal Loss and MS-SSIM Loss.

[0099] A hybrid loss function is used, combined with Lovász-Softmax Loss, Focal Loss and MS-SSIM Loss, to calculate the loss of the segmentation results from multiple levels, which alleviates the problems of unbalanced sample classification and difficult and easy samples, and enables the model to pay more attention to the differences in fuzzy boundaries and regional distribution to improve model performance. At the same time, the OneCycleLR learning rate scheduling strategy is applied to dynamically adjust the learning rate to improve model training efficiency.

[0100] Preferably, the decoder comprises a plurality of inputs.

[0101] There are three structures at the input end. The first structure includes a Max Pooling downsampling module and a Conv 3x3 module;

[0102] The second structure includes an Upsample upsampling module and a Conv 3x3 module;

[0103] The third structure is the Max Pooling downsampling module.

[0104] The multiple input ends included in the decoder may include only one structure, two structures, or three structures.

[0105] The input of the decoder contains multiple Max Pooling downsampling modules with different magnification factors and Upsample upsampling modules with different reduction factors. The purpose is to downsample / upsample feature maps (of different sizes) from different scales by different multiples so that they have the same size for splicing.

[0106] The feature map is first downsampled / upsampled by X times through Max Pooling X / Upsample X, then passed through the Conv3x3 module, and finally output to Concat for concatenation with other outputs.

[0107] After concatenation through Concat, it will go through ConvNeXt v2 Block, LayerNorm, and GELU to get the final output.

[0108] If the input feature map size is the same as the target size (such as Figure 6 If the Xen3 in , no downsampling / upsampling is required and only one Conv 3x3 module is included.

[0109] Preferably, S5 includes:

[0110] The model trained in S4 is used as the final unstructured road drivable area recognition model for deployment in the terminal device. The present invention uses the validation set divided in S1 to test the model performance and uses three performance indicators, namely mIoU, mPA and Dicemean, to measure the prediction performance of the model. The calculation methods of the three performance indicators are as follows:

[0111]

[0112]

[0113]

[0114] Among them, TP represents the correctly predicted positive sample, FP represents the incorrectly predicted positive sample, FN represents the incorrectly predicted negative sample, and k represents the number of classifications including background or empty classifications. According to the performance evaluation and result analysis, the model is adjusted and optimized as necessary, and then the training and evaluation process is repeated until the model reaches the expected performance standard. The test results are shown in Table 1:

[0115] Table 1 Model performance test results

[0116] Decoder Layers QUR mPA Dicemean 1 0.7267 0.8205 0.8240 2 0.7280 0.8176 0.8252 3 0.7208 0.8101 0.8200 4 0.7023 0.7958 0.8056 5 0.6686 0.7732 0.7791

[0117] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. The unstructured road drivable area recognition method based on the improved UNet3+ network model is characterized by: include: S1, obtaining unstructured road data, and obtaining a training set and a validation set based on the unstructured road data; S2, data augmentation for the training set; S3, normalizing the validation set and the training set after data augmentation to obtain a normalized validation set and training set; S4, using the training set obtained by the normalization process in S3 to train the unstructured road drivable area recognition model; S5, using the trained unstructured road drivable area recognition model to predict the unstructured road data that needs to be recognized.

2. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 1 is characterized in that: S1, including: Unstructured road data is obtained from a public dataset and divided into a training set and a validation set.

3. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 2 is characterized in that: Public datasets include Indian Driving Dataset, Mapillary Vistas, and BilbaoRaw.

4. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 1 is characterized in that S2 include: Data enhancement is performed by horizontally flipping, randomly Gaussian blurring, randomly adjusting brightness, and randomly adding noise to the images in the training set.

5. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 1 is characterized in that: S3 includes: The training set and validation set are normalized separately.

6. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 1 is characterized in that: The unstructured road drivable area recognition model consists of a Stem layer, 5 encoders, 5 Att layers, and 4 decoders; The 5 encoders are Enc1, Enc2, Enc3, Enc4 and Enc5; The five Att layers are Att1, Att2, Att3, Att4, and Att5; The 4 decoders are Dec1, Dec2, Dec3 and Dec4; The Stem layer is connected to Enc1, which is also connected to Att1; Enc2 is connected to Enc1, Att2, and Enc3 respectively; Enc3 is connected to Enc2, Att3, and Enc4 respectively; Enc4 is connected to Enc3, Att4, and Enc5 respectively; Enc5 is connected to Enc4 and Att5 respectively; Att1 is connected to Dec1, Dec2, Dec3, and Dec4 respectively; Att2 is connected to Dec2, Dec3, and Dec4 respectively; Att3 is connected to Dec3 and Dec4 respectively; Att4 is connected to Dec4; Att5 is connected to Dec1, Dec2, Dec3, and Dec4 respectively; Dec4 is connected to Dec1, Dec2, and Dec3 respectively; Dec3 is connected to Dec1 and Dec2 respectively; Dec2 is concatenated with Dec1.

7. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 6 is characterized in that: The Stem layer consists of Conv 7×7, LayerNorm, GELU, and ConvNext v2Block connected sequentially.

8. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 7 is characterized in that: The ConvNext v2 Block consists of Conv 7×7, LayerNorm, Conv 1×1, GELU, GRN, Conv 1×1, and DropPath connected sequentially.

9. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 6 is characterized in that: The encoder consists of LayerNorm, Conv 3×3, ConvNext v2 Block, LayerNorm and GELU connected sequentially.

10. The unstructured road drivable area identification method based on the improved UNet3+ network model according to claim 1 is characterized in that: During the training process, a hybrid loss function is used, which is a combination of Lovász-SoftmaxLoss, Focal Loss and MS-SSIM Loss.

Images