A non-paved road identification and classification system

Abstract

The application discloses a non-paved road identification and classification system, and relates to the technical field of intelligent driving, comprising the following modules: a front-end data acquisition module; a sensor data processing module: filtering the data of an inertial measurement unit to remove noise; adjusting the size of a camera image, graying and histogram equalization; filtering and denoising and down-sampling processing point cloud data; a road identification module: based on vehicle dynamics and vision-based road identification method fusion, adopting a convolutional neural network framework, for multi-modal input data from sensors, respectively via different feature extraction channels for feature extraction, and for different feature fusion tasks, multi-modal feature fusion is realized through different feature fusion modules, and the fused features are further input into a fully connected layer after convolution layer processing, and finally the classification result is obtained. It has the effects of high recognition accuracy, strong overall robustness of the system and wide applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a non-paved road identification and classification system. BACKGROUND

[0002] Currently, the development of autonomous vehicles has attracted widespread attention, but most research and development work has been focused on urban road and highway driving environments. With the development of autonomous driving technology, off-road vehicles are increasingly used in military reconnaissance, geological exploration, and field rescue. These scenarios are often accompanied by complex and variable road conditions, such as mud, sand, rock, and snow. These non-paved road types are diverse, and each type of road will have different effects on the vehicle's driving performance. For example, sand can easily cause the vehicle to sink, grassland can be slippery, and mud can become extremely difficult to travel after rain. Therefore, accurately identifying and classifying non-paved roads is crucial for unmanned vehicles, as it not only relates to the safe driving of the vehicle, but also affects the vehicle's dynamic control strategy. Non-paved road identification can also provide key information for vehicle navigation and path planning, improving the efficiency and success rate of the vehicle's driving in complex environments.

[0003] Under non-paved road conditions, the road conditions are complex and varied, and many sections often have no obvious road boundaries. Therefore, the detection difficulty is relatively large. In addition, the driving conditions of vehicles in off-road environments are often harsh, and vehicle sensors are easily disturbed by various environmental factors, affecting the perception effect. The vehicle's perception of road information through a single sensor solution will make the recognition result difficult to guarantee robustness and accuracy. SUMMARY

[0004] The purpose of the present application is to solve the above problems and provide a non-paved road identification and classification system.

[0005] To achieve the above purpose, the technical solution of the present application is as follows:

[0006] The present application provides a non-paved road identification and classification system, which includes the following modules:

[0007] Front-end data acquisition module: including camera, laser radar and inertial measurement unit;

[0008] Sensor data processing module: filtering the data of the inertial measurement unit to remove noise; adjusting the size of the camera image, grayscale and histogram equalization; filtering and denoising the point cloud data and reducing the sampling processing;

[0009] The road surface recognition module: based on the fusion of vehicle dynamics and vision-based road surface recognition method, a convolutional neural network framework is adopted, for the multi-modal input data from the sensor, respectively via different feature extraction channels for feature extraction, and for different feature fusion tasks through different feature fusion modules to realize multi-modal feature fusion, and the fusion features are further input into the fully connected layer after the convolution layer processing, and finally the classification result is obtained

[0010] The training and evaluation module:

[0011] Firstly, in the data preprocessing stage, we obtain image and point cloud data from the front-end data acquisition module, and combine with vehicle dynamics parameters. These data need to be divided into training set, validation set and test set, in order to facilitate the model tuning in the training process and finally evaluate the model performance. At the same time, according to the different types of unpaved road, the corresponding label is assigned to each sample. In the model training stage, a multi-modal fusion convolutional neural network model is constructed, which can adopt VGG architecture as the basis and add specific multi-modal feature fusion module for processing image and point cloud data. Select cross-entropy loss function and Adam optimizer, set appropriate learning rate and momentum and other hyperparameters to ensure that the model can effectively converge. Use the training set data to train the model, iteratively update the model parameters, and use the validation set to evaluate the model performance at the end of each epoch, record the best model version to prevent overfitting. If the model performance improvement is not obvious, early stopping method can be used to terminate the training in advance.

[0012] In the model evaluation stage, the test set data not involved in the training is used to evaluate the overall performance of the model, and the accuracy, recall rate, F1 score and other key performance indicators are calculated to fully understand the performance of the model. The confusion matrix is generated to visually show the prediction of each class, helping to understand the performance of the model in which class is better or worse, and according to this, analyze the performance difference of the model on different types of unpaved road, find out the reason of misclassification, and adjust the model or data preprocessing strategy accordingly.

[0013] Through the training and evaluation module, it can be ensured that the designed unpaved road recognition and classification system has high accuracy and practicality, so as to complete the design and implementation of the whole training and evaluation module.

[0014] The application is further provided: the camera is used to capture road surface images; the laser radar is used to construct a three-dimensional model of the road surface through high-precision distance measurement information; and the inertial measurement unit is used to measure acceleration and angular velocity, helping to identify the motion posture of the vehicle.

[0015] The vehicle dynamics road surface identification method is based on a Kalman filter, and is based on a vehicle dynamics model and a magic formula tire model.

[0016] The vehicle dynamics model is a three-degree-of-freedom vehicle dynamics model, which is used to represent the longitudinal, lateral and yaw motion of the vehicle.

[0017] The magic formula tire model uses a set of unified mathematical expressions to describe the mechanical properties of the tire, including longitudinal force, lateral force and turning moment.

[0018] The vision-based road surface identification includes the following steps:

[0019] Step S1, correspond the laser radar point cloud with the two-dimensional pixel points of the camera image; respectively perform camera intrinsic parameter calibration and camera-laser radar extrinsic parameter joint calibration; collect images and corresponding laser radar data containing different non-paved road types, and label the images;

[0020] Step S2, use a deep learning semantic segmentation algorithm to fuse laser radar point cloud data and camera image data to learn the regional features of the non-paved road; use a VGG16 deep learning model to extract features from the camera image; use a point cloud clustering algorithm to identify obstacles or other features on the road surface;

[0021] Step S3, after the data feature extraction is completed, a SuperFusion network is used to perform data-level fusion of the laser radar and the camera; the SuperFusion directly projects the point cloud data of the laser radar onto the camera image plane to form a sparse depth map; an image depth estimation method is used to improve the accuracy and robustness of the depth map in combination with the accurate depth information provided by the laser radar.

[0022] Compared with the prior art, the beneficial effects of the present scheme are: the non-paved road classification and identification method based on multi-source sensor fusion can bring significant technical effects, which not only reflect the improvement in the technical level, but also involve the performance improvement in the actual application scene. By fusing the data of camera images, laser radar point cloud data and IMU and other sensors, the advantages of each sensor can be used to complement the shortcomings, thereby improving the recognition accuracy of different road types.

[0023] By fusing camera images, lidar point cloud data, and IMU data from multiple sensors, the strengths of each sensor can be leveraged to complement its weaknesses, thereby improving the accuracy of identifying different road surface types. Even if a sensor fails or is affected by the environment, causing the data quality to decline, the data from other sensors can still provide effective information support, enhancing the overall robustness of the system. Multi-source sensor fusion can better cope with various weather conditions and changes in lighting, ensuring high recognition performance in different environments. The non-paved road classification and recognition method based on multi-source sensor fusion can not only be applied to autonomous vehicles, but also extended to agricultural machinery, military equipment, field exploration, and other fields, with wide applicability. The system can also receive feedback information in real time, continuously learn and adapt to new road types, gradually improving recognition accuracy and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the system architecture in the embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a three-degree-of-freedom vehicle dynamics model in the embodiment of the present application;

[0026] Figure 3 is a flowchart of the unscented Kalman filter algorithm in the embodiment of the present application;

[0027] Figure 4 is a VGG16 structure diagram in the embodiment of the present application;

[0028] Figure 5 is a fusion strategy based on dynamics and vision in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order for those skilled in the art to better understand the present application, the technical solutions of the present application will be further described in detail below in conjunction with the embodiments of the present application and the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0030] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below in conjunction with the embodiments.

[0031] Embodiment:

[0032] On unpaved roads, vehicles face various complex road conditions, such as mud, gravel, and grass. To improve vehicle adaptability and safety, this solution proposes an unpaved road surface recognition and classification system that utilizes data from cameras and LiDAR to identify and classify unpaved surfaces. This method aims to improve the accuracy of road surface classification by integrating information from multiple sensors, thereby improving vehicle control strategies. The system architecture of this invention is as follows: Figure 1 As shown.

[0033] The front-end data acquisition module includes a camera, a lidar unit, and an inertial measurement unit (IMU). The camera captures images of the road surface, while the lidar provides high-precision distance measurements to build a 3D model of the road surface. The IMU measures acceleration and velocity to help identify the vehicle's motion state.

[0034] Sensor data processing

[0035] The data from the inertial measurement unit is filtered to remove noise. Camera images are resized, converted to grayscale, and histogram equalized. Point cloud data is filtered, denoised, and downsampled.

[0036] Fusion Strategy Design

[0037] 1. Road surface recognition based on vehicle dynamics

[0038] The prerequisite for road surface recognition based on vehicle dynamics is that there must be accurate vehicle and tire models. This invention studies a road surface recognition method based on the unscented Kalman filter algorithm. Based on the vehicle dynamics model and the magic formula tire model, it makes full use of vehicle dynamic response information to establish an unscented Kalman filter estimator.

[0039] This invention uses a three-degree-of-freedom vehicle dynamics model to characterize the longitudinal, lateral, and yaw motions of a vehicle. The model is simple and clear and can integrate most of the parameters of the vehicle structure.

[0040] The Magic Formula tire model uses a unified set of mathematical expressions to describe various mechanical properties of tires, including longitudinal forces, lateral forces, and rotational torque. This uniformity makes the model easier to fit and reduces the number of different models that need to be managed. Parameters are easy to determine and it is applicable to a variety of working conditions.

[0041] The Unscented Kalman Filter (UKF) is a nonlinear state estimation method that approximates the probability distribution through a set of specifically selected sample points (called sigma points). The UKF does not require explicit calculation of the Jacobian matrix of the state transition equation and the observation equation, which makes the UKF easier to implement and more accurate in handling highly nonlinear systems. The implementation of the UKF is relatively simple, easy to understand and program, and easy to use with other sensor fusion techniques. The flowchart of the Unscented Kalman Filter algorithm is shown in FIG. Figure 3

[0042] 2. Vision-based road surface identification method

[0043] When collecting road surface data, in order to match the detection results of the camera and the lidar for the same target, it is necessary to first realize the correspondence between the lidar point cloud and the two-dimensional pixel points of the camera image. Therefore, camera intrinsic calibration and camera-lidar extrinsic calibration need to be performed respectively. Collect images and corresponding lidar data containing different types of unpaved road surfaces, and label the images.

[0044] A deep learning semantic segmentation algorithm is used to fuse lidar point cloud data and camera image data to learn the regional features of unpaved road surfaces, thereby achieving identification and classification. This patent uses a VGG16 deep learning model to extract features from camera images. A point cloud clustering algorithm is used to identify obstacles or other features on the road surface.

[0045] After the data features are extracted, the SuperFusion network is used for data-level fusion of the lidar and the camera. SuperFusion is a multi-level lidar-camera fusion network. This model not only supports short-range high-definition map prediction within 30 meters, but also provides long-range high-definition map prediction up to 90 meters to meet the needs of autonomous driving path planning and control tasks, thereby improving the smoothness and safety of autonomous driving. The point cloud data of the lidar is directly projected onto the camera image plane using SuperFusion to form a sparse depth map. Then, an image depth estimation method is used to combine the accurate depth information provided by the lidar to improve the accuracy and robustness of the depth map.

[0046] 3. Fusion method based on dynamics and vision

[0047] The road surface type identification algorithm also adopts a convolutional neural network framework. For multi-modal input data from three different sensors, feature extraction is performed through different feature extraction channels, and multi-modal feature fusion is achieved through different feature fusion modules for different feature fusion tasks. The fused features are further processed by convolution layers and input into fully connected layers to obtain the classification results.​Figure 5 are shown.

[0048] The above specific embodiments are only an explanation of the present application, which is not a limitation of the present application. Those skilled in the art can make modifications to the embodiments without creative contribution after reading the specification, but as long as the modifications are within the scope of the claims of the present application, they are protected by the patent law.

Claims

1. A non-paved road identification and classification system, characterized in that, The system comprises the following modules: Front-end data acquisition module: including camera, laser radar and inertial measurement unit; Sensor data processing module: filtering the data of the inertial measurement unit to remove noise; adjusting the size of the camera image, graying and histogram equalization; filtering and denoising point cloud data and downsampling processing; Road surface recognition module: based on the fusion of vehicle dynamics and vision-based road surface recognition method, a convolutional neural network framework is adopted, for multi-modal input data from sensors, feature extraction is performed through different feature extraction channels, and multi-modal feature fusion is realized through different feature fusion modules for different feature fusion tasks, and the fused features are further input into the fully connected layer after convolution layer processing, and finally the classification result is obtained; Training and evaluation module: in the model training stage, a multi-modal fusion convolutional neural network model is constructed, VGG architecture is adopted as the basis, and specific multi-modal feature fusion modules are added to process image and point cloud data; cross-entropy loss function and Adam optimizer are selected, and learning rate and momentum are set to ensure effective convergence of the model; the model is trained using the training set data, the model parameters are iteratively updated, and the performance of the model is evaluated using the validation set at the end of each epoch to record the best model version to prevent overfitting; in the model evaluation stage, the overall performance of the model is evaluated using the test set data that did not participate in the training, the accuracy, recall rate and F1 score indicators are calculated to fully understand the performance of the model, and the confusion matrix is generated to visually show the prediction of each class, helping to understand the performance of the model in each class, and analyzing the performance difference of the model in different types of non-paved road surfaces, finding out the reasons for misclassification, and adjusting the model or data preprocessing strategy accordingly; Through the training and evaluation module, the designed non-paved road surface recognition and classification system can ensure high accuracy and practicality, thereby completing the design and implementation of the entire training and evaluation module.

2. A non-paved road identification and classification system as claimed in claim 1, characterized in that: The camera is used to capture road surface images; the laser radar is used to construct a three-dimensional road surface model through high-precision distance measurement information; the inertial measurement unit is used to measure acceleration and angular velocity to help identify the motion posture of the vehicle.

3. A non-paved road identification and classification system as in claim 1, wherein: The vehicle dynamics-based road surface recognition method is based on unscented Kalman filter, and the vehicle dynamics model and magic formula tire model are used as the basis to fully utilize the vehicle dynamics response information to establish the unscented Kalman filter estimator.

4. A non-paved road identification and classification system as claimed in claim 3, characterized in that: The vehicle dynamics model selects a three-degree-of-freedom vehicle dynamics model to represent the longitudinal, lateral and yaw motion of the vehicle.

5. A non-paved road identification and classification system as in claim 3, wherein: The magic formula tire model uses a set of unified mathematical expressions to describe the mechanical properties of the tire, including longitudinal force, lateral force and turning moment.

6. A non-paved road identification and classification system as in claim 1, wherein, The vision-based road surface recognition comprises the following steps: Step S1, correspond the laser radar point cloud with the two-dimensional pixel points of the camera image; perform camera intrinsic parameter calibration and camera-laser radar extrinsic parameter joint calibration respectively; collect images and corresponding laser radar data containing different types of non-paved road surfaces, and label the images; Step S2, using a deep learning semantic segmentation algorithm, fusing laser radar point cloud data and camera image data to learn the area characteristics of the unpaved road; using a VGG16 deep learning model to extract features from the camera image; using a point cloud clustering algorithm to identify obstacles or other features on the road surface; Step S3, after the data feature extraction is completed, a SuperFusion network is used for data level fusion of the laser radar and the camera; the SuperFusion is used to directly project the point cloud data of the laser radar onto the camera image plane to form a sparse depth map; an image depth estimation method is used to improve the accuracy and robustness of the depth map in combination with the accurate depth information provided by the laser radar.

Images