Unmanned vehicle passability analysis method based on online self-supervised learning

Abstract

The application relates to an online self-supervised learning-based passability analysis method for an unmanned vehicle. The method comprises the following steps: generating BEV data of an environment according to relevant data obtained in real time by each sensor after time synchronization processing; constructing a ResNet18-UNet model and setting a loss function; performing deep-level feature extraction on the BEV data according to the trained ResNet18-UNet model, periodically recording the respective corresponding historical feature vectors of left and right wheels at a certain distance interval in the driving process of the unmanned vehicle at a historical moment, performing unsupervised clustering on the historical feature vectors, calculating the cosine similarity between each feature vector in a road surface-related feature vector diagram and all feature points in a high-level feature vector diagram at the current moment, mapping the obtained similarity value to a gray scale range of 0 to 255 to generate a similarity image, and performing unmanned vehicle passability prediction according to the similarity image. The method can improve the accuracy of unmanned vehicle passability analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to an unmanned vehicle passability analysis method based on online self-supervised learning. BACKGROUND

[0002] With the development of autonomous driving technology, accurately identifying passable and impassable terrains becomes a key challenge. Unlike the clearly distinguishable passable and impassable areas in indoor or road environments, the concept of passability in unstructured outdoor environments is more ambiguous. Unmanned vehicle driving requires strong and accurate perception and understanding of unstructured terrains, and often needs to cross multiple different surface types, which can include different types of passable and impassable vegetation, or surfaces with different properties, such as sand or soil. Therefore, it is particularly important to understand which surfaces are relatively low-risk for vehicles and which areas are relatively high-risk. However, manually assigning passability labels to vehicle perception data is a challenging task and is prone to errors. In addition, the complexity of terrain features and the diversity of the environment make it a difficult task to comprehensively collect various terrain data. Although there have been studies dedicated to generating annotated vehicle datasets, these datasets are usually limited to certain specific geographic locations and seasonal conditions.

[0003] Current passability analysis mainly relies on geometric, semantic or ontology perception features, which depend on the configuration of sensors and vary among different robots. Geometric passability analysis is suitable for rigid environments and evaluates the passability of terrain according to the obstacles, slopes or roughness of the terrain. The environment is usually represented in the form of a 2D, 2.5D or 3D map, while sensors such as RGB, near-infrared (NIR) or radar are often used to enhance the planning and inference of semantic information. In recent years, learning-based terrain classification methods have been widely studied. With the success of semantic segmentation models, multiple datasets and corresponding supervised terrain segmentation models have been released. For example, AI4Mars can generate a large annotated dataset for Mars terrain segmentation, but the workload of collecting such a large dataset is not feasible for many robotic applications. Some methods use both manual annotation and self-supervision, but the corresponding datasets are usually focused on specific environments and still require manual annotation. There are also methods that use anomaly detection using autoencoders to classify visual terrain features that the system has not traversed. The footprint of the vehicle is aggregated and projected into the image space to crop out the traversable area. Then, an autoencoder is used to reconstruct the only traversable area, forcing the model to fail in the untraversed area, which is not observed during training (i.e. out-of-distribution area). Therefore, they convert the reconstruction error into a passability score. This method is affected by changes in lighting and may cause visual artifacts due to the nature of the reconstruction loss, resulting in inaccurate predictions and affecting the safety of vehicle travel. SUMMARY

[0004] Therefore, it is necessary to provide an unmanned vehicle passability analysis method based on online self-supervised learning, which can improve the accuracy of unmanned vehicle passability analysis.

[0005] An unmanned vehicle passability analysis method based on online self-supervised learning, the method comprising:

[0006] According to the real-time acquisition of the relevant data of each sensor that has undergone time synchronization processing, BEV data of the environment is generated;

[0007] A ResNet18-UNet model is constructed and a loss function is set. The ResNet18-UNet model is trained using the loss function, and deep feature extraction is performed on the BEV data according to the trained ResNet18-UNet model to obtain a high-level feature vector map;

[0008] During the driving of the unmanned vehicle, historical feature vectors corresponding to left and right wheels are recorded periodically at certain driving distance intervals, and these historical feature vectors are subjected to online real-time unsupervised clustering processing to obtain several road surface related feature vectors, i.e., several road surface representations of short-term experience of the unmanned vehicle;

[0009] The cosine similarity of each feature vector in the several road surface related feature vectors and all feature points in the high-level feature vector graph at the current time is calculated, the obtained similarity value is mapped to a gray scale range of 0 to 255 to generate a similarity image, and the similarity image is used for unmanned vehicle passability prediction.

[0010] The above-mentioned unmanned vehicle passability analysis method based on online self-supervised learning first generates BEV data of the environment by real-time acquisition of time-synchronized data from various sensors, which enables the unmanned vehicle to rapidly perceive the dynamic changes of the surrounding environment, constructs and trains a ResNet18-UNet model, and extracts deep-level features from a large amount of BEV data in a short time. The ResNet18-UNet model can quickly process these data to provide key feature support for real-time decision-making, enabling the unmanned vehicle to make pass judgment in a timely manner based on environmental features. In addition, the historical feature vectors of the vehicle are recorded periodically during the driving of the unmanned vehicle and saved in a fixed-length queue, and the historical data are used to assist real-time decision-making, which not only considers the influence of the vehicle's own state and the past driving path on the current road condition judgment, but also ensures the timeliness of the data through the fixed-length queue to avoid excessive historical data interfering with real-time performance, enabling the unmanned vehicle to make decisions based on the latest and relevant information. Finally, the cosine similarity of the road surface related feature vectors and the current high-level feature vectors is calculated in real time to generate a similarity image for predicting passability, which can quickly judge the similarity between the current environment and the historical passable road surface, convert the complex feature comparison into intuitive image information, enable the unmanned vehicle to intuitively understand the forward pass condition in a short time, and make decisions such as continuing to drive, braking or avoiding in a timely manner, greatly improving the real-time response capability of the system and ensuring the safe and efficient driving of the unmanned vehicle under various complex road conditions. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A flowchart of an embodiment of an unmanned vehicle passability analysis method based on online self-supervised learning;

[0012] Figure 2 A schematic diagram of the BEV data generation process in an embodiment;

[0013] Figure 3 A schematic diagram of the Resnet18-Unet structure in an embodiment;

[0014] Figure 4 Fig. 2 is a schematic diagram of a base vector selection process in another embodiment;

[0015] Figure 5 Fig. 3 is a schematic diagram of a similarity result in one embodiment;

[0016] Figure 6 Fig. 4 is a schematic diagram of an internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0017] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0018] In one embodiment, as shown in Fig. 1, an online self-supervised learning based passability analysis method for unmanned vehicles is provided, including the following steps: Figure 1

[0019] Step 102, generating BEV data of the environment according to relevant data of each sensor which has been time-synchronized.

[0020] Data collection is performed through cameras, LiDARs and IMUs on board. In order to ensure the high quality of data, the time and space alignment between sensors is strictly ensured during the collection process. This means that in each frame of data, the data from different sensors is completely synchronized, so as to accurately reflect the environment state at the same time. The BEV data generation is as shown in Fig. 2. First, radar point cloud data, visible light image data and radar odometry data which have been time-synchronized are obtained from each sensor. Then, the radar point cloud data is converted from the radar coordinate system to the image coordinate system to obtain radar point cloud data matching the image RGB information. Finally, the radar odometry data is used to accumulate multiple frames of radar point cloud data with RGB information, and finally generate the BEV (Bird's Eye View) data of the environment. Figure 2

[0021] ​​The bird's eye view (BEV) data is generated according to the time-synchronized data obtained by each sensor in real time. In the automatic driving environment, real-time performance is crucial. The traditional method may cause incorrect judgment of the terrain due to data delay or asynchronization. During the rapid driving of the vehicle, if the sensor data cannot be updated and synchronized in time, when the vehicle encounters a suddenly appearing impassable area (such as a large pit on the road), it cannot respond in time. However, real-time acquisition and synchronization processing of sensor data to generate BEV data can enable the unmanned vehicle system to obtain complete environmental information in the first time, providing a basis for subsequent accurate analysis. The data acquisition method of the present application can better adapt to the dynamic changes of the environment, so that the unmanned vehicle can also perceive the environmental changes in time.

[0022] At the same time, the BEV data provides an overhead perspective of the environment, which can clearly show the terrain layout around the vehicle, including the boundaries and slopes of different terrains. Unlike previous methods that rely on manual annotation or partially limited datasets, it is directly converted from raw sensor data, does not rely on prior manual annotation, avoids the subjectivity and limitations of manual annotation, and can adapt to different terrain scenarios, whether it is a city road or an unstructured outdoor environment.

[0023] Step 104, constructing a ResNet18-UNet model and setting a loss function; training the ResNet18-UNet model using the loss function, and extracting deep features from the BEV data according to the trained ResNet18-UNet model to obtain a high-level feature vector graph.

[0024] The present application improves the traditional UNet, especially the structure optimization of the encoder part, such as Figure 3The encoder in the traditional UNet is replaced by four residual modules (layer1 to layer4) in ResNet18, each responsible for a downsampling operation. Unlike traditional max pooling and convolution operations, residual modules not only perform downsampling but also effectively extract deep features while preserving shallow features through residual connections. The advantage of this structure is that, through residual connections, the network can better avoid the problem of gradient vanishing, improve the stability during the training process, and more efficiently extract and integrate features at different levels. Unlike the encoder in the traditional UNet, which requires multiple convolution operations after downsampling, the design simplifies this process by stacking residual modules to achieve more efficient feature extraction. In this way, the network not only extracts deep features but also preserves more rich information between different levels, improving the network's expression ability for complex tasks. In practical application, vehicles generate a large amount of sensor data every second, and when driving in complex outdoor environments, the ResNet18-UNet model can quickly extract key features from BEV data to support subsequent passability analysis, ensuring that unmanned vehicles can make timely decisions. At the same time, it does not rely on manually labeled data but learns features from the data itself. Previous methods such as those based on geometric, semantic, or ontology perception features rely on sensor configurations and differ between different robots. The ResNet18-UNet model, by learning from a large amount of BEV data, can discover general terrain features, such as texture features of sand, color and shape features of grass, and effectively classify terrain. Moreover, using loss functions can optimize the model's parameters, making the model more accurate in terrain classification.

[0025] Step 106, during the driving of the unmanned vehicle, periodically record the historical feature vectors corresponding to the left and right wheels at certain driving distance intervals, and perform online real-time unsupervised clustering processing on these historical feature vectors to obtain several road surface related feature vectors, i.e., the short-term driving experience of the unmanned vehicle.

[0026] The radar image BEV map is input into the trained ResNet18-UNet model, and the high-level features of the image are extracted through the encoder part. The encoder part consists of four residual modules (layer1-layer4) of ResNet18, each of which will downsample the input image and extract its features. These features are then reconstructed layer by layer through the decoder part to generate a high-resolution feature vector map. In the generated feature vector map, select road surface related feature point vectors such as Figure 4As shown, first, a basic principle is followed: the area traveled by the vehicle is considered as the road surface area. Based on this assumption, during the vehicle's travel, the historical feature vectors corresponding to the left and right wheels are recorded every 2 meters, and these feature vectors are saved in a fixed-length historical feature vector queue. Next, unsupervised clustering is performed on the historical feature vectors in the queue to identify road-related features. Data points are assigned to existing clusters or new clusters are created for new data points through similarity measurement. The size of each cluster represents the number of feature vectors contained in the cluster, which will be an important basis for subsequent analysis. In particular, to eliminate road surface information that the vehicle has not sensed for a long time, we remove the feature vector clusters that have not been updated for a long time, effectively reducing redundant data. This strategy ensures that feature cluster updates can more accurately reflect current environmental changes, avoiding unnecessary historical information affecting subsequent analysis and decision-making. By recording historical feature vectors, the influence of the vehicle's own state and previous driving path can be better considered, making real-time decision-making more accurate. At the same time, the fixed-length queue can ensure the timeliness of the data, avoiding excessive historical data affecting real-time performance. At the same time, in the clustering process, the application does not rely on manual labeling, but mines information from the vehicle's own driving trajectory. In human driving examples, the vehicle's trajectory actually reflects which areas are passable. By clustering the historical feature vectors corresponding to the wheels, road-related feature patterns can be discovered. After clustering, these patterns can be extracted as a basis for determining road types, thereby effectively classifying terrain.

[0027] Step 108, calculate the cosine similarity of each feature vector in several road-related feature vectors and all feature points in the high-level feature vector map at the current time, map the obtained similarity value to the gray scale range of 0 to 255, generate a similarity image, and make a prediction of the passability of the unmanned vehicle according to the similarity image.

[0028] These cluster centers are used as reference vectors to calculate the similarity between other feature vectors and them. Specifically, for each feature vector in the reference vector, cosine similarity calculation is performed with the bird's eye view (BEV) feature map generated in the current frame. For each feature vector in the reference vector, the similarity scores of all feature points in the current BEV graph are calculated, and the maximum value of these similarity scores is selected as the final similarity result. Then, we visualize the calculated similarity graph by mapping the similarity value to the gray scale range of 0 to 255, thereby generating an intuitive similarity image, as shown in Figure 5Finally, the passability is predicted according to the similarity graph. The method for calculating the similarity in real time can quickly judge the similarity between the current environment and the historical passable road surface. In the real-time driving scene, the vehicle needs to judge whether the front is passable within a short time. By generating a similarity image, the matching situation of the current environment and the known passable road surface can be intuitively seen. When the similarity is high, it means that the current environment is likely to be passable, and the vehicle can continue to drive; when the similarity is low, the vehicle can take braking or avoidance measures in time. This way can quickly make passability prediction according to real-time data, and improve the real-time response ability of the system. At the same time, by generating a similarity image, the complex feature vector comparison result can be converted into intuitive image information, better terrain classification is performed, thereby improving the accuracy of terrain classification, and further improving the accuracy of the passability analysis of the unmanned vehicle.

[0029] The above unmanned vehicle passability analysis method based on online self-supervised learning first generates the BEV data of the environment by real-time acquisition of time-synchronized data of each sensor, and the BEV data of the environment is acquired in real time to enable the unmanned vehicle to quickly perceive the dynamic changes of the surrounding environment, construct and train the ResNet18-UNet model, and extract deep features from a large amount of BEV data within a short time. The ResNet18-UNet model can quickly process these data, provide key feature support for real-time decision-making, and enable the unmanned vehicle to make passability judgments in a timely manner based on environmental features. In addition, the historical feature vectors of the wheels are recorded periodically during the driving of the unmanned vehicle and saved in a fixed-length queue, and the historical data is used to assist real-time decision-making, which not only considers the influence of the vehicle state and the past driving path on the current road condition judgment, but also ensures the timeliness of the data through the fixed-length queue, avoids the interference of too much historical data on real-time performance, and enables the unmanned vehicle to make decisions based on the latest and relevant information. Finally, the cosine similarity of the road surface related feature vector and the current high-level feature vector is calculated in real time, a similarity image is generated to predict the passability, the similarity between the current environment and the historical passable road surface can be quickly judged, and the complex feature comparison is converted into intuitive image information, so that the unmanned vehicle can intuitively understand the passability of the front within a short time, make decisions such as continuing to drive, braking or avoiding in time, greatly improve the real-time response ability of the system, and ensure the safe and efficient driving of the unmanned vehicle under various complex road conditions.

[0030] In one of the embodiments, the related data includes radar point cloud data, visible light image data, and radar odometer data; the process of generating the BEV data includes:

[0031] The radar point cloud data is converted from the radar coordinate system to the image coordinate system to obtain radar point cloud data matched with the visible light image data information, and the radar point cloud data with RGB information is accumulated by multiple frames by using the radar odometer data to generate BEV data of the environment.

[0032] In one of the embodiments, the editor part of the ResNet18-UNet model includes four residual modules in the ResNet18, and the BEV data is subjected to deep feature extraction according to the trained ResNet18-UNet model to obtain a high-level feature vector map, including:

[0033] According to the fact that each of the four residual modules will down-sample the input image and extract the features thereof, the high-resolution high-level feature vector map is generated by layer-by-layer reconstruction through the decoder part.

[0034] In one of the embodiments, the historical moment is the previous moment of the current moment.

[0035] In one of the embodiments, the historical feature vectors are subjected to unsupervised clustering, including:

[0036] The historical feature vectors in the historical feature vector queue are classified into existing clusters or new clusters are created for new historical feature vectors by similarity measurement, and the size of each cluster represents the number of feature vectors contained in the cluster.

[0037] In one of the embodiments, the loss function is

[0038] .

[0039] It should be understood that, although Figure 1 the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 1 At least part of the steps in may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0040] In one embodiment, a computer device is provided, which can be a terminal, and the internal structure diagram thereof can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement an online self-supervised learning-based unmanned vehicle passability analysis method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0041] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0042] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM) and the like.

[0043] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not contradict each other, they shall be considered within the scope of the present disclosure.

[0044] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the scope of the present application. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An online self-supervised learning based passability analysis method for an unmanned vehicle, characterized in that, The method comprises: Real-time acquisition of relevant data of each sensor after time synchronization processing to generate BEV data of the environment; A ResNet18-UNet model is constructed and a loss function is set; the ResNet18-UNet model is trained using the loss function, and deep feature extraction is performed on the BEV data according to the trained ResNet18-UNet model to obtain a high-level feature vector graph; During the driving of the unmanned vehicle, historical feature vectors corresponding to the left and right wheels are recorded periodically at certain driving distance intervals, and online real-time unsupervised clustering processing is performed on the historical feature vectors to obtain several road-related feature vectors, i.e., several road representations of the short-term experience of the unmanned vehicle; The cosine similarity of each feature vector in the several road-related feature vector graphs and all feature points in the high-level feature vector graph at the current time is calculated, the obtained similarity value is mapped to a gray scale range of 0 to 255 to generate a similarity image, and the similarity image is used for unmanned vehicle passability prediction; The relevant data includes radar point cloud data, visible light image data, and radar odometry data; the process of generating BEV data comprises: The radar point cloud data is converted from the radar coordinate system to the image coordinate system to obtain radar point cloud data matching the visible light image data information, and the radar odometry data is used to accumulate multiple frames of radar point cloud data with RGB information to generate BEV data of the environment.

2. The method of claim 1, wherein, The editor part of the ResNet18-UNet model includes four residual modules in ResNet18, and deep feature extraction is performed on the BEV data according to the trained ResNet18-UNet model to obtain a high-level feature vector graph, which comprises: Each of the four residual modules will downsample the input image and extract its features, and then reconstruct layer by layer through the decoder part to generate a high-resolution high-level feature vector graph.

3. The method of claim 1, wherein, Unsupervised clustering of the historical feature vectors comprises: The historical feature vectors in the historical feature vector queue are classified into existing clusters or new clusters are created for new historical feature vectors through similarity measurement, and the size of each cluster represents the number of feature vectors contained in the cluster.

4. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 3.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3.

Images