Unmanned vehicle trafficability analysis method based on online self-supervised learning

Abstract

The invention relates to an unmanned vehicle trafficability analysis method based on online self-supervised learning. The method comprises the following steps: generating BEV data of an environment according to related data obtained by each sensor in real time and subjected to time synchronization processing; a ResNet18-UNet model is constructed, and a loss function is set; performing deep feature extraction on BEV data according to a trained ResNet18-UNet model, periodically recording historical feature vectors corresponding to a left wheel and a right wheel by taking a certain distance as an interval in the driving process of the unmanned vehicle at a historical moment, performing unsupervised clustering on the historical feature vectors, and obtaining a clustering result; and calculating the cosine similarity between each feature vector in the road surface related feature vector diagram and all feature points in the high-level feature vector diagram at the current moment, mapping the obtained similarity value to a gray scale range from 0 to 255, generating a similarity image, and predicting the trafficability of the unmanned vehicle according to the similarity image. By adopting the method, the accuracy of unmanned vehicle trafficability analysis can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for analyzing the drivability of an unmanned vehicle based on online self-supervised learning. Background Art

[0002] As autonomous driving technology develops, accurately identifying traversable and impassable terrain has become a key challenge. Unlike the clearly distinguished traversable and impassable areas in indoor or road environments, the concept of traversability in unstructured outdoor environments is more ambiguous. Autonomous vehicle driving requires strong and accurate perception and understanding of unstructured terrain, and often needs to traverse a variety of different surface types, which may include different types of traversable and impassable vegetation, or surfaces with different properties, such as sand or soil. Therefore, it is particularly important to understand which surfaces are less risky for vehicles and which areas are more risky. However, manually assigning traversability labels to vehicle perception data is a challenging task and prone to errors. In addition, the complexity of terrain features and the diversity of environments make it a difficult task to comprehensively collect data on various terrains. Although there have been studies dedicated to generating annotated vehicle datasets, these datasets are usually limited to certain specific geographical locations and seasonal conditions.

[0003] Current traversability analysis relies mainly on geometric, semantic, or proprioceptive features, which depend on the configuration of sensors and vary between robots. Geometric traversability analysis is applicable to rigid environments and evaluates the traversability of terrain based on obstacles, slopes, or roughness of the terrain. The environment is usually represented in the form of 2D, 2.5D, or 3D maps, and sensor modalities such as RGB, near-infrared (NIR), or radar are often used to enhance planning and inference of semantic information. In recent years, learning-based terrain classification methods have been widely studied. With the success of semantic segmentation models, multiple datasets and corresponding supervised terrain segmentation models have been released. For example, AI4Mars is able to generate a large annotated dataset for Martian terrain segmentation, but the workload of collecting such a large dataset is not feasible for many robotic applications. Some methods use manual annotation and self-supervision at the same time, but the corresponding datasets are usually focused on specific environments and still require manual annotation. Others have proposed anomaly detection methods using autoencoders to classify visual terrain features that have not been traversed by the system. The vehicle's footprints are aggregated and projected into the image space to crop out traversable areas. Then, they used an autoencoder to reconstruct only traversable areas, forcing the model to fail in untraversed areas, but these areas were not observed during training (i.e., out-of-distribution areas). Therefore, they converted the reconstruction error into a traversability score. This method is affected by lighting changes and may cause visual artifacts due to the nature of the reconstruction loss, resulting in inaccurate prediction results, affecting the safety of vehicle driving. Summary of the invention

[0004] Based on this, it is necessary to provide an unmanned vehicle drivability analysis method based on online self-supervised learning that can improve the accuracy of unmanned vehicle drivability analysis in response to the above technical problems.

[0005] A method for analyzing the drivability of an unmanned vehicle based on online self-supervised learning, the method comprising: Generate BEV data of the environment based on real-time acquisition of relevant data that has been time-synchronized by various sensors; Construct a ResNet18-UNet model and set a loss function; use the loss function to train the ResNet18-UNet model, perform deep feature extraction on the BEV data based on the trained ResNet18-UNet model, and obtain a high-level feature vector graph; During the driving process of the unmanned vehicle, the historical feature vectors corresponding to the left and right wheels are periodically recorded at a certain driving distance interval, and these historical feature vectors are subjected to online real-time unsupervised clustering to obtain several road surface related feature vectors, that is, several road surface representations of the unmanned vehicle's short-term driving experience; Calculate the cosine similarity between each feature vector in several road surface related feature vectors and all feature points in the high-level feature vector map at the current moment, map the obtained similarity value to the grayscale range of 0 to 255, generate a similarity image, and predict the drivability of the unmanned vehicle based on the similarity image.

[0006] The above-mentioned unmanned vehicle passability analysis method based on online self-supervised learning first generates environmental BEV data by acquiring time-synchronized data in real time through various sensors. By acquiring environmental BEV data in real time, the unmanned vehicle can quickly perceive the dynamic changes of the surrounding environment, build and train the ResNet18-UNet model, and perform deep feature extraction on a large amount of BEV data in a short time. The ResNet18-UNet model can quickly process these data and provide key feature support for real-time decision-making, so that the unmanned vehicle can make traffic judgments based on environmental characteristics in a timely manner. In addition, the wheel historical feature vectors are periodically recorded during the driving process of the unmanned vehicle and saved in a fixed-length queue. The historical data is used to assist real-time decision-making, which not only considers the impact of the vehicle's own state and past driving paths on the current road condition judgment, but also ensures the timeliness of the data through a fixed-length queue, avoiding excessive interference of historical data on real-time performance, so that the unmanned vehicle can make decisions based on the latest and relevant information. Finally, the cosine similarity between the road surface related feature vector and the current high-level feature vector is calculated in real time to generate a similarity image to predict passability. This can quickly determine the similarity between the current environment and the historically passable road surface, and convert complex feature comparisons into intuitive image information, allowing the unmanned vehicle to intuitively understand the traffic conditions ahead in a short period of time and make timely decisions such as continuing to drive, braking or avoiding. This greatly improves the system's real-time response capability and ensures that the unmanned vehicle can drive safely and efficiently under various complex road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A flowchart of a method for analyzing the drivability of an unmanned vehicle based on online self-supervised learning in one embodiment; Figure 2 A schematic diagram of a BEV data generation process in one embodiment; Figure 3 A schematic diagram of the Resnet18-Unet structure in one embodiment; Figure 4 A schematic diagram of a basis vector selection process in another embodiment; Figure 5 A schematic diagram of similarity results in one embodiment; Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0008] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0009] In one embodiment, Figure 1 As shown, a method for analyzing the drivability of an unmanned vehicle based on online self-supervised learning is provided, comprising the following steps: Step 102 , generating BEV data of the environment according to relevant data that have been time-synchronized and acquired in real time by various sensors.

[0010] Data collection is performed through onboard cameras, LiDAR, and inertial measurement units (IMUs). To ensure high data quality, the temporal and spatial alignment between sensors is strictly ensured during the collection process. This means that in each frame of data, the data from different sensors are completely synchronized, so that they can accurately reflect the state of the environment at the same time. Figure 2 As shown in the figure. First, radar point cloud data, visible light image data, and radar odometer data that have been time-synchronized are obtained from each sensor. Then, the radar point cloud data is converted from the radar coordinate system to the image coordinate system to obtain radar point cloud data that matches the image RGB information. Finally, using the radar odometer data, these radar point cloud data with RGB information are accumulated over multiple frames to finally generate the BEV (Bird's Eye View) data of the environment.

[0011] Bird's-eye view (BEV) data is generated by acquiring time-synchronized data from each sensor in real time. In an autonomous driving environment, real-time performance is critical. Traditional methods may lead to incorrect judgments about the terrain due to data delays or asynchrony. During rapid vehicle driving, if the sensor data cannot be updated and synchronized in a timely manner, the vehicle will not be able to respond in time when it encounters a sudden inaccessible area (such as a large pothole on the road). Real-time acquisition and synchronous processing of sensor data to generate BEV data allows the unmanned vehicle system to obtain complete environmental information at the first time, providing a basis for subsequent accurate analysis. The data acquisition method of this application can better adapt to dynamic changes in the environment, allowing unmanned vehicles to perceive environmental changes in a timely manner.

[0012] At the same time, BEV data provides an overhead perspective of the environment, which can clearly show the terrain layout around the vehicle, including the boundaries of different terrains, slopes and other information. Unlike previous datasets that rely on manual annotation or have some limitations, it is directly converted from the raw data obtained by the sensor, does not rely on prior manual annotation, avoids the subjectivity and limitations of manual annotation, and can adapt to different terrain scenarios, whether it is urban roads or unstructured outdoor environments.

[0013] Step 104, construct a ResNet18-UNet model and set a loss function; use the loss function to train the ResNet18-UNet model, perform deep feature extraction on the BEV data based on the trained ResNet18-UNet model, and obtain a high-level feature vector graph.

[0014] This application improves the traditional UNet, especially by optimizing the structure of the encoder. Figure 3As shown. The encoder in the traditional UNet is replaced by four residual modules (layer1 to layer4) in ResNet18, and each residual module is responsible for a downsampling operation. This change is different from the traditional maximum pooling and convolution operations. The residual module can not only downsample, but also effectively extract deep features and retain shallow features through residual connections. The advantage of this structure is that through residual connections, the network can better avoid the gradient vanishing problem, improve the stability during training, and more efficiently extract and fuse features at different levels. Unlike the encoder of the traditional UNet, multiple convolution operations are required after downsampling in the traditional UNet, and the design of this application simplifies this process and achieves more efficient feature extraction through the stacking of residual modules. In this way, the network can not only extract deep features, but also retain richer information between different levels, improving the network's ability to express complex tasks. In actual use, vehicles generate a large amount of sensor data every second. When the vehicle is driving in a complex outdoor environment, it needs to determine whether the front is sand, grass or other impassable areas every second. The ResNet18-UNet model can quickly extract key features from BEV data to provide support for subsequent passability analysis and ensure that the unmanned vehicle can make decisions in a timely manner. At the same time, it does not rely on manually labeled data, but learns features from the data itself. Previous methods, such as those based on geometric, semantic or ontological perception features, rely on sensor configuration and vary between different robots. The ResNet18-UNet model can discover common terrain features by learning a large amount of BEV data, and can learn the texture features of sand, the color and shape features of grass, etc., so as to effectively classify terrain. In addition, the loss function can be used to optimize the parameters of the model, making the model more accurate in terrain classification.

[0015] Step 106, during the driving process of the unmanned vehicle, the historical feature vectors corresponding to the left and right wheels are periodically recorded at a certain driving distance interval, and these historical feature vectors are subjected to online real-time unsupervised clustering processing to obtain several road surface related feature vectors, that is, several road surface representations of the unmanned vehicle's short-term driving experience.

[0016] The radar image BEV map is input into the trained ResNet18-UNet model, and the high-level features of the image are extracted through the encoder part. The encoder part consists of four residual modules (layer1-layer4) of ResNet18, each of which downsamples the input image and extracts its features. These features are then reconstructed layer by layer through the decoder part to generate a high-resolution feature vector map. In the generated feature vector map, feature point vectors related to the road surface are selected, such as Figure 4As shown in the figure, first of all, a basic principle is followed: the area where the vehicle has traveled is regarded as the road area. Based on this assumption, during the vehicle's driving process, the historical feature vectors corresponding to the left and right wheels are recorded every 2 meters, and these feature vectors are saved in a fixed-length historical feature vector queue. Next, the historical feature vectors in the queue are clustered unsupervised to identify features related to the road surface, and the data points are classified into existing clusters through similarity measurement, or new clusters are created for new data points. The size of each cluster represents the number of feature vectors contained in the cluster, and this information will serve as an important basis for subsequent analysis. In particular, in order to eliminate the road information that the unmanned vehicle has not perceived for a long time, we remove the feature vector clusters that have not been updated for a long time, thereby effectively reducing redundant data. This strategy ensures that the update of the feature cluster can more accurately reflect the current environmental changes and avoid the influence of unnecessary historical information on subsequent analysis and decision-making. By recording historical feature vectors, the impact of the vehicle's own state and the previous driving path can be better considered, making real-time decisions more accurate. At the same time, the fixed-length queue can ensure the timeliness of the data and avoid the real-time performance affected by too much historical data. At the same time, this application does not rely on manual labeling during the clustering process, but mines information from the vehicle's own driving trajectory. In the human driving example, the vehicle's trajectory actually reflects which areas are passable. By clustering the historical feature vectors corresponding to the wheels, characteristic patterns related to the road surface can be found. After clustering, these patterns can be extracted as a basis for judging the road surface type, thereby effectively classifying the terrain.

[0017] Step 108, calculate the cosine similarity between each feature vector of several road surface related feature vectors and all feature points in the high-level feature vector map at the current moment, map the obtained similarity value to a grayscale range of 0 to 255, generate a similarity image, and predict the drivability of the unmanned vehicle based on the similarity image.

[0018] These cluster centers are used as reference vectors to calculate the similarity between other feature vectors and them. Specifically, for each feature vector in the reference vector, the cosine similarity is calculated with the feature map of the bird's-eye view (BEV map) generated in the current frame. For each feature vector in the reference vector, its similarity score with all feature points in the current BEV map is calculated, and the maximum value of these similarity scores is selected as the final similarity result. Next, we visualize the calculated similarity map by mapping the similarity values ​​to a grayscale range of 0 to 255 to generate an intuitive similarity image, as shown in the figure. Figure 5As described above, finally, the passability prediction is performed based on the similarity graph. The present application can quickly determine the similarity between the current environment and the historical passable road surface by calculating the similarity in real time. In a real-time driving scenario, the vehicle needs to determine whether the road ahead is passable in a short time. By generating a similarity image, you can intuitively see the matching situation between the current environment and the known passable road surface. When the similarity is high, it means that the current environment is likely to be passable and the vehicle can continue to drive; when the similarity is low, the vehicle can take braking or avoidance measures in time. This method can quickly make passability predictions based on real-time data, improving the real-time response capability of the system. At the same time, by generating a similarity image, the complex feature vector comparison results can be converted into intuitive image information, and the terrain classification can be better performed, thereby improving the accuracy of terrain classification, and then improving the accuracy of the unmanned vehicle passability analysis.

[0019] The above-mentioned unmanned vehicle passability analysis method based on online self-supervised learning first generates environmental BEV data by acquiring time-synchronized data in real time through various sensors. By acquiring environmental BEV data in real time, the unmanned vehicle can quickly perceive the dynamic changes of the surrounding environment, build and train the ResNet18-UNet model, and perform deep feature extraction on a large amount of BEV data in a short time. The ResNet18-UNet model can quickly process these data and provide key feature support for real-time decision-making, so that the unmanned vehicle can make traffic judgments based on environmental characteristics in a timely manner. In addition, the wheel historical feature vectors are periodically recorded during the driving process of the unmanned vehicle and saved in a fixed-length queue. The historical data is used to assist real-time decision-making, which not only considers the impact of the vehicle's own state and past driving paths on the current road condition judgment, but also ensures the timeliness of the data through a fixed-length queue, avoiding excessive interference of historical data on real-time performance, so that the unmanned vehicle can make decisions based on the latest and relevant information. Finally, the cosine similarity between the road surface related feature vector and the current high-level feature vector is calculated in real time to generate a similarity image to predict passability. This can quickly determine the similarity between the current environment and the historically passable road surface, and convert complex feature comparisons into intuitive image information, allowing the unmanned vehicle to intuitively understand the traffic conditions ahead in a short period of time and make timely decisions such as continuing to drive, braking or avoiding. This greatly improves the system's real-time response capability and ensures that the unmanned vehicle can drive safely and efficiently under various complex road conditions.

[0020] In one embodiment, the relevant data includes radar point cloud data, visible light image data, and radar odometer data; and the process of generating BEV data includes: The radar point cloud data is converted from the radar coordinate system to the image coordinate system to obtain radar point cloud data that matches the visible light image data information. The radar odometer data is used to accumulate multiple frames of radar point cloud data with RGB information to generate BEV data of the environment.

[0021] In one embodiment, the editor part of the ResNet18-UNet model includes four residual modules in ResNet18, and performs deep feature extraction on BEV data according to the trained ResNet18-UNet model to obtain a high-level feature vector graph, including: Each of the four residual modules downsamples the input image and extracts its features. The decoder then reconstructs the image layer by layer to generate a high-resolution, high-level feature vector map.

[0022] In one embodiment, the historical moment is the moment before the current moment.

[0023] In one embodiment, performing unsupervised clustering on the historical feature vectors includes: The historical feature vectors in the historical feature vector queue are classified into existing clusters through similarity measurement, or new clusters are created for new historical feature vectors. The size of each cluster represents the number of feature vectors contained in the cluster.

[0024] In one embodiment, the loss function is .

[0025] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0026] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for analyzing the passability of an unmanned vehicle based on online self-supervised learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0027] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0028] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0029] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0030] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for analyzing the drivability of unmanned vehicles based on online self-supervised learning, characterized in that: The method comprises: Generate BEV data of the environment based on real-time acquisition of relevant data that has been time-synchronized by various sensors; Constructing a ResNet18-UNet model and setting a loss function; using the loss function to train the ResNet18-UNet model, performing deep-level feature extraction on the BEV data according to the trained ResNet18-UNet model to obtain a high-level feature vector graph; During the driving process of the unmanned vehicle, the historical feature vectors corresponding to the left and right wheels are periodically recorded at a certain driving distance interval, and these historical feature vectors are subjected to online real-time unsupervised clustering to obtain several road surface related feature vectors, that is, several road surface representations of the unmanned vehicle's short-term driving experience; The cosine similarity between each feature vector in the several road surface related feature vector maps and all feature points in the high-level feature vector map at the current moment is calculated, the obtained similarity value is mapped to a grayscale range of 0 to 255, a similarity image is generated, and the unmanned vehicle passability is predicted based on the similarity image.

2. The method according to claim 1, characterized in that The relevant data includes radar point cloud data, visible light image data and radar odometer data; the process of generating BEV data includes: The radar point cloud data is converted from the radar coordinate system to the image coordinate system to obtain radar point cloud data that matches the visible light image data information. The radar odometer data is used to accumulate multiple frames of radar point cloud data with RGB information to generate BEV data of the environment.

3. The method according to claim 1, characterized in that The editor part of the ResNet18-UNet model includes four residual modules in ResNet18. The BEV data is subjected to deep feature extraction according to the trained ResNet18-UNet model to obtain a high-level feature vector graph, including: Each of the four residual modules downsamples the input image and extracts its features. The decoder then reconstructs the image layer by layer to generate a high-resolution, high-level feature vector map.

4. The method according to claim 1, characterized in that The historical moment is the moment before the current moment.

5. The method according to claim 1, characterized in that Performing unsupervised clustering on the historical feature vectors includes: The historical feature vectors in the historical feature vector queue are classified into existing clusters through similarity measurement, or new clusters are created for new historical feature vectors. The size of each cluster represents the number of feature vectors contained in the cluster.

6. The method according to claim 1, characterized in that The loss function is 。 7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images