Self-adaptive perception positioning method based on unsupervised fusion BEV

Abstract

The invention provides a self-adaptive perception positioning method based on unsupervised fusion BEV, and the method comprises the following steps: collecting point cloud data of environment information nearby a vehicle in real time through employing a laser radar, and collecting image view data of the environment information nearby the vehicle in real time through employing a camera; the method comprises the steps of converting multi-view image view data and point cloud data into the same BEV space, respectively generating camera BEV features and point cloud BEV features, fusing the camera BEV features and the point cloud BEV features in a projection mode to obtain fused BEV features, performing semantic segmentation and instance segmentation on the fused BEV features, and generating a vectorized fused BEV map; learning query is introduced through query-based collaborative self-adaption, interaction between multi-view image view data and the fused BEV map and collaborative self-adaption of the multi-view image view data and the fused BEV map are achieved, and a spatial perception result is generated; and self-checking the spatial perception result, checking the rationality of the image, and completing the closed loop of the perception positioning process. According to the invention, the fusion BEV under unsupervised learning can rapidly detect and obtain the environment information, and autonomous verification is carried out, so that the accuracy of vehicle positioning is ensured, and the experience feeling of the vehicle is improved at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor information fusion, and in particular to an adaptive perception positioning method based on unsupervised fusion BEV. Background Art

[0002] Autonomous driving is a hot topic in the automotive industry today, and positioning technology plays a vital role in the process of realizing autonomous driving. Among them, the positioning solution based on cameras is the current mainstream solution. Through visual information, a lane line map is constructed to complete the autonomous positioning of the vehicle. The camera can detect and identify lane lines from video images and output them to the autonomous driving controller in real time. The advantages of cameras are strong real-time perception and relatively accurate detection. However, cameras also have disadvantages such as being easily affected by weather, light, etc., having a limited field of view, and being easily blocked by obstacles. The performance is poor in rainy, foggy, and windy weather, because it is limited by the processing speed of image information, and the performance is difficult to guarantee under complex road conditions such as obstructions and turns. Given the inherent characteristics of cameras, the driving safety of autonomous driving cannot be guaranteed under complex and diverse working conditions, and comfort and experience are even more unattainable.

[0003] At present, in order to solve the problem of limited perception positioning accuracy, various teams have adopted a variety of solutions. The perception positioning solution based on GNSS benefits from the current global coverage of GPS, has a large coverage range, and can achieve all-weather positioning. However, the positioning accuracy is affected by the quality of the collected satellite signals, and the accuracy will fluctuate significantly. The positioning based on IMU solves the problem of being restricted by external information, has high short-term accuracy, and can provide the scale of motion changes to the real world. The disadvantage is that the positioning error will accumulate over time, resulting in large deviations in long-distance navigation. The advantage of positioning based on laser radar is that it is insensitive to light, has high accuracy, and can directly obtain the relative distance between the outline of objects in the surrounding environment and the carrier. However, high-precision laser radars are expensive and are easily disturbed by rain and snow, which brings many uncertainties to autonomous driving. The camera-based positioning method achieves positioning by collecting images in sequence, and can estimate the pose changes between adjacent frames based on visual features. The disadvantage is that this is a relative positioning method, which is easily affected by visual conditions such as light and occlusion. The present invention aims to adapt to more diversified outdoor environments and establish a more stable perception positioning solution that can operate for a long time.

[0004] The invention patent with application number 202311171675.8 discloses a multimodal BEV surround perception method, including: collecting multi-eye images of the vehicle's surrounding environment, using a transformer-based backbone network to extract image features of the images, and obtaining BEV features corresponding to the multi-eye images; collecting laser radar original point cloud data and millimeter wave radar original point cloud data, performing temporal pre-fusion on the original point cloud data, performing feature extraction through a point cloud feature extraction backbone network, and obtaining BEV features corresponding to the laser radar point cloud data and BEV features corresponding to the millimeter wave radar point cloud data; fusing the obtained multimodal BEV features, and then fusing the multi-frame BEV features to obtain temporal BEV features, performing 3D target detection on the temporal BEV features, and outputting 3D target detection results. The above invention can improve the perception detection accuracy and generalization in multiple scenarios. However, the above research was conducted under a supervised setting and cannot be well expanded when processing various new data. Unsupervised domain adaptation (BEV) is a method that effectively learns from various unlabeled target data and addresses the challenges of domain adaptation (BEV) by leveraging the complementarity of image view features and fusing BEV features. Summary of the invention

[0005] In view of the technical problems of poor safety and reliability of existing camera-based autonomous driving, the present invention proposes an adaptive perception and positioning method based on unsupervised fusion BEV, which realizes the complementary advantages and cross-validation of fusion BEV and real-time image information under complex driving conditions, thereby improving the safety and reliability of autonomous driving.

[0006] In order to achieve the above object, the technical solution of the present invention is implemented as follows: an adaptive perception positioning method based on unsupervised fusion BEV, the steps of which are as follows:

[0007] S1. Use laser radar to collect point cloud data of the environment information near the vehicle in real time, and use camera to collect image view data of the environment information near the vehicle in real time;

[0008] S2. Convert the multi-view image view data and point cloud data into the same BEV space, generate camera BEV features and point cloud BEV features respectively, fuse the camera BEV features and point cloud BEV features by projection to obtain fused BEV features, perform semantic segmentation and instance segmentation on the fused BEV features and generate a vectorized fused BEV map;

[0009] S3, introducing a learnable query through query-based collaborative adaptation, realizing the interaction between the multi-view image view data obtained in step S1 and the fused BEV map obtained in step S2 and their collaborative adaptation, and generating a spatial perception result;

[0010] S4: Perform self-inspection on the spatial perception results, check the rationality of the image, and complete the closed loop of the perception and positioning process.

[0011] Preferably, the vectorized fused BEV map uses geometric shapes to represent elements in the map, semantic segmentation is used to identify different categories in the fused BEV map, and instance segmentation is used to distinguish different individuals in the same category.

[0012] Preferably, the image view data is projected onto the trolley coordinate system of the three-dimensional trolley to generate a camera BEV feature; two voxel-based models and one column-based model are selected to encode the point cloud data into the BEV space to obtain the point cloud BEV feature; the camera BEV feature and the point cloud BEV feature are connected, and the camera BEV feature and the point cloud BEV feature are fused with static weights to obtain a fused BEV feature;

[0013] Connecting the camera BEV features and the point cloud BEV features is to make a one-to-one correspondence between the point cloud BEV features and the camera BEV features collected at the same time. The implementation method is: jointly calibrate the lidar and the camera to establish a unified coordinate system; project the point cloud BEV features onto the camera plane, and establish a corresponding relationship by comparing the position of the projection points with the image feature points; use the ICP point cloud registration method to further refine the correspondence between the feature points in the camera BEV features and the point cloud BEV features and eliminate mismatches to obtain fused BEV features.

[0014] Preferably, the query-based collaborative adaptation includes query-based adversarial learning and query-based self-training, wherein the query-based adversarial learning utilizes useful information queried from image view data or fused BEV features to standardize the adversarial learning of query-based self-training, and the query-based self-training utilizes useful information queried from multi-view image view data and fused BEV features to standardize the self-training of query-based adversarial learning.

[0015] Preferably, the loss function of the query-based self-training is:

[0016]

[0017] Among them, the loss function L cls is the multi-label classification loss function calculated on the source data and the target data, p cls Represents the image query feature q ～iv The result of multi-label classification is Indicates the merging of object category labels in the source data to generate multi-label classification annotations for image views, L det () represents the three-dimensional loss function, p det It represents the result of multi-label classification with manually annotated fused BEV features. It is to merge the object category labels in the source data to generate BEV feature multi-label classification annotations.

[0018] Preferably, we introduce a learnable image view query q iv Image view feature decoder D iv , image view query q iv and image view features f iv The interaction between them generates the image query feature q ～iv =D iv (q iv ,f iv ), where f iv represents the result of encoding multiple image view data into image view features; D iv (,) represents the image view feature encoder;

[0019] The image query feature q ～iv Input into the multi-label classification head to predict the probability of each object category and obtain the image query feature q ～iv The result of multi-label classification is p cls =H cls (q ～iv );

[0020] Among them, the function H of the multi-label classification head cls () is derived from the multi-label classification loss function L cls (p cls ,y iv ) training, y iv Represents the multi-label classification annotations of image views obtained by manually annotating the semantics of image view items.

[0021] Preferably, the query-based adversarial learning uses two domain classifiers to measure the image view query features q ～iv and BEV query feature q ～bev The inter-domain distance of the measured domains is used for mutual regularization.

[0022] Preferably, a domain classifier is used C iv and C bev To measure the inter-domain distance between image view data and fused BEV features, the method is as follows:

[0023]

[0024] in, and Represents the image view multi-label classification annotation y iv BEV query features before and after decoding; and denotes the decoded BEV query features interacting with the BEV features of the source and target domains, respectively; iv represents the inter-domain distance of 2D image views, λ bev represents the inter-domain distance of 3D BEV features; C iv ,C bev Represents the domain classifier of image view data and fused BEV features respectively;

[0025] The mutual regularization between image view data and fused BEV features is expressed as:

[0026]

[0027] Among them, L al (·) is the adversarial learning loss function for cross-domain alignment, L qal Represents the regularization results of image view data and fused BEV features.

[0028] Preferably, the BEV query q fused with the BEV map bev Interact with the fused BEV features to generate the decoded BEV query features q ～bev =D bev (q bev ,f bev ), where f bev Denotes the result of encoding multi-view image view data into fused BEV features. ,D bev () represents the processing function of the fused BEV encoder;

[0029] Query-based adversarial learning leverages predictions from image view data or fused BEV features to denoise predictions from another feature:

[0030]

[0031] in, and represents the image view and prediction of BEV features before / after cross-view denoising, max(·) represents the maximum probability among the BEV predictions for each class;

[0032] The global class probability distribution is obtained by accumulating predictions that capture the denoising of comprehensive 2D and 3D information, and the global class probability distribution is used to promote pseudo-label generation; according to the estimated global class probability distribution, the previous ν Percentage predictions are used as pseudo labels for each class; predictions for denoised image views are thresholded using pseudo labels and prediction of BEV characteristics Get the pseudo label of an image view and pseudo labels of BEV features

[0033] The overall constraint objectives are:

[0034]

[0035] Among them, the intermediate parameter ε={ε 2d ,ε 3d},D={D iv ,D bev},H={H cls ,H det},C={C iv ,C bev};L iv represents the loss function of image view data, L bev represents the loss function of integrating BEV features; ε 2d ,ε 3d Denote image view data and fused BEV features, respectively, and D iv ,D bev Denotes the feature decoders of the two modalities, H cls ,H det Represent the multi-label classification results and 3D detection head, C iv ,C bev Represents the domain classifiers for image view information and BEV features respectively.

[0036] An adaptive perception and positioning system based on unsupervised fusion BEV, characterized by comprising: a fusion BEV module for outputting lane line information; a GPS receiver for receiving global positioning system satellite signals and determining ground space positions; an inertial sensor for outputting IMU signals; an unsupervised positioning controller for positioning perception results; an automatic driving control center for performing interaction and collaborative adaptation, self-checking of perception results and completing the perception and positioning process;

[0037] The fusion BEV module and the unsupervised positioning controller are both connected to the vehicle body CAN bus, and the unsupervised positioning controller is respectively connected to the fusion BEV module, the GPS receiver and the inertial sensor; at the same time, the GPS receiver, the inertial sensor, the fusion BEV module and the unsupervised positioning controller are all connected to the automatic driving control center;

[0038] The fusion BEV module is a forward-looking monocular camera and has six cameras and a 16-line laser radar set around the vehicle body.

[0039] The beneficial effects of the present invention are as follows: the BEV images generated by the fusion perception of multiple monocular cameras and sixteen-line laser radars and the real-time images output by the unsupervised positioning controller are used to perform unsupervised adaptive learning on the car jointly controlled by the two, and after cross-validation of the two data information, the normal driving of the vehicle is guaranteed to the greatest extent, thereby building a complete set of autonomous positioning systems suitable for diverse road conditions. On the basis of giving full play to the advantages of different sensors, the vehicle can be guaranteed to drive smoothly on the basis of the unsupervised positioning controller in the face of sensor disorder. The fusion BEV under unsupervised learning of the present invention can quickly detect environmental information and verify it autonomously, so that the accuracy of vehicle positioning is guaranteed, while improving the vehicle experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 The figure is a flow chart of the positioning method of the present invention.

[0042] Figure 2 Schematic diagram of the sensors required for the BEV fusion sensing unit in the present invention.

[0043] Figure 3 It is a structural schematic diagram of the adaptive perception and positioning system based on unsupervised fusion BEV of the present invention.

[0044] In the figure, 1- fusion BEV module (6 monocular cameras + 1 16-line laser radar); 2- unsupervised positioning controller; 3- autonomous driving controller; 4- GPS receiver; 5- inertial sensor IMU. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Example 1

[0047] Image view can extract a variety of image features: color features, texture features, shape features, and spatial relationship features. Color features are global features that describe the surface properties of the scene corresponding to an image or image region. Generally, color features are based on pixel features. Since color is insensitive to changes in the direction, size, etc. of an image or image region, color features cannot capture the local features of objects in an image well.

[0048] Various retrieval methods based on shape features can effectively use the objects of interest in the image for retrieval. However, the current shape-based retrieval methods still lack a relatively complete mathematical model, and the retrieval results are often unreliable when the object is deformed. To fully describe the target, it often requires high computing time and storage. In addition, the 3-D object represented by the 2-D image is actually just the projection of the object on a certain plane in space. The shape reflected from the 2-D image is often not the real shape of the 3-D object. Due to the change of viewpoint, various distortions may occur. Generally, there are two types of representation methods for shape features, one is contour features and the other is regional features. The contour features of the image are mainly aimed at the outer boundary of the object, while the regional features of the image are related to the entire shape area.

[0049] Spatial relationship features refer to the spatial position or relative direction relationship between multiple targets segmented from an image. These relationships can also be divided into connection / adjacency relationships, overlap / overlapping relationships, and inclusion / inclusion relationships. Usually, spatial position information can be divided into two categories: relative spatial position information and absolute spatial position information. There are two ways to extract image spatial relationship features: one method is to first automatically segment the image, divide the objects or color areas contained in the image, and then extract image features based on these areas and create an index; the other method is to simply divide the image evenly into several regular sub-blocks, and then extract features for each image sub-block and create an index.

[0050] In the case of point cloud and multi-view image input, the two streams extract features and transform them into the same BEV space respectively: 1) project the camera view features onto the 3D car coordinate features to generate the camera BEV features; 2) extract the point cloud BEV features from the point cloud. Then, the fusion module integrates the BEV features of the two modes. Finally, the task-specific head is built on the fused BEV features and the target value of the 3D target is predicted. Lift-Splat-Shoot is used as the camera stream to project the multi-view image features onto the 3D coordinate features to generate the camera BEV features. Similarly, for the lidar stream, three popular models, two voxel-based models and one column-based model, are selected to encode the point cloud information into the BEV space. In order to effectively fuse the BEV features from the camera and lidar sensors, a simple channel attention module is used to select the important fused features. In this way, the construction of the task-specific head is realized, laying the foundation for the subsequent steps.

[0051] Supervised learning is a training method in machine learning. It refers to the process of adjusting the parameters of the classifier using a set of samples of known categories to achieve the required performance. It is also called supervised training or teacher learning. It is a machine learning task of inferring a function from labeled training data.

[0052] The "supervision" in supervised learning does not mean that humans observe the correctness of the machine, but rather execute the following process: 1. Select a mathematical model suitable for the target task; 2. Give the machine a part of known questions and answers (i.e., training sets) for training simulation; 3. The machine summarizes the rules based on the content of the training set and forms its own methodology; 4. After the methodology is formed, humans give the machine new problems to be solved (test sets) and let it solve them. Compared with supervised learning, unsupervised learning does not require the true value of the predicted quantity for each sample, which makes it more flexible and efficient in some applications. The purpose of unsupervised learning is usually to discover the inherent structure of the data. Using unsupervised learning mechanisms in the field of autonomous driving allows unmanned vehicles to quickly adapt to various external environments, greatly improving the accuracy and stability of perception and positioning.

[0053] See also Figure 1 As shown, this embodiment discloses an adaptive perception positioning method based on unsupervised fusion BEV, comprising the following steps:

[0054] S1. Load the point cloud data collected by the laser radar in real time and the image view data collected by the camera in real time into the memory respectively.

[0055] The fusion BEV module contains two sensors: lidar and camera. After the lidar and camera collect environmental information in real time, they will process the collected point cloud data and image view data to obtain BEV information. At the same time, the two sets of data will be loaded into the memory for the next self-training. This step is to keep the image view data separately in preparation for the next step of collaborative adaptation.

[0056] S2. Convert the multi-view image view data and point cloud data into the same BEV space, generate camera BEV features and point cloud BEV features respectively, fuse the camera BEV features and point cloud BEV features through projection to obtain fused BEV features, and generate a vectorized fused BEV map based on the results of semantic segmentation and instance segmentation.

[0057] In the BEV space, semantic segmentation and instance segmentation are performed on the fused BEV features. Semantic segmentation is used to identify different categories in the map (such as roads, lane lines, traffic signs, etc.), while instance segmentation is used to distinguish different individuals in the same category (such as different vehicles, pedestrians, etc.). Vectorized fused BEV maps use geometric shapes (such as line segments, polygons, etc.) to represent elements in the map, which has higher accuracy and flexibility than traditional raster maps.

[0058] The lane information output by the fusion BEV module is preprocessed, including: in the case of point cloud data and multi-view image view data input, the camera stream and radar stream respectively convert the multi-view image view data and point cloud data into the same BEV space, which better ensures the integrity of the collected data compared with the traditional method. The image view data collected by the camera is projected onto the car coordinate system of the three-dimensional car to generate the camera BEV features. In the specific implementation, Lift-Splat-Shoot is used as the camera stream to project the multi-view image view data onto the car coordinate system to generate the camera BEV features. For the radar stream, three popular models are selected: two voxel-based models and one column-based model. The point cloud data is encoded into the BEV space to obtain the point cloud BEV features. In order to effectively fuse the camera BEV features and the point cloud BEV features, a dynamic fusion module is proposed to integrate the BEV features of the two modes. Given two features in the same spatial dimension, an intuitive idea is to connect them and fuse them with static weights. Therefore, the camera weight and radar weight are set to 0.5 to fuse the camera BEV features and the point cloud BEV features. Finally, the fused BEV features are saved.

[0059] When the laser radar receives environmental information, it generates corresponding point cloud BEV features, and the point cloud BEV features collected at the same time are matched one by one with the camera BEV features. First, the laser radar and camera need to be jointly calibrated, including determining their relative position (translation) and orientation (rotation) so that a unified coordinate system can be established. The calibration process usually involves collecting scene data containing common feature points and using these feature points to estimate the transformation matrix between sensors. The point cloud BEV features generated by the laser radar are preprocessed as necessary, including filtering, denoising, downsampling, etc., to reduce data redundancy and improve the efficiency of subsequent processing. Joint calibration makes the laser radar and camera form a complete perception system, projects the point cloud BEV features onto the camera plane, and establishes a correspondence by comparing the positions of the projection points and the image feature points. The ICP point cloud registration method is used to further refine the correspondence of the feature points and eliminate mismatches, thereby obtaining the result of the fusion of the camera BEV features and the point cloud BEV features.

[0060] S3. Through a new query-based collaborative adaptation, a learnable query is introduced to achieve the interaction between the multi-view image view data obtained in step S1 and the fused BEV map obtained in step S2 and their collaborative adaptation to generate a spatial perception result.

[0061] Intuitively, the global 3D information from the fused BEV map helps adapt to the multi-view image view data. And the local 2D information with less domain variation in the multi-view image view data helps adapt to the fused BEV map. Based on this concept, two query-based collaborative adaptation techniques are designed, namely query-based adversarial learning (QAL) and query-based self-training (QST). QAL / QST utilizes the road condition feature information queried from the multi-view image view data or the fused BEV map to regularize the "adversarial learning" / "self-training" of the other.

[0062] The complementarity of multi-view image view data and fused BEV features is used for unsupervised BEV perception adaptation, including two designs: query-based adversarial learning (QAL) and query-based self-training (QST). Query-based adversarial learning uses the useful information queried from image view data or fused BEV features to regularize the adversarial learning of QST, and query-based self-training uses the useful information queried from multi-view image view data and fused BEV features to regularize the self-training of QAL.

[0063] To capture the 2D information in image view features with less cross-domain, we introduce a learnable image view query q iv Image view feature decoder D iv Therefore, the image view query q iv and image view features fiv The interaction between them generates the image query feature q ～iv As shown below:

[0064] q ～iv =D iv (q iv ,f iv ) (1)

[0065] Among them, f iv Represents the result of encoding multi-camera images into image view features. iv (,) represents the image view feature encoder.

[0066] Then the image query feature q ～iv Input to the multi-label classification head H cls To predict the probability of each object category, the multi-label classification head H cls From the multi-label classification loss function L cls Training is as follows:

[0067] p cls =H cls (q ～iv ),L iv =L cls (p cls ,y iv ) (2)

[0068] Among them, y iv represents the multi-label classification annotation of the image view obtained by manually annotating the semantics of the image view items. cls represents the result of multi-label classification of image query features, H cls () represents the multi-label classification function. Multi-label classification loss function L cls is the loss function calculated for the two-dimensional source data and target data. For the source data x with labeled information s ,k s ,y s , by merging the source data y s Object category labels in Generate Image View Multi-label Classification Annotations For the unlabeled target data x T ,k T , generated according to the model prediction defined in the formula x represents multiple camera images, k represents the camera configuration, and y represents the annotations of the 3D object. s represents the annotated source data, x T Represents unlabeled target data.

[0069]

[0070] Among them, L qst represents the training loss of QST, Indicates the merging of object category labels in the source data to generate multi-label classification annotations for image views, p det represents the result of multi-label classification with manually annotated fused BEV features, L det () represents the three-dimensional loss function, It is to merge the object category labels in the source data to generate BEV feature multi-label classification annotations, that is, a more comprehensive collection of classification annotations.

[0071] Therefore, the decoded image view query feature q ～iv Query q from an image view iv Rich 2D semantic and location information is captured in , which contain less inter-domain variance than BEV features. BEVs typically need to fuse data from multiple sensors, which may become sparse or unavailable in certain areas or conditions. For example, in low-light or heavily occluded environments, cameras may not provide enough information; and radar performance may also be affected in severe weather or terrain conditions. This data sparsity increases inter-domain variance in BEVs because the available sensor data may vary in different scenarios and environments. In contrast, 2D images typically only require one camera to capture enough information, so they are more likely to maintain consistent performance under various conditions. Therefore, it can greatly benefit the adaptation of BEV features. In addition, the image view query q iv The multi-label classification training objective encourages capturing more balanced information between different categories, which is beneficial to the adaptation of BEV features, which are trained with single object category annotations and thus often suffer from obvious class imbalance problems.

[0072] In order to capture the global 3D information in the BEV feature, we directly use the ready-made BEV query q in (4) bev ,q bev is the annotation result of the fused BEV map, which can be obtained directly. It interacts with the fused BEV features to generate the decoded BEV query feature q ～bev Since the camera BEV feature is encoded with the camera configuration, the BEV query q bev is trained with 3D object annotations, and the decoded BEV query feature q ～bev Rich global 3D information is captured, including object locations in 3D BEV space, which can help adapt to image view features, where little global 3D information is captured.

[0073] q ～bev =D bev (q bev ,fbev ) (4)

[0074] Among them, f bev It represents the result of encoding multi-camera images into fused BEV features. bev () represents the processing function of the fused BEV encoder.

[0075] QAL utilizes the useful information from the query of image view or BEV features to regularize the adversarial learning of the other party. QAL uses two domain classifiers to measure the image view query feature q respectively. ～iv and BEV query feature q ～bev The inter-domain distance is measured, and the measured inter-domain distance is used for mutual regularization. Intuitively, the decoded image view query feature captures rich two-dimensional semantics and location information. Through deep learning and computer vision technology, first, the encoder is designed; then, the query feature is designed; then, the decoder, cross-modal interaction, training and optimization are performed, and finally visualization is performed to obtain the final result, which is the perfected fusion BEV feature. Compared with the two, the image view query feature has a smaller difference between domains, and the decoded BEV query feature q ～bev Capturing rich global three-dimensional information of the BEV space also leads to greater differences between BEV feature domains.

[0076] Therefore, QAL simultaneously alleviates the inter-domain differences of local 2D information in image view features and global 3D information in BEV features, both of which are crucial for localizing and recognizing objects and backgrounds in 3D space. In addition, the adversarial learning of 2D image view query features involves less 3D information, while the BEV query feature q obtained by formula (4) ～bev It can be effectively regularized by providing rich global 3D information, i.e., 2D image view adversarial learning is enhanced for samples with large inter-domain differences of 3D BEV, while it is weakened for samples with small inter-domain differences of 3D BEV. Similarly, the adversarial learning of 3D BEV query features can be regularized by image view query features that capture rich 2D semantic and position information.

[0077] In order to better compare the domain differences between image view data and fused BEV features, a domain classifier is used C iv and C bev To measure the inter-domain distance between 2D image views and 3D BEV features, the method is as follows:

[0078]

[0079] in, and They represent the multi-label classification annotations y of the image views in equation (1) respectively. iv BEV query features before and after decoding. and denote the decoded BEV query features interacting with the source and target domain BEV features in equation (4). iv represents the inter-domain distance of 2D image views, λ bev Represents the inter-domain distance of 3D BEV features.

[0080] The QAL mutual regularization between image views and BEV features can be expressed as:

[0081]

[0082] Among them, L al (·) is a widely adopted adversarial learning loss function for cross-domain alignment. L qal Represents the regularization result of image view information and BEV features.

[0083] QST utilizes the useful information from the image view and BEV feature queries to regularize their self-training. Intuitively, the decoded image view query feature q ～iv Captures rich 2D semantics and location information, with small differences between domains, while the decoded BEV query features Capturing rich global 3D information in the BEV space. Therefore, the two features complement each other and effectively regularize self-training by combining comprehensive 2D and 3D information.

[0084] Specifically, QST first utilizes the predictions from either image view or BEV feature to denoise the predictions from another feature:

[0085]

[0086] in, and represents the image view and the prediction of BEV features before / after cross-view denoising. The main intention of this step is cross-denoising, that is, the confirmation of the final view is completed through the information of multiple cameras. The operation max(·) represents the maximum probability in the BEV prediction of each category. Note that the above cross-view denoising operation is performed separately for each camera view.

[0087] Then, QST obtains the global class distribution by accumulating denoised predictions that capture comprehensive 2D and 3D information, and further exploits the global class distribution to facilitate pseudo-label generation. Specifically, the global class probability distribution is modeled as a Gaussian distribution where c is the category index. The mean is estimated by maintaining a first-in, first-out queue of historical forecasts. and standard deviation

[0088]

[0089] Where B represents the queue length, represents the denoised predicted BEV feature in formula (7) The j-th historical forecast in . Note that only Because it already contains the denoised predicted image view after cross denoising 2D information in .

[0090] Cumulative mean along the training process and variance To estimate the Gaussian distribution more stably

[0091]

[0092] Among them, μ c , Represent the mean and variance respectively. γ represents the exponential moving average coefficient of the smooth update. By trying different smooth coefficient values, observe their impact on the model or prediction results, and then select the optimal smooth coefficient. Usually multiple experiments and evaluations are required.

[0093] According to the estimated global class probability distribution, the first ν The percentage of predictions is used as the pseudo-label for each class. The threshold for pseudo-labels τc It can be determined by solving the following equation:

[0094]

[0095] Where Φ(·) is Gaussian distribution The cumulative distribution function of the standard normal distribution is selected. Note that the pseudo-label threshold {τ c |c=1,...,C} for the denoised image views and BEV predictions and Get the pseudo label of an image view and pseudo labels of BEV features Substitute the standard distribution function of the normal distribution into the equation to obtain the solution.

[0096] Therefore, pseudo-label generation methods have three desirable characteristics:

[0097] 1) The thresholds are state-of-the-art as they are determined dynamically during training based on the complementary 2D and 3D information captured by image views and BEV features;

[0098] 2) Alleviate the class imbalance problem by selecting the same percentage of pseudo labels for each class;

[0099] 3) It works online in real time.

[0100] Therefore, the training loss of QST can be expressed as:

[0101]

[0102] In summary, the overall constraint objectives can be formulated as follows:

[0103]

[0104] Where ε={ε 2d ,ε 3d},D={D iv ,D bev},H={H cls ,H det},C={C iv ,C bev}. iv Represents the constraint target of the image view, L bev Represents the constraint target of the fusion BEV model, and these two constraint targets are loss functions. 2d ,ε 3d They represent image view information and BEV feature information respectively. Both of them are information inherent in the image. iv ,D bev Describes the feature decoder of the two modalities, H cls ,H det They represent the multi-label classification results and 3D detection head respectively. Cls is the annotation result, det is the generated 3D detection head, which is the result of fusion BEV, and C iv ,C bev Represents the domain classifiers of image view information and BEV features. Under the condition that the constraints are met, the result corresponding to the minimum value is obtained.

[0105] Step S3 mainly cross-validates the fused BEV map and image view, supplements the omissions of the fused BEV map, and then generates the final result, namely the spatial perception result.

[0106] S4: self-check the spatial perception results, check the rationality of the image, and complete the closed loop of the perception and positioning process. By confirming the rationality of the output results of step S3, the continuity of the lanes is mainly aimed at to ensure that there are no multiple lanes without information. When there are more than 10 abnormal interruptions, the alarm is issued that the result is wrong.

[0107] Compared with the prior art, the present invention has the following advantages: the adaptive perception positioning method based on unsupervised fusion BEV of the present invention uses the BEV image generated by the fusion perception of multiple monocular cameras and sixteen-line laser radars and the real-time image output by the unsupervised positioning controller to perform unsupervised adaptive learning on the car jointly controlled by the two, and after cross-validation of the two data information, the normal driving of the vehicle is guaranteed to the greatest extent, thereby building a complete set of autonomous positioning systems suitable for diverse road conditions. On the basis of giving full play to the advantages of different sensors, it can also ensure the smooth driving of the vehicle on the basis of the unsupervised positioning controller in the face of sensor disorder. The fusion BEV under unsupervised learning can quickly detect and obtain environmental information, and verify it autonomously, so that the accuracy of vehicle positioning is guaranteed, while improving the vehicle experience.

[0108] Example 2

[0109] like Figure 3 As shown, an adaptive perception and positioning system based on unsupervised fusion BEV includes the following components: a fusion BEV module for outputting lane information; the fusion BEV module is connected to the vehicle body CAN bus. A GPS receiver is used to receive global positioning system satellite signals and determine the ground space position. An inertial sensor IMU is used to output IMU signals. An unsupervised positioning controller is used to complete hardware system initialization, positioning algorithm initialization, and positioning perception results; the unsupervised positioning controller is connected to the vehicle body CAN bus, and is connected to the fusion BEV module, the GPS receiver, and the inertial sensor. The automatic driving control center is used to perform the cross-validation function of step S3 in Example 1 and the navigation positioning loop detection function in step S4; at the same time, the GPS receiver, the inertial sensor, the fusion BEV module, and the unsupervised positioning controller are all connected to the automatic driving control center. This method can ensure a faster and more stable data flow and less data loss.

[0110] like Figure 2 As shown, the fusion BEV module is a forward-looking monocular camera and requires six cameras and a 16-line laser radar around the vehicle body for feature point fusion.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive perception positioning method based on unsupervised fusion BEV, characterized in that: The steps are as follows: S1. Use laser radar to collect point cloud data of the environment information near the vehicle in real time, and use camera to collect image view data of the environment information near the vehicle in real time; S2. Convert the multi-view image view data and point cloud data into the same BEV space, generate camera BEV features and point cloud BEV features respectively, fuse the camera BEV features and point cloud BEV features by projection to obtain fused BEV features, perform semantic segmentation and instance segmentation on the fused BEV features and generate a vectorized fused BEV map; S3, introducing a learnable query through query-based collaborative adaptation, realizing the interaction between the multi-view image view data obtained in step S1 and the fused BEV map obtained in step S2 and their collaborative adaptation, and generating a spatial perception result; S4: Perform self-inspection on the spatial perception results, check the rationality of the image, and complete the closed loop of the perception and positioning process.

2. The adaptive perception positioning method based on unsupervised fusion BEV according to claim 1 is characterized in that: The vectorized fused BEV map uses geometric shapes to represent elements in the map, semantic segmentation is used to identify different categories in the fused BEV map, and instance segmentation is used to distinguish different individuals in the same category.

3. The adaptive perception positioning method based on unsupervised fusion BEV according to claim 1 or 2, characterized in that: The image view data is projected onto the trolley coordinate system of the 3D trolley to generate the camera BEV features; two voxel-based models and one column-based model are selected to encode the point cloud data into the BEV space to obtain the point cloud BEV features; the camera BEV features and the point cloud BEV features are connected, and the camera BEV features and the point cloud BEV features are fused with static weights to obtain fused BEV features; Connecting the camera BEV features and the point cloud BEV features is to make a one-to-one correspondence between the point cloud BEV features and the camera BEV features collected at the same time. The implementation method is: jointly calibrate the lidar and the camera to establish a unified coordinate system; project the point cloud BEV features onto the camera plane, and establish a corresponding relationship by comparing the position of the projection points with the image feature points; use the ICP point cloud registration method to further refine the correspondence between the feature points in the camera BEV features and the point cloud BEV features and eliminate mismatches to obtain fused BEV features.

4. The adaptive perception positioning method based on unsupervised fusion BEV according to claim 3 is characterized in that: The query-based collaborative adaptation includes query-based adversarial learning and query-based self-training. The query-based adversarial learning uses useful information queried from image view data or fused BEV features to standardize the adversarial learning of query-based self-training, and the query-based self-training uses useful information queried from multi-view image view data and fused BEV features to standardize the self-training of query-based adversarial learning.

5. The adaptive perception positioning method based on unsupervised fusion BEV according to claim 4 is characterized in that: The loss function of the query-based self-training is: Among them, the loss function L cls is the multi-label classification loss function calculated on the source data and the target data, p cls Represents the image query feature q ～iv The result of multi-label classification is Indicates the merging of object category labels in the source data to generate multi-label classification annotations for image views, L det () represents the three-dimensional loss function, p det It represents the result of multi-label classification with manually annotated fused BEV features. It is to merge the object category labels in the source data to generate BEV feature multi-label classification annotations.

6. The adaptive perception positioning method based on unsupervised fusion BEV according to claim 5 is characterized in that: We introduce a learnable image view query q iv Image view feature decoder D iv , image view query q iv and image view features f iv The interaction between them generates the image query feature q ～iv = Div (q iv ,f iv ), where f iv represents the result of encoding multiple image view data into image view features; D iv (,) represents the image view feature encoder; The image query feature q ～iv Input into the multi-label classification head to predict the probability of each object category and obtain the image query feature q ～iv The result of multi-label classification is p cls =H cls (q ～iv ); Among them, the function H of the multi-label classification head cls () is derived from the multi-label classification loss function L cls (p cls ,y iv ) training, y iv Represents the multi-label classification annotations of image views obtained by manually annotating the semantics of image view items.

7. The adaptive perception positioning method based on unsupervised fusion BEV according to any one of claims 4 to 6, characterized in that: The query-based adversarial learning uses two domain classifiers to measure the image view query features q ～iv and BEV query feature q ～bev The inter-domain distance of the measured domains is used for mutual regularization.

8. The adaptive perception positioning method based on unsupervised fusion BEV according to claim 7 is characterized in that: Using Domain Classifier C iv and C bev To measure the inter-domain distance between image view data and fused BEV features, the method is as follows: in, and Represents the image view multi-label classification annotation y iv BEV query features before and after decoding; and denotes the decoded BEV query features interacting with the BEV features of the source and target domains, respectively; iv represents the inter-domain distance of 2D image views, λ bev represents the inter-domain distance of 3D BEV features; C iv ,C bev Represents the domain classifier of image view data and fused BEV features respectively; The mutual regularization between image view data and fused BEV features is expressed as: Among them, L al (·) is the adversarial learning loss function for cross-domain alignment, L qal Represents the regularization results of image view data and fused BEV features.

9. The adaptive perception positioning method based on unsupervised fusion BEV according to claim 8 is characterized in that: The BEV query q of the fused BEV map bev Interact with the fused BEV features to generate the decoded BEV query features q ～bev =D bev (q bev ,f bev ), where f bev Denotes the result of encoding multi-view image view data into fused BEV features. ,D bev () represents the processing function of the fused BEV encoder; Query-based adversarial learning leverages predictions from image view data or fused BEV features to denoise predictions from another feature: in, and represents the image view and prediction of BEV features before / after cross-view denoising, max(·) represents the maximum probability among the BEV predictions for each class; The global class probability distribution is obtained by accumulating predictions that capture the denoising of comprehensive 2D and 3D information, and the global class probability distribution is used to promote pseudo-label generation; according to the estimated global class probability distribution, the previous ν Percentage predictions as pseudo labels for each class; predictions for denoised image views using pseudo label thresholds and prediction of BEV characteristics Get the pseudo label of an image view and pseudo labels of BEV features The overall constraint objectives are: Among them, the intermediate parameter ε={ε 2d ,ε 3d },D={D iv ,D bev },H={H cls ,H det },C={C iv ,C bev };L iv represents the loss function of image view data, L bev represents the loss function of integrating BEV features; ε 2d ,ε 3d Denote image view data and fused BEV features, respectively, and D iv ,D bev Denotes the feature decoders of the two modalities, H cls ,H det Represent the multi-label classification results and 3D detection head, C iv ,C bev Represents the domain classifiers for image view information and BEV features respectively.

10. The positioning system according to the adaptive perception positioning method based on unsupervised fusion BEV according to claims 1-9, characterized in that: include: Fusion BEV module, used to output lane line information; GPS receiver, used to receive global positioning system satellite signals and determine the ground space position; inertial sensor, used to output IMU signal; unsupervised positioning controller, used to locate the perception results; automatic driving control center, used to perform interaction and collaborative adaptation, self-check of perception results and complete the perception positioning process; The fusion BEV module and the unsupervised positioning controller are both connected to the vehicle body CAN bus, and the unsupervised positioning controller is respectively connected to the fusion BEV module, the GPS receiver and the inertial sensor; at the same time, the GPS receiver, the inertial sensor, the fusion BEV module and the unsupervised positioning controller are all connected to the automatic driving control center; The fusion BEV module is a forward-looking monocular camera and has six cameras and a 16-line laser radar set around the vehicle body.

Images