Long-distance high-definition map online prediction method and device based on multi-sensor fusion

Abstract

The invention discloses a long-distance high-definition map online prediction method and device based on multi-sensor fusion. The method comprises the steps that sensors are installed and debugged; image data and point cloud data around the vehicle are collected in real time for subsequent processing and mapping; generating a mapping data set containing environment information by using the collected image data and point cloud data; constructing a high-definition map network model, namely constructing a high-precision aerial view feature map by combining LiDAR point cloud data and camera image data so as to support perception and decision of an automatic driving system; defining a loss function for training the high-definition map network model; and finally, generating a high-definition map in an actual automatic driving scene by utilizing the trained model, and supporting path planning and navigation tasks. According to the method, the environment perception capability of the automatic driving system can be remarkably improved, and more reliable data support can be provided for downstream path planning and motion control tasks, so that the driving safety of the vehicle and the user experience are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of high-precision maps, and in particular to a method and device for online prediction of long-distance high-definition maps based on multi-sensor fusion. Background Art

[0002] In the development of autonomous driving technology, the generation of high-precision semantic maps (HD Maps) is a crucial task. HD Maps contain semantic layers such as lane boundaries, road dividers, and pedestrian crosswalks, providing precise location information about nearby infrastructure, roads, and the environment, thereby helping autonomous vehicles to navigate safely.

[0003] Existing high-precision map generation technology is mainly based on sensor fusion, including multiple sensors such as LiDAR (laser radar) and cameras. Traditional methods usually build high-precision maps through the following steps: first, use LiDAR to obtain point cloud data, then use simultaneous localization and mapping (SLAM) algorithms to create a globally consistent map, and finally perform semantic annotation.

[0004] For example, the Chinese patent publication number CN107462897B, the invention name is "Three-dimensional mapping method based on laser radar", which mainly provides a three-dimensional mapping method based on laser radar, which mainly solves the problem that the existing 2D laser radar is difficult to draw detailed three-dimensional maps in real time without the assistance of other sensors. First, a point cloud with coordinate information is obtained; then the point cloud is filtered using a bilateral filtering algorithm; then the laser radar ranging algorithm calculates the displacement of each feature point in the laser radar point cloud; finally, the scanned point cloud is aligned to the world coordinate system using a drawing algorithm to form a three-dimensional point cloud map. This method has shortcomings in terms of real-time performance and three-dimensional data accuracy.

[0005] The Chinese patent publication number is CN114608561B, and the invention name is "A positioning and mapping method and system based on multi-sensor fusion". It mainly provides a positioning and mapping method and system based on multi-sensor fusion. After acquiring lidar data, inertial measurement data and video image data, by constructing a laser inertial odometer and a visual inertial fusion odometer, the lidar data and the inertial measurement data are fused, and the inertial measurement data and the video image data are fused, and then the sliding window optimization, loop detection and global pose optimization of the above two fusion results are performed to complete the positioning and construction of the map. Although this method integrates lidar data, inertial measurement data and video image data, the sparsity of LiDAR data and the lack of camera depth information are still significant problems in long-distance perception scenarios.

[0006] Although the above technologies perform well in short-range environment perception and map generation, they have the following significant drawbacks: Lack of real-time performance: Most high-precision map generation technologies are mainly carried out in offline environments, resulting in untimely map updates and inability to adapt to changes in the dynamic environment.

[0007] Limited perception distance: Existing methods can usually only generate maps within 30 meters, and the long-distance perception capability is insufficient. This limits the application of vehicles on highways and complex urban roads, and may lead to inaccurate path planning and increase driving risks.

[0008] Insufficient data fusion: Although some methods fuse data from multiple sensors, they fail to fully utilize the respective strengths of LiDAR and cameras, especially in long-range perception. Summary of the invention

[0009] The present invention proposes a long-distance high-definition map online prediction method and device based on multi-sensor fusion, which can solve at least one of the technical problems in the background technology.

[0010] To achieve the above object, the present invention adopts the following technical solutions: A long-distance high-definition map online prediction method based on multi-sensor fusion includes the following steps: Install and debug sensors; Through the deployed sensors, image data and point cloud data around the vehicle are collected in real time for subsequent processing and mapping; Generate a mapping dataset containing environmental information using the collected image data and point cloud data; Build a high-definition map network model, that is, by combining LiDAR point cloud data and camera image data, build a high-precision bird's-eye view feature map to support the perception and decision-making of the autonomous driving system; Define the loss function for training the HD map network model; The trained HD map network model is used to generate HD maps in actual autonomous driving scenarios to support path planning and navigation tasks.

[0011] Further, installing and debugging the sensor includes the following steps, Surround view cameras are installed on the front and rear bumpers, near the side mirrors, and in the center of the roof. LiDAR is installed on the roof to ensure that the entire area around the vehicle can be covered. Make sure all sensors are firmly mounted on the vehicle to avoid vibration or displacement; adjust the angle and position of each sensor to ensure maximum effective coverage; Each sensor is calibrated to ensure data accuracy.

[0012] Furthermore, real-time collection of image data and point cloud data around the vehicle includes the following steps: The camera collects surrounding video and static images, and the lidar generates point cloud data of the surrounding environment; A synchronous control system is used to ensure that the image and point cloud data are completely synchronized in time, and the data is preliminarily processed, including denoising the image and filtering the point cloud data.

[0013] Furthermore, building a high-definition map network model includes the following steps: Step 4-1: Use LiDAR point cloud data to build BEV feature map First, the radar point cloud data is divided into voxel grids; each voxel grid represents a small cubic area of ​​the point cloud data in 3D space; according to the preset voxel size , each point Assign to the corresponding voxels; voxelization calculation formula: ; Secondly, clustering algorithms are used to group points according to the voxels where they are located, ensuring that the point cloud data in space is effectively segmented to improve the efficiency of data processing; Again, the voxels are uniformly downsampled to reduce the computational burden; Then, each voxel block is input into the feature encoder for encoding, and the key features of the point cloud data are extracted through the encoder; Finally, the radar point cloud data is converted to the BEV perspective using the self-attention mechanism; Step 4-2: Plane project the LiDAR point cloud data This step uses the joint calibration data of LiDAR and camera to project the radar point cloud data onto the camera image. Through this step, the 3D point cloud data is aligned with the 2D image data for multimodal fusion in subsequent steps. At the same time, the radar point cloud data is used to generate a depth map for subsequent model supervision. Step 4-3: Use camera images to construct a bird’s-eye view feature map Image BEV feature extraction uses a LSS-like method to construct a bird's-eye view feature map from the 2D image data captured by the camera; By improving the camera perspective, the 2D image data is converted into a 3D feature map; then, these 3D features are projected onto the ground plane through ray projection to generate a bird's-eye view feature map; in the image mapping process, in order to improve the receptive field, convolution and dilated convolution are used to extract effective information from the feature map; at the same time, the depth map generated in step 4-2 is used to supervise the image enhancement operation; Step 4-4: Fusion of LiDAR-BEV feature map and image BEV feature map First, the point cloud BEV features and image BEV features are merged; secondly, the merged features are input into the fully connected layer to obtain the weights of each channel; finally, the weights of each channel are fused with the original merged features to obtain the fused feature map; Step 4-5: Input the fused feature map into the multi-task head to output the detection result This step adopts a multi-task head design, which includes a semantic segmentation head, an instance embedding head, and a direction prediction head. The semantic segmentation head identifies and classifies each pixel of the road and its surroundings. The instance embedding head generates instance embeddings for each independent object and distinguishes different instances of the same category. The direction prediction head predicts the direction of road elements. The final output high-definition map integrates the results of semantic segmentation, instance embedding, and direction prediction, and contains rich geographical and structural information.

[0014] Furthermore, the loss function is defined as follows: Step 5-1, semantic segmentation loss

[0015] The cross entropy loss is used to supervise the semantic segmentation task; its formula is:

[0016] in: N Indicates the total number of pixels; Indicates the i The true class of the pixel; Indicates i The predicted probability of pixels; Step 5-2: Instance Embedding Loss ; Include variance loss and distance loss ; Variance loss: calculates the distance between the embedding vector within the same instance and its average embedding vector, ensuring that points in the same instance are close to each other;

[0017] in: C Indicates the total number of instances; Indicates the instance c The number of points; Indicates i The embedding vector of each point; Representation instancec The average embedding vector of Is a hyperparameter that defines the tolerance range of variance loss; Distance loss: ensures that the average embedding vectors of different instances maintain a sufficient distance to prevent the embedding vectors of different instances from being confused;

[0018] in: is a hyperparameter that defines the minimum distance between instances; The instance embedding loss consists of a weighted sum of variance loss and distance loss;

[0019] Among them, α and β are weighting factors of variance loss and distance loss; Step 5-3: Road direction prediction loss

[0020] The road direction is predicted using a cross entropy loss, which is discretized into 36 categories evenly distributed around the circle; this loss is backpropagated only for those pixels that are on the lane and have a valid direction;

[0021] in: is the number of pixels with valid orientation; and are the true direction categories and predicted probabilities of these pixels respectively; Step 5-4. Depth prediction loss

[0022] Focal Loss is used to strengthen the penalty for wrong predictions by adjusting the γ parameter in the focal loss, thereby increasing the model's attention to difficult or error-prone samples.

[0023] in: is the adjustment parameter in the focal loss'' Step 5-5. Total loss L The total loss is the weighted sum of each loss, and the formula is as follows:

[0024] in, , , and It is the weight factor of each loss item, which is used to adjust the impact of different tasks on the total loss.

[0025] Furthermore, the high-definition map network model is trained, including the following steps: Step 6-1. Data preparation Dataset: Use a self-made driving scene dataset; Preprocessing: including point cloud processing of LiDAR data and normalization of camera images. LiDAR point cloud is projected onto the image plane to generate a sparse depth image. Step 6-2: Network initialization Camera branch: Use the pre-trained ResNet-101 as the backbone network of the camera branch. This model has been pre-trained on the MS-COCO dataset. LiDAR branch: Use the LiDAR extractor module to process LiDAR point cloud data, without pre-training, training from scratch; Step 6-3: Multi-level fusion training Data-level fusion: In the initial stage, point cloud projection features are combined with RGB image input network; Feature-level fusion: Applying cross-attention mechanism to guide long-range prediction of LiDAR BEV features through camera features; BEV-level fusion: Use the BEV alignment module to align and fuse BEV features from camera and LiDAR; Step 6-4. Application of loss function Simultaneously optimize multiple loss functions during training, including semantic segmentation loss, instance embedding loss, road direction prediction loss, and depth prediction loss; Adjust the weight factors of each loss function to balance the training focus of different tasks; Step 6-5: Optimizer and hyperparameter settings Use stochastic gradient descent SGD as the optimizer; Step 6-6: Evaluate and adjust Use the validation dataset to evaluate model performance during training, monitoring IoU and mean average precision metrics; Based on the validation results, the network parameters and training strategies are adjusted to improve the performance.

[0026] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0027] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0028] It can be seen from the above technical solutions that the present invention aims to overcome the above shortcomings in the prior art and proposes a long-distance high-definition map online prediction method based on multi-sensor fusion. This method generates a high-precision semantic map covering a longer distance (up to 90 meters) by fusing LiDAR and camera data at multiple levels. This technology can not only significantly improve the environmental perception capability of the autonomous driving system, but also provide more reliable data support for downstream path planning and motion control tasks, thereby improving the driving safety and user experience of the vehicle.

[0029] In summary, the present invention proposes an online prediction method for long-distance high-definition maps based on multi-sensor fusion, which has the following advantages: (1) The present invention constructs an online prediction method for high-definition maps based on LiDAR and camera data fusion, in which a dual-branch network is set to process LiDAR and camera data respectively, extract BEV features, and then input them into the BEV fusion module for feature fusion, which is used for subsequent multi-task head output of high-precision maps. This method supports online prediction and real-time update of maps, can adapt to complex and dynamic driving environments, and improves the safety and reliability of autonomous driving.

[0030] (2) The present invention processes data sources by integrating them from three aspects: data level, feature level, and BEV level, giving full play to the respective advantages of LiDAR and camera. Through the deep fusion of multiple sensors, the present invention can generate a high-definition map covering a range of 90 meters, significantly improving the long-distance perception capability of autonomous driving vehicles.

[0031] (3) The present invention adopts multiple fusion mechanisms to fuse various features in the network model, fully compressing multiple features while retaining the key information of various features to the maximum extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the process of constructing a network model according to an embodiment of the present invention; Figure 2 A schematic diagram of the process of constructing a BEV feature map using LiDAR point cloud data according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an encoder according to an embodiment of the present invention; Figure 4 A schematic diagram of converting radar point cloud data into a BEV perspective using a self-attention mechanism according to an embodiment of the present invention; Figure 5A schematic diagram of constructing a BEV feature map using a camera image according to an embodiment of the present invention; Figure 6 Schematic diagram of the fusion of LiDAR-BEV feature map and image BEV feature map in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0034] The present invention proposes an online prediction method for long-distance high-definition maps based on multi-sensor fusion, receiving surround view images and point cloud data, and the specific steps are as follows: Step 1: Install the sensor In order to achieve 360-degree environmental perception without blind spots, autonomous vehicles need to be equipped with multiple types of sensors, including surround-view cameras and lidar. The specific steps are as follows: Step 1-1: Select sensor location Surround view cameras are usually installed on the front and rear bumpers, near the side mirrors, and in the center of the roof of the vehicle. LiDAR is installed on the roof to ensure that the entire area around the vehicle can be covered.

[0035] Step 1-2: Install and debug the sensor Make sure all sensors are securely mounted on the vehicle to avoid vibration or displacement. Adjust the angle and position of each sensor to ensure maximum effective coverage.

[0036] Steps 1-3: Sensor Calibration and Synchronization Calibrate each sensor to ensure data accuracy. Implement spatiotemporal synchronization to ensure data from different sensors can be accurately aligned. Use timestamps and GPS signals to synchronize the data collection moments of each sensor.

[0037] Step 2: Collect image and point cloud data Through the deployed sensors, images and point cloud data around the vehicle are collected in real time for subsequent processing and mapping.

[0038] Step 2-1: Data Collection The camera collects video and static images of the surroundings. The lidar generates point cloud data of the surrounding environment.

[0039] Step 2-2: Data synchronization and processing Use a synchronous control system to ensure that the image and point cloud data are completely synchronized in time. Perform preliminary data processing, such as denoising the image and filtering the point cloud data.

[0040] Step 3: Create a map dataset The collected images and point cloud data are used to generate a mapping dataset containing environmental information.

[0041] Step 4: Build the network model This step aims to build a high-precision bird's-eye view (BEV) feature map by combining LiDAR point cloud data and camera image data to support the perception and decision-making of the autonomous driving system. The specific process is as follows Figure 1 shown.

[0042] Step 4-1: Construct BEV feature map using LiDAR point cloud data The process of this step is as follows Figure 2 As shown: First, the radar point cloud data is divided into voxel grids. Each voxel represents a small cubic area of ​​the point cloud data in 3D space. , each point Assigned to the corresponding voxel. Voxelization calculation formula: .

[0043] Secondly, clustering algorithms are used to group points according to the voxels where the points are located, ensuring that the point cloud data in space is effectively segmented to improve the efficiency of data processing. This method uses the DBSCAN algorithm to group points within each voxel.

[0044] Third, the voxels are uniformly downsampled to reduce the computational burden. This step not only helps reduce processing time, but also balances the problem of density inconsistency, thereby improving the reliability of feature extraction.

[0045] Then, each voxel block is input into the feature encoder for encoding, and the key features of the point cloud data are extracted through the encoder. The structure of the encoder is as follows Figure 3 shown.

[0046] The encoder first uses the self-attention mechanism module to identify the relationship and importance of each point in the point cloud, thereby extracting point-to-point features. Finally, the original voxels are integrated with the point-to-point features for subsequent analysis and application.

[0047] Finally, the radar point cloud data is converted to the BEV perspective using the self-attention mechanism, as shown in Figure 4 shown.

[0048] The BEV perspective starts by receiving image features Q and point cloud features K, V as input. First, these features are normalized to ensure data consistency. Then, random masks are applied to enhance the model’s robustness to partial information loss. Next, the features are converted to probability distributions through the Softmax function to calculate attention weights. Finally, the image and point cloud features are combined and the self-attention mechanism is used to generate point cloud BEV features for subsequent tasks.

[0049] Step 4-2: Project the LiDAR point cloud data onto a plane This step uses the joint calibration data of LiDAR and camera to project the radar point cloud data onto the camera image. Through this step, the 3D point cloud data is aligned with the 2D image data for multimodal fusion in the subsequent steps. At the same time, the radar point cloud data is used to generate a depth map for subsequent model supervision.

[0050] Step 4-3: Construct BEV feature map using camera images The processing flow of this step is as follows Figure 5 shown.

[0051] Image BEV feature extraction adopts a LSS-like (Lift, Splat, Shoot) method to construct the 2D image data captured by the camera into a bird's-eye view (BEV) feature map.

[0052] Specifically, by lifting the camera perspective, the 2D image data is converted into a 3D feature map; then, through ray projection (Splat), these 3D features are projected onto the ground plane to generate a BEV perspective feature map. In the image mapping (Shoot) process, in order to improve the receptive field, convolution and dilated convolution are used simultaneously to extract effective information from the feature map. At the same time, the depth map generated in step 4-2 is used to supervise the image lift operation.

[0053] Step 4-4: Fusion of LiDAR-BEV feature map and image BEV feature map The flowchart of this step is as follows Figure 6 shown.

[0054] First, the point cloud BEV features and image BEV features are merged (Concat); secondly, the merged features are input into the fully connected layer to obtain the weights of each channel; finally, the weights of each channel are stimulated and fused with the original merged features to obtain the fused feature map.

[0055] Step 4-5: Input the fused feature map into the multi-task head to output the detection result This step adopts a multi-task head design, which includes a semantic segmentation head (Semantic Segmentation Head), an instance embedding head (Instance Embedding Head) and a direction prediction head (Direction Prediction Head).

[0056] The semantic segmentation head identifies and classifies each pixel of the road and its surroundings, such as lane boundaries, sidewalks, traffic signs, etc. The instance embedding head generates instance embeddings for each individual object, distinguishing different instances of the same category (such as different lane lines). The direction prediction head predicts the direction of road elements, especially the direction of travel of the lane.

[0057] The final output HD map integrates the results of semantic segmentation, instance embedding and direction prediction, and contains rich geographical and structural information, which is crucial for the navigation, obstacle avoidance and path planning of the autonomous driving system.

[0058] Step 5: Define the loss function Step 5-1. Semantic segmentation loss ( ) Use cross entropy loss to supervise semantic segmentation tasks. This loss function evaluates the difference between the predicted semantic category and the true category, and is a common method for evaluating classification tasks. Its formula is:

[0059] in: N Indicates the total number of pixels.

[0060] Indicates the i The true class of the pixel (usually one-hot encoded).

[0061] Indicates i The predicted probability of each pixel.

[0062] Step 5-2. Instance Embedding Loss ( ) Contains variance loss ( ) and distance loss ( ).

[0063] Variance loss: Calculates the distance between the embedding vector within the same instance and its average embedding vector, ensuring that points in the same instance are close to each other.

[0064]

[0065] in: CIndicates the total number of instances.

[0066] Indicates the instance c points.

[0067] Indicates i The embedding vector of each point.

[0068] Representation instance c The average embedding vector of .

[0069] is a hyperparameter that defines the tolerance range of variance loss.

[0070] Distance loss: Ensure that the average embedding vectors of different instances maintain a sufficient distance between each other to prevent the embedding vectors of different instances from being confused.

[0071]

[0072] in: is a hyperparameter that defines the minimum distance between instances.

[0073] The instance embedding loss consists of a weighted sum of variance loss and distance loss.

[0074]

[0075] Among them, α and β are weighting factors of variance loss and distance loss.

[0076] Step 5-3. Road direction prediction loss ( ) The road direction is predicted using a cross entropy loss, which is discretized into 36 categories evenly distributed around the circle. This loss is backpropagated only for those pixels that are on the lane and have a valid direction.

[0077]

[0078] in: is the number of pixels with valid directions.

[0079] and are the true direction categories and predicted probabilities of these pixels, respectively.

[0080] Step 5-4. Depth prediction loss ( ) Focal Loss is used, which is a method to improve learning in the case of class imbalance. By adjusting the γ parameter in the focal loss (γ=2.0), the penalty for wrong predictions is strengthened and the model pays more attention to difficult or error-prone samples.

[0081]

[0082] in: is a tuning parameter in the focal loss, usually set to 2.0.

[0083] Step 5-5. Total loss (L) The total loss is the weighted sum of each loss, and the formula is as follows:

[0084] in, , , and is the weight factor of each loss term, which is used to adjust the impact of different tasks on the total loss. This combination allows the model to optimize multiple tasks simultaneously during training and improve overall performance.

[0085] Step 6: Set up the training process Step 6-1: Data preparation Dataset: Use a self-made driving scene dataset. The dataset contains LiDAR point cloud and camera image data.

[0086] Preprocessing: includes point cloud processing of LiDAR data and normalization of camera images. LiDAR point cloud is projected onto the image plane to generate a sparse depth image.

[0087] Step 6-2: Network initialization Camera branch: Use the pre-trained ResNet-101 as the backbone network of the camera branch. The model was pre-trained on the MS-COCO dataset.

[0088] LiDAR branch: Use the LiDAR extractor module to process LiDAR point cloud data, without pre-training, and train from scratch.

[0089] Step 6-3: Multi-level fusion training Data-level fusion: The initial stage combines point cloud projection features and RGB images into the network.

[0090] Feature Level Fusion: Applying cross-attention mechanism to guide long-range prediction of LiDAR BEV features through camera features.

[0091] BEV-level fusion: Use the BEV alignment module to align and fuse BEV features from camera and LiDAR.

[0092] Step 6-4: Application of loss function Multiple loss functions are optimized simultaneously during the training process, including semantic segmentation loss, instance embedding loss, road direction prediction loss, and depth prediction loss.

[0093] Adjust the weight factors of each loss function to balance the training focus of different tasks.

[0094] Step 6-5: Optimizer and hyperparameter settings Stochastic gradient descent (SGD) is used as the optimizer.

[0095] The initial learning rate is set to 0.1 and adjusted during the training process.

[0096] The total training cycle is 200 epochs.

[0097] Step 6-6: Evaluate and adjust The validation dataset is used during the training process to evaluate the model performance, monitoring the IoU and mean average precision metrics.

[0098] Based on the validation results, the network parameters and training strategies are adjusted to improve the performance.

[0099] Step 7: Application and Deployment After training is completed, the trained model is used to generate high-definition maps in actual autonomous driving scenarios to support path planning and navigation tasks.

[0100] In summary, the present invention proposes an online prediction method for long-distance high-definition maps based on multi-sensor fusion, which has the following characteristics: (1) The present invention is an online prediction method for long-distance high-definition maps based on multi-sensor fusion. It constructs an online prediction method for high-definition maps based on LiDAR and camera data fusion, in which a dual-branch network is set to process LiDAR and camera data respectively, extract BEV features, and then input them into the BEV fusion module for feature fusion, which is used for subsequent multi-task head output of high-precision maps. The method supports online prediction and real-time update of maps, can adapt to complex and dynamic driving environments, and improves the safety and reliability of autonomous driving.

[0101] (2) The present invention is an online prediction method for long-distance high-definition maps based on multi-sensor fusion. The data source is processed from three aspects: data level, feature level, and BEV level, giving full play to the respective advantages of LiDAR and camera. Through the deep fusion of multiple sensors, the present invention can generate a high-definition map covering a range of 90 meters, significantly improving the long-distance perception capability of autonomous driving vehicles.

[0102] (3) The present invention is an online prediction method for long-distance high-definition maps based on multi-sensor fusion. It adopts multiple fusion mechanisms to fuse various features in the network model, fully compressing multiple features while retaining the key information of various features to the maximum extent.

[0103] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0104] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0105] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the long-distance high-definition map online prediction methods based on multi-sensor fusion in the above-mentioned embodiments.

[0106] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above methods.

[0107] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0108] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0109] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0110] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A long-distance high-definition map online prediction method based on multi-sensor fusion, characterized in that: The following steps are included: Install and debug sensors; Through the deployed sensors, image data and point cloud data around the vehicle are collected in real time for subsequent processing and mapping; Generate a mapping dataset containing environmental information using the collected image data and point cloud data; Build a high-definition map network model, that is, by combining LiDAR point cloud data and camera image data, build a high-precision bird's-eye view feature map to support the perception and decision-making of the autonomous driving system; Define the loss function for training the HD map network model; The trained HD map network model is used to generate HD maps in actual autonomous driving scenarios to support path planning and navigation tasks.

2. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 1 is characterized by: Installing and commissioning the sensor includes the following steps, Surround view cameras are installed on the front and rear bumpers, near the side mirrors, and in the center of the roof. LiDAR is installed on the roof to ensure that the entire area around the vehicle can be covered. Make sure all sensors are firmly mounted on the vehicle to avoid vibration or displacement; adjust the angle and position of each sensor to ensure maximum effective coverage; Each sensor is calibrated to ensure data accuracy.

3. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 1 is characterized by: Real-time collection of image data and point cloud data around the vehicle includes the following steps: The camera collects surrounding video and static images, and the lidar generates point cloud data of the surrounding environment; A synchronous control system is used to ensure that the image and point cloud data are completely synchronized in time, and the data is preliminarily processed, including denoising the image and filtering the point cloud data.

4. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 1 is characterized by: Building a high-definition map network model includes the following steps: Step 4-1: Use LiDAR point cloud data to build BEV feature map First, the radar point cloud data is divided into voxel grids; each voxel grid represents a small cubic area of ​​the point cloud data in 3D space; according to the preset voxel size , each point Assign to the corresponding voxels; voxelization calculation formula: ； Secondly, clustering algorithms are used to group points according to the voxels where they are located, ensuring that the point cloud data in space is effectively segmented to improve the efficiency of data processing; Again, the voxels are uniformly downsampled to reduce the computational burden; Then, each voxel block is input into the feature encoder for encoding, and the key features of the point cloud data are extracted through the encoder; Finally, the radar point cloud data is converted to the BEV perspective using the self-attention mechanism; Step 4-2: Plane project the LiDAR point cloud data This step uses the joint calibration data of LiDAR and camera to project the radar point cloud data onto the camera image. Through this step, the 3D point cloud data is aligned with the 2D image data for multimodal fusion in subsequent steps. At the same time, the radar point cloud data is used to generate a depth map for subsequent model supervision. Step 4-3: Use camera images to construct a bird’s-eye view feature map Image BEV feature extraction uses a LSS-like method to construct a bird's-eye view feature map from the 2D image data captured by the camera; By improving the camera perspective, the 2D image data is converted into a 3D feature map; then, these 3D features are projected onto the ground plane through ray projection to generate a bird's-eye view feature map; in the image mapping process, in order to improve the receptive field, convolution and dilated convolution are used to extract effective information from the feature map; at the same time, the depth map generated in step 4-2 is used to supervise the image enhancement operation; Step 4-4: Fusion of LiDAR-BEV feature map and image BEV feature map First, the point cloud BEV features and image BEV features are merged; secondly, the merged features are input into the fully connected layer to obtain the weights of each channel; finally, the weights of each channel are fused with the original merged features to obtain the fused feature map; Step 4-5: Input the fused feature map into the multi-task head to output the detection result This step adopts a multi-task head design, which includes a semantic segmentation head, an instance embedding head, and a direction prediction head. The semantic segmentation head identifies and classifies each pixel of the road and its surroundings. The instance embedding head generates instance embeddings for each independent object and distinguishes different instances of the same category. The direction prediction head predicts the direction of road elements. The final output high-definition map integrates the results of semantic segmentation, instance embedding, and direction prediction, and contains rich geographical and structural information.

5. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 4 is characterized by: Defining the loss function includes, Step 5-1, semantic segmentation loss Use cross entropy loss to supervise the semantic segmentation task; The formula is: in: N Indicates the total number of pixels; Indicates the i The true class of the pixel; Indicates i The predicted probability of pixels; Step 5-2: Instance Embedding Loss ; Include variance loss and distance loss ; Variance loss: calculates the distance between the embedding vector within the same instance and its average embedding vector, ensuring that points in the same instance are close to each other; in: C Indicates the total number of instances; Indicates the instance c The number of points; Indicates i The embedding vector of each point; Representation instance c The average embedding vector of Is a hyperparameter that defines the tolerance range of variance loss; Distance loss: ensures that the average embedding vectors of different instances maintain a sufficient distance to prevent the embedding vectors of different instances from being confused; in: is a hyperparameter that defines the minimum distance between instances; The instance embedding loss consists of a weighted sum of variance loss and distance loss; Among them, α and β are weighting factors of variance loss and distance loss; Step 5-3: Road direction prediction loss The road direction is predicted using a cross entropy loss, which is discretized into 36 categories evenly distributed around the circle; this loss is backpropagated only for those pixels that are on the lane and have a valid direction; in: is the number of pixels with valid orientation; and are the true direction categories and predicted probabilities of these pixels respectively; Step 5-4. Depth prediction loss Focal Loss is used to strengthen the penalty for wrong predictions by adjusting the γ parameter in the focal loss, thereby increasing the model's attention to difficult or error-prone samples. in: is the adjustment parameter in the focal loss'' Step 5-5. Total loss L The total loss is the weighted sum of each loss, and the formula is as follows: in, , , and It is the weight factor of each loss item, which is used to adjust the impact of different tasks on the total loss.

6. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 1 is characterized by: Training the HD map network model includes the following steps: Step 6-1. Data preparation Dataset: Use a self-made driving scene dataset; Preprocessing: including point cloud processing of LiDAR data and normalization of camera images. LiDAR point cloud is projected onto the image plane to generate a sparse depth image. Step 6-2: Network initialization Camera branch: Use the pre-trained ResNet-101 as the backbone network of the camera branch. This model has been pre-trained on the MS-COCO dataset. LiDAR branch: Use the LiDAR extractor module to process LiDAR point cloud data, without pre-training, training from scratch; Step 6-3: Multi-level fusion training Data-level fusion: In the initial stage, point cloud projection features are combined with RGB image input network; Feature-level fusion: Applying cross-attention mechanism to guide long-range prediction of LiDAR BEV features through camera features; BEV-level fusion: Use the BEV alignment module to align and fuse BEV features from camera and LiDAR; Step 6-4. Application of loss function Simultaneously optimize multiple loss functions during training, including semantic segmentation loss, instance embedding loss, road direction prediction loss, and depth prediction loss; Adjust the weight factors of each loss function to balance the training focus of different tasks; Step 6-5: Optimizer and hyperparameter settings Use stochastic gradient descent SGD as the optimizer; Step 6-6: Evaluate and adjust Use the validation dataset to evaluate model performance during training, monitoring IoU and mean average precision metrics; Based on the validation results, the network parameters and training strategies are adjusted to improve the performance.

7. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 4 is characterized by: In step S4-1, the DBSCAN algorithm is used to group the points within each voxel.

8. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 5 is characterized by: In step S5-4, γ=2.

0.

9. The long-distance high-definition map online prediction method based on multi-sensor fusion according to claim 5 is characterized by: In step 6-5, the initial learning rate is set to 0.1 and adjusted during the training process; the total training cycle is 200 epochs.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

Images