Occupied grid map construction method, training method, device and electronic equipment

By employing multimodal data fusion and sparse voxel reconstruction, the problem of high computational power requirements in the construction of occupied raster maps is solved, achieving more efficient computation and more detailed raster map construction, which is suitable for obstacle recognition and path planning of autonomous vehicles.

CN119048704APending Publication Date: 2024-11-29CHONGQING CHANGAN AUTOMOBILE CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411222680.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In existing technologies, the construction of grid maps requires high-performance computing equipment, resulting in large computational and storage demands. Furthermore, sparse algorithms are mainly designed for single-modal image data, and their application to multimodal data fusion presents significant challenges in terms of computational power and workload.

Method used

We adopt an occupied grid map construction method based on multimodal data fusion. By fusing sparse voxels multiple times, we use an encoder and decoder to extract features and reconstruct grids. By combining image voxel and point cloud voxel features, we can achieve coarse-to-fine feature learning, reduce computing power requirements and improve computational efficiency.

Benefits of technology

It achieves reduced computational load and resource consumption, improved computational efficiency, and construction of more detailed and accurate occupancy grid maps in the context of multimodal data fusion, enabling the identification of object motion states and detailed features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048704A_ABST
    Figure CN119048704A_ABST
Patent Text Reader

Abstract

The invention relates to an occupied grid map construction method and device, a training method and device and electronic equipment. The occupied grid map construction method comprises the steps of obtaining multi-modal data of a target area; inputting the multi-modal data into an occupied raster map construction model, and extracting multi-modal data features based on the multi-modal feature data in an encoder of the occupied raster map construction model; in a decoder of the grid map construction model, obtaining fusion features based on the first object query and the multi-modal data features, predicting the occupancy probability of the grid based on the fusion features, and performing finer-grained grid reconstruction on the target grid based on the occupancy probability to obtain a second object query; and obtaining an occupied raster map of the target area based on fusion features obtained by the second object query and the multi-modal data features. According to the method, the occupied grid map is constructed based on the multi-modal data, the calculation amount can be reduced through the construction mode, and the requirements for the calculation power and the storage amount are low.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an occupancy grid map construction method, training method, device and electronic equipment. Background Art

[0002] With the advancement of sensor technology, computing power and artificial intelligence algorithms, occupancy grid maps have become one of the indispensable technologies in autonomous vehicles. Occupancy grid refers to dividing the continuous environment space into discrete grid cells and marking the state of each grid (occupied or free) according to feature information. Occupancy grid maps represent the environment in an intuitive grid form, which is easy to understand and analyze. Compared with other types of maps, occupancy grid maps have higher computational efficiency in path planning and obstacle avoidance algorithms.

[0003] In the related technologies, sparse algorithms have become a research hotspot in recent years for the construction of occupancy grid maps. However, current sparse algorithms are mainly designed and applied for single-modal image data. If these algorithms are extended to multimodal data fusion, there are still computing power challenges. Therefore, it is urgent to provide an occupancy grid map construction solution based on multimodal data fusion. Summary of the invention

[0004] One of the purposes of the present invention is to provide a method for constructing an occupancy grid map to solve the problem that the occupancy grid map construction in the prior art needs to rely on high-computing power equipment and the map construction requires a large amount of calculation and storage; the second purpose is to provide a training method for an occupancy grid map construction model; the third purpose is to provide a device; and the fourth purpose is to provide an electronic device.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for constructing an occupancy grid map, the method comprising:

[0007] Acquire multimodal data of the target area;

[0008] Inputting the multimodal data into an occupancy grid map construction model, and extracting multimodal data features based on the multimodal feature data in an encoder of the occupancy grid map construction model;

[0009] In the decoder of the grid map construction model, a fused feature is obtained based on the first object query and the multimodal data feature, and the occupancy probability of the grid is predicted based on the fused feature, and a finer-grained grid reconstruction of the target grid is performed based on the occupancy probability to obtain a second object query; and an occupancy grid map of the target area is obtained based on the fused feature obtained from the second object query and the multimodal data feature.

[0010] According to the above technical means, multimodal data is collected to construct an occupancy grid map, and the multimodal data is identified based on the occupancy grid map construction model, thereby obtaining the target occupancy grid map. Based on the need to process multimodal data, the sparse voxel multiple fusion method is used in the decoder of the grid map construction model to reduce the demand for computing power. The sparse voxel multiple fusion method is as follows: based on the first object query and the multimodal data feature, a fusion feature is obtained, and the occupancy probability of the grid is predicted based on the fusion feature. Based on the occupancy probability, a more fine-grained grid reconstruction is performed on the target grid to obtain a second object query, and based on the fusion feature obtained from the second object query and the multimodal data feature, an occupancy grid map of the target area is obtained, wherein a grid with high feature validity is selected based on the occupancy probability for more fine-grained grid reconstruction, and feature learning from coarse to fine can not only delete the grid of the air part, but also further refine the grid to obtain more detailed features, realize the construction of the grid map, and at the same time reduce the computing power demand, improve computing efficiency and reduce resource consumption.

[0011] Further, reconstructing the target grid in a finer granularity based on the occupancy probability to obtain a second object query includes:

[0012] Selecting a set number of target grids from large to small based on the occupancy probability to perform grid reconstruction with finer granularity;

[0013] The second object query is determined based on the reconstructed grid.

[0014] According to the above technical means, the grid with the highest occupancy probability is selected as the target grid, and the grid with low feature efficiency can be removed. Then, the target grid is reconstructed with a finer granularity to determine the second object query with the reconstructed grid. The image voxel features and point cloud voxel features are further learned based on the second object query, thereby realizing the learning of detailed parts, and reducing the amount of grid processing, thereby reducing the amount of calculation of the model.

[0015] Further, the decoder includes N decoding layers, the fusion feature is obtained based on the first object query and the multimodal data feature, the occupancy probability of the grid is predicted based on the fusion feature, and the target grid is reconstructed in a finer granularity based on the occupancy probability to obtain the second object query, including:

[0016] In the first decoding layer, a fusion feature is obtained based on the first object query and the multimodal data feature; and the occupancy probability is predicted based on the fusion feature to obtain the occupancy probability corresponding to each initialized grid; a first set number of first target grids are selected from large to small based on the occupancy probability, and the first target grids are upsampled to obtain a finer-grained reconstructed grid, and a second object query corresponding to the next decoding layer is determined based on the reconstructed grid;

[0017] In the nth decoding layer, a fusion feature is obtained based on the second object query and the multimodal data feature; and the occupancy probability is predicted based on the fusion feature to obtain the occupancy probabilities corresponding to the reconstructed grids corresponding to the n-1th decoding layer; based on the occupancy probability, a second set number of second target grids are selected from large to small, and the second target grids are upsampled to obtain a finer-grained reconstructed grid, and the second object query is updated based on the reconstructed grid, and the updated second object query is used as the object query corresponding to the n+1th decoding layer; wherein, when n is equal to N, an occupancy grid map of the target area is obtained, and N≥2.

[0018] According to the above technical means, through multiple decoding layers, grid reconstruction from coarse to fine to finer is achieved, which improves computing efficiency, reduces resource consumption, and ensures the learning of feature details.

[0019] Furthermore, the multimodal data features include image voxel features and point cloud voxel features, and a fusion feature is obtained based on the object query and the multimodal data features, including:

[0020] Obtaining the image voxel features and the point cloud voxel features of the corresponding positions according to an object query, wherein the object query includes a first object query or a second object query;

[0021] The image voxel features and the point cloud voxel features are fused based on attention learning to obtain fused features.

[0022] According to the above technical means, multimodal data feature fusion can realize feature learning in more dimensions, making the constructed occupancy grid map more detailed and accurate.

[0023] Furthermore, before obtaining the image voxel features and the point cloud voxel features of the corresponding positions according to the object query, it also includes:

[0024] Acquire at least one frame of image voxel features that are continuous with the image voxel features at the current moment according to the time series features, and fuse the at least one frame of image voxel features that are continuous with the image voxel features at the current moment to obtain the fused image voxel features;

[0025] And / or, based on the time series features, at least one frame of point cloud voxel features that are continuous with the point cloud voxel features at the current moment are obtained, and the at least one continuous frame of point cloud voxel features are fused into the point cloud voxel features at the current moment to obtain the fused point cloud voxel features, wherein the object query is performed based on the fused image voxel features and the fused point cloud voxel features.

[0026] According to the above technical means, by fusing the image voxel features of different time sequences and the point cloud voxel features of different time sequences, the features of the occluded objects can be supplemented, and the motion state and trajectory of the object can be identified, which is convenient for identifying the type of object and the changing trend on the occupancy grid map, the impact on vehicle driving, etc., and enriching the constructed occupancy grid map.

[0027] Furthermore, the multimodal feature data includes image data and radar point cloud data, and the multimodal data features extracted based on the multimodal feature data include:

[0028] The image data and the radar point cloud data are input into the encoder of the grid map construction model, and the following processing is performed in the encoder:

[0029] Obtaining depth image features according to the image data, and obtaining radar point cloud features according to the radar point cloud data;

[0030] A voxel conversion is performed based on the depth image features to obtain image voxel features, and a voxel conversion is performed based on the radar point cloud features to obtain point cloud voxel features.

[0031] Further, obtaining the depth image features according to the image data includes:

[0032] Projecting the radar point cloud data into the image data to obtain depth image data;

[0033] Depth image features are extracted based on the depth image data and the image data.

[0034] According to the above technical means, the geometric features in the image data are corrected through radar point cloud data to enrich the content of the image data, so that the model can recognize more effective features and enrich the content of the occupied grid map.

[0035] Further, obtaining radar point cloud features according to the radar point cloud data includes:

[0036] Extracting semantic feature points from the image data according to the position coordinates of the radar point cloud data;

[0037] Fusion of the semantic feature points with the radar point cloud data;

[0038] Based on the fused radar point cloud data, radar point cloud features are extracted.

[0039] According to the above technical means, the semantic features of radar point cloud data are repaired through image data, which facilitates the recognition of radar point cloud data and improves computing efficiency.

[0040] Further, the voxel conversion is performed based on the radar point cloud features to obtain point cloud voxel features, including:

[0041] Performing voxel conversion on the radar point cloud features to obtain initial point cloud voxel features;

[0042] Determine the point cloud sparsity corresponding to each position according to the voxelized features of the initial point cloud;

[0043] The voxels at the point cloud positions where the sparsity is greater than or equal to the first threshold are merged, and the voxels at the point cloud positions where the sparsity is greater than or equal to the second threshold are merged, and the merged voxel features are the point cloud voxel features.

[0044] According to the above technical means, merging relatively sparse voxels (voxels with low feature efficiency) together can reduce the voxel size and thus speed up the model reasoning speed. And the voxel merging method adopts nonlinear distribution, which can further speed up the model reasoning speed.

[0045] A training method for an occupancy grid map construction model, comprising:

[0046] Acquire a training sample, wherein the training sample includes multimodal data of a plurality of target areas and a real occupancy grid map corresponding to each of the multimodal data;

[0047] The multimodal data is input into an occupancy grid map construction model, and the following processing is performed in the occupancy grid map construction model to obtain an occupancy grid map of the target area: extracting multimodal data features based on the multimodal feature data; obtaining fused features based on a first object query and the multimodal data features, and predicting occupancy probabilities of grids based on the fused features, and reconstructing a target grid with finer granularity based on the occupancy probabilities to obtain a second object query; obtaining an occupancy grid map of the target area based on the fused features obtained from the second object query and the multimodal data features;

[0048] Determining a loss value of the occupancy grid map and a real occupancy grid map corresponding to the target area based on a total loss function;

[0049] Model parameters of the occupancy grid map construction model are adjusted according to the loss value.

[0050] Furthermore, the total loss function is:

[0051] L=L focal +L mask +L dice +L occ ;

[0052] Among them, Lfocal is the focus loss function, Ldice is the DICE loss function, Lmask is the mask loss function, and Locc is the sparse voxel fusion module loss function.

[0053] An occupancy grid map construction device, the device comprising:

[0054] A first acquisition module, used to acquire multimodal data of a target area;

[0055] The occupancy grid map construction module includes an encoder and a decoder, wherein the encoder is used to extract features from the input multimodal data to obtain multimodal data features; the decoder is used to obtain fused features according to a first object query and the multimodal data features, and predict the occupancy probability of a grid based on the fused features, and perform a more fine-grained grid reconstruction on a target grid based on the occupancy probability to obtain a second object query; and the occupancy grid map of the target area is obtained based on the fused features obtained from the second object query and the multimodal data features.

[0056] A training device for an occupancy grid map construction model, the device comprising:

[0057] A second acquisition module is used to acquire training samples, wherein the training samples include multimodal data of multiple target areas and a real occupancy grid map corresponding to each multimodal data;

[0058] An occupancy grid map construction model is used to perform the following processing on the input multimodal data to obtain an occupancy grid map of the target area: extracting multimodal data features based on the multimodal feature data; obtaining fused features based on a first object query and the multimodal data features, and predicting occupancy probabilities of grids based on the fused features, and reconstructing a target grid with finer granularity based on the occupancy probabilities to obtain a second object query; obtaining an occupancy grid map of the target area based on the fused features obtained from the second object query and the multimodal data features;

[0059] A training module is used to determine the loss value of the occupancy grid map and the real occupancy grid map corresponding to the target area based on the total loss function; and adjust the model parameters of the occupancy grid map construction model according to the loss value.

[0060] An electronic device, comprising: a memory and a processor;

[0061] The memory is used to store computer programs / instructions; the processor is used to implement the above-mentioned method according to the computer programs / instructions stored in the memory.

[0062] A computer-readable storage medium stores a computer program / instruction, wherein the computer program / instruction is used to implement the method described above when executed by a processor.

[0063] A computer program product, comprising a computer program / instruction, wherein the computer program / instruction implements the method as described above when executed by a processor.

[0064] Beneficial effects of the present invention:

[0065] (1) An occupancy grid map construction scheme based on multimodal data fusion realizes the fusion of multimodal data. Based on the sparse voxel fusion method, feature learning from coarse to fine can not only delete the grid of the air part, but also further refine the grid to obtain more detailed features and realize the construction of the grid map. At the same time, it reduces the computing power requirements, speeds up the model inference speed, improves the computing efficiency and reduces resource consumption.

[0066] (2) In the process of voxelization of multimodal data features, the voxel size is reduced by merging voxels of distant sparse points, which is beneficial to reducing the amount of calculation and increasing the ranging range;

[0067] (3) Multimodal data includes image data and radar point cloud data. The image data is geometrically corrected through radar point cloud data to enrich the depth information of the image data. The radar point cloud data is corrected through image data to enrich the semantic information of the radar point cloud data. This allows the decoder to obtain richer features, identify features more quickly, and improve computing efficiency.

[0068] (4) Use sparse sampling to obtain features in different modes and time series, complement the feature differences between different modes, and enhance the model segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of the structure of a vehicle to which the method mentioned in an embodiment of the present invention is applied;

[0070] Figure 2 A schematic diagram of the structure of an occupancy grid map construction model provided by an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of a geometry correction module provided by an embodiment of the present invention;

[0072] Figure 4A schematic diagram of a semantic correction module provided by an embodiment of the present invention;

[0073] Figure 5 A schematic diagram of a sparse voxel fusion module provided in an embodiment of the present invention;

[0074] Figure 6 A flowchart of a training method for an occupancy grid map construction model is provided for an embodiment of the present invention;

[0075] Figure 7 A schematic diagram of a flow chart of a method for constructing an occupancy grid map provided in one embodiment of the present invention;

[0076] Figure 8 A schematic diagram of a flow chart of a method for constructing an occupancy grid map provided by another embodiment of the present invention;

[0077] Fig. 9 A schematic diagram of a process for geometric correction of image data provided by an embodiment of the present invention;

[0078] Fig.10 A schematic diagram of a process for semantic correction of radar point cloud data provided by an embodiment of the present invention;

[0079] Fig.11 A schematic diagram of nonlinear voxel transformation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0081] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0082] With the advancement of sensor technology, computing power and artificial intelligence algorithms, occupancy grid maps have become one of the indispensable technologies in autonomous vehicles. Occupancy grid refers to dividing the continuous environment space into discrete grid cells and marking the state of each grid (occupied or free) according to feature information. Occupancy grid maps represent the environment in an intuitive grid form, which is easy to understand and analyze. Compared with other types of maps, occupancy grid maps have higher computational efficiency in path planning and obstacle avoidance algorithms.

[0083] Although occupancy grid maps have performed well in many aspects, they still have some limitations. High-resolution grid maps may lead to a sharp increase in computing and storage requirements. Therefore, in related technologies, sparse algorithms have become a research hotspot in recent years for the construction of occupancy grid maps. However, current sparse algorithms are mainly designed and applied for single-modal image data. If these algorithms are extended to multimodal data fusion, based on multimodal data including data from different sensors, such as vision, radar, and lidar, these data need to be integrated, involving data synchronization, feature fusion, and information consistency. There are challenges in computing power and large amount of calculation.

[0084] Based on this, an embodiment of the present application provides an occupancy grid map construction solution based on multimodal data fusion, introduces a completely sparse occupancy network in the Head part, and reconstructs the geometric structure of the scene through a sparse voxel fusion module, from sparse to dense, reducing the amount of calculation while ensuring the learning of detail voxels, thereby significantly saving computing resources while achieving multimodal time series fusion.

[0085] The specific implementation process and method are described in detail through the following embodiments.

[0086] Figure 1 This is a schematic diagram of the vehicle structure mentioned in the embodiment of the present application. As shown in the figure, the method for generating an occupation grid map is applied to a vehicle, and a plurality of image sensors CAN (such as a vehicle-mounted camera) and a radar sensor RADAR are arranged on the vehicle. As an example, the image sensor CAN is distributed in the left viewing angles of the vehicle, covering the four sides of the vehicle, namely, the front viewing angle, the front left viewing angle, the front right viewing angle, the front rear viewing angle, the rear left viewing angle and the rear right viewing angle. As an example, the radar sensor RADAR is arranged at the front viewing angle, the front left viewing angle, the front right viewing angle, the rear left viewing angle and the rear right viewing angle, etc. In addition to the above-mentioned configuration, the image sensor CAN and the radar sensor RADAR can also be other configurations, which are not limited here.

[0087] In the embodiment of the present application, multiple frames of image data and radar point cloud data can be obtained through the image sensor CAN and the radar sensor RADAR, and the image data and radar point cloud data are input into the occupancy grid map construction model to construct the occupancy grid map.

[0088] The image data captured by the image sensor CAN can fully cover the surrounding environment of the vehicle; the point cloud data collected by the laser radar system records the three-dimensional structure of the space around the vehicle in detail. In terms of the specific requirements for data collection, the collected data types are rich and the scenes are varied, so that the model can learn a variety of different traffic environments and situations.

[0089] Figure 2 A schematic diagram of the structure of the occupancy grid map construction model provided in the embodiment of the present application, such as Figure 2 As shown, the model includes: an encoder and a decoder, wherein the encoder is used to extract features from the input multimodal data to obtain multimodal data features. Optionally, the multimodal data includes but is not limited to image data and radar point cloud data, and the corresponding extracted multimodal data features include but are not limited to image voxel features and point cloud voxel features.

[0090] In the example where multimodal data includes image data and radar point cloud data, both image modality and radar modality have limitations, which have a certain impact on the construction of occupancy grid maps. For example, although image modality can capture rich semantic information such as color and texture, it lacks accurate geometric and spatial structure information. While radar modality can collect information on geometric features such as distance and speed, it is insufficient in expressing information at the semantic level.

[0091] Based on the above limitations, the decoder provided in the embodiment of the present application includes a geometric correction module and a semantic correction module, which are used to correct the limitations of the image modality and the radar modality.

[0092] like Figure 3 As shown, the geometric correction module is used to correct the image data, and the specific correction process can be:

[0093] First, each point in the radar point cloud data is accurately projected into the image data captured by multiple cameras to generate a series of sparse multi-view depth maps, which provide additional depth information for the scene. Then, the multi-view depth map is used to extract depth features through the feature extraction module of the encoder to obtain depth perception of the scene. The extracted depth features are stacked with the image features captured by the camera to achieve the fusion of the depth information in the radar point cloud data with the image data to obtain the depth image features. The depth image features include the rich color and texture information of the image and the precise spatial structure information provided by the lidar.

[0094] like Figure 4 As shown, the semantic correction module is used to correct the radar data, and the specific correction process can be:

[0095] First, based on the position of the point cloud in the radar point cloud data, the image data captured by multiple cameras is sampled to obtain semantic feature points, and then these semantic feature points are fused with the original radar point cloud data through stacking operations to obtain fused radar point cloud data. Finally, the radar point cloud features are extracted through the feature extraction module, and the radar point cloud features include spatial structure information and semantic information provided by the image data.

[0096] The decoder is used to learn the image voxel features and point cloud voxel features and fuse them to obtain the occupancy grid map.

[0097] Please refer to Figure 2 and Figure 5 As an example, the decoder includes a sparse voxel fusion module, which is used to reconstruct the occupancy grid map from coarse to fine and to combine image voxel features and radar voxel features at different time sequences to obtain the motion trajectory and state of objects in the scene, thereby improving the accuracy of the occupancy grid map.

[0098] Based on statistical analysis, it is found that in the vehicle driving scene, more than 90% of the voxel units in the scene collected by the camera or radar are actually air. When constructing the occupancy grid map, if the air part is decoded and fused, it will undoubtedly greatly increase the amount of invalid calculations, affecting the calculation efficiency and resource consumption. Therefore, the sparse voxel fusion module in the embodiment of the present application first queries the coarse-grained object, fuses the multimodal data, obtains the fusion feature, and then predicts the occupancy probability of the grid based on the fusion feature, and reconstructs the target grid with a finer granularity based on the occupancy probability to obtain the second object query; and then obtains the occupancy grid map of the target area based on the fusion feature obtained from the second object query and the multimodal data feature. Among them, after obtaining the second object query, it can continue to fuse based on the second object query and the multimodal data feature to obtain the fusion feature, and then predict the occupancy probability of the grid through the fusion feature, and reconstruct the target grid with a finer granularity based on the occupancy probability to update the second object query, and obtain the final occupancy grid map of the target area by updating the second object query once or multiple times (gradually reconstructing with finer granularity).

[0099] Optionally, when fusing multimodal data, the sparse voxel fusion module also fuses the image voxel features at different time sequences and the point cloud voxel features at different time sequences to obtain the features of the occluded object, or obtain the features corresponding to the object's motion trajectory and motion state. Based on the fusion of these features, an occupancy grid map with higher accuracy, clarity and more details can be obtained.

[0100] Based on the above constructed occupancy grid map construction model, training is performed by loading multimodal data. The training process of the occupancy grid map construction model is described below:

[0101] like Figure 6 As shown, the embodiment of the present application provides a training method for an occupancy grid map construction model, comprising:

[0102] S601, obtaining a training sample, where the training sample includes multimodal data of multiple target areas and a real occupancy grid map corresponding to each multimodal data;

[0103] This step takes the application in a vehicle as an example. As an example, the multimodal data includes image data and radar point cloud data. The image data is captured through a vehicle-mounted camera network (a network composed of multiple cameras), and the image data can fully cover the surrounding environment of the vehicle; the radar point cloud data is collected through a vehicle-mounted lidar system, and the radar point cloud data records in detail the three-dimensional structure of the space around the vehicle.

[0104] The training samples include multimodal data of multiple target areas. The target areas are determined according to different scenarios, so that the collected multimodal data types are rich and adaptable to changing scenarios. These data are used as inputs of the model so that the model can learn various traffic environments and scenarios. As an example, the real occupancy grid map can be obtained through simulation or other methods.

[0105] In this step, the collected data set is divided into two parts: 80% of the scenes are used as training samples in the training phase of the model to provide sufficient information for model learning. The remaining 20% ​​of the scenes are used as test samples for rigorous verification of the model to test its performance and generalization ability on unseen data. The data of each individual scene contains a video sequence of s1 seconds to s2 seconds. These video frames are annotated with 3D detection boxes at intervals of x seconds (that is, the corresponding labels, occupying the grid map), ensuring the continuity and accuracy of the annotations, where s1 can be selected as 20, s2 can be selected as 30, and x can be selected as 0.1s.

[0106] Furthermore, in order to achieve effective multimodal data fusion, all image data and corresponding radar point cloud data have undergone strict synchronization processing to ensure the time consistency between the two. Among them, when feature extraction and encoding are performed on image data and radar point cloud data, the image data and radar point cloud data can be time-synchronized to ensure time consistency.

[0107] The image data is acquired by multiple cameras, and the multiple frames of images in this embodiment are stacked to obtain the image data. Similarly, the radar point cloud data is also acquired by multiple radar sensors, and the multiple frames of radar point cloud are stacked to obtain the radar point cloud data.

[0108] S602, inputting the multimodal data into an occupancy grid map construction model, performing the following processing in the occupancy grid map construction model to obtain an occupancy grid map of the target area: extracting multimodal data features based on the multimodal feature data; obtaining fused features based on the first object query and the multimodal data features, and predicting the occupancy probability of the grid based on the fused features, and reconstructing the target grid with a finer granularity based on the occupancy probability to obtain a second object query; obtaining an occupancy grid map of the target area based on the fused features obtained from the second object query and the multimodal data features;

[0109] S603, determining loss values ​​of the occupancy grid map and the real occupancy grid map corresponding to the target area based on the total loss function;

[0110] S604: Adjust model parameters of the occupancy grid map construction model according to the loss value.

[0111] In this step, after the multimodal data is input into the occupancy grid map construction model, the multimodal data is processed based on the structure and model parameters of the occupancy grid map construction model to obtain the occupancy grid map of the target area, and then the model parameters of the occupancy grid map construction model are adjusted by calculating the loss of the occupancy grid map construction model and the true occupancy grid map, so that the loss of the occupancy grid map construction model is less than or equal to the preset threshold, or the loss function converges, so as to complete the training of the occupancy grid map construction model.

[0112] As an example, the occupancy grid map construction model includes an encoder and a decoder, where:

[0113] The encoder performs feature extraction on the multimodal data. The feature extraction process may refer to the specific description of the following embodiment.

[0114] The decoder includes a sparse voxel fusion module, which fuses multimodal data based on a sparse voxel algorithm, thereby reducing the amount of calculation and enabling timely occupancy grid map updates. Optionally, the processing of multimodal data by the sparse voxel fusion module can refer to the specific description of the following embodiment.

[0115] As an example, the total loss function of this embodiment is:

[0116] L=L focat +L mask +L dice +L occ ;

[0117] Among them, Lfocal is the focus loss function, Ldice is the DICE loss function, Lmask is the mask loss function, and Locc is the sparse voxel fusion module loss function.

[0118] In this embodiment, the occupancy head is used to classify whether the voxel predicted each time is occupied, and the classification head is used to predict the category of the occupied voxel. Optionally, the focal loss function Lfocal is used for classification, that is, this embodiment trains the model's loss of the occupancy probability of the grid based on the focal loss function Lfocal. Among them, the combination of DICE loss Ldice and BCE mask loss Lmask is used for mask prediction. Locc is the sparse voxel fusion module loss function, and Locc is used to train the model's loss for sparse fusion of multimodal data characteristics.

[0119] In this embodiment, the model parameters of the occupancy grid map construction model are adjusted according to the loss value until the loss value reaches a preset value or the loss function converges, and the model parameters (network weight parameters) of the occupancy grid map construction model are obtained.

[0120] Based on the above process, the training of the occupancy grid map construction model is realized, and the training amount of the occupancy grid map construction model can be reduced. The obtained occupancy grid map construction model realizes the fusion of multimodal data, and has high computing power requirements and low computing amount, thereby improving computing efficiency, thereby enabling the occupancy grid map to be updated in a timely manner.

[0121] After the occupancy grid map construction model is trained based on the above method, the process of implementing occupancy grid map construction based on the occupancy grid map construction model is described below:

[0122] Figure 7 A schematic diagram of a process of constructing an occupancy grid map provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the method includes:

[0123] S701, acquiring multimodal data of a target area;

[0124] In this step, taking multimodal data including image data and radar point cloud data as an example, the image data is obtained through a camera, and the radar point cloud data is obtained through a radar sensor.

[0125] Taking the application in vehicles as an example, cameras and radar sensors are both on-board devices, and the target area is the area currently scanned by the on-board devices during the driving process of the vehicle. During the driving process of the vehicle, an occupancy grid map corresponding to the target area is constructed based on the method proposed in the embodiment of the present application, so that the vehicle can perform obstacle recognition, lane line recognition, etc. based on the occupancy grid map, and then perform obstacle avoidance control, or perform automatic driving control based on lane lines.

[0126] Optionally, the image data includes multiple frames of images, which are acquired by cameras at different angles, such as Figure 1 As shown in FIG. 1 , the images are collected by cameras arranged at different angles on the vehicle. Multiple frames of images are stacked to form image data.

[0127] Optionally, the radar point cloud data includes multiple frames of radar point cloud images, which are also acquired by different radar sensors, such as Figure 1 As shown in the figure, it is collected by radar sensors arranged at different positions of the vehicle. Multiple frames of radar point cloud images are stacked to form radar point cloud data.

[0128] S702, inputting the multimodal data into an occupancy grid map construction model, and extracting multimodal data features based on the multimodal feature data in an encoder of the occupancy grid map construction model;

[0129] Optionally, taking multimodal data including image data and radar point cloud data as an example, the encoder performs feature extraction on the image data and the radar point cloud data respectively to obtain image voxel features and point cloud voxel features respectively.

[0130] Based on the need to construct an occupancy grid map in the embodiment of the present application, it is necessary to perform voxel conversion on the two-dimensional image features and the three-dimensional point cloud features during the feature extraction process.

[0131] As an example, the image data and radar point cloud data are input into the encoder of the raster map construction model, in which the following processing is performed:

[0132] According to the image data, the depth image features are obtained, and according to the radar point cloud data, the radar point cloud features are obtained;

[0133] Voxel conversion is performed based on the depth image features to obtain image voxel features, and voxel conversion is performed based on the radar point cloud features to obtain point cloud voxel features.

[0134] In this example, the encoder extracts the image data and the radar point cloud data respectively to obtain the depth image features and the radar point cloud features respectively, and then inputs the depth image features and the radar point cloud features into the decoder respectively.

[0135] S703, in the decoder of the grid map construction model, a fused feature is obtained based on the first object query and the multimodal data feature, and the occupancy probability of the grid is predicted based on the fused feature, and a finer-grained grid reconstruction of the target grid is performed based on the occupancy probability to obtain a second object query, and an occupancy grid map of the target area is obtained based on the fused feature obtained from the second object query and the multimodal data feature.

[0136] In this step, the first object query is a sparse object query, querying coarse-grained features, and the second object query is densified layer by layer to learn finer-grained features. The first object query is determined based on a preset initialization.

[0137] In this step, after the image voxel features and point cloud voxel features are obtained based on S702, the image voxel features and point cloud voxel features are used as Key and Value and input into the decoder together with the first object query. The decoder samples from the corresponding positions of the image voxel features and point cloud voxel features based on the first object query, and then fuses the sampled image voxel features and point cloud voxel features based on attention learning to obtain fused features. Since the first object query is a sparse object query, the number of grids obtained through learning through the first object query is small, the size is large, and the granularity corresponding to each grid is coarse (coarse voxel granularity).

[0138] Based on this, the decoder of the embodiment of the present application includes multiple decoding layers (Transformer layers). Through the learning of multiple decoding layers from coarse granularity to fine granularity, the grid of the air part can be deleted in the first decoding layer, and based on the learning in the nth decoding layer, more fine-grained features in the grid can be obtained. This is specifically achieved in the following ways:

[0139] At the end of each decoding layer, the occupancy probability of the grid is predicted based on the fusion features, and the occupancy probability of each grid is predicted. Then, the target grid is reconstructed at a finer granularity based on the occupancy probability, and the second object query is determined based on the reconstructed grid, where the grid of the air part is excluded by the target grid. The second object query is determined based on the finer granularity reconstruction of the target grid, which excludes the query of the air part, and learns finer multimodal data features based on finer coordinates to enrich the details of the occupancy grid map.

[0140] As an example, the target grid may be an occupied grid, for example, a grid with an occupation probability greater than 0 is the target grid.

[0141] As an example, the target grid may be a grid whose occupancy probability is greater than or equal to a preset probability, for example, a grid whose occupancy probability is greater than 30% to 50% is the target grid.

[0142] As an example, the target grids may be the top K grids (Top-K) in terms of occupancy probability, where K is a set number. For example, based on the occupancy probability sorting from large to small, a set number of grids are selected as the target grids.

[0143] After obtaining the second object query, the second object query is input into the second decoding layer, and then based on the second object query, samples are respectively taken from the corresponding positions of the image voxel features and the point cloud voxel features, and then the sampled image voxel features and point cloud voxel features are fused based on attention learning to obtain fused features. In one example, an occupancy grid map of the target area is determined based on the fused features. In another example, the decoder also includes an nth decoding layer, and the nth object query is obtained by the above-mentioned step S703, and then input into the nth decoding layer, and the image voxel features and point cloud voxel features are learned until the last decryption layer to obtain the occupancy grid map of the target area.

[0144] It should be noted that the decoder includes N decoding layers, where N ≥ 2. The following takes the decoder including three decoding layers as an example to describe in detail the process of constructing an occupancy grid map by the sparse voxel fusion decoder in the embodiment of the present application:

[0145] After extracting the multimodal data features, the multimodal data features are input into the first decoding layer;

[0146] In the first decoding layer, a fusion feature is obtained based on the first object query and the multimodal data feature; and the occupancy probability is predicted based on the fusion feature to obtain the occupancy probability corresponding to each initialized grid; a first set number of first target grids are selected based on the occupancy probability, and the first target grids are upsampled (for example, grids that are enlarged by 2 times in length, width, and depth respectively - a total of 8 times), to obtain a finer-grained reconstructed grid, and a second object query corresponding to the second decoding layer is determined based on the reconstructed grid (for example, the first set number is 500, and after 8 times magnification, 4000 is obtained, and the second object query includes 4000 objects);

[0147] In the second decoding layer, a fusion feature is obtained based on the second object query and the multimodal data feature; and the occupancy probability is predicted based on the fusion feature to obtain the occupancy probabilities corresponding to the reconstructed grids corresponding to the first decoding layer; a second set number of second target grids are selected from large to small based on the occupancy probability, and the second target grids are upsampled (same as above) to obtain a finer-grained reconstructed grid, and the second object query is updated based on the reconstructed grid, and the third object query is used as the object query corresponding to the third decoding layer (for example, the second set number is 2000, and after 8 times magnification, 16000 is obtained, and the updated second object query includes 16,000 objects).

[0148] In the third decoding layer, fused features are obtained based on the second object query and the multimodal data features, and an occupancy grid map of the target area is determined based on the fused features.

[0149] In other examples, if the decoder includes N decoding layers, then in the nth decoding layer, a fusion feature is obtained based on the second object query and the multimodal data feature; and the occupancy probability is predicted based on the fusion feature to obtain the occupancy probabilities corresponding to the reconstructed grids corresponding to the n-1th decoding layer; based on the occupancy probability, an nth set number of second target grids are selected from large to small, and the second target grids are upsampled to obtain a finer-grained reconstructed grid, and the second object query is updated based on the reconstructed grid, and the updated second object query is used as the object query corresponding to the n+1th decoding layer; wherein, when n is equal to N, an occupancy grid map of the target area is obtained.

[0150] As an example, the first object query (Query) is defined by a preset segmentation space, which can be selected as (200×200×16). In the initial stage of fine-grained processing, i.e., the first object query, its dimensions can be selected as (25×25×2,3), where the number 3 refers to the coordinate position in the 3D space, and 25 refers to the size of the segmentation space downsampled by 8 times.

[0151] In this step, the fusion features obtained based on the second object query and the multimodal data features include:

[0152] Obtaining image voxel features and point cloud voxel features of corresponding positions according to the second object query;

[0153] Based on attention learning, the image voxel features and point cloud voxel features are fused to obtain the fused features.

[0154] It should be noted that in the first decoding layer, since the first object query is obtained based on a preset initialization method, the first object query has no association with the three-dimensional space. Based on this, in the embodiment of the present application, the first object query, the multimodal data features and the 3D position coordinates of the image are input into the first decoding layer. Among them, in the 3D position encoding, a sine function is used to generate the code to establish a connection between the two-dimensional image and the three-dimensional space. The process of 3D position encoding is as follows:

[0155] First, a discrete camera viewing space is set, whose size is (W F ,H F ,D), where W F and H F Represent the width and height of the feature map respectively, and D represents the depth dimension.

[0156] In this viewing space, each point p j It can be expressed as p j =(u j ·d j , v j ·d j , d j, 1) T , where (u j , v j ) represents the two-dimensional coordinates of the image features, d j Represents the depth value perpendicular to the image plane, and T represents the transposition operation. According to different cameras i and their external parameters Ki, the three-dimensional coordinates of all points in the viewing space can be calculated, specifically using the following formula:

[0157]

[0158] The pseudo 3D features of the image are obtained by fusing the 3D coordinates with the image features, thereby achieving association between the first object query and the 3D coordinates.

[0159] In this embodiment, multimodal data is collected to construct an occupancy grid map, and the multimodal data is identified based on the occupancy grid map construction model, thereby obtaining a target occupancy grid map. Based on the need to process multimodal data, a sparse voxel multiple fusion method is used in the decoder of the grid map construction model to reduce the computing power requirement. The sparse voxel multiple fusion method is as follows: based on the first object query and the multimodal data feature, a fusion feature is obtained, and the occupancy probability of the grid is predicted based on the fusion feature. Based on the occupancy probability, a more fine-grained grid reconstruction is performed on the target grid to obtain a second object query, and based on the fusion feature obtained from the second object query and the multimodal data feature, an occupancy grid map of the target area is obtained, wherein a grid with high feature validity is selected based on the occupancy probability for more fine-grained grid reconstruction, and feature learning from coarse to fine can not only delete the grid of the air part, but also further refine the grid to obtain more detailed features, realize the construction of the grid map, and at the same time reduce the computing power requirement, improve computing efficiency and reduce resource consumption.

[0160] Figure 8 Another flowchart of the method for constructing an occupancy grid map provided in an embodiment of the present application is as follows: Figure 8 As shown, the method includes:

[0161] S801, acquiring multimodal data of a target area;

[0162] S802, inputting the multimodal data into an occupancy grid map construction model, and extracting multimodal data features based on the multimodal feature data in an encoder of the occupancy grid map construction model;

[0163] Among them, steps S801 to S802 and Figure 7 The implementation of steps S701 to S702 in the embodiment is similar, and will not be repeated here in this embodiment.

[0164] S803, in the decoder of the grid map construction model, obtaining at least one frame of image voxel features that are continuous with the image voxel features at the current moment according to the time series features, and fusing the continuous at least one frame of image voxel features into the image voxel features at the current moment to obtain the fused image voxel features;

[0165] In this step, the temporal feature represents the acquisition timing of the image. When encoding the image voxel features in the encoder, temporal feature encoding is also included.

[0166] In this step, the current moment refers to the moment when the multimodal data is currently input and processed. During the application process, multimodal data is collected in real time, and the grid map construction model is used to process the multimodal data in real time. Before fusing the image voxel features and point cloud voxel features at the current moment, the image voxel features with different time series characteristics are combined to predict the motion trajectory or motion state of the occluded objects and objects. Therefore, before the image voxel features and point cloud voxel features are fused, at least one frame of image voxel features that is continuous with the image voxel features at the current moment is obtained based on the time series features, and the image voxel features of multiple frames are fused.

[0167] Optionally, the process of fusing voxel features of multiple frames of images may be:

[0168] First, the radar coordinate system L(t) is used as the 3D segmentation space to transform the image voxel features into 3D space. Represents the transformation from the source coordinate system to the target coordinate system. Therefore, the transformation formula for converting the camera internal and external parameters of each perspective to the 3D space is:

[0169]

[0170] in, Represents the camera intrinsic parameter of the i-th perspective, Represents the transformation matrix from the camera of the i-th view to the 3D space L(t).

[0171] The voxel features of at least one continuous frame of image are fused to the voxel features of the image at the current moment, mainly based on the alignment of the coordinate origin. For example, the voxel features of the image corresponding to the current moment t and the voxel features of the image corresponding to the moment t-1 are fused. The coordinate origins of the two frames are not consistent. The 3D coordinates of the t-1 frame are aligned to the coordinate system of the tth frame. The specific transformation formula can be:

[0172]

[0173] in, It is the transformation matrix of the coordinate system between the voxel features of two consecutive frames of images.

[0174] Optionally, the transformation matrix of the transformation system between the voxel features of two consecutive frames of images can be obtained by the following formula:

[0175]

[0176] Among them, e(t) is the vehicle coordinate system at time t, e(t-1) is the vehicle coordinate system at time (t-1), and g is the global world coordinate system.

[0177] And / or, S804, in a decoder of the grid map construction model, obtaining at least one frame of point cloud voxel features that are continuous with the point cloud voxel features at the current moment based on the time series features, and fusing the continuous at least one frame of point cloud voxel features into the point cloud voxel features at the current moment to obtain the fused point cloud voxel features;

[0178] This step is similar to step S803, and also combines the point cloud voxel features of different time series features. The specific process is similar to step S803.

[0179] S805, in the decoder of the grid map construction model, based on the first object query, the fused image voxel features and the fused point cloud voxel features, a fused feature is obtained, and the occupancy probability of the grid is predicted based on the fused feature, and a more fine-grained grid reconstruction of the target grid is performed based on the occupancy probability to obtain a second object query; and based on the fused feature obtained from the second object query and the multimodal data features, an occupancy grid map of the target area is obtained.

[0180] Among them, step S805 and Figure 7 The implementation of step S703 in the embodiment is similar and will not be described in detail in this embodiment.

[0181] In this embodiment, before the image voxel features and the point cloud voxel features are fused, the image voxel features of different time series are fused, and the point cloud voxel features of different time series are fused, so that the characteristics of the occluded object, as well as the motion state and motion trajectory of the object can be obtained, and the type of object can be more accurately inferred. In this way, a more accurate occupancy grid map can be constructed, providing a more accurate reference for obstacle recognition and navigation.

[0182] Fig. 9 This is a schematic diagram of the process of geometric correction of image data proposed in the embodiment of the present application, such as Fig. 9 As shown, according to the image data, the deep image features are obtained, including:

[0183] S901, projecting the radar point cloud data into the image data to obtain depth image data;

[0184] First, each point in the radar point cloud data is accurately projected into the image data captured by multiple cameras to generate a series of sparse multi-view depth maps, which provide additional depth information for the scene. The multi-view depth map is then used by the encoder’s feature extraction module to extract depth features, thereby obtaining depth perception of the scene and forming depth image data. Among them, depth features include but are not limited to surface features such as color and texture, and information related to image depth or the three-dimensional structure of the scene.

[0185] S902: Extracting depth image features according to the depth image data and the image data.

[0186] The extracted depth features are stacked with the image features captured by the camera to fuse the depth information in the radar point cloud data into the image data to obtain the depth image features. The depth image features include the rich color and texture information of the image and the precise spatial structure information provided by the lidar.

[0187] In this embodiment, the image data is geometrically corrected based on the radar point cloud data so that the image data contains depth features. Compared with the initial image data collected by the camera, the depth image data has richer information, providing richer features for the subsequent occupancy grid map construction, and thus making it easier to identify obstacles or road information based on the occupancy grid map.

[0188] Fig.10 This is a flow chart of the semantic correction of image data proposed in the embodiment of the present application, as shown in FIG. Fig.10 As shown, according to the radar point cloud data, radar point cloud features are obtained, including:

[0189] S1001, extracting semantic feature points from the image data according to the position coordinates of the radar point cloud data;

[0190] Based on the position of the point cloud in the radar point cloud data, sampling is performed from the image data captured by multiple cameras to obtain semantic feature points.

[0191] S1002, fusing the semantic feature points with the radar point cloud data to obtain radar point cloud features;

[0192] Through stacking operation, these semantic feature points are fused with the original radar point cloud data to obtain fused radar point cloud data. Finally, the radar point cloud features are extracted by the feature extraction module, and the radar point cloud features include spatial structure information and semantic information provided by image data.

[0193] In this embodiment, semantic correction is performed on radar point cloud data based on image data to add semantic information to the radar point cloud data. When constructing an occupancy grid map, it is easier to identify the type of object reflected by the point cloud, thereby improving the efficiency of constructing the occupancy grid map.

[0194] Please refer to Fig.11 On the basis of the above embodiment, in step S702, when performing voxel conversion on the radar point cloud features, the radar point cloud features may be subjected to nonlinear voxel conversion, wherein the nonlinear voxel conversion refers to conversion into grids (also voxels) of inconsistent sizes. The specific implementation method is:

[0195] First, the radar point cloud features are voxelized to obtain the initial point cloud voxelized features. The voxel conversion here is performed according to the conventional voxel conversion method to obtain voxels of the same size.

[0196] After obtaining the initial point cloud voxel features, the point cloud sparsity corresponding to each position is determined according to the initial point cloud voxel features; the point cloud positions where the sparsity is greater than or equal to the first threshold are voxel merged, and the point cloud positions where the sparsity is greater than or equal to the second threshold are voxel merged, and the merged voxel features are the point cloud voxel features.

[0197] Because it takes a long time to occupy the grid network, the size of the voxelization directly affects the reasoning speed of the model. Due to the limitations of LiDAR scanning, the farther the point cloud is, the sparser it is, and the less effective the features provided by the sparse point cloud are. Therefore, this embodiment proposes the concept of nonlinear voxelization based on the sparse properties of distant features, aggregates the sparse voxels of the elements, and reduces the grid.

[0198] like Fig.11 As shown in the figure, the left picture is the radar point cloud feature. After the general voxelization module, the intermediate voxel feature (that is, the initial point cloud voxel feature) is obtained, where L represents the depth dimension, W represents the width dimension, and H represents the height dimension. Then, the distant voxels are aggregated, as shown in the figure on the right, which changes from the original size L×W×H to L / 2×W×H. The distant features are integrated and the voxel size is greatly reduced, thereby speeding up the model reasoning speed.

[0199] As an example, the sparsity of distant point cloud features is nonlinear, so the aggregation method also adopts a nonlinear distribution (Nonlinear Distribution) aggregation method (for example, sparsity less than a first threshold is aggregated together, and sparsity less than a second threshold is aggregated together, and the first threshold and the second threshold are different).

[0200] Taking China as an example, the aggregation method adopts the maximum pooling scheme, and the specific grid size is calculated by the following formula:

[0201]

[0202] Where b is the depth growth factor, and dx represents the depth of the xth grid (voxel). For example, when the growth factor is 1.1, the depth of the first grid (voxel) is 1.11-1=0.1 meters, and the depth of the second grid (voxel) is 0.1+(1.12-1)=0.31 meters.

[0203] This embodiment can speed up the reasoning speed of the model, reduce the amount of calculation, and make the construction of the multimodal occupancy grid map meet the timeliness requirements.

[0204] The present application also provides an occupancy grid map construction device, the device comprising:

[0205] A first acquisition module, used to acquire multimodal data of a target area;

[0206] The occupancy grid map construction module includes an encoder and a decoder. The encoder is used to extract features from the input multimodal data to obtain multimodal data features; the decoder is used to obtain fused features according to the first object query and the multimodal data features, and predict the occupancy probability of the grid based on the fused features, and reconstruct the target grid with finer granularity based on the occupancy probability to obtain a second object query; based on the fused features obtained from the second object query and the multimodal data features, an occupancy grid map of the target area is obtained.

[0207] The occupancy grid map construction device provided in the embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0208] The present application also provides a training device for an occupancy grid map construction model, the device comprising:

[0209] A second acquisition module is used to acquire training samples, where the training samples include multimodal data of multiple target areas and a real occupancy grid map corresponding to each multimodal data;

[0210] The occupancy grid map construction model is used to perform the following processing on the input multimodal data to obtain an occupancy grid map of the target area: extracting multimodal data features based on the multimodal feature data; obtaining fused features based on the first object query and the multimodal data features, and predicting the occupancy probability of the grid based on the fused features, and reconstructing the target grid with a finer granularity based on the occupancy probability to obtain a second object query; obtaining an occupancy grid map of the target area based on the fused features obtained from the second object query and the multimodal data features;

[0211] A training module is used to determine the loss values ​​of the occupancy grid map and the real occupancy grid map corresponding to the target area based on the total loss function; and adjust the model parameters of the occupancy grid map construction model according to the loss value.

[0212] The training device for the occupancy grid map construction model provided in the embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0213] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor;

[0214] The memory is used to store computer programs / instructions; the processor is used to execute the computer programs / instructions stored in the memory to implement the methods involved in the above embodiments.

[0215] The electronic device also includes a communication interface and a CAN bus, wherein the processor is used to provide computing power and control capabilities, and can be a GPU, CPU, NPU, MCU, FPGA, etc. The storage device includes an internal memory and a non-volatile memory. The non-volatile memory stores a computer program that implements the above method. The internal memory provides an environment for program startup and operation. The communication interface is used to communicate with an external terminal by wire or wireless.

[0216] In this embodiment, the electronic device may be an electronic component in a car, or may be the car.

[0217] The present invention also provides a computer-readable storage medium / computer program product, in which computer-readable storage medium stores computer execution instructions / the computer program product includes computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the methods involved in the above-mentioned embodiments.

[0218] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A method for constructing an occupancy grid map, characterized in that: The method comprises: Acquire multimodal data of the target area; Inputting the multimodal data into an occupancy grid map construction model, and extracting multimodal data features based on the multimodal feature data in an encoder of the occupancy grid map construction model; In the decoder of the grid map construction model, a fused feature is obtained based on the first object query and the multimodal data feature, and the occupancy probability of the grid is predicted based on the fused feature, and a finer-grained grid reconstruction of the target grid is performed based on the occupancy probability to obtain a second object query; and an occupancy grid map of the target area is obtained based on the fused feature obtained from the second object query and the multimodal data feature.

2. The method according to claim 1, characterized in that The reconstructing the target grid in a finer granularity based on the occupancy probability to obtain a second object query comprises: Selecting a set number of target grids from large to small based on the occupancy probability to perform grid reconstruction with finer granularity; The second object query is determined based on the reconstructed grid.

3. The method according to claim 1, characterized in that The decoder comprises N decoding layers, and the fusion feature is obtained based on the first object query and the multimodal data feature, and the occupancy probability of the grid is predicted based on the fusion feature, and the target grid is reconstructed in a finer granularity based on the occupancy probability to obtain the second object query, including: In the first decoding layer, a fusion feature is obtained based on the first object query and the multimodal data feature; and the occupancy probability is predicted based on the fusion feature to obtain the occupancy probability corresponding to each initialized grid; a first set number of first target grids are selected from large to small based on the occupancy probability, and the first target grids are upsampled to obtain a finer-grained reconstructed grid, and a second object query corresponding to the next decoding layer is determined based on the reconstructed grid; In the nth decoding layer, a fusion feature is obtained based on the second object query and the multimodal data feature; and the occupancy probability is predicted based on the fusion feature to obtain the occupancy probabilities corresponding to the reconstructed grids corresponding to the n-1th decoding layer; based on the occupancy probability, a second set number of second target grids are selected from large to small, and the second target grids are upsampled to obtain a finer-grained reconstructed grid, and the second object query is updated based on the reconstructed grid, and the updated second object query is used as the object query corresponding to the n+1th decoding layer; wherein, when n is equal to N, an occupancy grid map of the target area is obtained, and N≥2.

4. The method according to claim 1, characterized in that: The multimodal data features include image voxel features and point cloud voxel features, and a fusion feature is obtained based on the object query and the multimodal data features, including: Obtaining the image voxel features and the point cloud voxel features of the corresponding positions according to an object query, wherein the object query includes a first object query or a second object query; The image voxel features and the point cloud voxel features are fused based on attention learning to obtain fused features.

5. The method according to claim 4, characterized in that Before obtaining the image voxel features and the point cloud voxel features of the corresponding positions according to the object query, the method further includes: Acquire at least one frame of image voxel features that are continuous with the image voxel features at the current moment according to the time series features, and fuse the at least one frame of image voxel features that are continuous with the image voxel features at the current moment to obtain the fused image voxel features; And / or, based on the time series features, at least one frame of point cloud voxel features that are continuous with the point cloud voxel features at the current moment are obtained, and the at least one continuous frame of point cloud voxel features are fused into the point cloud voxel features at the current moment to obtain the fused point cloud voxel features, wherein the object query is performed based on the fused image voxel features and the fused point cloud voxel features.

6. The method according to any one of claims 1 to 5, characterized in that: The multimodal feature data includes image data and radar point cloud data, and the multimodal data features extracted based on the multimodal feature data include: The image data and the radar point cloud data are input into the encoder of the grid map construction model, and the following processing is performed in the encoder: Obtaining depth image features according to the image data, and obtaining radar point cloud features according to the radar point cloud data; A voxel conversion is performed based on the depth image features to obtain image voxel features, and a voxel conversion is performed based on the radar point cloud features to obtain point cloud voxel features.

7. The method according to claim 6, characterized in that The step of obtaining a depth image feature according to the image data comprises: Projecting the radar point cloud data into the image data to obtain depth image data; Depth image features are extracted based on the depth image data and the image data.

8. The method according to claim 6, characterized in that The step of obtaining radar point cloud features according to the radar point cloud data includes: Extracting semantic feature points from the image data according to the position coordinates of the radar point cloud data; Fusion of the semantic feature points with the radar point cloud data; Based on the fused radar point cloud data, radar point cloud features are extracted.

9. The method according to claim 6, characterized in that The voxel conversion based on the radar point cloud features to obtain point cloud voxel features includes: Performing voxel conversion on the radar point cloud features to obtain initial point cloud voxel features; Determining the point cloud sparsity corresponding to each position according to the voxelized features of the initial point cloud; The voxels at the point cloud positions where the sparsity is greater than or equal to the first threshold are merged, and the voxels at the point cloud positions where the sparsity is greater than or equal to the second threshold are merged, and the merged voxel features are the point cloud voxel features.

10. A training method for building an occupancy grid map model, characterized in that: include: Acquire a training sample, wherein the training sample includes multimodal data of a plurality of target areas and a real occupancy grid map corresponding to each of the multimodal data; The multimodal data is input into an occupancy grid map construction model, and the following processing is performed in the occupancy grid map construction model to obtain an occupancy grid map of the target area: extracting multimodal data features based on the multimodal feature data; Obtaining fused features based on the first object query and the multimodal data features, predicting occupancy probabilities of grids based on the fused features, and reconstructing target grids with finer granularity based on the occupancy probabilities to obtain a second object query; Obtaining an occupancy grid map of the target area based on a fused feature obtained from a second object query and the multimodal data features; Determining a loss value of the occupancy grid map and a real occupancy grid map corresponding to the target area based on a total loss function; Model parameters of the occupancy grid map construction model are adjusted according to the loss value.

11. The method according to claim 10, characterized in that The total loss function is: L=L focal +L mask +L dice +L occ ; Among them, Lfocal is the focus loss function, Ldice is the DICE loss function, Lmask is the mask loss function, and Locc is the sparse voxel fusion module loss function.

12. An occupancy grid map construction device, characterized in that: The device comprises: A first acquisition module, used to acquire multimodal data of a target area; The occupancy grid map construction module includes an encoder and a decoder, wherein the encoder is used to extract features from the input multimodal data to obtain multimodal data features; the decoder is used to obtain fused features according to a first object query and the multimodal data features, and predict the occupancy probability of a grid based on the fused features, and perform a more fine-grained grid reconstruction on a target grid based on the occupancy probability to obtain a second object query; and the occupancy grid map of the target area is obtained based on the fused features obtained from the second object query and the multimodal data features.

13. A training device for an occupancy grid map construction model, characterized in that: The device comprises: A second acquisition module is used to acquire training samples, wherein the training samples include multimodal data of multiple target areas and a real occupancy grid map corresponding to each multimodal data; An occupancy grid map construction model is used to perform the following processing on the input multimodal data to obtain an occupancy grid map of the target area: extracting multimodal data features based on the multimodal feature data; obtaining fused features based on a first object query and the multimodal data features, and predicting occupancy probabilities of grids based on the fused features, and reconstructing a target grid with finer granularity based on the occupancy probabilities to obtain a second object query; obtaining an occupancy grid map of the target area based on the fused features obtained from the second object query and the multimodal data features; A training module is used to determine the loss value of the occupancy grid map and the real occupancy grid map corresponding to the target area based on the total loss function; and adjust the model parameters of the occupancy grid map construction model according to the loss value.

14. An electronic device, characterized in that: The electronic device comprises: a memory and a processor; The memory is used to store computer programs / instructions; the processor is used to implement the method according to any one of claims 1 to 11 according to the computer programs / instructions stored in the memory.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program / instruction, and the computer program / instruction is used to implement the method according to any one of claims 1 to 11 when executed by a processor.

16. A computer program product, characterized in that The computer program product comprises a computer program / instructions, and when the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Citation Information

Cited By

  • Generation method, system and product for occupying grid scene

    CN119958588A

  • Equipment control method and device, electronic equipment and readable storage medium

    CN121011010A

  • Equipment control methods, devices, electronic equipment and readable storage media

    CN121011010B

  • Logistics vehicle obstacle sensing method and system based on sparse semantic occupancy network

    CN122392027A

  • Logistics vehicle obstacle perception method and system based on sparse semantic occupancy network

    CN122392027B