3D occupancy grid detection method, apparatus, device, and medium

Abstract

The present disclosure provides a 3D occupancy grid detection method, device, equipment and medium. In the 3D occupancy grid detection method of the present disclosure, after obtaining the current frame memory entry through the current frame point cloud data, the relevant memory entry is obtained through the memory library retrieval, the 3D grid probability with semantic information is obtained through the fusion features of the current frame memory entry and the relevant memory entry, and the current frame memory entry is stored in the memory library only when the scene feature of the current frame point cloud data belongs to the key frame. The present disclosure realizes the 3D occupancy grid detection based on point cloud through the memory storage mode, can effectively reduce the memory occupation and the calculation resource consumption without reducing the perception accuracy, improves the processing efficiency, and meets the real-time demand of automatic driving.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a 3D occupancy grid detection method, apparatus, device, and medium. Background Technology

[0002] In autonomous driving technology, accurately perceiving the vehicle's surroundings and detecting drivable areas is crucial. 3D occupancy grid detection can effectively represent the distribution of obstacles in the environment, providing a basis for determining the drivable area for the vehicle.

[0003] However, current 3D occupancy grid-based detection methods typically require storing and processing large amounts of raw voxel features, resulting in high memory consumption and computational overhead. According to tests on the Waymo dataset, storing just 10 frames of voxel features at a resolution of 256×256×32 requires approximately 3.8GB of memory. This large data volume not only leads to high memory and computational costs but also significantly increases processing complexity, making it difficult for 3D occupancy grid detection to meet the stringent real-time requirements of autonomous driving systems. Summary of the Invention

[0004] In view of this, the present disclosure provides a 3D occupancy grid detection method, apparatus, device and medium.

[0005] According to a first aspect of this disclosure, a 3D occupancy grid detection method is provided, the 3D occupancy grid detection method comprising:

[0006] The current frame point cloud data of the surrounding environment of the vehicle is obtained from the lidar mounted on the vehicle. Extract scene features from the current frame point cloud data, and use the scene features to generate a current frame memory entry; Retrieve relevant memory entries for the current frame memory entry from the memory bank, which stores historical memory entries, each of which is obtained from point cloud data of a historical frame; The fusion features are obtained by using the current frame memory entries and related memory entries, and the fusion features are used to obtain a 3D occupancy probability grid with semantic information; Whether a scene feature belongs to a keyframe is determined by calculating the scene entropy value or feature change rate of the scene feature. If the scene feature belongs to a keyframe, the memory bank is updated using the current frame memory entry. If the scene feature does not belong to a keyframe, the memory bank is not updated using the current frame memory entry.

[0007] In some embodiments of the first aspect of this disclosure, determining whether a scene feature belongs to a keyframe by calculating the scene entropy value or feature change rate of the scene feature includes: When the scene entropy value of the scene feature is greater than a preset entropy threshold or the feature change rate of the scene feature is greater than a preset change rate threshold, the scene feature is determined to belong to a keyframe. When the scene entropy value of the scene feature is less than or equal to the preset entropy threshold or the feature change rate of the scene feature is less than or equal to the preset change rate threshold, the scene feature is determined not to belong to a keyframe.

[0008] In some embodiments of the first aspect of this disclosure, the scene entropy value of the scene feature is calculated as follows: the feature values ​​in the scene feature are normalized to map to the range [0, 1], the normalized feature values ​​are quantized into N intervals, the interval index to which each feature value belongs is calculated, the number of feature values ​​in each interval is counted and the ratio of the number of feature values ​​in each interval to the total number of feature values ​​is calculated to obtain the probability of each interval, and the scene entropy value is calculated based on the probability of each interval, where N is the preset total number of quantized intervals.

[0009] In some embodiments of the first aspect of this disclosure, the feature change rate of the scene feature is calculated as follows: the feature difference is calculated using the current scene feature and the previously obtained scene feature; the L2 norm of the current scene feature and the L2 norm of the feature difference are calculated; the L2 norm of the feature difference is divided by the L2 norm of the current scene feature to obtain the relative change rate, which is the feature change rate of the scene feature.

[0010] In some embodiments of the first aspect of this disclosure, updating the memory bank using the current frame memory entry includes: Write the current frame memory entry into the memory bank and increment the number of valid memories in the memory bank by 1; Determine whether the effective memory count of the memory bank has reached a preset upper limit. If the number of valid memories in the memory bank reaches the upper limit, then the historical memory entry with the fewest uses or the lowest importance score in the memory bank is eliminated and the number of valid memories in the memory bank is updated.

[0011] In some embodiments of the first aspect of this disclosure, the historical frame memory entries in the memory bank are stored in timestamp order.

[0012] In some embodiments of the first aspect of this disclosure, the current frame memory entry includes a current frame memory key and a current frame memory value, the current frame memory value includes current frame memory features; the current frame memory key is obtained by performing global average pooling, dimensionality reduction projection and L2 normalization on the scene features, and the current frame memory features are obtained by performing CNN distillation compression on the scene features.

[0013] According to a second aspect of this disclosure, a 3D occupancy grid detection device is provided, the 3D occupancy grid detection device comprising: The data acquisition unit is used to acquire current frame point cloud data of the surrounding environment of the vehicle from the lidar mounted on the vehicle. A scene feature unit is used to extract scene features from the current frame point cloud data; A memory encoding unit is used to generate a current frame memory entry using the scene features; The memory retrieval unit is used to retrieve relevant memory entries of the current frame memory entry from the memory bank, which stores historical memory entries, each of which is obtained based on point cloud data of a historical frame. The feature fusion unit is used to obtain fused features by utilizing the current frame memory entries and related memory entries; A multi-task unit is used to obtain a 3D occupancy probability grid with semantic information by utilizing the fusion features; The keyframe determination unit is used to determine whether a scene feature belongs to a keyframe by calculating the scene entropy value or the feature change rate of the scene feature; The memory bank update unit is used to update the memory bank with the current frame memory entry when the scene feature belongs to a keyframe, and not to update the memory bank with the current frame memory entry when the scene feature does not belong to a keyframe.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium storing a program, the program including instructions that, when executed by a processor, cause the processor to perform the methods described above.

[0016] As can be seen from the above technical solutions, the embodiments of this disclosure can effectively reduce the amount of data in the 3D grid detection process, thereby effectively reducing memory and computing costs, reducing processing complexity, improving processing efficiency, and meeting the real-time requirements of autonomous driving without sacrificing perception accuracy. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the system architecture to which the embodiments of this disclosure apply; Figure 2 A schematic flowchart of the 3D occupancy grid detection method provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram of the structure of the 3D occupancy grid detection device provided in the embodiments of this disclosure; Figure 4 A schematic block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” as used in the embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0022] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0023] Figure 1 A schematic diagram of the architecture of a system to which embodiments of this disclosure apply is shown. See also Figure 1 The system to which this disclosure applies may include: an electronic device and a peripheral sensor assembly connected to the electronic device, the peripheral sensor assembly including radar, and the electronic device may be used to perform the 3D occupancy grid detection method described in the embodiments of this disclosure.

[0024] Radar can be used to collect point cloud data of the environment surrounding a vehicle. The radar can be, but is not limited to, lidar, millimeter-wave radar, or other types, and can be implemented as a single radar or a radar array. This disclosure does not limit the specific type, deployment method, or implementation form of the radar.

[0025] The electronic device can be implemented as, but is not limited to, a domain controller or other similar device, and the radar and electronic device can be connected communicatively or wiredly.

[0026] This disclosure can be applied to, but is not limited to, intelligent control of various devices such as multiple wheeled mobile robots, wheeled mobile robots, mobile robots, vehicles, aircraft, ships, Autonomous Rail Rapid Transit (ART) systems, and industrial automation equipment. Vehicles can be, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles), agricultural and industrial vehicles (e.g., harvesters, forklifts), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks), new energy vehicles (e.g., electric vehicles, hybrid vehicles), and special-purpose vehicles (e.g., garbage trucks, water trucks). In other words, the term "vehicle" in this disclosure is equivalent to the aforementioned various devices.

[0027] The embodiments disclosed herein can be applied to various scenarios such as urban transportation, highways, ports, mines, farms, closed parks, and industrial production. They are applicable to many aspects such as passenger travel, public transportation, logistics distribution, unmanned transportation, last-mile delivery, automated agricultural operations, and automated sanitation. This disclosure does not limit the application scenarios and applicable fields of the embodiments disclosed herein.

[0028] The systems applicable to the embodiments of this disclosure may include, but are not limited to, autonomous driving systems, intelligent driver assistance systems, etc. See also Figure 1 The system provided in this disclosure can be installed in a vehicle and used as, but is not limited to, an intelligent driver assistance system or an autonomous driving system.

[0029] Furthermore, those skilled in the art should understand that the systems to which the embodiments of this disclosure apply are not limited to... Figure 1 The architecture shown is not limited to the above-mentioned application scenarios.

[0030] Figure 2 A schematic flowchart of the 3D occupancy grid detection method provided in an embodiment of this disclosure is shown. See also Figure 2 The 3D occupancy grid detection method of this disclosure embodiment may include: Step 201: Obtain the current frame point cloud data of the environment around the vehicle from the lidar mounted on the vehicle. Step 202: Extract scene features from the current frame point cloud data and use the scene features to generate the current frame memory entry; Step 203: Retrieve relevant memory entries for the current frame memory entry from the memory bank. The memory bank stores historical memory entries, and each historical memory entry is obtained based on point cloud data of a historical frame. Step 204: Obtain fusion features using the current frame memory entries and related memory entries, and obtain a 3D occupancy probability grid with semantic information using the fusion features; Step 205: Determine whether a scene feature belongs to a keyframe by calculating the scene entropy value or feature change rate of the scene feature. If the scene feature belongs to a keyframe, update the memory bank using the current frame memory entry. If the scene feature does not belong to a keyframe, do not update the memory bank using the current frame memory entry.

[0031] The 3D occupancy grid detection method of this disclosure stores the current frame memory entry into a memory bank when the scene features of the current frame point cloud data belong to a keyframe, and does not store it in the memory bank when the scene features do not belong to a keyframe. Simultaneously, it combines the current frame memory entry with relevant memory entries in the memory bank to obtain a fused feature, and uses this fused feature to obtain the occupancy probability and semantic label of the 3D occupancy grid. Therefore, this disclosure achieves point cloud-based 3D occupancy grid detection through memory storage, which can significantly reduce the amount of data in the 3D occupancy grid detection process. It reduces memory usage and computational resource consumption without compromising perception accuracy, while effectively reducing complexity and improving processing efficiency, thereby meeting the real-time requirements of autonomous driving.

[0032] Furthermore, in this embodiment of the present disclosure, step 201 may further include: preprocessing the current frame point cloud data, which may include, but is not limited to, point cloud alignment, motion compensation, and noise reduction filtering. Thus, aligned and clean point cloud data can be obtained.

[0033] In the current frame point cloud data obtained in step 201, each frame can contain multiple points, and each point can have 9 original attributes. These 9 original attributes can include: spatial coordinates, reflection intensity, timestamp, dynamic flag, distance to the sensor, echo type, and reflectivity.

[0034] Further, the process of obtaining scene features in step 202 may include: voxelizing each frame of point cloud data in the current frame to obtain a voxel feature grid, which is the scene feature. Specifically, the aforementioned voxelization can be implemented using a voxelization engine. Voxelization may include: for the current frame point cloud data, determining the voxel grid and assigning each point in the current frame point cloud data to a voxel in the voxel grid; for each non-empty voxel, statistically analyzing the features of all points therein and combining these statistically obtained feature values ​​to obtain a fixed-dimensional feature vector of the non-empty voxel, which is the voxel feature. Among them, statistically analyzing the features of all points therein for each non-empty voxel may include: spatial coordinate statistics, reflection intensity statistics, dynamic marker statistics, distance statistics, echo type statistics, reflectivity statistics, and time statistics.

[0035] The scene features obtained in step 202 are a high-level, structured, and computable representation of the environment of the current frame point cloud data. The scene features contain high-level, multi-dimensional scene information, including not only the geometric and semantic features of static objects such as roads and buildings, but also the geometric and semantic features of dynamic objects such as vehicles and pedestrians.

[0036] Memory keys are used for fast scene matching. They are low-dimensional, typically 64-256 dimensions, facilitating rapid computation, and possess high discriminative power, distinguishing different scene types and relating to an abstract description of the scene. In step 202, the memory key for the current frame can be obtained through a lightweight key generator. That is, inputting scene features into this lightweight key generator will output the memory key for the current frame. This lightweight key generator may include a global average pooling module, a dimensionality reduction projection module, and an L2 normalization module connected in sequence. The scene features input into the lightweight key generator are processed through global average pooling, dimensionality reduction projection, and L2 normalization to obtain the memory key for the current frame. This lightweight key generator has the advantages of low computational cost, standardization, and good performance, and can efficiently and quickly generate memory keys that can distinguish different scenes.

[0037] In step 202, the current frame memory value can include memory features and metadata. Memory features can include geometric features and semantic features. Geometric features describe the 3D spatial structure of the scene, such as the shape, size, orientation, and surface normals of objects. Semantic features describe the meaning of the scene content, such as the category of objects (e.g., vehicles, pedestrians, trees) and the semantic labels of the scene (e.g., roads, sidewalks, buildings). These can be semantic labels for each voxel or a semantic graph of the entire scene. Memory features can be in vector form for easy storage and retrieval. Metadata can include, but is not limited to, timestamps, location, and scene tags.

[0038] In step 202, the scene features can be compressed using CNN distillation (compression ratio of 4:1) to obtain memory features. Memory features can capture key information in the scene features while removing redundancy and noise.

[0039] In step 203, when retrieving relevant memory entries for the current frame memory entry in the memory bank, memory keys in the memory bank with a similarity higher than a predetermined similarity threshold to the current frame memory key are retrieved. The memory features of these relevant memory entries are then extracted from the historical frame memory entries and the relevant memory entries for the current frame memory entry. Here, the similarity of memory keys can be obtained using various feature similarity calculation methods, such as cosine similarity.

[0040] In step 204, obtaining fusion features using the current frame memory entries and related memory entries may include: fusing the memory features in the current frame memory entries with the memory features in related memory entries to obtain fusion features.

[0041] Specifically, an exemplary process for obtaining fused features by fusing memory features from memory entries in the current frame with memory features from related memory entries may include the following steps c1 to c5: Step c1: Aggregate the memory features from the relevant memory entries: Specifically, the similarity between the memory features in each relevant memory entry and the memory features in the current frame is calculated. The aggregated memory features are obtained by weighting the memory features in all relevant memory entries by using the similarity between the memory features in each relevant memory entry and the memory features in the current frame as weights.

[0042] Step c2: Use a 1×1×1 convolution to align the dimensions of the aggregated memory features with those of the current frame memory features; Specifically, the number of channels in the current frame memory feature and the aggregated memory feature may differ. To facilitate subsequent fusion, the number of channels in both can be adjusted to be the same. For example, a 1×1×1 convolution (i.e., pointwise convolution) can be used to convert the number of channels in the aggregated memory feature to the number of channels in the current frame memory feature. A 1×1×1 convolution can change the number of channels without altering the spatial dimensions.

[0043] Step c3, calculate the gating graph; Specifically, the current frame memory features and aggregated memory features are concatenated along the channel dimension to obtain a feature map with twice the number of channels. This feature map is then processed using 3D convolution and the output values ​​are compressed to between 0 and 1 using a sigmoid activation function to obtain a gating map. The gating map is a spatial weight map used to indicate whether the current frame memory features or aggregated memory features should be trusted at each spatial location. The values ​​of the gating map are between 0 and 1, with values ​​closer to 1 indicating greater trust in the current feature and values ​​closer to 0 indicating greater trust in the memory features.

[0044] Step c4: Use a gating graph to fuse the current frame memory features and aggregated memory features to obtain fused features.

[0045] The current frame's memory features and aggregated memory features are weighted and fused using a gating graph. For each spatial location and each channel, a weighted sum of the feature values ​​in the current frame's memory features and the aggregated memory features is calculated, with the weights determined by the gating graph values ​​at that spatial location. That is, the weighted fusion is performed using the following formula: F_fused = Gate_map ⊙ F_current + (1 - Gate_map) ⊙ F_memory Where F_fused represents fused features, Gate_map represents gated graph, F_current represents current frame memory features, F_memory represents aggregated memory features, and ⊙ represents element-wise multiplication.

[0046] Step c5 involves post-processing the fused features.

[0047] Specifically, the fused features can be post-processed using a 3D convolution to fuse information between channels and enhance the expressive power of the features. This 3D convolution is lightweight, such as a 1×1×1 convolution or a 3×3×3 convolution.

[0048] The fused features obtained through the above method include both information from current observations and historical memories, and the information source for each location is adaptively selected through a gating mechanism. This gating fusion method avoids the high complexity of attention mechanisms, further improving computational efficiency.

[0049] In step 204, the fused features are processed by a 3D deconvolutional network and a 3D occupancy and semantic prediction head to obtain a 3D occupancy probability grid with semantic information.

[0050] A 3D deconvolutional network can be a three-level deconvolutional network. The first deconvolutional layer outputs an intermediate resolution feature map, the second deconvolutional layer outputs a higher resolution feature map, and the third deconvolutional layer outputs a feature map of the target size. By progressively restoring details through three levels of deconvolution, a better balance between computational cost and accuracy can be achieved.

[0051] The 3D probabilistic grid comprises multiple channels. One channel represents the occupancy probability, indicating the probability that the voxel is occupied by any object. The other channels represent semantic category probabilities, indicating the probability that the voxel belongs to each of several predefined categories when it is occupied. The occupancy probability can be calculated using the Sigmoid function, and the semantic category probability can be calculated using the Softmax function. Simultaneously predicting occupancy status and semantic category, and sharing feature extraction, can further improve efficiency.

[0052] For example, a 3D probability grid can be a 21-channel tensor, where channel 0 represents the occupancy probability and channels 1-20 represent the semantic category probabilities of 20 predefined semantic categories, respectively.

[0053] In step 205, if the scene entropy value of the scene feature is greater than a preset entropy threshold or the feature change rate of the scene feature is greater than a preset change rate threshold, the scene feature is determined to belong to a key frame; if the scene entropy value of the scene feature is less than or equal to a preset entropy threshold or the feature change rate of the scene feature is less than or equal to a preset change rate threshold, the scene feature is determined not to belong to a key frame.

[0054] Both the preset entropy threshold and the preset rate of change threshold can be set to empirical values. For example, the preset entropy threshold can be set to 0.6, and the preset rate of change threshold can be set to 0.4. In specific applications, both the preset entropy threshold and the preset rate of change threshold can be flexibly adjusted according to the needs of the application scenario.

[0055] For example, if the preset entropy threshold is set to 0.6 and the preset rate of change threshold is set to 0.4, then the following formula can be used to determine whether a scene feature belongs to a keyframe ( ).

[0056]

[0057] in, The rate of change of features representing scene characteristics. Scene entropy value, representing scene characteristics.

[0058] Specifically, the scene entropy value of the scene features is calculated as follows: the feature values ​​in the scene features are normalized to map to the range [0, 1], the normalized feature values ​​are quantized into N intervals, the interval index of each feature value is calculated, the number of feature values ​​in each interval is counted, and the ratio of the number of feature values ​​in each interval to the total number of feature values ​​is calculated to obtain the probability of each interval. The scene entropy value is obtained by calculating the probability of each interval.

[0059] The scene entropy value can be calculated using the following formula.

[0060] ,

[0061] in, This represents the probability that a feature value in the scene features appears in the i-th quantization interval. N is the preset total number of quantization intervals. For example, N=256.

[0062] It is the probability mass function of the feature value distribution in scene features, describing the frequency of different values ​​occurring in the scene features. This indicates that certain eigenvalues ​​occur frequently and are low in frequency. This indicates that certain eigenvalues ​​are rare. Specifically, when... When = 0, define ×log2( = 0.

[0063] Specifically, the feature change rate of scene features can be calculated as follows: calculate the feature difference using the current scene features and the previously obtained scene features. Calculate the L2 norm of the current scene features. and characteristic differences L2 norm The relative rate of change is obtained by dividing the L2 norm of the feature differences by the L2 norm of the current scene features. This relative rate of change is the feature change rate of the scene features.

[0064] In this embodiment, the memory bank is a collection of memory entries. A memory entry is a complete and independent environmental memory unit stored in the memory bank, recording a complete snapshot of information for a specific moment and scene. Each memory entry contains only one key-value pair, and each key-value pair corresponds to a scene memory. A scene memory is an abstract and compact representation obtained after processing multiple frames of point clouds, containing various information such as the scene's geometric structure, semantic labels, and temporal context. The structure of this memory entry is the same as the structure of the current frame memory entry mentioned above, and will not be described again.

[0065] The memory bank stores historical frame entries, which can be stored in chronological order. Each historical frame entry is obtained from a historical frame point cloud data, and the acquisition method is the same as that for the current frame entry, so it will not be repeated here. The last access time, access count, importance score, and other information of the historical frame entries in the memory bank can be updated in real time based on the retrieval status of the historical frame entries.

[0066] For example, assuming the memory bank contains m memory entries, the structure of the memory bank can be represented as follows: [ Memory entry 0: {Key0, Value0, Metadata0}, Memory entry 1: {Key1, Value1, Metadata1}, Memory entry 2: {Key2, Value2, Metadata2}, ... Memory Entries (m-1) : {Key (m-1) Value (m-1) Metadata (m-1)} ] For example, a memory entry can be represented as: Memory entry = { Key: A 256-dimensional vector; Value: A 512-dimensional vector; }

[0067] After a memory entry is stored in the memory bank, information such as spatiotemporal information, last access time, access count, and update count can be added. For example, the memory value data structure of a memory entry in the memory bank can be represented as follows: Memory value = { "Memory Characteristics": { "Geometric features": [...], # Spatial structure, shape, and location "Semantic features": [...], # Category, attribute, relationship }, "Spatiotemporal information": { "timestamp" Spatial location: [x, y, z], }, "Metadata": { "Creation Time": timestamp, "Last access time": timestamp, "Number of visits": int, Importance score: int, } } The state of the memory bank can be represented by the number of valid memories. If the number of valid memories in the current memory bank reaches the preset upper limit, it indicates that the memory bank is full. If the number of valid memories in the current memory bank has not reached the preset upper limit, it indicates that the memory bank is not full.

[0068] In practical applications, the preset upper limit represents the capacity of the memory bank, which can be pre-configured. For example, this upper limit can be configured to 128, meaning that the memory bank is full when the number of memory entries reaches 128. By configuring the upper limit of the memory bank, the memory size occupied by the memory bank can be effectively limited. It has been verified that too many memory entries will reduce retrieval speed, while 128 memory entries is a balance point, which can cover common typical scenarios of autonomous driving, and can also optimize retrieval efficiency and reduce computing resource consumption.

[0069] In step 205, the process of updating the memory bank using the current frame memory entry may include: writing the current frame memory entry into the memory bank and incrementing the effective memory count in the memory bank by 1; determining whether the effective memory count in the memory bank has reached a preset upper limit; if the effective memory count in the memory bank has reached the upper limit, then discarding the historical memory entry that has been used the least or has the lowest importance score in the memory bank and updating the effective memory count in the memory bank. Therefore, the least important memory entry can be discarded when the memory bank is full, thereby reducing memory usage.

[0070] The method described in this embodiment of the present disclosure has the following beneficial effects: 1) Using a memory bank to store scene features during the 3D occupancy raster detection process results in a small data storage requirement, which can significantly reduce memory usage; 2) The keyframe mechanism based on entropy or rate of change can ensure that only information-rich or rapidly changing scenes are remembered, avoiding the memory bank from being filled with a large number of trivial frames, and further reducing memory usage.

[0071] 3) The elimination strategy keeps the memory bank diverse and fresh, and can adapt to environmental changes.

[0072] 4) The computational complexity is reduced from O(N²) to O(N), and the processing time for one frame can be controlled within 100ms, which can meet the real-time perception requirements of autonomous driving.

[0073] This disclosure is particularly applicable to the following scenarios: 1) Handling repetitive scenarios: For example, when vehicles pass through the same intersections, garages, or company entrances every day, the memory bank can provide accurate "experience," predict the location of familiar obstacles, and even react in advance. 2) When the LiDAR is partially obstructed by large vehicles ahead, tunnel walls, etc., the complete historical structure of the area can be retrieved from the memory bank, greatly improving the continuity of perception. 3) Quickly understanding complex static environments: When encountering complex overpasses, roundabouts, or construction zones, it is not necessary to analyze each frame from scratch. Through retrieval, the understanding of similar structures can be quickly invoked, accelerating scene analysis. 4) Lightweight design, meeting automotive-grade requirements: The significant reduction in computation and memory means lower power consumption and heat dissipation requirements, which is beneficial to the long-term stable operation of automotive-grade chips.

[0074] Figure 3 A schematic diagram of the structure of the 3D occupancy grid detection device provided in an embodiment of this disclosure is shown. See also Figure 3 The 3D occupancy grid detection device 300 of this disclosure embodiment may include: The data acquisition unit 301 is used to acquire current frame point cloud data of the environment around the vehicle from the lidar mounted on the vehicle. Scene feature unit 302 is used to extract scene features from the current frame point cloud data; The memory encoding unit 303 is used to generate memory entries for the current frame using scene features; The memory retrieval unit 304 is used to retrieve relevant memory entries of the current frame memory entry from the memory bank. The memory bank stores historical memory entries, and each historical memory entry is obtained based on point cloud data of a historical frame. The feature fusion unit 305 is used to obtain fused features using the current frame memory entries and related memory entries; Multi-task unit 306 is used to obtain a 3D occupancy probability grid with semantic information by utilizing fused features; The keyframe determination unit 307 is used to determine whether a scene feature belongs to a keyframe by calculating the scene entropy value or feature change rate of the scene feature; The memory bank update unit 308 is used to update the memory bank with the current frame memory entry when the scene feature belongs to a keyframe, and not to update the memory bank with the current frame memory entry when the scene feature does not belong to a keyframe.

[0075] Furthermore, the keyframe determination unit 307 can be specifically used to determine that a scene feature belongs to a keyframe when the scene entropy value of the scene feature is greater than a preset entropy threshold or the feature change rate of the scene feature is greater than a preset change rate threshold; and to determine that a scene feature does not belong to a keyframe when the scene entropy value of the scene feature is less than or equal to a preset entropy threshold or the feature change rate of the scene feature is less than or equal to a preset change rate threshold.

[0076] Furthermore, the keyframe determination unit 307 can be used to calculate the scene entropy value of the scene features in the following way: normalize the feature values ​​in the scene features to map them to the range of [0, 1], quantize the normalized feature values ​​into N intervals, calculate the interval index for each feature value, count the number of feature values ​​in each interval and calculate the ratio of the number of feature values ​​in each interval to the total number of feature values ​​to obtain the probability of each interval, and calculate the scene entropy value based on the probability of each interval, where N is the preset total number of quantization intervals.

[0077] Furthermore, the keyframe determination unit 307 can be specifically used to calculate the feature change rate of scene features in the following manner: calculate the feature difference using the current scene features and the previously obtained scene features, calculate the L2 norm of the current scene features and the L2 norm of the feature difference, divide the L2 norm of the feature difference by the L2 norm of the current scene features to obtain the relative change rate, which is the feature change rate of scene features.

[0078] Furthermore, the memory bank update unit 308 can be specifically used to: write the current frame memory entry into the memory bank and increment the number of valid memories in the memory bank by 1; determine whether the number of valid memories in the memory bank has reached a preset upper limit; if the number of valid memories in the memory bank has reached the upper limit, then discard the historical memory entry that has been used the least or has the lowest importance score in the memory bank and update the number of valid memories in the memory bank.

[0079] Furthermore, the current frame memory entry includes a current frame memory key and a current frame memory value, and the current frame memory value includes current frame memory features; the memory encoding unit 303 can be specifically used to obtain the current frame memory key by performing global average pooling, dimensionality reduction projection and L2 normalization on the scene features, and to obtain the current frame memory features by performing CNN distillation compression on the scene features.

[0080] In specific applications, the 3D occupancy grid detection device 300 can be implemented through software, hardware, or a combination of both. For example, the 3D occupancy grid detection device 300 can be implemented as software running in the electronic device 400 described below.

[0081] Figure 4 A schematic structural diagram of an electronic device provided according to an embodiment of this disclosure is shown. See also... Figure 4 The electronic device 400 provided in this embodiment may include: one or more processors 401 and a memory 402. The memory 402 stores a computer program, which, when run by the one or more processors 401, causes the processors 401 to perform the aforementioned 3D occupancy grid detection method.

[0082] Processor 401 may be, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), or other processing units with data processing capabilities and / or instruction execution capabilities.

[0083] Memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the methods described above and / or other desired functions.

[0084] Depending on the specific application, the electronic device 400 may also include any other suitable components.

[0085] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the 3D occupancy grid detection method described above.

[0086] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0087] Furthermore, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when run by a processor, causes the processor to perform the steps in the 3D occupancy grid detection method described above.

[0088] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0089] The technical solutions provided in this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, those skilled in the art will recognize that, based on the ideas of this disclosure, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0090] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications or equivalent substitutions made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A 3D occupancy grid detection method, characterized in that, The 3D occupancy grid detection method includes: The current frame point cloud data of the surrounding environment of the vehicle is obtained from the lidar mounted on the vehicle. Extract scene features from the current frame point cloud data, and use the scene features to generate a current frame memory entry; Retrieve relevant memory entries for the current frame memory entry from the memory bank, which stores historical memory entries, each of which is obtained from point cloud data of a historical frame; The fusion features are obtained by using the current frame memory entries and related memory entries, and the fusion features are used to obtain a 3D occupancy probability grid with semantic information; Whether a scene feature belongs to a keyframe is determined by calculating the scene entropy value or feature change rate of the scene feature. If the scene feature belongs to a keyframe, the memory bank is updated using the current frame memory entry. If the scene feature does not belong to a keyframe, the memory bank is not updated using the current frame memory entry.

2. The method according to claim 1, characterized in that, The step of determining whether a scene feature belongs to a keyframe by calculating the scene entropy value or feature change rate of the scene feature includes: When the scene entropy value of the scene feature is greater than a preset entropy threshold or the feature change rate of the scene feature is greater than a preset change rate threshold, the scene feature is determined to belong to a keyframe. When the scene entropy value of the scene feature is less than or equal to the preset entropy threshold or the feature change rate of the scene feature is less than or equal to the preset change rate threshold, the scene feature is determined not to belong to a keyframe.

3. The method according to claim 1, characterized in that, The scene entropy value of the scene features is calculated as follows: the feature values ​​in the scene features are normalized to map to the range of [0, 1], the normalized feature values ​​are quantized into N intervals, the interval index of each feature value is calculated, the number of feature values ​​in each interval is counted and the ratio of the number of feature values ​​in each interval to the total number of feature values ​​is calculated to obtain the probability of each interval, and the scene entropy value is calculated based on the probability of each interval, where N is the preset total number of quantization intervals.

4. The method according to claim 1, characterized in that, The feature change rate of the scene features is calculated as follows: the feature difference is calculated using the current scene features and the previously obtained scene features; the L2 norm of the current scene features and the L2 norm of the feature difference are calculated; the L2 norm of the feature difference is divided by the L2 norm of the current scene features to obtain the relative change rate, which is the feature change rate of the scene features.

5. The method according to claim 1, characterized in that, Updating the memory bank using the current frame memory entry includes: Write the current frame memory entry into the memory bank and increment the number of valid memories in the memory bank by 1; Determine whether the effective memory count of the memory bank has reached a preset upper limit. If the number of valid memories in the memory bank reaches the upper limit, then the historical memory entry with the fewest uses or the lowest importance score in the memory bank is eliminated and the number of valid memories in the memory bank is updated.

6. The method according to claim 1, characterized in that, The historical frame memory entries in the memory bank are stored in timestamp order.

7. The method according to claim 1, characterized in that, The current frame memory entry includes a current frame memory key and a current frame memory value. The current frame memory value includes current frame memory features. The current frame memory key is obtained by performing global average pooling, dimensionality reduction projection, and L2 normalization on the scene features. The current frame memory features are obtained by performing CNN distillation and compression on the scene features.

8. A 3D occupancy grid detection device, characterized in that, The 3D occupancy grid detection device includes: The data acquisition unit is used to acquire current frame point cloud data of the surrounding environment of the vehicle from the lidar mounted on the vehicle. A scene feature unit is used to extract scene features from the current frame point cloud data; A memory encoding unit is used to generate a current frame memory entry using the scene features; The memory retrieval unit is used to retrieve relevant memory entries of the current frame memory entry from the memory bank, which stores historical memory entries, each of which is obtained based on point cloud data of a historical frame. The feature fusion unit is used to obtain fused features by utilizing the current frame memory entries and related memory entries; A multi-task unit is used to obtain a 3D occupancy probability grid with semantic information by utilizing the fusion features; The keyframe determination unit is used to determine whether a scene feature belongs to a keyframe by calculating the scene entropy value or the feature change rate of the scene feature; The memory bank update unit is used to update the memory bank with the current frame memory entry when the scene feature belongs to a keyframe, and not to update the memory bank with the current frame memory entry when the scene feature does not belong to a keyframe.

9. An electronic device, characterized in that, include: A processor and a memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images