Obstacle detection methods, devices, vehicles, and storage media

Abstract

This invention relates to an obstacle detection method, apparatus, vehicle, and storage medium. The method includes: acquiring point cloud information around the vehicle and extracting multiple point cloud features from the point cloud information; encoding each point cloud feature to obtain an encoded feature; aligning the encoded features of the same point cloud feature at all times; and fusing the aligned encoded features at all times to obtain a fused feature; and detecting obstacle information around the vehicle based on the fused feature. This solves the problems in related technologies, such as the loss of local features due to the Transformer structure, the high dependence of the temporal framework on positioning information calculated by other sensors, and the impact of positioning or occlusion on obstacle detection in special scenarios such as underground parking lots, leading to a decline in user experience.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to an obstacle detection method, device, vehicle, and storage medium. Background Technology

[0002] In autonomous driving, laser perception solutions play a crucial role. Based on point cloud encoding methods, laser perception solutions are mainly divided into three types: voxel-based laser perception, point cloud-based laser perception, and multi-view-based laser perception.

[0003] Among them, voxel-based laser sensing has been widely adopted in the industry due to its advantages in efficiency and deployment. For example, the PointPillars framework has become the mainstream choice for laser sensing tasks. Various improvement schemes based on this framework have emerged one after another, which has led to the rapid development of laser sensing in recent years. This has further attracted more hardware manufacturers to participate in the production and design of laser sensors, forming a virtuous cycle.

[0004] However, the laser sensing solutions of related technologies still have structural imperfections. They are highly dependent on positioning information and environmental scenarios, which may affect the stability of detection and the user experience. Summary of the Invention

[0005] This invention provides an obstacle detection method, device, vehicle, and storage medium to address issues such as the loss of local features caused by the Transformer structure in related technologies, the high dependence of the temporal framework on the positioning information calculated by other sensors, and the impact of positioning or occlusion on obstacle detection in special scenarios such as underground parking lots, which leads to a decline in user experience.

[0006] A first aspect of the present invention provides a method for detecting obstacles, comprising the following steps: acquiring point cloud information around a vehicle and extracting multiple point cloud features from the point cloud information; encoding each of the multiple point cloud features to obtain an encoded feature; aligning the encoded features of the same point cloud feature at all times; fusing the aligned encoded features at all times to obtain a fused feature; and detecting obstacle information around the vehicle based on the fused feature.

[0007] Based on the above technical means, the embodiments of the present invention can perform temporal fusion by aligning point clouds and encoding them, so that temporal information can be preserved. Since the final generated features contain information from both the current moment and the historical moment, the temporal features will provide some completion for distant, sparse, or occluded targets. Therefore, even obstacles in extreme or special scenarios can be guided by historical information, increasing detection stability. Thus, accurate obstacle detection can be achieved without relying on positioning information, freeing us from the limitations of positioning sensors and positioning accuracy. Furthermore, since it is free from positioning dependence, it can simultaneously meet the needs of online and offline applications and deployments, effectively meeting the needs of actual use and improving the user experience.

[0008] Optionally, aligning the encoded features of the same point cloud feature at all times includes: setting a self-learning matrix; using the self-learning matrix to learn the alignment relationship between point cloud features at different times; and using the alignment relationship to align the encoded features of the same point cloud feature at all times.

[0009] Based on the above technical means, the embodiments of the present invention can perform learnable encoding of temporal point cloud alignment and adopt corresponding feature fusion methods to preserve temporal information, provide guidance on historical information for obstacles in extreme or special scenarios, and increase detection stability.

[0010] Optionally, aligning the encoded features of the same point cloud feature across all times includes: extracting the neighboring features around the encoded feature at each time; and aligning the encoded features of the same point cloud feature across all times based on the neighboring features belonging to the encoded features of the same point cloud feature.

[0011] Based on the above technical means, the embodiments of the present invention can further employ a learnable downsampling feature extraction method to aggregate voxel neighborhood features before encoding, further aligning features so that each feature in the current frame is not only superimposed with features at the same position in historical frames, but is generated by taking and aggregating surrounding neighborhood features to produce superimposed features for each voxel, making the features more complete and increasing detection robustness.

[0012] Optionally, encoding each point cloud feature among the plurality of point cloud features to obtain encoded features includes: performing multi-scale feature grouping on all point cloud features to obtain grouped features at multiple scales; and encoding the grouped features at each scale to obtain encoded features at each scale.

[0013] Based on the above technical means, the embodiments of the present invention can adopt a multi-scale feature encoding method to group point cloud features at multiple scales, encode the grouped features at each scale, and finally obtain the encoded features at each scale. Thus, the embodiments of the present invention can fill in feature gaps of different degrees, give full play to the advantages of the self-attention mechanism, and meet the needs of practical use.

[0014] Optionally, the step of performing multi-scale feature grouping on all point cloud features to obtain grouped features at multiple scales includes: setting a preset window of target size; using the preset window to group all point cloud features to obtain grouped features at one scale; translating and / or scaling the preset window, and using the translated and / or scaled preset window to group all point cloud features to obtain grouped features at other scales.

[0015] Based on the above technical means, the embodiments of the present invention can use a method of grouping point cloud features using a preset window. By translating and / or scaling the preset window, grouped features of point cloud features at different scales are obtained. Feature extraction is then performed based on multi-scale grouping, preserving local features between voxels after voxelization, compensating for the problem of local feature loss caused by the encoder structure, and meeting the needs of practical use.

[0016] Optionally, grouping all point cloud features using a preset window after translation and / or scaling includes: setting a preset number of features in the group; setting a target number of translations and / or scalings based on the preset number; and ending the multi-scale feature grouping when the actual number of translations and / or scalings reaches the target number.

[0017] According to the above technical means, the embodiments of the present invention can preset the preset number of grouped features, determine the preset number of times the preset window is translated and / or scaled based on the number of grouped features, and then perform operations based on this number of times to obtain multi-scale features after the preset window changes accordingly. When the number of times reaches the target, the multi-scale feature grouping can be considered complete and the grouping operation ends; this avoids repetitive and redundant operations, the grouping results are more accurate and adaptable, and the reliability of multi-scale feature grouping is improved.

[0018] Optionally, encoding the grouped features at each scale includes performing multi-attention encoding on the grouped features at each scale.

[0019] Based on the above technical means, the embodiments of the present invention can obtain local information at different scales each time the encoder encodes, thereby making up for the problem of local feature loss caused by the encoder structure, giving full play to the advantages of the self-attention mechanism, and achieving the effect of taking into account both the global and local aspects.

[0020] Optionally, the actual number of fused encoded features is less than the maximum number of fused features during the training phase.

[0021] Based on the above technical means, the embodiments of the present invention can input a variable overlay frame rate, as long as the number of overlays is less than the maximum length.

[0022] Optionally, the step of extracting multiple point cloud features from the point cloud information includes: traversing all point clouds in the point cloud information and filtering point clouds that are not within a preset area; calculating the corresponding voxel coordinates based on the point cloud coordinates, and generating point cloud features based on the voxel coordinates.

[0023] Based on the above technical means, the embodiments of the present invention can perform point cloud filtering, voxelization and feature processing, which on the one hand shortens the encoding time; on the other hand, removes the dependence on intensity information, thereby eliminating the problem of inconsistent intensity scale caused by different laser hardware.

[0024] Optionally, the step of detecting obstacle information around the vehicle based on the fused features includes: inputting the fused features into the detection head, outputting the center position, size, and orientation information of the predicted bounding box of the obstacle, and generating the obstacle information based on one or more of the center position, size, and orientation information of the predicted bounding box.

[0025] Based on the above technical means, the embodiments of the present invention can add learnable aggregation processing before the final detection head to improve efficiency, shorten detection time, meet actual use needs, and enhance user experience.

[0026] Optionally, before detecting obstacle information around the vehicle based on the fused features, the method further includes: downsampling feature encoding the fused features to obtain encoded fused features.

[0027] Based on the above technical means, the embodiments of the present invention can record the location of the aggregation center, project each newly generated feature back onto the new bird's-eye view feature map, and use it as the input of the subsequent static target detection head, thereby improving detection efficiency and providing historical information guidance for obstacles in extreme or special scenarios.

[0028] Optionally, before detecting obstacle information around the vehicle based on the fused features, the method further includes: projecting a bird's-eye view onto the fused features to obtain the projected fused features.

[0029] Based on the above technical means, the embodiments of the present invention can make the fusion features not constrained by the field of view or viewpoint, nor restricted by the law of near-large and far-small, avoiding the high dependence on positioning information, ignoring the positioning or occlusion effects that may be caused by special scenarios such as underground parking lots, thus expanding the application range and improving the user experience.

[0030] A second aspect of the present invention provides an obstacle detection device, comprising: an acquisition module for acquiring point cloud information around a vehicle and extracting multiple point cloud features from the point cloud information; a fusion module for encoding each of the multiple point cloud features to obtain an encoded feature, aligning the encoded features of the same point cloud feature at all times, and fusing the aligned encoded features at all times to obtain a fused feature; and a detection module for detecting obstacle information around the vehicle based on the fused feature.

[0031] Optionally, the fusion module is further configured to: set a self-learning matrix; use the self-learning matrix to learn the alignment relationship between point cloud features at different times; and use the alignment relationship to align the encoded features of the same point cloud feature at all times.

[0032] Optionally, the fusion module is further configured to: extract the neighborhood features around the encoded features at each time step; align the encoded features of the same point cloud feature at all times based on the neighborhood features belonging to the encoded features of the same point cloud feature.

[0033] Optionally, the fusion module is further configured to: perform multi-scale feature grouping on all point cloud features to obtain grouped features at multiple scales; and encode the grouped features at each scale to obtain coded features at each scale.

[0034] Optionally, the fusion module is further configured to: set a preset window of target size; group all point cloud features using the preset window to obtain grouped features at one scale; translate and / or scale the preset window, and group all point cloud features using the translated and / or scaled preset window to obtain grouped features at other scales.

[0035] Optionally, the fusion module is further configured to: set a preset number of grouped features; set a target number of translations and / or scalings based on the preset number; and terminate the multi-scale feature grouping when the actual number of translations and / or scalings reaches the target number.

[0036] Optionally, the fusion module is further configured to: perform multi-attention encoding on the grouped features at each scale.

[0037] Optionally, the actual number of fused encoded features is less than the maximum number of fused features during the training phase.

[0038] Optionally, the acquisition module is further configured to: traverse all point clouds in the point cloud information, filter point clouds that are not in the preset area; calculate the corresponding voxel coordinates based on the point cloud coordinates, and generate point cloud features based on the voxel coordinates.

[0039] Optionally, the detection module is further configured to: input the fused features into the detection head, output the center position, size, and orientation information of the predicted bounding box of the obstacle, and generate the obstacle information based on one or more of the center position, size, and orientation information of the predicted bounding box.

[0040] Optionally, the detection module is further configured to: perform downsampling feature encoding on the fused features to obtain encoded fused features.

[0041] Optionally, the detection module is further configured to: project a bird's-eye view onto the fused features to obtain the projected fused features.

[0042] A third aspect of the present invention provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the obstacle detection method as described in the above embodiments.

[0043] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the obstacle detection method as described in the above embodiments.

[0044] The beneficial effects of this invention are:

[0045] (1) The embodiments of the present invention can perform temporal fusion by aligning point clouds and encoding, so that temporal information can be preserved. Since the final generated features contain information from both the current time and the historical time, the temporal features will provide some completion for distant sparse or occluded targets. Therefore, even obstacles in extreme or special scenarios can be guided by historical information, increasing detection stability. Thus, accurate detection of obstacles can be achieved without relying on positioning information, getting rid of the limitations of positioning sensors and positioning accuracy. Furthermore, since it is free from positioning dependence, it can simultaneously meet the needs of online and offline applications and deployments, effectively meeting the needs of actual use and improving the user experience.

[0046] (2) The embodiments of the present invention can encode the temporal point cloud alignment in a learnable manner and adopt the corresponding feature fusion method so that the temporal information can also be preserved, providing guidance on historical information for obstacles in extreme or special scenarios and increasing detection stability;

[0047] (3) In this embodiment of the invention, a learnable downsampling feature extraction method can be further adopted to aggregate the voxel neighborhood features before encoding, further aligning the features so that each feature of the current frame is not only superimposed with the features of the same position in the historical frame, but is generated by taking the surrounding neighborhood features and aggregating them to generate the superimposed features of each voxel, making the features more complete and increasing the detection robustness.

[0048] (4) The embodiments of the present invention can adopt a multi-scale feature encoding method to group point cloud features into multi-scale features, encode the grouped features at each scale, and finally obtain the encoded features at each scale. Thus, the embodiments of the present invention can fill in feature loss of different degrees, give full play to the advantages of self-attention mechanism, and meet the needs of actual use.

[0049] (5) The present invention can use a method of grouping point cloud features using a preset window. By translating and / or scaling the preset window, grouped features of point cloud features at different scales can be obtained. Based on multi-scale grouping, feature extraction is performed, local features between voxels are preserved after voxelization, and the problem of local feature loss caused by encoder structure is made up to meet the needs of actual use.

[0050] (6) The present invention embodiment can preset the preset number of grouped features, and based on the number of grouped features, determine the number of times the preset window is translated and / or scaled, and then perform operations based on the number of times to obtain multi-scale features after the preset window changes accordingly. When the number of times reaches the target, the multi-scale feature grouping can be considered complete and the grouping operation ends; avoids repeated and redundant operations, the grouping results are more accurate and adaptable, and improves the reliability of multi-scale feature grouping.

[0051] (7) The embodiments of the present invention can obtain local information at different scales each time the encoder encodes, thereby compensating for the loss of local features caused by the encoder structure, giving full play to the advantages of the self-attention mechanism, and achieving the effect of taking into account both the global and local aspects.

[0052] (8) In this embodiment of the invention, a variable overlay frame rate can be input, as long as the number of overlays is less than the maximum length;

[0053] (9) The embodiments of the present invention can perform point cloud filtering, voxelization and feature processing, which shortens the encoding time on the one hand; and removes the dependence on intensity information on the other hand, thereby eliminating the problem of inconsistent intensity scale caused by different laser hardware.

[0054] (10) In this embodiment of the invention, a learnable aggregation process can be added before the final detection head to improve efficiency, shorten detection time, meet actual use needs, and enhance user experience;

[0055] (11) The embodiments of the present invention can record the location of the aggregation center, and project each newly generated feature back onto the new bird's-eye view feature map as the input of the subsequent static target detection head, thereby improving the detection efficiency and providing historical information guidance for obstacles in extreme or special scenarios;

[0056] (12) The embodiments of the present invention can make the fusion features not constrained by the field of view or viewpoint, nor restricted by the law of near-large and far-small, avoiding the high dependence on positioning information, ignoring the positioning or occlusion effects that may be caused by special scenarios such as underground parking lots, thus having a wider range of applications and improving the user experience.

[0057] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0058] Figure 1 This is a flowchart of an obstacle detection method according to an embodiment of the present invention;

[0059] Figure 2 This is a model structure diagram for point cloud feature extraction according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the MSA (multi-head self attention) structure according to an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of multi-scale feature grouping according to an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of temporal feature fusion according to an embodiment of the present invention;

[0063] Figure 6 This is an example diagram of an obstacle detection device according to an embodiment of the present invention;

[0064] Figure 7 This is a schematic diagram of the structure of a vehicle according to an embodiment of the present invention. Detailed Implementation

[0065] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0066] In related technologies, laser perception solutions play a crucial role in autonomous driving: laser point clouds are widely used for various tasks such as obstacle perception, multi-sensor fusion, and localization mapping during vehicle operation due to their accurate ranging information; at the same time, the characteristics of laser sensors being insensitive to light and providing WYSIWYG are often used as a safety redundancy in perception tasks, providing strong safety assurance for driving vehicles.

[0067] Laser sensing solutions mainly rely on manually designed descriptors to extract features. However, this method often suffers from significant limitations in specific scenarios and low efficiency. Therefore, after deep learning was widely applied to computer vision, point cloud-based deep learning feature extraction solutions quickly replaced the previous manually designed descriptor-based feature extraction solutions. This enabled laser sensing tasks to run in real time on vehicles and had good generalization capabilities.

[0068] Laser sensing solutions in related technologies can be categorized into three types based on point cloud encoding methods: voxel-based laser sensing, point cloud-based laser sensing, and multi-view-based laser sensing. Among these, voxel-based laser sensing is widely adopted in industry due to its advantages in efficiency and deployment. For example, the PointPillars framework has become the mainstream choice for current laser sensing tasks. Various improved solutions based on this framework have emerged, leading to the rapid development of laser sensing in recent years. This, in turn, has attracted more hardware manufacturers to participate in the production and design of laser sensors, creating a virtuous cycle.

[0069] In summary, the main drawbacks of the laser sensing technology based on Transformer are: (1) the structure is relatively simple and fails to fully utilize the advantages of the self-attention mechanism proposed in Transformer; (2) the weakness of Transformer in local feature extraction is ignored; (3) the temporal framework is highly dependent on the positioning information calculated by other sensors; and (4) the positioning or occlusion effects caused by special scenarios such as underground parking lots are ignored.

[0070] The obstacle detection method, apparatus, vehicle, and storage medium of the present invention are described below with reference to the accompanying drawings.

[0071] Specifically, Figure 1 This is a schematic flowchart of an obstacle detection method provided in an embodiment of the present invention.

[0072] like Figure 1 As shown, the obstacle detection method includes the following steps:

[0073] In step S101, point cloud information around the vehicle is acquired, and multiple point cloud features are extracted from the point cloud information.

[0074] Point cloud information refers to a set of vectors in a three-dimensional coordinate system. In this embodiment of the invention, point cloud information around the vehicle can be obtained in at least one way, such as by using a vehicle body sensor, etc., without any specific limitation.

[0075] It is understood that the embodiments of the present invention can first obtain point cloud information around the vehicle body to obtain point cloud features, which facilitates the encoding of point cloud features in subsequent steps; wherein, the embodiments of the present invention can use at least one method to extract multiple point cloud features from the point cloud information, without making specific limitations.

[0076] As one possible approach, multiple point cloud features are extracted from the point cloud information, including: traversing all point clouds in the point cloud information and filtering out point clouds that are not within a preset area; calculating the corresponding voxel coordinates based on the point cloud coordinates, and generating point cloud features based on the voxel coordinates.

[0077] The preset area can be set according to the actual situation. For example, it can be set to a specific area around the vehicle body, without any specific restrictions.

[0078] It is understood that embodiments of the present invention can use a lightweight coding method to extract point cloud features, including point cloud filtering, voxelization, and feature processing. Therefore, embodiments of the present invention save the time consumption of traditional VFE (Stacked Voxel Feature Encoding) and eliminate the dependence on intensity information, thereby eliminating the influence caused by inconsistent intensity scales among different laser hardware. The process of extracting point cloud features in embodiments of the present invention can be specifically described as follows:

[0079] (1) Point cloud filtering: In this embodiment of the invention, the point cloud dimension can be 3, that is, the three-dimensional coordinate information of each point in the point cloud information; such as Figure 2 As shown, after traversing all points one by one, this embodiment of the invention can retain points containing point cloud features in the ROI (region of interest) region and filter out point clouds that are not in the preset ROI region, thereby obtaining coordinates containing point cloud feature information.

[0080] (2) Calculating the corresponding voxel features: In this embodiment of the invention, the point cloud coordinates can be used as a reference. The point cloud can be directly assigned a value, which can be set according to the actual situation, such as 1, etc., without specific limitations. Taking an assignment of 1 as an example, the calculation can yield results such as (L... x L y L z Binary voxel raster of 1 / 2 dimension;

[0081] (3) Feature processing: In this embodiment of the invention, L can be made x =W,L y =H,L z =C, directly obtain the feature map of the (H, W, C) dimension.

[0082] It should be noted that the lightweight encoding method used in the embodiments of the present invention needs to pay attention to the impact of binarization. In order to prevent the loss of point cloud features within voxels, the voxel size should be as small as possible when extracting point cloud features in the embodiments of the present invention, for example, it can be set to 0.1m, etc., without making specific limitations.

[0083] In step S102, each point cloud feature in multiple point cloud features is encoded to obtain encoded features, the encoded features of the same point cloud feature at all times are aligned, and the encoded features at all times after alignment are fused to obtain fused features.

[0084] In this embodiment, the actual number of fused encoding features is less than the maximum number of fused features during the training phase, meaning the superposition frame rate is variable. This embodiment can use at least one encoding method for encoding, such as using a Transformer MSA for encoding, etc., without specific limitations. The following embodiments will use Transformer as an example for illustration.

[0085] It is understood that embodiments of the present invention can use temporal fusion to align the encoded features of the same point cloud feature across all time points, thereby fusing historical frame information with current frame information to obtain a fused feature containing information from both the current and historical time points. This overcomes the limitations of positioning accuracy and provides some completion for distant, sparse, or occluded targets. Furthermore, embodiments of the present invention can use at least one method to achieve temporal fusion of point cloud features. For example, embodiments of the present invention can use a learnable spatiotemporal coding strategy, as detailed below:

[0086] In this embodiment of the invention, aligning the encoded features of the same point cloud feature at all times includes: setting a self-learning matrix; using the self-learning matrix to learn the alignment relationship between point cloud features at different times; and using the alignment relationship to align the encoded features of the same point cloud feature at all times.

[0087] It is understandable that, since it does not depend on localization, the cloud features of the same point in different frames of the present invention can be added to a learning matrix W before encoding, so that the model can adaptively learn the features at different times.

[0088] Specifically, embodiments of the present invention may use a method such as Figure 3 The Transformer-based MSA encoding shown adds a self-learning matrix W before the first Transformer encoding of each QKV group's features in different frames, i.e.:

[0089] feature_0 = MSA(Wgroup),

[0090] After multiple encoding steps, each voxel can acquire its own and its surrounding features, namely:

[0091] feature = MSA(group),

[0092] The final generated feature length is L; thus, the self-learning matrix can only learn the feature misalignment problem caused by vehicle motion, reducing the difficulty of model learning.

[0093] It should be noted that the above alignment process only considers static obstacles and does not take into account the feature misalignment problem that may be caused by obstacle movement. In order to further align features, the embodiments of the present invention can further superimpose historical frames of point cloud features. On the basis of superimposing features at the same position in the historical frames of point cloud features, features from the surrounding neighborhood can also be taken and aggregated to generate superimposed features for each voxel; specifically:

[0094] In this embodiment of the invention, aligning the encoded features of the same point cloud feature at all times includes: extracting the neighboring features around the encoded feature at each time; and aligning the encoded features of the same point cloud feature at all times based on the neighboring features belonging to the encoded features of the same point cloud feature.

[0095] It is understood that embodiments of the present invention can aggregate features from the surrounding neighborhood to generate superimposed features for each voxel, i.e.:

[0096] feature_voxel=Cat(feature_voxel(t)

[0097] +Maxpool(feature_voxels(t-1))+...+Maxpool(feature_voxels(tn))),

[0098] The generated feature length is nL.

[0099] It should be noted that, in the embodiments of the present invention, the voxel superposition of the surrounding neighborhood features can be performed directly after obtaining the point cloud features in step S101, or the voxel superposition of the surrounding neighborhood features can be performed after aligning the encoded features of the same point cloud feature at all times; the method of first aligning the temporal sequence and then aligning the neighborhood features can reduce the amount of computation and improve the alignment speed.

[0100] In this embodiment of the invention, encoding each point cloud feature among multiple point cloud features to obtain encoded features includes: performing multi-scale feature grouping on all point cloud features to obtain grouped features at multiple scales; and encoding the grouped features at each scale to obtain encoded features at each scale.

[0101] In this embodiment of the invention, encoding the grouped features at each scale may include multi-attention encoding of the grouped features at each scale.

[0102] It is understood that embodiments of the present invention can employ a multi-scale feature encoding method to group point cloud features at multiple scales, encode the grouped features at each scale, and finally obtain the encoded features at each scale. Thus, embodiments of the present invention can fill in feature gaps of varying degrees. Furthermore, embodiments of the present invention can use at least one method for multi-scale grouping, such as using a preset window for grouping, without specific limitations. The method of using a preset window for multi-scale grouping of point cloud features in embodiments of the present invention can be specifically as follows:

[0103] One possible approach is to perform multi-scale feature grouping on all point cloud features to obtain grouped features at multiple scales, including: setting a preset window with a target size; using the preset window to group all point cloud features to obtain grouped features at one scale; translating and / or scaling the preset window, and using the translated and / or scaled preset window to group all point cloud features to obtain grouped features at other scales.

[0104] The preset window can be preset in advance according to the actual situation. For example, it can be a window with an N, N dimension, without specific limitations. In the following embodiments, the preset window is (N, N) as an example for illustration.

[0105] It is understood that embodiments of the present invention can use a method of grouping point cloud features using a preset window. By translating and / or scaling the preset window, grouped features of point cloud features at different scales are obtained. Feature extraction is then performed based on multi-scale grouping, preserving local features between voxels after voxelization, thus meeting the needs of practical use. Specifically, the translation and scaling steps of the preset window in embodiments of the present invention can be as follows:

[0106] In this embodiment of the invention, all point cloud features are grouped using a preset window after translation and / or scaling, including: setting a preset number of grouped features; setting a target number of translations and / or scalings based on the preset number; and ending multi-scale feature grouping when the actual number of translations and / or scalings reaches the target number.

[0107] The preset quantity can be set according to the actual situation, and the target number of times can be specifically set according to the preset quantity, without any specific limitation.

[0108] It is understood that the embodiments of the present invention can preset the preset number of grouped features, and based on the preset number of grouped features, determine the preset number of times the window is translated and / or scaled, and then perform operations based on this number of times to obtain the multi-scale features after the preset window changes accordingly. When the number of times reaches the target, the multi-scale feature grouping can be considered complete, and the grouping operation ends. The following will specifically illustrate the multi-scale feature encoding method in the present invention through an embodiment:

[0109] For example, the multi-scale feature encoding method of the embodiments of the present invention can be specifically as follows: Figure 4 As shown, before each Transformer encoding, all voxels are grouped once, based on a preset window. This embodiment of the invention can translate and scale the preset window to different degrees, thereby obtaining different voxel groups, i.e.:

[0110] group = g(f(W)),

[0111] Where g is the grouping strategy, f is the window translation and scaling operation, and W is the preset window with an (N, N) dimension. The features corresponding to the voxels in each group are used as QKV encoded by Transformer. QKV is obtained by linear transformation of the input position encoding vector x. It can be understood that Q is the information to be queried, K is the queried vector, and V is the value obtained by the query.

[0112] Therefore, the embodiments of the present invention can solve the problem that the original Transformer framework only performs feature extraction on each token using a self-attention mechanism. By extracting features based on multi-scale grouping, the feature gaps between tokens to varying degrees are filled in, thereby effectively preserving the local features between voxels after voxelization and meeting the needs of practical use.

[0113] In step S103, obstacle information around the vehicle is detected based on fused features.

[0114] It is understood that, based on the features of the point cloud information obtained in step S102 above, the present invention can fuse the features to obtain fused features, and then detect obstacle information around the vehicle; wherein, the present invention can use at least one method to detect obstacle information around the vehicle, and there is no specific limitation thereto.

[0115] One possible approach is to detect obstacle information around a vehicle based on fused features, including: inputting fused features into a detection head, outputting the center position, size, and orientation information of the predicted bounding box of the obstacle, and generating obstacle information based on one or more of the center position, size, and orientation information of the predicted bounding box.

[0116] Understandably, during feature extraction, each token corresponds to a feature of a voxel. Before final detection, these features are transferred back to the BEV (Bird's Eye View). The original voxelized BEV view is usually quite large. To improve efficiency, this embodiment of the invention can add a learnable aggregation process before the final detection head, the principle of which is similar to the self-attention mechanism. Finally, the downsampled view is input into a general detection head to obtain the position, size, and orientation information of the detected obstacles.

[0117] Specifically: (1) The detection head configuration in this embodiment of the invention can be as follows: For the downsampled features already projected onto the BEV, this embodiment of the invention can input them into a standard detection head after a 3x3 convolution, thereby obtaining the center position, size, and orientation information of the obstacle prediction box. A standard detection head may include:

[0118] The heatmap of the detection box center position, which is a Gaussian normal distribution with the detection box center as the origin, or a heatmap with a certain normal distribution along the orientation, is supervised by FocalLoss; the regression values ​​of the length, width and height of the detection box, i.e. size prediction, are supervised by SmoothL1Loss; the regression value of the detection box orientation angle, i.e. heading prediction, is usually sin & cos regression, supervised by SmoothL1Loss.

[0119] (2) In this embodiment of the invention, before detecting obstacle information around the vehicle based on the fusion features, the method further includes: downsampling feature encoding of the fusion features to obtain the encoded fusion features.

[0120] Understandably, to avoid excessively large feature maps leading to low operating efficiency, the voxel size in the above embodiments should be as small as possible. Therefore, as Figure 5 As shown, after obtaining the temporal superposition features corresponding to each voxel, this embodiment of the invention can further use a learnable downsampling feature extraction method to aggregate the voxel neighborhood features before performing MSA encoding, that is:

[0121] feature_voxel=MSA(MaxPool(feature_voxels)),

[0122] The final feature length is also nL, and the position of the corresponding aggregation center is recorded.

[0123] (3) In this embodiment of the invention, before detecting obstacle information around the vehicle based on the fusion features, the method further includes: projecting a bird's-eye view onto the fusion features to obtain the projected fusion features.

[0124] It is understood that, through the steps of the above embodiments, the original BEV feature map of this invention changes in dimension from (H, W, C) to (H / / d, W / / d, nL), that is, H and W are downsampled by a factor of d, and C is further transformed into nL through feature extraction and temporal fusion. Based on the above embodiments, the position of the aggregation center is recorded, and each newly generated feature is back-projected onto the new BEV feature map as input for the subsequent static target detection head.

[0125] The obstacle detection method proposed according to embodiments of the present invention has at least the following advantages:

[0126] (1) The embodiments of the present invention can perform temporal fusion by aligning point clouds and encoding, so that temporal information can be preserved. Since the final generated features contain information from both the current time and the historical time, the temporal features will provide some completion for distant sparse or occluded targets. Therefore, even obstacles in extreme or special scenarios can be guided by historical information, increasing detection stability. Thus, accurate detection of obstacles can be achieved without relying on positioning information, getting rid of the limitations of positioning sensors and positioning accuracy. Furthermore, since it is free from positioning dependence, it can simultaneously meet the needs of online and offline applications and deployments, effectively meeting the needs of actual use and improving the user experience.

[0127] (2) The embodiments of the present invention can encode the temporal point cloud alignment in a learnable manner and adopt the corresponding feature fusion method so that the temporal information can also be preserved, providing guidance on historical information for obstacles in extreme or special scenarios and increasing detection stability;

[0128] (3) In this embodiment of the invention, a learnable downsampling feature extraction method can be further adopted to aggregate the voxel neighborhood features before encoding, further aligning the features so that each feature of the current frame is not only superimposed with the features of the same position in the historical frame, but is generated by taking the surrounding neighborhood features and aggregating them to generate the superimposed features of each voxel, making the features more complete and increasing the detection robustness.

[0129] (4) The embodiments of the present invention can adopt a multi-scale feature encoding method to group point cloud features into multi-scale features, encode the grouped features at each scale, and finally obtain the encoded features at each scale. Thus, the embodiments of the present invention can fill in feature loss of different degrees, give full play to the advantages of self-attention mechanism, and meet the needs of actual use.

[0130] (5) The present invention can use a method of grouping point cloud features using a preset window. By translating and / or scaling the preset window, grouped features of point cloud features at different scales can be obtained. Based on multi-scale grouping, feature extraction is performed, local features between voxels are preserved after voxelization, and the problem of local feature loss caused by encoder structure is made up to meet the needs of actual use.

[0131] (6) The present invention embodiment can preset the preset number of grouped features, and based on the number of grouped features, determine the number of times the preset window is translated and / or scaled, and then perform operations based on the number of times to obtain multi-scale features after the preset window changes accordingly. When the number of times reaches the target, the multi-scale feature grouping can be considered complete and the grouping operation ends; avoids repeated and redundant operations, the grouping results are more accurate and adaptable, and improves the reliability of multi-scale feature grouping.

[0132] (7) The embodiments of the present invention can obtain local information at different scales each time it is encoded, thereby compensating for the loss of local features caused by the encoder structure, giving full play to the advantages of the self-attention mechanism, and achieving the effect of taking into account both the global and local aspects.

[0133] (8) In this embodiment of the invention, a variable overlay frame rate can be input, as long as the number of overlays is less than the maximum length;

[0134] (9) The embodiments of the present invention can perform point cloud filtering, voxelization and feature processing, which shortens the encoding time on the one hand; and removes the dependence on intensity information on the other hand, thereby eliminating the problem of inconsistent intensity scale caused by different laser hardware.

[0135] (10) In this embodiment of the invention, a learnable aggregation process can be added before the final detection head to improve efficiency, shorten detection time, meet actual use needs, and enhance user experience;

[0136] (11) The embodiments of the present invention can record the location of the aggregation center, and project each newly generated feature back onto the new bird's-eye view feature map as the input of the subsequent static target detection head, thereby improving the detection efficiency and providing historical information guidance for obstacles in extreme or special scenarios;

[0137] (12) The embodiments of the present invention can make the fusion features not constrained by the field of view or viewpoint, nor restricted by the law of near-large and far-small, avoiding the high dependence on positioning information, ignoring the positioning or occlusion effects that may be caused by special scenarios such as underground parking lots, thus having a wider range of applications and improving the user experience.

[0138] Next, the obstacle detection device according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0139] Figure 6This is a block diagram of an obstacle detection device according to an embodiment of the present invention.

[0140] like Figure 6 As shown, the obstacle detection device 10 includes: an acquisition module 100, a fusion module 200, and a detection module 300.

[0141] Specifically, the acquisition module 100 is used to acquire point cloud information around the vehicle and extract multiple point cloud features from the point cloud information; the fusion module 200 is used to encode each point cloud feature in the multiple point cloud features to obtain encoded features, align the encoded features of the same point cloud feature at all times, and fuse the encoded features at all times after alignment to obtain fused features; the detection module 300 is used to detect obstacle information around the vehicle based on the fused features.

[0142] In this embodiment of the invention, the fusion module 200 is further configured to: set a self-learning matrix; use the self-learning matrix to learn the alignment relationship between point cloud features at different times; and use the alignment relationship to align the encoded features of the same point cloud feature at all times.

[0143] In this embodiment of the invention, the fusion module 200 is further configured to: extract the neighboring features around the encoded features at each time step; align the encoded features of the same point cloud feature at all times based on the neighboring features belonging to the encoded features of the same point cloud feature.

[0144] In this embodiment of the invention, the fusion module 200 is further configured to: perform multi-scale feature grouping on all point cloud features to obtain grouped features at multiple scales; and encode the grouped features at each scale to obtain encoded features at each scale.

[0145] In this embodiment of the invention, the fusion module 200 is further configured to: set a preset window of target size; group all point cloud features using the preset window to obtain grouped features at one scale; translate and / or scale the preset window, and group all point cloud features using the translated and / or scaled preset window to obtain grouped features at other scales.

[0146] In this embodiment of the invention, the fusion module 200 is further configured to: set a preset number of grouped features; set a target number of translations and / or scalings based on the preset number; and end multi-scale feature grouping when the actual number of translations and / or scalings reaches the target number.

[0147] In this embodiment of the invention, the fusion module 200 is further configured to: perform multi-attention encoding on the grouped features at each scale.

[0148] In this embodiment of the invention, the actual number of fused encoded features is less than the maximum number of fused features during the training phase.

[0149] In this embodiment of the invention, the acquisition module 100 is further configured to: traverse all point clouds in the point cloud information, filter point clouds that are not in the preset area; calculate the corresponding voxel coordinates based on the point cloud coordinates, and generate point cloud features based on the voxel coordinates.

[0150] In this embodiment of the invention, the detection module 300 is further configured to: input the fused features into the detection head, output the center position, size and orientation information of the predicted bounding box of the obstacle, and generate obstacle information based on one or more of the center position, size and orientation information of the predicted bounding box.

[0151] In this embodiment of the invention, the detection module 300 is further configured to: perform downsampling feature encoding on the fused features to obtain the encoded fused features.

[0152] In this embodiment of the invention, the detection module 300 is further configured to: project a bird's-eye view onto the fused features to obtain the projected fused features.

[0153] It should be noted that the foregoing explanation of the obstacle detection method embodiment also applies to the obstacle detection device of this embodiment, and will not be repeated here.

[0154] The obstacle detection device proposed in this embodiment of the invention can perform temporal fusion by aligning point clouds and encoding them, so that temporal information can be preserved. Since the final generated features contain information from both the current moment and the historical moment, the temporal features will provide some completion for distant, sparse, or occluded targets. Therefore, even obstacles in extreme or special scenarios can be guided by historical information, increasing detection stability. Thus, accurate obstacle detection can be achieved without relying on positioning information, freeing it from the limitations of positioning sensors and positioning accuracy. Furthermore, due to the independence from positioning dependence, it can simultaneously meet the needs of online and offline applications and deployments, effectively meeting the needs of actual use and improving the user experience.

[0155] Figure 7 This is a schematic diagram of a vehicle provided in an embodiment of the present invention. The vehicle may include:

[0156] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0157] When the processor 702 executes the program, it implements the obstacle detection method provided in the above embodiments.

[0158] Furthermore, the vehicle also includes:

[0159] Communication interface 703 is used for communication between memory 701 and processor 702.

[0160] The memory 701 is used to store computer programs that can run on the processor 702.

[0161] The memory 701 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0162] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0163] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0164] The processor 702 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0165] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the obstacle detection method described above.

[0166] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0167] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0168] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0169] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0170] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0171] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting obstacles, characterized in that, Includes the following steps: Acquire point cloud information around the vehicle and extract multiple point cloud features from the point cloud information; Encode each point cloud feature among the multiple point cloud features to obtain an encoded feature, align the encoded features of the same point cloud feature at all times, and fuse the aligned encoded features at all times to obtain a fused feature. Based on the fused features, obstacle information around the vehicle is detected; The encoded features of the same point cloud feature across all time points include: Set a self-learning matrix; The self-learning matrix is ​​used to learn the alignment relationship between point cloud features at different times, and the alignment relationship is used to align the encoded features of the same point cloud feature at all times. The process of encoding each point cloud feature from the plurality of point cloud features to obtain encoded features includes: Multi-scale feature grouping is performed on all point cloud features to obtain grouped features at multiple scales; Encode the grouped features at each scale to obtain the encoded features at each scale; The encoding of the grouped features at each scale includes: Multi-attention encoding is performed on the grouped features at each scale.

2. The obstacle detection method according to claim 1, characterized in that, The encoded features of the same point cloud feature across all time points include: Extract the neighborhood features surrounding the encoded features at each time step; Based on the coded features of the neighboring features belonging to the same point cloud feature, the coded features of the same point cloud feature are aligned at all times.

3. The obstacle detection method according to claim 1, characterized in that, The process of grouping all point cloud features into multiple scales to obtain grouped features includes: Sets the target window size; The preset window is used to group all point cloud features to obtain grouped features at one scale. The preset window is translated and / or scaled, and all point cloud features are grouped using the translated and / or scaled preset window to obtain grouped features at other scales.

4. The obstacle detection method according to claim 3, characterized in that, The method of grouping all point cloud features using a preset window after translation and / or scaling includes: Set the preset number of the grouping features; The target number of translations and / or scalings is set according to the preset number; The multi-scale feature grouping ends when the actual number of translations and / or scalings reaches the target number.

5. The obstacle detection method according to claim 1, characterized in that, The actual number of fused encoded features is less than the maximum number of fused features during the training phase.

6. The obstacle detection method according to claim 1, characterized in that, The extraction of multiple point cloud features from the point cloud information includes: Traverse all point clouds in the point cloud information and filter out point clouds that are not within the preset area; Calculate the corresponding voxel coordinates based on the point cloud coordinates, and generate point cloud features based on the voxel coordinates.

7. The obstacle detection method according to claim 1, characterized in that, The method of detecting obstacle information around the vehicle based on the fused features includes: The fused features are input into the detection head, which outputs the center position, size, and orientation information of the predicted bounding box of the obstacle. The obstacle information is generated based on one or more of the center position, size, and orientation information of the predicted bounding box.

8. The obstacle detection method according to claim 1, characterized in that, Before detecting obstacle information around the vehicle based on the fused features, the process also includes: The fused features are downsampled and encoded to obtain the encoded fused features.

9. The obstacle detection method according to claim 1 or 7, characterized in that, Before detecting obstacle information around the vehicle based on the fused features, the process also includes: The fused features are projected onto a bird's-eye view to obtain the projected fused features.

10. An obstacle detection device, characterized in that, include: The acquisition module is used to acquire point cloud information around the vehicle and extract multiple point cloud features from the point cloud information; The fusion module is used to encode each point cloud feature among the multiple point cloud features to obtain an encoded feature, align the encoded features of the same point cloud feature at all times, and fuse the aligned encoded features at all times to obtain a fused feature. The detection module is used to detect obstacle information around the vehicle based on the fused features; Based on the fused features, obstacle information around the vehicle is detected; The encoded features of the same point cloud feature across all time points include: Set a self-learning matrix; The self-learning matrix is ​​used to learn the alignment relationship between point cloud features at different times, and the alignment relationship is used to align the encoded features of the same point cloud feature at all times. The process of encoding each point cloud feature from the plurality of point cloud features to obtain encoded features includes: Multi-scale feature grouping is performed on all point cloud features to obtain grouped features at multiple scales; Encode the grouped features at each scale to obtain the encoded features at each scale; The encoding of the grouped features at each scale includes: Multi-attention encoding is performed on the grouped features at each scale.

11. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the obstacle detection method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the obstacle detection method as described in any one of claims 1-9.

Images