Target detection method and apparatus

Abstract

This application relates to a target detection method and apparatus. The method includes: voxelizing point cloud data acquired by a lidar according to polar coordinates to obtain processed data; extracting features from the processed data to obtain a first 2D feature map; aligning the first 2D feature map based on global information interaction to obtain a second 2D feature map; extracting 2D features from the second 2D feature map to obtain first 2D feature information; adjusting and aggregating the first 2D feature information based on geometric and instance-level information to obtain second 2D feature information; and performing target detection based on the second 2D feature information to obtain a detection result. This application enables global feature information interaction, reducing the computational load of global interaction and aligning feature information. After feature information extraction, geometric cues and object-level information can be introduced into the features to improve regression ability.

Description

Technical Field

[0001] This application relates to the fields of autonomous driving and driver assistance, and in particular to a target detection method and device. Background Technology

[0002] With economic development, the number of cars worldwide is constantly increasing, and the incidence of traffic accidents is also rising sharply, posing a huge threat to people's lives and property. Human factors are the main cause of traffic accidents, and how to reduce human error is an important issue in improving driving safety. Therefore, Advanced Driving Assistance Systems (ADAS) and Autonomous Driving Systems (ADS) have attracted the attention of major companies worldwide. Related companies have invested heavily in the research and development and deployment of related technologies. How to provide a more accurate target detection method is an urgent technical problem to be solved. Summary of the Invention

[0003] In view of this, a target detection method and device are proposed.

[0004] In a first aspect, embodiments of this application provide a target detection method, the method comprising:

[0005] The point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain the processed data;

[0006] Feature extraction is performed on the processed data to obtain a first 2D feature map;

[0007] The second 2D feature map is obtained by aligning the first 2D feature map based on global information interaction.

[0008] 2D feature extraction is performed on the second 2D feature map to obtain the first 2D feature information;

[0009] Based on the geometric information and instance-level information of the first 2D feature information, the first 2D feature information is adjusted and aggregated to obtain the second 2D feature information;

[0010] Target detection is performed based on the second 2D feature information to obtain the detection result.

[0011] The target detection method provided in the first aspect conforms to the characteristics of LiDAR scanning. The voxel density distribution of the processed data after voxelization is more uniform, and it can focus on nearby objects; that is, the detection performance of nearby objects degrades slowly when the resolution is reduced. The interaction of global feature information reduces the computational cost of global interaction and helps improve deformation problems, allowing feature information to align. After feature information extraction, the first 2D feature information is adjusted and aggregated based on its geometric and instance-level information, which can introduce geometric cues and object-level information into the features, improving regression ability.

[0012] In one possible implementation, the point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain processed data, including:

[0013] The point cloud data is divided into 3D spaces according to distance, angle, and height to obtain processed data.

[0014] According to the first aspect or one possible implementation of the first aspect, it can be applied to streaming detection and can meet the characteristics of LiDAR scanning. For example, the voxel density of the processed data obtained after voxelization can be more uniformly distributed, and it can focus on nearby objects. That is, when the resolution is reduced, the detection performance of nearby objects decreases slowly.

[0015] In one possible implementation, the first 2D feature map is aligned based on global information interaction to obtain a second 2D feature map, including:

[0016] Based on nearest-neighbor nonmaximum suppression, multiple first key points of column features in each column of the first 2D feature map are determined;

[0017] Based on the first attention mechanism, the column features of each column are compressed into the feature information of each first key point in the column to obtain multiple second key points. In the first attention mechanism, the features of each first key point are used as requests and the column features of the column are used as keywords and values.

[0018] The second key points corresponding to each column are divided into multiple non-overlapping key point windows, and the feature information of each second key point in each key point window is interacted based on the second attention mechanism to obtain multiple third key points. In the second attention mechanism, the features of the second key points are used as requests, keywords and values.

[0019] Based on the third attention mechanism, the feature information of the third key point in each column feature is diffused to the column feature of the column, resulting in a second 2D feature map.

[0020] Based on the first aspect or one possible implementation thereof, selecting key points to represent column features for global feature information interaction can reduce the computational cost of global interaction. Information interaction helps improve deformation problems, and by using window shifting to expand the receptive field, feature information at both near and far distances can be aligned as much as possible.

[0021] In one possible implementation, the first 2D feature information is adjusted and aggregated based on the geometric information and instance-level information of the first 2D feature information to obtain second 2D feature information, including:

[0022] The segmentation information of the first 2D feature information is predicted using the segmentation branch, and the center offset of the first 2D feature information is predicted using the regression branch.

[0023] Based on the segmentation information, the center offset, and the position code corresponding to the first 2D feature information, the geometric information corresponding to the first 2D feature information is determined.

[0024] The geometric information and the first 2D feature information are divided into multiple non-overlapping information windows, and the geometric information and the first 2D feature information in each information window are interacted based on a fourth attention mechanism to obtain the second 2D feature information. The fourth attention mechanism is a self-attention mechanism.

[0025] According to the first aspect or one possible implementation of the first aspect, the first 2D feature information is adjusted and aggregated based on the geometric information and instance-level information of the first 2D feature information, which can introduce geometric clues and object-level information into the features and improve the regression ability.

[0026] Secondly, embodiments of this application provide a target detection device, the device comprising:

[0027] The global feature alignment module is used to perform voxelization on the point cloud data acquired by the lidar according to polar coordinates to obtain processed data; to extract features from the processed data to obtain a first 2D feature map; and to align the first 2D feature map based on global information interaction to obtain a second 2D feature map.

[0028] The 2D feature extraction module is used to extract 2D features from the second 2D feature map to obtain the first 2D feature information;

[0029] A geometry-aware detection head is used to adjust and aggregate the first 2D feature information based on the geometric information and instance-level information of the first 2D feature information to obtain the second 2D feature information.

[0030] The target detection module is used to perform target detection based on the second 2D feature information and obtain the detection result.

[0031] In one possible implementation, the point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain processed data, including:

[0032] The point cloud data is divided into 3D spaces according to distance, angle, and height to obtain processed data.

[0033] In one possible implementation, the first 2D feature map is aligned based on global information interaction to obtain a second 2D feature map, including:

[0034] Based on nearest-neighbor nonmaximum suppression, multiple first key points of column features in each column of the first 2D feature map are determined;

[0035] Based on the first attention mechanism, the column features of each column are compressed into the feature information of each first key point in the column to obtain multiple second key points. In the first attention mechanism, the features of each first key point are used as requests and the column features of the column are used as keywords and values.

[0036] The second key points corresponding to each column are divided into multiple non-overlapping key point windows, and the feature information of each second key point in each key point window is interacted based on the second attention mechanism to obtain multiple third key points. In the second attention mechanism, the features of the second key points are used as requests, keywords and values.

[0037] Based on the third attention mechanism, the feature information of the third key point in each column feature is diffused to the column feature of the column, resulting in a second 2D feature map.

[0038] In one possible implementation, the first 2D feature information is adjusted and aggregated based on the geometric information and instance-level information of the first 2D feature information to obtain second 2D feature information, including:

[0039] The segmentation information of the first 2D feature information is predicted using the segmentation branch, and the center offset of the first 2D feature information is predicted using the regression branch.

[0040] Based on the segmentation information, the center offset, and the position code corresponding to the first 2D feature information, the geometric information corresponding to the first 2D feature information is determined.

[0041] The geometric information and the first 2D feature information are divided into multiple non-overlapping information windows, and the geometric information and the first 2D feature information in each information window are interacted based on a fourth attention mechanism to obtain the second 2D feature information. The fourth attention mechanism is a self-attention mechanism.

[0042] The beneficial effects of the target detection device provided in the second aspect and various possible implementations of the second aspect are the same as those of the target detection method provided in the first aspect and various possible implementations of the first aspect. To avoid redundancy, they will not be described in detail here.

[0043] Thirdly, embodiments of this application provide a target detection device, comprising:

[0044] processor;

[0045] Memory used to store processor-executable instructions;

[0046] The processor is configured to implement one or more of the target detection methods described above, or multiple possible implementations of the first aspect, when executing the instructions.

[0047] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the target detection methods described in the first aspect or various possible implementations of the first aspect.

[0048] Fifthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes one or more of the target detection methods described in the first aspect or various possible implementations of the first aspect.

[0049] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0050] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0051] Figure 1 A schematic diagram of an autonomous driving system in related technologies is shown.

[0052] Figure 2 A schematic diagram of the model framework provided by related technology is shown.

[0053] Figure 3 The graphs showing the performance of the related technology at different resolutions are illustrated.

[0054] Figure 4 A schematic diagram of the model framework provided by related technology 2 is shown.

[0055] Figure 5a A schematic diagram of the point cloud distribution after voxelization based on the Cartesian coordinate system is shown.

[0056] Figure 5b A schematic diagram of the point cloud distribution after voxelization based on polar coordinates is shown.

[0057] Figure 6a A schematic diagram of the delay of a full-scan lidar is shown.

[0058] Figure 6b A schematic diagram of the delay of a quarter-scan lidar is shown.

[0059] Figure 7a A schematic diagram of flow cytometry detection based on a Cartesian coordinate system is shown.

[0060] Figure 7b A schematic diagram of flow cytometry detection based on polar coordinates is shown.

[0061] Figure 8a A schematic diagram of the characteristic deformations in polar coordinate system voxelization is shown.

[0062] Figure 8b A schematic diagram of the characteristic deformations in polar coordinate system voxelization is shown.

[0063] Figure 9a A flowchart of a target detection method according to an embodiment of this application is shown.

[0064] Figure 9b A schematic flowchart of a target detection method according to an embodiment of this application is shown.

[0065] Figure 10 A flowchart illustrating step S103 of a target detection method according to an embodiment of this application is shown.

[0066] Figure 11 A flowchart illustrating step S105 of a target detection method according to an embodiment of this application is shown.

[0067] Figure 12a A schematic diagram illustrating the detection performance of a target detection method according to an embodiment of this application at different resolutions is shown.

[0068] Figure 12bA schematic diagram illustrating the detection performance of a target detection method according to an embodiment of this application at different resolutions is shown.

[0069] Figure 13 A schematic diagram of the structure of an electronic device 100 is shown. Detailed Implementation

[0070] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0071] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0072] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0073] Figure 1 A schematic diagram of an autonomous driving system in related technologies is shown. For example... Figure 1 As shown, autonomous or assisted driving systems mainly consist of five components: perception data acquisition, target detection, target tracking, multi-sensor fusion, and control.

[0074] Perception data acquisition can include collecting road surface data through various devices such as cameras, lasers, and iBeacons. Laser scanning typically collects external environmental feedback at a frequency of 10 FPS. Camera data acquisition generally collects external scene information at a rate of 25 or 30 FPS. Target detection can include detecting dynamic targets and obstacles (pedestrians, vehicles) as well as static targets (traffic signs, lane lines, freespace, etc.). Target tracking can smooth detection results and can also be used for speed measurement and prediction of target trajectories. Multi-sensor fusion allows each sensor to play its role, achieving a fusion result superior to any single sensor's result. Planning and control can utilize the comprehensive obstacle information output by multi-sensor fusion to make reasonable path planning and control the vehicle's driving status. This part decides how the vehicle should move and is the control center of the autonomous vehicle.

[0075] The object detection method provided in this application can be applied to the perception data acquisition part of autonomous driving or assisted driving related products, and can also be applied to object detection in other scenarios. This application does not limit its application in this regard. The following uses autonomous driving as an example to illustrate the problems existing in object detection in related technologies one and two, as well as the object detection method and device provided in this application.

[0076] 3D object detection technology is a crucial component of autonomous driving perception data acquisition systems. With decreasing costs, LiDAR sensors are now widely used in autonomous vehicles to obtain precise positioning information. Unlike 2D images acquired by camera sensors, the point clouds obtained from LiDAR scans are disordered and sparse; therefore, selecting effective representation methods for point cloud data is one of the key challenges in point cloud 3D object detection.

[0077] Related Technology 1:

[0078] Figure 2 A schematic diagram of the model framework provided by related technology is shown. For example... Figure 2 As shown in the first related technology, a model framework based on CenterPoint is provided. First, the 3D point cloud is voxelized in the Cartesian coordinate system. Then, features are extracted through a 3D backbone network. The 3D features are transformed into a bird's-eye view. After 2D feature extraction and detection head prediction, the 3D bounding box is predicted. Figure 3 The graphs showing the performance of the related technology at different resolutions are illustrated. Figure 3 As shown, the related technology has the following problems: when using the Cartesian coordinate system to represent point cloud data, the voxel density distribution after voxelization is uneven and the voxel size will increase. That is, when the resolution is reduced, the detection performance shows an exponential downward trend.

[0079] Related Technology 2:

[0080] Figure 4 A schematic diagram of the model framework provided by related technology two is shown. For example... Figure 4 As shown in the second related technology, a model framework based on the Polarstream method is provided, employing a streaming detection scheme and a polar coordinate system. To address the problems of the Cartesian coordinate system, the second related technology uses a polar coordinate system to voxelize the 3D point cloud. This is because the polar coordinate system can effectively alleviate the problems of the Cartesian system. The polar coordinate system divides the 3D space according to distance and angle, which is more in line with the characteristics of the LiDAR scanning environment. The voxel density distribution is more uniform, and this division method focuses more on nearby objects. When the resolution is reduced, the detection performance of nearby objects decreases slowly. Furthermore, the representation obtained based on the polar coordinate system voxelization is more suitable for the streaming detection scheme.

[0081] Figure 5a A schematic diagram of the point cloud distribution after voxelization based on the Cartesian coordinate system is shown. Figure 5b A schematic diagram of the point cloud distribution after voxelization based on polar coordinates is shown. (Example) Figure 5a and Figure 5b As shown, since point cloud distribution typically has more near-field points and fewer far-field points, uniform voxel partitioning based on Cartesian coordinates results in a more uneven voxel density distribution than that based on polar coordinates. Furthermore, the detection performance of voxelization based on Cartesian coordinates degrades significantly when the resolution is reduced. Therefore, voxelization based on polar coordinates can alleviate the significant performance degradation problem of voxelization based on Cartesian coordinates when the resolution is reduced.

[0082] Figure 6a A schematic diagram of the latency of a full-sweep lidar is shown. Figure 6b A schematic diagram of the latency of a 1 / 4-sweep lidar is shown. Figure 6a This is a non-flow detection method; the lidar scans one full circle to obtain one frame of point cloud data, resulting in a relatively long system latency. For example... Figure 6b This is a type of streaming detection. Streaming detection schemes assume that the point cloud acquired by commonly used rotating lidar is actually a streaming data (i.e., data is acquired while scanning), thus eliminating the concept of point cloud data "frames" and dividing the point cloud data into multiple slices. Figure 6b The system processes a quarter of the data obtained from the LiDAR scan, and then processes that data sequentially. After the point cloud data of each slice is scanned, this portion of data is processed first for model inference. Based on this setup, the system latency can be significantly reduced; ideally, the system latency can be directly divided by the number of slices. Figure 7a A schematic diagram of flow cytometry detection based on a Cartesian coordinate system is shown. Figure 7b A schematic diagram of flow cytometry detection based on polar coordinates is shown. Figure 7a As shown, because the data is divided into sectors, using the commonly used Cartesian coordinate system will result in wasted memory and calculation of blank areas. Figure 7b As shown, the polar coordinate system is more suitable for streaming detection by rotating lidar, and will not cause memory waste or calculation of blank areas.

[0083] like Figure 4As shown, since Related Technique 2 uses a polar coordinate system to represent the sliced ​​point cloud data, it proposes an interpolation strategy for Cartesian positions and distance-level hierarchical convolution and normalization to address the deformation problem after voxelization based on polar coordinates. The interpolation strategy is used in the classification head, encoding the Cartesian positions corresponding to the feature maps and then performing a linear transformation on the 2D feature maps. Distance-level hierarchical convolution and normalization use different convolution kernels to process features at different distances. Related Technique 2 improves upon the feature deformation problem, but it doesn't solve it perfectly. A major problem with polar coordinate detectors is feature deformation; due to uneven voxel division, objects at different distances deform differently. Figure 8a , Figure 8b A schematic diagram of the characteristic deformations in polar coordinate system voxelization is shown. For example... Figure 8a , Figure 8b As shown, the deformation of the same object varies at different distances and orientations, and the number of pixels it occupies also differs. This makes translation-invariant CNNs difficult to handle such non-rectangular features, leading to global feature misalignment and regression difficulties. Therefore, related technique two has the following problems: the interpolation strategy performs a linear transformation on the 2D feature map, while feature deformation is a non-linear transformation and cannot completely solve the deformation problem. In addition, the method of using different convolution kernels to process features at different distances is somewhat cumbersome, and features at the boundaries may be discontinuous.

[0084] To address the aforementioned technical problems, this application provides a target detection method and apparatus. Using a polar coordinate system aligns with the characteristics of LiDAR scanning, the voxel density distribution of the processed data after voxelization is more uniform, and it allows for focus on nearby objects; that is, the detection performance of nearby objects degrades slowly when the resolution is reduced. Key points are selected to represent column features for global feature information interaction, reducing the computational load of global interaction. Information interaction helps improve deformation problems and allows feature information alignment. After feature information extraction, the first 2D feature information is adjusted and aggregated based on its geometric and instance-level information, introducing geometric cues and object-level information into the features, thus improving regression capabilities.

[0085] Figure 9a A flowchart of a target detection method according to an embodiment of this application is shown. Figure 9b A schematic flowchart of a target detection method according to an embodiment of this application is shown. Figure 9a , Figure 9b As shown, the target detection method provided in this application includes steps S101-S106 and can be applied to electronic devices.

[0086] In step S101, the point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain the processed data.

[0087] In one possible implementation, step S101 may include: dividing the point cloud data into 3D spaces according to distance, angle, and height to obtain processed data.

[0088] First, point cloud data can be acquired using a lidar system. This point cloud data can be a single frame of data from a lidar scan, or it can be slices of point cloud data obtained through streaming detection. Streaming detection can process the already scanned point cloud data simultaneously, without requiring a full lidar scan to obtain a single frame. Then, the point cloud data is voxelized using polar coordinates to obtain the processed data. For example, it can be... Figure 4 The 3D point cloud space (i.e., point cloud data) of the cylinder is divided into multiple polar cylinders (i.e., processed data). Using polar coordinates to voxelize point cloud data is suitable for streaming detection, and the use of polar coordinates can conform to the characteristics of LiDAR scanning. For example, the voxel density of the processed data obtained after voxelization can be more uniformly distributed, and it can focus on nearby objects. That is, when the resolution is reduced, the detection performance of nearby objects degrades slowly.

[0089] In step S102, feature extraction is performed on the processed data to obtain a first 2D feature map.

[0090] First, 3D features can be extracted from the processed data obtained in step S101 using 3D sparse convolution to obtain 3D point cloud features. Then, the height dimension (i.e., the height of the pole) of the 3D point cloud features can be merged with the feature dimension obtained from the 3D feature extraction, that is, the 3D point cloud features can be converted into a first 2D feature map from a bird's-eye view. At this time, the 3D point cloud features may be deformed during the conversion into the first 2D feature map. The deformation may affect the accuracy of 2D feature extraction. Therefore, after the conversion into the first 2D feature map and before 2D feature extraction, information interaction can be added to align global features and improve the deformation problem.

[0091] In step S103, the first 2D feature map is aligned based on global information interaction to obtain the second 2D feature map.

[0092] Figure 10 A flowchart illustrating step S103 of a target detection method according to an embodiment of this application is shown. Figure 10 As shown, step S103 may include: an information compression step, an information interaction step, and an information diffusion step. The following is combined with... Figure 10 The steps of information compression, information interaction, and information dissemination are explained.

[0093] In the "information compression step", multiple first key points of column features of each column in the first 2D feature map can be determined based on nearest neighbor nonmaximum suppression.

[0094] First, based on the column information of each column of the first 2D feature map, multiple key points with the largest eigenvalues ​​in each column can be determined. Then, a nearest neighbor non-maximum suppression operation can be performed radially. This involves initially selecting a first key point with the largest eigenvalue, but subsequently avoiding selecting any first key points adjacent to it. This ensures that the final selected first key points with the largest eigenvalues ​​are not adjacent to each other, preventing the selection of first key points on different targets and guaranteeing that each first key point can be selected for different targets.

[0095] like Figure 10 As shown, in the "information interaction step," the column features of each column can be compressed into the feature information of each first key point in the column based on the first attention mechanism, resulting in multiple second key points. In the first attention mechanism, the features of each first key point are used as the query, and the column features of the column are used as the key and value.

[0096] Specifically, column features of each column can be determined based on column information of each column in the first 2D feature map. For each column, a first attention mechanism can be used to condense and merge the column features of that column onto the plurality of first key points, resulting in a plurality of second key points. The feature information of each second key point includes the feature information of the corresponding first key point and the information that the column features of the column in which it belongs have been condensed and merged into that second key point. The number of first key points to be selected for each column can be set according to actual needs, and this application does not impose any restrictions on this. For example, such as Figure 10 As shown, you can select the two first key points with the largest feature values ​​in each column, and then merge the column features of each column into the two first key points of that column.

[0097] Where, assume f i1 ∈R N×C f represents the feature information of the first key point in the i-th column. i ′∈R A×C For the column feature of column i, the first attention mechanism can be expressed as the following formula:

[0098] Q i1 =f i1 W q1 ,K i1 =f′ i W k1 V i1 =f′ i W v1 ,

[0099]

[0100] Among them, Q i1For the request corresponding to the i-th column, K i1 V is the keyword corresponding to the i-th column. i1 f is the value corresponding to the i-th column. i1 f represents the feature information of the first key point in the i-th column. i ′ represents the column feature of the i-th column, W q1 W k1 W v1 These are linear mappings corresponding to queries, keys, and values, respectively. i2 This refers to the feature information of the second key point in the i-th column. It can be based on W... q1 W k1 W v1 f i1 and f′ i Calculate Q i1 K i1 V i1 E(p) is a relative positional code, which can be represented as:

[0101] E(p) = ReLU((p) i1 -p′ i )×W pos )

[0102] in, Represents dimensions, This represents the regularity ratio corresponding to the i-th column. This represents the similarity between the regularization ratio corresponding to the i-th column and the relative position code E(p). i1 p represents the coordinates of the first key point in the i-th column in the Cartesian coordinate system. i ′ represents the coordinate position of the column feature in the i-th column in the polar coordinate system. W pos This represents a linear transformation of the coordinate positions. ReLU is a commonly used activation function in artificial neural networks, often referring to the ramp function in mathematics. It is used to output the nonlinear result of a neuron from a previous layer of the neural network after a linear transformation to the next layer or as the output of the entire neural network. By calculating the column features corresponding to each first keypoint using the aforementioned first attention mechanism, the column features of the column containing the first keypoint can be merged into the first keypoint, resulting in f. i2 That is, to obtain each second keypoint f containing the column features of each corresponding column. i2 .

[0103] like Figure 10As shown, in the "information interaction step": the second key points corresponding to each column can be divided into multiple non-overlapping key point windows, and information interaction is performed on the feature information of each second key point in each key point window based on the second attention mechanism to obtain multiple third key points. In the second attention mechanism, the features of the second key points are used as requests, keywords, and values.

[0104] Following the "information interaction step," multiple second keypoints corresponding to the point cloud data are obtained. Based on a pre-set keypoint window size, these second keypoints are divided into multiple non-overlapping keypoint windows. A second attention mechanism is then used to interact with the feature information of the second keypoints within each keypoint window. This second attention mechanism can be a self-attention mechanism, used to determine the correlation between different parts of the entire input. Figure 5 shows the feature deformation after voxelization in polar coordinates. The feature deformation is greater near the azimuth angle region. Therefore, the aforementioned information interaction step can be added in the azimuth angle region to introduce a sufficient receptive field, thereby aiding in feature alignment.

[0105] Where, assume f i2 ∈R N×C f′ represents the feature information of the second key point in the i-th column. i ∈R A×C For the column feature of column i, the second attention mechanism can be expressed as the following formula:

[0106] Q i2 =f i2 W q2 ,K i2 =f′ i W k2 V i2 =f′ i W v2 ,

[0107]

[0108] Among them, Q i2 For the request corresponding to the i-th column, K i2 V is the keyword corresponding to the i-th column. i2 f is the value corresponding to the i-th column. i2 f′ represents the feature information of the second keypoint in the i-th column. i For the column feature of column i, W q2 W k2 W v2 These are linear mappings corresponding to requests, keywords, and values, respectively. i3 This refers to the feature information of the third key point in the i-th column. It can be based on W... q2 W k2 Wv2 f i2 and f′ i Q can be calculated i2 K i2 V i2 E(p) is a relative positional code, which can be represented as:

[0109] E(p) = ReLU((p) i2 -p′ i )×W pos )

[0110] in, Represents dimensions, This represents the regularity ratio corresponding to the i-th column. This represents the similarity between the regularization ratio and the relative positional encoding E(p) corresponding to the i-th column. i2 p represents the coordinates of the feature information of the second key point in the i-th column in the Cartesian coordinate system. i ′ represents the coordinate position of the column feature in the i-th column in the polar coordinate system. W pos This represents a linear transformation of the coordinate positions. ReLU is a commonly used activation function in artificial neural networks, often referring to the ramp function in mathematics. It is used to output the nonlinear result of a neuron from a previous layer of the neural network after a linear transformation to the next layer or as the output of the entire neural network. The feature information of each second keypoint is calculated using the aforementioned second attention mechanism. Each second keypoint can interact with each other, resulting in f. i3 That is, obtaining the third keypoint f, which has interacted with the feature information of other second keypoints. i3 .

[0111] In one embodiment of this application, the number of times the "information interaction step" is executed can be set as needed. The size of the keypoint window can be the same or different in different execution processes, and this application does not impose any restrictions on this. For example, the above "information interaction step" can be repeated twice. In the second information interaction step, multiple third keypoints obtained in the first information interaction step can be divided into several non-overlapping keypoint windows as new second keypoints. Here, the division method of the keypoint windows can be different from the division method of the first information interaction step to expand the receptive field. This method can be called window shifting. Then, a second attention mechanism can be used to perform information interaction steps on the second keypoints of the information in each of the new keypoint windows. Each new second keypoint in the new keypoint window can interact with each other's feature information to obtain third keypoints that have interacted with the feature information of other new second keypoints.

[0112] like Figure 10As shown, in the "information diffusion step", the feature information of the third key point in each column feature can be diffused to the column feature of the column based on the third attention mechanism to obtain the second 2D feature map.

[0113] In this process, the feature information from the key points can be diffused back to the corresponding columns. In one embodiment of this application, a third attention mechanism is used to diffuse the feature information from each of the third key points obtained in the above steps back to the corresponding columns, resulting in a second 2D feature map after global feature alignment.

[0114] The attention mechanism parameters in the third attention mechanism are opposite to those in the first attention mechanism. Specifically, in the first attention mechanism, the key point is the query, and the column features are the key and value. In the third attention mechanism, the key point is the key and value, and the column feature is the query. Aside from the different numerical values ​​used in the formula, the third attention mechanism is structurally similar to the first attention mechanism, and will not be elaborated further here. Thus, each column receives feature information calculated and diffused back from the third key point through the third attention mechanism, interacting with other surrounding column features, and globally aligned, forming a second 2D feature map. In this application, selecting key points to represent column features for global feature information interaction reduces the computational load of global interaction. Information interaction at the azimuth angle helps improve deformation problems, and the window shifting method expands the receptive field, maximizing the alignment of feature information at both near and far distances.

[0115] In step S104, 2D feature extraction is performed on the second 2D feature map to obtain the first 2D feature information.

[0116] The second 2D feature map obtained through the above-mentioned global information interaction alignment process can solve the feature deformation problem of the first 2D feature map. Then, when performing 2D feature extraction on the second 2D feature map, more accurate 2D feature information can be obtained.

[0117] In step S105, the first 2D feature information is adjusted and aggregated based on the geometric information and instance-level information of the first 2D feature information to obtain the second 2D feature information.

[0118] Figure 11 A flowchart illustrating step S105 of a target detection method according to an embodiment of this application is shown. Figure 11 As shown, step S105 may include: geometry-aware prediction and geometry-aware aggregation. The following is combined with... Figure 11 This section explains geometric perception prediction and geometric perception aggregation.

[0119] In "geometric perception prediction", the segmentation information and center offset of the first 2D feature information can be predicted.

[0120] After obtaining the first 2D feature information from the second 2D feature map through 2D feature extraction, the first 2D feature information can be input into the geometry-aware detection head. The geometry-aware detection head can adjust and aggregate the first 2D feature information based on its geometric and instance-level information to introduce geometric cues and object-level information into the features, thereby improving regression capabilities. The geometry-aware detection head may include auxiliary branches, which may include segmentation branches and regression branches. The segmentation branch can predict segmentation information F in the second 2D feature map based on the first 2D feature information. seg The segmentation information may include the category of each pixel. The regression branch can predict the center offset F of the second 2D feature map based on the first 2D feature information. offset The center offset can include the offset of the center point of the object in the second 2D feature map. The two branches can be optimized using focal loss (a loss function to address the problem of extreme imbalance between positive and negative samples in object detection) and smooth L1 loss (a loss function to address gradient explosion at outliers), respectively.

[0121] In "Geometric Aware Aggregation," the segmentation information and center offset obtained from "Geometric Aware Prediction" can be used as prediction information. Based on the prediction information and the position encoding corresponding to the first 2D feature information, the geometric information corresponding to the first 2D feature information is determined. The position encoding represents the relative position of each pixel between Cartesian and polar coordinate systems.

[0122] Among these, geometric information, i.e., fused segmentation information F, can be determined first. seg Center offset F offset and position code F pos As geometric information F geo It can be expressed as the following formula:

[0123] F geo =MLP(Cat([F seg ,F offset ,F pos ]))

[0124] MLP stands for Multilayer Perceptron, which consists of multiple perceptrons interconnected and has multiple layers. It can be regarded as the basic form of neural network and has a strong fitting ability.

[0125] In one possible implementation, a "step similar to the information interaction step in step S103" can be included. Based on a pre-set information window size, the geometric information and the first 2D feature information can be divided into multiple non-overlapping information windows. Then, based on a fourth attention mechanism, information interaction is performed between the geometric information and the first 2D feature information in each information window to obtain the second 2D feature information. The fourth attention mechanism can be a self-attention mechanism.

[0126] Among them, geometric information F geo and the first 2D feature information F neck The information is divided into non-overlapping information windows. A fourth attention mechanism is used to interact the geometric information and the first 2D feature information of each information window to obtain the second 2D feature information. The second 2D feature information includes information exchanged from geometric information.

[0127] The fourth attention mechanism can be a self-attention mechanism, which allows information interaction within each information window. For example, for the j-th information window, it can be represented by the following formula:

[0128]

[0129]

[0130]

[0131]

[0132] Among them, Q j For the request corresponding to column j, K j It is the keyword corresponding to column j, V j It is the value corresponding to column j. For the first 2D feature information in the j-th column, For the geometric information of the j-th column, W q2 W k2 W v2 These are linear mappings between requests, keywords, and values, respectively. This can be done based on W. q W k W v , Q can be calculated j K j V j . Represents dimensions, This represents the regularity ratio corresponding to the j-th column. This represents the similarity value of the regularization ratio corresponding to the j-th column. The geometric information and the first 2D feature information of each information window can be calculated using the fourth attention mechanism mentioned above. The geometric information can be fused into the first 2D feature information to obtain the second 2D feature information.

[0133] Similar to the global information interaction alignment, the information interaction between geometric information and the first 2D feature information in each information window can be repeated multiple times in the detection head. The size of the information window can be the same or different in different execution processes, and this application does not impose any restrictions on this. For example, the above information interaction can be repeated twice. In the second information interaction, the geometric information in each column and the 2D feature information obtained from the first interaction can be divided into multiple non-overlapping information windows that are different from the first information interaction to expand the receptive field. Then, a fourth attention mechanism can be used to continue to fuse geometric information into the 2D feature information obtained from the first interaction of each information window to obtain the second 2D feature information with sufficient interaction.

[0134] In step S106, target detection is performed based on the second 2D feature information to obtain the detection result.

[0135] The second 2D feature information obtained after the above information interaction can be input into the key point-based detection head in the CenterPoint method, and the classification and regression results are obtained after passing through the segmentation branch and the regression branch respectively.

[0136] In one embodiment of this application, comparisons were made with the CenterPoint and PolarStream methods on the publicly available datasets Waymo and Once, as shown in Tables 1 and 2, respectively. On the Waymo dataset, our method outperforms the CenterPoint 3D detector in Cartesian coordinates. Due to feature deformation, directly transforming the center point to polar coordinates results in a significant performance degradation, while our method achieves significant gains of 4.93% L2 mAP and 4.02% L2 mAP compared to the CenterPoint and PolarStream methods, respectively. Similar results were obtained on the Once dataset, where our method achieves a 2.43% mAP improvement compared to the CenterPoint 3D detector in Cartesian coordinates, and also shows significant gains compared to the CenterPoint and PolarStream methods in polar coordinates.

[0137] Table 1. 3D Detection Results on the Waymo Dataset

[0138]

[0139] Table 1 shows the 3D detection results on the Waymo dataset, listing four methods: the first row uses the CenterPoint method in Cartesian coordinates, the second row uses the CenterPoint method in polar coordinates, the PolarStream method, and the object detection method (PARTNER) of this application. The input to the backbone network is the voxel result, using different coordinate systems; the PolarStream method and the object detection method of this application use polar coordinates. Taking the first row as an example, using the CenterPoint method, voxelization is performed based on Cartesian coordinates. In Vehicle LEVEL 1 detection, the 3D object detection model achieves a detection metric (mAP) of 75.58% and an evaluation metric (APH) of 75.01% on the Waymo dataset. In Vehicle Level 2 (VLE), the 3D object detection model achieved a detection metric (mAP) rate of 67.00% and an evaluation metric (APH) rate of 66.52% on the Waymo dataset.

[0140] Table 2. 3D Detection Results of the Once Dataset

[0141]

[0142] Table 2 shows the 3D detection results of the Once dataset, listing four methods: the CenterPoint method based on Cartesian coordinates in the first row, the CenterPoint method based on polar coordinates in the second row, the PolarStream method, and the object detection method (PARTNER) of this application. The detected targets included vehicles, pedestrians, and cyclists. The overall detection accuracy, detection accuracy at distances of 0-30 meters, 30-50 meters, and 50 meters and further were obtained.

[0143] Figure 12a , Figure 12b A schematic diagram illustrating the detection performance of a target detection method according to an embodiment of this application at different resolutions is shown. Figure 12a , Figure 12bThe CenterPoint method in Cartesian and polar coordinate systems was compared, and the results of the method provided in this application at different resolutions were presented. It can be seen that as the resolution decreases, the Cartesian detector exhibits an exponential decline, while the polar detector shows a linear decline. This is because the polar coordinate system divides the nearby scene into finer subdivisions, and the performance of nearby objects is still well maintained when the resolution decreases. The distortion problem of the polar coordinate system leads to poor detector performance at high resolutions, while the method provided in this application consistently maintains relatively optimal performance. This fully demonstrates the superiority of the polar coordinate detector at low resolutions.

[0144] Furthermore, the polar coordinate system provides a more suitable representation in streaming detection schemes. As shown in Table 3, under the streaming detection scheme, this application outperforms the PolarStream method under different regions, and the detection performance after slicing even has a certain gain.

[0145] Table 3 Performance comparison of Polar series under flow cytometry detection schemes

[0146]

[0147] As can be seen, global information interaction alignment (i.e., step S103) can alleviate the feature deformation problem after voxelization based on polar coordinates, resulting in better 2D feature map representation. As shown in Table 4, adding global information interaction alignment to the polar coordinate detector baseline can achieve a performance gain of 2.27% L2 mAPH. Adjusting and aggregating 2D feature information based on geometric and instance-level information in the detection head is beneficial to improving regression ability. As shown in Table 4, the geometric perception detection head of the present invention can further achieve a performance gain of 2.85% L2 mAPH.

[0148] Table 4 shows the gains in detection performance due to module design.

[0149]

[0150] Embodiments of this application provide a target detection device, the device comprising: a global feature alignment module, a 2D feature extraction module, a geometric perception detection head, and a target detection module.

[0151] A global feature alignment module is used to voxelize the point cloud data acquired by the LiDAR according to polar coordinates to obtain processed data. Feature extraction is performed on the processed data to obtain a first 2D feature map. The first 2D feature map is then aligned based on global information interaction to obtain a second 2D feature map.

[0152] The 2D feature extraction module is used to extract 2D features from the second 2D feature map to obtain the first 2D feature information.

[0153] A geometry-aware detection head is used to adjust and aggregate the first 2D feature information based on the geometric information and instance-level information of the first 2D feature information to obtain the second 2D feature information.

[0154] The target detection module is used to perform target detection based on the second 2D feature information and obtain the detection result.

[0155] In one possible implementation, the point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain processed data, including:

[0156] The point cloud data is divided into 3D spaces according to distance, angle, and height to obtain processed data.

[0157] In one possible implementation, the first 2D feature map is aligned based on global information interaction to obtain a second 2D feature map, including:

[0158] Based on nearest-neighbor nonmaximum suppression, multiple first key points of column features in each column of the first 2D feature map are determined;

[0159] Based on the first attention mechanism, the column features of each column are compressed into the feature information of each first key point in the column to obtain multiple second key points. In the first attention mechanism, the features of each first key point are used as requests and the column features of the column are used as keywords and values.

[0160] The second key points corresponding to each column are divided into multiple non-overlapping key point windows, and the feature information of each second key point in each key point window is interacted based on the second attention mechanism to obtain multiple third key points. In the second attention mechanism, the features of the second key points are used as requests, keywords and values.

[0161] Based on the third attention mechanism, the feature information of the third key point in each column feature is diffused to the column feature of the column, resulting in a second 2D feature map.

[0162] In one possible implementation, the first 2D feature information is adjusted and aggregated based on the geometric information and instance-level information of the first 2D feature information to obtain second 2D feature information, including:

[0163] The segmentation information of the first 2D feature information is predicted using the segmentation branch, and the center offset of the first 2D feature information is predicted using the regression branch.

[0164] Based on the segmentation information, the center offset, and the position code corresponding to the first 2D feature information, the geometric information corresponding to the first 2D feature information is determined.

[0165] The geometric information and the first 2D feature information are divided into multiple non-overlapping information windows, and the geometric information and the first 2D feature information in each information window are interacted based on a fourth attention mechanism to obtain the second 2D feature information. The fourth attention mechanism is a self-attention mechanism.

[0166] The implementation methods and beneficial effects of the various modules and detection heads of the target detection device provided in this application embodiment can be referred to the relevant descriptions of the corresponding steps in the above target detection method. To avoid redundancy, they will not be repeated here.

[0167] The aforementioned target detection device can be an electronic device 100. Figure 13 A schematic diagram of an electronic device 100 is shown. The electronic device 100 may include at least one of the following: mobile phone, foldable electronic device, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device. This application does not impose any special limitations on the specific type of the electronic device 100.

[0168] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) connector 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, lidar (not shown), and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0169] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0170] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0171] The processor can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions, and then implement the above target detection method after acquiring point cloud data from the lidar.

[0172] Embodiments of this application provide a target detection device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0173] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0174] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0175] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0176] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0177] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0178] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0179] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0180] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0182] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0183] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0184] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A target detection method, characterized in that, The method includes: The point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain the processed data; Feature extraction is performed on the processed data to obtain a first 2D feature map; The second 2D feature map is obtained by aligning the first 2D feature map based on global information interaction. 2D feature extraction is performed on the second 2D feature map to obtain the first 2D feature information; Based on the geometric information and instance-level information of the first 2D feature information, the first 2D feature information is adjusted and aggregated to obtain the second 2D feature information; Target detection is performed based on the second 2D feature information to obtain the detection result; The second 2D feature map is obtained by aligning the first 2D feature map based on global information interaction, including: Based on nearest-neighbor nonmaximum suppression, multiple first key points of column features in each column of the first 2D feature map are determined; Based on the first attention mechanism, the column features of each column are compressed into the feature information of each first key point in the column to obtain multiple second key points; The second key points corresponding to each column are divided into multiple non-overlapping key point windows, and the feature information of each second key point in each key point window is interacted based on the second attention mechanism to obtain multiple third key points. Based on the third attention mechanism, the feature information of the third key point in each column feature is diffused to the column feature of the column, resulting in a second 2D feature map.

2. The method according to claim 1, characterized in that, The point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain the processed data, including: The point cloud data is divided into 3D spaces according to distance, angle, and height to obtain processed data.

3. The method according to claim 1, characterized in that, In the first attention mechanism, each first key point feature is used as the request and the column feature of the column in which it is located is used as the keyword and value; in the second attention mechanism, the second key point feature is used as the request, keyword and value. Based on the third attention mechanism, the feature information of the third key point in each column feature is diffused to the column feature of the column, resulting in a second 2D feature map.

4. The method according to claim 1, characterized in that, Based on the geometric and instance-level information of the first 2D feature information, the first 2D feature information is adjusted and aggregated to obtain the second 2D feature information, including: The segmentation information of the first 2D feature information is predicted using the segmentation branch, and the center offset of the first 2D feature information is predicted using the regression branch. Based on the segmentation information, the center offset, and the position code corresponding to the first 2D feature information, the geometric information corresponding to the first 2D feature information is determined. The geometric information and the first 2D feature information are divided into multiple non-overlapping information windows, and the geometric information and the first 2D feature information in each information window are interacted based on a fourth attention mechanism to obtain the second 2D feature information. The fourth attention mechanism is a self-attention mechanism.

5. A target detection device, characterized in that, The device includes: The global feature alignment module is used to perform voxelization on the point cloud data acquired by the lidar according to polar coordinates to obtain processed data; to extract features from the processed data to obtain a first 2D feature map; and to align the first 2D feature map based on global information interaction to obtain a second 2D feature map. The 2D feature extraction module is used to extract 2D features from the second 2D feature map to obtain the first 2D feature information; A geometry-aware detection head is used to adjust and aggregate the first 2D feature information based on the geometric information and instance-level information of the first 2D feature information to obtain the second 2D feature information. The target detection module is used to perform target detection based on the second 2D feature information and obtain the detection result; The second 2D feature map is obtained by aligning the first 2D feature map based on global information interaction, including: Based on nearest-neighbor nonmaximum suppression, multiple first key points of column features in each column of the first 2D feature map are determined; Based on the first attention mechanism, the column features of each column are compressed into the feature information of each first key point in the column to obtain multiple second key points; The second key points corresponding to each column are divided into multiple non-overlapping key point windows, and the feature information of each second key point in each key point window is interacted based on the second attention mechanism to obtain multiple third key points. Based on the third attention mechanism, the feature information of the third key point in each column feature is diffused to the column feature of the column, resulting in a second 2D feature map.

6. The apparatus according to claim 5, characterized in that, The point cloud data acquired by the lidar is voxelized according to polar coordinates to obtain the processed data, including: The point cloud data is divided into 3D spaces according to distance, angle, and height to obtain processed data.

7. The apparatus according to claim 5, characterized in that, In the first attention mechanism, each first key point feature is used as the request and the column feature of the column in which it is located is used as the keyword and value; in the second attention mechanism, the second key point feature is used as the request, keyword and value.

8. The apparatus according to claim 5, characterized in that, Based on the geometric and instance-level information of the first 2D feature information, the first 2D feature information is adjusted and aggregated to obtain the second 2D feature information, including: The segmentation information of the first 2D feature information is predicted using the segmentation branch, and the center offset of the first 2D feature information is predicted using the regression branch. Based on the segmentation information, the center offset, and the position code corresponding to the first 2D feature information, the geometric information corresponding to the first 2D feature information is determined. The geometric information and the first 2D feature information are divided into multiple non-overlapping information windows, and the geometric information and the first 2D feature information in each information window are interacted based on a fourth attention mechanism to obtain the second 2D feature information. The fourth attention mechanism is a self-attention mechanism.

9. A target detection device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-4 when executing the instructions.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-4.

11. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that, When the computer-readable code is run in an electronic device, the processor in the electronic device performs the method according to any one of claims 1-4.

Images