Three-dimensional target recognition method, device and vehicle

Abstract

This application discloses a three-dimensional target recognition method, apparatus, and vehicle. The method includes: performing channel-level pooling operations on LiDAR features and visual features respectively to obtain a first feature; performing convolution and normalization operations on the first feature to obtain spatial attention weights; weighting the visual features based on the spatial attention weights of the LiDAR features to obtain weighted visual features; weighting the LiDAR features based on the spatial attention weights of the visual features to obtain weighted LiDAR features; and concatenating the weighted visual features and the weighted LiDAR features to obtain a first fusion feature; and recognizing the target to be recognized based on the first fusion feature. This application, through a cross-attention mechanism, deeply explores the spatial complementarity of LiDAR features and visual features, improving the completeness and robustness of target recognition in complex scenes.

Description

Technical Field

[0001] This application relates to the field of target recognition technology, and in particular, to a three-dimensional target recognition method, apparatus, and vehicle. Background Technology

[0002] 3D object recognition is one of the core technologies in the field of computer vision. It is used to detect and identify objects from three-dimensional data and output information such as their category and three-dimensional bounding box.

[0003] When using LiDAR and vision sensors for 3D target recognition, multimodal feature fusion is mainly achieved through feature stitching and addition. However, the above fusion methods are prone to feature redundancy and overwhelming of key information, making it difficult to improve the accuracy and robustness of 3D target recognition. Summary of the Invention

[0004] The embodiments of this application provide a three-dimensional target recognition method, apparatus, and vehicle to solve the technical problem that existing multimodal fusion methods are prone to feature redundancy and overwhelming of key information.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to a first aspect of this application, a three-dimensional target recognition method is provided, the method comprising: The system acquires point cloud data of the target to be identified based on LiDAR detection and image data of the target to be identified based on a visual sensor, performs feature extraction on the point cloud data to obtain LiDAR features, and performs feature extraction on the image data to obtain visual features. For each feature, channel-level pooling is performed on the feature to obtain a first feature, and convolution and normalization operations are performed on the first feature to obtain the spatial attention weight of the feature, wherein the feature is the LiDAR feature or the visual feature; The visual features are weighted based on the spatial attention weights of the lidar features to obtain weighted visual features. The lidar features are weighted based on the spatial attention weights of the visual features to obtain weighted lidar features. The weighted visual features and the weighted lidar features are then concatenated to obtain a first fusion feature. The target to be identified is identified based on the first fusion feature.

[0007] In some embodiments of this application, based on the foregoing scheme, the channel-level pooling operation on the feature includes: The second feature is obtained by performing channel-level max pooling on the feature, and the third feature is obtained by performing channel-level average pooling on the feature. The second feature and the third feature are combined to obtain the first feature.

[0008] In some embodiments of this application, based on the foregoing scheme, after concatenating the weighted visual features and the weighted radar features to obtain the first fused feature, the method further includes: The first fused feature is subjected to global pooling, full connection and normalization operations in sequence to obtain the channel attention weights. After weighting the first fused feature based on the channel attention weights, a convolution operation is performed to obtain the second fused feature; The target to be identified is identified based on the second fusion feature.

[0009] In some embodiments of this application, based on the foregoing scheme, a fully connected operation is performed on the first fusion feature, including: After performing a global pooling operation on the first fused feature, the first full-connection operation, the activation operation, and the second full-connection operation are performed sequentially.

[0010] In some embodiments of this application, based on the foregoing scheme, the step of identifying the target to be identified according to the second fusion feature includes: The target to be identified is identified based on the second fusion feature to obtain the identification result of the target to be identified. The identification result includes a contour mask and a three-dimensional detection box. The contour mask is used to characterize the irregularity of the target to be identified, and the three-dimensional detection box is used to characterize the regularity of the target to be identified.

[0011] In some embodiments of this application, based on the foregoing scheme, both the lidar features and the visual features are in a bird's-eye view space.

[0012] In some embodiments of this application, based on the foregoing scheme, the step of extracting features from the point cloud data to obtain lidar features includes: The point cloud data is preprocessed to obtain voxel features; The voxel features are encoded to obtain columnar features; The column features are mapped onto the bird's-eye view space to obtain spatial features; The spatial features are subjected to multi-scale feature fusion to obtain the lidar features.

[0013] In some embodiments of this application, based on the foregoing scheme, the step of extracting visual features from the image data includes: Image feature extraction is performed on the image data to obtain multi-scale fusion features; The multi-scale fusion features are aggregated into the bird's-eye view space, and the aggregated multi-scale fusion features are subjected to bird's-eye view feature extraction to obtain the visual features.

[0014] According to a second aspect of this application, a three-dimensional target recognition device is provided, the device comprising: The first acquisition unit acquires point cloud data of the target to be identified based on lidar detection and image data of the target to be identified based on visual sensor detection, performs feature extraction on the point cloud data to obtain lidar features, and performs feature extraction on the image data to obtain visual features. The first obtaining unit performs channel-level pooling on each feature to obtain a first feature, and performs convolution and normalization operations on the first feature to obtain the spatial attention weight of the feature, wherein the feature is the LiDAR feature or the visual feature. The second obtaining unit weights the visual features based on the spatial attention weights of the lidar features to obtain weighted visual features, weights the lidar features based on the spatial attention weights of the visual features to obtain weighted lidar features, and concatenates the weighted visual features and the weighted lidar features to obtain a first fusion feature. The first identification unit identifies the target to be identified based on the first fusion feature.

[0015] In some embodiments of this application, based on the foregoing scheme, the first obtaining unit includes a first pooling unit, which is used to perform channel-level pooling operations on the features. The first pooling unit is configured as follows: The second pooling unit performs channel-level max pooling on the feature to obtain the second feature, and performs channel-level average pooling on the feature to obtain the third feature. The third obtaining unit concatenates the second feature and the third feature to obtain the first feature.

[0016] In some embodiments of this application, based on the foregoing scheme, the device further includes a second identification unit, which is configured to, after the weighted visual features and the weighted radar features are stitched together to obtain the first fused feature, perform the following steps: The fourth unit performs global pooling, full connection, and normalization operations on the first fused feature in sequence to obtain the channel attention weights. The fifth unit performs a convolution operation on the first fused feature based on the channel attention weights to obtain the second fused feature. The third identification unit identifies the target to be identified based on the second fusion feature.

[0017] In some embodiments of this application, based on the foregoing scheme, the fourth obtaining unit includes a first operation unit, which is used to perform a fully connected operation on the first fused feature. The first operation unit is configured as follows: The second operation unit performs a global pooling operation on the first fused feature, followed by a first full-connection operation, an activation operation, and a second full-connection operation in sequence.

[0018] In some embodiments of this application, based on the foregoing scheme, the third identification unit is configured as follows: The fourth identification unit performs target identification on the target to be identified based on the second fusion feature to obtain the identification result of the target to be identified. The identification result includes a contour mask and a three-dimensional detection box. The contour mask is used to characterize the irregularity of the target to be identified, and the three-dimensional detection box is used to characterize the regularity of the target to be identified.

[0019] In some embodiments of this application, based on the foregoing scheme, both the lidar features and the visual features are in a bird's-eye view space.

[0020] In some embodiments of this application, based on the foregoing scheme, the first acquisition unit includes a sixth obtaining unit, which is used to extract features from the point cloud data to obtain lidar features. The sixth obtaining unit is configured as follows: The seventh unit performs preprocessing operations on the point cloud data to obtain voxel features; The eighth unit encodes the voxel features to obtain columnar features; The ninth unit maps the column features to the bird's-eye view space to obtain spatial features; The tenth unit performs a multi-scale feature fusion operation on the spatial features to obtain the lidar features.

[0021] In some embodiments of this application, based on the foregoing scheme, the first acquisition unit further includes an eleventh obtaining unit, which is used to extract visual features from the image data. The obtaining unit is configured as follows: The twelfth unit performs image feature extraction on the image data to obtain multi-scale fusion features; The thirteenth unit aggregates the multi-scale fusion features into the bird's-eye view space, and performs bird's-eye view feature extraction on the aggregated multi-scale fusion features to obtain the visual features.

[0022] According to a third aspect of this application, a vehicle is provided, comprising: a processor for performing the method described in any embodiment of the first aspect of this application.

[0023] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, the computer program including executable instructions that, when executed by a processor, implement the method described in any embodiment of the first aspect of this application.

[0024] According to a fifth aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions of the processors, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any embodiment of the first aspect of this application.

[0025] The beneficial effects of this application are as follows: The spatial attention weights of LiDAR features and visual features are determined. The visual features are weighted based on the spatial attention weights of the LiDAR features, and the LiDAR features are weighted based on the spatial attention weights of the visual features. These weights are then concatenated to obtain the first fused feature. The first fused feature deeply explores the spatial complementarity of LiDAR features and visual features, which can enhance the target information of the target to be identified that is not significant in the LiDAR features and visual features. It focuses more on key spatial domains, effectively avoids target omissions caused by the lack of single-modal features, and improves the completeness and robustness of target recognition in complex scenarios.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 A flowchart of a three-dimensional target recognition method according to an embodiment of this application is shown; Figure 2 This paper shows an overall schematic diagram of a three-dimensional target recognition method according to an embodiment of the present application; Figure 3 This application shows Figure 2 First partial schematic diagram; Figure 4 This application shows Figure 2 A second partial schematic diagram; Figure 5 This application shows Figure 2 A schematic diagram of the third part; Figure 6 This application shows Figure 2 The fourth partial schematic diagram; Figure 7 This application shows Figure 2 A partial schematic diagram of the fifth part; Figure 8 This application shows Figure 2 A schematic diagram of the sixth part; Figure 9 This application shows Figure 2 The seventh partial schematic diagram; Figure 10 This application shows Figure 2 The eighth part of the schematic diagram; Figure 11 A block diagram of a three-dimensional target recognition device according to an embodiment of this application is shown; Figure 12 A schematic diagram of a computer-readable storage medium in an embodiment of this application is shown; Figure 13 A schematic diagram of the system structure of an electronic device in an embodiment of this application is shown. Detailed Implementation

[0028] Figure 1 A flowchart of a three-dimensional target recognition method according to an embodiment of this application is shown. See also: Figure 1 This paper provides a method for recognizing three-dimensional targets, where three-dimensional targets can be understood as 3D targets. The method includes at least S1 to S4, which are described in detail below: In step S1, point cloud data of the target to be identified based on LiDAR detection and image data of the target to be identified based on visual sensor detection are acquired. Feature extraction is performed on the point cloud data to obtain LiDAR features, and feature extraction is performed on the image data to obtain visual features.

[0029] In step S2, for each feature, channel-level pooling is performed on the feature to obtain a first feature. Then, convolution and normalization operations are performed on the first feature to obtain the spatial attention weight of the feature. The feature is either the LiDAR feature or the visual feature. The first feature can be normalized using a sigmoid activation function.

[0030] In step S3, the visual features are weighted based on the spatial attention weights of the lidar features to obtain weighted visual features, the lidar features are weighted based on the spatial attention weights of the visual features to obtain weighted lidar features, and the weighted visual features and the weighted lidar features are spliced ​​together to obtain a first fusion feature. In step S4, the target to be identified is identified based on the first fusion feature.

[0031] Figure 2 This paper presents an overall schematic diagram of a three-dimensional target recognition method according to an embodiment of the present application. Figure 3 This application shows Figure 2 The first partial schematic diagram, in Figure 2 In this context, Lidar refers to a laser radar, Camera 1, Camera 2, etc., are visual sensors, VFE is voxel feature encoding, Scatter is scattering, Backbone+FPN is a backbone network plus feature pyramid, Img_Backbone+FPN is an image backbone network plus feature pyramid, AttentionTransform is an attention transformation operation, and BEV_Backbone+FPN is a bird's-eye view backbone network plus feature pyramid. Figure 3 middle, Features of lidar For visual features, channel Maxpool is channel-level max pooling, channel Avgpool is channel-level average pooling, C is concatenation, Conv3×3 is convolution operation, and σ is normalization operation. Spatial attention weights for lidar features. The spatial attention weights for visual features are shown, with × indicating weighting. The first fusion feature is that SEblock is the SE module, i.e., the extrusion excitation module. For the second fusion feature, Globalpool is the global pooling function, ReLU is the activation function, and sigmoid is the normalization operation. Figure 3 It can be understood as Figure 2 The Fusion_Block part in it.

[0032] See Figure 2 and Figure 3In some implementations, the channel-level pooling operation on the feature includes: performing channel-level max pooling on the feature to obtain a second feature, and performing channel-level average pooling on the feature to obtain a third feature; and concatenating the second feature and the third feature to obtain the first feature.

[0033] In some implementations, the spatial attention weights are calculated using the following formula: ,in, Spatial attention weights for lidar features. It is the Sigmoid activation function. For a 3×3 convolution operation, For splicing operations, This is a channel-level max pooling operation. Features of lidar This is a channel-level average pooling operation. Spatial attention weights for visual features It is a visual feature.

[0034] In some implementations, the calculation formula for the first fusion feature is as follows: ,in, The first fusion feature, For splicing operations, Spatial attention weights for lidar features. Features of lidar Spatial attention weights for visual features It is a visual feature.

[0035] In this way, through the cross-attention mechanism, the spatial complementarity between LiDAR features and visual features can be deeply explored. For example, for small targets at a distance, the point cloud data is clear, while the image data is semantically ambiguous due to its low resolution. At this time, the LiDAR features are significant, while the visual features are not, and the attention weight of the corresponding region is large. After weighted fusion, the target information that was originally not significant in the visual features can be strengthened, allowing the model to focus more on key spatial regions, effectively avoiding target misses caused by the lack of single-modal features, and improving the completeness and robustness of target recognition in complex scenarios.

[0036] In some implementations, after concatenating the weighted visual features and the weighted radar features to obtain the first fused feature, the method further includes: performing global pooling, fully connected operation, and normalization operation sequentially on the first fused feature to obtain channel attention weights; performing convolution operation on the first fused feature based on the channel attention weights to obtain the second fused feature; and identifying the target to be identified based on the second fused feature.

[0037] In some implementations, performing a full-connection operation on the first fused feature includes: after performing a global pooling operation on the first fused feature, performing a first full-connection operation, an activation operation, and a second full-connection operation in sequence.

[0038] In some implementations, the second fusion feature is calculated using the following formula: ,in, This is the second fusion feature. For a 3×3 convolution operation, For SE module, The first fusion feature, It is the sigmoid activation function. For fully connected operation, The ReLU activation function is used. This is a global pooling operation.

[0039] Thus, a global pooling operation is performed on the first fused feature to extract the average feature of each channel. Then, channel attention weights are calculated through two fully connected operations and the Sigmoid function. The channel attention weights are used to enhance key channel features and suppress redundant channels. Finally, the channel attention weights are multiplied by the first fused feature to enhance the channels. Then, a 3×3 convolution operation is performed to further extract features, resulting in a second fused feature that combines spatial and channel features.

[0040] In some embodiments, identifying the target to be identified based on the second fusion feature includes: performing target identification on the target to be identified based on the second fusion feature to obtain an identification result of the target to be identified, wherein the identification result includes a contour mask and a three-dimensional detection box, wherein the contour mask is used to characterize the irregularity of the target to be identified, and the three-dimensional detection box is used to characterize the regularity of the target to be identified.

[0041] Thus, contour masks are used to handle irregular targets, such as flower beds and winding paths, while 3D bounding boxes are used to handle regular targets, such as pedestrians and vehicles, thereby improving the perception of various targets in complex scenes and providing more effective information for downstream tasks.

[0042] In some implementations, both the lidar features and the visual features are located in a bird's-eye view space. The bird's-eye view space can be understood as the BEV space.

[0043] In some embodiments, the step of extracting features from the point cloud data to obtain lidar features includes: preprocessing the point cloud data to obtain voxel features; encoding the voxel features to obtain column features; mapping the column features to the bird's-eye view space to obtain spatial features; and performing multi-scale feature fusion on the spatial features to obtain the lidar features. The column features are Pillar features.

[0044] In some implementations, the preprocessing operation on the point cloud data to obtain voxel features includes: loading the point cloud data and its ground truth information; performing data augmentation on the point cloud data and its ground truth information; filtering out point cloud data that exceeds the boundary and retaining the ground truth information of point cloud data within the boundary; and converting the point cloud data into voxel features. Data augmentation may include random rotation and scaling. Filtering out point cloud data that exceeds the boundary and retaining the ground truth information of point cloud data within the boundary can be understood as filtering out point clouds and targets that exceed the boundary. In the voxel features, each voxel may be limited to 32 points, and the position coordinates (coord) of each voxel are recorded.

[0045] In some embodiments, the voxel features include multiple voxels. Encoding the voxel features to obtain cylinder features includes: obtaining a first deviation feature based on the positional deviation of all point clusters of each voxel; obtaining a second deviation feature based on the deviation of all points from the coordinate center point of the voxel; concatenating the first deviation feature, the second deviation feature, and the voxel features; and then extracting cylinder features to obtain the cylinder features. The positional deviation is the deviation of each point of each voxel from the mean center of all point clusters; the mean center can also be understood as the centroid.

[0046] Figure 4 This application shows Figure 2 The second partial schematic diagram, Figure 4 It can be understood as Figure 2 For the VFE section, see Figure 4 For voxel features, in (num_voxels, 32, 4), num_voxels represents the total number of voxels, 32 represents the number of points, and 4 represents the dimension. See [link to documentation]. Figure 4Based on the positional deviation of all point clusters for each voxel, a first deviation feature is obtained, i.e., the positional deviation of all point clusters is calculated to obtain the first deviation feature, which is represented by (num_voxels,32,3). Based on the deviation of all points from the center point of the voxel coordinates, a second deviation feature is obtained, i.e., the deviation of all points from the center point of the voxel coordinates is calculated to obtain the second deviation feature, which is represented by (num_voxels,32,2). (num_voxels,32,4), (num_voxels,32,3), and (num_voxels,32,2) are concatenated by feature dimensions to obtain (num_voxels,32,9), which is then fed into the PFE feature extraction network for feature extraction to obtain the cylinder feature (num_voxels,32).

[0047] For example, Figure 5 This application shows Figure 2 The third partial schematic diagram, Figure 5 It can be understood as Figure 2 See the Scatter section. Figure 5 The PointPillarScatter module is used to map Pillar features onto the BEV space grid, where the BEV space size is set to (640, 256). The Pillar features are output via VFE, and Coords represents the coordinates of each voxel calculated during data preprocessing. Using coordinate indexing, the Pillar features can be projected onto the BEV space plane to obtain the spatial features. Figure 2 When applied to vehicles in this application, the spatial feature is the driving space feature; Figure 6 This application shows Figure 2 The fourth partial schematic diagram, Figure 6 It can be understood as Figure 2 For the Backbone+FPN section, see [link / reference]. Figure 6 The Downblock downsampling module is used to compress spatial dimensions and increase the number of channels, downsampling spatial features by 2x, 4x, and 8x. The downsampled features are then aligned using a 1×1 convolution through a lateral connection in the lateral layer. Lower-layer features are upsampled through the upblock to restore spatial dimensions and then added to the upper-layer features. The fused features are then smoothed by the smoothing module to unify the feature dimensions to 32. Finally, the upfinal upsampling module outputs a unified feature map size (128, 320). After concatenation, the final LiDAR feature map is output. Figure 6 The LiDAR features (B, 96, 128, 320).

[0048] In some embodiments, the step of extracting visual features from the image data includes: performing image feature extraction operations on the image data to obtain multi-scale fusion features; aggregating the multi-scale fusion features into the bird's-eye view space; and performing bird's-eye view feature extraction operations on the aggregated multi-scale fusion features to obtain the visual features.

[0049] For example, Figure 7 This application shows Figure 2 The fifth partial schematic diagram, Figure 7 It can be understood as Figure 2 For the Img_Backbone+FPN section, see [link / reference]. Figure 7 Taking two cameras as input images as an example, it uses DLA-Net34 as the backbone network for feature extraction, then performs lateral connections on the downsampled results of 8x, 16x, and 32x, and finally performs upsampling and summation to output multi-scale fused features. Figure 8 This application shows Figure 2 The sixth partial diagram, Figure 8 It can be understood as Figure 2 For the AttentionTransform part, see Figure 8 First, pre-generate 3D position reference points ref_3d in BEV space. Each pillar contains N points. Figure 8 In the first step, N is 15. Second, the generated 3D position reference points are projected onto each camera image to find the corresponding 2D position reference points (reference_points_cam) and the mask (bev_mask). Third, based on bev_mask, 2D position reference points (reference_points) that contain features within the image range are selected, and their position numbers (index_query_per_img) are recorded. Fourth, the input image features (Camera_feature) are extracted, and position encoding information is added to output the encoded feature (feat_flatten). Fifth, the corresponding encoded feature (feat_flatten) at the reference_points position is obtained through the grid_sample function, and the feature information within the effective area (Output) is output. Sixth, finally, the image features are filled into the BEV space based on index_query_per_img, and the number of channels is changed to 256 through convolution to obtain the final feature slots in the BEV space. This means that the multi-scale fused features are aggregated into the bird's-eye view space. Figure 9 This application shows Figure 2 The seventh partial schematic diagram, Figure 9 It can be understood as Figure 2For the BEV_Backbone+FPN section, see [link / reference]. Figure 9 Further feature extraction is performed on the extracted BEV features. Its structure is basically similar to that of the Backbone+FPN of LiDAR. The output features are visual features, that is, the bird's-eye view feature extraction operation is performed on the aggregated multi-scale fused features to obtain the visual features.

[0050] In some embodiments, before performing image feature extraction on the image data to obtain multi-scale fusion features, the method further includes: preprocessing the image data, including: loading images and labels from multiple cameras, data augmentation, and normalization processing, wherein data augmentation includes: translation, rotation, and color transformation.

[0051] In some embodiments, after extracting features from the point cloud data to obtain LiDAR features, the method further includes: performing pixel value segmentation and bounding box regression operations on the LiDAR features, and then backpropagating these operations to update the operation parameters of the encoding operation, mapping the cylinder features to the bird's-eye view space, and the multi-scale feature extraction operation. After extracting features from the image data to obtain visual features, the method further includes: performing pixel value segmentation and bounding box regression operations on the visual features, and then backpropagating these operations to update the operation parameters of the image feature extraction operation, aggregating the multi-scale fused features to the bird's-eye view space, and the bird's-eye view feature extraction operation to obtain the visual features.

[0052] For example, Figure 10 This application shows Figure 2 The eighth part of the schematic diagram, Figure 10 It can be understood as Figure 2 For the detection head and segmentation head parts, see Figure 10 The head structure is divided into two parts: the pp_heavy_head detection head and the SegHead_pcseg segmentation head. The detection head first uses a 3×3 convolution to extract features, and then uses three 1×1 convolutions to obtain the output of class probability, box size, and orientation class. The class probability is the predicted score for each class, the box probability includes the prediction of the center point (x,y,z), size (w,h,l), and heading angle θ, and the orientation class is used to predict whether the heading angle is forward or backward. The number of output channels is related to the number of pre-set anchors. The segmentation head first uses upsampling to restore the feature map scale to the size set in the BEV space, and then outputs the predicted segmentation result through convolution operation, where the number of channels is the number of segmented classes.

[0053] In this application, spatial attention weights for LiDAR features and visual features are determined. Visual features are weighted based on the spatial attention weights of LiDAR features, and LiDAR features are weighted based on the spatial attention weights of visual features. These weights are then concatenated to obtain a first fusion feature. The first fusion feature deeply explores the spatial complementarity of LiDAR features and visual features, which can enhance the target information of the target to be identified that is not significant in LiDAR features and visual features. It focuses more on key spatial domains, effectively avoids target omissions caused by the lack of single-modal features, and improves the completeness and robustness of target recognition in complex scenarios.

[0054] Figure 11 A block diagram of a three-dimensional target recognition device according to an embodiment of this application is shown. See also: Figure 11 According to a second aspect of this application, a three-dimensional target recognition device 100 is provided, the device comprising: The first acquisition unit 101 acquires point cloud data of the target to be identified based on lidar detection and image data of the target to be identified based on visual sensor detection, performs feature extraction on the point cloud data to obtain lidar features, and performs feature extraction on the image data to obtain visual features. The first obtaining unit 102 performs channel-level pooling on each feature to obtain a first feature, and performs convolution and normalization operations on the first feature to obtain the spatial attention weight of the feature, wherein the feature is the LiDAR feature or the visual feature. The second obtaining unit 103 weights the visual features based on the spatial attention weights of the lidar features to obtain weighted visual features, weights the lidar features based on the spatial attention weights of the visual features to obtain weighted lidar features, and concatenates the weighted visual features and the weighted lidar features to obtain a first fusion feature. The first identification unit 104 identifies the target to be identified based on the first fusion feature.

[0055] In some embodiments, the first obtaining unit includes a first pooling unit, which is used to perform channel-level pooling operations on the feature. The first pooling unit is configured as follows: a second pooling unit performs channel-level max pooling on the feature to obtain a second feature, and performs channel-level average pooling on the feature to obtain a third feature; and a third obtaining unit concatenates the second feature and the third feature to obtain the first feature.

[0056] In some embodiments, the device further includes a second identification unit, which is configured as follows after the weighted visual features and the weighted radar features are stitched together to obtain a first fused feature: a fourth obtaining unit performs global pooling, fully connected operation, and normalization operation sequentially on the first fused feature to obtain channel attention weights; a fifth obtaining unit performs convolution operation on the first fused feature based on the channel attention weights to obtain a second fused feature; and a third identification unit identifies the target to be identified based on the second fused feature.

[0057] In some embodiments, the fourth obtaining unit includes a first operation unit, which is used to perform a full connection operation on the first fused feature. The first operation unit is configured as: a second operation unit, which, after performing a global pooling operation on the first fused feature, sequentially performs a first full connection operation, an activation operation, and a second full connection operation.

[0058] In some embodiments, the third identification unit is configured as a fourth identification unit, which performs target identification on the target to be identified based on the second fusion feature to obtain the identification result of the target to be identified. The identification result includes a contour mask and a three-dimensional detection box. The contour mask is used to characterize the irregularity of the target to be identified, and the three-dimensional detection box is used to characterize the regularity of the target to be identified.

[0059] In some implementations, both the lidar features and the visual features are located in a bird's-eye view space.

[0060] In some embodiments, the first acquisition unit includes a sixth obtaining unit, which is used to extract features from the point cloud data to obtain lidar features. The sixth obtaining unit is configured as follows: a seventh obtaining unit performs preprocessing operations on the point cloud data to obtain voxel features; an eighth obtaining unit performs encoding operations on the voxel features to obtain column features; a ninth obtaining unit maps the column features to the bird's-eye view space to obtain spatial features; and a tenth obtaining unit performs multi-scale feature fusion operations on the spatial features to obtain the lidar features.

[0061] In some embodiments, the first acquisition unit further includes an eleventh obtaining unit, which is used to extract visual features from the image data. The obtaining unit is configured as follows: a twelfth obtaining unit performs image feature extraction on the image data to obtain multi-scale fusion features; and a thirteenth obtaining unit aggregates the multi-scale fusion features into the bird's-eye view space and performs bird's-eye view feature extraction on the aggregated multi-scale fusion features to obtain the visual features.

[0062] According to a third aspect of this application, a vehicle is provided, comprising: a processor for performing the method described in any embodiment of the first aspect of this application.

[0063] Based on the same inventive concept, as a fourth aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program including executable instructions that, when executed by a processor, implement the method described in any embodiment of the first aspect of this application.

[0064] In some possible implementations, various aspects of this application may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this application.

[0065] refer to Figure 12 As shown, a program product 200 for implementing the above-described method according to an embodiment of this application is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0066] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0067] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0068] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0069] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0070] In another respect, this application also provides an electronic device capable of implementing the above-described method.

[0071] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0072] The following reference Figure 13 To describe an electronic device 300 according to this embodiment of the present application. Figure 13 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0073] like Figure 13 As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, and a bus 330 connecting different system components (including storage unit 320 and processing unit 310).

[0074] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the "Embodiment Methods" section above according to various exemplary embodiments of this application.

[0075] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0076] Storage unit 320 may also include a program / utility 324 having a set (at least one) of program modules 325, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0077] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0078] Electronic device 300 can also communicate with one or more external devices 400 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with the electronic device 300, and / or with any device that enables the electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. Figure 4 As shown, network adapter 360 communicates with other modules of electronic device 300 via bus 330. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0079] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and embodiments are within the scope and spirit of this application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Furthermore, the functional units may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0080] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A three-dimensional target recognition method, characterized in that, The method includes: The system acquires point cloud data of the target to be identified based on LiDAR detection and image data of the target to be identified based on a visual sensor, performs feature extraction on the point cloud data to obtain LiDAR features, and performs feature extraction on the image data to obtain visual features. For each feature, channel-level pooling is performed on the feature to obtain a first feature, and convolution and normalization operations are performed on the first feature to obtain the spatial attention weight of the feature, wherein the feature is the LiDAR feature or the visual feature; The visual features are weighted based on the spatial attention weights of the lidar features to obtain weighted visual features. The lidar features are weighted based on the spatial attention weights of the visual features to obtain weighted lidar features. The weighted visual features and the weighted lidar features are then concatenated to obtain a first fusion feature. The target to be identified is identified based on the first fusion feature.

2. The three-dimensional target recognition method according to claim 1, characterized in that, The channel-level pooling operation on the features includes: The second feature is obtained by performing channel-level max pooling on the feature, and the third feature is obtained by performing channel-level average pooling on the feature. The second feature and the third feature are combined to obtain the first feature.

3. The three-dimensional target recognition method according to claim 1, characterized in that, After concatenating the weighted visual features and the weighted radar features to obtain the first fused feature, the method further includes: The first fused feature is subjected to global pooling, full connection and normalization operations in sequence to obtain the channel attention weights. After weighting the first fused feature based on the channel attention weights, a convolution operation is performed to obtain the second fused feature; The target to be identified is identified based on the second fusion feature.

4. The three-dimensional target recognition method according to claim 3, characterized in that, Performing a fully connected operation on the first fused feature includes: After performing a global pooling operation on the first fused feature, the first full-connection operation, the activation operation, and the second full-connection operation are performed sequentially.

5. A three-dimensional target recognition method according to claim 3, characterized in that, The step of identifying the target based on the second fusion feature includes: The target to be identified is identified based on the second fusion feature to obtain the identification result of the target to be identified. The identification result includes a contour mask and a three-dimensional detection box. The contour mask is used to characterize the irregularity of the target to be identified, and the three-dimensional detection box is used to characterize the regularity of the target to be identified.

6. The three-dimensional target recognition method according to claim 1, characterized in that, Both the lidar features and the visual features are located in a bird's-eye view space.

7. A three-dimensional target recognition method according to claim 6, characterized in that, The step of extracting features from the point cloud data to obtain lidar features includes: The point cloud data is preprocessed to obtain voxel features; The voxel features are encoded to obtain columnar features; The column features are mapped onto the bird's-eye view space to obtain spatial features; The spatial features are subjected to multi-scale feature fusion to obtain the lidar features.

8. A three-dimensional target recognition method according to claim 6, characterized in that, The step of extracting visual features from the image data includes: Image feature extraction is performed on the image data to obtain multi-scale fusion features; The multi-scale fusion features are aggregated into the bird's-eye view space, and the aggregated multi-scale fusion features are subjected to bird's-eye view feature extraction to obtain the visual features.

9. A three-dimensional target recognition device, characterized in that, The device includes: The first acquisition unit acquires point cloud data of the target to be identified based on lidar detection and image data of the target to be identified based on visual sensor detection, performs feature extraction on the point cloud data to obtain lidar features, and performs feature extraction on the image data to obtain visual features. The first obtaining unit performs channel-level pooling on each feature to obtain a first feature, and performs convolution and normalization operations on the first feature to obtain the spatial attention weight of the feature, wherein the feature is the LiDAR feature or the visual feature. The second obtaining unit weights the visual features based on the spatial attention weights of the lidar features to obtain weighted visual features, weights the lidar features based on the spatial attention weights of the visual features to obtain weighted lidar features, and concatenates the weighted visual features and the weighted lidar features to obtain a first fusion feature. The first identification unit identifies the target to be identified based on the first fusion feature.

10. A vehicle, characterized in that, include: A processor for performing the method according to any one of claims 1-8.

Images