Multi-scale space-time hybrid sampling fusion method and system for three-dimensional target detection

Through the multi-scale space-time hybrid sampling and fusion method, combined with Transformer and other technologies, LiDAR and image data are effectively integrated, which solves the problems of overfitting and feature misalignment in multimodal network training in three-dimensional target detection of autonomous driving, and improves detection accuracy and positioning accuracy.

CN118941899AActive Publication Date: 2024-11-12SUZHOU UNIV

Patent Information

Application Number
CN202410842787.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-12
Estimated Expiration
2044-06-27

AI Technical Summary

Technical Problem

In the existing three-dimensional object detection technology of autonomous driving, it is difficult to effectively integrate LiDAR and image data in the training of multimodal networks, resulting in overfitting, uneven generalization, and misalignment between LiDAR bird's eye view features and image features, reducing detection performance.

Method used

Using multi-scale spatial and temporal hybrid sampling fusion method, the dual-stream spatial and temporal feature extraction network generates adaptive spatial and temporal sampling features, and the point cloud coordinate features are aggregated into BEV features by transformer and other variable conversion modules, and multimodal features are captured through the adaptive hybrid mechanism. The spatial position relationship between them is combined with multi-scale cross-transformer for feature fusion to realize 3D object detection.

Benefits of technology

It improves detection accuracy and target positioning accuracy, enhances the detection ability of objects of different sizes and categories, and solves the problem that multimodal data cannot be effectively acquired during downscale transformation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941899A_ABST
    Figure CN118941899A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-scale space-time hybrid sampling fusion method and system for three-dimensional target detection, and relates to the technical field of automatic driving target detection. The method comprises the following steps: receiving image data and original point cloud data; performing feature extraction on the image data and the original point cloud data by adopting a double-flow spatio-temporal feature extraction network, and generating adaptive spatio-temporal sampling features of a sampling position under query guidance by using scale adaptive force of a multi-scale receptive field aggregation feature; according to the adaptive space-time sampling features, sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through a Transform equivariant conversion module. Through combination of enhancement of global time scale feature relation and design of a local multi-scale sensing field, the detection precision of objects of different sizes and categories is improved, different single-scale features are converted into multi-scale coupling representation through multi-scale cross Transformer, and more space-time sensing information is extracted to improve the detection precision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving target detection technology, and specifically to a multi-scale spatiotemporal hybrid sampling fusion method and system for three-dimensional target detection. Background Art

[0002] In the field of autonomous driving, 3D object detection is a crucial problem that enables vehicles to accurately perceive and interpret their surroundings. Cameras and LiDAR sensors play an important role in achieving 3D perception tasks for autonomous vehicles, and each has unique advantages and disadvantages. LiDAR sensors provide high-quality distance information, but their echoes become sparse as the distance increases, especially in terms of resolution of distant objects. In contrast, cameras provide rich texture and color information but lack direct 3D depth measurement capabilities, so combining these two sensors can obtain more comprehensive information. However, current fusion methods face some challenges. First, the training of multimodal networks is more challenging than that of single-modal networks because they have features of different scales, a large number of parameters, and are prone to overfitting problems. Second, the overfitting and generalization speeds between different modalities are different, and end-to-end fusion models may lead to suboptimal solutions. In addition, the misalignment between LiDAR bird's-eye view (BEV) features and image features in current fusion architectures may degrade performance.

[0003] To address the above issues, existing techniques use sequential fusion methods, which can fuse image predictions at the input of LiDAR-only detectors, or by fusing the output of 3D LiDAR detectors with image predictions. However, current sequential methods have limited fusion capabilities and may be strictly limited to 2D predictions, or only fuse at sparse projected point clouds. Finally, effective LiDAR data enhancement techniques can accelerate the convergence of LiDAR-only methods, but cannot be directly applied to fusion methods due to the lack of correspondence between LiDAR ground truth objects added in image data. To this end, we propose a multi-scale spatiotemporal hybrid sampling fusion method and system for 3D object detection. Summary of the invention

[0004] According to a first aspect of the present invention, an object of the present invention is to provide a multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection, which specifically includes the following steps:

[0006] Receiving image data and raw point cloud data;

[0007] A two-stream spatiotemporal feature extraction network is used to extract features from image data and raw point cloud data. The scale-adaptive force of the multi-scale receptive field aggregation features is used to generate adaptive spatiotemporal sampling features of the sampling position under the guidance of the query.

[0008] According to the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer equivariant transformation module;

[0009] The adaptive hybrid mechanism is used to capture the spatial position relationship between multimodal features and align point clouds with image features.

[0010] The image features, point cloud features and BEV features are fused into a multi-scale coupled feature representation with more spatial perception through a multi-scale cross Transformer, and then 3D object detection is achieved through a learning network of RPN and classification regression.

[0011] Furthermore, regarding the generation of adaptive spatiotemporal sampling features of sampling positions, the specific method is as follows:

[0012] (1) We design a scale-adaptive attention mechanism based on vision-based Transformer to learn the appropriate receptive field under the guidance of the query. The specific design method is as follows:

[0013] (1.1) Given a set of learnable queries, each query parameter is defined as [(x, y, z); (w, l, h); (θ)], where (x, y, z) represents the coordinate position, (w, l, h) represents the dimension vector, and (θ) represents the vector angular velocity; z is initially set to 0; the other parameters (x, y, w, l, θ) come from a random Gaussian distribution;

[0014] (1.2) Attach query features to each query box to encode rich instance features and calculate the full pairwise distance between query centers in BEV space n represents the number of queries:

[0015]

[0016] Where (x i ,y i ), (x j ,y j ) represent the coordinates of the i-th and j-th query centers respectively. The attention mechanism not only focuses on the similarity of query features, but also pays attention to the distance between queries;

[0017] (2) For each frame of the feature image, adaptive spatiotemporal sampling is used to adaptively generate a set of sampling offsets {(Δa i , Δb i, Δc i )}, and convert the query pillars to 3D sampling points via these offsets:

[0018]

[0019] Adaptive spatiotemporal sampling points are used to adapt query pillars and query features to cover objects of different sizes;

[0020] (3) For the self-motion and object motion modes, time alignment is performed by warping the sampling points according to the motion, as follows:

[0021] (3.1) Processing object motion: In autonomous driving, the instantaneous speed is equal to the average speed in a short time window, so the speed vector [v a , v b ] Adaptively warp samples to previous timestamps:

[0022] a ti =a i +v a ·(T t -T0), (4)

[0023] b ti =b i +v b ·(T t -T0), (5)

[0024] Where T t represents the timestamp of the previous frame at time t, and T0 represents the timestamp of the current frame. Since the velocity vector is defined in the BEV plane, c t,i With c i are the same;

[0025] (3.2) According to the self-pose distortion sampling points provided by the dataset, the sampling points are first transformed to the global coordinate system, and then transformed to the local coordinate system of the frame at time t:

[0026]

[0027] Where E t = [R|t] is the self-pose of the frame at time t;

[0028] For each feature target at different moments, the target’s distortion sampling points {[a′ t.i , b′ t.i , c′ t.i ]} Projected to each view, because there will be overlap between adjacent views, the projection point will hit one or more views, and the view at this time is named , for each projected view k, there are multi-scale feature maps [F k,j |j∈{1,2,…,N feat}], the features are bilinearly interpolated in the image plane Sampling is done and then weighting is done on the scale axis:

[0029]

[0030] Where N feat is the number of multi-scale feature maps, P k is the projection function of view k; ω ij is the weight of the i-th point on the j-th feature map, which is generated from the query features through a linear transformation.

[0031] Furthermore, the working principle of the scale-adaptive attention mechanism is as follows:

[0032]

[0033] In the formula, is the scale-adaptive attention itself, d is the channel dimension; θ and τ are standard parameters that control the local and global receptive fields of each query, respectively; when θ = 0, self-attention has only a global receptive field; as θ gradually increases, the attention weights of queries located far away gradually decrease, and the receptive field narrows and focuses on nearby targets.

[0034] Furthermore, based on the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer equivariant transformation module, as follows:

[0035] (1) The original points are encoded into transformed equivariant pixel features through shared equivariant sparse convolution. In terms of specific representation, given n 2 Transformation Action Transform point cloud p into N 2 A collection of different points In each voxel set, the feature average of all the original feature points inside is taken as input;

[0036] The voxels are transformed into Encoded as transformation-equivalent pixel features The transformation formula is as follows:

[0037]

[0038] Transformation of isomorphic pixel features Contains diverse features under different rotation and reflection transformations.

[0039] (2) The voxel features are compressed into BEV features through Transformer equivariant BEV conversion. Compressed into BEV features along high latitudes Align the transformed BEV features to a unified coordinate system:

[0040] (2.1) In the BEV coordinate system Determine a set of scene-level grid points in Based on BEV characteristics The transformation action method is to transform this set of grid points Transform to the BEV coordinate system and generate a new set of grid points

[0041] (2.2) Apply a series of bilinear interpolation Ψ on the BEV coordinate system to obtain a set of alignment features The conversion formula is as follows:

[0042]

[0043] like exist There is no corresponding boundary feature point in The interpolation result of is filled with zero values;

[0044] (2.3) Align features on a feature map of scale N The compact feature representation F is obtained by the maximum pooling operation MP(·) * , as follows:

[0045]

[0046] Furthermore, an adaptive hybrid mechanism is used to capture the spatial position relationship between multimodal features and align the point cloud with the image features, as follows:

[0047] The adaptive mixing mechanism includes adaptive channel mixing and adaptive spatial mixing in turn:

[0048] (1) Adaptive channel mixing adaptively enhances channel semantics by transforming features in the channel dimension using query-based dynamic weights. For a given set of adaptive spatiotemporal sampling matrices Where t represents the current time axis position, i represents the current query coordinates, and channel C = d feat Divided into g groups, each with d channels feat / g, through the channel grouping and mixing mechanism, the decoder generates a g·pin offset vector for the query, thereby capturing the correlation between the feature space and the channel dimension, as follows:

[0049]

[0050] F ACM =ReLU(LayerNorm(f t,i ·W C )), (12)

[0051] In the formula is the channel mixed feature output. The linear layer processes each set of matrices separately. The layer normalization is applied to the two dimensions of the mixed output. The dynamic weight W C Shared between different sampling points in 3D space;

[0052] (2) Adaptive spatial mixing (ASM), the process can be described as transposing the channel mixing feature matrix and applying the dynamic kernel to its spatial dimension, as follows:

[0053]

[0054] F ASM =ReLU(LayerNorm(f t,i ·w p )), (14)

[0055] Where w p is a dynamic weight, shared among different channels, where is the spatial mixed feature output, C out is the number of spatial mixing output modes. Adaptive spatial mixing follows adaptive channel mixing.

[0056] (3) Voxelize the output channel mixed features and spatial mixed features, Input feature refinement encoding module:

[0057]

[0058] Where, '⊙' is the Hadamard product, j is the voxel Internal sample point p j The index of β is the characteristic linear transformation function, which is used to generate the characteristic vector β(p j ), the feature vector and the adaptive weight vector α(V i ·p j ) aggregation, ξ is the softmax function, and the attention weight function α calculates the weight V i and p j The weight between

[0059] (4) Combine the encoded channel mixed features and spatial mixed features into F u , as follows:

[0060]

[0061] in, For link operations.

[0062] Furthermore, the image features, point cloud features and BEV features are fused into a multi-scale coupled feature representation with more spatial perception through a multi-scale cross Transformer, as follows:

[0063] (1) For the point-by-point mixed feature F u The coordinates are projected to obtain the corresponding image feature block coordinates P I , point cloud feature block coordinates P p and BEV feature block coordinates P B , the projection equation is as follows:

[0064] P I =D matrix ·R matrix ·T L→C ·F u , (18)

[0065] P p =T C→L ·F u , (19)

[0066] P B =T L→B ·F u , (20)

[0067] Where D matrix , R matrix They are the camera calibration matrix and correction matrix, T L→C is the transformation matrix from LiDAR coordinates to camera, T C→L is the LiDAR coordinate to camera transformation matrix, T L→B is the LiDAR coordinate to BEV conversion matrix;

[0068] (2) Image feature block coordinates P I , point cloud feature block coordinates P p and BEV feature block coordinates P B Input is converted into query Q * , key K * Sum value V * , as shown below:

[0069]

[0070] In the formula is a linear projection matrix. In order to adaptively learn the weight transformation of different modal features for complementary enhancement, the query matrices of image, point cloud and BEV features are further cross-fused through a dynamic fusion mechanism. The new fused query matrix As shown below:

[0071]

[0072] (3) New fusion query With the key K of another mode * Combined and fused through a dynamic weight fusion module to obtain the corresponding multimodal cross fusion weights.

[0073]

[0074] (4) After obtaining the new weight score through the normalization function, the new image, point cloud and BEV feature F′ are obtained through dot product calculation * , as shown below:

[0075]

[0076] (5) Aggregate the features with cross-modal multi-fusion interactions into a new feature as follows:

[0077]

[0078] According to a second aspect of the present invention, the present invention provides a multi-scale spatiotemporal hybrid sampling fusion system for three-dimensional target detection, comprising:

[0079] A receiving unit, used for receiving image data and original point cloud data;

[0080] A generation unit, for extracting features from image data and raw point cloud data using a two-stream spatiotemporal feature extraction network, by using the scale-adaptive force of multi-scale receptive field aggregation features and generating adaptive spatiotemporal sampling features of sampling positions under the guidance of queries;

[0081] Aggregation unit, used to aggregate sparse point cloud coordinate features into lightweight and compact BEV features through the Transformer equivariant transformation module according to the adaptive spatiotemporal sampling features;

[0082] An alignment unit, used to capture the spatial position relationship between multimodal features using an adaptive hybrid mechanism and align point clouds with image features;

[0083] The multi-feature fusion unit is used to fuse image features, point cloud features and BEV features into a multi-scale coupled feature representation with more spatial perception through a multi-scale cross Transformer, and then realize 3D object detection through the RPN and classification regression learning network.

[0084] Furthermore, the generation unit includes a design module, a conversion module and a time alignment execution module:

[0085] Design module for designing scale-adaptive attention mechanism through vision-based Transformer to learn appropriate receptive fields under the guidance of queries;

[0086] A conversion module, for adaptively generating a set of sampling offsets from the query features by using adaptive spatiotemporal sampling for the feature image of each frame, and converting the query pillars into 3D sampling points through these offsets;

[0087] The time alignment execution module is used to perform time alignment by warping sampling points according to motion in self-motion and object motion modes.

[0088] According to a third aspect of the present invention, the present invention provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the above-mentioned multi-scale spatiotemporal mixed sampling fusion method for three-dimensional target detection is adopted.

[0089] According to a fourth aspect of the present invention, the present invention provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the above-mentioned multi-scale spatiotemporal mixed sampling fusion method for three-dimensional target detection.

[0090] The present invention has at least the following beneficial effects:

[0091] 1. In order to solve the problem that representative features cannot be effectively obtained during the downscaling transformation of multimodal data, the present invention proposes a target detection algorithm based on multimodal cross fusion of Transformer and BEV, which aims to combine the spatial coordinate features of point cloud data with the temporal perception information of continuous images by using the BEV features after multi-scale adaptive fusion, and convert different single-scale features into multi-scale coupled representations through multi-scale cross Transformer, so as to extract more spatiotemporal perception information and improve detection accuracy.

[0092] 2. Since the Transformer's multi-head self-attention (MHSA) lacks a clear multi-scale design, the scale connection between global and local features is hindered. To this end, the present invention proposes a two-stream spatiotemporal feature extraction network. By using the scale adaptability of multi-scale receptive field aggregation features and generating adaptive spatiotemporal sampling features of sampling positions under the guidance of queries, this information is integrated into the feature-level information of the Transformer. The enhancement of the global time scale feature connection is combined with the design of the local multi-scale perception field, which improves the detection accuracy of objects of different sizes and categories.

[0093] 3. In order to solve the positioning and matching problem caused by the offset of point cloud and image features, the present invention designs a Transformer equivariant transformation module, which aims to aggregate sparse point cloud coordinate features into a lightweight and compact representation, and uses an adaptive hybrid mechanism to capture the spatial position relationship between multimodal features, so as to better align point cloud and image features, thereby improving the accuracy of target positioning.

[0094] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention;

[0096] Figure 2 It is a schematic diagram of the principle of the dual-stream spatiotemporal feature extraction network of the present invention;

[0097] Figure 3 Schematic diagram of the principle of the Transformer equivariant transformation module of the present invention;

[0098] Figure 4 Schematic diagram of the principle of the adaptive mixing mechanism of the present invention;

[0099] Figure 5 It is a schematic diagram of the principle of multiple cross-fusion of the present invention;

[0100] Figure 6 This is an example of qualitative results on the KITTI dataset of Embodiment 1 of the present invention;

[0101] Figure 7 Schematic diagram of the comparison of BEV results of different 3D object detection algorithms in Example 1 of the present invention;

[0102] Figure 8 This is a comparison diagram of the precision-recall curves of vehicle detection results on the KITTI dataset in Example 1 of the present invention;

[0103] Fig. 9 This is an example of qualitative results on the NuScenses dataset in Example 1 of the present invention;

[0104] Fig.10 A real vehicle platform for real testing in the first embodiment of the present invention;

[0105] Fig.11 This is the qualitative result of the real vehicle platform of Example 1 of the present invention;

[0106] Fig.12 Comparison of average accuracy between the baseline and the method described in the present invention with different object modules;

[0107] Fig.13 This is a comparison of the number of decoder layers of the KITTI validation set in Example 1 of the present invention. DETAILED DESCRIPTION

[0108] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0109] Embodiment 1:

[0110] See also Figure 1-3 The present invention provides a technical solution: a multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection, which specifically includes the following steps:

[0111] S1. Receive image data and raw point cloud data, which are obtained by collecting synchronous data from a vehicle-mounted camera and a laser radar;

[0112] S2. A two-stream spatiotemporal feature extraction network is used to extract features from image data and raw point cloud data. The scale adaptation of multi-scale receptive field aggregation features is used to generate adaptive spatiotemporal sampling features of sampling positions under the guidance of queries, as follows:

[0113] (1) We design a scale-adaptive attention mechanism based on vision-based Transformer to learn the appropriate receptive field under the guidance of the query. The specific design method is as follows:

[0114] (1.1) Given a set of learnable queries, each query parameter is defined as [(x, y, z); (w, l, h); (θ)], where (x, y, z) represents the coordinate position, (w, l, h) represents the dimension vector, and (θ) represents the vector angular velocity; z is initially set to 0; the other parameters (x, y, w, l, θ) come from a random Gaussian distribution;

[0115] (1.2) Attach query features to each query box to encode rich instance features and calculate the full pairwise distance between query centers in BEV space n represents the number of queries:

[0116]

[0117] Where (x i ,y i ), (x j ,y j ) represent the coordinates of the i-th and j-th query centers respectively. The attention mechanism not only focuses on the similarity of query features, but also pays attention to the distance between queries;

[0118] The scale-adaptive attention mechanism works as follows:

[0119]

[0120] In the formula, is the scale-adaptive attention itself, d is the channel dimension; θ and τ are standard parameters that control the local and global receptive fields of each query, respectively; when θ = 0, self-attention has only a global receptive field; as θ gradually increases, the attention weights of queries located far away gradually decrease, and the receptive field narrows and focuses on nearby targets;

[0121] (2) For each frame of the feature image, adaptive spatiotemporal sampling is used to adaptively generate a set of sampling offsets {(Δa i , Δb i , Δc i )}, and convert the query pillars to 3D sampling points via these offsets:

[0122]

[0123] Adaptive spatiotemporal sampling points are more flexible than the deformable attention in BEVFormer because they can adapt to query pillars and query features to cover objects of different sizes. These points are not limited to the query because we do not limit the range of sampling offsets;

[0124] (3) Furthermore, temporal alignment is performed by warping the sampling points according to the motion;

[0125] In autonomous driving, there are two kinds of motion: ego motion and object motion. Ego motion describes the motion of the car from its own perspective as it moves through the environment, while object motion refers to the motion of other objects in the environment as they move around the autonomous car.

[0126] For both ego-motion and object-motion modes, temporal alignment is performed by warping the sampling points according to the motion, as follows:

[0127] (3.1) Processing object motion: In autonomous driving, the instantaneous speed is equal to the average speed in a short time window, so the speed vector [v a , v b ] Adaptively warp samples to previous timestamps:

[0128] a ti =a i +v a ·(T t -T0), (4)

[0129] b ti =b i +v b ·(T t -T0), (5)

[0130] Where T t represents the timestamp of the previous frame at time t, and T0 represents the timestamp of the current frame. Since the velocity vector is defined in the BEV plane, c t,i With c i are the same;

[0131] (3.2) According to the self-pose distortion sampling points provided by the dataset, the sampling points are first transformed to the global coordinate system, and then transformed to the local coordinate system of the frame at time t:

[0132]

[0133] Where E t = [R|t] is the self-pose of the frame at time t;

[0134] For each feature target at different moments, the target’s distortion sampling points {[a′ t.i , b′ t.i , c′ t.i ]} Projected to each view, because there will be overlap between adjacent views, the projection point will hit one or more views, and the view at this time is named , for each projected view k, there are multi-scale feature maps [F k,j |j∈{1,2,…,N feat}], the features are bilinearly interpolated in the image plane Sampling is done and then weighting is done on the scale axis:

[0135]

[0136] Where N featis the number of multi-scale feature maps, P k is the projection function of view k; ω ij is the weight of the i-th point on the j-th feature map, which is generated from the query feature through a linear transformation;

[0137] S3. Based on the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact bird’s-eye view (BEV) features through the Transformer equivariant transformation module, as follows:

[0138] (1) Efficiently encode the original points into transformed equivariant pixel features through shared equivariant sparse convolution, which has the following two characteristics: (a) highly shared weights between transformation channels; (b) input transformations with different rotation angles and reflections;

[0139] In terms of specific expression, given N 2 Transformation Action Transform point cloud p into N 2 A collection of different points In each voxel set, the feature average of all the original feature points inside is taken as input;

[0140] The voxels are transformed into Encoded as transformation-equivalent pixel features The transformation formula is as follows:

[0141]

[0142] Transformation of isomorphic pixel features Contains diverse features under different rotation and reflection transformations;

[0143] (2) Transformation of isomorphic pixel features It contains a large number of different rotation and reflection transformation channels. In order to solve the calculation and memory pressure caused by a large number of transformation channels, it is proposed to compress voxel features into BEV features through Transformer equivariant BEV conversion to reduce the amount of data;

[0144] Voxel features Compressed into BEV features along high latitudes The converted BEV features are located in different coordinate systems due to different transformation channels. Therefore, the converted BEV features need to be aligned to a unified coordinate system:

[0145] (2.1) In the BEV coordinate system Determine a set of scene-level grid points in Based on BEV characteristics The transformation action method is to transform this set of grid points Transform to the BEV coordinate system and generate a new set of grid points

[0146] (2.2) Apply a series of bilinear interpolation Ψ on the BEV coordinate system to obtain a set of alignment features The conversion formula is as follows:

[0147]

[0148] like exist There is no corresponding boundary feature point in The interpolation result of is filled with zero values;

[0149] (2.3) Align features on a feature map of scale N The compact feature representation F is obtained by the maximum pooling operation MP(·) * , as follows:

[0150]

[0151] S4. Use the adaptive hybrid mechanism to capture the spatial position relationship between multimodal features and align the point cloud with the image features as follows:

[0152] After obtaining sampling features from different time nodes and positions, the next key is how to fuse the sampling features to obtain highly coupled multimodal information, thereby realizing the alignment of point cloud and image information. To this end, this embodiment designs an adaptive mixing mechanism based on adamixer, which includes adaptive channel mixing and adaptive spatial mixing in sequence;

[0153] (1) Adaptive channel mixing adaptively enhances channel semantics by transforming features in the channel dimension using query-based dynamic weights. For a given set of adaptive spatiotemporal sampling matrices Where t represents the current time axis position, i represents the current query coordinates, and channel C = d feat Divided into g groups, each with d channels feat / g; Through the channel grouping mixing mechanism, the decoder can generate a g·pin offset vector for the query to enrich the diversity of sampling points and exploit the richer spatial structure of these points, thereby capturing the correlation of feature space and channel dimensions:

[0154]

[0155] F ACM =ReLU(LayerNorm(f t,i ·W C)), (12)

[0156] In the formula is the channel mixed feature output. The linear layer processes each set of matrices separately. The layer normalization is applied to the two dimensions of the mixed output. The dynamic weight W C Shared between different sampling points in 3D space;

[0157] (2) In order to make the query adaptive to the spatial structure of different sampled features, adaptive spatial mixing (Adaptive spatial mixing, ) is introduced. The process can be described as transposing the channel mixing feature matrix and applying a dynamic kernel to its spatial dimension:

[0158]

[0159] F ASM =ReLU(LayerNorm(f t,i ·w p )), (14)

[0160] Where w p is a dynamic weight, shared among different channels, where is the spatial mixed feature output, C out is the number of spatial mixing output modes. Adaptive spatial mixing follows adaptive channel mixing.

[0161] (3) Voxelize the output channel mixed features and spatial mixed features, i∈1,2,…N. Input feature refinement encoding module:

[0162]

[0163] Where, '⊙' is the Hadamard product, j is the voxel Internal sample point p j The index of β is the characteristic linear transformation function, which is used to generate the characteristic vector β(p j ), the feature vector and the adaptive weight vector α(V i ·p j ) aggregation, ξ is the softmax function, and the attention weight function α calculates the weight V i and p j The weight between

[0164] (4) Combine the encoded channel mixed features and spatial mixed features into F u , as follows:

[0165]

[0166] in, For link operations;

[0167] S5. Through the multi-scale cross Transformer, the image features, point cloud features and BEV features are fused into a multi-scale coupled feature representation with more spatial perception, and then 3D object detection is realized through the learning network of RPN and classification regression, as follows:

[0168] (1) For the point-by-point mixed feature F u The coordinates are projected to obtain the corresponding image feature block coordinates P I , point cloud feature block coordinates P p and BEV feature block coordinates P B , the projection equation is as follows:

[0169] P I =D matrix ·R matrix ·T L→C ·F u , (18)

[0170] P p =T C→L ·F u , (19)

[0171] P B =T L→B ·F u , (20)

[0172] Where D matrix , R matrix They are the camera calibration matrix and correction matrix, T L→C is the transformation matrix from LiDAR coordinates to camera, T C→L is the LiDAR coordinate to camera transformation matrix, T L→B is the LiDAR coordinate to BEV conversion matrix;

[0173] (2) Image feature block coordinates P I , point cloud feature block coordinates P p and BEV feature block coordinates P B Input is converted into query Q * , key K * Sum value V * , as shown below:

[0174]

[0175] In the formula is a linear projection matrix. In order to adaptively learn the weight transformation of different modal features for complementary enhancement, the query matrices of image, point cloud and BEV features are further cross-fused through a dynamic fusion mechanism. The new fused query matrix As shown below:

[0176]

[0177] (3) New fusion query It is combined with the key K* of another modality and passed through a dynamic weight fusion module to obtain the corresponding multimodal cross fusion weights.

[0178]

[0179] (4) After obtaining the new weight score through the normalization function, the new image, point cloud and BEV feature F' are obtained through dot product calculation * , as shown below:

[0180]

[0181] (5) Aggregate the features with cross-modal multi-fusion interactions into a new feature as follows:

[0182]

[0183] The specific process of multi-cross fusion Transformer is shown in Table 1:

[0184]

[0185]

[0186] S6. The overall loss function is as follows:

[0187] The overall loss function includes the classification loss Lcls for objectness and category, the regression loss Lreg for prediction box, the prediction loss Ldff for the two-stream spatiotemporal feature extraction network, and the conversion loss Ltrans for Transformer equivariant BEV conversion. The overall loss function is as follows:

[0188] Ltotal=λclsLcls+λregLreg+λdffLdff+λtransLtrans(34)where λcls, λreg, λdff, and λtran represent the regularization weight coefficients of classification loss, regression loss, prediction loss, and conversion loss, respectively.

[0189] In order to balance the imbalance of positive and negative samples, focus loss is used to balance the samples of classification tasks and prediction tasks respectively, and smoothed-L1 is used to balance the regression loss. Focus loss is also used to process the conversion loss of Transformer equivariant BEV conversion:

[0190]

[0191] Where ρ is the 2D pixel feature region with valid depth labels, is the accurate lidar true value.

[0192] Next, the technical solution of the present invention is further described in conjunction with a specific embodiment, but the protection scope of the present invention is not limited to this embodiment:

[0193] 1. Experimental results and analysis

[0194] This paper evaluates the proposed SMSFusion algorithm on the challenging KITTI3D

[28] and NuScenes

[29] target detection datasets and the latest open dataset Waymo

[30] . It is also compared with other SOTA methods in previous target detection. In addition, extensive ablation experiments are conducted to evaluate the role of different modules. Finally, the proposed method is used on a real autonomous driving vehicle platform to verify the effectiveness of the algorithm in real road scenes.

[0195] 1.1 Experimental Results on KITTI Dataset

[0196] Table 2 is a comparison of the detection performance of the technical solution of this embodiment and other target detection algorithms in the KITTI test set car category. It can be seen from Table 2 that the comprehensive detection performance of the technical solution of this embodiment is better than other current 3D target detection algorithms; compared with the baseline ACF-Net of the technical solution of this embodiment, the average detection accuracy AP of different levels of the technical solution of this embodiment is 3D The average precision AP is improved by 1.53 / 0.58 / 0.33 respectively. Compared with the excellent TED

[37] , the average precision AP is 3D Improved by 0.24.

[0197] Table 3 shows the comparison results of the detection performance of the technical solution of this embodiment on the pedestrian and bicycle categories in the KITTI test set. It can be seen from Table 2 that, compared with the baseline ACF-Net of the technical solution of this embodiment, the average detection accuracy of the pedestrian target of the technical solution of this embodiment is improved by 1.10 / 0.03 / 0.06 at different levels of difficulty, while the detection results of the bicycle target are improved by 0.46 / 0.66 / 1.43 at different levels of difficulty compared with the baseline. The detection schematic diagram of the technical solution of this embodiment on the KITTI dataset is shown in Figure 6 shown.

[0198] Table 2 Detection performance of car category in KITTI test set

[0199] Table2DetectionperformanceofautomotivecategoriesontheKITTI

[0200] testset

[0201]

[0202]

[0203] Table 3 Detection performance of pedestrian and bicycle categories in the KITTI test set

[0204]

[0205] To verify the 3D orientation of the detection results, the 3D bounding box is projected onto BEV. The BEV detection results are compared with the 3D object detection algorithm. Figure 7 As shown in Figure 2, SMSFusion has the best mAP values ​​on all subsets, especially compared with M3DETR

[39] , which is improved by 6.51%, 4.32%, and 3.9% on the “easy”, “medium”, and “difficult” subsets, respectively. The BEV detection results show that the object localization performance is effectively improved by fusing complex spatial information.

[0206] Comparison of the precision-recall curves of vehicle detection results on the KITTI dataset is shown in the figure below. Figure 8 Compared with other algorithms, the proposed FFAFormer algorithm shows higher precision at high recall, which shows that the proposed algorithm has overall robustness and stability.

[0207] 1.2 Experimental results on the NuScences dataset

[0208] In order to further test the effectiveness and versatility of the technical solution of this embodiment, the verification algorithm has good detection performance for three-dimensional targets. The algorithm was verified on the more challenging NuScences dataset. As shown in Table 4, the detection results of the technical solution of this embodiment and other 3D target detection algorithms on different categories of targets on NuScences are compared. It can be seen from the table that in terms of the important indicators NDS and mAP in the NuScence dataset, the results are improved by 0.2 / 1.1 compared with the optimal VFF

[46] . It can be seen that the detection ability of the technical solution of this embodiment on different datasets is still robust. The detection results of the technical solution of this embodiment on the NuScences dataset are shown in the figure. Fig. 9 shown.

[0209] Table 4 Performance comparison on NuScenes dataset

[0210]

[0211] 1.3 Experimental results on the Waymo dataset

[0212] This embodiment also compares the proposed algorithm with the baseline ACF-Net

[44] on the Waymo dataset. The results are shown in Table 5. The results show that the technical solution of this embodiment performs well in different levels of tests on the Waymo dataset. Compared with ACF-Net

[44] , the mAP / mAPH on L1 / L2 are improved by 1.10 / 0.78 and 1.14 / 0.86 respectively. The improvement is particularly significant when detecting smaller targets such as pedestrians and bicycles, indicating that the technical solution of this embodiment has excellent performance in detecting small targets at a long distance.

[0213] Table 5 Performance comparison on the Waymo dataset

[0214]

[0215] 1.4 Experiments on a real vehicle platform

[0216] Simulation experiments conducted only on the dataset may not fully demonstrate the performance of SMSFusion in real scenarios. Therefore, a series of tests were conducted on the real vehicle platform, such as Fig.10 As shown in the figure, it includes 80-beam LiDAR and stereo cameras, millimeter-wave radar and two GNSS. 3D object detection is mainly performed by LiDAR and vehicle cameras. The qualitative results of the real vehicle platform in a certain city are shown in the figure. Fig.11 shown.

[0217] In order to more clearly demonstrate the effectiveness of each module of the proposed algorithm, for different target detection objects, the average accuracy of different object modules between the baseline and the proposed algorithm is compared. Fig.12 As shown in Table 6, the detection results of different vehicle targets on the real vehicle platform are compared. It can be seen from Table 6 that in the complex and changeable real environment, the addition of various modules of the proposed algorithm can effectively improve the overall performance of the model.

[0218] Table 6 Performance of vehicle categories on the real vehicle platform

[0219]

[0220] 1.5 Ablation Experiment

[0221] A large number of ablation experiments are performed to verify the effectiveness of each component of the proposed algorithm:

[0222] The effectiveness of each module is shown in Table 7. Experiment (a) is based on the CenterPoint

[44] experimental data. The average precision results of 40 car recall locations calculated on the KITTI validation set are compared. As shown in Table 7, each module component can provide effective assistance to the accuracy of target detection results of different difficulty levels.

[0223] Table 7. The impact of different parts on the KITTI validation set.

[0224] The results are evaluated by AP calculated over 40 car-class recall locations.

[0225]

[0226] In order to verify whether the adaptive hybrid mechanism of this embodiment is more efficient than the conventional hybrid query mechanism (CHQM), the variable mask offset and weight of the conventional hybrid query mechanism are determined only by a single camera query, while the adaptive hybrid mechanism is jointly updated and determined by feature queries in different domains to obtain more accurate variable mask offsets and feature weights. As shown in Table 8, compared with the baseline and single-camera query on the KITTI validation set, the average detection accuracy of the adaptive hybrid mechanism is improved by 7% and 12%, respectively.

[0227] Table 8 Comparison of different query types on the KITTI validation set.

[0228]

[0229] Verify the effectiveness of each part of the multi-cross fusion transformer, as shown in Table 9. Experiment (A) associates the camera features with the voxel features through the original naive fusion method, and fuses the voxel features of the lidar through simple summation. Experiments (B) and (C) add dynamic fusion mechanism and multi-modal cross fusion mechanism in turn. The results show that each component of the multi-cross fusion transformer module plays an improvement role.

[0230] Table 9 Ablation study of multi-cross-fusion transformer.

[0231] The results are evaluated using AP calculated with 40 recall positions.

[0232]

[0233] FEN-DSS module behavior analysis. The performance trend of the KITTI validation set is related to the number of decoder layers, such as Fig.13 As shown, in the KITTI dataset, the performance steadily improves as the number of layers reaches four, indicating that the adaptive spatiotemporal sampling mechanism continuously improves the quality of features for 3D object detection.

[0234] By evaluating the technical solution of this embodiment at different distance levels and occlusion levels, it can be known under what circumstances this embodiment improves the baseline the most, as shown in Table 10. Object detection affected by different distances and occlusions has a significant improvement, which is because the multi-scale depth quasi-prediction module enhances the interactive depth of point features of different scales, thereby refining the features.

[0235] Table 10 Performance at different distances and different degrees of occlusion, evaluation results and the medium level of the car class in 3DAP calculation 40 recall positions

[0236]

[0237] In summary, in order to solve the problem that multimodal data cannot adapt to objects of different sizes and categories during the downscaling process, which ultimately leads to the loss of positioning information. This embodiment proposes a multi-scale spatiotemporal hybrid sampling fusion method (SMSFusion) for three-dimensional target detection. First, through a dual-stream spatiotemporal feature extraction network, adaptive spatiotemporal sampling features are generated under the guidance of queries. At the same time, a Transformer equivariant transformation module is designed and an adaptive hybrid mechanism is used to better align point clouds with image features. Finally, multiple cross-Transformers fuse multiple unimodal data into a hybrid representation with more spatial perception, thereby achieving accurate 3D object detection. Experimental results on KITTI, NuScenes, and Waymo datasets show that SMSFusion can handle challenging detection tasks well. SMSFusion has also been applied to autonomous driving platforms for real road testing. The results show that this method has sufficient generalization ability in complex real-world environments.

[0238] Embodiment 2:

[0239] The present invention provides a multi-scale spatiotemporal hybrid sampling and fusion system for three-dimensional target detection, comprising:

[0240] A receiving unit, used for receiving image data and original point cloud data;

[0241] A generation unit, for extracting features from image data and raw point cloud data using a two-stream spatiotemporal feature extraction network, by using the scale-adaptive force of multi-scale receptive field aggregation features and generating adaptive spatiotemporal sampling features of sampling positions under the guidance of queries;

[0242] Aggregation unit, used to aggregate sparse point cloud coordinate features into lightweight and compact BEV features through the Transformer equivariant transformation module according to the adaptive spatiotemporal sampling features;

[0243] An alignment unit, used to capture the spatial position relationship between multimodal features using an adaptive hybrid mechanism and align point clouds with image features;

[0244] The multi-feature fusion unit is used to fuse image features, point cloud features and BEV features into a multi-scale coupled feature representation with more spatial perception through a multi-scale cross Transformer, and then realize 3D object detection through the RPN and classification regression learning network.

[0245] Furthermore, the generation unit includes a design module, a conversion module and a time alignment execution module:

[0246] Design module for designing scale-adaptive attention mechanism via vision-based Transformer to learn appropriate receptive fields under the guidance of queries;

[0247] A conversion module, for adaptively generating a set of sampling offsets from the query features by using adaptive spatiotemporal sampling for the feature image of each frame, and converting the query pillars into 3D sampling points through these offsets;

[0248] The time alignment execution module is used to perform time alignment by warping sampling points according to motion in self-motion and object motion modes.

[0249] Specifically, the above-mentioned receiving unit, generating unit, aggregation unit, alignment unit and multi-feature fusion unit can be embedded in a computer processing system, and the computer calls the above-mentioned units to complete the task of equivalent modeling based on the multi-scale spatiotemporal mixed sampling fusion method for three-dimensional target detection provided above; the above-mentioned receiving unit, generating unit, aggregation unit, alignment unit and multi-feature fusion unit can perform operations according to the specific steps given in the above-mentioned multi-scale spatiotemporal mixed sampling fusion method for three-dimensional target detection.

[0250] It should be noted that it should be understood that the division of the various units of the above system is only the division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated, and these units can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some units can also be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware. For example, the generation unit can be a separately established processing element, or it can be integrated in a chip of the above-mentioned device. In addition, it can also be stored in the memory of the above-mentioned device in the form of program code, and a processing element of the above-mentioned device calls and executes the functions of the above signal processing unit. The implementation of other units is similar. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0251] For example, the above units may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more digital singnal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a certain unit above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0252] Embodiment three:

[0253] The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the above-mentioned multi-scale spatiotemporal mixed sampling fusion method for three-dimensional target detection is adopted.

[0254] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory. For example, the terminal device can also include input and output devices, network access devices and buses.

[0255] Furthermore, the processor may be a central processing unit (CPU). Of course, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. may also be used according to actual usage. The general-purpose processor may be a microprocessor or any conventional processor, etc., and this application does not limit this.

[0256] Embodiment 4:

[0257] The present invention provides a storage medium containing computer executable instructions, which are used to execute the above-mentioned multi-scale spatiotemporal mixed sampling fusion method for three-dimensional target detection when executed by a computer processor.

[0258] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer-readable medium includes but is not limited to the above-mentioned components.

[0259] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0260] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on", "installed on", "fixed on" or "set on" another element, it can be directly on the other element or there can also be a centered element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be a centered element at the same time. The terms "vertical", "horizontal", "up", "down", "left", "right" and similar expressions used herein are only for illustrative purposes and are not intended to be the only implementation method.

[0261] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0262] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. Multi-scale spatiotemporal hybrid sampling fusion method for 3D object detection, It is characterized in that The specific steps include: Receiving image data and raw point cloud data; A two-stream spatiotemporal feature extraction network is used to extract features from image data and raw point cloud data. The scale-adaptive force of the multi-scale receptive field aggregation features is used to generate adaptive spatiotemporal sampling features of the sampling position under the guidance of the query. According to the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer equivariant transformation module; The adaptive hybrid mechanism is used to capture the spatial position relationship between multimodal features and align point clouds with image features. The image features, point cloud features and BEV features are fused into a multi-scale coupled feature representation with more spatial perception through a multi-scale cross Transformer, and then 3D object detection is achieved through a learning network of RPN and classification regression.

2. The multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection according to claim 1, It is characterized in that Regarding the adaptive spatiotemporal sampling features for generating sampling positions, the specific method is as follows: (1) We design a scale-adaptive attention mechanism based on vision-based Transformer to learn the appropriate receptive field under the guidance of the query. The specific design method is as follows: (1.1) Given a set of learnable queries, each query parameter is defined as [(x, y, z); (w, l, h); (θ)], where (x, y, z) represents the coordinate position, (w, l, h) represents the dimension vector, and (θ) represents the vector angular velocity; z is initially set to 0; the other parameters (x, y, w, l, θ) come from a random Gaussian distribution; (1.2) Attach query features to each query box to encode rich instance features and calculate the full pairwise distance between query centers in BEV space n represents the number of queries: Where (x i ,y i ), (x j ,y j ) represent the coordinates of the i-th and j-th query centers respectively. The attention mechanism not only focuses on the similarity of query features, but also pays attention to the distance between queries; (2) For each frame of the feature image, adaptive spatiotemporal sampling is used to adaptively generate a set of sampling offsets {(Δa i , Δb i , Δc i )}, and convert the query pillars to 3D sampling points via these offsets: Adaptive spatiotemporal sampling points are used to adapt query pillars and query features to cover objects of different sizes; (3) For the self-motion and object motion modes, time alignment is performed by warping the sampling points according to the motion, as follows: (3.1) Processing object motion: In autonomous driving, the instantaneous speed is equal to the average speed in a short time window, so the speed vector [v a , v b ] Adaptively warp samples to previous timestamps: a ti =a i +v a ·(T t -T 0 ), (4) b ti =b i +v b ·(T t -T 0 ), (5) Where T t Indicates the timestamp of the previous frame at time t, and T 0 Indicates the timestamp represented by the current frame. Since the velocity vector is defined in the BEV plane, c t,i With c i are the same; (3.2) According to the self-pose distortion sampling points provided by the dataset, the sampling points are first transformed to the global coordinate system, and then transformed to the local coordinate system of the frame at time t: Where E t = [R|t] is the self-pose of the frame at time t; For each feature target at different moments, the target’s distortion sampling points {[a′ t.i , b′ t.i , c′ t.i ]} Projected to each view, because there will be overlap between adjacent views, the projection point will hit one or more views, and the view at this time is named For each projected view k, there are multi-scale feature maps [F k,j |j∈{1,2,…,N feat }], the features are bilinearly interpolated in the image plane Sampling is done and then weighting is done on the scale axis: Where N feat is the number of multi-scale feature maps, P k is the projection function of view k; ω ij is the weight of the i-th point on the j-th feature map, which is generated from the query features through a linear transformation.

3. The multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection according to claim 2, Features: The working principle of the scale-adaptive attention mechanism is as follows: In the formula, is the scale-adaptive attention itself, d is the channel dimension; θ and τ are standard parameters that control the local and global receptive fields of each query, respectively; when θ = 0, self-attention has only a global receptive field; as θ gradually increases, the attention weights of queries located far away gradually decrease, and the receptive field narrows and focuses on nearby targets.

4. The multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection according to claim 3, It is characterized in that According to the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer equivariant transformation module, as follows: (1) Encode the original points into transformed equivariant pixel features through shared equivariant sparse convolution. In terms of specific representation, given N 2 Transformation Action Transform point cloud p into N 2 A collection of different points In each voxel set, the feature average of all the original feature points inside is taken as input; The voxels are transformed into Encoded as transformation-equivalent pixel features The transformation formula is as follows: Transformation of isomorphic pixel features Contains diverse features under different rotation and reflection transformations (2) The voxel features are compressed into BEV features through Transformer equivariant BEV conversion. Compressed into BEV features along high latitudes Align the transformed BEV features to a unified coordinate system: (2.1) In the BEV coordinate system Determine a set of scene-level grid points in Based on BEV characteristics The transformation action method is to transform this set of grid points Transform to the BEV coordinate system and generate a new set of grid points (2.2) Apply a series of bilinear interpolation ψ(,) in the BEV coordinate system to obtain a set of alignment features The conversion formula is as follows: like exist There is no corresponding boundary feature point in The interpolation result of is filled with zero values; (2.3) Align features on a feature map of scale N The compact feature representation F is obtained by the maximum pooling operation MP(·) * , as follows:

5. The multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection according to claim 4, It is characterized in that The spatial position relationship between multimodal features is captured using an adaptive hybrid mechanism to align point clouds with image features as follows: The adaptive mixing mechanism includes adaptive channel mixing and adaptive spatial mixing in turn: (1) Adaptive channel mixing adaptively enhances channel semantics by transforming features in the channel dimension using query-based dynamic weights. For a given set of adaptive spatiotemporal sampling matrices Where t represents the current time axis position, i represents the current query coordinates, and channel C = d feat Divided into g groups, each with d channels feat / g, through the channel grouping and mixing mechanism, the decoder generates a g·pin offset vector for the query, thereby capturing the correlation between the feature space and the channel dimension, as follows: F ACM =ReLU(LayerNorm(f t,i ·W C )), (12) In the formula is the channel mixed feature output. The linear layer processes each set of matrices separately. The layer normalization is applied to the two dimensions of the mixed output. The dynamic weight W C Shared between different sampling points in 3D space; (2) Adaptive spatial mixing (ASM), the process can be described as transposing the channel mixing feature matrix and applying the dynamic kernel to its spatial dimension, as follows: F ASM =ReLU(LayerNorm(f t,i ·w p )), (14) Where w p is a dynamic weight, shared among different channels, where is the spatial mixed feature output, C out is the number of spatial mixing output modes. Adaptive spatial mixing follows adaptive channel mixing. (3) Voxelize the output channel mixed features and spatial mixed features, i∈1,2,…N. Input feature refinement encoding module: Where, '⊙' is the Hadamard product, j is the voxel Internal sample point p j The index of β is the characteristic linear transformation function, which is used to generate the characteristic vector β(p j ), the feature vector and the adaptive weight vector α(V i ·p j ) aggregation, ξ is the softmax function, and the attention weight function α calculates the weight V i and p j The weight between (4) Combine the encoded channel mixed features and spatial mixed features into F u , as follows: in, For link operations.

6. The multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection according to claim 5, It is characterized in that The image features, point cloud features and BEV features are fused into a multi-scale coupled feature representation with more spatial perception through a multi-scale cross Transformer, as follows: (1) For the point-by-point mixed feature F u The coordinates are projected to obtain the corresponding image feature block coordinates P I , point cloud feature block coordinates P p and BEV feature block coordinates P B , the projection equation is as follows: P I =D matrix ·R matrix ·T L→C ·F u (18) P p =T C→L ·F u , (19) P B =T L→B ·F u , (20) Where D matrix , R matrix They are the camera calibration matrix and correction matrix, T L→C is the transformation matrix from LiDAR coordinates to camera, T C→L is the LiDAR coordinate to camera transformation matrix, T L→B is the LiDAR coordinate to BEV conversion matrix; (2) Image feature block coordinates P I , point cloud feature block coordinates P p and BEV feature block coordinates P B Input is converted into query Q * , key K * Sum value V * , as shown below: In the formula is a linear projection matrix. In order to adaptively learn the weight transformation of different modal features for complementary enhancement, the query matrices of image, point cloud and BEV features are further cross-fused through a dynamic fusion mechanism. The new fused query matrix As shown below: (3) New fusion query With the other mode key K * Combined and fused through a dynamic weight fusion module to obtain the corresponding multimodal cross fusion weights. (4) After obtaining the new weight score through the normalization function, the new image, point cloud and BEV feature F′ are obtained through dot product calculation * , as shown below: (5) Aggregate the features with cross-modal multi-fusion interactions into a new feature as follows:

7. A multi-scale spatiotemporal hybrid sampling fusion system for 3D object detection. It is characterized in that include: A receiving unit, used for receiving image data and original point cloud data; A generation unit, for extracting features from image data and raw point cloud data using a two-stream spatiotemporal feature extraction network, by using the scale-adaptive force of multi-scale receptive field aggregation features and generating adaptive spatiotemporal sampling features of sampling positions under the guidance of queries; Aggregation unit, used to aggregate sparse point cloud coordinate features into lightweight and compact BEV features through the Transformer equivariant transformation module according to the adaptive spatiotemporal sampling features; An alignment unit, used to capture the spatial position relationship between multimodal features using an adaptive hybrid mechanism and align point clouds with image features; The multi-feature fusion unit is used to fuse image features, point cloud features and BEV features into a multi-scale coupled feature representation with more spatial perception through a multi-scale cross Transformer, and then realize 3D object detection through the RPN and classification regression learning network.

8. The multi-scale spatiotemporal hybrid sampling and fusion system for three-dimensional target detection according to claim 7, Features: The generation unit includes a design module, a conversion module and a time alignment execution module: Design module for designing scale-adaptive attention mechanism via vision-based Transformer to learn appropriate receptive fields under the guidance of queries; A conversion module, for adaptively generating a set of sampling offsets from the query features by using adaptive spatiotemporal sampling for the feature image of each frame, and converting the query pillars into 3D sampling points through these offsets; The time alignment execution module is used to perform time alignment by warping sampling points according to motion in self-motion and object motion modes.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, It is characterized in that The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection described in any one of claims 1 to 6 is adopted.

10. A storage medium containing computer executable instructions, It is characterized in that The computer executable instructions are used to execute the multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional object detection as described in any one of claims 1 to 6 when executed by a computer processor.

Citation Information

Patent Citations

  • Three-dimensional target detection method and system based on point cloud-image multi-cross mixing and storage medium

    CN116664856A

  • Multi-modal fusion 3D-BEV target detection method and system for unmanned aerial vehicle cluster tasks

    CN117671541A

Cited By

  • Multi-agent sparse three-dimensional sampling point learning method and system

    CN119579909A

  • 3D target detection method and device applied to intelligent networked automobile, medium and product

    CN121789169A