Multi-scale spatio-temporal hybrid sampling fusion method and system for three-dimensional target detection

Abstract

The application discloses a multi-scale space-time hybrid sampling fusion method and system for three-dimensional target detection, and relates to the technical field of automatic driving target detection.The application comprises the following steps: receiving image data and original point cloud data; using a double-flow space-time feature extraction network to perform feature extraction on the image data and the original point cloud data, using scale adaptive force of multi-scale receptive field aggregation features, and generating adaptive space-time sampling features of sampling positions under the guidance of query; and according to the adaptive space-time sampling features, aggregating sparse point cloud coordinate features into light and compact BEV features through a variable conversion module such as a Transformer.The application combines the enhancement of global time scale feature connection with the design of a local multi-scale perception field, improves the detection accuracy of objects of different sizes and categories, converts different single-scale features into multi-scale coupled representations through a multi-scale cross-Transformer, and extracts more space-time perception information to improve the detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving target detection technology, specifically to a multi-scale spatiotemporal hybrid sampling and fusion method and system for three-dimensional target detection. Background Technology

[0002] In the field of autonomous driving, 3D object detection is a crucial problem, enabling vehicles to accurately perceive and interpret their surroundings. Cameras and LiDAR sensors play important roles in achieving 3D perception in autonomous vehicles, each with its own unique advantages and disadvantages. LiDAR sensors provide high-quality distance information, but their echoes become sparser with increasing distance, particularly affecting the resolution of distant objects. In contrast, cameras provide rich texture and color information but lack direct 3D depth measurement capabilities. Therefore, combining these two sensors can yield more comprehensive information. However, current fusion methods face several challenges. First, training multimodal networks is more challenging than single-modal networks due to their features at different scales, larger parameter count, and susceptibility to overfitting. Second, overfitting and generalization speeds differ between modalities, and end-to-end fusion models may lead to suboptimal solutions. Furthermore, the misalignment between LiDAR bird's-eye view (BEV) features and image features in current fusion architectures can degrade performance.

[0003] To address the aforementioned issues, existing technologies employ sequential fusion methods, which can fuse image predictions at the input of LiDAR-only detectors or by fusing the output of 3D LiDAR detectors with image predictions. However, current sequential methods have limited fusion capabilities, potentially being severely restricted by 2D predictions or only performing fusion at sparse projected point clouds. Finally, effective LiDAR data augmentation techniques can accelerate the convergence of LiDAR-only methods, but since the LiDAR ground reality objects added to the image data lack corresponding relationships, they cannot be directly applied to fusion methods. Therefore, we propose a multi-scale spatiotemporal hybrid sampling fusion method and system for 3D target detection. Summary of the Invention

[0004] According to a first aspect of the present invention, the object of the present invention is to provide a multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection, specifically including the following steps:

[0006] Receive image data and raw point cloud data;

[0007] A dual-stream spatiotemporal feature extraction network is used to extract features from image data and raw point cloud data. The scale adaptation force of feature aggregation by multi-scale receptive field is used to generate adaptive spatiotemporal sampling features of sampling location under query guidance.

[0008] Based on the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer equivariant transformation module.

[0009] An adaptive fusion mechanism is used to capture the spatial relationships between multimodal features, aligning point clouds with image features;

[0010] Image features, point cloud features, and BEV features are fused into a multi-scale coupled feature representation with greater spatial awareness by using a multi-scale cross-transformer. Then, 3D object detection is achieved through a learning network of RPN and classification regression.

[0011] Furthermore, the specific method for generating adaptive spatiotemporal sampling features for sampling locations is as follows:

[0012] (1) A scale-adaptive attention mechanism based on vision is designed using Transformer to learn an appropriate receptive field under the guidance of the query. The specific design method is as follows:

[0013] (1.1) Given a set of learnable queries, where each query parameter is defined as [(x, y, z); (w, l, h); (θ)], where (x, y, z) represents the coordinate position, (w, l, h) represents the dimension vector, and (θ) represents the vector angular velocity; z is initially set to 0; the other parameters (x, y, w, l, θ) are from a random Gaussian distribution;

[0014] (1.2) Query features are appended to each query box to encode rich instance features, and the all-pair distance between query centers in the BEV space is calculated. n represents the number of queries:

[0015]

[0016] In the formula (x i y i ), (x j y j ) represent the coordinates of the i-th and j-th query centers respectively. In addition to focusing on the similarity of query features, the attention mechanism also focuses on the distance between queries.

[0017] (2) For each frame of the feature image, adaptive spatiotemporal sampling is used to adaptively generate a set of sampling offsets {(Δa)} for the query features. i , Δb i, Δc i These offsets are used to convert the query pillars into 3D sampling points:

[0018]

[0019] Adaptive spatiotemporal sampling points are used to adapt to query pillars and query features, covering objects of different sizes;

[0020] (3) For self-motion and object motion modes, time alignment is performed based on motion distortion sampling points, as follows:

[0021] (3.1) Handling object motion: In autonomous driving, the instantaneous velocity is equal to the average velocity within a short time window, therefore the velocity vector [v] in the query is used. a v b Adaptively warp the sampling points to previous timestamps:

[0022] a ti =a i +v a ·(T t -T0), (4)

[0023] b ti =b i +v b ·(T t -T0), (5)

[0024] In the formula T t Tt represents the timestamp of the previous frame at time t, while T0 represents the timestamp of the current frame. Since the velocity vector is defined in the BEV plane, ct t，i With c i They are the same;

[0025] (3.2) Based on the self-pose distortion sampling points provided by the dataset, the sampling points are first transformed to the global coordinate system, and then sequentially transformed to the local coordinate system of the frame at time t:

[0026]

[0027] In the formula E t =[R|t] is the self-pose of the frame at time t;

[0028] For each feature target at different times, the distortion sampling points {[a′] of the target are obtained using the camera's intrinsic and extrinsic parameters. t.i b′ t.i , c′ t.i Projecting onto each view, due to overlap between adjacent views, the projection point will hit one or more views. These views will then be named... For each projected view k, there is a multi-scale feature map [F] from the image backbone. k，j |j∈{1,2,…,N} feat The features will be interpolated in the image plane using bilinear interpolation. Sampling is performed, and then weighted on the scale axis:

[0029]

[0030] Where N feat It is the number of multi-scale feature maps, P k It is the projection function of view k; ω ij It is the weight of the i-th point on the j-th feature map, which is generated from the query features through a linear transformation.

[0031] Furthermore, the working principle of the scale-adaptive attention mechanism is as follows:

[0032]

[0033] In the formula, It is the scale-adaptive attention itself, where d is the channel dimension; θ and τ are the standard parameters that control the local and global receptive fields of each query, respectively; when θ = 0, the self-attention only has the global receptive field; as θ gradually increases, the attention weight of queries located far away gradually decreases, and the receptive field narrows and focuses on nearby targets.

[0034] Furthermore, based on the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer equivariant transformation module, as follows:

[0035] (1) The original points are encoded into transformed equivariant features by sharing equivariant sparse convolution. In terms of specific representation, given n 2 Transformation action Transform the point cloud p into N 2 A set of differences In each voxel set, the average feature value of all original feature points within it is used as input;

[0036] By sharing the equivariant sparse convolution σ(·), voxels are... Encoding as transform-equivariant features The transformation formula is as follows:

[0037]

[0038] Transformation of variant features It contains diverse features under different rotation and reflection transformations.

[0039] (2) Voxel features are compressed into BEV features through the Transformer isovariant BEV transformation. Compressed into BEV characteristics along high latitudes Align the transformed BEV features to a unified coordinate system:

[0040] (2.1) In the BEV coordinate system Determine a set of scene-level grid points Based on BEV characteristics The transformation action method will change this set of grid points Transform to the BEV coordinate system to generate a new set of grid points.

[0041] (2.2) Apply a series of bilinear interpolation Ψ to the BEV coordinate system to obtain a set of alignment features. The conversion formula is as follows:

[0042]

[0043] like exist If there is no corresponding boundary feature point, then The interpolation results are filled with zero values;

[0044] (2.3) Align features on a feature map of scale N A compact feature representation F is obtained by using the max-pooling operation MP(·). * The details are as follows:

[0045]

[0046] Furthermore, an adaptive fusion mechanism is used to capture the spatial relationships between multimodal features, aligning the point cloud with image features, as follows:

[0047] The adaptive blending mechanism includes adaptive channel blending and adaptive spatial blending in sequence:

[0048] (1) Adaptive channel mixing adaptively enhances channel semantics by transforming features along the channel dimension using query-based dynamic weights. For a given set of adaptive spatiotemporal sampling matrices... Where t represents the current time axis position, i represents the current query coordinate, and channel C = d feat Divide into g groups, each with d channels. feat / g, through a channel grouping and mixing mechanism, the decoder generates a g·pin offset vector for the query, thereby capturing the correlation between the feature space and the channel dimension, as detailed below:

[0049]

[0050] F ACM =ReLU(LayerNorm(f t，i ·W C (12)

[0051] In the formula It is a channel-mixed feature output. The linear layer processes each matrix separately, and layer normalization is applied to the two dimensions of the mixed output, where the dynamic weights W... C Shared between different sampling points in 3D space;

[0052] (2) Adaptive spatial mixing (ASM) can be described as transposing the channel mixing feature matrix and applying a dynamic kernel to its spatial dimension, as follows:

[0053]

[0054] F ASM =ReLU(LayerNorm(f t，i ·w p (14)

[0055] In the formula w p It is a dynamic weight that is shared between different channels, where It is a spatial blending feature output, C out It represents the number of spatial blending output modes; adaptive spatial blending follows adaptive channel blending.

[0056] (3) Voxelize the output channel blending features and spatial blending features. Input feature refinement encoding module:

[0057]

[0058] In the formula, '⊙' represents the Hadamard product, and j is a voxel. Inner sample point p j The index, β, is the characteristic linear transformation function used to generate the feature vector β(p). j The feature vector and the adaptive weight vector α(V) i ·p j In the aggregation, ξ is the softmax function, and the attention weight function α calculates the weights V. i and p j The weights between them;

[0059] (4) Combine the encoded channel blending features and spatial blending features into F u The details are as follows:

[0060]

[0061] in, For linking operations.

[0062] Furthermore, image features, point cloud features, and BEV features are fused into a multi-scale coupled feature representation with greater spatial awareness through a multi-scale cross-transformer, as detailed below:

[0063] (1) For point-by-point mixing features F u Projecting the coordinates, we obtain the corresponding image feature block coordinates P. I Point cloud feature block coordinates P p and BEV feature block coordinates P B The projection equation is as follows:

[0064] P I =D matrix ·R matrix ·T L→C ·F u (18)

[0065] P p =T C→L ·F u (19)

[0066] P B =T L→B ·F u (20)

[0067] In the formula D matrix R matrix These are the camera's calibration matrix and correction matrix, T L→C It is the transformation matrix from LiDAR coordinates to camera coordinates, T C→L It is the LiDAR coordinate to camera transformation matrix, T L→B It is the LiDAR coordinate to BEV transformation matrix;

[0068] (2) Image feature block coordinates P I Point cloud feature block coordinates P p and BEV feature block coordinates P B Input converted to query Q * Key K * Sum V * As shown below:

[0069]

[0070] In the formula It is a linear projection matrix. To adaptively learn the weight transformations of different modal features for complementary enhancement, the query matrices of image, point cloud, and BEV features are further cross-fused through a dynamic fusion mechanism, resulting in a new fused query. As shown below:

[0071]

[0072] (3) The new merge query With another modality of bond K * They are combined and the corresponding multimodal cross-fusion weights are obtained through the dynamic weight fusion module.

[0073]

[0074] (4) After obtaining the new weight scores through the normalization function, the new image, point cloud, and BEV features F′ are obtained through dot product calculation. * As shown below:

[0075]

[0076] (5) Aggregate features with cross-modal multi-fusion interactions into a new feature, as shown below:

[0077]

[0078] According to a second aspect of the present invention, the present invention provides a multi-scale spatiotemporal hybrid sampling fusion system for three-dimensional target detection, comprising:

[0079] The receiving unit is used to receive image data and raw point cloud data;

[0080] The generation unit is used to extract features from image data and raw point cloud data using a dual-stream spatiotemporal feature extraction network. It generates adaptive spatiotemporal sampling features at the sampling location under query guidance by using the scale adaptation force of multi-scale receptive field aggregation features.

[0081] The aggregation unit is used to aggregate sparse point cloud coordinate features into lightweight and compact BEV features based on adaptive spatiotemporal sampling features through the Transformer equivariant transformation module.

[0082] Alignment units are used to capture the spatial relationships between multimodal features using an adaptive blending mechanism, aligning point clouds with image features;

[0083] The multi-feature fusion unit is used to fuse image features, point cloud features, and BEV features into a multi-scale coupled feature representation with greater spatial awareness through a multi-scale cross-transformer, and then achieve 3D object detection through a learning network of RPN and classification regression.

[0084] Furthermore, the generation unit includes a design module, a transformation module, and a time-aligned execution module:

[0085] The design module is used to design scale-adaptive attention mechanisms based on vision-based Transformers to learn appropriate receptive fields under the guidance of queries;

[0086] The conversion module is used to adaptively generate a set of sampling offsets for the query features by using adaptive spatiotemporal sampling on the feature image of each frame, and convert the query pillars into 3D sampling points through these offsets;

[0087] The time alignment execution module is used to perform time alignment based on motion distortion sampling points in both self-motion and object motion modes.

[0088] According to a third aspect of the present invention, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor, and when the processor loads and executes the computer program, it employs the above-described multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection.

[0089] According to a fourth aspect of the present invention, the present invention provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the above-described multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection.

[0090] This invention has at least the following beneficial effects:

[0091] 1. This invention addresses the problem that representative features cannot be effectively obtained during the downscaling transformation of multimodal data. It proposes a multimodal cross-fusion target detection algorithm based on the combination of Transformer and BEV. The algorithm aims to combine the spatial coordinate features of point cloud data with the temporal awareness information of continuous images by using the BEV features after multi-scale adaptive fusion. By using a multi-scale cross Transformer, different single-scale features are converted into multi-scale coupled representations, and more spatiotemporal awareness information is extracted to improve detection accuracy.

[0092] 2. Since the Multi-Head Self-Attention (MHSA) of Transformer lacks an explicit multi-scale design, it hinders the scale connection between global and local features. To address this, this invention proposes a dual-stream spatiotemporal feature extraction network. By using the scale-adaptive force of multi-scale receptive fields to aggregate features, and generating adaptive spatiotemporal sampling features at sampling locations under query guidance, this information is integrated into the feature-level information of Transformer. The enhancement of global temporal scale feature connections combined with the design of local multi-scale receptive fields improves the detection accuracy of objects of different sizes and categories.

[0093] 3. This invention addresses the localization and matching problem caused by the offset between point cloud and image features by designing a Transformer isomorphic transformation module. This module aims to aggregate sparse point cloud coordinate features into a lightweight and compact representation and utilize an adaptive blending mechanism to capture the spatial relationship between multimodal features, thereby better aligning point cloud and image features and improving the accuracy of target localization.

[0094] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the overall framework of the method described in this invention;

[0096] Figure 2 This is a schematic diagram illustrating the principle of the dual-stream spatiotemporal feature extraction network of the present invention;

[0097] Figure 3 This is a schematic diagram illustrating the principle of the Transformer equivalent transformation module of the present invention;

[0098] Figure 4 This is a schematic diagram illustrating the principle of the adaptive hybridization mechanism of the present invention;

[0099] Figure 5 This is a schematic diagram illustrating the principle of multi-cross-fusion in this invention;

[0100] Figure 6 This is an example of qualitative results from the KITTI dataset in Embodiment 1 of the present invention;

[0101] Figure 7 This is a schematic diagram comparing the BEV results of different 3D target detection algorithms in Embodiment 1 of the present invention;

[0102] Figure 8 This is a comparison chart of the precision-recall curves of vehicle detection results on the KITTI dataset in Embodiment 1 of the present invention.

[0103] Figure 9 This is an example of qualitative results on the NuScenses dataset from Embodiment 1 of the present invention;

[0104] Figure 10 This is a real vehicle platform used for actual testing in Embodiment 1 of the present invention;

[0105] Figure 11 This is the qualitative result of the actual vehicle platform in Embodiment 1 of the present invention;

[0106] Figure 12 This serves as a baseline for comparing the average accuracy of different object modules between the baseline and the method described in this invention;

[0107] Figure 13 This is a comparison of the number of decoder layers in the KITTI verification set according to Embodiment 1 of the present invention. Detailed Implementation

[0108] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0109] Example 1:

[0110] Please see Figure 1-3 This invention provides a technical solution: a multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection, specifically including the following steps:

[0111] S1. Receive image data and raw point cloud data. The image data and raw point cloud data are acquired by collecting synchronous data from the vehicle-mounted camera and the lidar.

[0112] S2. A dual-stream spatiotemporal feature extraction network is used to extract features from image data and raw point cloud data. The feature aggregation scale adaptability is achieved by using multi-scale receptive fields, and adaptive spatiotemporal sampling features for the sampling location are generated under query guidance, as detailed below:

[0113] (1) A scale-adaptive attention mechanism based on vision is designed using Transformer to learn an appropriate receptive field under the guidance of the query. The specific design method is as follows:

[0114] (1.1) Given a set of learnable queries, where each query parameter is defined as [(x, y, z); (w, l, h); (θ)], where (x, y, z) represents the coordinate position, (w, l, h) represents the dimension vector, and (θ) represents the vector angular velocity; z is initially set to 0; the other parameters (x, y, w, l, θ) are from a random Gaussian distribution;

[0115] (1.2) Query features are appended to each query box to encode rich instance features, and the all-pair distance between query centers in the BEV space is calculated. n represents the number of queries:

[0116]

[0117] In the formula (x i y i ), (x j y j ) represent the coordinates of the i-th and j-th query centers respectively. In addition to focusing on the similarity of query features, the attention mechanism also focuses on the distance between queries.

[0118] The working principle of the scale-adaptive attention mechanism is as follows:

[0119]

[0120] In the formula, It is the scale-adaptive attention itself, where d is the channel dimension; θ and τ are the standard parameters that control the local and global receptive fields of each query, respectively; when θ = 0, the self-attention only has the global receptive field; as θ gradually increases, the attention weight of queries located far away gradually decreases, and the receptive field narrows and focuses on nearby targets.

[0121] (2) For each frame of the feature image, adaptive spatiotemporal sampling is used to adaptively generate a set of sampling offsets {(Δa)} for the query features. i , Δb i , Δc i These offsets are used to convert the query pillars into 3D sampling points:

[0122]

[0123] Adaptive spatiotemporal sampling points are more flexible than deformable attention in BEVFormer because they can adapt to query pillars and query features, thus covering objects of different sizes. These points are not limited to queries because we do not restrict the range of sampling offsets;

[0124] (3) In addition, time alignment is performed based on motion distortion sampling points;

[0125] In autonomous driving, there are two types of motion: self-motion and object motion. Self-motion describes the movement of the car from its own perspective as it drives through the environment, while object motion refers to the movement of other objects in the environment as they move around the autonomous vehicle.

[0126] For both self-motion and object motion modes, time alignment is performed based on motion distortion sampling points, as follows:

[0127] (3.1) Handling object motion: In autonomous driving, the instantaneous velocity is equal to the average velocity within a short time window, therefore the velocity vector [v] in the query is used. a v b Adaptively warp the sampling points to previous timestamps:

[0128] a ti =a i +v a ·(T t -T0), (4)

[0129] b ti =b i +v b ·(T t -T0), (5)

[0130] In the formula T t Tt represents the timestamp of the previous frame at time t, while T0 represents the timestamp of the current frame. Since the velocity vector is defined in the BEV plane, ct t，i With c i They are the same;

[0131] (3.2) Based on the self-pose distortion sampling points provided by the dataset, the sampling points are first transformed to the global coordinate system, and then sequentially transformed to the local coordinate system of the frame at time t:

[0132]

[0133] In the formula E t =[R|t] is the self-pose of the frame at time t;

[0134] For each feature target at different times, the distortion sampling points {[a′] of the target are obtained using the camera's intrinsic and extrinsic parameters. t.i b′ t.i , c′ t.i Projecting onto each view, due to overlap between adjacent views, the projection point will hit one or more views. These views will then be named... For each projected view k, there is a multi-scale feature map [F] from the image backbone. k，j |j∈{1,2,…,N} feat The features will be interpolated in the image plane using bilinear interpolation. Sampling is performed, and then weighted on the scale axis:

[0135]

[0136] Where N featIt is the number of multi-scale feature maps, P k It is the projection function of view k; ω ij It is the weight of the i-th point on the j-th feature map, which is generated from the query features through a linear transformation;

[0137] S3. Based on the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact bird's-eye view (BEV) features through the Transformer isomorphic transformation module, as follows:

[0138] (1) The original points are efficiently encoded into transform equivariant features by sharing equivariant sparse convolution, which has the following two characteristics: (a) there are highly shared weights between transform channels; (b) input transforms with different rotation angles and reflections.

[0139] In terms of specific manifestation, given N 2 Transformation action Transform the point cloud p into N 2 A set of differences In each voxel set, the average feature value of all original feature points within it is used as input;

[0140] By sharing the equivariant sparse convolution σ(·), voxels are... Encoding as transform-equivariant features The transformation formula is as follows:

[0141]

[0142] Transformation of variant features It includes diverse features under different rotation and reflection transformations;

[0143] (2) Transformation of isomorphic features It contains a large number of different rotation and reflection transformation channels. In order to solve the computational and memory pressure caused by the large number of transformation channels, it is proposed to compress voxel features into BEV features through Transformer isovariant BEV transformation to reduce the amount of data.

[0144] voxel characteristics Compressed into BEV characteristics along high latitudes The transformed BEV features reside in different coordinate systems due to different transformation channels. Therefore, it is necessary to align the transformed BEV features to a unified coordinate system.

[0145] (2.1) In the BEV coordinate system Determine a set of scene-level grid points Based on BEV characteristics The transformation action method will change this set of grid points Transform to the BEV coordinate system to generate a new set of grid points.

[0146] (2.2) Apply a series of bilinear interpolation Ψ to the BEV coordinate system to obtain a set of alignment features. The conversion formula is as follows:

[0147]

[0148] like exist If there is no corresponding boundary feature point, then The interpolation results are filled with zero values;

[0149] (2.3) Align features on a feature map of scale N A compact feature representation F is obtained by using the max-pooling operation MP(·). * The details are as follows:

[0150]

[0151] S4. Utilize an adaptive fusion mechanism to capture the spatial relationships between multimodal features, aligning the point cloud with image features, as detailed below:

[0152] Having acquired sampling features from different time points and locations, the key next step is to fuse these sampling features to obtain highly coupled multimodal information, thereby achieving alignment between point cloud and image information. To this end, this embodiment designs an adaptive mixing mechanism based on adamixer, which includes adaptive channel mixing and adaptive spatial mixing in sequence.

[0153] (1) Adaptive channel mixing adaptively enhances channel semantics by transforming features along the channel dimension using query-based dynamic weights. For a given set of adaptive spatiotemporal sampling matrices... Where t represents the current time axis position, i represents the current query coordinate, and channel C = d feat Divide into g groups, each with d channels. feat / g; Through the channel grouping and mixing mechanism, the decoder can generate a g·pin offset vector for the query to enrich the diversity of sampling points and utilize the richer spatial structure of these points, thereby capturing the correlation between the feature space and the channel dimension:

[0154]

[0155] F ACM =ReLU(LayerNorm(f t，i ·W C(12)

[0156] In the formula It is a channel-mixed feature output. The linear layer processes each matrix separately, and layer normalization is applied to the two dimensions of the mixed output, where the dynamic weights W... C Shared between different sampling points in 3D space;

[0157] (2) In order to enable the query to adapt to the spatial structure of different sampling features, adaptive spatial mixing is introduced. The process can be described as transposing the channel mixing feature matrix and applying a dynamic kernel to its spatial dimension:

[0158]

[0159] F ASM =ReLU(LayerNorm(f t，i ·w p (14)

[0160] In the formula w p It is a dynamic weight that is shared between different channels, where It is a spatial blending feature output, C out It represents the number of spatial blending output modes; adaptive spatial blending follows adaptive channel blending.

[0161] (3) Voxelize the output channel blending features and spatial blending features. i∈1,2,…N. Input feature refinement encoding module:

[0162]

[0163] In the formula, '⊙' represents the Hadamard product, and j is a voxel. Inner sample point p j The index, β, is the characteristic linear transformation function used to generate the feature vector β(p). j The feature vector and the adaptive weight vector α(V) i ·p j In the aggregation, ξ is the softmax function, and the attention weight function α calculates the weights V. i and p j The weights between them;

[0164] (4) Combine the encoded channel blending features and spatial blending features into F u The details are as follows:

[0165]

[0166] in, For link operations;

[0167] S5. Image features, point cloud features, and BEV features are fused into a multi-scale coupled feature representation with greater spatial awareness through a multi-scale cross-transformer. Then, 3D object detection is achieved through a learning network of RPN and classification regression, as detailed below:

[0168] (1) For point-by-point mixing features F u Projecting the coordinates, we obtain the corresponding image feature block coordinates P. I Point cloud feature block coordinates P p and BEV feature block coordinates P B The projection equation is as follows:

[0169] P I =D matrix ·R matrix ·T L→C ·F u (18)

[0170] P p =T C→L ·F u (19)

[0171] P B =T L→B ·F u (20)

[0172] In the formula D matrix R matrix These are the camera's calibration matrix and correction matrix, T L→C It is the transformation matrix from LiDAR coordinates to camera coordinates, T C→L It is the LiDAR coordinate to camera transformation matrix, T L→B It is the LiDAR coordinate to BEV transformation matrix;

[0173] (2) Image feature block coordinates P I Point cloud feature block coordinates P p and BEV feature block coordinates P B Input converted to query Q * Key K * Sum V * As shown below:

[0174]

[0175] In the formula It is a linear projection matrix. To adaptively learn the weight transformations of different modal features for complementary enhancement, the query matrices of image, point cloud, and BEV features are further cross-fused through a dynamic fusion mechanism, resulting in a new fused query. As shown below:

[0176]

[0177] (3) The new merge query It is combined with the key K* of another modality and the corresponding multimodal cross-fusion weights are obtained through the dynamic weight fusion module.

[0178]

[0179] (4) After obtaining the new weight scores through the normalization function, the new image, point cloud, and BEV features F' are obtained through dot product calculation. * As shown below:

[0180]

[0181] (5) Aggregate features with cross-modal multi-fusion interactions into a new feature, as shown below:

[0182]

[0183] The specific process of multi-cross-fusion Transformer is shown in Table 1:

[0184]

[0185]

[0186] S6. Regarding the overall loss function, see below:

[0187] The overall loss function includes classification loss Lcls for object and category, regression loss Lreg for predicted bounding boxes, prediction loss Ldff for the two-stream spatiotemporal feature extraction network, and transformation loss Ltrans for the Transformer's equivalent BEV transformation. The overall loss function is as follows:

[0188] Ltotal=λclsLcls+λregLreg+λdffLdff+λtransLtrans(34) where λcls, λreg, λdff, and λtrans represent the regularization weight coefficients of classification loss, regression loss, prediction loss, and transformation loss, respectively.

[0189] To address the imbalance between positive and negative samples, focus loss is used to balance the samples for the classification and prediction tasks respectively, and smoothed-L1 is used to balance the regression loss. Focus loss is also used to handle the transformation loss of the Transformer's equivalent BEV transformation.

[0190]

[0191] Where ρ is a 2D pixel feature region with an effective depth label. It is the accurate true value of the lidar.

[0192] The technical solution of the present invention will be further described below with reference to specific embodiments, but the scope of protection of the present invention is not limited to this embodiment:

[0193] 1. Experimental Results and Analysis

[0194] This paper evaluates and tests the proposed SMSFusion algorithm on the challenging KITTI3D

[28] and NuScenes

[29] object detection datasets and the state-of-the-art open dataset Waymo

[30] . It also compares the algorithm with other state-of-the-art methods in object detection. Furthermore, extensive ablation experiments were conducted to evaluate the effects of different modules. Finally, the proposed method was applied to a real autonomous vehicle platform to verify its effectiveness in real-world road scenarios.

[0195] 1.1 Experimental results on the KITTI dataset

[0196] Table 2 compares the detection performance of the technical solution of this embodiment with other object detection algorithms in the KITTI test set for the car category. As shown in Table 2, the overall detection performance of the technical solution of this embodiment is superior to other current 3D object detection algorithms. Compared with the baseline ACF-Net of the technical solution of this embodiment, the average detection accuracy (AP) at different levels of the technical solution of this embodiment is significantly higher. 3D The average accuracy AP was improved by 1.53 / 0.58 / 0.33 respectively. Compared with the high-performance TED

[37] , the average accuracy AP was improved. 3D It increased by 0.24.

[0197] Table 3 shows the comparison results of the detection performance of the technical solution of this embodiment on the KITTI test set for pedestrians and bicycles. As shown in Table 2, compared with the baseline ACF-Net of the technical solution of this embodiment, the average accuracy of pedestrian target detection of the technical solution of this embodiment is improved by 1.10 / 0.03 / 0.06 at different difficulty levels, while the detection results of bicycle targets are improved by 0.46 / 0.66 / 1.43 at different difficulty levels compared with the baseline. The detection diagram of the technical solution of this embodiment on the KITTI dataset is shown below. Figure 6 As shown.

[0198] Table 2. Detection performance of car categories in the KITTI test set.

[0199] Table2DetectionperformanceofautomotivecategoriesontheKITTI

[0200] testset

[0201]

[0202]

[0203] Table 3 shows the detection performance of pedestrians and bicycles in the KITTI test set.

[0204]

[0205] To verify the 3D orientation of the detection results, a 3D bounding box was projected onto the BEV. The BEV detection results were compared with those of a 3D object detection algorithm, and the results are as follows. Figure 7 As shown. SMSFusion has the best mAP value on all subsets, especially compared with M3DETR

[39] , it improves by 6.51%, 4.32% and 3.9% on the “easy”, “medium” and “hard” subsets, respectively. BEV detection results show that by fusing complex spatial information, the positioning performance of objects is effectively improved.

[0206] A comparison of the precision-recall curves for vehicle detection results on the KITTI dataset, as follows: Figure 8 As shown in the figure, the proposed FFAFormer algorithm exhibits higher accuracy with high recall compared to other algorithms, indicating that the proposed algorithm has overall robustness and stability.

[0207] 1.2 Experimental results on the NuScences dataset

[0208] In order to further test the effectiveness and versatility of the technical solution of this embodiment and verify that the algorithm has good detection performance for three-dimensional targets, the algorithm was tested on the more challenging NuScences dataset. As shown in Table 4, the technical solution of this embodiment is compared with other 3D target detection algorithms on NuScences for different categories of targets. As can be seen from the table, on the important indicators NDS and mAP in the NuScences dataset, it is improved by 0.2 / 1.1 compared with the best VFF

[46] . It can be seen that the technical solution of this embodiment still has good robustness in detection ability on different datasets. The detection results of the technical solution of this embodiment on the NuScences dataset are shown in the figure below. Figure 9 As shown.

[0209] Table 4 Performance comparison on the NuScenes dataset

[0210]

[0211] 1.3 Experimental results on the Waymo dataset

[0212] This embodiment also compares the proposed algorithm with the baseline ACF-Net

[44] on the Waymo dataset. The results are shown in Table 5. The results show that the technical solution of this embodiment performs well in different levels of tests on the Waymo dataset. Compared with ACF-Net

[44] , the mAP / mAPH on L1 / L2 is improved by 1.10 / 0.78 and 1.14 / 0.86 respectively. The improvement is even greater when detecting smaller targets such as pedestrians and bicycles, showing that the technical solution of this embodiment has excellent performance for detecting small targets at a distance.

[0213] Table 5 shows the performance comparison on the Waymo dataset.

[0214]

[0215] 1.4 Experiments on a real vehicle platform

[0216] Simulation experiments on datasets alone may not fully demonstrate the performance of SMSFusion in real-world scenarios. Therefore, a series of tests were conducted on a real vehicle platform, such as... Figure 10 As shown, it includes 80-beam LiDAR, a stereo camera, millimeter-wave radar, and two GNSS units. 3D object detection is primarily performed using LiDAR and vehicle cameras. The qualitative results of the actual vehicle platform in a certain city's urban area are as follows: Figure 11 As shown.

[0217] To more clearly demonstrate the effectiveness of each module of the proposed algorithm, the average accuracy of modules with different target detection objects is compared between the baseline and the proposed algorithm. Figure 12 As shown in Table 6, the detection results of different vehicle targets on the real vehicle platform are compared. As can be seen from Table 6, in the complex and ever-changing real environment, the addition of each module of the proposed algorithm can effectively improve the overall performance of the model.

[0218] Table 6 Performance of vehicle categories on the real-world platform

[0219]

[0220] 1.5 Ablation Experiment

[0221] The effectiveness of each component of the proposed algorithm was verified through extensive ablation experiments:

[0222] The effectiveness of each module is shown in Table 7. Experiment (a) is based on the experimental data of CenterPoint

[44] . The average accuracy results calculated on 40 car recall locations on the KITTI validation set are compared. As shown in Table 7, each module component can effectively help improve the accuracy of target detection results at different difficulty levels.

[0223] Table 7 shows the impact of different parts on the KITTI validation set.

[0224] The results were evaluated using APs calculated from 40 automotive recall locations.

[0225]

[0226] To verify whether the adaptive hybrid mechanism in this embodiment is more efficient than the conventional hybrid query mechanism (CHQM), the variable mask offset and weights of the conventional hybrid query mechanism are determined by a single camera query, while the adaptive hybrid mechanism is jointly updated and determined by feature queries from different domains, thereby obtaining more accurate variable mask offsets and feature weights. As shown in Table 8, on the KITTI validation set, the average detection accuracy of the adaptive hybrid mechanism is improved by 7% and 12% respectively compared to the baseline and single-camera query.

[0227] Table 8 compares different query types on the KITTI validation set.

[0228]

[0229] The effectiveness of each component of the multi-cross-fusion transformer was verified, as shown in Table 9. Experiment (A) correlated camera features with voxel features using a naive fusion method and fused the voxel features of the lidar using simple summation. Experiments (B) and (C) added a dynamic fusion mechanism and a multi-modal cross-fusion mechanism respectively. The results showed that each component of the multi-cross-fusion transformer module played an improvement role.

[0230] Table 9. Ablation study of multi-cross-fusion transformers.

[0231] The results were evaluated using AP calculated from 40 recall locations.

[0232]

[0233] FEN-DSS module behavior analysis. The performance trend of the KITTI validation set is related to the number of decoder layers, such as... Figure 13 As shown, in the KITTI dataset, the performance steadily improves as the number of layers reaches four, demonstrating that the adaptive spatiotemporal sampling mechanism continuously improves the quality of features used for 3D object detection.

[0234] By evaluating the technical solution of this embodiment at different distance levels and occlusion degrees, it can be determined under what conditions this embodiment improves the baseline the most, as shown in Table 10. Object detection affected by different distances and occlusions shows significant improvement. This is because the multi-scale depth prediction module enhances the interaction depth of point features at different scales, thereby refining the features.

[0235] Table 10 shows the performance and evaluation results at different distances and degrees of obstruction, and the average level of 40 recall locations for automobiles calculated by 3DAP.

[0236]

[0237] In summary, to address the problem of multimodal data failing to adapt to objects of different sizes and categories during downscaling, ultimately leading to the loss of localization information, this embodiment proposes a multi-scale spatiotemporal hybrid sampling fusion method (SMSFusion) for 3D object detection. First, an adaptive spatiotemporal sampling feature is generated under query guidance through a dual-stream spatiotemporal feature extraction network. Simultaneously, a Transformer isomorphic transformation module is designed, and an adaptive fusion mechanism is used to better align point cloud and image features. Finally, a multi-cross Transformer fuses multiple unimodal data into a hybrid representation with greater spatial awareness, thereby achieving accurate 3D object detection. Experimental results on the KITTI, NuScenes, and Waymo datasets demonstrate that SMSFusion can handle challenging detection tasks well. SMSFusion has also been applied to an autonomous driving platform for real-world road testing, showing that the method has sufficient generalization ability in complex real-world environments.

[0238] Example 2:

[0239] This invention provides a multi-scale spatiotemporal hybrid sampling fusion system for three-dimensional target detection, comprising:

[0240] The receiving unit is used to receive image data and raw point cloud data;

[0241] The generation unit is used to extract features from image data and raw point cloud data using a dual-stream spatiotemporal feature extraction network. It generates adaptive spatiotemporal sampling features at the sampling location under query guidance by using the scale adaptation force of multi-scale receptive field aggregation features.

[0242] The aggregation unit is used to aggregate sparse point cloud coordinate features into lightweight and compact BEV features based on adaptive spatiotemporal sampling features through the Transformer equivariant transformation module.

[0243] Alignment units are used to capture the spatial relationships between multimodal features using an adaptive blending mechanism, aligning point clouds with image features;

[0244] The multi-feature fusion unit is used to fuse image features, point cloud features, and BEV features into a multi-scale coupled feature representation with greater spatial awareness through a multi-scale cross-transformer, and then achieve 3D object detection through a learning network of RPN and classification regression.

[0245] Furthermore, the generation unit includes a design module, a transformation module, and a time-aligned execution module:

[0246] The design module is used to design scale-adaptive attention mechanisms based on vision-based Transformers to learn appropriate receptive fields under the guidance of queries;

[0247] The conversion module is used to adaptively generate a set of sampling offsets for the query features by using adaptive spatiotemporal sampling on the feature image of each frame, and convert the query pillars into 3D sampling points through these offsets;

[0248] The time alignment execution module is used to perform time alignment based on motion distortion sampling points in both self-motion and object motion modes.

[0249] Specifically, the aforementioned receiving unit, generating unit, aggregation unit, alignment unit, and multi-feature fusion unit can be embedded into a computer processing system. The computer, based on the aforementioned multi-scale spatiotemporal hybrid sampling fusion method for 3D target detection, calls upon each of the aforementioned units to complete the task of equivalent modeling. The aforementioned receiving unit, generating unit, aggregation unit, alignment unit, and multi-feature fusion unit can perform operations according to the specific steps given by the aforementioned multi-scale spatiotemporal hybrid sampling fusion method for 3D target detection.

[0250] It should be noted that the division of the various units in the above system is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These units can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some units can be implemented by processing element calls to software, while others are implemented in hardware. For example, the generation unit can be a separate processing element, or it can be integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and its functions can be called and executed by a processing element of the device. The implementation of other units is similar. Furthermore, these units can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0251] For example, these units can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a unit is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these units can be integrated together to form a system-on-a-chip (SOC).

[0252] Example 3:

[0253] The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it employs the above-mentioned multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection.

[0254] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server, and the terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0255] Furthermore, the processor can be a central processing unit (CPU). Of course, depending on the actual application, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., can also be used. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0256] Example 4:

[0257] The present invention provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the above-described multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection.

[0258] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0259] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0260] For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on," "mounted on," "fixed to," or "set on" another element, it may be directly on the other element or there may be an intermediate element present. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible embodiments.

[0261] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0262] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

Claims

1. A multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection, characterized in that, Specifically, the following steps are included: Receive image data and raw point cloud data; A dual-stream spatiotemporal feature extraction network is used to extract features from image data and raw point cloud data. The scale adaptation force of feature aggregation by multi-scale receptive field is used to generate adaptive spatiotemporal sampling features of sampling location under query guidance. Based on the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer equivariant transformation module. An adaptive fusion mechanism is used to capture the spatial relationships between multimodal features, aligning point clouds with image features; Image features, point cloud features, and BEV features are fused into a multi-scale coupled feature representation with greater spatial awareness by using a multi-scale cross-transformer, and then 3D object detection is achieved through a learning network of RPN and classification regression. Based on the adaptive spatiotemporal sampling features, the sparse point cloud coordinate features are aggregated into lightweight and compact BEV features through the Transformer isomorphic transformation module, as follows: (1) The original points are encoded into transformed equivariant features by sharing equivariant sparse convolution. In terms of specific representation, given Transformation action , will dot clouds Transform into A set of different points In each voxel set, the average feature value of all original feature points inside is used as input; By sharing equivariant sparse convolutions voxels Encoding as transform-equivariant features The transformation formula is as follows: . (8) Transformation Equivalent Features Includes diverse features under different rotation and reflection transformations (2) The voxel features are compressed into BEV features through the Transformer isovariant BEV transformation. Compressed into BEV characteristics along high latitudes Align the transformed BEV features to a unified coordinate system: (2.1) In the BEV coordinate system Determine a set of scene-level grid points Based on BEV characteristics The transformation action method will change this set of grid points Transform to the BEV coordinate system to generate a new set of grid points. ; (2.2) Apply a series of bilinear interpolations in the BEV coordinate system Obtain a set of alignment features The conversion formula is as follows: . (9) like exist If there is no corresponding boundary feature point, then The interpolation results are filled with zero values; (2.3) At a scale of Alignment features on feature map Through max pool operation To obtain a compact feature representation The details are as follows: . (10)。 2. The multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection according to claim 1, characterized in that, The specific method for generating adaptive spatiotemporal sampling features for sampling locations is as follows: (1) A scale-adaptive attention mechanism based on vision is designed to learn the appropriate receptive field under the guidance of the query. The specific design method is as follows: (1.1) Given a set of learnable queries, where each query parameter is defined as ,in Indicates coordinate position, Represents a dimension vector. Represents vector angular velocity; Initially set to 0; other parameters From a random Gaussian distribution; (1.2) Query features are appended to each query box to encode rich instance features, and the all-pair distance between query centers in the BEV space is calculated. , where n represents the number of queries: , (1) In the formula , They represent the first and the The coordinates of each query center; the attention mechanism focuses not only on the similarity of query features, but also on the distance between queries. (2) For each frame of feature image, adaptive spatiotemporal sampling is used to adaptively generate a set of sampling offsets for the query features. These offsets are used to convert the query pillars into 3D sampling points: . (3) Adaptive spatiotemporal sampling points are used to adapt to query pillars and query features, covering objects of different sizes; (3) For self-motion and object motion modes, time alignment is performed based on motion distortion sampling points, as follows: (3.1) Handling object motion: In autonomous driving, the instantaneous speed is equal to the average speed within a short time window, therefore the velocity vector in the query is used. Adaptively warp the sampling points to previous timestamps: , (4) , (5) In the formula Indicates the previous frame The timestamp represented by the moment, and This represents the timestamp of the current frame. Since the velocity vector is defined in the BEV plane, therefore... and They are the same; (3.2) Based on the self-pose distortion sampling points provided by the dataset, first transform the sampling points to the global coordinate system, and then transform them sequentially to... The local coordinate system of the frame at that time: , (6) In the formula yes The self-position of the frame in which the moment is located; For each feature target at different times, the distortion sampling points of the target are obtained using the camera's intrinsic and extrinsic parameters. When projecting onto each view, due to overlap between adjacent views, the projection point will hit one or more views. These views will then be named... For each projected view It has multi-scale feature maps from the image backbone. Features are interpolated in the image plane using bilinear interpolation. Sampling is performed, and then weighted on the scale axis: (7) in It is the number of multi-scale feature maps. It is a view The projection function; It is the first On the feature map, the first The weights of each point are generated from the query features through a linear transformation.

3. The multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection according to claim 2, characterized in that: The working principle of the scale-adaptive attention mechanism is as follows: (2) In the formula, It is scale-adaptive attention itself. It refers to the channel dimension; and These are the standard parameters that control the local and global receptive fields of each query; when At that time, self-attention only has the global receptive field; when As the number of queries increases, the attention weight of queries located at a distance gradually decreases, and the receptive field narrows and focuses on nearby targets.

4. The multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection according to claim 3, characterized in that, An adaptive fusion mechanism is used to capture the spatial relationships between multimodal features, aligning the point cloud with image features, as follows: The adaptive blending mechanism includes adaptive channel blending and adaptive spatial blending in sequence: (1) Adaptive channel fusion adaptively enhances channel semantics by transforming features along the channel dimension using query-based dynamic weights, for a given set of adaptive spatiotemporal sampling matrices. ,in Indicates the current timeline position. Indicates the current query coordinates; Channel C is represented as: Divide channel C into Groups, each group has channels. Through a channel grouping and mixing mechanism, the decoder generates an offset vector for the query, and the offset vector is... This allows us to capture the correlation between the feature space and the channel dimension, as detailed below: , (11) , (12) In the formula It is a channel-mixed feature output. The linear layer processes each matrix separately, and layer normalization is applied to the two dimensions of the mixed output, where dynamic weights... Shared between different sampling points in 3D space; (2) Adaptive spatial blending, the process of which can be described as transposing the channel blending feature matrix and applying a dynamic kernel to its spatial dimension, as follows: , (13) , (14) In the formula It is a dynamic weight that is shared between different channels, where It is a spatial blending feature output. This refers to the number of spatial blending output modes; adaptive spatial blending follows adaptive channel blending. (3) Voxelize the output channel blending features and spatial blending features: , (15) , (16) In the formula, ' 'For the accumulation of Hadama, It is a voxel Internal sample points index, It is a characteristic linear transformation function used to generate eigenvectors. The feature vector and the adaptive weight vector polymerization, It's the softmax function, the attention weight function. Calculate weights and The weights between them; This indicates that the channel blending feature is mapped to the first... The set of channel mixing point features within an individual element; (4) Merge the encoded channel blending features and spatial blending features into The details are as follows: , (17) in,' 'For link operations.' 5. The multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection according to claim 4, characterized in that, Image features, point cloud features, and BEV features are fused into a multi-scale coupled feature representation with greater spatial awareness using a multi-scale cross-transformer, as detailed below: (1) Point-by-point mixing features Projecting the coordinates yields the coordinates of the corresponding image feature blocks. Point cloud feature block coordinates and BEV feature block coordinates The projection equation is as follows: (18) , (19) , (20) In the formula , These are the camera's calibration matrix and correction matrix, It is the transformation matrix from LiDAR coordinates to camera coordinates. It is the LiDAR coordinate to camera transformation matrix. It is the LiDAR coordinate to BEV transformation matrix; (2) Image feature block coordinates Point cloud feature block coordinates and BEV feature block coordinates Input converted to query , , ,key , , Sum , , As shown below: ， ， , (21) ， ， , (22) ， ， , (23) In the formula , , , , , , , , It is a linear projection matrix. The query matrix of image, point cloud, and BEV features is further cross-fused through a dynamic fusion mechanism to create a new fused query. , , As shown below: , (24) , (25) , (26) (3) The new fusion query , , Bond with other modes , , Combined, and obtained through a dynamic weight fusion module, the corresponding multimodal cross-fusion weights are obtained: , (27) , (28) , (29) (4) After obtaining the new weight scores through the normalization function, new image, point cloud and BEV features are obtained through dot product calculation. , , As shown below: , (30) , (31) , (32) (5) Aggregate features with cross-modal multi-fusion interactions into a new feature, as shown below: . (33)。 6. A multi-scale spatiotemporal hybrid sampling fusion system for three-dimensional target detection, used to implement the multi-scale spatiotemporal hybrid sampling fusion method for three-dimensional target detection as described in any one of claims 1 to 5, characterized in that, include: The receiving unit is used to receive image data and raw point cloud data; The generation unit is used to extract features from image data and raw point cloud data using a dual-stream spatiotemporal feature extraction network. It generates adaptive spatiotemporal sampling features at the sampling location under query guidance by using the scale adaptation force of multi-scale receptive field aggregation features. The aggregation unit is used to aggregate sparse point cloud coordinate features into lightweight and compact BEV features through the Transformer equivariant transformation module based on adaptive spatiotemporal sampling features. Alignment units are used to capture the spatial relationships between multimodal features using an adaptive blending mechanism, aligning point clouds with image features; The multi-feature fusion unit is used to fuse image features, point cloud features, and BEV features into a multi-scale coupled feature representation with more spatial awareness through a multi-scale cross-transformer, and then achieve 3D object detection through a learning network of RPN and classification regression.

7. The multi-scale spatiotemporal hybrid sampling and fusion system for three-dimensional target detection according to claim 6, characterized in that: The generation unit includes a design module, a transformation module, and a time-aligned execution module: The design module is used to design scale-adaptive attention mechanisms based on vision-based Transformers to learn appropriate receptive fields under the guidance of queries; The conversion module is used to adaptively generate a set of sampling offsets for the query features by using adaptive spatiotemporal sampling on the feature image of each frame, and convert the query pillars into 3D sampling points through these offsets; The time alignment execution module is used to perform time alignment based on motion distortion sampling points in both self-motion and object motion modes.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs the multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection as described in any one of claims 1 to 5.

9. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the multi-scale spatiotemporal hybrid sampling and fusion method for three-dimensional target detection as described in any one of claims 1 to 5.

Images