3D target detection method and device based on inter-modal implicit spatial enhancement and space-time interactive coding

Abstract

The invention discloses a 3D target detection method and device based on inter-modal implicit spatial enhancement and space-time interactive coding. The method comprises the steps of obtaining training data and constructing a 3D target detection architecture; rGB image features and point cloud features are obtained through a feature extraction network, and the features are input into an implicit space enhancement fusion coding network to generate aerial view image features containing rich geometric information; fusing the image aerial view features and the point cloud aerial view features in space and time dimensions through a space-time interactive feature fusion network to obtain final multi-modal fusion features; and sending the fusion features into a classification head to realize target detection and identification. According to the invention, high-precision target detection is carried out by fully mining the complementary characteristics of the LiDAR point cloud and the RGB image.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional target detection, and in particular to a 3D target detection method and device based on implicit spatial enhancement between modalities and spatiotemporal interaction coding. Background Art

[0002] In recent years, 3D object detection technology has attracted great attention in fields such as robotics and autonomous driving. Traditional 3D object detection methods usually rely on single-modality data, such as LiDAR point clouds or RGB images. These methods perform well in specific scenarios, but have limitations in complex and dynamic environments. For example, LiDAR point clouds perform well in outdoor environments, but have difficulty detecting small objects under occlusion and complex backgrounds; while RGB images provide rich texture information, they perform poorly in depth perception and in severe weather conditions. Therefore, effectively integrating multimodal data for accurate perception is crucial in autonomous driving, and multimodal fusion has become a key method to solve this problem.

[0003] At present, multimodal fusion models are mainly divided into two methods: viewpoint conversion-based splicing fusion and cross-modal interaction-based fusion. These methods aim to align LiDAR point clouds with RGB image features, achieve effective feature fusion, and use fused features for bounding box and category prediction. However, existing methods face many challenges in 3D object detection.

[0004] First, RGB images lack depth information, and usually rely on explicit depth estimation methods to align LiDAR data, but errors are prone to occur in complex environments, resulting in inaccurate conversion of image features to bird's-eye view (BEV) space, affecting the quality of fusion features and detection results. Second, although LiDAR point clouds provide accurate three-dimensional spatial information, they lack color and texture, which limits the ability to classify and recognize targets. In addition, point clouds are sparse, which limits the detection of small objects. Finally, existing fusion models often use simple splicing, which fails to fully exploit the complementary information of LiDAR and RGB images, limits the ability to express multimodal features, and faces sensor failures and data corruption problems in practical applications, affecting their reliability. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a 3D object detection method and device based on implicit spatial enhancement between modalities and spatiotemporal interaction coding.

[0006] The present invention proposes a 3D target detection method based on implicit spatial enhancement and spatiotemporal interactive coding between modalities. Through the implicit spatial enhancement fusion coding module, the spatial information and attention mechanism of LiDAR are used to enhance the geometric details of RGB image features, and the LiDAR point cloud is projected to the image plane for feature correspondence through the camera internal and external parameter matrix. In addition, spatiotemporal interactive feature fusion is adopted, and the deformable attention mechanism is used to capture the feature relationship between modalities in the spatial and temporal dimensions, aggregate multimodal semantic information, generate enhanced bird's-eye view features, and fuse historical frame information to improve detection accuracy.

[0007] In a first aspect, the present invention provides a 3D target detection method based on implicit spatial enhancement between modalities and spatiotemporal interaction coding, the method comprising:

[0008] S1. Obtain one-to-one corresponding multi-view RGB image data and LiDAR point cloud data as training samples, and construct a 3D target detection architecture, which includes a feature extraction network, an implicit spatial enhancement fusion coding network, a spatiotemporal interactive feature fusion network, and a classification head;

[0009] S2. Input a set of corresponding multi-view RGB image data and LiDAR point cloud data in the training sample into the feature extraction network for feature extraction to obtain RGB image features and LiDAR point cloud features;

[0010] S3. The implicit spatial enhancement fusion coding network includes a preliminary interaction unit and a fine interaction unit. In the preliminary interaction unit, the LiDAR point cloud data is projected into the RGB image coordinate system using the intrinsic parameter matrix and the extrinsic parameter matrix of the camera, and the spatial depth information of the RGB image features is dynamically enhanced through a variable cross-attention mechanism. In the fine interaction unit, the original RGB image features are fused with the enhanced RGB image features through a residual network to obtain the RGB image features under a bird's-eye view.

[0011] S4. The LiDAR point cloud features and the RGB image features from a bird's-eye view are respectively input into a spatiotemporal interactive feature fusion network, wherein the spatiotemporal interactive feature fusion network includes a spatial fusion unit and a temporal fusion unit. In the spatial fusion unit, the LiDAR point cloud features and the RGB image features from a bird's-eye view are subjected to deformable cross-attention to obtain preliminary bird's-eye view features, and in the temporal fusion unit, the preliminary bird's-eye view features are aggregated using a deformable self-attention mechanism to obtain the final fused features.

[0012] S5. Input the final fusion feature into the classification head to obtain the 3D target detection result, construct a loss function based on the detection result, and when the loss function meets the set conditions, save the current parameters of the 3D target detection architecture to obtain the 3D target detection model.

[0013] S6. Input a set of corresponding multi-view RGB image data and LiDAR point cloud data into the trained 3D target detection model to obtain the target detection result.

[0014] The preliminary interaction unit described in step S3 includes a projection operation layer and a spatial coding layer. In the projection operation layer, the camera's intrinsic parameter matrix and extrinsic parameter matrix are used to project the LiDAR point cloud data onto the RGB image features to establish a coordinate system correspondence; in the spatial coding layer, the precise spatial information of the LiDAR data is combined with a variable cross-attention mechanism to perform operations to obtain enhanced image features.

[0015] The fine interaction unit described in step S3 includes two residual modules, and the original RGB image features are fused with the enhanced RGB image features through a residual network to obtain the RGB image features from a bird's-eye view.

[0016] The spatial fusion unit described in step S4 includes a query generation and position encoding layer, a deformable cross-attention layer, and a residual layer. The query generation and position encoding layer generates query vectors corresponding to specific spatial positions in the ROI grid and performs position encoding on them to capture the relative spatial information of these positions. The deformable cross-attention layer uses these query vectors to fuse the point cloud bird's-eye view features and the image bird's-eye view features. Finally, the residual layer performs residual connection to obtain preliminary bird's-eye view fusion features.

[0017] Among them, the temporal fusion unit described in step S4 includes an initialization layer, a self-attention update layer, a feedforward network layer and a fusion layer. The preliminary bird's-eye view fusion feature uses the self-attention mechanism in the initialization layer to initialize the query vector; then, the deformable attention mechanism is used in the self-attention update layer to integrate the encoding features of the historical frame into the query vector to update the query vector. The updated query vector is processed by the feedforward network layer to further enrich its feature representation, and finally the final bird's-eye view fusion feature is obtained by fusing the temporal information through the fusion layer.

[0018] The second aspect of the present invention relates to a 3D target detection device based on implicit spatial enhancement and spatiotemporal interaction coding between modalities, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the 3D target detection method based on implicit spatial enhancement and spatiotemporal interaction coding between modalities of the present invention.

[0019] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the 3D target detection method based on implicit spatial enhancement between modalities and spatiotemporal interaction coding of the present invention.

[0020] The advantages of the present invention are as follows:

[0021] The present invention uses the spatial information of LiDAR data for implicit depth estimation through implicit spatial enhancement fusion coding and variable cross-attention mechanism, improves the expressiveness of RGB image features and dynamically adjusts the interaction strength between RGB image features and LiDAR point cloud features. In addition, the present invention also introduces a spatiotemporal interactive feature fusion method, which captures the relationship between modalities and aggregates semantic information in spatial and temporal dimensions through deformable self-attention and deformable cross-attention mechanisms, thereby improving detection accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flow chart of a 3D object detection method based on implicit spatial enhancement between modalities and spatiotemporal interaction coding according to an embodiment of the present application;

[0024] Figure 2 is an overall structural diagram of a 3D object detection model according to an embodiment of the present application;

[0025] Figure 3 is a structural diagram of an implicit spatial enhancement fusion coding network according to an embodiment of the present application;

[0026] Figure 4 It is a structural diagram of a spatiotemporal interactive feature fusion network according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent.

[0030] Embodiment 1

[0031] This example proposes a 3D target detection method based on implicit spatial enhancement and spatiotemporal interactive coding between modalities. Through the implicit spatial enhancement fusion coding module, the spatial information and attention mechanism of LiDAR are used to enhance the geometric details of RGB image features, and the LiDAR point cloud is projected to the image plane for feature correspondence through the camera internal and external parameter matrix. In addition, spatiotemporal interactive feature fusion is adopted, and the deformable attention mechanism is used to capture the feature relationship between modalities in spatial and temporal dimensions, aggregate multimodal semantic information, generate enhanced bird's-eye view features, and fuse historical frame information to improve detection accuracy. Specifically, refer to Figure 1 and Figure 2 , the method comprising:

[0032] S1. Obtain one-to-one corresponding multi-view RGB image data and LiDAR point cloud data as training samples, and construct a 3D target detection architecture, which includes a feature extraction network, an implicit spatial enhancement fusion coding network, a spatiotemporal interactive feature fusion network, and a classification head;

[0033] S2. Input a set of corresponding multi-view RGB image data and LiDAR point cloud data in the training sample into the feature extraction network for feature extraction to obtain RGB image features and LiDAR point cloud features;

[0034] S3. The implicit spatial enhancement fusion coding network includes a preliminary interaction unit and a fine interaction unit. In the preliminary interaction unit, the LiDAR point cloud data is projected into the RGB image coordinate system using the intrinsic parameter matrix and the extrinsic parameter matrix of the camera, and the spatial depth information of the RGB image features is dynamically enhanced through a variable cross-attention mechanism. In the fine interaction unit, the original RGB image features are fused with the enhanced RGB image features through a residual network to obtain the RGB image features under a bird's-eye view.

[0035] S4. The LiDAR point cloud features and the RGB image features from a bird's-eye view are respectively input into a spatiotemporal interactive feature fusion network, wherein the spatiotemporal interactive feature fusion network includes a spatial fusion unit and a temporal fusion unit. In the spatial fusion unit, the LiDAR point cloud features and the RGB image features from a bird's-eye view are subjected to deformable cross-attention to obtain preliminary bird's-eye view features, and in the temporal fusion unit, the preliminary bird's-eye view features are aggregated using a deformable self-attention mechanism to obtain the final fused features.

[0036] S5. Input the final fusion feature into the classification head to obtain the 3D target detection result, construct a loss function based on the detection result, and when the loss function meets the set conditions, save the current parameters of the 3D target detection architecture to obtain the 3D target detection model.

[0037] S6. Input a set of corresponding multi-view RGB image data and LiDAR point cloud data into the trained 3D target detection model to obtain the target detection result.

[0038] In this solution, the multi-view RGB images are multiple images taken by multiple cameras at the same location, which together reflect the complete situation of all targets in the area. The LiDAR point cloud data is a set of spatial points generated by scanning by a three-dimensional laser radar device, and each point contains the three-dimensional coordinate information of the target.

[0039] In this solution, the feature extraction network consists of an RGB feature extraction unit and a LiDAR point cloud feature extraction unit. The RGB feature extraction unit is used to extract features from multi-view RGB images, and the LiDAR point cloud feature extraction unit is used to extract features from LiDAR point cloud data.

[0040] Specifically, the RGB feature extraction unit and the LiDAR point cloud feature extraction unit in this solution can both use any convolutional neural network architecture for feature extraction.

[0041] In this solution, the structure of the implicit spatial enhancement fusion coding network is as follows: Figure 3 As shown, the implicit spatial enhancement fusion coding network is described below:

[0042] In the step of "the preliminary interaction unit projects the LiDAR point cloud data into the RGB image coordinate system using the camera's intrinsic and extrinsic matrix, and dynamically enhances the spatial depth information of the RGB image features through a variable cross-attention mechanism", the preliminary interaction unit includes a projection operation layer, a spatial encoding layer, and a feature fusion layer. In the projection operation layer, the camera's intrinsic and extrinsic matrix is ​​used to project the LiDAR point cloud data onto the RGB image features to establish a coordinate system correspondence; in the spatial encoding layer, the precise spatial information of the LiDAR data is combined with a variable cross-attention mechanism to perform operations to obtain enhanced image features.

[0043] Specifically, the LiDAR point cloud data is mapped to the two-dimensional image coordinate system where the image features are located through the camera's intrinsic parameter matrix and extrinsic parameter matrix, and a spatial correspondence between the coordinate systems is established. The specific formula is as follows:

[0044]

[0045] Where K represents the camera's intrinsic parameter matrix, [R|t] represents the extrinsic parameter matrix from the LiDAR coordinate system to the camera coordinate system, {(x, y, z)} represents the laser point, and {(u, v)} represents the point converted from the laser point to the camera coordinate system.

[0046] Furthermore, the feature point set of the LiDAR point cloud is defined as P LiDAR ={p 1 ,p 2 ,…p N}, at LiDAR point p i = {x, y, z} and its corresponding image feature sampling point q i ={u,v}, a variable cross attention mechanism is applied to construct richer image features. The specific formula is as follows:

[0047]

[0048] in, Represents the LiDAR feature point p i As a query, each LiDAR feature point is projected onto the image plane, and its corresponding sampling point q i As key K and value V in In. F enhance_c represents the initial enhanced features, represents the element of the LiDAR feature at index position (i, j), Atten(·) represents the attention mechanism, Represents the element of the image feature at index position (i, j), d k Indicates the dimension of the key vector.

[0049] In the step of "in the fine interaction unit, the original RGB image features are fused with the enhanced RGB image features through a residual network to obtain the RGB image features from a bird's-eye view", the fine interaction unit includes two residual modules, and the original RGB image features are fused with the enhanced RGB image features through a residual network to obtain the RGB image features from a bird's-eye view.

[0050] Specifically, the first residual module is responsible for preliminary processing of the enhanced image features, extracting features through operations such as convolutional layers, batch normalization, and ReLU activation functions. The second residual module further processes the features to ensure the retention of key information and the enhancement of feature representation. The specific formula is as follows:

[0051] F final_c =F sec (F first (F enhance_c )) (3)

[0052] Among them, F(x) represents the residual function, F first (x) represents the feature output after the first residual unit, and x is the input feature.

[0053] Through the above units, the implicit spatial enhancement fusion coding module can effectively utilize LiDAR data to provide accurate spatial information, and enhance image features through implicit depth estimation, thereby improving the accuracy and robustness of three-dimensional target detection.

[0054] In this scheme, the structure of the spatiotemporal interactive feature fusion network is as follows: Figure 4 As shown, the spatiotemporal interactive feature fusion network is described below:

[0055] In the step of "obtaining preliminary bird's-eye view features after applying deformable cross-attention to the LiDAR point cloud features and the RGB image features under the bird's-eye view in the spatial fusion unit", the spatial fusion unit includes a query generation and position encoding layer, a deformable cross-attention layer and a residual layer. The query generation and position encoding layer generates query vectors corresponding to specific spatial positions in the ROI grid and performs position encoding on them to capture the relative spatial information of these positions. The deformable cross-attention layer uses these query vectors to fuse the point cloud bird's-eye view features and the image bird's-eye view features. Finally, the residual layer performs residual connection to obtain preliminary bird's-eye view fusion features.

[0056] Specifically, we introduce query vectors in the spatial dimension and use the cross-attention mechanism to adaptively fuse RGB image and LiDAR features, maximizing the complementary advantages of different modalities and capturing spatial information from multiple angles. First, the spatial region of the region of interest (ROI) is divided into an H×W grid array, where each grid cell corresponds to a specific position in the ROI space. A set of learnable parameter sets Q is introduced as query vectors, and each query vector q i Mapped to a specific grid unit, and encoded based on its position information in the spatial coordinates, so that the model understands the relative spatial information of each query vector, thereby more accurately locating and identifying objects during feature fusion. In addition, a spatial cross-attention mechanism based on deformable attention is introduced to minimize the dependence on computing resources. The specific formula is as follows:

[0057] Q i =CA(B c ,B l )=DefAtteni(B c ,B l ) (4)

[0058] Among them, B l represents LiDAR features, B c Represents the final image features. The encoded features are divided into N regular grids, and each grid point corresponds to a query vector Q i , DefAtteni(·) represents the deformable attention mechanism. CA(·) represents the cross attention mechanism.

[0059] In the step of "aggregating the preliminary bird's-eye view features using a deformable self-attention mechanism in the temporal fusion unit to obtain the final fused features", the temporal fusion unit includes an initialization layer, a self-attention update layer, a feedforward network layer, and a fusion layer. The preliminary bird's-eye view fused features use the self-attention mechanism in the initialization layer to initialize the query vector; then, the deformable attention mechanism is used in the self-attention update layer to integrate the encoding features of the historical frames into the query vector to update the query vector. The updated query vector is processed by the feedforward network layer to further enrich its feature representation, and finally the final bird's-eye view fused features are obtained by fusing the temporal information through the fusion layer.

[0060] Specifically, the preliminary bird's-eye view fusion features are in the initialization layer, and the query vector is initialized based on the features of the current frame using the self-attention mechanism; then, the deformable attention mechanism is used in the self-attention update layer to integrate the encoding features of the historical frames into the query vector to update the query vector. The specific formula is as follows:

[0061]

[0062] Where, SA(·) represents the self-attention mechanism, which is used to initialize the query vector of the current frame. SA(·) represents the self-attention mechanism. DefAtteni(·) represents the deformable attention mechanism. FFN(·) represents the feedforward network. represents the i-th query vector of the previous frame, Q t represents the query vector of frame t, F t Indicates the encoding features of the current frame.

[0063] Furthermore, the updated query vector is processed by the temporal attention mechanism in the feedforward network layer to further enrich its feature representation, and finally the fusion layer fuses the temporal information to obtain the final bird's-eye view fusion feature. The specific formula is as follows:

[0064]

[0065] Among them, TSA(·) represents the temporal self-attention mechanism. FFN(·) represents the feedforward network, DefAtteni(·) represents the deformable attention mechanism, Q t represents the query vector for frame t. represents the i-th query vector of the current frame, F t-j represents the coding feature of the previous frame, and F represents the final temporal coding feature.

[0066] Specifically, the spatiotemporal interactive feature fusion network uses the above-mentioned spatial fusion unit and temporal fusion unit to establish the connection between the two modal features by applying the deformable attention mechanism in different dimensions, and aggregates semantic information from the multimodal context to obtain an enhanced final feature representation. In the spatial dimension, by introducing a learnable query vector and a spatial cross-attention mechanism, the preliminary BEV feature generation of RGB images and LiDAR features is achieved. Subsequently, in the temporal dimension, the self-attention mechanism and feedforward network are used to integrate the historical frame features and perform multi-layer temporal fusion encoding to obtain the final fused features.

[0067] Specifically, the final fusion feature is input into the classification head, the classification head performs bounding box regression, attribute and classification prediction to obtain a 3D object detection result, and a loss function is constructed based on the detection result. When the loss function meets the set conditions, the current parameters of the 3D object detection architecture are saved.

[0068] Specifically, this scheme uses the data in the nuScenes dataset as training samples and validation sets, and proves the robustness and effectiveness of the model by verifying the 3D object detection model.

[0069] Embodiment 2

[0070] The present embodiment relates to a 3D target detection device based on implicit spatial enhancement and spatiotemporal interaction coding between modalities, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the 3D target detection method based on implicit spatial enhancement and spatiotemporal interaction coding between modalities of Example 1.

[0071] Embodiment 3

[0072] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the 3D target detection method based on implicit spatial enhancement between modalities and spatiotemporal interaction coding of the second embodiment is implemented.

[0073] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A 3D object detection method based on implicit spatial enhancement between modalities and spatiotemporal interaction coding, characterized in that: The following steps are involved: S1: Obtain one-to-one corresponding multi-view RGB image data and LiDAR point cloud data as training samples, and construct a 3D object detection architecture, which includes a feature extraction network, an implicit spatial enhancement fusion coding network, a spatiotemporal interactive feature fusion network, and a classification head; S2: Input a set of corresponding multi-view RGB image data and LiDAR point cloud data in the training sample into the feature extraction network for feature extraction to obtain RGB image features and LiDAR point cloud features; S3: The implicit spatial enhancement fusion coding network includes a preliminary interaction unit and a fine interaction unit. In the preliminary interaction unit, the LiDAR point cloud data is projected into the RGB image coordinate system using the intrinsic parameter matrix and the extrinsic parameter matrix of the camera, and the spatial depth information of the RGB image features is dynamically enhanced through a variable cross-attention mechanism. In the fine interaction unit, the original RGB image features are fused with the enhanced RGB image features through a residual network to obtain the RGB image features under a bird's-eye view. S4: The LiDAR point cloud features and the RGB image features from a bird's-eye view are respectively input into a spatiotemporal interactive feature fusion network, wherein the spatiotemporal interactive feature fusion network includes a spatial fusion unit and a temporal fusion unit. In the spatial fusion unit, the LiDAR point cloud features and the RGB image features from a bird's-eye view are subjected to deformable cross-attention to obtain preliminary bird's-eye features. In the temporal fusion unit, the preliminary bird's-eye features are aggregated using a deformable self-attention mechanism to obtain the final fused features. S5: Input the final fusion feature into the classification head to obtain the 3D target detection result, construct a loss function based on the detection result, and when the loss function meets the set conditions, save the current parameters of the 3D target detection architecture to obtain the 3D target detection model. S6: Input a set of corresponding multi-view RGB image data and LiDAR point cloud data into the trained 3D object detection model to obtain the object detection result.

2. The 3D object detection method based on inter-modal implicit spatial enhancement and spatiotemporal interaction coding according to claim 1, characterized in that: The preliminary interaction unit described in step S3 includes a projection operation layer and a spatial coding layer. In the projection operation layer, the camera's intrinsic parameter matrix and extrinsic parameter matrix are used to project the LiDAR point cloud data onto the RGB image features to establish a coordinate system correspondence; in the spatial coding layer, the precise spatial information of the LiDAR data is combined with a variable cross-attention mechanism to perform operations to obtain enhanced image features.

3. The 3D object detection method based on inter-modal implicit spatial enhancement and spatiotemporal interaction coding according to claim 1, characterized in that: The fine interaction unit described in step S3 includes two residual modules, which fuse the original RGB image features with the enhanced RGB image features through a residual network to obtain the RGB image features from a bird's-eye view.

4. The 3D object detection method based on inter-modal implicit spatial enhancement and spatiotemporal interaction coding according to claim 1, characterized in that: The spatial fusion unit described in step S4 includes a query generation and position encoding layer, a deformable cross-attention layer, and a residual layer. The query generation and position encoding layer generates query vectors corresponding to specific spatial positions in the ROI grid and performs position encoding on them to capture the relative spatial information of these positions. The deformable cross-attention layer uses these query vectors to fuse the point cloud bird's-eye view features and the image bird's-eye view features. Finally, the residual layer performs residual connection to obtain preliminary bird's-eye view fusion features.

5. The 3D object detection method based on inter-modal implicit spatial enhancement and spatiotemporal interaction coding according to claim 1, characterized in that: The temporal fusion unit described in step S4 includes an initialization layer, a self-attention update layer, a feedforward network layer, and a fusion layer. The preliminary bird's-eye view fusion feature uses the self-attention mechanism in the initialization layer to initialize the query vector; then, the deformable attention mechanism is used in the self-attention update layer to integrate the encoding features of the historical frames into the query vector to update the query vector. The updated query vector is processed by the feedforward network layer to further enrich its feature representation, and finally the final bird's-eye view fusion feature is obtained by fusing the temporal information through the fusion layer.

6. A 3D object detection device based on implicit spatial enhancement between modalities and spatiotemporal interaction coding, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the 3D target detection method based on implicit spatial enhancement between modalities and spatiotemporal interaction coding as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the 3D target detection method based on implicit spatial enhancement between modalities and spatiotemporal interaction coding described in any one of claims 1 to 5 is implemented.

Images