Monocular 3D target detection method based on depth perception enhanced Mama

Abstract

The invention belongs to the technical field of monocular 3D target detection, and particularly relates to a monocular 3D target detection method based on depth perception enhancement Mama, and an FMF module in a model constructed by the method focuses on effectively reserving and fusing image information from different scales under relatively low computing resource consumption. By accurately regulating and controlling the information flow, the FMF module can enhance the adaptability and robustness of the model to the scale change while keeping the details of the image. And the DMB module globally integrates the depth information and the visual information by adopting a novel self-adaptive strategy by taking the features fused from the image features as input. The depth fusion strategy not only improves the accuracy of depth estimation, but also enhances the performance of the model under different visual angles and environmental conditions. According to the method, the accuracy and stability of monocular 3D target detection can be ensured on the basis of controlling the cost, so that the development of the fields of robot navigation, automatic driving, multi-target tracking and the like is effectively promoted.

Description

Technical Field

[0001] The present invention belongs to the technical field of monocular 3D target detection, and in particular relates to a monocular 3D target detection method based on depth perception enhanced Mamba. Background Art

[0002] Monocular 3D object detection is a technology that detects and identifies the position and posture of target objects in three-dimensional space from images captured by a single camera.

[0003] In recent years, 2D object detection technology based on convolutional neural networks has developed rapidly. It is widely used in different fields, but in areas such as robot navigation, autonomous driving, and multi-target tracking, it is necessary to obtain detection results that are closer to the real world, and 2D object detection is difficult to achieve. Therefore, in order to obtain detection results that are closer to the real world, many scholars use distance sensors (such as lidar) or binocular cameras as information input devices to obtain accurate depth information. Although higher performance is achieved, it faces the problem of high hardware costs. In contrast, obtaining high-precision 3D detection results from a single image can greatly save computing and equipment costs. At present, some methods that only use monocular cameras have been proposed, and certain effects have been achieved by using the geometric relationship constraints between 2D and 3D. However, since a single image cannot directly obtain depth information, the performance is far from satisfactory, which has attracted the research attention of many scholars.

[0004] In recent years, the field of depth estimation has developed rapidly, and the accuracy of monocular depth estimation models has also improved. Therefore, some scholars use mature monocular depth estimation models to generate estimated depth information to assist monocular 3D target detection tasks. The pseudo-LiDAR-based method converts the estimated depth map into a point cloud, simulates the real LiDAR signal, and then uses a mature LiDAR detector for 3D target detection. There are also fusion-based methods that extract the image features and depth features of the target separately for fusion to perform 3D target detection. Although these methods can locate the target well, they may face the risk of losing valuable features during feature extraction and inaccurate estimation of depth images. In contrast, Transform-based detectors proposed in recent years, such as MonoDTR, can greatly reduce the setting of hyperparameters. At present, some work has shown its superior performance in monocular 3D object detection. MonoDTR uses an image for feature extraction, and then uses the output of multi-scale fusion as the input of other modules. However, in the fusion process, most of them use element-by-element addition or splicing methods, which makes it difficult to fully utilize feature information of different scales. At the same time, both operations are prone to generate a large amount of redundant information, weakening the complementarity between features at different levels. The Depth-Aware Transformer module proposed by MonoDTR can better capture the long-distance dependencies between image information through the self-attention mechanism, and the parallel computing of the multi-head attention mechanism greatly improves the efficiency of training and reasoning, but it may be affected by memory limitations and efficiency, resulting in reduced processing efficiency. In addition, the above model can only process unidirectional sequence data and cannot process image data that lacks a specific direction. In addition, when the Transformer model processes longer sequences, the resource requirements will show a trend of rapid expansion.

[0005] In summary, how to ensure the accuracy and stability of monocular 3D target detection while controlling costs, so as to effectively promote the development of fields such as robot navigation, autonomous driving and multi-target tracking, has become a problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a monocular 3D target detection method based on depth perception enhanced Mamba, which can ensure the accuracy and stability of monocular 3D target detection on the basis of controlling costs, thereby effectively promoting the development of robot navigation, autonomous driving, multi-target tracking and other fields.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A monocular 3D target detection method based on depth perception enhanced Mamba, comprising the following steps:

[0009] S1. Build a MonoDMB model for monocular 3D target detection. The MonoDMB model includes a backbone network, an FMF module, a DAF module, a DMB module, and a detection head. The workflow of the MonoDMB model includes:

[0010] First, the backbone network is used to extract features from the input image to obtain multiple initial features of different scales;

[0011] Then, the initial features of different scales are fused through the FMF module to obtain enhanced fused features; the FMF module includes an initial fusion submodule and a detail fusion submodule; the initial fusion submodule is used to process the initial features extracted by the backbone network according to a preset feature fusion method and then perform concat splicing to obtain the initial fused features; the detail fusion submodule is used to process the initial fused features through a set of parallel deep convolutions to capture rich information across multiple scales, so that the detailed context information of the features of each scale can be better integrated;

[0012] Then, the convolution layer is used to obtain the image visual features of the enhanced fusion features. At the same time, the DAF module is used to obtain the depth perception features of the enhanced fusion features through the Depth-Assisted Perception mechanism. The image visual features and depth perception features are summed element by element to obtain the comprehensive features.

[0013] Afterwards, the DMB module is used to globally integrate the depth information and visual information according to the preset adaptive strategy of the comprehensive features to obtain the final integrated features;

[0014] Finally, the detection head is used to perform 3D object detection on the final integrated features;

[0015] S2, training the MonoDMB model;

[0016] S3. Use the trained MonoDMB model to perform actual monocular 3D target detection.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. An FMF module is set up in this method. Its initial fusion submodule fuses initial features of different scales through a preset feature fusion method (such as concat splicing), which helps the model capture multi-level information in the image. Its detail fusion submodule uses parallel deep convolution to process the initial fusion features, which can further capture detailed contextual information across scales and improve the richness and accuracy of feature representation. This multi-scale fusion strategy helps the model better understand the target structure in complex scenes. The FMF module uses specific feature focusing and feature diffusion mechanisms to diffuse features rich in contextual information to each detection scale, effectively avoiding noise interference.

[0019] The FMF proposed in this method can not only capture the long-distance dependencies between different scales and fully focus on the contextual information of different layers to enable better fusion and enhancement, but also does not adopt the attention mechanism, which significantly reduces the computational cost.

[0020] 2. Through the Depth-Assisted Perception mechanism, the model's ability to understand the depth of the target can be significantly improved by combining the image's visual features and depth perception features. This is crucial for monocular 3D target detection, because depth information is the key to reconstructing 3D spatial positions. The element-by-element summation method effectively combines the two features, allowing the model to use both the image's visual information and depth information during detection, improving the accuracy and robustness of detection.

[0021] 3. This method innovatively integrates the DMB module into monocular 3D target detection. The DMB module globally integrates depth information and visual information according to a preset adaptive strategy, which can further optimize feature representation and enable the model to better locate targets in complex 3D space, thereby significantly enhancing the performance of subsequent depth prediction heads. This adaptive integration strategy not only improves the accuracy of depth estimation, but also optimizes the performance of the model under different viewing angles and environmental conditions.

[0022] 4. The FMF module and DMB module proposed in this method can be easily plug-and-play into the existing image-only based framework to improve the performance.

[0023] 5. Through comprehensive experiments on the KITTI dataset, it is verified that the model of this method can achieve real-time detection and is significantly better than the existing advanced monocular detection methods.

[0024] In summary, this method can ensure the accuracy and stability of monocular 3D target detection while controlling costs, thereby effectively promoting the development of fields such as robot navigation, autonomous driving and multi-target tracking.

[0025] Preferably, the initial fusion submodule includes an Aconv unit, a 1×1 convolution layer, and an upsampling layer; when the initial fusion submodule is working, each initial feature is processed in parallel through the Aconv unit, the 1×1 convolution layer, and the upsampling layer;

[0026] Among them, the Aconv unit includes an average pooling-convolution AC subunit and a maximum pooling-convolution MC subunit, which are used to process the initial features in parallel to capture different feature information; the 1×1 convolution layer is used to reduce the dimension of the initial features so that they can be concat-joined with the output of the Aconv unit; the upsampling layer is used to adjust the resolution of feature maps of different scales so that they can have the same size when concat-joined.

[0027] With this setup, the parallel processing mechanism of the average pooling-convolution AC subunit and the maximum pooling-convolution MC subunit enables the initial features to be extracted from different angles (average pooling and maximum pooling). This parallel processing method helps to capture richer and more diverse feature information and improves the representation ability of the model. The AC and MC subunits use average pooling and maximum pooling respectively to reduce the spatial dimension and retain important information, and then further extract features through the convolution layer, which helps the model learn more representative feature representations.

[0028] The 1×1 convolution layer performs feature dimensionality reduction, which helps reduce the amount of computation and the complexity of the model while retaining key information. Through dimensionality reduction, the feature map output by this layer matches the output of the Aconv unit in dimension, which facilitates the subsequent concat splicing operation. The concatenated features combine the diversity of the original features, the average pooling characteristics of the AC subunit, the maximum pooling characteristics of the MC subunit, and the dimensionality reduction characteristics of the 1×1 convolution layer, making the fused features richer and more comprehensive. The upsampling layer is used to adjust the resolution of feature maps of different scales to ensure that they have the same size when concat spliced. It enables features of different scales to be fused at the same spatial resolution, thereby making full use of information at each scale.

[0029] In summary, the initial fusion submodule achieves refined processing and efficient fusion of initial features through the collaborative work of Aconv units, 1×1 convolutional layers, and upsampling layers. This multi-scale, multi-path feature fusion method helps to improve the model's representation ability and detection performance, making the MonoDMB model perform better in monocular 3D object detection tasks.

[0030] Preferably, the working process of the detail fusion submodule includes:

[0031] Firstly, the deep separable convolution DWConv technology is used to process multiple channels of the initial fusion feature F in parallel to obtain multiple channel processing features optimized by DWConv.

[0032] Then, these channel processing features optimized by DWConv are stacked according to the 0th dimension to obtain stacked features; the stacking refers to merging multiple feature maps into a larger feature map in the channel dimension;

[0033] Then, the initial fusion feature F is added and integrated with the stacked feature using the convolution and skip residual connection mechanism to obtain the detailed fusion feature F feature ;

[0034] Finally, the size of the detailed fusion features is doubled by applying the transposed convolution TConv to obtain the final enhanced fusion feature map.

[0035] In this way, through the deep separable convolution DWConv technology, multiple channels of the initial fusion features are processed in parallel and efficiently. DWConv reduces the amount of calculation and the number of parameters while maintaining good feature extraction capabilities, which enables the features of each channel to be optimized in a targeted manner without affecting other channels, thereby improving the diversity and representation of features. Stacking the channel processing features optimized by DWConv is essentially merging and enhancing features in the channel dimension. This stacking operation not only expands the number of channels of the feature map, but also enables the fusion of information between different channels, thereby enhancing the expressiveness of the feature map. The initial fusion feature F is added and integrated with the stacked features using the convolution and skip residual connection mechanism. The residual connection helps to retain the important information of the original features while allowing the network to learn the residual between input and output, which helps to alleviate the problem of gradient vanishing or explosion in deep networks, promotes the flow of information, and makes the fused features more detailed and rich in information. The resulting enhanced fusion feature map not only retains the important information of the original features, but also improves the richness and representation of features through a series of optimization and enhancement steps.

[0036] Preferably, the comprehensive features are converted into a flattened two-dimensional sequence and then processed using the DMB module.

[0037] Such a setting, by flattening the comprehensive features into a two-dimensional sequence, can eliminate the differences in the original features in spatial dimensions (such as height, width), making the feature representation more uniform in form. This uniformity helps to simplify the subsequent processing flow and enables the DMB module to process these features more efficiently. In the form of a two-dimensional sequence, the DMB module can more easily capture the association between depth information and visual information because all information is organized on the same plane. This enables the DMB module to more effectively use depth information to guide the processing of visual information, thereby improving the model's ability to understand the spatial position of 3D targets.

[0038] Preferably, the process of converting the comprehensive features into a flattened two-dimensional sequence includes:

[0039] Using the preset window size and step size, the comprehensive feature F d ∈R C×H×W Sliding window operation is performed on the image to divide it into multiple local area patches; each patch is flattened from three dimensions (k×k×C) to a one-dimensional vector, and then all one-dimensional vectors are combined into a vector with a dimension of (p, k 2 C), where p is the total number of patches; then, these vectors are converted into vectors of dimension E through linear projection to obtain the final tokens sequence of dimension (p, E).

[0040] This setting converts the comprehensive features into two-dimensional sequence tokens, which makes it easier for the subsequent DMB module to process these tokens, thereby effectively capturing the long-term dependencies between elements in the sequence, and further exploring the potential connections and patterns between features. This is especially important for processing features with complex structures and dynamic changes. By converting each patch vector into a vector of fixed dimension E through linear projection, all tokens can have the same representation, which is convenient for subsequent unified processing.

[0041] Preferably, the working process of the DMB module includes:

[0042] First, the input two-dimensional sequence tokens are normalized and then processed through two parallel branches;

[0043] In the first branch, tokens are linearly projected into a vector D of dimension E, and then deformable convolution DCN is used for feature extraction, followed by SiLU activation function, DSSM layer and Norm processing to obtain the first feature; the DSSM layer is used to enhance the semantic dependency between tokens;

[0044] In the second branch, tokens are also linearly projected into a vector O of dimension E, and then processed using the SiLU activation function to obtain the second feature;

[0045] Afterwards, the features output by the two branches are aggregated by element-wise multiplication to obtain the fused features;

[0046] Finally, the fused features are processed and projected back to the size of the original input image to obtain an output image with the same shape as the input image.

[0047] With this setup, tokens are processed in the first branch through deformable convolution (DCN), which can dynamically adjust the sampling position of the convolution kernel according to the local characteristics of the feature map, thereby more flexibly capturing the geometric deformation and scale changes in the image. This feature makes the DMB module more robust and effective in extracting features, and can enhance the representation ability of features. Through the DSSM layer, the model can learn the deep semantic relationship between tokens, making the association between features closer and more accurate. This is of great significance for improving the semantic understanding and reasoning ability of the model.

[0048] The DMB module uses two parallel branches to process tokens, and each branch uses a different processing method, so that the features output by the two branches are diverse and complementary. This design helps the model capture and represent features from multiple angles, thereby improving the performance of the model. The features output by the two branches are aggregated by element-wise multiplication to achieve deep fusion of features. This aggregation method can retain the key information in the two branches and make full use of the complementarity between them to generate a richer and more comprehensive feature representation. Finally, the fused features are processed and projected back to the size of the initial input image, thereby maintaining the same shape as the input image. This step ensures the spatial consistency of the model output, making the model more intuitive and convenient when processing image tasks.

[0049] In summary, the DMB module significantly improves the performance of the model through a series of operations such as feature enhancement, semantic dependency enhancement, feature diversity and complementarity, feature aggregation and fusion, and maintaining spatial consistency. Whether in feature extraction, semantic understanding or image reconstruction, the DMB module can bring significant advantages to the model.

[0050] Preferably, the DSSM layer includes two branches, wherein the first branch includes a forward branch and a reverse branch; the process of the DSSM layer performing feature processing includes:

[0051] In the forward branch of the first branch, the features are transformed by the linear layer and mapped to a new feature space to enhance their expressiveness. After that, the 1D convolution layer is used with the SiLU activation function to introduce nonlinear characteristics and enhance the learning ability of the features. It is combined with the SSM layer to promote feature extraction. The reverse branch of the first branch performs the same processing sequence as the forward branch in the opposite direction.

[0052] In the second branch, first, the features are transformed by the linear layer; then they are processed using the SiLU activation function to maintain the nonlinear characteristics of feature propagation;

[0053] Then, the output features of the two branches are fused through the Hadamard product;

[0054] Finally, the fused features are reshaped back to their original dimensions, ready to be processed by the next layer of the network.

[0055] Such a setting, in the forward branch, not only increases the depth of the model, but also introduces nonlinear characteristics through the 1D convolution layer with the SiLU activation function. This helps the model capture complex patterns and relationships in the input data, thereby improving the learning and generalization capabilities of the model. The forward branch and the reverse branch perform the same processing sequence in opposite directions. This two-way processing method can extract and fuse feature information from different angles, forming a complementary relationship, which helps to understand and represent the input data more comprehensively. The output features of the two branches are fused through the Hadamard product, while retaining the original feature information, the interaction and fusion between the features are achieved, which helps to generate richer and more meaningful feature representations. Finally, the fused features are reshaped back to their original dimensions. This step ensures that the feature vector maintains a consistent dimension when it is input to the next layer of the network, which facilitates subsequent processing. At the same time, this also provides better input data for the learning of subsequent layers of the network, which helps to further improve the performance of the entire model.

[0056] Preferably, the detection head adopts a single-stage detection method to determine the bounding box by using pre-set 2D-3D anchor points;

[0057] Each anchor point is set including 2D bounding box parameters [x 2d ,y 2d ,w 2d ,h 2d ] and 3D bounding box parameters [x p ,y p ,z,w 3d ,h 3d ,l 3d ,θ]; where [x 2d ,y 2d ] represents the center point of the 2D box, [w 2d,h 2d ] is the size of the 2D box; [x p ,y p ] represents the position where the center of the 3D object is projected onto the image, [w 3d ,h 3d ,l 3d ] is the size of the 3D box, parameter z refers to the distance from the center of the 3D object to the camera, and θ represents the observation angle;

[0058] During training, all true labels are mapped to 2D space to calculate the intersection over union (IoU) with each 2D anchor point; when the IoU exceeds 0.5, the anchor point is selected for optimal assignment with the corresponding 3D box;

[0059] For each anchor point, a set of 2D parameters [t x ,t y ,t w ,t h ] and a set of 3D parameters [t x ,t y ,t w ,t h ,t l ,t z ,t θ ], these parameters are used to quantify the deviation of 2D and 3D bounding boxes and output a classification score cls for each category; by combining the reference position of the anchor point and the prediction result of the network, the bounding box of the target is restored.

[0060] With this setting, the stage detection method is more efficient than the two-stage detection method (such as generating candidate regions first and then performing classification and position regression). By setting the 2D and 3D bounding box parameters for each anchor point, the two-dimensional image plane position and three-dimensional spatial position of the target can be captured simultaneously. This joint optimization helps to improve the accuracy of target detection, especially in scenarios that require accurate spatial information (such as autonomous driving, robot navigation, etc.). This method can efficiently utilize training data by mapping the true label to 2D space to calculate the intersection over union (IoU) with the anchor point, and selecting and optimizing the allocation of anchor points based on the IoU value. Only when the IoU between the anchor point and the true label exceeds a certain threshold (such as 0.5), it will be used in the subsequent optimization process, which reduces unnecessary calculations and accelerates training convergence.

[0061] In addition, a set of 2D and 3D parameters are predicted for each anchor point to quantify the deviation of the bounding box, which enables the model to adjust the position and size of the bounding box more accurately. This refined deviation prediction helps to generate a bounding box that fits the target more closely, thereby improving the accuracy of detection. By outputting a classification score for each category, this method can detect multiple categories of objects at the same time. This multi-category detection capability makes this method more general and flexible, and suitable for a variety of complex practical application scenarios.

[0062] Preferably, in S2, when training the MonoDMB model, the loss function includes the classification loss L cls , regression loss L reg and the depth loss L dep .

[0063] Preferably, the classification loss L cls and regression loss L reg , the expression is:

[0064]

[0065] In the formula, α is the weight coefficient for balancing positive and negative samples, which is used to adjust the importance of different categories; γ is the focusing parameter for adjusting the weight of easy-to-classify samples, which is used to reduce the weight of easy-to-classify samples and make the model pay more attention to those difficult-to-classify samples; δ is the threshold parameter.

[0066] Preferably, the depth loss L dep The expression is:

[0067]

[0068] Where P is the pixel area with valid depth labels on the image; D represents the depth value predicted by the model. represents the depth label, and FL represents the calculation using the focal loss function.

[0069] With this setup, the MonoDMB model achieves comprehensive performance improvement in target detection tasks by optimizing classification loss, regression loss, and depth loss simultaneously. Classification loss improves the accuracy of target recognition, regression loss optimizes the accuracy of target positioning, and depth loss enhances the model's perception in three-dimensional space. This multi-loss joint optimization strategy makes the MonoDMB model more robust and effective in dealing with complex and changeable real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0071] Figure 1This is the overall architecture diagram of the MonoDMB model in this method;

[0072] Figure 2 This is a diagram of the FMF module architecture in Example 1;

[0073] Figure 3 This is a DMB module architecture diagram in Example 1;

[0074] Figure 4 Schematic diagram of the M3D-RPN model after fusion in Example 2;

[0075] Figure 5 This is a schematic diagram of a qualitative example on the KITTI validation set in Example 2. DETAILED DESCRIPTION

[0076] The following is a further detailed description through specific implementation methods:

[0077] Embodiment 1

[0078] This embodiment discloses a monocular 3D target detection method based on depth perception enhanced Mamba, including the following steps:

[0079] S1. Build the MonoDMB model for monocular 3D target detection.

[0080] like Figure 1 As shown in FIG. 1 , the MonoDMB model includes a backbone network, an FMF module, a DAF module, a DMB module, and a detection head; the workflow of the MonoDMB model includes:

[0081] First, the backbone network is used to extract features from the input image and obtain multiple initial features of different scales.

[0082] Then, the initial features of different scales are fused through the FMF module to obtain enhanced fused features; the FMF module includes an initial fusion submodule and a detail fusion submodule; the initial fusion submodule is used to process the initial features extracted by the backbone network according to a preset feature fusion method and then concat them to obtain the initial fused features; the detail fusion submodule is used to process the initial fused features through a set of parallel deep convolutions to capture rich information across multiple scales, so that the detailed contextual information of the features of each scale can be better integrated.

[0083] Then, the convolution layer is used to obtain the image visual features of the enhanced fusion features. At the same time, the DAF module is used to obtain the depth perception features of the enhanced fusion features through the Depth-Assisted Perception mechanism. The image visual features and depth perception features are summed element by element to obtain the comprehensive features.

[0084] Afterwards, the DMB module is used to globally integrate the depth information and visual information according to the preset adaptive strategy of the comprehensive features to obtain the final integrated features.

[0085] Finally, the detection head is used to perform 3D object detection on the final integrated features.

[0086] About FMF Modules

[0087] Existing multi-scale feature fusion methods mainly focus on the fusion of contextual information between different layers. However, due to the limitation of the receptive field size, it is difficult to fuse the contextual information of different layers together, and adding an attention mechanism easily increases the risk of additional computational burden. To alleviate this situation, this method proposes a global focusing FMF module for 3D object detection based on monocular images. Compared with existing multi-scale fusion methods, the FMF proposed in this method can not only capture the long-range dependencies between different scales, fully focus on the contextual information of different layers, and better integrate and enhance them, but also does not use an attention mechanism, which significantly reduces the computational cost.

[0088] In specific implementation, the architecture of the FMF module is as follows Figure 2 The left box is the initial fusion submodule, and the right box is the detail fusion submodule.

[0089] The initial fusion submodule includes Aconv unit, 1×1 convolution layer and upsampling layer. When the initial fusion submodule works, each initial feature is processed in parallel through Aconv unit, 1×1 convolution layer and upsampling layer.

[0090] Among them, the Aconv unit includes an average pooling-convolution AC subunit and a maximum pooling-convolution MC subunit, which are used to process the initial features in parallel to capture different feature information; the 1×1 convolution layer is used to reduce the dimension of the initial features so that they can be concat-joined with the output of the Aconv unit; the upsampling layer is used to adjust the resolution of feature maps of different scales so that they can have the same size when concat-joined.

[0091] In this way, the parallel processing mechanism of the average pooling-convolution AC subunit and the maximum pooling-convolution MC subunit enables the initial features to be extracted from different angles (average pooling and maximum pooling). This parallel processing method helps to capture richer and more diverse feature information and improves the representation ability of the model. The AC and MC subunits use average pooling and maximum pooling respectively to reduce the spatial dimension and retain important information, and then further extract features through the convolution layer, which helps the model learn more representative feature representations. The 1×1 convolution layer performs feature dimensionality reduction, which helps to reduce the amount of calculation and reduce the complexity of the model while retaining key information. Through dimensionality reduction, the feature map output by this layer matches the output of the Aconv unit in dimension, which is convenient for the subsequent concat splicing operation. The spliced ​​features combine the diversity of the original features, the average pooling characteristics of the AC subunit, the maximum pooling characteristics of the MC subunit, and the dimensionality reduction characteristics of the 1×1 convolution layer, making the fused features richer and more comprehensive. The upsampling layer is used to adjust the resolution of feature maps of different scales to ensure that they have the same size when concat splicing. It enables features of different scales to be fused at the same spatial resolution, thereby making full use of information at each scale. In summary, the initial fusion submodule achieves refined processing and efficient fusion of initial features through the collaborative work of Aconv units, 1×1 convolutional layers, and upsampling layers. This multi-scale, multi-path feature fusion method helps to improve the model's representation ability and detection performance, allowing the MonoDMB model to perform better in monocular 3D object detection tasks.

[0092] The working process of the detail fusion submodule includes:

[0093] Firstly, the deep separable convolution DWConv technology is used to process multiple channels of the initial fusion feature F in parallel to obtain multiple channel processing features optimized by DWConv.

[0094] Then, these channel processing features optimized by DWConv are stacked according to the 0th dimension to obtain stacked features; the stacking refers to merging multiple feature maps into a larger feature map in the channel dimension;

[0095] Then, the initial fusion feature F is added and integrated with the stacked feature using the convolution and skip residual connection mechanism to obtain the detailed fusion feature F feature ;

[0096] Finally, the size of the detailed fusion feature is doubled by applying the transposed convolution TConv to obtain the final enhanced fusion feature map F out .

[0097] The expression of the working process of the detail fusion submodule is:

[0098]

[0099] Where Tconv represents transposed convolution; F feature represents detailed fusion features; F represents initial fusion features; Indicates element-wise sum element by element; DW i It represents depth-wise separable convolution; Stack represents stacking; Reduction represents dimension reduction.

[0100] In this way, through the deep separable convolution DWConv technology, multiple channels of the initial fusion features are processed in parallel and efficiently. DWConv reduces the amount of calculation and the number of parameters while maintaining good feature extraction capabilities, which enables the features of each channel to be optimized in a targeted manner without affecting other channels, thereby improving the diversity and representation of features. Stacking the channel processing features optimized by DWConv is essentially merging and enhancing features in the channel dimension. This stacking operation not only expands the number of channels of the feature map, but also enables the fusion of information between different channels, thereby enhancing the expressiveness of the feature map. The initial fusion feature F is added and integrated with the stacked features using the convolution and skip residual connection mechanism. The residual connection helps to retain the important information of the original features while allowing the network to learn the residual between input and output, which helps to alleviate the problem of gradient vanishing or explosion in deep networks, promotes the flow of information, and makes the fused features more detailed and rich in information. The resulting enhanced fusion feature map not only retains the important information of the original features, but also improves the richness and representation of features through a series of optimization and enhancement steps.

[0101] About DMB Module

[0102] In order to obtain depth perception features for subsequent fusion with visual features, this method draws on the idea of ​​MonoDTR and introduces the DAF module on this basis. The DAF module effectively learns and refines depth-related feature information by introducing accurate depth maps as auxiliary supervision signals. This ensures that the DMB module can more comprehensively and deeply integrate depth perception features and visual features, thereby capturing and optimizing the subtle information after the fusion of the two, and improving the overall detection performance and accuracy.

[0103] In specific implementation, after the comprehensive features are converted into a flattened two-dimensional sequence, the DMB module is used for processing. The process of converting the comprehensive features into a flattened two-dimensional sequence includes:

[0104] Using the preset window size and step size, the comprehensive feature F d ∈R C×H×WSliding window operation is performed on the image to divide it into multiple local area patches; each patch is flattened from three dimensions (k×k×C) to a one-dimensional vector, and then all one-dimensional vectors are combined into a vector with a dimension of (p, k 2 C), where p is the total number of patches; then, these vectors are converted into vectors of dimension E through linear projection to obtain the final tokens sequence of dimension (p, E).

[0105] In this way, by flattening the comprehensive features into a two-dimensional sequence, the differences in the original features in spatial dimensions (such as height and width) can be eliminated, making the feature representation more unified in form. This uniformity helps to simplify the subsequent processing flow and enables the DMB module to process these features more efficiently. After converting the comprehensive features into two-dimensional sequence tokens, it is convenient for the subsequent DMB module to process these tokens, thereby effectively capturing the long-term dependencies between elements in the sequence, so as to further explore the potential connections and laws between features. This is especially important for processing features with complex structures and dynamic changes. By converting each patch vector into a vector of fixed dimension E through linear projection, all tokens can have the same representation, which is convenient for subsequent unified processing.

[0106] The DMB module architecture is as follows Figure 3 (a) The working process of the DMB module includes:

[0107] First, the input two-dimensional sequence tokens are normalized and then processed through two parallel branches;

[0108] In the first branch, tokens are linearly projected into a vector D of dimension E, and then deformable convolution DCN is used for feature extraction, followed by SiLU activation function, DSSM layer and Norm processing to obtain the first feature; among them, the DSSM layer is used to enhance the semantic dependency between tokens.

[0109] In the second branch, tokens are also linearly projected into a vector O of dimension E, and then processed using the SiLU activation function to obtain the second feature.

[0110] Afterwards, the features output by the two branches are aggregated by element-wise multiplication to obtain the fused features;

[0111] Finally, the fused features are processed and projected back to the size of the original input image to obtain an output image with the same shape as the input image.

[0112] Among them, the DSSM layer in the first branch is as follows Figure 3As shown in (b), in the forward branch of the first branch, the features are transformed by the linear layer and mapped to a new feature space to enhance their expressiveness. After that, a 1D convolution layer with SiLU activation function is used to introduce nonlinear characteristics and enhance the learning ability of the features. This process is combined with the SSM layer to promote the effective extraction of features. The reverse branch performs the same processing sequence as the forward branch in the opposite direction to ensure the comprehensive communication and refinement of feature information, thereby deepening the model's understanding of details. As for the second branch, it also starts with a linear mapping step to retain more global coarse-grained information, which is crucial to avoid the gradient explosion phenomenon that may occur during training. It is followed by the SiLU activation function to maintain the nonlinear characteristics of feature propagation. The features of the two branches are then fused through the Hadamard product, which is an aggregation method that can effectively combine the advantages of features from different perspectives. Finally, the fused features are reshaped back to their original dimensions and are ready to enter the next layer of the network for processing.

[0113] In this way, tokens are processed in the first branch through deformable convolution (DCN), which can dynamically adjust the sampling position of the convolution kernel according to the local characteristics of the feature map, so as to more flexibly capture the geometric deformation and scale changes in the image. This feature makes the DMB module more robust and effective in extracting features and can enhance the representation ability of features. Through the DSSM layer, the model can learn the deep semantic relationship between tokens, making the association between features closer and more accurate. This is of great significance for improving the semantic understanding and reasoning ability of the model. The DMB module uses two parallel branches to process tokens, and each branch uses a different processing method, so that the features output by the two branches are diverse and complementary. This design helps the model capture and represent features from multiple angles, thereby improving the performance of the model. The features output by the two branches are aggregated by element-wise multiplication, realizing deep fusion of features. This aggregation method can retain the key information in the two branches and make full use of their complementarity to generate a richer and more comprehensive feature representation. Finally, the fused features are processed and projected back to the size of the initial input image, so as to maintain the same shape as the input image. This step ensures the spatial consistency of the model output, making the model more intuitive and convenient when processing image tasks.

[0114] The DMB module significantly improves the performance of the model through a series of operations such as feature enhancement, semantic dependency enhancement, feature diversity and complementarity, feature aggregation and fusion, and maintaining spatial consistency. Whether in feature extraction, semantic understanding or image reconstruction, the DMB module can bring significant advantages to the model.

[0115] About the sensor head

[0116] In the specific implementation, the detection head adopts a single-stage detection method to determine the bounding box by using the pre-set 2D-3D anchor points;

[0117] Each anchor point is set including 2D bounding box parameters [x 2d ,y 2d ,w 2d ,h 2d ] and 3D bounding box parameters [x p ,y p ,z,w 3d ,h 3d ,l 3d ,θ]; where [x 2d ,y 2d ] represents the center point of the 2D box, [w 2d ,h 2d ] is the size of the 2D box; [x p ,y p ] represents the position where the center of the 3D object is projected onto the image, [w 3d ,h 3d ,l 3d ] is the size of the 3D box, parameter z refers to the distance from the center of the 3D object to the camera, and θ represents the observation angle;

[0118] During training, all true labels are mapped to 2D space to calculate the intersection over union (IoU) with each 2D anchor point; when the IoU exceeds 0.5, the anchor point is selected for optimal assignment with the corresponding 3D box;

[0119] For each anchor point, a set of 2D parameters [t x ,t y ,t w ,t h ] and a set of 3D parameters [t x ,t y ,t w ,t h ,t l ,t z ,t θ ], these parameters are used to quantify the deviation of 2D and 3D bounding boxes and output a classification score cls for each category; by combining the reference position of the anchor point and the prediction result of the network, the bounding box of the target is restored.

[0120] In this way, the stage detection method is more efficient than the two-stage detection method (such as generating candidate regions first and then performing classification and position regression). By setting 2D and 3D bounding box parameters for each anchor point, the 2D image plane position and 3D spatial position of the target can be captured simultaneously. This joint optimization helps improve the accuracy of target detection, especially in scenarios that require accurate spatial information (such as autonomous driving, robot navigation, etc.). By mapping the true label to 2D space to calculate the intersection over union (IoU) with the anchor point, and selecting and optimizing the allocation of anchor points based on the IoU value, this method can efficiently utilize the training data. Only when the IoU between the anchor point and the true label exceeds a certain threshold (such as 0.5), it is used in the subsequent optimization process, which reduces unnecessary calculations and accelerates training convergence. In addition, a set of 2D and 3D parameters are predicted for each anchor point to quantify the deviation of the bounding box, which enables the model to adjust the position and size of the bounding box more accurately. This refined deviation prediction helps to generate a bounding box that fits the target more closely, thereby improving the accuracy of detection. By outputting a classification score for each category, this method can detect multiple categories of objects at the same time. This multi-category detection capability makes the method more general and flexible, suitable for various complex practical application scenarios.

[0121] S2. Train the MonoDMB model.

[0122] In specific implementation, when training the MonoDMB model, the loss function includes the classification loss L cls , regression loss L reg and the depth loss L dep .

[0123] Classification loss L cls and the regression loss L reg The focal loss and smooth L1 loss are used respectively, and the expressions are:

[0124]

[0125]

[0126] In the formula, α is the weight coefficient for balancing positive and negative samples, which is used to adjust the importance of different categories; γ is the focusing parameter for adjusting the weight of easy-to-classify samples, which is used to reduce the weight of easy-to-classify samples and make the model pay more attention to those difficult-to-classify samples; δ is the threshold parameter.

[0127] Depth loss L dep The expression is:

[0128]

[0129] Where P is the pixel area with valid depth labels on the image; D represents the depth value predicted by the model. represents the depth label, and FL represents the calculation using the focal loss function.

[0130] With this setup, the MonoDMB model achieves comprehensive performance improvement in target detection tasks by optimizing classification loss, regression loss, and depth loss simultaneously. Classification loss improves the accuracy of target recognition, regression loss optimizes the accuracy of target positioning, and depth loss enhances the model's perception in three-dimensional space. This multi-loss joint optimization strategy makes the MonoDMB model more robust and effective in dealing with complex and changeable real-world scenarios.

[0131] S3. Use the trained MonoDMB model to perform actual monocular 3D target detection.

[0132] The FMF module is set in this method. Its initial fusion submodule fuses the initial features of different scales through a preset feature fusion method (such as concat splicing), which helps the model capture multi-level information in the image. Its detail fusion submodule uses parallel deep convolution to process the initial fusion features, which can further capture detailed contextual information across scales and improve the richness and accuracy of feature representation. This multi-scale fusion strategy helps the model better understand the target structure in complex scenes. The FMF module uses a specific feature focusing and feature diffusion mechanism to diffuse features rich in contextual information to each detection scale, effectively avoiding noise interference. The FMF proposed in this method can not only capture the long-distance dependencies between different scales, fully focus on the contextual information of different layers, and better integrate and enhance them, but also does not use the attention mechanism, significantly reducing the computational cost. In addition, through the Depth-Assisted Perception mechanism, combined with image visual features and depth perception features, the model's ability to understand the depth of the target can be significantly improved. This is crucial for monocular 3D target detection, because depth information is the key to reconstructing 3D spatial position. The element-by-element summation method effectively combines the two features, allowing the model to simultaneously utilize the visual information and depth information of the image during detection, thereby improving the accuracy and robustness of detection.

[0133] In addition, this method innovatively integrates the DMB module into monocular 3D target detection. The DMB module globally integrates depth information and visual information according to a preset adaptive strategy, which can further optimize the feature representation and enable the model to better locate the target in a complex 3D space, thereby significantly enhancing the performance of the subsequent depth prediction head. This adaptive integration strategy not only improves the accuracy of depth estimation, but also optimizes the performance of the model under different viewing angles and environmental conditions. The FMF module and DMB module proposed in this method can be easily plug-and-play with existing image-based frameworks to improve performance. This method can ensure the accuracy and stability of monocular 3D target detection while controlling costs, thereby effectively promoting the development of fields such as robot navigation, autonomous driving, and multi-target tracking.

[0134] Embodiment 2

[0135] In order to better illustrate the effect of this method, the following experimental verification is carried out.

[0136] Experimental setup

[0137] In order to verify the effectiveness of this method, the KITTI 3D object detection dataset, which is widely used in 3D object detection tasks, was selected as the evaluation platform. The dataset contains a total of 7481 training images and 7518 test images. In the experiment, the training set was divided into two subsets: a training subset containing 3712 images and a validation subset containing 3769 images. Under this partitioning framework, ablation experiments were carried out to systematically analyze and verify the contribution of each component to the improvement of model performance, ensuring the reliability of the experimental results and the rationality of the methodology.

[0138] In terms of evaluation methods, both for 3D object detection and bird's-eye view (BEV) detection tasks, Average Precision (AP) is used as the key indicator to measure performance. In order to reduce the potential impact of bias, the metric AP40 for 40 recall positions is selected. In the benchmark test, the detection difficulty is divided into three levels: "easy", "medium" and "difficult" according to the size of the target, the degree of occlusion and the truncation condition. The ranking of each method is mainly based on the 3D AP value at the "medium" difficulty (Mod.) consistent with the KITTI benchmark. At the same time, following the official settings, the intersection-over-union ratio thresholds for target detection of car, cyclist and pedestrian categories are set to 0.7, 0.5 and 0.5 respectively.

[0139] The specific operations in the implementation process are as follows: The Adam optimizer is used to train the network model. The entire training process will last for 120 cycles, and the amount of data processed in each batch is 12 samples. The initial learning rate is set to 0.0001 and dynamically adjusted by cosine annealing. At each pixel position in the feature map, 48 anchor boxes are designed, covering 3 different aspect ratios (i.e., 0.5, 1.0, and 1.5). At the same time, according to the exponential function 24×2 i / 4 Generate 12 anchor boxes of different height sizes, where i increases from 0 to 15. For the parameter setting of the 3D anchor box, first calculate the mean and variance of the 3D real bounding box in the training dataset as a reference for the prior statistical characteristics of the anchor box. In order to reduce the time consumption of inference, the top 100 pixels of each image are cropped in the preprocessing stage, and all images are uniformly scaled to 288×1280. In the training phase, random horizontal flipping technology is used for data enhancement. In the inference phase, the model will eliminate those prediction results with a confidence level lower than 0.75, and use the non-maximum suppression (NMS) algorithm with an IoU threshold of 0.4 to eliminate redundant predictions.

[0140] Main Results

[0141] In the car category results of the KITTI test set, as shown in Table 1, this method is compared with other advanced monocular 3D detection methods in recent years. It can be found that MonoDMB has achieved the best performance on AP3D medium objects, which is the most important indicator for KITTI. This is because medium samples are usually small in size, so the previous algorithm fusion features are prone to fusion of too much coarse-grained information, resulting in detection errors. In contrast, MonoDMB alleviates the excessive coarse-grained information in the feature fusion process by using the FMF module, and focuses the information more on the details, thereby bringing significant performance improvements.

[0142] Table 1

[0143]

[0144] Experiments were conducted on the KITTI validation set for the car category under different IoU thresholds and task conditions, and the results are shown in Table 2. This method achieves better performance than some methods that only use images. Specifically, compared with MonoDTR, under an IoU threshold of 0.5, this method shows advantages in simple, medium, and difficult settings. The model achieves significant increases of 1.22, 2.21, and 1.00 percentage points in 3D average precision (AP3D), respectively, and 1.92, 1.43, and 2.19 percentage points in bird's-eye view average precision (APBEV), respectively. These improvements not only highlight the superiority of the model in accuracy, but also further verify its robust performance and wide applicability in complex environments.

[0145] Table 2

[0146]

[0147] Table 3

[0148]

[0149] Table 3 shows the results for pedestrian and cyclist categories in the KITTI test set. The performance of the model of this method on pedestrian and cyclist categories is further illustrated. It can be clearly seen that these two categories are more difficult to detect than the car category, mainly because of their small size and non-rigid body, which makes it more difficult to accurately locate the position. Overall, the model significantly outperforms other advanced methods in the pedestrian category, achieving an improvement of about 15%. In terms of 3D detection of cyclists, it also achieved results comparable to CaDDN in simple difficulty, and surpassed other methods in medium and difficult difficulty. The results shown in Table 3 verify the excellent versatility and scalability of the model of this method. Therefore, this method can easily achieve accurate detection of objects with different appearances.

[0150] Run time analysis

[0151] The average running time of the model is measured to reach 40 frames per second on a single Nvidia 4090 GPU with batch size 1 to process the entire validation set, which shows the efficiency of our method. Compared with the current state-of-the-art methods, the MonoDMB model is 15 times faster than CaDDN. The main reasons are as follows: CaDDN generates a bird's-eye view representation by predicting the depth map for 3D detection, which requires a more complex architecture to achieve accurate depth prediction and is therefore slower. Fusion methods usually use two separate backbone networks to extract image and depth features, which is very time-consuming. In addition, the depth estimator requires additional inference time, which is not included in Table 1. In contrast, the model of our method uses a lightweight FMF module specifically for enhancing the detailed expression of features. At the same time, through the DMB module, it cleverly integrates depth perception features with visual features. This strategy not only greatly enhances the feature processing capability of the model, but also significantly reduces the running time.

[0152] Ablation Study

[0153] In order to verify the effectiveness of the components proposed in the model of this method. The ablation experiments shown in Table 4 were carried out, in which the Baseline model was used as the baseline model. (a) Baseline. (b) Based on the baseline model, the feature fusion module is replaced with FMF to observe the impact of this feature fusion method on the 3D object detection performance. (c) Based on the baseline model, the DMB module is used to perceive the image features, and its impact on the results is observed based on only the image perception features. (d) The visual context features are integrated using convolution operations, and then added to the depth perception feature elements, and the results are observed again. (e) MonoDMB, which combines all the modules mentioned above.

[0154] Table 4

[0155]

[0156] As shown in Table 4, the experimental results reveal the contribution and role of each module to the overall performance. From the experimental results, it can be seen that after introducing the FMF module into the Baseline, the average precision of 3D detection (AP3D) is significantly improved from 19.35 / 15.47 / 12.83 to 20.71 / 16.95 / 13.26, which highlights the positive role of the FMF module in enhancing feature details and promoting detection performance. When the DMB module is integrated into the Baseline, AP3D jumps to 24.32 / 18.89 / 14.12. Compared with the original configuration, this shows that DMB is significantly more efficient than existing methods in integrating depth information and visual information, which promotes a significant improvement in the accuracy of 3D object detection. Continuing to use the DMB module in combination with convolution in the Baseline, AP3D climbs to 25.03 / 19.62 / 14.14. This result not only emphasizes the effectiveness of Conv integrating contextual features, but also confirms the effective strategy of the DMB module in integrating visual and depth features. Finally, the MonoDMB model, which integrates all optimization modules, achieved a gain of 6.39 / 5.31 / 3.89 compared to the Baseline, verifying the excellent results of this method in improving detection accuracy and enhancing model stability and robustness. The above ablation experiments prove the rationality of the design of each module and the effectiveness of this method in monocular 3D object detection tasks.

[0157] In order to verify that the modules of this method can be easily integrated into the existing monocular image-only 3D object detection model to help improve performance, we selected the current advanced monocular 3D object detection models - M3D-RPN and MonoDLE - and based on their official codes, we connected the modules (FMF and DMB) proposed by this method into them. Figure 4 The fused M3D-RPN model is shown. In practice, the following steps are taken: the features of these models (before the detection head) are used as the initial features. Then, these features are processed using FMF and DMB to generate the final integrated features. Finally, the detection head of the original model is used to detect 3D objects using these integrated features. The core idea of ​​this process is that the modules proposed in this method (FMF and DMB) can be plug-and-play and directly combined with existing models without making large-scale changes to the original model structure. This flexibility is one of the significant advantages of the modules of this method. As shown in Table 5, with the help of FMF and DMB modules, these detectors achieve significant performance improvements on the KITTI validation set. This result not only proves the flexibility of this method, but also shows its applicability to a variety of 3D object detection models. In addition, these improvements show that this method can improve the performance of the original model without affecting its architecture.

[0158] Table 5

[0159]

[0160] Qualitative Results

[0161] Figure 5 Qualitative examples on the KITTI validation set are shown to intuitively verify the effect of this method. In the figure, the blue box represents the prediction of MonoDMB and the green box represents the label. As can be seen from the figure, the predicted bounding box and the actual label show a high degree of consistency and close fit, indicating that the accuracy of the prediction of the model of this method has been significantly improved. Finally, this model allows real-time reasoning. The framework of the model of this method achieves a state-of-the-art trade-off between performance and latency.

[0162] Conclusion

[0163] This method proposes a novel monocular 3D target detection method. By introducing a multi-scale enhanced fusion module (FMF), the method uses a specific feature focusing and feature diffusion mechanism to diffuse features rich in contextual information to each detection scale, effectively avoiding noise interference. The DMB module is innovatively integrated into monocular 3D target detection, which can efficiently integrate the contextual information of the image. Through a new adaptive strategy, the depth and visual information are combined to significantly enhance the performance of the subsequent depth prediction head. This strategy not only improves the accuracy of depth estimation, but also optimizes the performance of the model under different viewing angles and environmental conditions. Comprehensive experiments conducted on the KITTI dataset verify that the model of this method can achieve real-time detection and is significantly better than the existing advanced monocular detection methods.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A monocular 3D target detection method based on depth perception enhanced Mamba, characterized in that: The following steps are involved: S1. Build a MonoDMB model for monocular 3D target detection. The MonoDMB model includes a backbone network, an FMF module, a DAF module, a DMB module, and a detection head. The workflow of the MonoDMB model includes: First, the backbone network is used to extract features from the input image to obtain multiple initial features of different scales; Then, the initial features of different scales are fused through the FMF module to obtain enhanced fused features; the FMF module includes an initial fusion submodule and a detail fusion submodule; the initial fusion submodule is used to process the initial features extracted by the backbone network according to a preset feature fusion method and then perform concat splicing to obtain the initial fused features; the detail fusion submodule is used to process the initial fused features through a set of parallel deep convolutions to capture rich information across multiple scales, so that the detailed context information of the features of each scale can be better integrated; Then, the convolution layer is used to obtain the image visual features of the enhanced fusion features. At the same time, the DAF module is used to obtain the depth perception features of the enhanced fusion features through the Depth-Assisted Perception mechanism. The image visual features and depth perception features are summed element by element to obtain the comprehensive features. Afterwards, the DMB module is used to globally integrate the depth information and visual information according to the preset adaptive strategy of the comprehensive features to obtain the final integrated features; Finally, the detection head is used to perform 3D object detection on the final integrated features; S2, training the MonoDMB model; S3. Use the trained MonoDMB model to perform actual monocular 3D target detection.

2. The monocular 3D target detection method based on depth perception enhanced Mamba according to claim 1, characterized in that: The initial fusion submodule includes Aconv unit, 1×1 convolution layer and upsampling layer. When the initial fusion submodule works, each initial feature is processed in parallel through Aconv unit, 1×1 convolution layer and upsampling layer. Among them, the Aconv unit includes an average pooling-convolution AC subunit and a maximum pooling-convolution MC subunit, which are used to process the initial features in parallel to capture different feature information; the 1×1 convolution layer is used to reduce the dimension of the initial features so that they can be concat-joined with the output of the Aconv unit; the upsampling layer is used to adjust the resolution of feature maps of different scales so that they can have the same size when concat-joined.

3. The monocular 3D target detection method based on depth perception enhanced Mamba as claimed in claim 2, characterized in that: The working process of the detail fusion submodule includes: Firstly, the deep separable convolution DWConv technology is used to process multiple channels of the initial fusion feature F in parallel to obtain multiple channel processing features optimized by DWConv. Then, these channel processing features optimized by DWConv are stacked according to the 0th dimension to obtain stacked features; the stacking refers to merging multiple feature maps into a larger feature map in the channel dimension; Then, the initial fusion feature F is added and integrated with the stacked feature using the convolution and skip residual connection mechanism to obtain the detailed fusion feature F feature ; Finally, the size of the detailed fusion features is doubled by applying the transposed convolution TConv to obtain the final enhanced fusion feature map.

4. The monocular 3D target detection method based on depth perception enhanced Mamba as claimed in claim 1, characterized in that: After the comprehensive features are converted into a flattened two-dimensional sequence, they are processed using the DMB module.

5. The monocular 3D target detection method based on depth perception enhanced Mamba as claimed in claim 4, characterized in that: The process of converting the comprehensive features into a flattened two-dimensional sequence includes: Using the preset window size and step size, the comprehensive feature F d ∈R C×H×W Sliding window operation is performed on the image to divide it into multiple local area patches; each patch is flattened from three dimensions (k×k×C) to a one-dimensional vector, and then all one-dimensional vectors are combined into a vector with a dimension of (p, k 2 C), where p is the total number of patches; then, these vectors are converted into vectors of dimension E through linear projection to obtain the final tokens sequence of dimension (p, E).

6. The monocular 3D target detection method based on depth perception enhanced Mamba as claimed in claim 5, characterized in that: The working process of the DMB module includes: First, the input two-dimensional sequence tokens are normalized and then processed through two parallel branches; In the first branch, tokens are linearly projected into a vector D of dimension E, and then deformable convolution DCN is used for feature extraction, followed by SiLU activation function, DSSM layer and Norm processing to obtain the first feature; the DSSM layer is used to enhance the semantic dependency between tokens; In the second branch, tokens are also linearly projected into a vector O of dimension E, and then processed using the SiLU activation function to obtain the second feature; Afterwards, the features output by the two branches are aggregated by element-wise multiplication to obtain the fused features; Finally, the fused features are processed and projected back to the size of the original input image to obtain an output image with the same shape as the input image.

7. The monocular 3D target detection method based on depth perception enhanced Mamba as claimed in claim 6, characterized in that: The DSSM layer includes two branches, wherein the first branch includes a forward branch and a reverse branch; the feature processing process of the DSSM layer includes: In the forward branch of the first branch, the features are transformed by the linear layer and mapped to a new feature space to enhance their expressiveness. After that, the 1D convolution layer is used with the SiLU activation function to introduce nonlinear characteristics and enhance the learning ability of the features. It is combined with the SSM layer to promote feature extraction. The reverse branch of the first branch performs the same processing sequence as the forward branch in the opposite direction. In the second branch, first, the features are transformed by the linear layer; then they are processed using the SiLU activation function to maintain the nonlinear characteristics of feature propagation; Then, the output features of the two branches are fused through the Hadamard product; Finally, the fused features are reshaped back to their original dimensions, ready to be processed by the next layer of the network.

8. The monocular 3D target detection method based on depth perception enhanced Mamba as claimed in claim 1, characterized in that: The detection head adopts a single-stage detection method to determine the bounding box by using pre-set 2D-3D anchor points; Each anchor point is set including 2D bounding box parameters [x 2d ,y 2d ,w 2d ,h 2d ] and 3D bounding box parameters [x p ,y p ,z,w 3d ,h 3d ,l 3d ,θ]; where [x 2d ,y 2d ] represents the center point of the 2D box, [w 2d ,h 2d ] is the size of the 2D box; [x p ,y p ] represents the position where the center of the 3D object is projected onto the image, [w 3d ,h 3d ,l 3d ] is the size of the 3D box, parameter z refers to the distance from the center of the 3D object to the camera, and θ represents the observation angle; During training, all true labels are mapped to 2D space to calculate the intersection over union (IoU) with each 2D anchor point; when the IoU exceeds 0.5, the anchor point is selected for optimal assignment with the corresponding 3D box; For each anchor point, a set of 2D parameters [t x ,t y ,t w ,t h ] and a set of 3D parameters [t x ,t y ,t w ,t h ,t l ,t z ,t θ ], these parameters are used to quantify the deviation of 2D and 3D bounding boxes and output a classification score cls for each category; by combining the reference position of the anchor point and the prediction result of the network, the bounding box of the target is restored.

9. The monocular 3D target detection method based on depth perception enhanced Mamba as claimed in claim 1, characterized in that: In S2, when training the MonoDMB model, the loss function includes the classification loss L cls , regression loss L reg and the depth loss L dep .

10. The monocular 3D target detection method based on depth perception enhanced Mamba according to claim 9, characterized in that: Classification loss L cls and the regression loss L reg The expression is: In the formula, α is the weight coefficient for balancing positive and negative samples, which is used to adjust the importance of different categories; γ is the focusing parameter for adjusting the weight of easy-to-classify samples, which is used to reduce the weight of easy-to-classify samples and make the model pay more attention to those difficult-to-classify samples; δ is the threshold parameter; Depth loss L dep The expression is: Where P is the pixel area with valid depth labels on the image; D represents the model predicted depth value, represents the depth label, and FL represents the focus loss function.

Images