Monocular 3D target detection method and system based on edge information

Abstract

The invention discloses a monocular 3D target detection method and system based on edge information. The method comprises the following steps: generating an initial feature map for a given monocular image by using a feature backbone network; respectively extracting global visual features and edge information from the features by using a visual encoder and an edge predictor; the features output by the visual encoder are separated into low-level features and high-level features through a feature separation module; the edge module uses the edge information to enhance the low-level features, then fuses the enhanced low-level features with the high-level features, and transmits the fused features to the decoder; in the decoder, an edge information acquisition layer is used for acquiring edge information, an inter-query self-attention layer is used for interaction between the edge information, a visual cross attention layer is used for interaction between edge features and visual features, finally, an edge perception query is generated through a feedforward neural network, and finally, a detection head is used for detection. Through experiments, the method provided by the invention obtains good performance, and the target detection area is more aggregated without the assistance of additional data.

Description

Technical Field

[0001] The present invention relates to the field of image data processing, and in particular to a monocular 3D target detection method and system based on edge information. Background Art

[0002] In recent years, the field of computer vision has made remarkable progress, especially in object detection. The 3D object detection task aims to locate and identify objects in 3D space and has attracted widespread attention due to its important applications in autonomous driving, robotics, and augmented reality. Although 3D object detection methods based on LiDAR point clouds and multi-view images have achieved good performance, their cost is still high. In contrast, monocular 3D object detection predicts 3D attributes from a single image and does not rely on depth measurement or multi-view geometry, which can greatly save computational and equipment costs, and thus has attracted increasing research attention. Monocular 3D object detection has always been a challenging task in autonomous driving. Most existing methods follow the traditional 2D detector, which first locates the center of the object and then predicts 3D attributes from neighboring features. However, using only local visual features is not enough to understand the 3D spatial structure at the scene level. In the process of monocular 3D object detection, the acquisition of depth and depth features is also crucial. In contrast, newly proposed Transformer-based detectors such as MonoDETR do not require non-maximum suppression (NMS) or rule-based label assignment and have demonstrated superior performance in monocular 3D object detection. Given an input image with random query points, MonoDETR first operates on the visual and depth maps to search for relative keypoints for each query, and then integrates the features of these keypoints to predict 3D properties.

[0003] Transformer and its follow-up work have achieved great success in end-to-end 2D object detection. Inspired by this, recent research has begun to focus on using Transformer-based methods for 3D object detection. For example, MonoDTR introduces LiDAR point clouds as auxiliary supervision for its Transformer pipeline and uses the learned deep features as input queries to the decoder without any additional data. However, MonoDTR is not a DETR-based method. DETR3D and PETR (v2) are both 3D object detection methods that follow the DETR framework, but they use multiple images as input. MonoDETR is a monocular 3D object detection method based on the DETR framework. It designs a depth-guided decoder that uses a deep cross-attention layer to extract global depth cues and a deformable attention layer to aggregate local visual features for the query. Although deformable attention can find the relative number of key points of the query, there is still a serious problem of noisy points on difficult objects. Aiming at the two main problems of the prior art: the introduction of additional data and the insufficient aggregation of the target detection area, the present invention aims to make the target detection area more aggregated without introducing additional data, saving costs and improving performance. Summary of the invention

[0004] The purpose of the present invention is to address the problems in the prior art and disclose a monocular 3D target detection method and system based on edge information. In the case where the monocular 3D target detection is performed using edge information instead of using a depth map, no additional data is required for assistance, which reduces costs and makes the target detection area more concentrated.

[0005] A monocular 3D target detection method based on edge information, the steps are as follows:

[0006] S1, the backbone network generates low-level features with a sampling rate of 1 / 8 and high-level features with a sampling rate of 1 / 32 from the input monocular image;

[0007] Specifically, the monocular image I∈R H×W×3 As input, H and W represent the height and width of the image respectively; the backbone network is used to extract features, and the backbone network samples the multi-scale image in sequence, which are feature maps with a sampling rate of 1, a feature map with a sampling rate of 1 / 2, a feature map with a sampling rate of 1 / 4, a feature map with a sampling rate of 1 / 8, a feature map with a sampling rate of 1 / 16, and a feature map with a sampling rate of 1 / 32.

[0008] The multi-scale feature maps obtained from the last three stages of the backbone network are feature maps with a sampling rate of 1 / 8, feature maps with a sampling rate of 1 / 16, and feature maps with a sampling rate of 1 / 32. Among them, the feature map with a sampling rate of 1 / 8 is selected as the low-level feature, and the feature map with a sampling rate of 1 / 32 is selected as the high-level feature. The downsampling ratios relative to the input monocular image are 1 / 8 and 1 / 32 respectively;

[0009] S2, using the visual encoder and edge predictor to extract global visual features and edge information from high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 respectively;

[0010] The visual encoder is composed of three encoders, each of which contains a self-attention layer and a feedforward neural network, and the high-level features with a sampling rate of 1 / 32 pass through the three encoders in sequence;

[0011] Specifically, the working method of the edge predictor is as follows: using a convolutional neural network (CNN) as an edge predictor to extract edge information, and obtaining edge information from a feature map with a sampling rate of 1 / 8 output by the backbone network;

[0012] Specifically, the working method of the visual encoder is as follows: a feature map with a sampling rate of 1 / 32 is used as the input of the visual encoder, and the specific expression is as follows:

[0013]

[0014] In the formula, H and W represent the height and width of the image respectively, C is the number of channels, and f v e is the global visual feature;

[0015] S3, separating the global visual features output by the visual encoder into high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 through a feature separation module;

[0016] S4, the high-level features with a sampling rate of 1 / 32 separated by the feature separation module are input into the visual cross-attention layer in the encoder. At the same time, the edge module uses the edge information to enhance the low-level features and fuses them with the high-level features to generate queries that are valuable for target prediction and input into the edge information acquisition layer in the decoder. The specific steps are as follows:

[0017] S4.1, input the high-level features with a sampling rate of 1 / 32 separated by the feature separation module into the visual cross-attention layer in the encoder;

[0018] S4.2, input the low-level features with a sampling rate of 1 / 8 and the high-level features with a sampling rate of 1 / 32 separated by the feature separation module into the edge module;

[0019] S4.3, using edge information to fuse the low-level features of 1 / 8 sampling rate separated by the feature separation module, and enhancing the low-level features;

[0020] S4.4, the enhanced low-level features of 1 / 8 sampling rate are fused with the high-level features of 1 / 32 sampling rate to generate queries that are valuable for target prediction and passed to the edge information acquisition layer in the decoder;

[0021] The decoder consists of four layers, from bottom to top: edge information acquisition layer, inter-query self-attention layer, visual cross-attention layer and feedforward neural network; the edge information acquisition layer is used to acquire edge information, the inter-query self-attention layer is used for interaction between edge information, and the visual cross-attention layer is used for interaction between edge features and visual features. Finally, the feedforward neural network generates edge-aware queries, which are detected by the detection head.

[0022] S5, detecting and calculating the loss through the detection head;

[0023] Specifically, after the decoder, the edge-aware queries are fed into a multi-layer perceptron (MLP) for 3D attribute prediction, which includes: object category, 2D size, projected 3D center, 3D size and orientation.

[0024] For example, these predicted perspective attributes are converted into 3D space bounding boxes using camera parameters, which does not require post-processing of non-maximum suppression (NMS) or pre-defined anchors. During training, unordered queries are matched with ground-truth labels and the loss is calculated for the paired queries.

[0025] The specific loss calculation process of detecting and calculating the loss by the detection head is as follows:

[0026] S5.1. Calculate the training loss as follows:

[0027] The monocular 3D object detection method based on edge information (MonoDOE) is an end-to-end network, and all components are jointly trained according to a composite loss function. In order to correctly match each query with the ground-truth label, the Hungarian algorithm is used to find the global optimal match after calculating the loss of each query-label pair. For each pair of queries, the losses of the six attributes of object category, 2D size and projected 3D center, depth, 3D size and orientation angle are integrated into two groups. The first group contains object category, 2D size loss and projected 3D center loss, because these attributes mainly focus on the 2D visual appearance of the image; the second group contains depth loss, 3D size loss and orientation angle loss, and the third group is the 3D spatial attributes of the target; they are represented as and

[0028] Specifically, the 2D object loss uses focal loss, and the specific expression is as follows:

[0029]

[0030] In the formula, Represents object category, 2D size loss, projected 3D center loss, Represents the generalized intersection loss, which needs to be taken into account when calculating 2D objects, where λ1 is the balancing weight of the object category, λ2 is the balancing weight of the 2D size loss, λ3 is the balancing weight of the projected 3D center loss, and λ4 is the balancing weight of the generalized intersection loss;

[0031] For 3D object loss Following MonoDLE predicts 3D size (h 3d , W 3d , L 3d ) and orientation angle α, for depth prediction, the average of the two depth values ​​is used to represent the predicted depth. The specific expression is as follows:

[0032]

[0033] Where, d reg is the depth value directly regressed by a detector head, d geo The specific expression is the relationship between 2D and 3D dimensions as follows:

[0034]

[0035] Where f is the focal length of the camera, t is the distance from the target point to the imaging plane, and b is the brightness;

[0036] For the entire predicted depth, the Laplace arbitrary uncertainty loss is used to form the final depth loss. The specific expression is as follows:

[0037]

[0038] In the formula, σ is related to d reg The standard deviation of the predictions, d gt is the ground truth depth value.

[0039] The final can be expressed as:

[0040]

[0041] In the formula, and Represents 3D size loss, direction angle loss and depth loss respectively, where λ s is the balance weight of 3D size loss, λ6 is the balance weight of direction angle loss, and λ7 is the balance weight of depth loss;

[0042] Since the network predicts 3D attributes less accurately than 2D attributes, especially at the beginning of training, The value of is unstable and will interfere with the matching process, so use as a match for each query-label pair;

[0043] S5.2. Calculate the overall loss and sum the losses of the two groups;

[0044] After the matching process, valid pairs are obtained from N queries, and the overall loss formula for training images is:

[0045]

[0046] Where N gt represents the number of ground truth objects,

[0047] A monocular 3D object detection system based on edge information includes: a feature extraction module, a feature separation module, an edge module, and a decoder module.

[0048] The feature extraction module performs the following operations:

[0049] S1, the backbone network generates low-level features with a sampling rate of 1 / 8 and high-level features with a sampling rate of 1 / 32 from the input monocular image;

[0050] S2, using the visual encoder and edge predictor to extract global visual features and edge information from high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 respectively;

[0051] The feature separation module performs the following operations:

[0052] S3, separating the global visual features output by the visual encoder into high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 through a feature separation module;

[0053] The edge module performs the following operations:

[0054] S4, the high-level features with a sampling rate of 1 / 32 separated by the feature separation module are input into the visual cross attention layer in the encoder. At the same time, the edge module uses the edge information to enhance the low-level features, and then fuses them with the high-level features and inputs them into the edge information acquisition layer in the decoder;

[0055] The decoder module performs the following operations:

[0056] S5. Perform detection through the detection head and calculate the loss.

[0057] Specifically, the feature extraction module includes: a backbone network (Backbone), an edge predictor (EdgePredictor);

[0058] The backbone network is used to transform a monocular image I∈R H×W×3 As input, the framework uses the ResNet-50 backbone network to extract features;

[0059] The edge predictor uses a convolutional neural network (CNN) as an edge predictor to extract edge information instead of directly using the Canny edge detector method; because it is necessary to address the complexity of the target domain data to ensure that robust and accurate edge information can be extracted under complex conditions without introducing too much additional calculations.

[0060] The feature separation module (FSM) is used to separate the global visual features output by the visual encoder into low-level features and high-level features.

[0061] The edge module aims to enhance low-level features through edge information obtained, and fuse low-level features with high-level features to enrich decoder queries; the edge module enhances low-level features, uses 1x1 convolution as a channel to embed edge information into low-level features, and finally generates enhanced low-level features, denoted as L′;

[0062] The decoder module includes, from bottom to top: an edge information acquisition layer, an inter-query self-attention layer, a visual cross-attention layer, and a feedforward neural network, wherein the visual cross-attention layer includes a cross-attention module;

[0063] The high-level features in the decoder and the newly generated low-level features are processed in the cross-attention module, and the formula is as follows:

[0064] Cross-Attn(Q q , K k , V v )=H

[0065] K k , V v =Maxpool(1×1Conv(L′))

[0066] Where H represents the high-level features from the encoder, Q q , K k , V vThey are query, key, and value respectively, and L′ is the enhanced low-level feature; the purpose of fusion is to ensure the comprehensiveness and richness of the input and promote the performance improvement in the subsequent decoding process;

[0067] Q q , K k , V v The specific expression is as follows:

[0068] Q q =Linear(q)

[0069] K,V=Linear(K k，v )

[0070] In the formula, Q q ∈R N×C ,K,V∈R H×W×C , q is the edge-aware query before enhancement; K k，v is the key corresponding to q, and then the query result is combined with the result generated by the edge module (Flatten) to obtain the edge-aware query q′, which is formulated as:

[0071] q′=S(Q q , Flatten(Q))

[0072] Where S is the combination function.

[0073] Such a mechanism enables each object query to adaptively capture the region guided by the edge on the image, thereby aggregating the region of interest and better locating the object in the image; the edge-aware query is then fed to the edge information acquisition layer. The inter-query self-attention layer is used for feature interaction between objects, and the visual cross-attention layer is used to collect visual semantics from high-level features with a sampling rate of 1 / 32.

[0074] Beneficial Effects of the Invention

[0075] Compared with the existing monocular 3D target detection technology, the present invention does not use additional data to assist in the detection process, thus greatly saving costs. At the same time, the present invention makes the target detection area more aggregated by fusing the enhanced low-level features with the high-level features to generate queries, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is the overall flow chart of the monocular 3D object detection method (MonoDOE) based on edge information;

[0077] Figure 2 It is a schematic diagram of the working of the edge module;

[0078] Figure 3It is a comparison chart between the monocular 3D object detection method based on edge information (MonoDOE) and MonoDETR. DETAILED DESCRIPTION

[0079] The present invention is further described in detail below in conjunction with specific embodiments.

[0080] Example 1

[0081] like Figure 1 As shown, a monocular 3D target detection method based on edge information includes the following steps:

[0082] S1, the backbone network generates low-level features with a sampling rate of 1 / 8 and high-level features with a sampling rate of 1 / 32 from the input monocular image;

[0083] Specifically, the monocular image I∈R H×W×3 As input, H and W represent the height and width of the image respectively; the backbone network is used to extract features, and the backbone network samples the multi-scale image in sequence, which are feature maps with a sampling rate of 1, a feature map with a sampling rate of 1 / 2, a feature map with a sampling rate of 1 / 4, a feature map with a sampling rate of 1 / 8, a feature map with a sampling rate of 1 / 16, and a feature map with a sampling rate of 1 / 32.

[0084] The multi-scale feature maps obtained from the last three stages of the backbone network are feature maps with a sampling rate of 1 / 8, feature maps with a sampling rate of 1 / 16, and feature maps with a sampling rate of 1 / 32. Among them, the feature map with a sampling rate of 1 / 8 is selected as the low-level feature, and the feature map with a sampling rate of 1 / 32 is selected as the high-level feature. The downsampling ratios relative to the input monocular image are 1 / 8 and 1 / 32 respectively;

[0085] S2, using the visual encoder and edge predictor to extract global visual features and edge information from high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 respectively;

[0086] The visual encoder is composed of three encoders, each of which contains a self-attention layer and a feedforward neural network, and the high-level features with a sampling rate of 1 / 32 pass through the three encoders in sequence;

[0087] Specifically, the working method of the edge predictor is as follows: using a convolutional neural network (CNN) as an edge predictor to extract edge information, and obtaining edge information from a feature map with a sampling rate of 1 / 8 output by the backbone network;

[0088] Specifically, the working method of the visual encoder is as follows: a feature map with a sampling rate of 1 / 32 is used as the input of the visual encoder, and the specific expression is as follows:

[0089]

[0090] In the formula, H and W represent the height and width of the image respectively, C is the number of channels, and f v e is the global visual feature;

[0091] S3, separating the global visual features output by the visual encoder into high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 through a feature separation module;

[0092] S4, the high-level features with a sampling rate of 1 / 32 separated by the feature separation module are input into the visual cross-attention layer in the encoder. At the same time, the edge module uses the edge information to enhance the low-level features and fuses them with the high-level features to generate queries that are valuable for target prediction and input into the edge information acquisition layer in the decoder. The specific steps are as follows:

[0093] S4.1, input the high-level features with a sampling rate of 1 / 32 separated by the feature separation module into the visual cross-attention layer in the encoder;

[0094] S4.2, input the low-level features with a sampling rate of 1 / 8 and the high-level features with a sampling rate of 1 / 32 separated by the feature separation module into the edge module;

[0095] S4.3, using edge information to fuse the low-level features of 1 / 8 sampling rate separated by the feature separation module, and enhancing the low-level features;

[0096] S4.4, the enhanced low-level features of 1 / 8 sampling rate are fused with the high-level features of 1 / 32 sampling rate to generate queries that are valuable for target prediction and passed to the edge information acquisition layer in the decoder;

[0097] The decoder consists of four layers, from bottom to top: edge information acquisition layer, inter-query self-attention layer, visual cross-attention layer and feedforward neural network; the edge information acquisition layer is used to acquire edge information, the inter-query self-attention layer is used for interaction between edge information, and the visual cross-attention layer is used for interaction between edge features and visual features. Finally, the feedforward neural network generates edge-aware queries, which are detected by the detection head.

[0098] S5, detecting and calculating the loss through the detection head;

[0099] Specifically, after the decoder, the edge-aware queries are fed into a multi-layer perceptron (MLP) for 3D attribute prediction, which includes: object category, 2D size, projected 3D center, 3D size and orientation.

[0100] For example, these predicted perspective attributes are converted into 3D space bounding boxes using camera parameters, which does not require post-processing of non-maximum suppression (NMS) or pre-defined anchors. During training, unordered queries are matched with ground-truth labels and the loss is calculated for the paired queries.

[0101] The specific loss calculation process of detecting and calculating the loss by the detection head is as follows:

[0102] S5.1. Calculate the training loss as follows:

[0103] The edge-based monocular 3D object detection method (MonoDOE) is an end-to-end network, and all components are jointly trained according to a composite loss function. In order to correctly match each query with the ground-truth label, the Hungarian algorithm is used to find the global optimal match after calculating the loss for each query-label pair. For each pair of queries, the losses of the six attributes of object category, 2D size and projected 3D center, depth, 3D size and orientation are integrated into two groups. The first group contains object category, 2D size and projected 3D center, because these attributes mainly focus on the 2D visual appearance of the image; the second group contains depth, 3D size and orientation, and the second group is the 3D spatial attributes of the target; denoted as and Then sum the losses of the two groups;

[0104] Specifically, the 2D object loss uses focal loss, and the specific expression is as follows:

[0105]

[0106] In the formula, Represents object category, 2D size loss, projected 3D center loss, Represents the generalized intersection loss, which needs to be taken into account when calculating 2D objects, where λ1 is the balancing weight of the object category, λ2 is the balancing weight of the 2D size loss, λ3 is the balancing weight of the projected 3D center loss, and λ4 is the balancing weight of the generalized intersection loss;

[0107] For 3D object loss Following MonoDLE predicts 3D size (h 3d , W 3d , L 3d ) and orientation angle α, for depth prediction, the average of the two depth values ​​is used to represent the predicted depth. The specific expression is as follows:

[0108]

[0109] Where, d reg is the depth value directly regressed by a detector head, d geo The specific expression is the relationship between 2D and 3D dimensions as follows:

[0110]

[0111] Where f is the focal length of the camera, t is the distance from the target point to the imaging plane, and b is the brightness;

[0112] For the entire predicted depth, the Laplace arbitrary uncertainty loss is used to form the final depth loss. The specific expression is as follows:

[0113]

[0114] In the formula, σ is related to d reg The standard deviation of the predictions, d gt is the ground truth depth value.

[0115] The final can be expressed as:

[0116]

[0117] In the formula, and They represent 3D size loss, angular loss, and depth loss respectively, where λ5 is the balancing weight of 3D size loss, λ6 is the balancing weight of angular loss, and λ7 is the balancing weight of depth loss;

[0118] Since the network predicts 3D attributes less accurately than 2D attributes, especially at the beginning of training, The value of is unstable and will interfere with the matching process, so use as a match for each query-label pair;

[0119] S5.2. Calculate the overall loss;

[0120] After the matching process, valid pairs are obtained from N queries, and the overall loss formula for training images is:

[0121]

[0122] Where Ngt represents the number of ground truth objects,

[0123] A monocular 3D object detection system based on edge information includes: a feature extraction module, a feature separation module, an edge module, and a decoder module.

[0124] The feature extraction module performs the following operations:

[0125] S1, the backbone network generates low-level features with a sampling rate of 1 / 8 and high-level features with a sampling rate of 1 / 32 from the input monocular image;

[0126] S2, using the visual encoder and edge predictor to extract global visual features and edge information from high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 respectively;

[0127] The feature separation module performs the following operations:

[0128] S3, separating the global visual features output by the visual encoder into high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 through a feature separation module;

[0129] The edge module performs the following operations:

[0130] S4, the high-level features with a sampling rate of 1 / 32 separated by the feature separation module are input into the visual cross attention layer in the encoder. At the same time, the edge module uses the edge information to enhance the low-level features, and then fuses them with the high-level features and inputs them into the edge information acquisition layer in the decoder;

[0131] The decoder module performs the following operations:

[0132] S5. Perform detection through the detection head and calculate the loss.

[0133] Specifically, the feature extraction module includes: a backbone network (Backbone), an edge predictor (EdgePredictor);

[0134] The backbone network is used to transform a monocular image I∈R H×W×3 As input, the framework uses the ResNet-50 backbone network to extract features;

[0135] The edge predictor uses a convolutional neural network (tCNN) as an edge predictor to extract edge information instead of directly using the Canny edge detector method; because it is necessary to address the complexity of the target domain data to ensure that robust and accurate edge information can be extracted under complex conditions without introducing too much additional calculations.

[0136] The feature separation module (FSM) is used to separate the global visual features output by the visual encoder into low-level features and high-level features.

[0137] like Figure 2As shown, the edge module aims to enhance low-level features through the obtained edge information and fuse low-level features with high-level features to enrich decoder queries; the edge module enhances low-level features, uses 1x1 convolution as a channel to embed edge information into low-level features, and finally generates enhanced low-level features, denoted as L′;

[0138] The decoder module includes, from bottom to top: an edge information acquisition layer, an inter-query self-attention layer, a visual cross-attention layer, and a feedforward neural network, wherein the visual cross-attention layer includes a cross-attention module;

[0139] The high-level features in the decoder and the newly generated low-level features are processed in the cross-attention module, and the formula is as follows:

[0140] Cross-Attn(Q q , K k , V v )=H

[0141] K k , V v =Maxpool(1×1Conv(L′))

[0142] Where H represents the high-level features from the encoder, Q q , K k , V v They are query, key, and value respectively, and L′ is the enhanced low-level feature; the purpose of fusion is to ensure the comprehensiveness and richness of the input and promote the performance improvement in the subsequent decoding process;

[0143] Q q , K k , V v The specific expression is as follows:

[0144] Q q =Linear(q)

[0145] K,V=Linear(K k，v )

[0146] In the formula, Q q ∈R N×C ,K,V∈R H×W×C , q is the edge-aware query before enhancement; K k，v is the key corresponding to q, and then the query result is combined with the result generated by the edge module (Flatten) to obtain the edge-aware query q′, which is formulated as:

[0147] q′=S(Q q , Flatten(Q))

[0148] Where S is the combination function.

[0149] Such a mechanism enables each object query to adaptively capture the region guided by the edge on the image, thereby aggregating the region of interest and better locating the object in the image; the edge-aware query is then fed to the edge information acquisition layer. The inter-query self-attention layer is used for feature interaction between objects, and the visual cross-attention layer is used to collect visual semantics from high-level features with a sampling rate of 1 / 32.

[0150] like Figure 3 As shown in the figure, compared with the prior art MonoDETR, the monocular 3D target detection method and system (MonoDOE) based on edge information introduces an edge predictor to predict the information of the target edge to replace the depth predictor, and uses the edge information as the input of the edge module to generate a token, thereby enhancing the query in the decoder. At the same time, a feature separation module (FSM) is used to separate the low-level features and high-level features in the output of the visual encoder to better distinguish the foreground target and the background, and use the target detection area to be more aggregated. Therefore, the monocular 3D target detection method and system (MonoDOE) based on edge information has improved performance compared to the baseline.

[0151] On the KITTI3D dataset, the proposed MonoDOE is evaluated on target detection, and ResNet-50 is used as the feature backbone to extract features. The specific results are shown in Table 1:

[0152] Table 1 Comparison results

[0153]

[0154]

[0155] Table 1 shows the comparison results of the “car” category on the KITTI test set and validation set. MonoDOE shows excellent performance on the KITTI test set and validation set. From the results, the following conclusions are drawn:

[0156] 1. Among all the methods, the method of the present invention achieves the best accuracy on medium-difficulty and difficult targets, and improves AP by 1.0% on medium-difficulty targets in the test set compared with MonoDETR, the second best method in the figure. 3D The minimum improvement was 0.58%, and AP 3DThe maximum improvement is 1.83%. This is because difficult targets are usually small in size, so the query points estimated by previous algorithms tend to deviate from the object, resulting in detection errors. In contrast, the edge module can well estimate the scale edge information of the query point, bringing significant performance improvements. In addition, the proposed method does not require the use of additional data for assistance. Even compared with the previously proposed method that uses additional data, the performance is better than some methods, which greatly saves costs.

[0157] 2. The method of the present invention also achieves good performance on the bird's eye view. This is because the method mainly focuses on evaluating the relative position of the car with respect to the street in the bird's eye view, which is highly relevant to estimating the position of the query point.

Claims

1. A monocular 3D target detection method based on edge information, characterized in that: The following steps are involved: S1, the backbone network generates low-level features with a sampling rate of 1 / 8 and high-level features with a sampling rate of 1 / 32 from the input monocular image; S2, using the visual encoder and edge predictor to extract global visual features and edge information from high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 respectively; S3, separating the global visual features output by the visual encoder into high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 through a feature separation module; S4, the high-level features with a sampling rate of 1 / 32 separated by the feature separation module are input into the visual cross attention layer in the encoder. At the same time, the edge module uses the edge information to enhance the low-level features, and then fuses them with the high-level features and inputs them into the edge information acquisition layer in the decoder; S5. Perform detection through the detection head and calculate the loss.

2. The monocular 3D target detection method based on edge information according to claim 1, characterized in that: The visual encoder consists of three encoders, each of which contains a self-attention layer and a feedforward neural network, and high-level features with a sampling rate of 1 / 32 pass through the three encoders in sequence.

3. The monocular 3D target detection method based on edge information according to claim 1, characterized in that: The specific steps of inputting the high-level features with a sampling rate of 1 / 32 separated by the feature separation module into the visual cross attention layer in the encoder, and at the same time, the edge module uses the edge information to enhance the low-level features, fuse them with the high-level features, and input them into the edge information acquisition layer in the decoder are as follows: S4.1, input the high-level features with a sampling rate of 1 / 32 separated by the feature separation module into the visual cross-attention layer in the encoder; S4.2, input the low-level features with a sampling rate of 1 / 8 and the high-level features with a sampling rate of 1 / 32 separated by the feature separation module into the edge module; S4.3, using edge information to fuse the low-level features of 1 / 8 sampling rate separated by the feature separation module, and enhancing the low-level features; S4.4, fuse the enhanced low-level features of 1 / 8 sampling rate with the high-level features of 1 / 32 sampling rate.

4. The monocular 3D target detection method based on edge information according to claim 3, characterized in that: The decoder consists of four layers, from bottom to top: edge information acquisition layer, inter-query self-attention layer, visual cross-attention layer and feedforward neural network.

5. The monocular 3D target detection method based on edge information according to claim 1, characterized in that: The specific steps of detecting and calculating the loss by the detection head are as follows: S5.

1. Calculate the training loss, including: 2D object loss, 3D target loss, the specific expressions are as follows: The specific expression of 2D object loss is as follows: In the formula, and They represent object category, 2D size loss, projected 3D center loss and generalized intersection loss respectively, where λ1 is the balancing weight of object category, λ2 is the balancing weight of 2D size loss, λ3 is the balancing weight of projected 3D center loss, and λ4 is the balancing weight of generalized intersection loss; The specific expression of 3D target loss is as follows: In the formula, and They represent 3D size loss, angular loss, and depth loss respectively, where λ5 is the balancing weight of 3D size loss, λ6 is the balancing weight of angular loss, and λ7 is the balancing weight of depth loss; S5.

2. Calculate the overall loss. The specific expression is as follows: Where N gt Represents the number of ground truth objects.

6. A monocular 3D object detection system based on edge information, characterized in that: The system comprises: a feature extraction module, an edge module, a feature separation module, and a decoder module; wherein the feature extraction module comprises: a backbone network and an edge predictor; the decoder module comprises four layers, which are, from bottom to top: an edge information acquisition layer, an inter-query self-attention layer, a visual cross-attention layer, and a feedforward neural network; wherein the visual cross-attention layer comprises a cross-attention module; The feature extraction module performs the following operations: S1, the backbone network generates low-level features with a sampling rate of 1 / 8 and high-level features with a sampling rate of 1 / 32 from the input monocular image; S2, using the visual encoder and edge predictor to extract global visual features and edge information from high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 respectively; The feature separation module performs the following operations: S3, separating the global visual features output by the visual encoder into high-level features with a sampling rate of 1 / 32 and low-level features with a sampling rate of 1 / 8 through a feature separation module; The edge module performs the following operations: S4, the high-level features with a sampling rate of 1 / 32 separated by the feature separation module are input into the visual cross attention layer in the encoder. At the same time, the edge module uses the edge information to enhance the low-level features, and then fuses them with the high-level features and inputs them into the edge information acquisition layer in the decoder; The decoder module performs the following operations: S5. Perform detection through the detection head and calculate the loss.

7. The monocular 3D target detection system based on edge information according to claim 6, characterized in that: The specific expression of the low-level features enhanced by the high-level features of 1 / 32 sampling rate in the decoder in the cross attention module is as follows: Cross-Attn(Q q ,K k ,V v )=H K k ,V v =Maxpool(1×1Conv(L′)) Where H represents the high-level features from the encoder, Q q ,K k ,V v are query, key, and value respectively, and L' is the enhanced low-level feature.

Images