A Transformer-based Monocular 3D Object Detection Method

Abstract

This invention discloses a Transformer-based monocular 3D object detection method, belonging to the technical field of computation, estimation, or counting. The method includes the following steps: S1: Constructing a basic network structure, which includes a feature extraction module, an encoding / decoding module, and a loss function module; S2: Verifying and adjusting the basic network structure using a monocular 3D object detection dataset, and constructing the final monocular 3D object detection model using the adjusted structure; S3: Retraining the monocular 3D object detection model obtained in S2; S4: Performing detection using the model trained in S3. This invention can also optionally utilize depth-based data augmentation to process the dataset for model optimization. This invention effectively improves the detection accuracy in monocular 3D object detection algorithms, while using depth transformation on the input feature data for depth encoding / decoding, reducing runtime latency and memory usage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D target detection, and particularly discloses a monocular 3D target detection method based on a Transformer. BACKGROUND

[0002] 3D target detection technology can provide important information such as length, width, height, depth and rotation angle of an object in a three-dimensional scene, and plays a crucial role in automatic driving tasks such as path planning and collision avoidance. At present, mainstream 3D target detection technologies include four types of methods based on monocular RGB images, binocular RGB, laser point clouds and image point cloud fusion. The method based on laser point cloud and image point cloud fusion needs to use a point cloud map, which is collected by a high-precision laser radar. These two methods have high recognition accuracy but are expensive. The environment perception scheme based on a monocular sensor has a very high landing advantage, only a monocular camera is needed, the cost is the lowest, and the running speed is the fastest, but the accuracy needs to be improved.

[0003] The Transformer arises from the field of natural language processing (NLP), and its proposal solves the problem that a recurrent network model cannot be trained in parallel and needs a large amount of storage resources to remember the entire sequence. The Transformer uses a non-cyclic network structure, performs parallel computation through an encoder-decoder and a self-attention mechanism, greatly shortens the training time, and achieves the best performance at that time. The successful application of the Transformer in the field of NLP provides a new possibility for visual feature learning. Visual models based on the Transformer have achieved effects comparable to or even better than convolutional neural networks in the fields of image classification, target detection, image segmentation, video understanding, image generation and point cloud analysis. The target detection algorithm based on the Transformer has reached or even exceeded the SOTA (state-of-the-art) model algorithm based on the convolutional neural network, which provides the possibility of improvement for a key factor required in the 3D target detection algorithm, i.e., high accuracy.

[0004] In summary, it is necessary to propose a monocular 3D target detection algorithm based on the Transformer, and the network structure of the algorithm should meet the real-time requirements in practical applications. SUMMARY

[0005] The application aims to solve the technical problem of improving the accuracy of 3D target detection algorithm based on monocular sensor, and achieve the application purpose of meeting the accuracy and real-time requirements of monocular 3D target detection algorithm while reducing the cost.

[0006] The application adopts the following technical solutions to achieve the above application purpose.

[0007] A monocular 3D target detection method based on Transformer comprises the following steps:

[0008] Step one, constructing a network basic structure, which comprises a feature extraction module, a coding and decoding module, and a loss function module;

[0009] Step two, verifying and adjusting the network basic structure through a monocular 3D target detection dataset, and constructing a final monocular 3D target detection model using the adjusted network basic structure;

[0010] Step three, retraining the final monocular 3D target detection model obtained in step two;

[0011] Step four, detecting using the final monocular 3D target detection model obtained in step three.

[0012] Further, the monocular 3D target detection method based on Transformer further comprises processing the monocular 3D target detection dataset using a depth-based data enhancement method, and step three uses the monocular 3D target detection dataset after data enhancement to train and optimize the final monocular 3D target detection model.

[0013] Further, the feature extraction module in step one comprises n stage extraction modules connected in sequence, each stage extraction module comprises a plurality of basic feature extraction units connected in sequence, and the output channel numbers of the basic feature extraction units in the same stage extraction module are the same.

[0014] Further, the basic feature extraction unit determines the basic structure corresponding to the candidate operation through the verification of step two, and the candidate operation of the basic feature extraction unit includes but is not limited to 3*3 convolution, 5*5 convolution, 7*7 convolution, 3*3 deformable convolution, 5*5 deformable convolution, 7*7 deformable convolution, Xception operation, and residual operation.

[0015] Further, the coding and decoding module in step one comprises m stage coding and decoding units, sequentially marked as E1-En, the input data of the mth stage coding and decoding module is the target instance feature map output by the n-m+1th stage basic feature extraction module to the nth stage basic feature extraction module, and the final target detection result is obtained through the attention mechanism and the front feedback network.

[0016] Further, the coding and decoding unit comprises a depth coding and decoding unit and a visual coding and decoding unit. The input of the depth coding and decoding unit is generated by using the following linearly increasing distribution function on the corresponding target instance feature map. Wherein d c represents the calculated increasing depth value, d min represents the minimum depth value in the specified range, d max represents the maximum depth value in the specified range, d i represents the input i-th target instance feature map depth value, and D represents the set depth feature map layer number.

[0017]

[0018] Further, the following loss function is used in network design. represents the loss function when 2D prediction, IoU represents the overlap degree, respectively represent the proportion of the Euclidean distance of the predicted true value box and the real true value box in the center point, length and width. represents the focal loss when depth prediction, and the focal loss is calculated by summing and averaging the predicted depth D(u, v) and the true depth of each point on the feature map composed of W F ,H F . respectively represent the cross-entropy loss of classification and rotation angle. represents the overall regularization loss function, λ cls , λ 2d , λ depth , λ θ is the regularization factor.

[0019]

[0020]

[0021]

[0022] Further, step three re-trains the final monocular 3D target detection model obtained in step two, specifically including the following steps:

[0023] Step 3.1: Pre-training the final monocular 3D target detection model constructed on the monocular 3D target detection data set;

[0024] Step 3.2: Obtain 2D detection accuracy result mAP according to the coincidence degree of the pre-trained detection result and the data provided by the verification set 2d and 3D detection accuracy result mAP 3d ;

[0025] Step 3.3: Optimize the network structure by using 2D detection accuracy result mAP 2d and 3D detection accuracy result mAP 3d and loss function value Loss. That is, introduce normalization parameters α1, α2, α3, and perform weighted summation. Finally, take the model with the highest weighted result as the final network model.

[0026] Further, the weighted normalization calculation function is as follows:

[0027] F = α1mAP 2d + α2mAP 3d + α3Loss

[0028] Further, the steps of processing monocular 3D target detection data set by using depth-based data enhancement method include the following steps:

[0029] Step I: Extract target instances in the training set by a pre-trained semantic segmentation model;

[0030] Step II: Calculate the related parameters of the instances scaled down / scaled up to the corresponding size by the following proportional conversion formula, (X, Y, Z) and represent the 3D coordinates before and after scaling, respectively, wherein is artificially set, v is the height of the extracted instance, h horizon is the horizontal line height of the input image, f is the camera focal length of the monocular 3D target detection data set, p 2d is [u, v, 1] T represents the mapping matrix of the 2D coordinates obtained by projecting the 3D center position coordinates of the target instance, (c u , c v ) represents the 2D coordinates of the instance placement position, and the center position is artificially set;

[0031] Step III: Put the scaled down / scaled up instances back into the original training set image.

[0032]

[0033]

[0034]

[0035] The application has the following beneficial effects by adopting the technical scheme:

[0036] (1) The monocular 3D target detection method provided by the application encodes the feature map through the Transformer structure to predict the detection result, replaces the traditional detection method of predicting by using geometric formula or pre-training model, effectively improves the detection accuracy in the monocular 3D target detection algorithm, and uses depth conversion on the input feature data of depth encoding and decoding, reduces the running delay, and reduces the occupation of memory.

[0037] (2) The monocular 3D target detection method provided by the application can select to use the data enhancement mode based on depth information to further improve the detection accuracy, and can be better applied to the actual monocular 3D target detection task. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the monocular 3D target detection algorithm based on Transformer provided in embodiment 1.

[0039] Figure 2 The schematic diagram of the network basic structure provided in embodiment 1.

[0040] Figure 3 The structural schematic diagram of the feature extraction module provided in embodiment 1.

[0041] Figure 4 The structural schematic diagram of the encoding and decoding module provided in embodiment 1.

[0042] Figure 5 The flowchart of the data enhancement mode provided in embodiment 1. DETAILED DESCRIPTION

[0043] In order to make the purposes, technical schemes and advantages of the application clearer, the technical scheme of the application will be described clearly and completely in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0044] Embodiment 1

[0045] Reference Figures 1-5The embodiment provides a monocular 3D target detection algorithm based on a Transformer, which can be used in the field of automatic driving. The algorithm uses image data collected by a single camera to perform 3D target detection on vehicles and pedestrians by searching for an optimal model structure in a training set, and gives the size of a target vehicle or pedestrian, the distance from the camera, and the rotation angle and other information. The implementation flowchart of the algorithm is shown in Figure 1 The specific process is as follows and includes S1-S4.

[0046] S1, constructing a network basic structure

[0047] The network basic structure shown in Figure 2 includes a feature extraction module, a coding and decoding module, and a loss function module.

[0048] The feature extraction module is used to extract the features of vehicles and pedestrians in a picture. As shown in Figure 2 The feature extraction module is composed of n stage basic feature extraction modules connected in sequence, and the n stage basic feature extraction modules are sequentially denoted as C1-Cn. The value of the parameter n can be manually adjusted according to actual needs. When more attention is paid to the speed of the model and high precision is not required, the value of n can be appropriately reduced, and vice versa. Specifically, the value of n is 5. Each stage basic feature extraction module includes a basic feature extraction unit connected in sequence, and the value of a can be manually adjusted according to actual conditions. Specifically, the number of basic feature extraction units in the five stage basic feature extraction modules C1-C5 is 1, 9, 12, 18 and 9, respectively. The output channel number of the basic feature extraction units in the same stage basic feature extraction module needs to be a multiple of 64, but the specific value can be adjusted according to actual conditions. Specifically, the output channel number of the basic feature extraction units in the five stage basic feature extraction modules C1-C5 is 64, 128, 512, 1024 and 2048, respectively.

[0049] The number of basic feature extraction units is determined according to the weighted sum value in S3, as shown in Figure 3As shown, the candidate operations for the basic feature extraction unit include: 3*3 convolution, 5*5 convolution, 7*7 convolution, 3*3 deformable convolution, 5*5 deformable convolution, 7*7 deformable convolution, 3*3 depthwise separable convolution, and residual blocks, where the residual blocks optionally contain 1*1 convolutional layers. Deformable convolution differs from ordinary convolution in that its grid shape is deformable; each grid point can be moved by a learnable offset, and the weights and convolution operations are applied to these moved grid points. Deformable convolution addresses the limitation of traditional convolution, which, due to its fixed shape, cannot fully adapt to the shape of the target. In autonomous driving, there are often situations where the shape of the target changes significantly in the image due to changes in the target's angle. Deformable convolution can improve the performance of monocular 3D target detection with a slight increase in computational load. 3*3 depthwise separable convolution can be divided into two parts: 3*3 channel-wise convolution and pointwise convolution. The input feature image first undergoes a 3*3 channel-wise convolution operation and then a pointwise convolution operation to obtain the output. In channel-wise convolution, each channel is convolved by only one kernel, and the number of output channels of the feature map after channel-wise convolution is exactly the same as the number of input channels. Pointwise convolution is a convolution operation with a 1*1 kernel size. 3*3 depthwise separable convolution can significantly reduce computation. Residual blocks can optimize the network and accelerate network training.

[0050] The encoding / decoding module is used to encode and decode the extracted features' depth and visual information through the attention mechanism in the transformer, predicting the detection results and replacing traditional geometric formula prediction or pre-trained model prediction. For example... Figure 2 As shown, the encoding / decoding module includes m stages, denoted as E1-Em. The value of parameter m can be set according to parameter n of the feature extraction module, where m <= n. Specifically, the value of m is set to 3. Each stage of the encoding / decoding module contains b depth encoding / decoding units and c visual encoding / decoding units. The depth encoding / decoding units and visual encoding / decoding units can be computed in parallel. The b depth encoding / decoding units output 3D coordinate information prediction values, and the c visual encoding / decoding units output 2D coordinate information prediction values. The 3D coordinate information prediction values ​​and the 2D coordinate information prediction values ​​constitute the detection result prediction value. Specifically, the values ​​of a and b are both set to 1. For the encoding / decoding modules E1-Em, the input data of the current encoding / decoding module corresponds sequentially to the output data of m-1 basic feature extraction modules from basic feature extraction module C(n-m+1) to basic feature extraction module Cn. The data input to the depth encoding / decoding unit must first undergo depth transformation through a linearly increasing distribution, that is, the following linearly increasing distribution function is used to generate the input m-1 target instance feature maps, where d c d represents the calculated incremental depth value. min d represents the minimum depth value within the specified range. maxd represents the maximum depth value within the specified range. i This represents the depth value of the feature map of the i-th target instance, where D represents the set number of depth feature layers.

[0051]

[0052] The loss function module is used to calculate the error between the predicted value and the true value of the detection result, including a classification loss calculation unit, a 2D loss calculation unit, a depth loss calculation unit, and an angle loss calculation unit.

[0053]

[0054]

[0055]

[0056] in, This represents the loss function for 2D prediction, and the loss value is calculated using the 2D loss calculation unit; IoU represents the overlap. represents the ratio of the Euclidean distances between the predicted ground truth box and the true ground truth box at the center point, length, and width, respectively; b represents the coordinates of the center point of the predicted ground truth box. gt represents the coordinates of the center point of the true ground truth box, c represents the diagonal distance of the minimum closure containing both the predicted and true ground truth boxes, and w represents the width of the predicted ground truth box. gt The width of the truth box, c w h represents the width of the smallest closure containing both the predicted and true truth boxes, and h represents the height of the predicted truth boxes. gt c represents the height of the truth box. h It represents the height of the smallest closure that simultaneously contains both the predicted truth boxes and the true truth boxes. The focus loss represents the depth prediction, in which W... F H F Calculate the predicted depth D(u,v) and ground truth depth for each point on the constructed feature map. The focus loss is summed and averaged, which is achieved through the depth loss calculation unit, W. F H F The width and height of the target instance feature map are represented by , and D(u,v) represents the predicted depth of the point with coordinates (u,v) in the target instance feature map. The ground truth depth of the point with coordinates (u,v) in the feature map of the target instance represents the ground truth depth. Represents taking D(u,v), The focus of the loss. These represent the cross-entropy loss for classification and rotation angle, respectively. The regularization loss function representing the whole, λ cls ,λ 2d ,λ depth ,λ θ It is the regularization factor, which can be set to 0.25, 0.25, 0.25, 0.25 in sequence.

[0057] S2. Validate and adjust the basic network structure using a monocular 3D object detection dataset to construct the final monocular 3D object detection model.

[0058] Adjustments to the basic network structure include, but are not limited to: adjusting the number of stages in the basic feature extraction module, the number of basic feature extraction units contained in each stage's basic feature extraction module, and the candidate operations for each basic feature extraction unit.

[0059] S3. Retrain the final monocular 3D object detection model constructed in step S2 to achieve optimal detection accuracy.

[0060] The final monocular 3D object detection model was trained using a monocular 3D object detection dataset, and network parameters, including but not limited to the loss function regularization factor and normalization parameters, were adjusted. The basic network structure was validated and tuned using the 2D detection accuracy results (mAP). 2d and 3D detection accuracy results mAP 3d The network structure is optimized using the loss function value (Loss). 2D detection accuracy (mAP) results are also analyzed. 2d and 3D detection accuracy results mAP 3d The overlap between the detection results obtained from pre-training on the final monocular 3D object detection model and the data provided in the validation set is compared. This is achieved by introducing normalization parameters α1, α2, and α3 and performing a weighted sum. Finally, the model with the highest weighted sum is selected as the final monocular 3D object detection model. Specifically, the normalization parameters α1, α2, and α3 can be set to 0.4, 0.4, and 0.2, respectively.

[0061] The weighted normalization calculation function described above is as follows:

[0062] F=α1mAP 2d +α2mAP 3d +α3Loss

[0063] S4. Use the final monocular 3D target detection model obtained in step S3 to perform detection.

[0064] Preferably, the monocular 3D object detection method proposed in this application further includes the step of processing the monocular 3D object detection dataset using a depth-based data augmentation method. Step S3 uses the data-augmented dataset to train the final monocular 3D object detection model, which can optimize the network parameters.

[0065] like Figure 5 As shown, the data augmentation method consists of three steps: 1) Extracting target instances from the monocular 3D object detection dataset using a pre-trained semantic segmentation model; 2) Calculating the relevant parameters (X, Y, Z) of instances scaled down / scaled to the corresponding size using the following scaling formula. These represent the 3D coordinates before and after scaling, where Designed by humans, specifically v is the height of the extracted instance, v horizon Let f be the horizontal line height of the input image, f be the camera focal length of the monocular 3D object detection dataset, and p be the horizontal line height. 2d [u,v,1] T The mapping matrix representing the projection of the 3D center position coordinates of the target instance into 2D coordinates, (c u ,c v ) represents the 2D coordinates of the instance placement location, with the instance center position as the placement location, which is set manually; 3) put the scaled-down / scaled instance back into the original monocular 3D object detection dataset.

[0066]

[0067]

[0068]

[0069] In the specific implementation of monocular 3D object detection for autonomous driving, the experimental environment consisted of Ubuntu 18.04, an Intel Xeon E5 CPU, DDR4 256GB RAM, and an NVIDIA 2080Ti GPU. The PyTorch deep learning framework was used for network training. The dataset used was the KITTI 3D object detection dataset, jointly produced by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America. It is currently the largest international dataset for evaluating algorithms in autonomous driving scenarios. KITTI includes real-world image data collected from urban, rural, and highway scenes, with each image containing up to 15 vehicles and 30 pedestrians, along with varying degrees of occlusion and truncation. The entire dataset consists of 389 pairs of stereo images and optical flow maps, 39.2km visual ranging sequences, and over 200,000 images of 3D-annotated objects. We selected only the RGB images as input data and the 3D bounding box annotations as labels. The dataset was divided into training, validation, and test sets in a 6:1:3 ratio.

[0070] First, a network structure encompassing all possible operations is constructed, meaning that every operation in all basic feature extraction units within this network model participates in the computation. The specific data transmission process is as follows: An input image with a resolution of 1280*380 is passed through the first basic feature extraction unit, where it undergoes operations such as 1*1 and 3*3 ordinary convolutions. The output parameters are then processed using BatchNorm and ReLU activation operations to obtain the output feature map. Subsequent basic feature extraction units follow the same principle. The network structure includes feature extraction modules C1-C5, with the number of output channels for each stage being 64, 128, 512, 1024, and 2048, respectively. After each stage's feature extraction module completes its operation and obtains the corresponding output feature map, the feature map is fed into the corresponding visual and depth encoding / decoding units for encoding and decoding. The visual encoding / decoding module outputs the 2D coordinates of the detected object, while the depth encoding / decoding module outputs the 3D coordinates. During training, the prediction results are fed into the loss function module for loss calculation. The network is then optimized by weighting the detection accuracy and the loss function value. During the verification phase, the network will eventually output the coordinates of the eight vertices of the target's 3D bounding box in the image.

[0071] In summary, the Transformer-based monocular 3D object detection algorithm designed in this embodiment has the following advantages compared with existing technologies: 1. It uses depth feature map transformation instead of depth pre-estimation models or complex geometric priors for depth estimation, reducing depth estimation errors. 2. The network model has advantages such as small memory footprint, high detection accuracy, and low latency. It can be well applied to practical monocular 3D object detection tasks.

[0072] Any aspects not detailed in this application are well-known to those skilled in the art.

[0073] The preferred embodiments of this application have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the inventive concept of this application without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the inventive concept of this application through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A Transformer-based monocular 3D object detection method, characterized in that, The method comprises the following steps: Step one, constructing a network basic structure, the network basic structure comprising: a feature extraction module for extracting a feature map of a target instance, the feature extraction module being composed of n stage basic feature extraction modules connected in sequence, each stage basic feature extraction module comprising a basic feature extraction unit connected in sequence, the output channel numbers of the basic feature extraction units in the same stage basic feature extraction module being the same, n and a being integers greater than 1, The coding module is configured to encode and decode the depth and visual information of the target instance feature map by using the attention mechanism in the transformer and output a prediction detection result. The coding module includes m stage coding modules. The input data of the mth stage coding module is the target instance feature map output by the n-m+1th stage basic feature extraction module to the nth stage basic feature extraction module. Each stage coding module includes b depth coding and decoding units and c visual coding and decoding units. The b depth coding and decoding units are configured to encode and decode the depth information of the target instance feature map by using the attention mechanism in the transformer and the feedforward network, and output 3D coordinate information prediction values. The c visual coding and decoding units are configured to encode and decode the visual information of the target instance feature map by using the attention mechanism in the transformer and the feedforward network, and output 2D coordinate information prediction values. The input data of the depth coding and decoding unit is obtained by converting the depth of the target instance feature map, and a linearly increasing distribution function is used The target instance feature map is converted in depth. m, b and c are integers greater than 1, and m is less than or equal to n. represents the calculated increasing depth value, represents the minimum depth value, represents the maximum depth value, represents the depth value of the ith target instance feature map, represents the number of layers of the target instance depth feature map, and a loss function module for calculating a loss function value according to a predicted detection result and a true value; Step two, verifying and adjusting the network basic structure through a monocular 3D target detection dataset to construct a final monocular 3D target detection model; Step three, retraining the final monocular 3D target detection model constructed in step two until the final monocular 3D target detection model reaches a target detection precision; Step four, detecting by using the final monocular 3D target detection model obtained in step three.

2. The Transformer-based monocular 3D object detection method of claim 1, wherein, The loss function module comprises a classification loss calculation unit, a 2D loss calculation unit, a depth loss calculation unit and an angle loss calculation unit, and the result of weighted summation of the loss function values calculated by each unit is the loss function value. 3.The Transformer-based monocular 3D object detection method of claim 1, wherein, The basic feature extraction unit performs operations on the input feature map, including but not limited to ordinary convolution, deformable convolution, depth separable convolution and residual estimation.

4. The Transformer-based monocular 3D object detection method of claim 2, wherein, The loss function of the 2D loss calculation unit is: The loss function of the depth loss calculation unit is: wherein, represents the loss function in 2D prediction, represents the overlap degree, respectively represent the proportion of the Euclidean distance of the center point, length and width of the predicted ground truth and the real ground truth, represents the center point coordinate of the predicted ground truth, represents the center point coordinate of the real ground truth, represents the diagonal distance of the minimum closed range containing the predicted ground truth and the real ground truth at the same time, represents the width of the predicted ground truth, represents the width of the real ground truth, represents the width of the minimum closed range containing the predicted ground truth and the real ground truth at the same time, represents the height of the predicted ground truth, represents the height of the real ground truth, represents the height of the minimum closed range containing the predicted ground truth and the real ground truth at the same time, represents the focal point loss in depth prediction, represents the width and height of the target instance feature map, represents the predicted depth of the point with coordinates in the target instance feature map, represents the ground truth depth of the point with coordinates in the target instance feature map, represents the focal point loss taking .

5. The Transformer-based monocular 3D object detection method of claim 4, wherein, The step three re-trains the final monocular 3D target detection model constructed in step two by using a monocular 3D target detection dataset, and adjusts network parameters including but not limited to loss function regularization factors and normalization parameters, specifically: the monocular 3D target detection dataset is used to pre-train the final monocular 3D target detection model constructed, and the 2D detection accuracy result and the 3D detection accuracy result and the loss function value obtained by pre-training are weighted and summed, the pre-training process is repeated after adjusting network parameters including but not limited to loss function regularization factors and normalization parameters, and the final monocular 3D target detection model with the highest weighted sum result is taken as the monocular 3D target detection model meeting the target detection accuracy requirement.

6. The Transformer-based monocular 3D object detection method of claim 5, wherein, The method further comprises a step of processing the monocular 3D target detection dataset by using a depth-based data enhancement method.

7. The Transformer-based monocular 3D object detection method of claim 6, wherein, The step of processing the monocular 3D target detection dataset by using the depth-based data enhancement method specifically comprises: extracting target instances in the monocular 3D target detection dataset by using a pre-trained semantic segmentation model; scaling the extracted target instance by means of coordinate conversion, wherein, and respectively represent the 3D coordinates of the target instance before and after scaling, is a mapping matrix for projecting the 3D center position coordinates of the target instance to 2D coordinates, is the height of the extracted instance, is the horizontal line height of the input feature image, is the camera focal length of the monocular 3D target detection dataset, is the 2D coordinates of the center position of the target instance; putting the scaled target instances back into the monocular 3D target detection dataset.

Images