Monocular camera visual angle vehicle three-dimensional information detection method based on deep learning

Abstract

The invention discloses a monocular camera visual angle vehicle three-dimensional information detection method based on deep learning, and the method comprises the steps: obtaining a vehicle three-dimensional detection data set in a monocular camera, carrying out the data processing of the vehicle three-dimensional detection data set, carrying out the supplementary marking, building a pre-trained vehicle three-dimensional detection network model based on a Center Net network structure, and carrying out the detection of the visual angle vehicle three-dimensional information of the monocular camera. Fusing a weighted multi-scale feature map fusion strategy into the vehicle three-dimensional detection network model, inputting a vehicle labeling three-dimensional detection data set into the target vehicle three-dimensional detection network model for training, and obtaining coordinates of a vehicle center point in an image, image coordinates of a vehicle three-dimensional frame vertex and three-dimensional size information of the vehicle, the three-dimensional centroid of the vehicle is calculated in combination with camera calibration parameters, the accurate vehicle type, the accurate three-dimensional space position, the accurate three-dimensional frame vertex and the accurate three-dimensional size can be obtained through image recognition, and accurate positioning of the vehicle in the three-dimensional space is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning. Background Art

[0002] At present, the number of vehicles on the roads in my country is showing a growing trend. With the substantial increase in the number of vehicles, various traffic safety, traffic congestion and other problems have become increasingly prominent.

[0003] In the field of smart transportation, vehicle perception in traffic monitoring scenarios is an important part of smart vehicle-road collaboration. In traffic monitoring scenarios, monocular cameras have always dominated due to their low price and wide deployment, and are also a hot topic in the research of vehicle 3D detection. However, due to their inherent imaging mechanism, monocular cameras cannot obtain depth information, which also makes them a huge challenge in vehicle 3D detection. How to obtain accurate depth information is crucial for accurate vehicle 3D detection. Current monocular camera-based vehicle 3D detection methods can be divided into the following four categories: (1) Methods based on geometric information, including feature points, CAD models, and vanishing points. (2) Methods based on deep learning for 2D target detection and geometric information constraints. (3) Methods based on depth map estimation. (4) Methods based on deep learning for 3D feature estimation. In recent years, with the rapid development of deep learning technology, the detection of vehicle 3D information using deep learning technology has shown stronger processing performance than traditional methods. However, compared with the on-board perspective methods, the number of roadside perspective vehicle 3D detection methods is relatively small, and the authoritative data sets are relatively small, which is not conducive to the research and verification of vehicle 3D perception methods from the perspective of monocular cameras in vehicle-road collaboration systems. Summary of the invention

[0004] The purpose of the present invention is to solve the above problems and design a method for detecting three-dimensional information of vehicles from a monocular camera perspective based on deep learning.

[0005] To achieve the above-mentioned purpose, the technical solution of the present invention is that, further, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the method for detecting three-dimensional information of a vehicle from a monocular camera perspective comprises the following steps:

[0006] Acquire a vehicle three-dimensional detection data set from a monocular camera, perform data processing on the vehicle three-dimensional detection data set, and then perform additional marking to obtain a vehicle annotated three-dimensional detection data set;

[0007] A pre-trained vehicle 3D detection network model is established based on the CenterNet network structure, and a weighted multi-scale feature map fusion strategy is integrated into the vehicle 3D detection network model to obtain a target vehicle 3D detection network model;

[0008] Inputting the vehicle annotated three-dimensional detection data set into the target vehicle three-dimensional detection network model for training to obtain vehicle type data;

[0009] The coordinates of the vehicle center point in the image, the image coordinates of the vertices of the vehicle's three-dimensional box, and the three-dimensional size information of the vehicle are obtained. The three-dimensional center of mass of the vehicle is calculated in combination with the camera calibration parameters to obtain the vehicle's three-dimensional spatial positioning information.

[0010] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the step of obtaining a three-dimensional vehicle detection dataset from a monocular camera and performing data processing on the three-dimensional vehicle detection dataset to obtain an initial three-dimensional vehicle detection dataset includes:

[0011] By acquiring the BrnoComp dataset and the collected urban scene data, a vehicle 3D detection dataset is obtained;

[0012] The vehicle three-dimensional detection data set source is processed in sequence by random contrast, color gamut transformation, horizontal flipping and perspective transformation to obtain an initial vehicle three-dimensional detection data set;

[0013] The vehicles in the initial vehicle 3D detection dataset are supplemented with annotations using the annotation tool LabelImg-3D to obtain vehicle type, vehicle center point image coordinates, vehicle 3D box vertex image coordinates and vehicle 3D size information, thereby obtaining a vehicle annotated 3D detection dataset.

[0014] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, a pre-trained three-dimensional vehicle detection network model is established based on the CenterNet network structure, and a weighted multi-scale feature map fusion strategy is integrated into the three-dimensional vehicle detection network model to obtain a three-dimensional detection network model of a target vehicle, including:

[0015] The vehicle 3D detection network model at least includes a backbone feature extraction network, multi-scale feature fusion and a multi-task detection head;

[0016] The backbone feature extraction network includes an Hourglass network, a DLA network and a ResNet network, wherein the Hourglass network has an hourglass structure, the DLA network has a deep aggregation layer structure, and the ResNet network has a residual structure;

[0017] The multi-scale feature fusion includes three deconvolution layers with stride=2, which fuse the multi-scale feature maps into a fused feature map in a weighted manner. The fused feature map integrates the information of the vehicle at different scales.

[0018] The multi-task detection head contains 3 outputs.

[0019] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the vehicle annotated three-dimensional detection data set is input into the target vehicle three-dimensional detection network model for training to obtain vehicle type data, including:

[0020] Scale the vehicle annotated 3D detection dataset to P∈I X×Y×3 Input network, where X=Y=512;

[0021] Downsampling the vehicle annotated 3D detection dataset to extract features, with a downsampling factor of R=4, to obtain feature maps C3, C4, and C5 with sizes of 64×64, 32×32, and 16×16, respectively;

[0022] Using convolution and feature fusion techniques, a feature pyramid P is constructed F = {P3, P4, P5, P6, P7}; Deconvolution upsamples the feature maps of different sizes in the feature pyramid to unify the feature pyramid into a feature set of the same size

[0023] Using the weighted feature fusion strategy, as shown in formula (6), the feature set is fused into a fused feature map containing multi-scale information of the vehicle

[0024] The formula of the weighted feature fusion strategy is as follows:

[0025]

[0026] in, w k is the weight of the feature map, which is set to 0.2 by default.

[0027] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the step of inputting the vehicle annotated three-dimensional detection data set into the target vehicle three-dimensional detection network model for training to obtain vehicle type data further includes:

[0028] Design a multi-task regression detection head, which includes four branches: vehicle type classification, vehicle center point regression, vehicle 3D box vertex regression and vehicle 3D size regression;

[0029] A full convolution layer and an attention mechanism module are introduced into the vehicle type classification branch, and the vehicle center point regression, vehicle 3D box vertex regression and vehicle 3D size regression utilize the full convolution layer;

[0030] The output of the vehicle type classification branch is defined as the existence probability of each type of vehicle at each pixel point, forming a vehicle type heat map Where class represents the number of vehicle types;

[0031] The outputs of the remaining three regression branches are defined as the coordinate offsets of the center point of the vehicle's 3D box 3D box vertex coordinates and three-dimensional dimensions

[0032] Normalize the coordinates of the vehicle's 3D center point and 3D box vertices.

[0033] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the step of inputting the vehicle annotated three-dimensional detection data set into the target vehicle three-dimensional detection network model for training to obtain vehicle type data further includes:

[0034] The loss function of the target vehicle three-dimensional detection network model consists of two parts: a basic loss function and an enhanced loss function;

[0035] The basic loss function includes vehicle type classification, vehicle center point offset regression, vehicle 3D box vertex regression and vehicle 3D size regression loss;

[0036] The enhanced loss function includes reprojection loss and vehicle IoU loss.

[0037] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the basic loss function includes vehicle type classification, vehicle center point offset regression, vehicle three-dimensional box vertex regression and vehicle three-dimensional size regression loss, and also includes:

[0038] Vehicle type classification loss. In order to optimize the problem of imbalanced positive and negative samples in the dataset, the vehicle type classification loss L class Use FocalLoss, as shown in formula (7):

[0039]

[0040] Among them, N represents the number of positive samples, α and β are hyperparameters for adjusting the loss weights of positive and negative samples, and h xyc Represents the use of Gaussian kernel function The actual vehicle target obtained is in the feature map H c The response value in is, σ is the standard deviation calculated from the actual size of the vehicle target in the image coordinate system

[0041] The vehicle center point offset regression loss uses the L1 loss strategy, as shown in formula (8):

[0042]

[0043] Among them, E xy Indicates whether there is a vehicle target at x,y in the output feature map, d center Represents the real coordinates of the center point of the vehicle relative to the scaled input image size X,Y, Represents the true coordinates of the center point of the vehicle in the high-resolution feature map, Indicates the predicted vehicle center point offset;

[0044] The vehicle 3D box vertex regression loss uses the L1 loss strategy, as shown in formula (9):

[0045]

[0046] Among them, d vertex Represents the real coordinates of the vehicle key points relative to the scaled input image size X,Y, Represents the predicted coordinates of the vehicle's key points;

[0047] The three-dimensional regression loss of the vehicle is calculated using the L1 loss strategy, as shown in formula (10):

[0048]

[0049] in, A feature map representing the true 3D size of the vehicle, Feature map representing the 3D dimensions of vehicle prediction.

[0050] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the enhanced loss function includes reprojection loss and vehicle IoU loss, and further includes:

[0051] Reprojection loss, taking a single vehicle as an example, the image coordinates of the predicted and real vehicle 3D box vertices are and The corresponding world coordinate prediction value is The true value is The projected world and image coordinates of the vertices of the predicted vehicle 3D box are and k=1,2,…,8, the predicted vehicle 3D size is Already obtained on the network On the basis of Through coordinate transformation Then and A set of constraints is formed and the L1 loss strategy is used, as shown in formula (11):

[0052]

[0053] in, Represents the three-dimensional feature map of the vehicle obtained by using the camera calibration Cail and network prediction The calculated vehicle 3D frame reprojection vertex feature map, Represents the predicted vehicle 3D box vertex feature map relative to the original unscaled image size;

[0054] Vehicle IoU loss, assuming that the minimum enclosing rectangles of the predicted and real vehicle 3D box vertices in the image are F forecast and F real , using the L1 loss strategy, as shown in formula (12):

[0055]

[0056] Among them, F xy and It represents the predicted value and true value of the minimum bounding rectangle calculated by the vertex feature map of the vehicle 3D box relative to the original unscaled image size, and IoU represents the calculation strategy of the intersection over union loss.

[0057] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the loss function of the target vehicle three-dimensional detection network model is composed of a basic loss function and an enhanced loss function, and further includes:

[0058] Through the basic loss function and the enhanced loss function, the multi-task loss function can be defined as shown in formula (13):

[0059] L=λ class L class +λ center L center +λ vertex L vertex +λ size L size +λ project L project +λ cariou L cariou (13);

[0060] Among them, in order to balance the weight of each part of the loss, take λ class =1,λ center =1,λ vertex =1,λ size =1,λ project =1,λ cariou =1.

[0061] Furthermore, in the above-mentioned method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the step of inputting the vehicle annotated three-dimensional detection data set into the target vehicle three-dimensional detection network model for training to obtain vehicle type data further includes:

[0062] The vehicle annotated 3D detection dataset was scaled to 512×512 size during training, and the ratio of the training set to the validation set was set to 9:1;

[0063] The training consists of 100 rounds in total. The backbone network is frozen for the first 60 rounds and the batch size is set to 16. The training is unfrozen for the last 40 rounds and the batch size is set to 8.

[0064] The Adam optimizer is used in the training process, and the learning rate is dynamically adjusted. The initial setting is 0.001. The training is terminated when the verification loss tends to balance, and the vehicle type data is obtained.

[0065] The beneficial effect is that by acquiring the vehicle three-dimensional detection data set in the monocular camera, the vehicle three-dimensional detection data set is processed and supplemented with marks to obtain the vehicle annotated three-dimensional detection data set; a pre-trained vehicle three-dimensional detection network model is established based on the CenterNet network structure, and a weighted multi-scale feature map fusion strategy is integrated into the vehicle three-dimensional detection network model to obtain the target vehicle three-dimensional detection network model; the vehicle annotated three-dimensional detection data set is input into the target vehicle three-dimensional detection network model for training to obtain vehicle type data; the coordinates of the center point of the vehicle in the image, the image coordinates of the vertices of the vehicle three-dimensional frame and the three-dimensional size information of the vehicle are obtained, and the three-dimensional centroid of the vehicle is calculated in combination with the camera calibration parameters to obtain the vehicle three-dimensional space positioning information. The accurate vehicle type, three-dimensional space position, three-dimensional frame vertices and three-dimensional size can be obtained by recognizing the image, and the precise positioning of the vehicle in the three-dimensional space can be realized. The method has good performance in three-dimensional detection, positioning and size prediction, improves the accuracy and robustness of vehicle detection, and can take into account real-time performance in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0067] Figure 1 It is a schematic diagram of a first embodiment of a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning in an embodiment of the present invention;

[0068] Figure 2A schematic diagram of the CenterNet network structure of a method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning in an embodiment of the present invention;

[0069] Figure 3 The overall framework diagram of the algorithm of a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning in an embodiment of the present invention;

[0070] Figure 4 This is a structural diagram of a target vehicle three-dimensional detection network model of a method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning in an embodiment of the present invention;

[0071] Figure 5 The figure is a schematic diagram of weighted feature map fusion of a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0073] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0074] The present invention will be described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning, the method for detecting three-dimensional information of a vehicle from a monocular camera perspective comprises the following steps:

[0075] Step 101, obtaining a vehicle three-dimensional detection data set in a monocular camera, performing data processing on the vehicle three-dimensional detection data set, and performing supplementary marking to obtain a vehicle annotated three-dimensional detection data set;

[0076] Specifically, in this embodiment, an image set is constructed: a vehicle 3D detection dataset based on a monocular camera is constructed, and the data comes from the BrnoComp dataset and self-collected urban scenes. The dataset contains 5 types of scenes under typical perspectives, and the vehicle types include three types of cars, trucks, and buses. The training and validation sets contain 14,593 images, and the test set contains 2,237 images.

[0077] Labeled image set: Since the original data of the scene in the dataset only contains images and the three-dimensional spatial position information of the vehicle provided by the lidar, a self-made labeling tool LabelImg-3D is used to supplement the annotation of the vehicles in the image set to obtain the vehicle type, the image coordinates of the vehicle center point, the image coordinates of the vehicle three-dimensional box vertices and the vehicle three-dimensional size information, thereby obtaining complete labeling information.

[0078] Step 102: establishing a pre-trained vehicle 3D detection network model based on the CenterNet network structure, integrating a weighted multi-scale feature map fusion strategy into the vehicle 3D detection network model, and obtaining a target vehicle 3D detection network model;

[0079] Specifically, in this embodiment, CenterNet is selected as the basic network structure for vehicle three-dimensional information detection. The network structure CenterNet is one of the representatives of the no-priori box model. The network regards each target as a center point and predicts the attributes related to the target. Taking CenterNet under two-dimensional detection as an example, its network structure consists of three parts: the backbone feature extraction network, the enhanced feature extraction module and the detection head. Figure 2 CenterNet is the network structure diagram. Assume that the input image is P∈I X×Y×3 , X and Y are the width and height of the image respectively.

[0080] Its main feature extraction networks include Hourglass, DLA and ResNet. The Hourglass network has an hourglass structure, the DLA network has a deep aggregation layer structure, and the ResNet network has a residual structure. All three have good feature extraction capabilities. After the main feature extraction network, a feature map with a downsampling multiple of R is obtained.

[0081] The enhanced feature extraction module is implemented by upsampling and contains 3 deconvolution layers with stride=2. By enhancing the feature extraction module, a high-resolution feature map can be obtained. This high-resolution feature map is equivalent to dividing the image into When the center point of the target falls within a certain grid, the grid is responsible for the prediction task. Each grid is responsible for predicting the attributes of a target.

[0082] The detection head contains three outputs, which are obtained by P" passing through the detection head three times through 3×3 and 1×1 convolutions, including: detecting the existence of the target, using the center point heat map Indicates that C represents the number of class target categories; the center point offset of the detected target is used Indicates; the size of the detection target relative to P″, using express.

[0083] Training phase:

[0084] In this stage, the projection of the center point of each category target on the high-resolution feature map is recorded as Use Gaussian kernel function Calculate the true value of the center point heat map where σ p Refers to the standard deviation of target size adaptation. After obtaining the true value of the heat map, the focal loss is used to train the heat map, as shown in formula (1):

[0085]

[0086] Among them, α and β refer to the hyperparameters of Focal Loss, and the reference settings are α=2 and β=4, and T is the number of objects in the image.

[0087] Offset from center point Use L1 regression loss, as shown in formula (2):

[0088]

[0089] For the target size attribute, let the two-dimensional box of the nth target be (x n ′,y n ′,x n ″,y n ″), the center point is The target size is N = (x′ n ′-x′ n ,y′ n ′-y′ n ). Then the target size As shown in formula (3):

[0090]

[0091] Then the total loss function L det It can be defined as shown in formula (4):

[0092] L det =L heat +λ offset L offset +λ size L size (4)

[0093] Among them, λ size and λ offset To balance the loss coefficient, it is usually set to 0.1 and 1.

[0094] Reasoning process:

[0095] After training, in the network inference process, the target peak position in the heat map is first extracted. The coordinates of the n extracted center points of category c The integer coordinates of the center point are Record the center point response value is the confidence level of the detected target, then The upper left and lower right coordinates of the predicted target box at can be expressed by formula (5):

[0096]

[0097] in, Indicates the center point The predicted value of the offset at Indicates the center point The predicted value of the size at .

[0098] The vehicle 3D detection network for monocular cameras consists of three parts: backbone feature extraction network, multi-scale feature fusion and multi-task detection head. First, the original image is scaled to P∈I X×Y×3 The input network, where X = Y = 512, is then downsampled to extract features, with a downsampling factor of R = 4.

[0099] Backbone feature extraction network. The backbone feature extraction network uses ResNet-50 with a residual structure, which has good feature extraction capabilities.

[0100] Multi-scale feature fusion. After being processed by the backbone feature extraction network, a series of hierarchical feature maps are obtained. Among these feature maps, high-resolution feature maps retain more accurate local details, which is very beneficial for the detection of small targets; while low-resolution feature maps gather higher-level semantic information and are more suitable for the detection of large targets. In actual traffic scenarios, in the images captured by the monocular camera on the side of the road, vehicles may appear at various positions on the road, and their sizes are also different in the image. In order to enhance the network's detection ability for vehicles of different sizes, we propose an innovative weighted multi-scale feature map fusion strategy. This method fuses multi-scale feature maps into a fused feature map in a weighted manner, which integrates the information of vehicles at different scales. The process of weighted feature map fusion is demonstrated. In this way, we improve the accuracy and robustness of vehicle detection.

[0101] Step 103: input the vehicle annotated 3D detection data set into the target vehicle 3D detection network model for training to obtain vehicle type data;

[0102] After the backbone feature extraction network, we obtain feature maps C3, C4, and C5 with sizes of 64×64, 32×32, and 16×16 respectively. In order to further utilize these feature maps, we use convolution and feature fusion techniques to construct a feature pyramid P F ={P3,P4,P5,P6,P7}. Unlike YOLO and other methods that directly use the feature pyramid, our method performs deconvolution upsampling on the feature maps of different sizes in the feature pyramid to unify them into feature sets of the same size. Next, we adopted a weighted feature fusion strategy, as shown in formula (6), to fuse these feature maps into a fused feature map containing multi-scale information of the vehicle This method not only enables us to obtain richer feature information, but also reduces the amount of computation in the subsequent prediction process and improves the overall efficiency of the network.

[0103]

[0104] in, w k is the weight of the feature map, which is set to 0.2 by default.

[0105] Multi-task detection head. Based on the multi-scale fusion feature map, we designed a multi-task regression detection head for the needs of vehicle three-dimensional detection under the perspective of roadside monitoring. This detection head contains four branches: vehicle type classification, vehicle center point regression, vehicle three-dimensional box vertex regression, and vehicle three-dimensional size regression. In order to enhance the network's ability to distinguish different types of vehicles, we introduced a full convolutional layer and an attention mechanism module in the vehicle type classification branch, while the other three regression branches only use a full convolutional layer. Drawing on the idea of ​​CenterNet, we define the output of the vehicle type classification branch as the probability of existence of each type of vehicle at each pixel point to form a vehicle type heat map Where class represents the number of vehicle types. The outputs of the other three regression branches are defined as the coordinate offset of the center point of the vehicle's 3D box. 3D box vertex coordinates And three-dimensional dimensions In addition, in order to maintain the stability of the training process, we normalized the coordinates of the 3D center point and 3D box vertices of the vehicle. This design enables our network to more accurately detect and identify vehicles of different sizes and types, thus meeting the actual needs of roadside monitoring.

[0106] During the training process of the vehicle 3D detection network, the loss function consists of two parts: the basic loss function and the enhanced loss function, with a total of six components. The basic loss function includes vehicle type classification, vehicle center point offset regression, vehicle 3D box vertex regression, and vehicle 3D size regression losses, which are necessary during the training process. In particular, the latter three types of regression losses can also be called vehicle 3D information regression losses. The enhanced loss function is an optional loss function that embeds the loss of spatial constraints, including reprojection constraints using camera calibration priors and vehicle IoU constraints. These enhanced loss functions are designed to improve the accuracy of vehicle 3D detection and spatial positioning. Through this combination, our training process can take various factors into account more comprehensively to optimize the performance of the model.

[0107] (1) Vehicle type classification loss. In order to optimize the problem of imbalance between positive and negative samples in the dataset, the vehicle type classification loss L class Use FocalLoss, as shown in formula (6):

[0108]

[0109] Among them, N represents the number of positive samples, α and β are hyperparameters for adjusting the loss weights of positive and negative samples, and h xyc Represents the use of Gaussian kernel function The actual vehicle target obtained is in the feature map H c The response value in is , and σ is the standard deviation calculated from the actual size of the vehicle target in the image coordinate system.

[0110] (2) Vehicle center point offset regression loss. Use the L1 loss strategy, as shown in formula (8):

[0111]

[0112] Among them, E xy Indicates whether there is a vehicle target at x,y in the output feature map, d center Represents the real coordinates of the center point of the vehicle relative to the scaled input image size X,Y, Represents the true coordinates of the center point of the vehicle in the high-resolution feature map, Indicates the predicted vehicle center point offset.

[0113] (3) Vehicle 3D box vertex regression loss. Use the L1 loss strategy, as shown in formula (9):

[0114]

[0115] Among them, d vertex Represents the real coordinates of the vehicle key points relative to the scaled input image size X,Y, Represents the predicted vehicle key point coordinates.

[0116] (4) Vehicle 3D size regression loss. Use the L1 loss strategy, as shown in formula (10):

[0117]

[0118] in, A feature map representing the true 3D size of the vehicle, Feature map representing the 3D dimensions of vehicle prediction.

[0119] (5) Reprojection loss. Taking a single vehicle as an example, let the image coordinates of the vertices of the predicted and real vehicle 3D boxes be and The corresponding world coordinate prediction value is The true value is The projected world and image coordinates of the vertices of the predicted vehicle 3D box are and The predicted vehicle 3D size is Already obtained on the network On the basis of Through coordinate transformation Then and A set of constraints is formed, where blue, red, and green represent the predicted box, the true box, and the projected box, respectively. The L1 loss strategy is used, as shown in formula (11):

[0120]

[0121] in, Represents the three-dimensional feature map of the vehicle obtained by using the camera calibration Cail and network prediction The calculated vehicle 3D frame reprojection vertex feature map, Represents the feature map of predicted vehicle 3D box vertices relative to the original unscaled image size.

[0122] (6) Vehicle IoU loss. Assume that the minimum enclosing rectangles of the predicted and real vehicle 3D box vertices in the image are F forecast and F real , blue represents the predicted box, red represents the real box, and the green shadow represents the overlapping area of ​​the two, which is used to calculate the vehicle IoU to form a set of constraints. The L1 loss strategy is used, as shown in formula (12):

[0123]

[0124] Among them, F xy and It represents the predicted value and true value of the minimum bounding rectangle calculated by the vertex feature map of the vehicle 3D box relative to the original unscaled image size. IoU represents the calculation strategy of intersection over union loss. This method adopts CarIoU loss.

[0125] Through the above six loss functions, the multi-task loss function can be defined as shown in formula (13):

[0126] L=λ class L class +λ center L center +λ vertex L vertex +λ size L size +λ project L project +λ cariou L cariou (13)

[0127] In order to balance the weights of the losses in each part, λ is taken in the experiment. class =1,λ center =1,λ vertex =1,λ size =1,λ project =1,λ cariou =1.

[0128] Step 104: Obtain the coordinates of the center point of the vehicle in the image, the image coordinates of the vertices of the vehicle's three-dimensional frame, and the three-dimensional size information of the vehicle, calculate the three-dimensional center of mass of the vehicle in combination with the camera calibration parameters, and obtain the three-dimensional spatial positioning information of the vehicle.

[0129] Data enhancement. During the training process, data enhancement strategies such as random contrast and color gamut transformation, horizontal flipping, and perspective transformation are used to expand the richness of training data set samples, simulate more roadside monitoring perspectives and scene environments, and prevent overfitting. This is a schematic diagram of the data enhancement strategy used in this method. In order to further enhance the adaptability of the network to the scene, perspective transformation is used to simulate the change of camera perspective based on the camera imaging principle. After data enhancement, the training validation set can be expanded to 4 times the original size.

[0130] In this embodiment, network training. After data enhancement, the data set is trained by deep learning neural network. The experiment uses PyTorch deep learning framework, Core i7-8700 CPU and GTX 1080ti GPU. The image is scaled to 512×512 during training, and the ratio of training set to validation set is set to 9:1. The training includes a total of 100 rounds. The backbone network is frozen for training in the first 60 rounds, and the Batchsize is set to 16. The training is unfrozen for the next 40 rounds, and the Batchsize is set to 8. The Adam optimizer is used during training, and the learning rate is dynamically adjusted, and the initial setting is 0.001. End the training when the validation loss tends to balance. Obtain the optimal training model to prevent overfitting.

[0131] Specifically, in this embodiment, the detection results. The original image to be detected obtained by the camera is passed into the training model, and the model outputs the existence of the target in the grid; the offset of the target relative to the center grid; the size of the target relative to the high-resolution feature; decode and draw, and use a cube grid to envelop the detected target on the image. The left side is the three-dimensional detection result of the vehicle, represented by a colored wireframe, and the right side is the heat map of the center point distribution of the vehicle, and the response center is represented by red.

[0132] The beneficial effect is that by acquiring the vehicle three-dimensional detection data set in the monocular camera, the vehicle three-dimensional detection data set is processed and supplemented with tags to obtain the vehicle annotated three-dimensional detection data set; a pre-trained vehicle three-dimensional detection network model is established based on the CenterNet network structure, and a weighted multi-scale feature map fusion strategy is integrated into the vehicle three-dimensional detection network model to obtain the target vehicle three-dimensional detection network model; the vehicle annotated three-dimensional detection data set is input into the target vehicle three-dimensional detection network model for training to obtain vehicle type data; the coordinates of the center point of the vehicle in the image, the image coordinates of the vertices of the vehicle three-dimensional frame, and the three-dimensional size information of the vehicle are obtained, and the three-dimensional center of mass of the vehicle is calculated in combination with the camera calibration parameters to obtain the vehicle three-dimensional space positioning information. The accurate vehicle type, three-dimensional space position, three-dimensional frame vertices and three-dimensional size can be obtained by recognizing the image, and the precise positioning of the vehicle in the three-dimensional space can be realized. The method has good performance in three-dimensional detection, positioning and size prediction, improves the accuracy and robustness of vehicle detection, and can take into account real-time performance in practical applications.

[0133] like Figure 2 The above is a schematic diagram of the CenterNet network structure of a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning;

[0134] Step 3, CenterNet target detection.

[0135] Specifically, in this embodiment, the network structure: This example selects CenterNet as the basic network structure for vehicle three-dimensional information detection. The network structure CenterNet is one of the representatives of the no-priori box model. The network regards each target as a center point and predicts the attributes related to the target. Taking CenterNet under two-dimensional detection as an example, its network structure consists of three parts: the backbone feature extraction network, the enhanced feature extraction module and the detection head. Figure 2 CenterNet is the network structure diagram. Assume that the input image is P∈I X×Y×3 , X and Y are the width and height of the image respectively.

[0136] Its main feature extraction networks include Hourglass, DLA and ResNet. The Hourglass network has an hourglass structure, the DLA network has a deep aggregation layer structure, and the ResNet network has a residual structure. All three have good feature extraction capabilities. After the main feature extraction network, a feature map with a downsampling multiple of R is obtained.

[0137] The enhanced feature extraction module is implemented by upsampling and contains 3 deconvolution layers with stride=2. By enhancing the feature extraction module, a high-resolution feature map can be obtained. This high-resolution feature map is equivalent to dividing the image into When the center point of the target falls within a certain grid, the grid is responsible for the prediction task. Each grid is responsible for predicting the attributes of a target.

[0138] The detection head contains three outputs, which are obtained by P" passing through the detection head three times through 3×3 and 1×1 convolutions, including: detecting the existence of the target, using the center point heat map Indicates that C represents the number of class target categories; the center point offset of the detected target is used Indicates; the size of the detection target relative to P″, using express.

[0139] like Figure 3 As shown, the overall algorithm framework diagram of a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning;

[0140] Specifically, the core algorithm of the method in this embodiment includes two steps: camera calibration and vehicle 3D detection. Figure 3As shown in the figure. First, we perform the camera calibration process to establish an accurate mapping relationship between the two-dimensional image space and the three-dimensional world space. Then, we use the designed vehicle three-dimensional detection network to identify the vehicle type and obtain the coordinates of the vehicle center point in the image, the image coordinates of the vertices of the vehicle three-dimensional box, and the three-dimensional size information of the vehicle. Finally, combined with the camera calibration parameters obtained previously, we calculate the three-dimensional center of mass of the vehicle, thereby achieving accurate positioning of the vehicle in three-dimensional space.

[0141] like Figure 4 The above is a structural diagram of a target vehicle three-dimensional detection network model of a vehicle three-dimensional information detection method from a monocular camera perspective based on deep learning;

[0142] Specifically, the vehicle 3D detection network for a monocular camera in this embodiment includes three parts: a backbone feature extraction network, a multi-scale feature fusion network, and a multi-task detection head. Figure 4 As shown, first scale the original image to P∈I X ×Y×3 The input network, where X = Y = 512, is then downsampled to extract features, with a downsampling factor of R = 4.

[0143] like Figure 5 The above is a schematic diagram of weighted feature map fusion of a method for detecting three-dimensional information of a vehicle from a monocular camera perspective based on deep learning;

[0144] Specifically, in this embodiment, multi-scale features are fused. After being processed by the backbone feature extraction network, a series of hierarchical feature maps are obtained. Among these feature maps, the high-resolution feature maps retain more accurate local details, which is very beneficial for the detection of small targets; while the low-resolution feature maps gather higher-level semantic information and are more suitable for the detection of large targets. In actual traffic scenarios, in the images captured by the monocular camera on the side of the road, vehicles may appear at various positions on the road, and their sizes are also different in the image. In order to enhance the network's detection capabilities for vehicles of different sizes, we propose an innovative weighted multi-scale feature map fusion strategy. This method fuses multi-scale feature maps into a fused feature map in a weighted manner. The fused feature map integrates the information of the vehicle at different scales. Figure 5 As shown in Figure 2, the process of weighted feature map fusion is demonstrated. In this way, we improve the accuracy and robustness of vehicle detection.

[0145] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning, characterized in that: The method for detecting three-dimensional information of a vehicle from a monocular camera perspective comprises the following steps: Acquire a vehicle three-dimensional detection data set from a monocular camera, perform data processing on the vehicle three-dimensional detection data set, and then perform additional marking to obtain a vehicle annotated three-dimensional detection data set; A pre-trained vehicle 3D detection network model is established based on the CenterNet network structure, and a weighted multi-scale feature map fusion strategy is integrated into the vehicle 3D detection network model to obtain a target vehicle 3D detection network model; Inputting the vehicle annotated three-dimensional detection data set into the target vehicle three-dimensional detection network model for training to obtain vehicle type data; The coordinates of the vehicle center point in the image, the image coordinates of the vertices of the vehicle's three-dimensional box, and the three-dimensional size information of the vehicle are obtained. The three-dimensional center of mass of the vehicle is calculated in combination with the camera calibration parameters to obtain the vehicle's three-dimensional spatial positioning information.

2. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 1, characterized in that: The step of acquiring a vehicle three-dimensional detection data set from a monocular camera and performing data processing on the vehicle three-dimensional detection data set to obtain an initial vehicle three-dimensional detection data set includes: By acquiring the BrnoComp dataset and the collected urban scene data, a vehicle 3D detection dataset is obtained; The vehicle three-dimensional detection data set source is processed by random contrast, color gamut transformation, horizontal flipping and perspective transformation in sequence to obtain an initial vehicle three-dimensional detection data set; The vehicles in the initial vehicle 3D detection dataset are supplemented with annotations using the annotation tool LabelImg-3D to obtain vehicle type, vehicle center point image coordinates, vehicle 3D box vertex image coordinates and vehicle 3D size information, thereby obtaining a vehicle annotated 3D detection dataset.

3. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 1, characterized in that: The pre-trained vehicle 3D detection network model is established based on the CenterNet network structure, and a weighted multi-scale feature map fusion strategy is integrated into the vehicle 3D detection network model to obtain a target vehicle 3D detection network model, including: The vehicle 3D detection network model at least includes a backbone feature extraction network, multi-scale feature fusion and a multi-task detection head; The backbone feature extraction network includes an Hourglass network, a DLA network and a ResNet network, wherein the Hourglass network has an hourglass structure, the DLA network has a deep aggregation layer structure, and the ResNet network has a residual structure; The multi-scale feature fusion includes three deconvolution layers with stride=2, which fuse the multi-scale feature maps into a fused feature map in a weighted manner. The fused feature map integrates the information of the vehicle at different scales. The multi-task detection head contains 3 outputs.

4. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 1, characterized in that: The step of inputting the vehicle annotated three-dimensional detection data set into the target vehicle three-dimensional detection network model for training to obtain vehicle type data includes: Scale the vehicle annotated 3D detection dataset to P∈I X×Y×3 Input network, where X=Y=512; Downsampling the vehicle annotated 3D detection dataset to extract features, with a downsampling factor of R=4, to obtain feature maps C3, C4, and C5 with sizes of 64×64, 32×32, and 16×16, respectively; Using convolution and feature fusion techniques, a feature pyramid P is constructed F = {P3, P4, P5, P6, P7}; Deconvolution upsamples the feature maps of different sizes in the feature pyramid to unify the feature pyramid into a feature set of the same size Using the weighted feature fusion strategy, as shown in formula (6), the feature set is fused into a fused feature map containing multi-scale information of the vehicle The formula of the weighted feature fusion strategy is as follows: in, w k is the weight of the feature map, which is set to 0.2 by default.

5. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 1, characterized in that: The step of inputting the vehicle annotated 3D detection data set into the target vehicle 3D detection network model for training to obtain vehicle type data further includes: Design a multi-task regression detection head, which includes four branches: vehicle type classification, vehicle center point regression, vehicle 3D box vertex regression and vehicle 3D size regression; A full convolution layer and an attention mechanism module are introduced into the vehicle type classification branch, and the vehicle center point regression, vehicle 3D box vertex regression and vehicle 3D size regression utilize the full convolution layer; The output of the vehicle type classification branch is defined as the existence probability of each type of vehicle at each pixel point, forming a vehicle type heat map Where class represents the number of vehicle types; The outputs of the remaining three regression branches are defined as the coordinate offsets of the center point of the vehicle's 3D box 3D box vertex coordinates and three-dimensional dimensions Normalize the coordinates of the vehicle's 3D center point and 3D box vertices.

6. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 1, characterized in that: The step of inputting the vehicle annotated 3D detection data set into the target vehicle 3D detection network model for training to obtain vehicle type data further includes: The loss function of the target vehicle three-dimensional detection network model consists of two parts: a basic loss function and an enhanced loss function; The basic loss function includes vehicle type classification, vehicle center point offset regression, vehicle 3D box vertex regression and vehicle 3D size regression loss; The enhanced loss function includes reprojection loss and vehicle IoU loss.

7. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 6, characterized in that: The basic loss function includes vehicle type classification, vehicle center point offset regression, vehicle 3D box vertex regression and vehicle 3D size regression loss, and also includes: Vehicle type classification loss. In order to optimize the problem of imbalanced positive and negative samples in the dataset, the vehicle type classification loss L class Use FocalLoss, as shown in formula (7): Among them, N represents the number of positive samples, α and β are hyperparameters for adjusting the loss weights of positive and negative samples, and h xyc Represents the use of Gaussian kernel function The actual vehicle target obtained is in the feature map H c The response value in is, σ is the standard deviation calculated from the actual size of the vehicle target in the image coordinate system The vehicle center point offset regression loss uses the L1 loss strategy, as shown in formula (8): Among them, E xy Indicates whether there is a vehicle target at x,y in the output feature map, d center Represents the real coordinates of the center point of the vehicle relative to the scaled input image size X,Y, Represents the true coordinates of the center point of the vehicle in the high-resolution feature map, Indicates the predicted vehicle center point offset; The vehicle 3D box vertex regression loss uses the L1 loss strategy, as shown in formula (9): Among them, d vertex Represents the real coordinates of the vehicle key points relative to the scaled input image size X,Y, Represents the predicted coordinates of the vehicle's key points; The three-dimensional regression loss of the vehicle is calculated using the L1 loss strategy, as shown in formula (10): in, A feature map representing the true 3D size of the vehicle, Feature map representing the 3D dimensions of vehicle prediction.

8. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 6, characterized in that: The enhanced loss function includes reprojection loss and vehicle IoU loss, and also includes: Reprojection loss, taking a single vehicle as an example, the image coordinates of the predicted and real vehicle 3D box vertices are and The corresponding world coordinate prediction value is The true value is The projected world and image coordinates of the predicted vehicle 3D box vertices are and The predicted vehicle 3D size is Already obtained on the network On the basis of Through coordinate transformation Then and A set of constraints is formed and the L1 loss strategy is used, as shown in formula (11): in, Represents the three-dimensional feature map of the vehicle obtained by using the camera calibration Cail and network prediction The calculated vehicle 3D frame reprojection vertex feature map, Represents the predicted vehicle 3D box vertex feature map relative to the original unscaled image size; Vehicle IoU loss, assuming that the minimum enclosing rectangles of the predicted and real vehicle 3D box vertices in the image are F forecast and F real , using the L1 loss strategy, as shown in formula (12): Among them, F xy and It represents the predicted value and true value of the minimum bounding rectangle calculated by the vertex feature map of the vehicle 3D box relative to the original unscaled image size, and IoU represents the calculation strategy of the intersection over union loss.

9. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 6, characterized in that: The loss function of the target vehicle three-dimensional detection network model consists of two parts: a basic loss function and an enhanced loss function, and also includes: Through the basic loss function and the enhanced loss function, the multi-task loss function can be defined as shown in formula (13): L=λ class L class +λ center L center +λ vertex L vertex +λ size L size +λ project L project +λ cariou L cariou (13); Among them, in order to balance the weight of each part of the loss, take λ class =1,λ center =1,λ vertex =1,λ size =1,λ project =1,λ cariou =1.

10. The method for detecting three-dimensional vehicle information from a monocular camera perspective based on deep learning as claimed in claim 1, characterized in that: The step of inputting the vehicle annotated 3D detection data set into the target vehicle 3D detection network model for training to obtain vehicle type data further includes: The vehicle annotated 3D detection dataset was scaled to 512×512 size during training, and the ratio of the training set to the validation set was set to 9:1; The training consists of 100 rounds in total. The backbone network is frozen for training in the first 60 rounds with the batch size set to 16, and then unfrozen for training in the last 40 rounds with the batch size set to 8. The Adam optimizer is used in the training process, and the learning rate is dynamically adjusted. The initial setting is 0.

001. The training is terminated when the verification loss tends to balance, and the vehicle type data is obtained.

Images