Monocular 3D vehicle detection method based on point density under roadside view angle

Abstract

The invention relates to the technical field of image data processing, in particular to a monocular 3D vehicle detection method based on point density under a roadside view angle, which comprises the following steps: acquiring a roadside view angle image; performing vehicle target detection on the roadside view angle image to obtain two-dimensional frame information; performing vehicle contour segmentation on the roadside view angle image to obtain vehicle contour pixel point coordinates; contour point density analysis is carried out based on the two-dimensional frame information and the vehicle contour pixel point coordinates, a three-dimensional frame is drawn, under the condition that errors are equal to those of other methods, the method does not need any prior data and complex network structures, a 3D target detection task can be completed by combining a traditional method with an existing deep learning model, and the method is simple and convenient to operate. The problem that an existing vehicle detection method is poor in detection effect is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a monocular 3D vehicle detection method based on point density from a roadside perspective. Background Art

[0002] With the rapid development of autonomous driving research, target detection, as one of the important contents of environmental perception, provides decision-making and planning information for autonomous driving vehicles. However, two-dimensional target detection can only provide target category and location information, and cannot provide information such as orientation angle, object size, and three-dimensional posture for environmental perception tasks. In practical applications, its detection effect is far from the requirements of the work task. Three-dimensional target detection introduces depth information in positioning and size regression. By processing the input data of different sensors, it outputs the category and precise location of the target object, thereby assisting the vehicle in obstacle avoidance and path planning. The necessity of three-dimensional target detection is self-evident.

[0003] Current research on 3D target detection relies on data from LiDAR or camera sensors. LiDAR can directly obtain 3D spatial data of the environment, but it has not been widely used in road systems due to its high cost, limited range, inability to describe target texture features, and poor performance in rainy and snowy weather. Depth cameras and binocular cameras can obtain depth maps and left and right images. Based on the depth image method, the image and depth map are fused with the neural network to generate depth perception features. Based on the binocular image method, the left and right parallax is used to calculate the depth, but the complexity of the detection system increases its implementation cost. In contrast, monocular 3D target detection has attracted widespread attention due to its advantages such as low hardware cost, low power consumption, and easy deployment in real scenes.

[0004] Existing monocular 3D target detection research relies on monocular images and camera parameters, but it is impossible to regress the 3D object size, orientation angle, stereo pose and other information from a single RGB image, and faces two major challenges: distance perception and target occlusion. Monocular 3D detection loses some scene depth information when projecting scene information onto a two-dimensional image plane, and due to the scale distance blur caused by perspective projection, the pixel error in the image space will be magnified in the 3D space, resulting in deviations in the object's pose information. To address this problem, some studies have used depth features to improve the accuracy of monocular 3D target detection, combining depth estimation, LiDAR guidance, direct regression and other methods to regress the target's three-dimensional geometric features. However, these methods require additional data as assistance and lack accurate depth information.

[0005] In addition, in the RGB images of vehicle-mounted perception, since the sensor is installed at a relatively low height, usually on the top of the vehicle, its perception range is limited and the target is easily obstructed, vehicle-mounted perception still faces safety challenges and uncontrolled threats. To address this problem, consider installing the camera on a pole a few meters above the ground on the roadside, using roadside sensors to perceive vehicle targets in the road scene, using their long-range, global perspective to cover the blind spots of the field of vision, spatially expand the vehicle's perception field, and broadcast the information of the roadside camera to all surrounding vehicles. In existing monocular 3D detection research, the main vehicle-mounted 3D perception datasets are KITTI3D, Nuscenes3D, etc., and there are relatively few studies on roadside perspectives.

[0006] In summary, the detection effect of existing vehicle detection methods is poor. Summary of the invention

[0007] The purpose of the present invention is to provide a monocular 3D vehicle detection method based on point density from a roadside perspective, aiming to solve the problem of poor detection effect of existing vehicle detection methods.

[0008] To achieve the above-mentioned purpose, the present invention provides a monocular 3D vehicle detection method based on point density under roadside perspective, comprising the following steps: acquiring a roadside perspective image; performing vehicle target detection on the roadside perspective image to obtain two-dimensional frame information; performing vehicle contour segmentation on the roadside perspective image to obtain vehicle contour pixel point coordinates; performing contour point density analysis based on the two-dimensional frame information and the vehicle contour pixel point coordinates to draw a three-dimensional frame.

[0009] Among them, the vehicle target detection is performed on the roadside perspective image to obtain two-dimensional frame information, including: constructing a Yolov7 model; constructing vehicle 3D data, and performing data cleaning and data enhancement on the vehicle 3D data to obtain a training data set; using the training data set to train the Yolov7 model to obtain a target detection model; inputting the roadside perspective image into the target detection model for training to obtain two-dimensional frame information.

[0010] The Yolov7 model consists of two parts: an aggregation network and a connection-based model scaling.

[0011] Among them, the vehicle contour segmentation of the roadside perspective image to obtain the vehicle contour pixel point coordinates includes: constructing a Bisenetv2 model; using the training data set to train the Bisenetv2 model to obtain a contour segmentation model; inputting the roadside perspective image into the contour segmentation model for training to obtain the vehicle contour pixel point coordinates.

[0012] The Bisenetv2 model consists of two main branches, namely the detail branch and the semantic branch, and a guided aggregation layer is used to enhance the feature representation of the interconnected and fused two branch types.

[0013] Among them, the contour point density analysis is performed based on the two-dimensional frame information and the vehicle contour pixel point coordinates to draw a three-dimensional frame, including: using the HSV color channel to separate and extract the horizontal and vertical coordinates of the vehicle contour pixel point coordinates; matching the two-dimensional frame information and the horizontal and vertical coordinates of the contour pixel points for clustering, and identifying all contour points under the vehicle ID; drawing the horizontal and vertical coordinate distribution histogram and density map of the contour points, providing horizontal and vertical size references, inferring the vehicle orientation and finding the reference point according to the number of point density distribution, and drawing the three-dimensional frame of the vehicle based on this.

[0014] The present invention provides a monocular 3D vehicle detection method based on point density under a roadside perspective, which obtains a roadside perspective image; performs vehicle target detection on the roadside perspective image to obtain two-dimensional frame information; performs vehicle contour segmentation on the roadside perspective image to obtain vehicle contour pixel point coordinates; performs contour point density analysis based on the two-dimensional frame information and the vehicle contour pixel point coordinates to draw a three-dimensional frame. When the error is comparable to that of other methods, this method does not require any prior data and complex network structure, and can complete the 3D target detection task by using traditional methods combined with existing deep learning models, thereby solving the problem of poor detection effect of existing vehicle detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a schematic diagram of a monocular 3D vehicle detection method based on point density from a roadside perspective provided by the present invention.

[0017] Figure 2 , Figure 3 , Figure 4 It is a density analysis diagram of vehicle contour points in different forms.

[0018] Figure 5 is a fixed 3D corner map.

[0019] Figure 6 It is the detection result of Deep3DBox.

[0020] Figure 7is the test result of VWH.

[0021] Figure 8 is the detection result of this method.

[0022] Fig. 9 It is the evaluation index error analysis of Deep3DBox.

[0023] Fig.10 It is the error analysis of the evaluation index of VWH.

[0024] Fig.11 It is the evaluation index error analysis of this method.

[0025] Fig.12 This is a flow chart of a monocular 3D vehicle detection method based on point density from a roadside perspective provided by the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0027] See also Figures 1 to 12 The present invention provides a monocular 3D vehicle detection method based on point density under a roadside perspective, comprising the following steps: S1 obtains a roadside perspective image.

[0028] S2 performs vehicle target detection on the roadside perspective image to obtain two-dimensional frame information.

[0029] Specifically, a Yolov7 model is constructed; vehicle 3D data is constructed, and the vehicle 3D data is cleaned and enhanced to obtain a training data set; the Yolov7 model is trained using the training data set to obtain a target detection model; the roadside view image is input into the target detection model for training to obtain two-dimensional frame information.

[0030] The Yolov7 model consists of two parts: an aggregation network and a connection-based model scaling. Based on ELAN, the transition layer architecture remains unchanged and only the architecture of the computing module is changed. E-ELAN is proposed, which combines different features in a variety of ways such as expansion, shuffling, and merging to enhance the network learning ability. The composite model scaling method of the cascade model is adopted. When the depth factor of the computing block is scaled, the output channel of the computing block changes, and the width factor scaling is performed on the transition layer with the same change amount to maintain the properties and optimal structure of the model.

[0031] Yolov7 has made great efforts to improve target detection accuracy and model performance. Convolutional reparameterization uses the gradient flow propagation path to combine different networks. Auxiliary training uses a deep supervised target detector architecture, adding additional auxiliary heads and shallow network weights guided by auxiliary losses in the middle layer of the network. In addition, batch normalization using a combined structure is directly connected to the convolution layer to integrate the mean and variance of batch normalization into the bias and weight of the convolution layer during the inference phase; implicit knowledge in Yolo is combined with convolutional feature maps and addition and multiplication methods, and simplified to a vector through pre-calculation during the inference phase, which can be combined with the bias and weight of the previous and next convolutional layers; EMA is used as the final inference model and other training techniques to improve model performance.

[0032] S3 performs vehicle contour segmentation on the roadside perspective image to obtain the pixel coordinates of the vehicle contour.

[0033] Specifically, a Bisenetv2 model is constructed; the Bisenetv2 model is trained using the training data set to obtain a contour segmentation model; the roadside view image is input into the contour segmentation model for training to obtain the pixel coordinates of the vehicle contour.

[0034] The Bisenetv2 model consists of two main branches, namely the detail branch and the semantic branch, and the guided aggregation layer is used to enhance the feature representation of the interconnected and fused two branch types. The detail branch uses the VGG network to stack each layer. The output feature map extracted by this branch is 1 / 8 of the original input. It captures low-level details through wide channels and shallow layers and generates high-resolution feature representations; the semantic branch instantiation includes Stem Block, Context Embedding Block, Gather-and-Expansion Layer, which performs downsampling, global average pooling and residual connection to embed context information, aggregation and expansion, and obtains high-level semantic context information through narrow channels and deep layers. Finally, bilateral guided aggregation is used to fuse the complementary information of the two branches, and the context information of the semantic branch is used to guide the feature response of the detail branch. Different scale feature representations are captured under different scale guidance to encode multi-scale information.

[0035] S4 performs contour point density analysis based on the two-dimensional frame information and the vehicle contour pixel point coordinates to draw a three-dimensional frame.

[0036] Specifically, when there are occlusions, overlaps, etc. between the detected objects, or when a small remote target is mapped to a very small pixel area, the target detection model performs poorly. The segmentation model distinguishes different object instances at the pixel level and can provide more accurate object boundary information. Therefore, this paper combines the segmentation model with the detection model, aiming to use the pixel-level granularity of the vehicle contour and the target detection category identification for distinction and analysis. First, the vehicle two-dimensional frame information is obtained according to the Yolov7 model, and the vehicle contour pixel coordinates are obtained according to the Bisenetv2 model. The hsv color channel is used to separate and extract the horizontal and vertical coordinates of the vehicle contour pixel coordinates; the two-dimensional frame information and the horizontal and vertical coordinates of the contour pixel are matched for clustering, and all contour points under the vehicle ID are identified; the horizontal and vertical coordinate distribution histograms and density maps of the contour points are drawn to provide horizontal and vertical size references, and the vehicle orientation is inferred and the reference point is found according to the number of point density distribution, and the three-dimensional frame of the vehicle is drawn based on this.

[0037] (1) Vehicle contour point segmentation: The input of this paper is an RGB image. The input image is scaled to 1024×512i, m0 represents the scaled image, and then the Yolov7 model is loaded for target detection. The detection result of the current image frame is obtained, boxes = {boxi}, which is defined as

[0038] box i =[left i ,top i ,right i ,bottom i ,conf i ,dclass i ] (1)

[0039] Among them, i∈n, n represents the number of all detected vehicle IDs in the current frame, (lefti,topi) and (righti,bottomi) represent the pixel coordinates of the top left and bottom right of the ith detected target, confi represents the confidence of the ith detected target, and dclassi represents the detection category of the ith detected target. Then, based on the Bisenetv2 model, the category segmentation is performed, and the vehicle contour pixels are extracted and ANDed to obtain an image with a normal background color and a black vehicle contour. The horizontal and vertical coordinates xy of the black pixel are extracted and defined as

[0040] xy={p j} (2)

[0041] Where j∈m, m represents the number of all contour points in the current frame, matching the two-dimensional detection box boxi=[lefti,topi,righti,bottomi] and the vehicle contour pixel pj=(pxj,pyj). When pxj≥lefti&pxj≤righti&pyj≥topi&pyj≤bottomi, it means that the pixel pj falls within the two-dimensional box boxi. The contour points falling within the two-dimensional box boxi belong to the same vehicle ID, which is defined as

[0042] car i ={[xlist i ,ylist i ]} (3)

[0043] Among them, cari represents the vehicle ID of the current frame, and xlisti and ylisti represent the horizontal and vertical coordinate lists under the vehicle ID. This can realize the division of the vehicle contour points of the current frame, and then perform density analysis under the specific vehicle ID.

[0044] (2) Vehicle body 2D information reference: Define boxesBisenet = {bboxi}, where bboxi = [blefti, btopi, brighti, bbottomi] represents the 2D box composed of vehicle contour points under the vehicle ID, where

[0045] bleft i =min(xlist i ) (4)

[0046] btop i =min(ylist i ) (5)

[0047] bright i =max(xlist i ) (6)

[0048] bbottom i =max(ylist i ) (7)

[0049] The horizontal coordinate xlisti of the vehicle outline pixel point is used as the histogram to draw the data, and righti-lefti is used as the number of groups.

[0050] xn i ,xbin i ,xp i =plt.hist(xlist i ,bright i -bleft i ) (8)

[0051] Among them, xni represents the number of elements in the horizontal coordinate histogram group of the vehicle outline pixel point, xbini represents the horizontal coordinate interval range, xpi represents the data contained in each interval range, and mxni=max(xni) represents the coordinate value with the highest horizontal coordinate density of the vehicle ID. Within the range of 10 pixels around the highest value, the critical points bleftxi and brightxi where the horizontal coordinate rises and falls sharply are obtained. Then the reference size expression of the horizontal body two-dimensional information is:

[0052] x i =max(|brightx i -bleft i |,|bright i -bleftx i |) (9)

[0053] For ordinate processing, since the density distribution of ordinate contour points is generally not much different and there is no significant rise or fall, the ordinate mean ymeani = mean(ylisti) is taken, and the reference size expression of the two-dimensional information in the vertical direction is:

[0054] yh i =max(|ymean i -btop i |,|bbottom i -ymean i |) (10)

[0055] In a single vehicle ID area, the vehicle contour point density analysis is as follows: Figure 2 As shown, Figure 1 represents the statistical number of pixel points in the horizontal coordinate of the vehicle contour point, the horizontal axis represents the horizontal coordinate value of the contour point, and the vertical axis represents the number corresponding to the horizontal coordinate value of the contour point; Figure 2 represents the horizontal and vertical coordinate density map and histogram, the top is the density map, the X-Density horizontal axis represents the horizontal coordinate value of the contour point, the X-Density vertical axis represents the density of the number corresponding to the horizontal coordinate value of the contour point, the Y-Density horizontal axis represents the vertical coordinate value of the contour point, and the Y-Density vertical axis represents the density of the number corresponding to the vertical coordinate value of the contour point; the bottom is the histogram, the X-Hist horizontal axis represents the horizontal coordinate value of the contour point, the X-Hist vertical axis represents the number corresponding to the horizontal coordinate value of the contour point in a small range, the Y-Hist horizontal axis represents the vertical coordinate value of the contour point, and the Y-Hist vertical axis represents the number corresponding to the vertical coordinate value of the contour point in a small range; Figure 3 represents the horizontal and vertical coordinate distribution map, the horizontal axis represents the horizontal coordinate value of the contour point, and the vertical axis represents the vertical coordinate value of the contour point, which approximately shows the overall contour characteristics of the vehicle.

[0056] (3) 3D frame drawing: For vehicle orientation and reference point selection, this method does not provide information such as the front and rear of the vehicle, yaw angle, pitch angle, etc., and cannot be considered based on the vehicle's driving direction and global direction. The vehicle orientation is only divided at the 2D pixel level, and the 3D frame corners are fixed according to the distance from the camera. The 3D corner point diagram is shown in Figure 3. Under the vehicle ID, the number of pixel coordinate values ​​in the horizontal coordinate histogram of the two intervals [blefti, bleftxi] and (brightxi, brighti], totalLefti and totalRighti, are counted respectively. If totalRighti>totalLefti, it means that the vehicle orientation is as follows Figure 3 As shown in the left figure, the upper left corner is set as the reference point, and the pixel expression of the octagonal point of the three-dimensional frame is shown in Table 1; otherwise, it is expressed as Figure 3 As shown in the right figure, the upper right corner is set as the reference point, and the pixel expression of the octagonal point of the three-dimensional frame is shown in Table 2.

[0057] Table 1 Pixel coordinates of the octagonal points of the three-dimensional frame ( Figure 3 Left side facing)

[0058]

[0059] Table 2 Pixel coordinates of the octagonal points of the three-dimensional frame ( Figure 3 Right side facing)

[0060]

[0061] Defining des = {deti} means drawing all 3D frame information in the current frame, deti means the 3D frame information under a specific vehicle ID, which is defined as

[0062] det i =[e ij ,bleft i ,btop i ,bbottom i ,conf i ,dclass i ] (11)

[0063] Where eij represents the coordinates of the octagonal point corresponding to the vehicle cari, that is, an element consisting of abscissa u and ordinate v, e = (u, v), j∈{1,2,…,8}.

[0064] Return [im0,des,boxes,boxesBisenet] as the output result of this method for the current frame. When drawing the 3D box, traverse the detection result des in the current frame and extract

[0065] boxes3D i =[e ij ] (12)

[0066] boxes3Di is the two-dimensional projection pixel coordinates of the three-dimensional box under the i-th vehicle ID, according to Figure 3 By connecting the pixel coordinates of the octagonal points in sequence as shown, a three-dimensional frame can be drawn.

[0067] In addition, the vehicle 3D datasets used in this paper are self-made datasets and Rope3D datasets. The self-made datasets are used for prior model training, and the Rope3D datasets are used to evaluate the experimental results. In training the target detection and contour segmentation network models, data cleaning, data enhancement, and other methods are used to effectively screen and supplement data scenarios, making the dataset preparation work of the prior model more sufficient. Dataset scenario

[0068] The self-made dataset is collected from a variety of intersection scenes based on the roadside camera perspective. There are 9771 pictures in total with an image resolution of 1920 pixels * 1080 pixels. The labelImg annotation tool is used to manually annotate the detection category two-dimensional box, and the labelme annotation tool is used to manually annotate the detection category contour to generate the corresponding label data file for training the target detection model and contour segmentation model.

[0069] The Rope3D dataset is aimed at 3D perception tasks from the roadside perspective. It is the first large-scale, multi-perspective roadside dataset with both image and point cloud 3D joint annotations. It has a total of 50,009 frames of image data and corresponding 2D & 3D annotation results, with 26 different scenes, covering different environmental conditions such as sunny days, rainy days, foggy days, daytime, and nighttime. Based on this dataset, this paper randomly selects 529 images and corresponding label files for vehicle 3D box error evaluation.

[0070] Data cleaning and data enhancement: For the Rope3D dataset, data cleaning mainly involves deleting the records in the label file label that are missing three-dimensional information due to the target being far away. For the self-made dataset, data cleaning mainly includes two stages, one is after data collection, and the other is after data annotation. After data collection, through manual screening and visual review, pictures that do not conform to the roadside perspective scene are screened out, pictures with few vehicles to be detected, pictures that are blocked, and pictures with severe truncation are screened out, and pictures where the road vehicle information cannot be seen by the naked eye due to bad weather such as dusk and fog are screened out; after data annotation, the script is used to screen and control the training set, and pictures that do not contain or contain less of the detection target category in the road information picture are screened out, and pictures whose categories do not meet the annotation requirements and whose effects do not meet the annotation specifications are screened out and re-annotated. Considering that it is difficult to annotate vehicle targets when the traffic volume is large, and because of the large errors caused by manual annotation, the targets that are far away and cannot be clearly annotated due to dense vehicles are blocked, and the image is masked with a color similar to the road surface.

[0071] Considering that the actual road scenes are complex and changeable, and the scene display effects are different due to weather and time, the limited scenes in the data set cannot simulate all roadside perspective scenes, so data enhancement is performed to expand the number of data set samples to prevent network overfitting. Considering the poor recognition effect of remote small targets, the two-dimensional frame vehicle targets with a horizontal width pixel difference of less than 24 and a vertical height pixel difference of less than 16 are defined as small targets. The small targets in the data set are extracted and spliced ​​into multiple images, and the corresponding label files are generated according to the width and height of the extracted small target images. Data enhancement is performed during the training process, using the commonly used methods such as Blur blur, VerticalFlip horizontal flip, Resize zoom, HueSaturationValue hue saturation value in the Albumentations data enhancement library, and the values ​​in the label files also change accordingly.

[0072] 3. Experimental results and analysis: The Python version of this method is 3.7, the pytorch version is 1.7.1, and the preliminary model is prepared to train on Xavier and test the vehicle contour segmentation and vehicle target detection model effects. The experimental configuration GRID driver version is 470.82.01, CUDA version is 11.4, and the GPU type is Persistence-M driver resident mode. The entire code and experiment are deployed and run on the laboratory server, and the experimental configuration is as follows.

[0073] Table 3 Experimental environment configuration

[0074]

[0075] 1. Prior model.

[0076] This algorithm combines two methods: vehicle target detection and vehicle contour segmentation. The detection accuracy of the two models has a decisive influence on the experimental effect of this method. First, the two models are trained based on a self-made data set as a preliminary model preparation.

[0077] (1) Vehicle target detection model: The 9771 dataset images are divided into training set and validation set in the ratio of 8:2. In the training process of the Yolov7 network model for vehicle target detection, the residual network is set to resnet101, and the detection types include seven types: car, truck, two-wheeled vehicle, person, bus, license plate, and tricycle. The initialization range of the three anchor box sizes is [12, 16, 19, 36, 40, 28], [36, 75, 76, 55, 72, 146], and [142, 110, 192, 243, 459, 401], respectively. The training iteration round is 300, and the input image cropping and scaling size is [640, 640]. The other network model default parameters are kept unchanged, and 8 images are sent to the network for training each time.

[0078] In order to improve the accuracy of the target detection model, data cleaning, small target splicing, data enhancement and other methods are used, and the model parameters are continuously adjusted. The model effect is evaluated by precision rate P, recall rate R, F1 score, precision rate and recall rate relationship PR curve. The results are shown in Table 4. "√" in the table means that this part is adopted, and if it is not checked, it means that this part is not adopted. The target detection model with the best evaluation effect is used as the preliminary model preparation for the subsequent methods.

[0079] Table 4 Experimental evaluation of vehicle target detection model

[0080]

[0081] (2) Vehicle contour segmentation model: 9771 dataset images are divided into training set and validation set in the ratio of 8:2. In the training process of the vehicle contour segmentation network model, the network type is set to bisenetv2, and the dataset annotation types include 8 categories: car, truck, two-wheeled vehicle, person, bus, license plate, tricycle, and road. The number of initial attention heads is 4, the initial learning rate is 5e-3, the weight decay is 5e-4, the step threshold is 1000, the maximum number of iterations is 150000, the feature layer pixel point is equivalent to the original input image pixel point range size [0.25, 2], the input image cropping scale size is [512, 1024], and 8 images are sent to the network for training each time. Its model accuracy is 83%.

[0082] 2. Vehicle 3D perception evaluation.

[0083] (1) Evaluation indicators: The open source 3D datasets for autonomous driving include KITTI3D dataset, Nuscenes3D dataset, Rope3D dataset, etc. KITTI3D and Nuscenes3D are from the vehicle perspective, while Rope3D is from the roadside perspective. The evaluation indicators of KITTI3D are the accuracy of 2D detection box bbox, the accuracy of detection box under BEV view bev, the accuracy of 3D detection box, and the accuracy of target rotation angle detected by aos; the evaluation indicators of Nuscenes3D are the distance mAP of the center point of the 2D box under BEV view, the 2D Euclidean distance mATE of the center point, the scale error mASE after the center point and orientation are aligned, the orientation error mAOE, the speed error mAVE, the attribute error mAAE, and the weighted average NDS of the above errors; the evaluation indicators of Rope3D are the average precision mAP, the average ground center point similarity ACS, the average orientation angle similarity AOS, the average area similarity AAS, and the average bottom four point similarity AGS.

[0084] This algorithm uses the 2D pixel coordinates of the octagonal points to draw a 3D frame. Its output does not have intuitive 3D orientation angle values, 3D dimensions, depth and other data information, so the evaluation index is only considered in the pixel coordinate system. When evaluating the experimental results, the real frame data of the Rope3D dataset is converted according to the set input format and compared with the output data of the algorithm. Inspired by the evaluation indicators of the above datasets, this paper sets the evaluation indicators as A3DS, A2DS, AGS, ALS, APS, AH, AW, and AL.

[0085] A3DS represents the average 3D box similarity error,

[0086]

[0087] Where D represents the set of true positive samples, |D| represents the total number of true positive objects, K=16 is the number of horizontal and vertical coordinates of the eight corner points of the three-dimensional box, sg and Respectively represent the true value and predicted value of the coordinates of the boundary points of the 3D box, represents the 3D ground-truth box norm.

[0088] A2DS represents the average two-dimensional box similarity error,

[0089]

[0090] Where D represents the set of true positive samples, |D| represents the total number of true positive objects, K=4 represents the number of horizontal and vertical coordinates of the upper left and lower right points of the two-dimensional box, sg and Respectively represent the true value and predicted value of the coordinates of the boundary points of the two-dimensional box, represents the 2D ground-truth box norm.

[0091] Considering that three mutually perpendicular planes can determine the three-dimensional information of a cube, the average four-point similarity error of three views is used as the evaluation index. AGS means average top view Figure 4 Point similarity error, ALS means average left view Figure 4 Point similarity error, APS means average primary view Figure 4 Point similarity error, AGS, ALS, APS formula expression is the same as A2DS, D represents the true positive sample set, |D| represents the total number of true positive objects, K = 4 is the top view, left view, front view Figure 2 The number of horizontal and vertical coordinates of the upper left and lower right points of the dimension box, sg and Respectively represent top view, left view, and front view Figure 2 The true and predicted values ​​of the coordinates of the boundary points of the dimension box, represents the 2D ground-truth box norm.

[0092] The length, width and height are important parameter information of the 3D frame. In this paper, the distance between the 2D projection point coordinates of the 3D frame corner points is used to measure the length, width and height information under the 2D pixels. AH represents the average height similarity error, AW represents the average width similarity error, and AL represents the average length similarity error.

[0093]

[0094] Where D represents the set of true positive samples, |D| represents the total number of true positive objects, Δsc represents the Euclidean distance between the predicted value and the true value, and Cs represents the norm of the height true value. The formula expressions of AW and AL are the same as those of AH and will not be repeated here.

[0095] (2) Experimental comparative analysis: This paper sets the IoU threshold of the 2D projection of the 3D detection box and the 2D detection box to 0.5, and tests it on the Rope3D dataset. Based on the evaluation indicators in 4.2.1, the corresponding experimental results are given and compared with the Deep3DBox, BEVHeight, VWH, VLH, and VWL methods. The experimental results are shown in Table 5, where FPS represents the rate of the method. The total average method is used, and the global frame number is divided by the global time. The unit is frame / second.

[0096] Table 5 Experimental comparison method error and rate (IoU ≥ 0.5)

[0097]

[0098] The evaluation indicators are measured from three dimensions: volume, surface, and value. A3DS is the three-dimensional volume similarity error, A2DS is the two-dimensional projection similarity error, AGS, ALS, and APS are the three-view similarity errors, and AH, AW, and AL are the size value errors. Their error fluctuations are more sensitive than those of volume and surface errors, and the mean value of the experimental parameters fluctuates greatly compared to the actual vehicle size, so the overall value is biased high.

[0099] Deep3DBox and BEVHeight are methods based on deep features, while VWH, VLH, and VWL are methods based on camera self-calibration and vanishing points. Deep3DBox and BEVHeight have known camera internal and external parameters, corner world coordinates, actual size, bottom plane equation, etc., and their error sources are mainly model training accuracy, size mean and actual vehicle size error. VWH, VLH, and VWL have known camera internal and external parameters, camera installation height, actual size, etc., and their detection errors mainly come from vanishing point position error, size mean and actual vehicle size error. This method is based on pure vision processing, and the input data is only RGB image, and no other prior data is required.

[0100] From the experimental results, we can see that when the IoU threshold is 0.5, the A3DS error on the Rope3D test set is 9.3%, the A2DS error is 1.9%, the average error of the three views is 16.2%, and the average error of the length, width and height is 36.3%. The large error of the length, width and height is because the comparison method knows the mean value of the vehicle size, while this method does not. Compared with Deep3DBox, BEVHeight, VWH, VLH, and VWL methods, this method does not require any prior data and complex network structure, and can achieve better performance based only on pure image data, while the error is the same as other methods.

[0101] (3) Visualization result analysis: In order to more comprehensively evaluate the effectiveness of this method, this paper visualizes some detection results in the Rope3D dataset and compares the method in this paper with the 3D detection boxes drawn by Deep3DBox and VWH. Figure 3 As shown. By comparing the detection results in the image, it can be seen that Deep3DBox fails to show a good pitch angle relationship for near-end objects, and has poor performance in detecting the direction of far-end objects; VWH has poor detection effect on the size information of near-end objects, and can detect small targets with dense far-end targets, but the drawn three-dimensional box will be severely occluded and will cause false detection; this method has good performance in detecting the direction and size of near-end objects, and has obvious advantages over the above two methods, which shows the effectiveness of the contour point density analysis method proposed in this paper, while the performance of far-end target drawing is poor, which is mainly affected by the accuracy of the vehicle contour segmentation model and the vehicle target detection model.

[0102] At the same time, the evaluation index errors of the three methods are visualized, such as Figure 4 As shown in the figure, the overall error fluctuation and the specific error of each detection category can be observed more intuitively. The horizontal axis in the figure represents the number of all three-dimensional boxes drawn by the method, and the vertical axis represents the error value of each evaluation index. From the experimental results, it can be seen that the total number of targets detected by this method is less than that of Deep3DBox and VWH. The main reason is that the two-dimensional box of the remote target is offset, and the contour points do not match the two-dimensional box well, resulting in partial data loss. This indirectly verifies the influence of the accuracy of the vehicle contour segmentation model and the vehicle target detection model on this method.

[0103] Conclusion: As a key technology in the field of autonomous driving, three-dimensional object detection provides information such as orientation angle, object size, and stereoscopic posture for environmental perception tasks, and has attracted much attention from academia and industry. This paper focuses on monocular 3D vehicle detection from the roadside perspective, extracts two-dimensional features based on the prior models of object detection and object segmentation, and processes 3D tasks based on traditional methods. The main research results of this paper are as follows:

[0104] This method does not rely on any additional data, only uses the RGB image from the roadside perspective to obtain the 3D morphological information of the vehicle target. Compared with the monocular 3D method based on depth features, camera self-calibration and vanishing points, this method frees the heavy reliance on prior data such as perception data and sensor parameter data, and does not require the design of a complex network model structure.

[0105] This method uses the roadside perspective to perform 3D perception of vehicle targets. Compared with the limited perception range and easy occlusion of targets under the vehicle-mounted perspective, it can achieve beyond-line-of-sight and cross-space perception of vehicle 3D morphological parameters in traffic scenes, expand the vehicle perception field and broadcast the information of the roadside camera to all surrounding vehicles, ensuring safety through cost perception and cost-effectiveness.

[0106] From the experimental results and analysis, it can be seen that this method does not require camera internal and external parameters, corner world coordinates, actual size, bottom plane equation and other information when ensuring that the evaluation index is within the allowable error range, compared with Deep3DBox and BEVHeight, and does not require camera internal and external parameters, camera installation height, actual size and other information compared with VWH, VLH, and VWL. The traditional method is adopted, combining the target detection model and the contour segmentation model, taking the image contour as the basic feature, and using two-dimensional features and statistical analysis to process the 3D detection task. In order to avoid the a priori model error being amplified in this method, data cleaning, small target splicing, data enhancement, and adjusting the anchor box size and model parameters are used to improve the detection accuracy of the Yolov7 model. At the same time, this method also has certain shortcomings. It uses low-level two-dimensional image features and statistical analysis to process, and its three-dimensional understanding ability of the scene is limited; it draws a pixel-level three-dimensional box, which does not interact with the physical size of the real world; it only considers vehicle targets, and does not perceive other traffic participants such as pedestrians and non-motor vehicles. Subsequently, traffic targets will be expanded and deep learning technology will be introduced to achieve real-time perception of detection targets from a roadside perspective.

[0107] What is disclosed above is only a preferred embodiment of a monocular 3D vehicle detection method based on point density from a roadside perspective of the present invention. Of course, this cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A monocular 3D vehicle detection method based on point density from a roadside perspective, characterized in that: The following steps are involved: Acquire roadside perspective images; Performing vehicle target detection on the roadside perspective image to obtain two-dimensional frame information; Segmenting the vehicle contour on the roadside perspective image to obtain the pixel coordinates of the vehicle contour; Based on the two-dimensional frame information and the vehicle outline pixel point coordinates, a contour point density analysis is performed to draw a three-dimensional frame.

2. The monocular 3D vehicle detection method based on point density from a roadside perspective as claimed in claim 1, characterized in that: The performing vehicle target detection on the roadside perspective image to obtain two-dimensional frame information includes: Build the Yolov7 model; Constructing vehicle 3D data, and performing data cleaning and data enhancement on the vehicle 3D data to obtain a training data set; Using the training data set to train the Yolov7 model to obtain a target detection model; The roadside perspective image is input into the target detection model for training to obtain two-dimensional frame information.

3. The monocular 3D vehicle detection method based on point density from a roadside perspective as claimed in claim 2, characterized in that: The Yolov7 model consists of two parts: an aggregation network and a connection-based model scaling.

4. The monocular 3D vehicle detection method based on point density from a roadside perspective as claimed in claim 3, characterized in that: The step of segmenting the vehicle contour on the roadside perspective image to obtain the pixel coordinates of the vehicle contour includes: Build the Bisenetv2 model; The Bisenetv2 model is trained using the training data set to obtain a contour segmentation model; The roadside perspective image is input into the contour segmentation model for training to obtain the pixel coordinates of the vehicle contour.

5. The monocular 3D vehicle detection method based on point density from a roadside perspective as claimed in claim 4, characterized in that: The Bisenetv2 model consists of two main branches, namely the detail branch and the semantic branch, and a guided aggregation layer is used to enhance the interconnected and fused feature representations of the two branch types.

6. The monocular 3D vehicle detection method based on point density from a roadside perspective as claimed in claim 5, characterized in that: The performing contour point density analysis based on the two-dimensional frame information and the vehicle contour pixel point coordinates to draw a three-dimensional frame includes: Using the hsv color channel to separate and extract the horizontal and vertical coordinates of the pixel coordinates of the vehicle outline; Match the two-dimensional frame information and the horizontal and vertical coordinates of the contour pixel points to perform clustering, and identify all contour points under the vehicle ID; Draw the horizontal and vertical coordinate distribution histogram and density map of the contour points to provide horizontal and vertical dimension references, infer the vehicle direction and find the reference point based on the number of point density distribution, and draw the three-dimensional frame of the vehicle based on this.

Images