Monocular vision-based vehicle 3D localization method, device, vehicle, and storage medium

Abstract

This invention provides a vehicle 3D localization method, device, vehicle, and storage medium based on monocular vision. It acquires images in real-time from a monocular camera, processes them using a multi-task recognition model to obtain first target information, acquires parameter information corresponding to the monocular camera, and calculates second target information of the target vehicle based on the first target data. Finally, it tracks the target vehicle based on the second target information. Even in environments with limited texture, it can obtain camera poses with high accuracy, thereby achieving high-accuracy vehicle poses. This reduces the impact of environmental texture variations on vehicle localization, significantly improving the robustness of the vehicle localization system. By using the same scale feature layer for vehicle sideline detection and vehicle detection, each feature point corresponds to a set of detection results, eliminating the need for subsequent matching and improving target tracking efficiency.

Description

Technical Field

[0001] This invention relates to the field of vehicle position detection and tracking technology, and in particular to a vehicle 3D positioning method, device, vehicle, and storage medium based on monocular vision. Background Technology

[0002] In ADAS (Advanced Driving Assistance System) monocular vision, accurate detection of vehicles ahead is an essential function of the ADAS system, and a prerequisite for functions such as path planning, motion prediction, and collision avoidance. However, 2D (two-dimensional) vehicle detection can no longer meet the needs of further improving the intelligence of ADAS systems. Therefore, it is necessary to study 3D (three-dimensional) bounding box detection technology for vehicles ahead in monocular vision.

[0003] Using vehicle 3D information as training data, a regression algorithm is used to obtain the vehicle's 3D shape. The single RGB image 3D object detection method, described in "Monocular 3D Object Detection and BOX Fitting trained end-to-end using intersecton over union loss," takes a single RGB image as input and directly passes it through a CNN (Convolutional Neural Network) to output the predicted object category, 2D bounding box position, object distance, object skew angle, 3D bounding box dimensions (length, width, height), and the coordinates of the eight vertices of the 3D bounding box projected onto the 2D bounding box. Non-Maximum Suppression (NMS) is used to extract the best 2D bounding box. 3D BOX Fitting transforms this into three pieces of information: object category, 2D bounding box, and 3D bounding box, each corresponding one-to-one with the labeled ground truth (real, valid values). The IOU loss is then optimized for network regression training. This method requires labeled data to have ground truth information such as length, width, height, and orientation.

[0004] However, in reality, training deep learning methods to obtain target 3D bounding boxes requires a large amount of ground truth data. If the data is insufficient, the trained model often overfits and exhibits weak generalization ability. In typical road conditions, with many vehicles ahead, it's difficult to obtain the actual physical dimensions and orientation of vehicles ahead using only a monocular camera without LiDAR data fusion. LiDAR is expensive, and requires joint calibration with the monocular camera, followed by further data fusion to obtain the relevant physical dimensions and orientation information of the vehicles ahead. Therefore, acquiring 3D data of the vehicles ahead is difficult and insufficient for training. Training deep learning methods to obtain target 3D bounding boxes requires a large amount of hard-to-obtain 3D data of the vehicles ahead. Furthermore, calibrated 3D data has poor universality; changing the camera (intrinsic parameter changes) or having deviations in installation position (extrinsic parameter changes) severely impacts the model's performance. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a vehicle 3D localization method, device, vehicle, and storage medium based on monocular vision. It employs a monocular vehicle 3D target detection and tracking method based on multi-task collaboration using convolutional neural networks. This method integrates vehicle body keypoint regression, side body line angle regression, vehicle posture classification, target detection, and feature vector generation. It allows for accurate acquisition of the target vehicle's 3D bounding box even with a certain degree of occlusion, exhibits strong anti-interference capabilities, and can effectively detect and track target vehicles in 3D.

[0006] Specifically, the present invention provides a vehicle 3D detection and tracking method based on monocular vision, comprising the following steps:

[0007] S1: Acquire real-time images from a monocular camera, the real-time images including real-time images of at least one target vehicle in front of the vehicle captured by the monocular camera of the vehicle.

[0008] S2: Based on the real-time acquired image, the first target information is obtained by processing it through a multi-task recognition model. The first target information includes the 2D full-body frame of the target vehicle, the target vehicle feature vector, the target vehicle body posture, and the target vehicle body side line information.

[0009] S3: Obtain the parameter information corresponding to the monocular camera, and calculate the second target information of the target vehicle in combination with the first target data. The second target information includes the 3D full-body frame of the target vehicle.

[0010] S4: Track the target vehicle based on the second target information.

[0011] In step S2, the first target information is obtained by processing it through a multi-task recognition model, specifically including:

[0012] S21: The real-time acquired images are preprocessed using a convolutional neural network model to obtain a heat map.

[0013] S22: Select feature points in the heatmap that are above a preset threshold as target feature data, obtain the vehicle type parameters corresponding to the feature points, and calculate the 2D detection box classification list according to the CenterNet target detection algorithm.

[0014] S23: For each 2D detection box, output the vehicle body pose information at the corresponding position, and obtain the pose of the target vehicle through the argmax operator; if the pose is Front_Side or Rear_Side, perform side line detection to obtain the coordinates of the side line endpoints A and B; otherwise, return to S22 until all target vehicle detection boxes have been processed.

[0015] The preprocessing in step S21 specifically includes:

[0016] The image size at the input end of the convolutional neural network model is set according to the camera resolution ratio and network structure parameters for the real-time acquired images.

[0017] Based on the image size ratio at the input of the convolutional neural network model, the multi-layer feature nodes are normalized; ghost features are obtained by performing linear transformation of image features using a ghost module, and then the weighted values ​​of ghost features at different scales are fused using the BiFPN feature fusion algorithm to output multi-layer feature nodes; the multi-layer feature nodes are normalized to obtain a heatmap.

[0018] Step S3, which calculates the second target information of the target vehicle, specifically includes:

[0019] S31: Calibrate the camera's internal and external parameters respectively, and establish the image coordinates to the ground plane vehicle coordinate system with the camera projection as the origin.

[0020] S32: Select the bottom endpoint and midpoint of the 2D detection frame, and obtain the coordinate mapping of the endpoint and midpoint of the target vehicle detection frame in the ground plane vehicle coordinate system.

[0021] S33: Calculate the target vehicle's 3D coordinate system parameters in the ground plane vehicle coordinate system. The 3D coordinate system parameters include at least the vehicle length, vehicle width, vehicle height, and angle relative to the origin.

[0022] Step S4 specifically includes: tracking the 3D target according to the deepsort target tracking method.

[0023] As another preferred embodiment, the present invention also provides a vehicle 3D detection and tracking device based on monocular vision, the device comprising at least:

[0024] The first acquisition module is used to acquire real-time images from a monocular camera, the real-time images including real-time images of at least one target vehicle in front of the vehicle captured by the monocular camera of the vehicle.

[0025] The first computing unit is used to process the real-time acquired image through a multi-task recognition model to obtain first target information. The first target information includes the 2D full-body frame corresponding to the target vehicle, the target vehicle feature vector, the target vehicle body posture, and the target vehicle body side line information.

[0026] The second acquisition module is used to acquire parameter information corresponding to the monocular camera.

[0027] The second calculation unit combines the parameter information corresponding to the monocular camera and the first target data to calculate the second target information of the target vehicle.

[0028] The tracking unit performs 3D target tracking based on the second target information using the deepsort target tracking method.

[0029] The first computing unit further includes:

[0030] The preprocessing module uses a convolutional neural network model to preprocess real-time acquired images.

[0031] The data filtering module is used to select feature points above a preset threshold from the heatmap obtained by the preprocessing module as target feature data.

[0032] The first detection module is used to obtain the vehicle type parameters corresponding to the feature points and calculate the 2D detection box classification list according to the CenterNet object detection algorithm.

[0033] The second detection module outputs the vehicle body posture information at the corresponding position for each 2D detection box, and obtains the posture of the target vehicle through the argmax operator; if the posture is Front_Side or Rear_Side, side line detection is performed to obtain the coordinates of the side line endpoints A and B.

[0034] The second computing unit further includes:

[0035] The calibration module is used to calibrate the camera's intrinsic and extrinsic parameters and establish the image coordinates to the ground plane vehicle coordinate system with the camera projection as the origin;

[0036] The coordinate mapping module is used to obtain the coordinate mapping of the endpoints and midpoints of the target vehicle detection frame in the ground plane vehicle coordinate system based on the selected bottom endpoints and midpoints of the 2D detection frame.

[0037] The parameter calculation module calculates the target vehicle's 3D coordinate system parameters in the ground plane vehicle coordinate system. The 3D coordinate system parameters include at least the vehicle length, vehicle width, vehicle height, and angle relative to the origin.

[0038] As another preferred embodiment, the present invention also provides a car equipped with a monocular camera and a vehicle 3D detection and tracking device based on monocular vision. The device uses the vehicle 3D detection and tracking method based on monocular vision as described above to process real-time images of at least one target vehicle in front of the car, which are acquired in real time by the monocular camera.

[0039] As another preferred embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vehicle 3D detection and tracking method based on monocular vision as described above.

[0040] In summary, this invention provides a vehicle 3D positioning method, device, vehicle, and storage medium based on monocular vision. It obtains first target information by acquiring images in real time from a monocular camera and processing them through a multi-task recognition model. Then, it acquires parameter information corresponding to the monocular camera, calculates second target information of the target vehicle by combining it with the first target data, and finally tracks the target vehicle based on the second target information.

[0041] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0042] (1) This invention utilizes a monocular camera on a vehicle to acquire images in real time while the vehicle is moving, and extracts feature points and feature lines from the images. The 2D bounding box of the vehicle is obtained using these feature points and feature lines, and then the coordinate mapping of the endpoints and midpoints of the target vehicle detection box in the ground plane vehicle coordinate system is obtained accordingly, thereby obtaining the 3D coordinate system parameters of the target vehicle. During the image tracking process, the optimized camera pose of the monocular camera is obtained based on the fusion optimization of feature points and feature lines. Even in environments with less texture, the camera pose with high accuracy can still be obtained, and thus the vehicle pose with high accuracy can be obtained. This can reduce the impact of environmental texture changes on vehicle positioning and greatly improve the robustness of the vehicle positioning system.

[0043] (2) The present invention uses the same scale feature layer for vehicle side line detection and vehicle detection. Each feature point corresponds to a set of detection results, and no subsequent matching is required.

[0044] (3) The vehicle side line detection in this invention has strong anti-interference ability and allows for a certain degree of occlusion. The vehicle body posture classification task allows various scenarios to be processed in a targeted manner.

[0045] (4) This invention can output the cuboid target of the preceding vehicle in the current vehicle 3D coordinate system. This invention combines vehicle 3D detection and feature detection to improve target tracking efficiency. Attached Figure Description

[0046] Figure 1 This is a flowchart of the vehicle 3D detection and tracking method based on monocular vision described in this invention.

[0047] Figure 2 This is a diagram of the convolutional neural network model described in this invention.

[0048] Figure 3 This is a schematic diagram of the ghost module described in this invention.

[0049] Figure 4 This is a schematic diagram of a method for calculating the side line of a vehicle body according to an embodiment of the present invention.

[0050] Figure 5 This is a schematic diagram of a method for calculating the side line of a vehicle body according to another embodiment of the present invention.

[0051] Figure 6 This is a schematic diagram of a method for calculating the side line of a vehicle body according to another embodiment of the present invention.

[0052] Figure 7 This is a schematic diagram of the target feature vector output in another embodiment of the present invention.

[0053] Figure 8 This is a schematic diagram of the vehicle coordinate system in another embodiment of the present invention.

[0054] Figure 9 This is a schematic diagram of image-to-vehicle coordinate system mapping in another embodiment of the present invention.

[0055] Figure 10 This is a projection analysis diagram illustrating the case where the side line of the front vehicle is not visible in another embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention provides a vehicle 3D detection and tracking method based on monocular vision, comprising the following steps:

[0058] S1: Acquire real-time images from a monocular camera, the real-time images including real-time images of at least one target vehicle in front of the vehicle captured by the monocular camera of the vehicle.

[0059] S2: Based on the real-time acquired image, the first target information is obtained by processing it through a multi-task recognition model. The first target information includes the 2D full-body frame of the target vehicle, the target vehicle feature vector, the target vehicle body posture, and the target vehicle body side line information.

[0060] S3: Obtain the parameter information corresponding to the monocular camera, and calculate the second target information of the target vehicle in combination with the first target data. The second target information includes the 3D full-body frame of the target vehicle.

[0061] S4: Track the target vehicle based on the second target information.

[0062] In step S2, the first target information is obtained by processing it through a multi-task recognition model, specifically including:

[0063] S21: The real-time acquired images are preprocessed using a convolutional neural network model to obtain a heat map.

[0064] The preprocessing specifically includes: setting the image size of the input end of the convolutional neural network model based on the camera resolution ratio and network structure parameters of the real-time acquired image; normalizing the multi-layer feature nodes according to the image size ratio of the input end of the convolutional neural network model; using a ghost module to perform linear transformation of image features to obtain ghost features, and then using the BiFPN feature fusion algorithm to fuse the weighted values ​​of ghost features at different scales to output multi-layer feature nodes; and normalizing the multi-layer feature nodes to obtain a heatmap.

[0065] like Figure 2-3 As shown, the convolutional neural network model includes: input units, a backbone network, feature fusion, and multi-task detection, specifically:

[0066] The input unit acquires images captured in real time by a monocular camera.

[0067] The backbone network uses a ResNet network structure based on GhostNet modules. GhostNet modules derive ghost features from existing features through simple linear transformations. This is shown in the lower half of the Output diagram above. Compared to MobileNet, GhostNet modules require less computation; the computational load can be reduced by appropriately decreasing the number of convolutional feature layers, resulting in richer feature representations. This makes it suitable for applications in autonomous driving, where real-time performance is critical.

[0068] Feature fusion: Google's BiFPN feature fusion algorithm is used. Unlike the traditional approach of directly stacking features of different scales, BiFPN can weight features of different scales for fusion, allowing the network to learn the weights of different input features. BiFPN is simple and easy to use. During feature fusion, BiFPN can be used multiple times depending on the complexity of the detection task and the hardware's computing power. For example, a 3-layer BiFPN can be used... Figure 4 As shown.

[0069] Multi-task detection: You can select one (e.g., P3_out), multiple, or all of the output feature nodes of BiFPNs for detection. When selecting some output feature layers, other output feature nodes can be deleted.

[0070] The following example uses P3_out for multi-task detection to illustrate the multi-task detection function in this invention:

[0071] Input: The input size should be designed based on the application's camera resolution ratio and network structure. Assuming a camera resolution of 1920x1080 (width:height = 3:2), the network structure above requires the input width and height to be multiples of 128. Based on this, the input size can be set (example sizes, including but not limited to):

[0072] Input_w = 128 * 6 = 768;

[0073] Input_h = 128 * 4 = 512;

[0074] Set the number of feature output layers P3_out to P3_out_c = 192. Based on the input size, the size of P3_out is:

[0075] P3_out_chw = [P3_out_c, Input_w / 8, Input_w / 8] = [192, 96, 64].

[0076] S22: Select feature points in the heat map that are higher than a preset threshold as target feature data, obtain the vehicle type parameters corresponding to the feature points, calculate the feature point distribution heat map, the target center relative feature point deviation regression value and the target box width and height regression value according to the CenterNet target detection algorithm, and then calculate the 2D detection box classification list, and record the 2D detection boxes at the corresponding coordinate positions on the heat map.

[0077] The 2D object detection uses the CenterNet algorithm. Each vehicle type (with 10 categories, but not limited to 10 categories) corresponds to an output sequence (heatmap, center deviation regression, width and height regression). The heatmap uses the focal_loss loss function, while the center deviation regression and width and height regression use the L2 loss function.

[0078] The 2D bounding box classification list uses a 6-class softmax structure and employs the cross-entropy loss function.

[0079] Vehicle body posture classification illustrate Front All you can see is the front of the car (your line of sight is directly facing the front of the car). Rear All I could see was the rear of the car (my line of sight was directly facing the rear of the car). Left_Side The car was seen turning sideways (the direction of the target vehicle was perpendicular to the line of sight), with the front of the car facing left. Right_Side The car was seen turning sideways (the direction of the target vehicle was perpendicular to the line of sight), with the front of the car facing right. Front_Side The front and side profiles of the car are visible. Rear_Side The rear and side profiles are visible.

[0080] S23: For each 2D detection box, output the vehicle body pose information at the corresponding position, and obtain the pose of the target vehicle through the argmax operator; if the pose is Front_Side or Rear_Side, perform side line detection to obtain the coordinates of the side line endpoints A and B; otherwise, return to S22 until all target vehicle detection boxes have been processed.

[0081] In the side line detection process, each feature location (96 x 64) corresponds to one detection result. This invention uses the L2 loss function to determine the side line, specifically providing three schemes for determining the vehicle body side line:

[0082] Option 1:

[0083] like Figure 4 As shown, let the coordinate deviation (reg_x, reg_y) of the midpoint C of the line segment containing endpoints A and B of the side line relative to the center feature point F be the distance line segment length L, and the angle R of the line segment relative to the x-axis be in the range [0, 1]. The formula for calculating the side line detection is as follows:

[0084] C_x = Fx + reg_x;

[0085] C_y = Fy + reg_y;

[0086] A_x = C_x - length / 2 * cos(angle*180°);

[0087] A_y = C_y + length / 2 * sin(angle*180°);

[0088] B_x = C_x + length / 2 * cos(angle*180°);

[0089] B_y = C_y - length / 2 * sin(angle*180°).

[0090] The advantage of this approach is that it does not rely on the accuracy of the 2D detection box.

[0091] Option 2:

[0092] like Figure 5 As shown, taking the endpoints A and B of the side line on the 2D detection box as prior information, the relative position ratio of the bottom edge endpoints reg_x is output, where 0 indicates the left end of the bottom line and 1 indicates the right end. The relative position of the side edge endpoints reg_y is also output, where [0, 0.5] indicates the left side of the box (0 above, 0.5 below) and (0.5, 1] ​​indicates the right side of the box (0.5 above, 1 below). The formula for calculating the side line detection is:

[0093] A_x = Box_x + Box_w * reg_x;

[0094] A_y = Box_y + Box_h;

[0095] When reg_y ≤ 0.5,

[0096] B_x = Box_x;

[0097] B_y = Box_y + Box_h * reg_y;

[0098] When reg_y > 0.5,

[0099] B_x = Box_x + Box_w;

[0100] B_y = Box_y + Box_h * (reg_y – 0.5).

[0101] The advantages of this approach are: it utilizes information from 2D bounding boxes; and it requires less regression work.

[0102] Option 3:

[0103] like Figure 6 As shown, the side line detection includes:

[0104] Using the fact that the endpoints A and B of the side line are located on the 2D detection box as prior information, the relative position ratio of the bottom edge endpoints reg_x is output, where 0 indicates the left end of the bottom line and 1 indicates the right end of the bottom line; the line segment relative to the x-axis r has a value range of [0, 1]; the side line detection formula is as follows:

[0105] A_x = Box_x + Box_w * reg_x;

[0106] A_y = Box_y + Box_h;

[0107] When angle < 90°

[0108] B_x = Box_x + Box_w;

[0109] B_y = Box_y + Box_h - Box_w * (1 - reg_x) * tan(angle*180°);

[0110] When angle > 90°

[0111] B_x = Box_x;

[0112] B_y = Box_y + Box_h + Box_w * reg_x * tan(angle*180°).

[0113] The advantages of this solution are: Figure 6 As shown, when the front vehicle's posture is Front_Side or Rear_Side, the side line direction (angle) is obvious. Furthermore, when the side line is visible in the forward view, the angle value cannot be within a small neighborhood centered at 0.5 (i.e., 90°), thus avoiding any abrupt changes. Endpoint A can be obtained by intersecting the bounding box with the outer perpendicular line of the taillight or headlight, making the feature clear. Both angle and reg_x have values ​​in the range [0, 1], facilitating network regression learning.

[0114] Feature output such as Figure 7 As shown. Each feature location (96 x 64) corresponds to an n-dimensional feature vector. Taking n=64 as an example (this value is not limited to specific applications). During training, referring to the ArcFace algorithm, the feature vector is followed by a tracking ID classifier. The number of classes is equal to the number of IDs in the training sample set, ensuring that targets with the same ID in the training set are projected into the same ID class, and targets with different IDs are projected into different ID classes. In this way, target feature vectors with individual differences can be obtained. When saving the model, only the target feature vector output is saved.

[0115] Step S3 of the present invention, which calculates the second target information of the target vehicle, specifically includes:

[0116] S31: Calibrate the camera's intrinsic and extrinsic parameters respectively, and establish the image coordinates to the ground plane vehicle coordinate system with the camera projection as the origin, such as... Figure 8 As shown.

[0117] S32: Select the bottom endpoint and midpoint of the 2D detection frame, and obtain the coordinate mapping of the endpoint and midpoint of the target vehicle detection frame in the ground plane vehicle coordinate system.

[0118] Let point E be the endpoint of the bottom line of the frame on the side of the invisible vehicle. We obtain the following coordinates in the vehicle coordinate system: A'(A'_x, A'_y), B'(B'_x, B'_y), E'(E'_x, E'_y). We then calculate the vehicle length, vehicle width, vehicle height, the angle of the vehicle in front, and the coordinate of A' to obtain the current vehicle's 3D coordinate system.

[0119] S33: Take the bottom endpoints A and E of the 2D detection frame and the midpoint M (M_x, M_y), and calculate the 3D coordinate system parameters of the target vehicle in the ground plane vehicle coordinate system. The 3D coordinate system parameters include at least the vehicle length, vehicle width, vehicle height, and angle relative to the origin.

[0120] Step S4 specifically includes: tracking the 3D target according to the deepsort target tracking method.

[0121] Preferably, in this embodiment, it is assumed that the camera is a pinhole camera; if not, distortion correction can be performed to convert it to a pinhole camera image. That is, as follows: Figure 9 As shown, the coordinate mapping of points on the ground in the image to the corresponding coordinates in the vehicle coordinate system is calculated, where point E is the endpoint of the bounding box line on the side of the invisible vehicle. The resulting coordinates in the vehicle coordinate system are: A'(A'_x, A'_y), B'(B'_x, B'_y), E'(E'_x, E'_y).

[0122] Then, the calculation yields:

[0123] Vehicle length:

[0124] .

[0125] Relative to the vehicle's x-axis, the angle of the vehicle in front: (range (-90°, 90°), leftward deviation is positive, rightward deviation is negative):

[0126] .

[0127] Since the rear frame of the vehicle is projected onto the camera at the same distance, according to the principle of pinhole imaging, the aspect ratio in the image coordinates is consistent with the actual aspect ratio, thus the vehicle height is:

[0128] .

[0129] In the image coordinate system The Euclidean distance between two points.

[0130] When the vehicle's orientation θ > 0, the right endpoint E of the rear frame's bottom line can be approximated as the projection of the actual E_r onto the line x = A_x. Therefore, the vehicle width is:

[0131] .

[0132] Based on the attitude classification of the vehicles ahead, the angle θ_final between the facing direction of the preceding vehicle and the facing direction of the current vehicle (x-axis direction) can then be obtained.

[0133] When car_pose = Front_Side, θ_final =θ + 180°.

[0134] When car_pose = Rear_Side (e.g.) Figure 9 (Example in the text), θ_final = θ.

[0135] In summary, based on the length, width, and height of the preceding vehicle, the vehicle's direction, and coordinate A', the current vehicle's 3D coordinate system is obtained (in...). Figure 7 Add a cuboid target to the coordinate system (vertical to the ground and upward along the z-axis) until all vehicle detection boxes have been processed.

[0136] In step S33, the target vehicle's 3D coordinate system parameters in the ground plane vehicle coordinate system are calculated. The specific process is as follows:

[0137] Take the bottom endpoints A and E of the 2D frame of the vehicle and its midpoint M (M_x, M_y), as follows: Figure 10 As shown:

[0138] The formula for calculating the midpoint M(M_x, M_y) is:

[0139] M_x = Box_x + Box_w / 2.

[0140] M_y = Box_y + Box_h.

[0141] The projection of point M into the vehicle coordinate system is M'(M'_x, M'_y), which gives:

[0142] .

[0143] Distance between A' and E': .

[0144] Assuming the front vehicle's length to width ratio is 3:1, we get:

[0145] When car_pose = Front, θ_final =θ + 180°; car_W = dis(A', E');

[0146] .

[0147] When car_pose = Rear, θ_final =θ; car_W = dis(A', E');

[0148] car_L = 3 * car_W.

[0149] When car_pose = Left_Side, θ_final =θ + 90°; car_L= dis (A', E');

[0150] ;car_W =car_L / 3.

[0151] When car_pose = Right_Side,

[0152] θ_final =θ + 270°; car_L = dis(A', E');

[0153] ;car_W =car_L / 3.

[0154] As can be seen, based on the length, width, and height of the preceding vehicle, the vehicle's direction, and the coordinate A', the cuboid target in the current vehicle's 3D coordinate system can be obtained.

[0155] As another preferred embodiment, the present invention also provides a vehicle 3D detection and tracking device based on monocular vision, the device comprising at least:

[0156] The first acquisition module is used to acquire real-time images from a monocular camera, the real-time images including real-time images of at least one target vehicle in front of the vehicle captured by the monocular camera of the vehicle.

[0157] The first computing unit is used to process the real-time acquired image through a multi-task recognition model to obtain first target information. The first target information includes the 2D full-body frame corresponding to the target vehicle, the target vehicle feature vector, the target vehicle body posture, and the target vehicle body side line information.

[0158] The second acquisition module is used to acquire parameter information corresponding to the monocular camera.

[0159] The second calculation unit combines the parameter information corresponding to the monocular camera and the first target data to calculate the second target information of the target vehicle.

[0160] The tracking unit performs 3D target tracking based on the second target information using the deepsort target tracking method.

[0161] The first computing unit further includes:

[0162] The preprocessing module uses a convolutional neural network model to preprocess real-time acquired images.

[0163] The data filtering module is used to select feature points above a preset threshold from the heatmap obtained by the preprocessing module as target feature data.

[0164] The first detection module is used to obtain the vehicle type parameters corresponding to the feature points and calculate the 2D detection box classification list according to the CenterNet object detection algorithm.

[0165] The second detection module outputs the vehicle body posture information at the corresponding position for each 2D detection box, and obtains the posture of the target vehicle through the argmax operator; if the posture is Front_Side or Rear_Side, side line detection is performed to obtain the coordinates of the side line endpoints A and B.

[0166] The second computing unit further includes:

[0167] The calibration module is used to calibrate the camera's intrinsic and extrinsic parameters and establish the image coordinates to the ground plane vehicle coordinate system with the camera projection as the origin.

[0168] The coordinate mapping module is used to obtain the coordinate mapping of the endpoints and midpoints of the target vehicle detection frame in the ground plane vehicle coordinate system based on the selected bottom endpoints and midpoints of the 2D detection frame.

[0169] The parameter calculation module calculates the target vehicle's 3D coordinate system parameters in the ground plane vehicle coordinate system. The 3D coordinate system parameters include at least the vehicle length, vehicle width, vehicle height, and angle relative to the origin.

[0170] As another preferred embodiment, the present invention also provides a car equipped with a monocular camera and a vehicle 3D detection and tracking device based on monocular vision. The device uses the vehicle 3D detection and tracking method based on monocular vision as described above to process real-time images of at least one target vehicle in front of the car, which are acquired in real time by the monocular camera.

[0171] As another preferred embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vehicle 3D detection and tracking method based on monocular vision as described above.

[0172] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0173] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0175] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0176] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] Although the invention has been described in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.

Claims

1. A monocular vision based vehicle 3D detection and tracking method, characterized in that, Including the following steps: S1: Acquire real-time images from a monocular camera, the real-time images including real-time images of at least one target vehicle in front of the vehicle captured by the monocular camera of the vehicle. S2: Based on the real-time acquired image, the first target information is obtained by processing it through a multi-task recognition model. The first target information includes the 2D full-body frame of the target vehicle, the target vehicle feature vector, the target vehicle body posture, and the target vehicle body side line information. S3: Obtain the parameter information corresponding to the monocular camera, and calculate the second target information of the target vehicle in combination with the first target information. The second target information includes the 3D full-body frame of the target vehicle. S4: Track the target vehicle based on the second target information; In step S2, the first target information is obtained by processing it through a multi-task recognition model, specifically including: S21: The real-time acquired images are preprocessed using a convolutional neural network model to obtain a heat map; S22: Select feature points in the heat map that are higher than a preset threshold as target feature data, obtain the vehicle type parameters corresponding to the feature points, and calculate the 2D detection box classification list according to the CenterNet target detection algorithm. S23: For each 2D detection box, output the vehicle body pose information at the corresponding position, and obtain the pose of the target vehicle through the argmax operator; if the pose is Front_Side or Rear_Side, perform side line detection to obtain the coordinates of the side line endpoints A and B; otherwise, return to S22 until all target vehicle detection boxes have been processed.

2. The monocular vision based vehicle 3D detection and tracking method of claim 1, wherein, The preprocessing in step S21 specifically includes: The image size at the input end of the convolutional neural network model is set according to the camera resolution ratio and network structure parameters for the real-time acquired images; Based on the image size ratio at the input of the convolutional neural network model, the multi-layer feature nodes are normalized; ghost features are obtained by performing linear transformation of image features using a ghost module, and then the weighted values ​​of ghost features at different scales are fused using the BiFPN feature fusion algorithm to output multi-layer feature nodes; the multi-layer feature nodes are normalized to obtain a heatmap.

3. The monocular vision based vehicle 3D detection and tracking method of claim 2, wherein, Step S3, which calculates the second target information of the target vehicle, specifically includes: S31: Calibrate the camera's internal and external parameters respectively, and establish the image coordinates to the ground plane vehicle coordinate system with the camera projection as the origin; S32: Select the bottom endpoint and midpoint of the 2D detection frame, and obtain the coordinate mapping of the endpoint and midpoint of the target vehicle detection frame in the ground plane vehicle coordinate system accordingly; S33: Calculate the target vehicle's 3D coordinate system parameters in the ground plane vehicle coordinate system. The 3D coordinate system parameters include at least the vehicle length, vehicle width, vehicle height, and angle relative to the origin.

4. The vehicle 3D detection and tracking method based on monocular vision according to claim 3, characterized in that, Step S4 specifically includes: tracking the 3D target according to the deepsort target tracking method.

5. The apparatus for vehicle 3D detection and tracking based on monocular vision according to any one of claims 1-4, characterized in that, The device includes at least: The first acquisition module is used to acquire real-time images captured by a monocular camera, the real-time images including real-time images captured by the monocular camera of the vehicle of at least one target vehicle in front. The first computing unit is used to process the real-time acquired image through a multi-task recognition model to obtain first target information. The first target information includes the 2D full-body frame of the target vehicle, the target vehicle feature vector, the target vehicle body posture, and the target vehicle body side line information. The second acquisition module is used to acquire parameter information corresponding to the monocular camera; The second calculation unit calculates the second target information of the target vehicle by combining the parameter information corresponding to the monocular camera and the first target information; The tracking unit performs 3D target tracking based on the second target information using the deepsort target tracking method.

6. The apparatus according to claim 5, characterized in that, The first computing unit further includes: The preprocessing module uses a convolutional neural network model to preprocess real-time acquired images; The data filtering module is used to select feature points above a preset threshold in the heat map obtained by the preprocessing module as target feature data. The first detection module is used to obtain the vehicle type parameters corresponding to the feature points and calculate the 2D detection box classification list according to the CenterNet object detection algorithm. The second detection module outputs the vehicle body posture information at the corresponding position for each 2D detection box, and obtains the posture of the target vehicle through the argmax operator; if the posture is Front_Side or Rear_Side, side line detection is performed to obtain the coordinates of the side line endpoints.

7. The apparatus according to claim 6, characterized in that, The second computing unit further includes: The calibration module is used to calibrate the camera's intrinsic and extrinsic parameters and establish the image coordinates to the ground plane vehicle coordinate system with the camera projection as the origin; The coordinate mapping module is used to obtain the coordinate mapping of the endpoints and midpoints of the target vehicle detection frame in the ground plane vehicle coordinate system based on the selected bottom endpoints and midpoints of the 2D detection frame. The parameter calculation module calculates the target vehicle's 3D coordinate system parameters in the ground plane vehicle coordinate system. The 3D coordinate system parameters include at least the vehicle length, vehicle width, vehicle height, and angle relative to the origin.

8. A car equipped with a monocular camera, characterized in that, It is also equipped with a vehicle 3D detection and tracking device based on monocular vision. The device uses the vehicle 3D detection and tracking method based on monocular vision as described in any one of claims 1-4 to process the real-time image of at least one target vehicle in front of the car, which is captured in real time by the monocular camera.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the monocular vision-based 3D vehicle detection and tracking method as described in any one of claims 1-4.

Images