Three-dimensional bounding box detection method for traffic participants under vehicle-mounted monocular vision

Abstract

The invention relates to the technical field of environmental perception of automatic driving and advanced driving assistance systems, and provides a three-dimensional bounding box detection method for traffic participants under vehicle-mounted monocular vision. The method comprises the following steps: extracting a midpoint of two ankles as a grounding point by utilizing human body posture estimation for a pedestrian, obtaining a contact point of a tire and the ground through wheel detection and grounding point fine positioning for a vehicle, finishing distortion removal and plane projection in combination with camera calibration parameters, and mapping to a ground coordinate system and a vehicle coordinate system through a homography matrix. On the basis, in combination with priori such as axle distance, wheel track and vehicle body size, geometric reasoning complementation is performed on part of visible grounding points, the center position, size and orientation of the target are estimated, and a three-dimensional bounding box unified in a vehicle coordinate system is constructed. Three-dimensional perception of multiple types of traffic participants can be realized only by depending on the monocular camera, the hardware cost and the calibration complexity are reduced, and the detection precision in a shielding scene is improved.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology for autonomous driving and advanced driver assistance systems, and in particular to a method for detecting three-dimensional bounding boxes of traffic participants under vehicle-mounted monocular vision. Background Technology

[0002] In autonomous driving technology, real-time and accurate perception of traffic participants such as vehicles, pedestrians, and non-motorized vehicles is a prerequisite for achieving safe navigation and decision-making. Perception systems typically need to acquire rich information about traffic participants in three-dimensional space, including their precise location, physical dimensions (length, width, and height), and orientation (attitude).

[0003] In existing technologies, the mainstream approach to achieving high-precision 3D perception is multi-sensor fusion, which integrates sensors such as LiDAR, millimeter-wave radar, and stereo cameras. LiDAR can directly provide high-precision 3D point clouds, enabling accurate measurement of the target's distance and shape. However, this type of multi-sensor fusion solution also has significant drawbacks: firstly, the hardware costs of sensors such as LiDAR and high-precision stereo cameras are extremely high; secondly, the extrinsic parameter calibration process between multiple sensors (especially LiDAR and cameras) is complex, and it is difficult to maintain calibration accuracy for extended periods under the influence of bumps and vibrations during actual vehicle operation, thus increasing the system's maintenance complexity and uncertainty.

[0004] To overcome the aforementioned cost and complexity issues, 3D perception technology based on low-cost vehicle cameras (especially monocular vision) has gradually become a research hotspot and industry development trend in the fields of autonomous driving and intelligent driving. However, while detection methods based solely on 2D images can provide the category of the target and its pixel location in the image, they struggle to acquire timely and accurate 3D information such as the position, orientation, and size of traffic participants in real 3D space.

[0005] In summary, there is an urgent need for a low-cost (monocular vision-based), high-efficiency (low computational overhead), and generalizable 3D bounding box detection method that can be applied to different types of traffic participants, so as to meet the needs of autonomous driving scenarios for 3D information perception of traffic participants while reducing hardware costs and system complexity. Summary of the Invention

[0006] In view of this, in order to solve the problems of high hardware cost and complex calibration and maintenance of existing multi-sensor fusion solutions, and the difficulty of existing monocular vision methods based solely on two-dimensional detection in acquiring the three-dimensional bounding box information of different types of traffic participants in a low-cost, high-efficiency, and robust manner, this invention provides a three-dimensional bounding box detection method for traffic participants under vehicle-mounted monocular vision. This method outputs the three-dimensional bounding box of each traffic participant in the vehicle coordinate system, including center coordinates, size, and pose, relying only on a single vehicle-mounted camera. This meets the engineering application requirements of low cost, real-time, and mass production in autonomous driving scenarios, improves the accuracy of ground contact point positioning, and enhances the detection robustness under occlusion conditions.

[0007] To achieve the above objectives, the present invention provides a method for detecting three-dimensional bounding boxes of traffic participants under vehicle-mounted monocular vision, characterized by the following steps: (1) Detecting the bounding boxes and categories of traffic participants based on the target detection model: Using a pre-trained target detection model, inference is performed on the images acquired by the vehicle-mounted monocular camera to detect the two-dimensional bounding boxes and their corresponding category labels of all traffic participants in the images; (2) Detecting the grounding points of traffic participants: For each of the two-dimensional bounding boxes, determine the pixel coordinates of one or more grounding points of the traffic participant in the image coordinate system within its corresponding image area; (3) Obtain camera intrinsic parameters and perform distortion correction processing on the docking location: obtain the camera intrinsic parameter matrix and camera distortion coefficient by calibrating the checkerboard image, perform distortion correction mapping on the grounding point pixel coordinates detected in step (2), and obtain the corrected two-dimensional grounding point. (4) Obtain the homography matrix and calculate the actual coordinates of the ground point on the ground plane in the vehicle coordinate system: Obtain the projection homography matrix from the camera distortion-free image plane to the ground plane through the surround view system calibration, project the corrected ground point onto the ground plane of the ground coordinate system to obtain the ground coordinates of the ground point in the ground coordinate system, and convert the ground coordinates into the ground plane coordinates of the ground point in the vehicle coordinate system according to the preset rotation and translation relationship between the ground coordinate system and the vehicle coordinate system. (5) Reasoning to complete the invisible grounding points and constructing a three-dimensional bounding box: The three-dimensional bounding box is a regular cube structure, and its bottom vertex is obtained by reasoning from the grounding points of traffic participants. For vehicle-type traffic participants, when only some grounding points are detected, the invisible grounding points are inferred and completed based on the vehicle structure information, vehicle type prior, and visible grounding point information to obtain the vehicle center point and orientation. Based on the vehicle center point, vehicle body size, and height information, a complete three-dimensional bounding box of traffic participants is constructed.

[0008] Furthermore, the object detection model mentioned in step (1) is a pre-trained YOLO object detection model, and the set of two-dimensional bounding boxes output by the model is represented as follows: Each bounding box bi contains a vector: , These represent: the pixel coordinates of the bounding box center point (xc, yc), the width w and height h of the bounding box, the target category c, and the confidence score p, respectively.

[0009] Furthermore, in step (2), for pedestrians, a human posture recognition model is used to detect key points of the human body, and the midpoint of the two ankles is extracted as the grounding point. Let the coordinates of the left ankle be... The coordinates of the right ankle are If both points are visible, then the grounding point is... Midpoint: .

[0010] Furthermore, in step (2), for vehicle-type targets, a multi-level processing method is used to determine the grounding point, including: (1) For each vehicle, extract the ROI corresponding to its two-dimensional bounding box, and detect the two-dimensional bounding boxes of all wheels within the ROI using the wheel detection model. (2) For each wheel bounding box, crop image blocks and use a grounding point fine localization network to output a low-resolution target heatmap of the image within the two-dimensional wheel bounding box. Assuming the actual grounding point is The target heat map It follows a two-dimensional Gaussian distribution: .

[0011] Furthermore, to ensure that the positioning point is located at the actual contact position between the tire and the ground, a vertical weighting function is introduced during heatmap inference. Let the height of the wheel rim be... Within the image patch The coordinate range is The weighting function is defined as follows: Where α is the gain coefficient; the heat map prediction result is denoted as Mpred(x,y), and the peak index of the weighted heat map corresponds to the location of the grounding point. ,satisfy:

[0012] The peak coordinates By adding the ROI offset and mapping it back to the pixel coordinates of the original image, the pixel coordinates of the visible wheel's contact point can be obtained. .

[0013] Furthermore, in step (3), the camera intrinsic parameters are obtained by calibrating the checkerboard image. The calibration results include: the transformation matrix K from the camera coordinate system to the pixel coordinate system, and the camera distortion coefficients D. The grounding point Pimg detected in step (2) is subjected to distortion correction mapping F^-1 to obtain the corrected two-dimensional point: .

[0014] Furthermore, in step (4), the corrected grounding point is represented in homogeneous coordinates. The projection homography matrix from the camera-distorted image plane to the ground plane is obtained through the surround view system calibration. In the ground coordinate system, the coordinates of the grounding point are: ,satisfy: , Right now:

[0015] By eliminating scale factor Solve for the ground coordinates .

[0016] Furthermore, in step (4), a preset rotation is performed between the ground coordinate system and the vehicle coordinate system (e.g., with the rear axle center as the origin, the front direction of the vehicle as the X-axis, and the left side of the vehicle as the Y-axis). Peaceful relocation Calculate the ground plane coordinates of the grounding point in the vehicle coordinate system: .

[0017] Furthermore, in step (5), for vehicle-type traffic participants, prior vehicle structure information (including wheelbase L) is used. wb Wheelbase W tk Vehicle length L body , body width W body Vehicle height H body The set of detected and projected ground grounding points is used for reasoning and completion, along with the information of visible grounding points. ,in The coordinates are in the vehicle coordinate system; When 3 grounding points are known At that time, based on the geometric properties of a rectangle, the fourth grounding point was completed using a vector method. : , And calculate the vehicle's ground center point:

[0018] The longitudinal direction vector of the vehicle is determined by the slope of the line connecting the grounding points on the same side and the line connecting the grounding points on the same axis. and horizontal direction vector This determines the vehicle's orientation.

[0019] Furthermore, in step (5), when only two grounding points are detected... When calculating the Euclidean distance between two points... , And combined with the prior wheelbase of the vehicle model and wheelbase Set threshold : like The wheels were determined to be on the same side. like These are determined to be wheels on the same axle; This allows us to deduce the vehicle's center point and orientation, and complete the other two grounding points; Based on vehicle center and orientation angle Combined with vehicle body size Calculate the four corner points at the bottom of the 3D bounding box on the ground. ,satisfy:

[0020] Based on the four corner points on the ground, Z-axis coordinates are added according to the height of vehicles or pedestrians (the average value is determined according to the category) to construct the top four corner points, and finally a complete three-dimensional bounding box containing 8 corner points is obtained.

[0021] The present invention, by adopting the above technical solution, has at least the following beneficial effects: I. This invention relies solely on an in-vehicle monocular camera to acquire image information. Through target detection, ground point detection, and geometric projection, it can complete the detection of the three-dimensional bounding box of traffic participants. It does not require expensive or complex sensors such as LiDAR or multi-view cameras. While ensuring three-dimensional perception capabilities, it can significantly reduce the hardware cost and calibration and deployment difficulty of the entire perception system, making it easy to modify and apply to existing vehicle models.

[0022] II. This invention features a sophisticated design for the extraction and localization of ground contact points. For pedestrians, key points on both ankles are extracted using a human posture recognition model, with the midpoint between these two points serving as the ground contact point. For vehicles, the actual contact position between the tires and the ground is precisely located through wheel detection, heatmap regression, and vertical weight correction. By combining camera distortion correction and homography matrix mapping operations, the ground contact point can be accurately transformed into both the ground coordinate system and the vehicle coordinate system. This ensures that the bottom surface of the generated 3D bounding box is aligned with the ground height, more closely approximating the actual physical location, thereby improving the positional accuracy and reliability of height and size estimation of the 3D bounding box.

[0023] Third, this invention utilizes prior parameters of the vehicle model structure and performs geometric reasoning on a set of contact points in the vehicle coordinate system: when only some contact points are obtained, the missing contact points are filled in by analyzing the geometric relationships between three or two points, thereby calculating the vehicle's center position and orientation; for pedestrians, only one ankle contact point is needed to construct their 3D bounding box. Therefore, even with partial occlusion or incomplete field of view, a complete and reasonably posed 3D bounding box can still be output, demonstrating strong overall robustness.

[0024] Fourth, this invention performs distortion correction and projection calculations only on a limited number of grounding points, rather than performing uniform geometric correction on the entire image. At the same time, it achieves fast mapping from the image plane to the ground plane through homography matrix. Compared with the scheme based on dense 3D reconstruction, it has less computation and higher operating efficiency, making it suitable for real-time deployment on hardware platforms with limited computing power, such as vehicle terminals.

[0025] Fifth, this invention outputs a unified three-dimensional bounding box for various traffic participants such as pedestrians, passenger cars, non-motorized vehicles, and trucks in the same vehicle coordinate system. The center point, size, and orientation have consistent definition specifications, which facilitates direct calling and integration by subsequent modules such as environmental modeling, motion prediction, and risk assessment. It is conducive to building a unified vehicle perception and decision-making system with clear interfaces and has high engineering application value. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an embodiment of the three-dimensional bounding box detection method of the present invention; Figure 2 This is a flowchart of Embodiment 2 of the three-dimensional bounding box detection method of the present invention; Figure 3 This is a schematic diagram of a specific implementation example of the present invention. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0029] Example 1 like Figure 1 As shown in the figure, this embodiment provides a method for detecting the three-dimensional bounding boxes of traffic participants under vehicle-mounted monocular vision, characterized by the following steps: (1) Detecting the bounding boxes and categories of traffic participants based on the target detection model: Using a pre-trained target detection model, inference is performed on the images acquired by the vehicle-mounted monocular camera to detect the two-dimensional bounding boxes and their corresponding category labels of all traffic participants in the images; (2) Detecting the grounding points of traffic participants: For each of the two-dimensional bounding boxes, determine the pixel coordinates of one or more grounding points of the traffic participant in the image coordinate system within its corresponding image area; (3) Obtain camera intrinsic parameters and perform distortion correction processing on the docking location: obtain the camera intrinsic parameter matrix and camera distortion coefficient by calibrating the checkerboard image, perform distortion correction mapping on the grounding point pixel coordinates detected in step (2), and obtain the corrected two-dimensional grounding point. (4) Obtain the homography matrix and calculate the actual coordinates of the ground point on the ground plane in the vehicle coordinate system: Obtain the projection homography matrix from the camera distortion-free image plane to the ground plane through the surround view system calibration, project the corrected ground point onto the ground plane of the ground coordinate system to obtain the ground coordinates of the ground point in the ground coordinate system, and convert the ground coordinates into the ground plane coordinates of the ground point in the vehicle coordinate system according to the preset rotation and translation relationship between the ground coordinate system and the vehicle coordinate system. (5) Reasoning to complete the invisible grounding points and constructing a three-dimensional bounding box: The three-dimensional bounding box is a regular cube structure, and its bottom vertex is obtained by reasoning from the grounding points of traffic participants. For vehicle-type traffic participants, when only some grounding points are detected, the invisible grounding points are inferred and completed based on the vehicle structure information, vehicle type prior, and visible grounding point information to obtain the vehicle center point and orientation. Based on the vehicle center point, vehicle body size, and height information, a complete three-dimensional bounding box of traffic participants is constructed.

[0030] Through the sequential processing of the above steps, this embodiment can obtain the three-dimensional bounding box information of traffic participants by relying solely on the vehicle-mounted monocular camera. This reduces the complexity and cost of sensor configuration while meeting the real-time and accuracy requirements of vehicle-mounted online perception.

[0031] As one implementation method, in this embodiment, the target detection model in step (1) is a pre-trained YOLO target detection model, and the set of two-dimensional bounding boxes output by the model is represented as follows: , Each bounding box bi contains a vector: , These represent: the pixel coordinates of the bounding box center point (xc, yc), the width w and height h of the bounding box, the target category c, and the confidence score p, respectively.

[0032] By outputting candidate bounding boxes and category information of traffic participants through a unified target detection model, subsequent grounding point detection and 3D bounding box construction can be processed on the same set of candidate targets, reducing the impact of missed detections and false detections on the overall 3D perception results, and facilitating the selection of more reliable targets for subsequent calculations based on confidence levels.

[0033] As one implementation method, in step (2) of this embodiment, for pedestrians, a human posture recognition model is used to detect key points of the human body, and the midpoint of the two ankles is extracted as the grounding point. Let the coordinates of the left ankle be... The coordinates of the right ankle are If both points are visible, then the grounding point is... Midpoint: .

[0034] Using the key points of both ankles output by the human posture recognition model as the basis for pedestrian grounding points, the grounding points can fit the actual landing position of the feet. Compared with the method of inferring only based on the bottom edge of the two-dimensional bounding box, the grounding point positioning accuracy is higher and it has better adaptability to changes in walking posture, body tilt and other factors.

[0035] Furthermore, in step (2), for vehicle-type targets, a multi-level processing method is used to determine the grounding point, including: (1) For each vehicle, extract the ROI corresponding to its two-dimensional bounding box, and detect the two-dimensional bounding boxes of all wheels within the ROI using the wheel detection model. (2) For each wheel bounding box, crop image blocks and use a grounding point fine localization network to output a low-resolution target heatmap of the image within the two-dimensional wheel bounding box. Assuming the actual grounding point is The target heat map It follows a two-dimensional Gaussian distribution: .

[0036] By first detecting the wheel position within the vehicle's two-dimensional bounding box and then regressing the grounding point heatmap within the local wheel image, the search range is gradually narrowed down to the area near the tire. This approach ensures precise grounding point localization while effectively suppressing interference from other vehicle structures and background noise on the localization results.

[0037] As one implementation method, in this embodiment, to ensure that the positioning point is located at the actual contact position between the tire and the ground, a vertical weighting function is introduced during heatmap inference. Let the height of the wheel rim be... Within the image patch The coordinate range is The weighting function is defined as follows: Where α is the gain coefficient; the heat map prediction result is denoted as Mpred(x,y), and the peak index of the weighted heat map corresponds to the location of the grounding point. ,satisfy:

[0038] The peak coordinates By adding the ROI offset and mapping it back to the pixel coordinates of the original image, the pixel coordinates of the visible wheel's contact point can be obtained. .

[0039] By introducing weights that gradually increase vertically on the heatmap, the response near the bottom of the image, that is, closer to the tire-ground interface area, is enhanced, thereby making the predicted grounding point closer to the actual landing point and reducing the offset caused by factors such as shadows, wheel covers and road texture.

[0040] As one implementation method, in step (3) of this embodiment, the camera intrinsic parameters are obtained by calibrating the checkerboard image. The calibration results include: the transformation matrix K from the camera coordinate system to the pixel coordinate system, and the camera distortion coefficient D. The grounding point Pimg detected in step (2) is subjected to distortion correction mapping F^-1 to obtain the corrected two-dimensional point: .

[0041] This embodiment only performs distortion correction mapping on the pixel coordinates of the docking location, without performing global distortion correction on the entire image. This reduces the computational load of geometric correction and ensures that the coordinates of key points participating in subsequent projection calculations conform to the ideal imaging model, thereby improving the accuracy of 3D geometric reasoning.

[0042] Furthermore, in step (4), the corrected grounding point is represented in homogeneous coordinates. The projection homography matrix from the camera-distorted image plane to the ground plane is obtained through the surround view system calibration. In the ground coordinate system, the coordinates of the grounding point are: ,satisfy: , Right now:

[0043] By eliminating scale factor Solve for the ground coordinates .

[0044] By using a homography matrix to establish a mapping relationship between the camera's distorted image plane and the ground plane, the grounding point can be accurately converted from pixel coordinates to its actual position in the ground coordinate system. This ensures that the 3D bounding box subsequently constructed in the vehicle coordinate system is consistent with the real ground position and facilitates fusion with map information and other sensor data.

[0045] As one implementation method, in this embodiment, step (4) involves a preset rotation between the ground coordinate system and the vehicle coordinate system (e.g., with the rear axle center as the origin, the front direction of the vehicle as the X-axis, and the left side of the vehicle as the Y-axis). Peaceful relocation Calculate the ground plane coordinates of the grounding point in the vehicle coordinate system: .

[0046] By pre-setting the rotation and translation relationship between the ground coordinate system and the vehicle coordinate system, each ground point is uniformly represented in a coordinate system with the vehicle as the reference. This is beneficial for subsequent processing such as environmental modeling, collision risk assessment, and path planning based on the vehicle itself.

[0047] Furthermore, in step (5), for vehicle-type traffic participants, prior vehicle structure information (including wheelbase L) is used. wb Wheelbase W tk Vehicle length L body , body width W body Vehicle height H body The set of detected and projected ground grounding points is used for reasoning and completion, along with the information of visible grounding points. ,in The coordinates are in the vehicle coordinate system; When 3 grounding points are known At that time, based on the geometric properties of a rectangle, the fourth grounding point was completed using a vector method. : , And calculate the vehicle's ground center point:

[0048] The longitudinal direction vector of the vehicle is determined by the slope of the line connecting the grounding points on the same side and the line connecting the grounding points on the same axis. and horizontal direction vector This determines the vehicle's orientation.

[0049] By introducing prior vehicle parameters such as wheelbase, track width, and vehicle length, width, and height, and performing geometric reasoning on multiple ground points in the vehicle coordinate system, the remaining ground points can be completed and the vehicle center and orientation can be inferred even when only some ground points are obtained. This allows for the reliable generation of the vehicle's 3D bounding box even when the wheels are partially obscured or the field of view is incomplete, thus improving the applicability and stability of the method in complex traffic scenarios.

[0050] As one implementation method, in step (5) of this embodiment, when only two grounding points are detected... When calculating the Euclidean distance between two points... , And combined with the prior wheelbase of the vehicle model and wheelbase Set threshold : like The wheels were determined to be on the same side. like These are determined to be wheels on the same axle; This allows us to deduce the vehicle's center point and orientation, and complete the other two grounding points; Based on vehicle center and orientation angle Combined with vehicle body size Calculate the four corner points at the bottom of the 3D bounding box on the ground. ,satisfy:

[0051] Based on the four corner points on the ground, the Z-axis coordinates are added according to the height of the vehicle or pedestrian (the average value is determined according to the category) to construct the four top corner points, and finally a complete three-dimensional bounding box containing 8 corner points is obtained.

[0052] By comparing the geometric distance between the two grounding points with the prior wheelbase and track width of the vehicle model, it is still possible to distinguish between wheels on the same side and wheels on the same axle even in the extreme case where only two grounding points are detected. Based on this, the four corner points of the ground are completed and a complete three-dimensional bounding box is generated, so that this embodiment still has good robustness in scenarios with severe occlusion or limited field of view.

[0053] Example 2 like Figure 2 As shown, in order to solve one of the problems existing in the prior art, the present invention proposes a method for recognizing 3D bounding boxes of traffic participants that includes distance and orientation information, comprising the following steps.

[0054] I. Detecting bounding boxes and their categories for all traffic participants in an image based on YOLO.

[0055] The model uses a pre-trained YOLO object detection model (such as YOLOv5, YOLOv8, etc.) to infer from the input image, extracting the 2D bounding boxes and their corresponding category labels for all traffic participants in the image. These traffic participants may include, but are not limited to, pedestrians, passenger vehicles, non-motorized vehicles, and trucks. The set of 2D bounding boxes output by the model is represented as follows: Each bounding box Contains vectors: , respectively, represent: the pixel coordinates of the center point of the bounding box, the width and height of the bounding box, the target category (pedestrian, vehicle, etc.), and the confidence score.

[0056] 2. Detect the grounding point of traffic participants.

[0057] In order to accurately map the target from image space to the real ground coordinate system, it is necessary to determine its grounding point location in the image.

[0058] For pedestrians, human pose recognition models (such as HRNet and OpenPose) are used to detect key points on the human body, and the midpoint of both ankles is extracted as the grounding point. This method is highly adaptable to pose changes and is suitable for recognizing dynamic or partially occluded pedestrians. Let the coordinates of the left ankle be... The coordinates of the right ankle are If both points are visible, then the grounding point... Midpoint: .

[0059] For vehicle targets, a multi-level processing approach is adopted. First, for each vehicle, the Region of Interest (ROI) corresponding to its 2D bounding box is extracted. Within this ROI, the 2D bounding boxes of all wheels are detected using a wheel detection model (such as YOLOv5, YOLOv8, etc.). Then, for each wheel bounding box, an image patch is cropped, and a fine-grained grounding point localization network (such as the HMNet model based on heatmap regression) is used to output a low-resolution heatmap of the image within the 2D bounding box of the wheel, assuming the actual grounding point is... The target heatmap output by the network It follows a two-dimensional Gaussian distribution: To ensure that the positioning point is located at the actual contact point between the tire and the ground, a vertical weighting function is introduced during heatmap inference. Let the height of the wheel rim be... Within the image patch coordinate range The weighting function is defined as follows: ,in This is a gain factor (e.g., 0.5) that gives higher weight to pixels at the bottom of the image. The peak index of the heatmap corresponds to the grounding point location: The peak coordinates of the heatmap By adding the ROI offset and mapping it back to the pixel coordinates of the original image, the pixel coordinates of the visible wheel's contact point can be obtained. .

[0060] 3. Obtain the camera's intrinsic parameters and perform distortion correction on the image.

[0061] The camera's intrinsic parameters are obtained by calibrating a checkerboard image. The calibration results include: the transformation matrix K from the camera coordinate system to the pixel coordinate system and the camera distortion coefficient D.

[0062] For the grounding point detected in step two Perform distortion mapping (For example, using a fisheye camera to call the cv2.fisheye.undistortPoints() function in OpenCV) to obtain the corrected 2D points. This method only performs calculations on a finite set of key points, avoiding distortion correction of the entire image frame and reducing computational overhead.

[0063] 4. Obtain the homography matrix and calculate the actual coordinates of the ground plane of the ground point in the vehicle coordinate system.

[0064] By calibrating with a surround-view system, the projection homography matrix from the camera-distorted image plane to the ground plane is obtained. .

[0065] The corrected grounding point obtained in step three Using the projection homography matrix Projected onto the ground coordinate system (Z=0), the coordinates are... ,Right now: Specifically, By eliminating the scale factor Solve for the ground coordinates .

[0066] Based on a preset offset (rotation) between the ground coordinate system and the vehicle coordinate system (e.g., with the rear axle center as the origin, the front direction of the vehicle as the X-axis, and the left side of the vehicle as the Y-axis). Peaceful relocation ), calculate the ground plane coordinates of the grounding point in the vehicle coordinate system. .

[0067] 5. Use reasoning to complete the invisible ground points to obtain the 3D bounding box.

[0068] The 3D bounding box is a regular cube structure, with the bottom vertex derived from the contact points of traffic participants. Since pedestrians occupy a smaller area, they only need one ankle contact point. Vehicles, however, require multiple contact points.

[0069] For invisible grounding points of vehicles, the information is inferred and completed using vehicle structure information (model prior, such as wheelbase Lwb, track width Wtk, body length Lbody, body width Wbody, and body height Hbody) and visible grounding point information.

[0070] For targets like vehicles with multiple grounding points, if some grounding points are not visible and only 2 or 3 grounding points are detected, then inference is performed based on the vehicle structure and visible grounding point information to complete the data. A geometric inference method is used to complete the data for the fourth grounding point, obtaining the vehicle's center point and orientation, thus obtaining the bottom four corner points of the 3D bounding box on the ground. Let the set of detected and projected ground grounding points be... ,in These are the coordinates in the vehicle coordinate system.

[0071] Scenario A: Given three grounding points, the remaining fourth point can be uniquely determined using vector methods, thus obtaining the vehicle's center point and orientation. Based on the geometric properties of a rectangle, the fourth point is completed using vector addition. : Calculate the vehicle's ground center point: The vehicle's direction vector is determined using the slope of the line connecting two points on the same side and along the same axis. (Vertical) and (Horizontal).

[0072] Case B: Given two grounding points, determine the Euclidean distance between points P1 and P2. And based on the prior wheelbase and track width of the vehicle model, a threshold is set. This allows us to deduce whether the two points are on the same side or coaxial. If If they are determined to be wheels on the same side; The wheels are identified as coaxial. This allows us to determine the vehicle's center point and orientation. Then, using prior knowledge of the vehicle model, we complete the other two contact points through vector reasoning.

[0073] Based on vehicle center and orientation angle Combined with vehicle body size Calculate the four bottom corner points:

[0074] By using the bottom four corner points of the 3D bounding box on the ground, adding the height of the vehicle or pedestrian (the average value is determined according to the category), and adding the Z-axis coordinate, the top four corner points are constructed, and finally a complete 3D bounding box of traffic participants containing 8 corner points is obtained.

[0075] Compared with the prior art, this second embodiment has the following advantages: This second embodiment relies on a single vehicle-mounted camera to acquire image data. It can estimate the vehicle's three-dimensional bounding box, attitude, and distance through monocular vision. Compared with systems that require multi-sensor fusion solutions such as LiDAR, stereo cameras, or millimeter-wave radar, it significantly reduces the types and number of sensors, lowers the overall vehicle hardware cost, and simplifies the engineering complexity of multi-sensor calibration, synchronization, and maintenance, making it easier to promote and apply in mass-produced vehicles.

[0076] This second embodiment employs a two-stage processing strategy: first detecting the wheel bounding box, then precisely locating the contact point within a local area. This overcomes the inaccurate positioning problem caused by the traditional method of roughly estimating the bottom edge position based solely on a two-dimensional detection box. By applying a higher weight to the bottom region of the contact point heatmap, the response near the tire-ground interface is enhanced, thereby more accurately identifying the actual contact point between the tire and the ground. This effectively eliminates interference from structures such as the wheel hub and axle, providing more accurate and stable foundational data for subsequent coordinate transformation and three-dimensional bounding box construction.

[0077] This second embodiment explicitly considers situations where some wheels are obscured or grounding point information is incomplete. By introducing prior vehicle parameters such as wheelbase and track width, when only two grounding points can be detected, the remaining grounding points are inferred and completed based on the geometric relationship between the two points and the "same side / coaxial" judgment result. Even in extreme conditions such as only one side of the vehicle is observed or the wheels are severely obscured, the vehicle's three-dimensional bounding box with a complete structure and reasonable posture can still be reconstructed, thereby significantly improving the applicability and robustness of the method in complex road scenarios.

[0078] Based on Examples 1 and 2, a specific implementation case is provided: The camera is mounted on the top right side of the vehicle, and its recognition performance is as follows: Figure 3 As shown. Figure 3 Taking the car in the middle as an example, YOLO identifies it as a car with a confidence level of 92%. The pixel coordinates of the two wheel contact points on the image are (816,756) and (1312,772), and their ground plane coordinates in the vehicle's coordinate system are calculated to be (1.98,-2.11) m and (-0.76,-2.09) m. Based on the prior measurements of the vehicle model—a track width of 1.6 meters, a wheelbase of 2.8 meters, a length of 3.6 meters, and a width of 1.9 meters—it is determined that the two wheels touch down on the same side. The vehicle center point's coordinates on the ground plane in the vehicle coordinate system are calculated to be (0.60, -2.90) m, and the yaw angle of the front / rear of the vehicle is -0.42° (almost directly in front of the vehicle). Therefore, the coordinates of the four bottom corners of the 3D bounding box on the ground are (2.40, -3.86) m, (2.41, -1.96) m, (-1.20, -3.84) m, and (-1.19, -1.94) m. Since the height of a sedan is typically 1.6 m, the pose data of the 3D bounding box of this traffic participant can be calculated.

[0079] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting 3D bounding boxes of traffic participants under vehicle-mounted monocular vision, characterized in that: Includes the following steps: (1) Detecting the bounding boxes and categories of traffic participants based on the target detection model: Using a pre-trained target detection model, inference is performed on the images acquired by the vehicle-mounted monocular camera to detect the two-dimensional bounding boxes and their corresponding category labels of all traffic participants in the images; (2) Detecting the grounding points of traffic participants: For each of the two-dimensional bounding boxes, determine the pixel coordinates of one or more grounding points of the traffic participant in the image coordinate system within its corresponding image area; (3) Obtain camera intrinsic parameters and perform distortion correction processing on the docking location: obtain the camera intrinsic parameter matrix and camera distortion coefficient by calibrating the checkerboard image, perform distortion correction mapping on the grounding point pixel coordinates detected in step (2), and obtain the corrected two-dimensional grounding point. (4) Obtain the homography matrix and calculate the actual coordinates of the ground point on the ground plane in the vehicle coordinate system: Obtain the projection homography matrix from the camera distortion-free image plane to the ground plane through the surround view system calibration, project the corrected ground point onto the ground plane of the ground coordinate system to obtain the ground coordinates of the ground point in the ground coordinate system, and convert the ground coordinates into the ground plane coordinates of the ground point in the vehicle coordinate system according to the preset rotation and translation relationship between the ground coordinate system and the vehicle coordinate system. (5) Reasoning to complete the invisible grounding points and constructing a three-dimensional bounding box: The three-dimensional bounding box is a regular cube structure, and its bottom vertex is obtained by reasoning from the grounding points of traffic participants. For vehicle-type traffic participants, when only some grounding points are detected, the invisible grounding points are inferred and completed based on the vehicle structure information, vehicle type prior, and visible grounding point information to obtain the vehicle center point and orientation. Based on the vehicle center point, vehicle body size, and height information, a complete three-dimensional bounding box of traffic participants is constructed.

2. The method according to claim 1, characterized in that: The object detection model mentioned in step (1) is a pre-trained YOLO object detection model, and the set of two-dimensional bounding boxes output by the model is represented as follows: ; Each bounding box bi contains a vector: , These represent: the pixel coordinates of the bounding box center point (xc, yc), the width w and height h of the bounding box, the target category c, and the confidence score p, respectively.

3. The method according to claim 1 or 2, characterized in that: In step (2), for pedestrians, a human pose recognition model is used to detect key points on the human body, and the midpoint of the two ankles is extracted as the grounding point. Let the coordinates of the left ankle be... The coordinates of the right ankle are If both points are visible, then the grounding point is... Midpoint: 。 4. The method according to claim 1 or 2, characterized in that: In step (2), for vehicle-type targets, a multi-level processing method is used to determine the grounding point, including: (1) For each vehicle, extract the ROI corresponding to its two-dimensional bounding box, and detect the two-dimensional bounding boxes of all wheels within the ROI using the wheel detection model. (2) For each wheel bounding box, crop image blocks and use a grounding point fine localization network to output a low-resolution target heatmap of the image within the two-dimensional wheel bounding box. Assuming the actual grounding point is The target heat map It follows a two-dimensional Gaussian distribution: .

5. The method according to claim 4, characterized in that: To ensure that the positioning point is located at the actual contact position between the tire and the ground, a vertical weighting function is introduced during heatmap inference. Let the height of the wheel rim be... Within the image patch The coordinate range is The weighting function is defined as follows: Where α is the gain coefficient; the heat map prediction result is denoted as Mpred(x,y), and the peak index of the weighted heat map corresponds to the location of the grounding point. ,satisfy The peak coordinates By adding the ROI offset and mapping it back to the pixel coordinates of the original image, the pixel coordinates of the visible wheel's contact point can be obtained. .

6. The method according to claim 1, characterized in that: In step (3), the camera intrinsic parameters are obtained by calibrating the checkerboard image. The calibration results include: the transformation matrix K from the camera coordinate system to the pixel coordinate system, and the camera distortion coefficients D. The grounding point Pimg detected in step (2) is subjected to distortion correction mapping F^-1 to obtain the corrected two-dimensional point: .

7. The method according to claim 1 or 6, characterized in that: In step (4), the corrected grounding point is represented in homogeneous coordinates. The projection homography matrix from the camera-distorted image plane to the ground plane is obtained through the surround view system calibration. In the ground coordinate system, the coordinates of the grounding point are: ,satisfy: ,Right now: By eliminating scale factor Solve for the ground coordinates .

8. The method according to claim 7, characterized in that: In step (4), the preset rotation between the ground coordinate system and the vehicle coordinate system (e.g., with the rear axle center as the origin, the front direction of the vehicle as the X-axis, and the left side of the vehicle as the Y-axis) is used. Peaceful relocation Calculate the ground plane coordinates of the grounding point in the vehicle coordinate system: 。 9. The method according to claim 1, characterized in that: In step (5), for vehicle-type traffic participants, prior vehicle structure information (including wheelbase L) is used. wb Wheelbase W tk Vehicle length L body , body width W body Vehicle height H body The set of detected and projected ground grounding points is used for reasoning and completion, along with the information of visible grounding points. ,in The coordinates are in the vehicle coordinate system; When 3 grounding points are known At that time, based on the geometric properties of a rectangle, the fourth grounding point was completed using a vector method. : And calculate the vehicle's ground center point: The longitudinal direction vector of the vehicle is determined by the slope of the line connecting the grounding points on the same side and the line connecting the grounding points on the same axis. and horizontal direction vector This determines the vehicle's orientation.

10. The method according to claim 9, characterized in that: In step (5), when only two grounding points are detected When calculating the Euclidean distance between two points... And combined with the prior wheelbase of the vehicle model and wheelbase Set threshold : like The wheels were determined to be on the same side. like These are determined to be wheels on the same axle; This allows us to deduce the vehicle's center point and orientation, and complete the other two grounding points; Based on vehicle center and orientation angle Combined with vehicle body size Calculate the four corner points at the bottom of the 3D bounding box on the ground. ,satisfy: , Based on the four corner points on the ground, Z-axis coordinates are added according to the height of vehicles or pedestrians (the average value is determined according to the category) to construct the top four corner points, and finally a complete three-dimensional bounding box containing 8 corner points is obtained.

Images