A method, apparatus, domain controller, and medium for monocular vision-based 3D object detection

Abstract

This invention relates to the field of autonomous driving and discloses a monocular vision-based 3D object detection method, apparatus, domain controller, and medium. The method includes: training a target detection model using sample images, wherein each sample target in the sample images is marked with a 2D bounding box and its ground landing point; acquiring a target image and identifying the target in the target image using the target detection model, while simultaneously marking the ground landing point of the target; transforming the marked ground landing points to the vehicle coordinate system to obtain the physical landing points; and generating 3D detection information without limiting the height of the target using the physical landing points. This invention solves the problem that traditional 3D object detection schemes cannot simultaneously achieve low cost and high detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and specifically to a monocular vision three-dimensional object detection method, device, domain controller, and medium. Background Technology

[0002] 3D object detection is a key technology for autonomous driving perception systems. The position and 3D size of an object in the world coordinate system are important bases for functions such as collision warning, lane changing and obstacle trajectory prediction, and are crucial for subsequent decision-making and planning of intelligent driving vehicles.

[0003] Currently, 3D object detection technologies applied to intelligent driving vehicles are mainly divided into three schools of thought: LiDAR-based 3D object detection, 3D object detection based on the fusion of radar 3D and visual 2D, and 3D object detection based on pure vision. LiDAR-based 3D object detection is characterized by high accuracy and strong adaptability; however, identifying target categories and accurate 3D information requires LiDAR with high point cloud density. Whether mechanical or solid-state / semi-solid-state LiDAR, the configuration cost is high, and mechanical LiDAR also suffers from significant latency issues. 3D object detection schemes based on the fusion of radar 3D and visual 2D offer greater flexibility, with the highest upper limit under high-resolution sensor configurations, but the cost and complexity are also higher. Although it is possible to use low-resolution radar and camera 2D fusion, leveraging the high accuracy of radar in distance and orientation measurement and the advantages of camera in target recognition to achieve complementarity, the fusion of sensors with different frequencies increases the complexity of the detection algorithm and latency instability. 3D object detection based on pure vision can be divided into two categories: multi-view and monocular. Multi-view methods acquire images of the same object using multiple cameras at different angles. The depth information of the object can be calculated based on the differences in viewpoints between the images, thus enabling 3D object detection. However, multi-view methods require a large number of cameras, resulting in higher costs. Monocular methods, on the other hand, use only a single camera for 3D object detection. They offer advantages such as lower cost and relatively lower system complexity. However, a monocular camera provides a 2D image of the scene after perspective transformation, lacking scene depth information. Distance measurement accuracy is relatively low, and the complete 3D bounding box of the object can only be estimated from the 2D bounding box using a data-driven AI model. The estimated 3D information thus has poor stability.

[0004] Therefore, there is an urgent need for a three-dimensional object detection solution that is both low-cost and highly accurate. Summary of the Invention

[0005] In view of this, the present invention provides a monocular vision three-dimensional object detection method, device, domain controller and medium to solve the problem that traditional three-dimensional object detection schemes cannot simultaneously achieve low cost and high detection accuracy.

[0006] In a first aspect, the present invention provides a monocular vision three-dimensional object detection method, which obtains a target detection model through training sample images, wherein the sample targets in the sample images are marked with two-dimensional bounding boxes and the ground landing points of the sample targets; an image to be detected is acquired, and the target detection model is used to identify the target in the image to be detected, while marking the ground landing points of the target; the marked ground landing points are transformed into the vehicle coordinate system to obtain the physical landing points; and three-dimensional detection information without limiting the height of the target is generated through the physical landing points.

[0007] Optionally, the sample image is obtained as follows: The parameters of the monocular camera on the vehicle body are calibrated to obtain the inverse perspective transformation matrix of the monocular camera; an actual scene image is acquired using the monocular camera, and the two-dimensional bounding box of the sample target is marked in the actual scene image; the actual scene image is transformed into a bird's-eye view using the inverse perspective transformation matrix, and the four ground landing points and their order are marked in the bird's-eye view, where the order of the landing points is used to represent the front-back direction of the target; the bird's-eye view marked with the ground landing points is inversely transformed back to the actual scene image using the inverse perspective transformation matrix, obtaining the corresponding images of the four ground landing points in the actual scene image. Like coordinates, and simultaneously marking the visibility of ground landing points in the actual scene image; obtaining a public 3D detection dataset, and projecting the rectangular 3D bounding box of the sample target in the public 3D detection dataset onto a virtual 2D image at an arbitrary angle; marking the 2D bounding box of the sample target in the virtual 2D image using the projected rectangular 3D bounding box; projecting the four bottom corner points of the rectangular 3D bounding box as ground landing points onto the virtual 2D image to obtain the image coordinates of the four bottom corner points in the virtual 2D image, and simultaneously marking the order and visibility of the four bottom corner points; the sample image is composed of the marked actual scene image and the marked virtual 2D image.

[0008] Optionally, the step of calibrating the parameters of the monocular camera on the vehicle body to obtain the inverse perspective transformation matrix of the monocular camera includes: calibrating the distortion parameters of the monocular camera; acquiring a calibration image through the monocular camera and correcting the distortion of the calibration image; selecting no fewer than four ground pixels on the corrected calibration image, wherein the ground pixels are before the vanishing point and no more than two ground pixels are on the same straight line; measuring the physical coordinates of the ground points corresponding to the ground pixels in the vehicle body coordinate system; and solving the PnP problem using the ground pixels and the physical coordinates of the ground points to obtain the inverse perspective transformation matrix.

[0009] Optionally, the step of training the target detection model through sample images includes: using the two-dimensional bounding boxes, ground landing points, landing point order, and visibility marked in the sample images as labels, training a deep neural network through the sample images to obtain the target detection model used to identify two-dimensional bounding boxes, ground landing points, landing point order, and landing point visibility in two-dimensional images.

[0010] Optionally, generating three-dimensional detection information that does not limit the height of the detection target through the physical landing points includes: extracting visible landing points from the physical landing points, and calculating the length, width, center position, and orientation of the detection target through the order of the visible landing points and physical coordinates; generating a three-dimensional model that does not limit the height of the detection target through the length and width of the detection target; and composing the three-dimensional detection information through the three-dimensional model, the center position, and the orientation.

[0011] Optionally, the step of calculating the length, width, center position, and orientation of the detected target using the order and physical coordinates of the visible landing points includes: when there are at least three visible landing points, generating the long side and wide side of the detected target by connecting the visible landing points according to their order, and calculating the values ​​of the long side and wide side using the physical coordinates of the visible landing points respectively; determining the orientation of the detected target based on the order of the visible landing points and the long side; calculating the geometric center coordinates using the physical coordinates of the visible landing points as the center position; when there are two visible landing points, estimating a virtual landing point using the two visible landing points; and calculating the length, width, center position, and orientation of the detected target using the two visible landing points and the virtual landing point as three visible landing points.

[0012] Optionally, estimating a virtual landing point using two visible landing points includes: determining the positional relationship between the two visible landing points according to their order, wherein the positional relationship includes the two visible landing points being on the same side and the two visible landing points being diagonal points; when the two visible landing points are on the same side, calculating the value of the side using the physical coordinates of the two visible landing points; extending any visible landing point in an invisible direction perpendicular to the side according to the value of the side, and estimating the physical coordinates of the virtual landing point based on the endpoint of the extension; when the two visible landing points are diagonal points, using the invisible landing point closest to the camera's viewpoint among the physical landing points as the virtual landing point.

[0013] Secondly, the present invention provides a monocular vision three-dimensional object detection device, the device comprising: a model training module for training a target detection model through sample images, wherein sample targets in the sample images are marked with two-dimensional bounding boxes and ground landing points of the sample targets; a landing point recognition module for acquiring an image to be detected and identifying the target in the image to be detected through the target detection model, while marking the ground landing points of the target; a coordinate transformation module for transforming the marked ground landing points to a vehicle coordinate system to obtain physical landing points; and a three-dimensional information generation module for generating three-dimensional detection information without limiting the height of the target based on the physical landing points.

[0014] Thirdly, the present invention provides a domain controller, characterized in that it includes: a perception processing unit, a decision processing unit, a control processing unit, and a communication unit, wherein the perception processing unit, the decision processing unit, the control processing unit, and the communication unit are communicatively connected to each other, the perception processing unit stores computer instructions, and the perception processing unit executes the method of the first aspect or any corresponding embodiment by executing the computer instructions.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0016] The technical solution provided in this application has the following advantages:

[0017] The technical solution provided in this application creates a large number of sample images by labeling the two-dimensional bounding boxes and ground landing points of the sample targets in the sample images. Then, a target detection model is trained using these sample images. In the prediction stage, images to be detected are acquired using a monocular camera on the vehicle body. The target detection model then identifies the targets in the images and labels their ground landing points. These labeled ground landing points are then transformed into the vehicle body coordinate system to obtain the physical landing points. Finally, three-dimensional detection information is generated using the physical landing points, which only restricts the planar contours of the targets without restricting their height. The target detection model used in this application only needs to detect the two-dimensional bounding boxes of objects and identify the landing points of objects in the image, ignoring the object's height. The target detection model in this method does not need to directly estimate the depth of objects or scenes. The research and application of two-dimensional detection technology are relatively mature, ensuring prediction accuracy. This approach also reduces the difficulty of data labeling for three-dimensional object detection, allowing for the convenient and low-cost acquisition of a large amount of labeled data by modifying existing monocular three-dimensional object detection databases, thereby improving the accuracy of two-dimensional bounding box detection and landing point identification.

[0018] Furthermore, the estimation of the object's 3D information directly converts pixel coordinates to physical coordinates in the vehicle body coordinate system, reducing the coordinate system transformation process and allowing for better control of the overall 3D detection system error. The algorithm used in this embodiment does not depend on the camera's installation and configuration, exhibiting high versatility and practicality across different scenarios. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a monocular vision three-dimensional object detection method according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of a sample image according to an embodiment of the present invention;

[0022] Figure 3 This is a bird's-eye view schematic diagram according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of a monocular vision three-dimensional object detection device according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the structure of a domain controller according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The monocular vision-based 3D object detection method provided in this embodiment can be used in the aforementioned domain controller. Figure 1 This is a flowchart of a monocular vision three-dimensional object detection method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0027] Step S101: The target detection model is trained using sample images. The sample targets in the sample images are all marked with two-dimensional bounding boxes and the landing points of the sample targets on the ground.

[0028] Specifically, this invention provides a novel monocular vision detection method that does not require measuring the depth of an object in an image, thereby generating accurate 3D object information. First, this invention requires pre-training an object detection model. This model can be deployed using various deep neural networks, such as Mask R-CNN, YOLO series networks, etc., and is only an example, not a limitation. The trained object detection model is used to identify targets from a 2D image and mark the ground landing points of the targets. In this invention, the process of identifying targets involves selecting the target in the 2D image using a 2D bounding box, and then marking the ground landing points of the selected targets. The 2D bounding box includes, but is not limited to, circular bounding boxes, rectangular bounding boxes, triangular bounding boxes, or irregular bounding boxes that closely follow the target's contour. This invention uses a rectangular bounding box. A concrete example is the 2D bounding box detection part, which can employ a two-stage 2D detection network such as Faster R-CNN, CenterNet, Mask R-CNN, and their derivatives, or a one-stage 2D detection network such as SSD and YOLO. For the ground landing point part, the mask generation part of the classic Mask R-CNN network can be replaced by a ground landing point recognition network. It's important to note that the target's landing point on the ground is not necessarily the point where the target actually contacts the ground. Figure 2 As shown, a truck consists of a cab and a trailer. The trailer is considered a separate target, and the point where it actually contacts the ground is the tire. Obviously, this point cannot represent the trailer's outline in space. In this embodiment, the trailer's ground landing point is the projection point of its four vertices onto the ground, i.e., the projection point of the trailer's maximum outline. Therefore, in the drivable planar area, irregular contour changes in the height direction are ignored, and 3D detection information is generated based on the maximum outline projected onto the ground, ensuring absolute safety in the vehicle's path planning and speed planning schemes. Based on this, the sample targets in the sample images obtained in this step are all marked with 2D bounding boxes and the ground landing points of the sample targets. It should be noted that during the model training phase, the feature map range cropped from the sample images should be larger than the 2D bounding box range, because some ground landing points may fall outside the 2D bounding box due to perspective transformation. Generally, a size of 1.25 times the length and width of the 2D bounding box can be used.

[0029] In some optional embodiments, the sample images provided in the embodiments of the present invention are obtained in the following manner:

[0030] Step a1: Perform parameter calibration on the monocular camera on the vehicle body to obtain the inverse perspective transformation matrix of the monocular camera.

[0031] Step a2: Acquire images of the actual scene using a monocular camera and mark the two-dimensional bounding boxes of the sample targets in the actual scene images.

[0032] Step a3: Transform the actual scene image into a bird's-eye view using an inverse perspective transformation matrix, and mark the four ground landing points and their order in the bird's-eye view. The order of the landing points is used to characterize the front and back directions of the target.

[0033] Step a4: Transform the bird's-eye view marked with ground landing points into the actual scene image using the inverse perspective transformation matrix to obtain the image coordinates of the four ground landing points in the actual scene image, and mark the visibility of the ground landing points in the actual scene image.

[0034] Step a5: Obtain the publicly available 3D detection dataset and project the rectangular 3D bounding boxes of the sample targets in the publicly available 3D detection dataset onto a virtual 2D image at any angle.

[0035] Step a6: Mark the two-dimensional bounding box of the sample target in the virtual two-dimensional image using a projected rectangular three-dimensional bounding box.

[0036] Step a7: Project the four bottom corner points of the rectangular 3D border as ground landing points onto the virtual 2D image to obtain the image coordinates of the four bottom corner points in the virtual 2D image, and mark the order and visibility of the four bottom corner points.

[0037] Step a8: The sample image is composed of the labeled actual scene image and the labeled virtual two-dimensional image.

[0038] Specifically, in this embodiment of the invention, the sample images are mainly obtained through two methods: images actually captured by the vehicle's monocular camera and publicly available 3D detection datasets.

[0039] To ensure the accuracy of ground landing point labeling for images actually acquired by a monocular camera, the camera's parameters are first calibrated, including: 1. Calibrating the camera's intrinsic and distortion parameters using a calibration board; 2. Performing distortion correction on the images; 3. Calculating the inverse perspective transformation matrix of the monocular camera (the inverse perspective transformation matrix can be obtained by solving the PnP problem using physical points in world coordinates and pixels in image coordinates). Afterwards, images of the actual scene are acquired using the vehicle's monocular camera, and the two-dimensional bounding boxes of the sample targets are marked in the actual scene images, such as... Figure 2 As shown. Then, as... Figure 3As shown, the actual scene image is transformed into a bird's-eye view through an inverse perspective transformation matrix. The bird's-eye view has a one-to-one correspondence with the ground physical coordinates, thus marking the four ground landing points of the sample target in the bird's-eye view. This ensures that the marked points match the physical coordinates and that they always land on the ground, not suspended in mid-air. Furthermore, in some application scenarios, 3D object detection requires not only obtaining the object's 3D contour in space but also its orientation. For example, in one scenario, a vehicle uses a monocular camera to detect the 3D contours and orientations of other vehicles to predict their travel routes and determine whether a collision is likely. Based on this, this embodiment also marks the order of the landing points to characterize the target's front-to-back direction, facilitating further determination of the object's orientation in some optional implementations. For example, the user marks four ground landing points counter-clockwise according to four preset sequential labels: left front, right front, right rear, and left rear, setting left front and right front as the front of the sample target, and right rear and left rear as the rear of the sample target. Next, the bird's-eye view marked with ground landing points is transformed back to the actual scene image using an inverse perspective transformation matrix, yielding the image coordinates of the four ground landing points in the actual scene image. This embodiment, through the calculation method of the inverse perspective transformation matrix, eliminates the need for mutual conversion between the global coordinate system and the vehicle coordinate system; it directly calculates the actual ground global coordinates into the vehicle coordinate system using only the image captured by the vehicle's camera. Furthermore, this embodiment also marks the visibility of the ground landing points in the actual scene image. Visibility refers to whether the marked ground landing points are obscured by other objects, such as… Figure 2 The ground landing point in the middle left rear is blocked by the tire, so this ground landing point is marked as invisible, while other ground landing points are marked as visible. By marking the visibility, some optional implementations can generate three-dimensional detection information of the object based on the visible ground landing points with higher confidence, thereby further improving the accuracy of the three-dimensional detection information.

[0040] For publicly available 3D detection datasets, such as the Kitti and Nuscenes datasets for autonomous driving, these datasets can be viewed 360 degrees in physical world coordinates, and the targets in them are all annotated with rectangular 3D bounding boxes using high-line-count LiDAR. In this embodiment of the invention, the rectangular 3D bounding boxes in the publicly available 3D detection dataset are projected onto a virtual 2D image at any angle, thereby quickly expanding the number of sample images. Then, the 2D bounding boxes of the sample targets are marked in the virtual 2D image using the projected rectangular 3D bounding boxes. For example, the rectangular 3D bounding boxes of the sample targets in the publicly available 3D detection dataset are projected onto a plane at any angle; this plane is the virtual 2D image. The eight vertices of the rectangular 3D bounding boxes are projected onto the virtual 2D image, and then a minimum bounding box is taken as the annotation of the 2D bounding box, which can encompass the projection of the eight vertices. The rectangular 3D bounding boxes in the publicly available 3D detection dataset are in real world coordinates and usually contain all the components of the object. Therefore, the four vertices of the bottom surface of the rectangular 3D bounding box are equivalent to four landing corners, and their projections on the image can be used as the four ground landing points required by this embodiment of the invention. This approach allows for the rapid acquisition of a large number of accurate annotations from publicly available datasets. Considering that the target detection model trained in this embodiment is purely 2D detection and its prediction is independent of sensor parameter configuration, data from publicly available 3D detection datasets can be incorporated into the training without being affected by the scene. Furthermore, the order and visibility of the four bottom corner points in the virtual 2D image are also labeled to facilitate the determination of the orientation of 3D objects in some optional implementations. Additionally, 3D detection information of the object is generated based on the visible ground landing points with higher confidence, further improving the accuracy of the 3D detection information.

[0041] Finally, the above two types of samples are used to create a sample image that includes both the target's two-dimensional bounding box and the target's ground landing point.

[0042] In some alternative implementations, step a1 above includes:

[0043] Step b1: Calibrate the distortion parameters of the monocular camera.

[0044] Step b2: Acquire calibration images using a monocular camera and perform distortion correction on the calibration images.

[0045] Step b3: Select no fewer than four ground pixels on the calibrated image after correction. The ground pixels should be before the vanishing point and no more than two ground pixels should be on the same straight line.

[0046] Step b4: Measure the physical coordinates of the ground point corresponding to the ground pixel in the vehicle coordinate system.

[0047] Step b5: Solve the PnP problem using ground pixels and the physical coordinates of ground points to obtain the inverse perspective transformation matrix.

[0048] Specifically, to further improve the accuracy of the inverse perspective transformation matrix, this embodiment modifies the calculation process of the inverse perspective transformation matrix. Steps b1 and b2 are existing technologies used to improve the accuracy of the calibration image, and their principles will not be elaborated further. Since the main purpose of the inverse perspective transformation matrix in this embodiment is to determine the landing point of the spatial contour of any object on the ground, the pixels selected on the corrected calibration image in this embodiment must be limited to the ground to reduce the influence of any pixel in the image (e.g., a pixel whose actual position is suspended) on the ground coordinate transformation. In addition, considering that the goal of solving the inverse perspective transformation matrix is ​​to obtain the transformation between the ground plane and the image by solving the transformation between point pairs, which is a transformation between surfaces, the selected marker points need to form a plane as much as possible for the transformation to be accurate. If all marker points are on a straight line, the transformation between point pairs is not unique and cannot accurately reflect the transformation of the plane, so it cannot be used for the transformation calculation of other points. Therefore, no less than four ground pixels are selected on the corrected calibration image, and the ground pixels are before the vanishing point of the ground (ensuring that the ground pixels do not overlap), and no more than two ground pixels are on the same straight line. Next, the physical coordinates of the ground points corresponding to the ground pixels are measured in the vehicle coordinate system. Then, the PnP problem is solved using the ground pixels and the physical coordinates of the ground points to obtain a more accurate inverse perspective transformation matrix.

[0049] In some optional embodiments, based on the above-described method of acquiring sample images, step S101 specifically includes:

[0050] Step c1: Using the 2D bounding boxes, ground landing points, landing point order, and visibility marked in the sample images as labels, train a deep neural network through the sample images to obtain a target detection model for recognizing 2D bounding boxes, ground landing points, landing point order, and landing point visibility in 2D images.

[0051] Specifically, the above sample images can be used to train a target detection model for recognizing two-dimensional bounding boxes, ground landing points, landing point order, and landing point visibility in two-dimensional images. The training process is existing technology, and the improvement lies in the different training samples. The principle of the samples is described in the above embodiments and will not be repeated here.

[0052] Step S102: Obtain the image to be detected, and identify the target in the image to be detected using the target detection model, while marking the ground landing point of the target.

[0053] Specifically, in the actual inference phase, the image to be detected is captured in real time by a monocular camera on the vehicle body. Then, the target detection model trained in step S101 above identifies the target in the image to be detected, selects the target by a two-dimensional bounding box, and marks the ground landing points of the target. In addition, in a specific embodiment, in order to reduce the complexity of the ground landing point annotation, four ground landing points are used, that is, the landing points of the four bottom corners of the smallest circumscribed cube of the object on the ground.

[0054] In some alternative implementations, the target detection model trained by step c1 above identifies the image to be detected and can obtain the two-dimensional bounding box, ground landing point, landing point order and landing point visibility of the target from the image to be detected, so that subsequent steps can determine the three-dimensional contour, orientation and position of the target.

[0055] Step S103: Convert the marked ground landing points to the vehicle coordinate system to obtain the physical landing points.

[0056] Specifically, after marking the landing points of the target in the image to be detected through the above steps, these landing points are still in the image coordinate system. At this point, it is necessary to transform the ground landing points to the vehicle coordinate system to obtain the physical landing points. Only then can useful 3D detection information be generated in the space around the vehicle based on the physical landing points, in order to facilitate vehicle path planning. The transformation of the ground landing points to the vehicle coordinate system can be achieved by coordinate system transformation between the image coordinate system and the vehicle coordinate system.

[0057] In some alternative implementations, the marked ground landing points can also be transformed into physical landing points using the inverse perspective transformation matrix provided in the above embodiments. There is no need to convert between the global coordinate system and the vehicle coordinate system. The actual ground global coordinates can be directly calculated into the vehicle coordinate system using the image captured by the vehicle camera, thereby improving the conversion efficiency.

[0058] Step S104: Generate three-dimensional detection information without limiting the height of the target by using physical landing points.

[0059] Specifically, finally, this embodiment of the invention generates three-dimensional detection information that does not limit the height of the target based on the physical landing point determined in the above embodiments. For example: Figure 2After converting the four ground landing points shown into physical landing points, the height of the object is ignored. It is assumed that the line connecting the four physical landing points forms an infinitely high plane, thus obtaining a three-dimensional model with infinite height. Considering that vehicle path planning treats the road surface as a plane, similar to a port scenario where the drivable area is flat, autonomous driving does not need to consider the height of obstacles ahead; regardless of the obstacle's height, it must be avoided. Therefore, the height of the detected target can be considered infinitely high. Thus, this embodiment only needs to detect the four landing corners of the target with a height of 0 to obtain sufficient information for planning and control. Simultaneously, the target detection model included in the method does not need to directly estimate the depth of the object or scene; it only needs to detect the ground landing points. The research and application of two-dimensional detection technology are relatively mature, ensuring prediction accuracy. No additional cameras or radar are needed. Therefore, this application provides a monocular vision three-dimensional object detection method that can balance low production cost and high accuracy.

[0060] In some alternative implementations, step S104 includes:

[0061] Step d1: Extract the visible landing points from the physical landing points, and calculate the length, width, center position and orientation of the detected target based on the order of the visible landing points and the physical coordinates;

[0062] Step d2: Generate a 3D model of the detected target without limiting its height by detecting its length and width;

[0063] Step d3 involves using the 3D model, center position, and orientation to form 3D detection information.

[0064] Specifically, the target detection model trained according to step c1 above identifies the image to be detected. Besides marking the four ground landing points of the target, it can also output the order and visibility of these landing points. For example, during annotation, visibility is added to each landing point. The visibility prediction result output by the target detection model through the activation function is a score in the range of 0 to 1, representing a measure of visibility and also a measure of the confidence of the landing point detection. For example, a score greater than 0.5 indicates that the landing point is visible. Once a landing point is obstructed, the accuracy of the predicted location using image information decreases. Using this method indiscriminately will lead to inaccurate target location. Therefore, this embodiment extracts visible landing points from the physical landing points based on their visibility (e.g., landing points with a score greater than a threshold of 0.5; the threshold needs to be determined based on the actual application scenario, this is just an example and not a limitation). Then, the forward and backward direction and the edge of the target are determined by the order of the visible landing points. Finally, the length, width, center position (the center position refers to the physical coordinates of the target's center), and orientation of the detected target are calculated using the physical coordinates of each visible landing point. This allows for the generation of a 3D model that does not limit the height of the detected target by measuring its length and width. Furthermore, the 3D model, its center position, and its orientation are used to construct 3D detection information, thereby improving the accuracy of the 3D detection information.

[0065] In some alternative implementations, step d1 above includes:

[0066] Step e1: When the number of visible landing points is no less than three, the long side and wide side of the detected target are generated by connecting the visible landing points in order, and the values ​​of the long side and wide side are calculated by using the physical coordinates of the visible landing points respectively.

[0067] Step e2: Determine the orientation of the target to be detected based on the order of the visible landing points and the long side.

[0068] Step e3: Calculate the geometric center coordinates using the physical coordinates of the visible landing point, and use them as the center position.

[0069] Step e4: When there are two visible landing points, estimate a virtual landing point using the two visible landing points.

[0070] Step e5: Calculate the length, width, center position, and orientation of the detected target by using the two visible landing points and the virtual landing point as three visible landing points.

[0071] Specifically, the number of visible landing points is generally divided into two cases: one is that there are no fewer than three visible landing points (usually three), and the other is that there are only two visible landing points. Therefore, for the above two cases, this embodiment uses two types of processes to determine the length, width, center position and orientation of the detection target, so as to improve the accuracy of the three-dimensional detection information.

[0072] When there are at least three visible landing points, there is more information about them, making it easier to calculate the 3D model, center position, and orientation. Specifically, based on the order of the visible landing points, the long and wide sides of the detected target are obtained by connecting them. The values ​​of the long and wide sides are then calculated using the physical coordinates of the visible landing points. For example, there are at least three visible landing points: left front, right front, and right rear. Based on the order of these three landing points, connecting the left front and right front points gives the wide side of the detected target, and connecting the right front and right rear points gives the long side. Since the visible landing points have been transformed into the vehicle body coordinate system, their physical coordinates are known. The specific values ​​of the long and wide sides can be calculated using these coordinates. The long and wide sides are then used to combine and generate a 3D model with no height restrictions in space. Next, the orientation of the detected target is determined based on the order and long side of the visible landing points. For example, if the order of the visible landing points is left front, right front, and right rear, the long side of the line connecting right front and right rear can determine the angle of the detected target in the vehicle coordinate system. The names right front and right rear can be used to determine the front and rear of the detected target. For example, right rear and left rear are predefined as the orientation of the detected target, thus determining the orientation of the detected target based on the order and long side of the visible landing points. If the detected target is a car, the future driving direction of the car can be predicted based on the orientation, determining whether the detected target will collide with the vehicle, thus making sufficient preparations for the vehicle's path planning. Finally, the geometric center coordinates are calculated using the physical coordinates of the visible landing points as the center position, thereby determining the accurate position of the 3D model in the vehicle coordinate system. For example, three consecutive visible landing points from left to right are p1, p2, and p3, with corresponding physical coordinates of (x1, y1), (x2, y2), and (x3, y3), respectively. The x-axis points towards the front of the vehicle, and the y-axis points towards the left side of the vehicle. The coordinates of the center of the bottom surface can be represented as ((x1+x3) / 2, (y1+y2+y3) / 3, 0). Point p2 should be the point closest to the longitudinal direction of the vehicle, and points p1 and p3 are to the left and right of point p2, respectively.

[0073] When there are only two visible landing points, this embodiment of the invention estimates a virtual landing point using the two visible landing points, and then uses the two visible landing points and the virtual landing point as three visible landing points. The length, width, center position, and orientation of the detected target are calculated following the steps described above for the three visible landing points. For example, an invisible landing point can be introduced as a virtual landing point for calculation. It should be noted that invisible landing points have varying confidence levels; therefore, in this embodiment, one invisible landing point is introduced as a virtual landing point to avoid multiple invisible landing points affecting the accuracy of the 3D detection information.

[0074] Specifically, in one embodiment, estimating a virtual landing point using two visible landing points is further divided into two cases: one where both visible landing points are on the same side, and the other where the two visible landing points are diagonal points on opposite sides of a diagonal. Therefore, step e4 above includes:

[0075] Step f1: Determine the positional relationship between the two visible landing points based on their order. The positional relationship includes the two visible landing points being on the same side and the two visible landing points being diagonal points.

[0076] Step f2: When both visible landing points are on the same side, calculate the value of the side using the physical coordinates of the two visible landing points.

[0077] Step f3: Extend the virtual landing point in an invisible direction from any visible landing point according to the value of one side perpendicular to the one side, and estimate the physical coordinates of the virtual landing point based on the endpoint of the extension.

[0078] Step f4: When two visible landing points are diagonal, the invisible landing point that is closest to the camera's viewpoint among the physical landing points is used as the virtual landing point.

[0079] Specifically, when two visible landing points are both on a single side, such as when the line connecting the two visible landing points is the longer or shorter side, the corresponding viewpoint of the monocular camera is that the vehicle is directly in front of, behind, to the left of, or to the right of the target. Therefore, the physical coordinates of the two visible landing points are used to calculate the value of one side. Then, using the value of one side, an extension is made perpendicular to the side from any visible landing point in an invisible direction, and the physical coordinates of the virtual landing point are estimated based on the endpoint of the extension. For example, if the line connecting the two visible landing points is the longer side, and the vehicle is to the left of the longer side, the right side of the longer side is invisible to the vehicle. Therefore, starting from any visible landing point, an extension is made perpendicular to the side for the same length, and the endpoint is a virtual landing point. In other words, the shape formed by the virtual landing point and the visible landing point on the ground is a square. Then, by combining the physical coordinates of each landing point according to the calculation process for three visible landing points, the length, width, center position, orientation, and 3D model of the target can be determined. On the one hand, the method for estimating landing points is simple and easy to implement, and has high computational efficiency; on the other hand, regardless of whether the invisible landing points identified by the target detection model are accurate, they can be disregarded, reducing the complexity of the analysis. Since the other two invisible landing corners are behind the field of view, the virtual landing points estimated in this way will not affect driving safety.

[0080] When the two visible landing points are diagonal points, for example: assuming Figure 2A person is standing in front of the right front landing point, the right front landing point is invisible, and the left rear landing point is blocked by a tire, leaving only the left front and right rear landing points visible. In this scenario, a square cannot be used to estimate the virtual landing points. This embodiment of the invention uses the invisible landing point closest to the camera's viewpoint among the physical landing points as the virtual landing point. Because the invisible landing point closest to the camera's viewpoint is relatively close, it usually has less occlusion and higher confidence. Therefore, using the invisible landing point closest to the camera's viewpoint as the virtual landing point can obtain the most accurate 3D detection information even under the worst detection conditions.

[0081] The technical solution provided in this application creates a large number of sample images by labeling the two-dimensional bounding boxes and ground landing points of the sample targets in the sample images. Then, a target detection model is trained using these sample images. In the prediction stage, images to be detected are acquired using a monocular camera on the vehicle body. The target detection model then identifies the targets in the images and labels their ground landing points. These labeled ground landing points are then transformed into the vehicle body coordinate system to obtain the physical landing points. Finally, three-dimensional detection information is generated using the physical landing points, which only restricts the planar contours of the targets without restricting their height. The target detection model used in this application only needs to detect the two-dimensional bounding boxes of objects and identify the landing points of objects in the image, ignoring the object's height. The target detection model in this method does not need to directly estimate the depth of objects or scenes. The research and application of two-dimensional detection technology are relatively mature, and the prediction accuracy is guaranteed. This reduces the difficulty of data labeling for three-dimensional object detection and allows for the convenient and low-cost acquisition of a large amount of labeled data by modifying existing monocular three-dimensional object detection databases, thereby improving the accuracy of two-dimensional bounding box detection and landing point identification.

[0082] This embodiment provides a monocular vision three-dimensional object detection device, such as... Figure 4 As shown, it includes:

[0083] The model training module 401 is used to train the target detection model using sample images. All target objects in the sample images are marked with two-dimensional bounding boxes and their landing points on the ground. For details, please refer to the relevant description of step S101 in the above method embodiment; it will not be repeated here.

[0084] The landing point recognition module 402 is used to acquire the image to be detected, identify the target in the image using a target detection model, and simultaneously mark the ground landing points of the detected targets. For details, please refer to the relevant description of step S102 in the above method embodiment, which will not be repeated here.

[0085] The coordinate transformation module 403 is used to transform the marked ground landing points to the vehicle coordinate system to obtain the physical landing points. For details, please refer to the relevant description of step S103 in the above method embodiment, which will not be repeated here.

[0086] The 3D information generation module 404 is used to generate 3D detection information without limiting the height of the detection target based on the physical landing point. For details, please refer to the relevant description of step S104 in the above method embodiment, which will not be repeated here.

[0087] In this embodiment, a monocular vision three-dimensional object detection device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0088] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0089] This invention also provides a domain controller having the above-described features. Figure 4 The diagram shows a path planning parameter tuning device.

[0090] Figure 5 This invention illustrates a domain controller according to an embodiment of the present invention. The domain controller includes at least a perception processing unit 901, a decision processing unit 902, a control processing unit 903, and a communication unit 904. The perception processing unit 901, the decision processing unit 902, the control processing unit 903, and the communication unit 904 can communicate with each other via a bus or other means. Figure 5 Taking the bus method as an example.

[0091] In this embodiment, the perception processing unit 901 and the decision processing unit 902 each include an independent processor. The perception processing unit 901 and the decision processing unit 902 may each include an independent memory, or they may use a shared memory.

[0092] In this embodiment of the invention, the perception processing unit 901 is mainly applied to engineering machinery scenarios. Its main function is to perform perception fusion processing on sensor data to obtain environmental information of the current environment of the engineering machinery, and then send the environmental information to the control processing unit 903 or the decision processing unit 902 according to the data type of the environmental information signal.

[0093] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above method embodiments. The perception processing unit 901 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above method embodiments.

[0094] The function of the decision processing unit 902 is to formulate driving or operation strategies by combining information such as the surrounding environment, operation scenario, and vehicle status, and finally issue control commands.

[0095] The main function of the control processing unit 903 is to convert between different types of signals, such as communication protocol conversion (CAN, Ethernet, LIN, etc.), AD conversion (sensor input), and DA conversion (control drive). For example, to convert the signal scanned by the LiDAR into point cloud data, the control processing unit 903 can be an MCU with chips such as the Texas Instruments (TI) TDA4VM, Mobileye's EyeQ series, Renesas' R-CAR H3, or Horizon Robotics' Journey series.

[0096] The main function of the communication unit 904 is to conduct wireless communication, including but not limited to 5G / 4G network communication, Wi-Fi communication, and satellite communication, and to communicate with the cloud server. Its main functions include uploading device-related status and information to the cloud service, requesting the cloud server to assist in calculation and processing, downloading data from the cloud server, and performing OTA software upgrades on the controller. It also communicates with nearby devices, receiving status information from other devices and collaboratively completing tasks. The communication unit 110 of the control module can be a 5G module, Wi-Fi module, Bluetooth module, etc.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The implemented program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0098] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A monocular vision-based three-dimensional object detection method, characterized in that, The method includes: The target detection model is obtained by training on sample images, wherein the sample targets in the sample images are all marked with two-dimensional bounding boxes and the landing points of the sample targets on the ground; The image to be detected is acquired, and the target detection model is used to identify the target in the image to be detected, while marking the ground landing point of the target. Transform the marked ground landing points into the vehicle coordinate system to obtain the physical landing points; Three-dimensional detection information is generated from the physical landing point without limiting the height of the detection target; The sample images were obtained in the following manner: The parameters of the monocular camera on the vehicle body are calibrated to obtain the inverse perspective transformation matrix of the monocular camera; The monocular camera acquires images of the actual scene, and the two-dimensional bounding boxes of the sample targets are marked in the actual scene images; The actual scene image is transformed into a bird's-eye view using the inverse perspective transformation matrix, and the four ground landing points and their order are marked on the bird's-eye view. The order of the landing points is used to characterize the front and back directions of the target. The bird's-eye view marked with ground landing points is transformed into the actual scene image by the inverse perspective transformation matrix, and the image coordinates of the four ground landing points in the actual scene image are obtained. At the same time, the visibility of the ground landing points is marked in the actual scene image. Obtain a publicly available 3D detection dataset, and project the rectangular 3D bounding boxes of the sample targets in the publicly available 3D detection dataset onto a virtual 2D image at an arbitrary angle; In the virtual two-dimensional image, the two-dimensional border of the sample target is marked by a projected rectangular three-dimensional border; The four bottom corner points of the rectangular three-dimensional border are projected onto the virtual two-dimensional image as ground landing points to obtain the image coordinates of the four bottom corner points in the virtual two-dimensional image, and the order and visibility of the four bottom corner points are marked at the same time. The sample image is composed of a labeled real-world scene image and a labeled virtual two-dimensional image; The target detection model obtained by training sample images includes: Using the two-dimensional bounding boxes, ground landing points, landing point order, and visibility marked in the sample images as labels, a deep neural network is trained through the sample images to obtain the target detection model used to identify two-dimensional bounding boxes, ground landing points, landing point order, and landing point visibility in two-dimensional images. The process of generating three-dimensional detection information that does not limit the height of the target through the physical landing point includes: Extract the visible landing points from the physical landing points, and calculate the length, width, center position and orientation of the detected target based on the order of the visible landing points and the physical coordinates; A three-dimensional model without limiting the height of the detected target is generated based on the length and width of the detected target; The three-dimensional detection information is composed of the three-dimensional model, the center position, and the orientation. The calculation of the length, width, center position, and orientation of the detected target based on the order and physical coordinates of the visible landing points includes: When the number of visible landing points is not less than three, the long side and wide side of the detected target are generated by connecting the visible landing points in order, and the values ​​of the long side and wide side are calculated by the physical coordinates of the visible landing points respectively. The orientation of the detected target is determined based on the order of the visible landing points and the longer side; The geometric center coordinates are calculated using the physical coordinates of the visible landing point and used as the center position. When there are two visible landing points, a virtual landing point is estimated using the two visible landing points. The length, width, center position, and orientation of the detected target are calculated by using the two visible landing points and the virtual landing point as three visible landing points.

2. The method according to claim 1, characterized in that, The parameter calibration of the monocular camera on the vehicle body to obtain the inverse perspective transformation matrix of the monocular camera includes: The distortion parameters of the monocular camera are calibrated; The calibration image is acquired by the monocular camera, and the distortion of the calibration image is corrected. Select no fewer than four ground pixels on the calibrated image after correction. The ground pixels are located before the vanishing point and no more than two ground pixels are on the same straight line. Measure the physical coordinates of the ground point corresponding to the ground pixel in the vehicle coordinate system; The inverse perspective transformation matrix is ​​obtained by solving the PnP problem using the ground pixels and the physical coordinates of the ground points.

3. The method according to claim 1, characterized in that, The method of estimating a virtual landing point using two visible landing points includes: The positional relationship between two visible landing points is determined based on the order of the two visible landing points. The positional relationship includes the two visible landing points being on the same side and the two visible landing points being diagonal points. When two visible landing points are both on one side, the value of one side is calculated using the physical coordinates of the two visible landing points; By extending the virtual landing point in an invisible direction from any visible landing point, perpendicular to the value of one side, and estimating the physical coordinates of the virtual landing point based on the endpoint of the extension; When two visible landing points are diagonally opposite, the invisible landing point that is closest to the camera's viewpoint among the physical landing points is taken as the virtual landing point.

4. A monocular vision three-dimensional object detection device, characterized in that, The device includes: The model training module is used to train an object detection model using sample images. Each sample object in the sample images is marked with a 2D bounding box and its ground landing points. The sample images are acquired as follows: The parameters of the monocular camera on the vehicle body are calibrated to obtain the inverse perspective transformation matrix of the monocular camera; Real-world scene images are acquired using the monocular camera, and the 2D bounding boxes of the sample objects are marked in the real-world scene images; The real-world scene images are transformed into a bird's-eye view using the inverse perspective transformation matrix, and the four ground landing points and their order are marked in the bird's-eye view, where the order of the landing points represents the front-back direction of the object; The bird's-eye view marked with the ground landing points is inversely transformed into a [missing information - likely a specific image or model] using the inverse perspective transformation matrix. The process involves: obtaining the image coordinates of four ground landing points in a real-world scene image and marking their visibility; acquiring a publicly available 3D detection dataset and projecting the rectangular 3D bounding boxes of the sample targets in the dataset onto a virtual 2D image at an arbitrary angle; marking the 2D bounding boxes of the sample targets in the virtual 2D image using the projected rectangular 3D bounding boxes; projecting the four bottom corner points of the rectangular 3D bounding boxes as ground landing points onto the virtual 2D image to obtain their image coordinates, and marking their order and visibility; and finally, combining the marked real-world scene image and the marked virtual 2D image to form the sample image. The process of training the target detection model using sample images includes: using the two-dimensional bounding boxes, ground landing points, landing point order, and visibility marked in the sample images as labels, training a deep neural network using the sample images to obtain the target detection model used to identify two-dimensional bounding boxes, ground landing points, landing point order, and landing point visibility in two-dimensional images. The landing point recognition module is used to acquire the image to be detected, and to identify the target in the image to be detected through the target detection model, while marking the ground landing point of the target; The coordinate transformation module is used to transform the marked ground landing points to the vehicle coordinate system to obtain the physical landing points; A 3D information generation module is used to generate 3D detection information that does not limit the height of the detection target based on the physical landing points. The generation of 3D detection information that does not limit the height of the detection target based on the physical landing points includes: extracting visible landing points from the physical landing points, and calculating the length, width, center position, and orientation of the detection target based on the order and physical coordinates of the visible landing points; generating a 3D model that does not limit the height of the detection target based on the length and width of the detection target; and assembling the 3D detection information based on the 3D model, the center position, and the orientation. The step of calculating the length, width, center position, and orientation of the detected target using the order and physical coordinates of the visible landing points includes: when there are at least three visible landing points, connecting the visible landing points in order to generate the long side and wide side of the detected target, and calculating the values ​​of the long side and wide side using the physical coordinates of the visible landing points respectively; determining the orientation of the detected target based on the order of the visible landing points and the long side; calculating the geometric center coordinates using the physical coordinates of the visible landing points as the center position; when there are two visible landing points, estimating a virtual landing point using the two visible landing points; and using the two visible landing points and the virtual landing point as three visible landing points to calculate the length, width, center position, and orientation of the detected target.

5. A domain controller, characterized in that, include: The system comprises a perception processing unit, a decision processing unit, a control processing unit, and a communication unit, wherein the perception processing unit, the decision processing unit, the control processing unit, and the communication unit are interconnected and communicate with each other. The perception processing unit stores computer instructions, and the perception processing unit executes the computer instructions to perform the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 3.

Images