Monocular camera ranging method and computer program product

Abstract

The application provides a ranging method of a monocular camera and a computer program product. The ranging method of the monocular camera comprises the following steps: acquiring a detection image and internal parameters of a camera; determining an initial ranging value, a detection frame of a target object and a real size of the target object according to the detection image; determining a direction angle of the target object according to the initial ranging value and the detection image; determining three-dimensional coordinates of a contour of the target object according to the real size, the initial ranging value and the direction angle; generating a projection area of the target object according to the three-dimensional coordinates of the contour of the target object and the internal parameters; and determining a final ranging value according to the projection area and the detection frame. The ranging method of the embodiment can adjust the monocular camera in real time through the determined final ranging value, is convenient and fast, has high reliability, and improves the safety of vehicle driving.

Description

Technical Field

[0001] This application relates to the technical field of vehicle ranging, and more particularly to a ranging method and computer program product using a monocular camera. Background Technology

[0002] In advanced driver assistance systems (ADAS), vehicles need to be able to perceive their surroundings using information obtained from sensors, such as determining the distance between the vehicle and surrounding obstacles. Common methods for distance measurement include monocular cameras, millimeter-wave radar, lidar, and binocular cameras. Monocular cameras are generally considered to have poor accuracy in distance measurement. If binocular or other distance measurement methods are used to evaluate the distance measured by a monocular camera, the different data acquisition scenarios may introduce significant errors, preventing the acquisition of accurate distance measurements from the monocular camera. Summary of the Invention

[0003] This application provides a ranging method and computer program product for a monocular camera to solve the problems existing in related technologies. The technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a monocular camera ranging method, including:

[0005] Acquire the detection image and the camera's internal parameters;

[0006] Based on the detected image, determine the initial ranging value, the detection box of the target object, and the true size of the target object;

[0007] Determine the orientation angle of the target object based on the initial ranging value and the detected image;

[0008] Based on the actual dimensions, initial distance measurement, and orientation angle, determine the three-dimensional coordinates of the target object's outline;

[0009] Generate the projection area of ​​the target object based on the three-dimensional coordinates and internal parameters of the target object's outline;

[0010] The final distance measurement value is determined based on the projection area and the detection frame.

[0011] Secondly, embodiments of this application provide a computer program product, wherein the computer program / instructions are executed by a single processor or by multiple processors in cooperation to implement any of the methods provided in the embodiments of this disclosure.

[0012] The advantages or beneficial effects of the above technical solutions include at least the following:

[0013] In this embodiment, an initial ranging value is first determined by detecting the image, and simultaneously, the detection box and the actual size of the target object are determined based on the detection image. Since the orientation angle of the target object affects the three-dimensional coordinates of its contour, the orientation angle needs to be determined first. After determining the orientation angle, the three-dimensional coordinates of the target object's contour are determined by combining the actual size of the target object and the initial ranging value. After determining the three-dimensional coordinates of the target object's contour, the three-dimensional coordinates of the target object's contour are converted into two-dimensional coordinates of the image, i.e., the projection area of ​​the target object, using the camera's internal parameters. The detection box of the target object is determined by detecting the image. Using the detection box as a reference, the projection area of ​​the target object determined by the initial ranging value can be evaluated and adjusted, thereby effectively determining an accurate ranging value as the final ranging value. That is, the ranging method of this embodiment does not require additional equipment and effectively obtains an accurate final ranging value from a monocular camera in the same scene. The process of determining the final ranging value of the monocular camera can be performed in real time during vehicle movement, and the monocular camera can be adjusted in real time based on the final ranging value, which is convenient, fast, and highly reliable, improving vehicle driving safety.

[0014] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0015] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0016] Figure 1 This is a schematic diagram of an inverse projection ranging method according to an embodiment of the related art;

[0017] Figure 2 This is a schematic diagram of a reverse projection ranging method according to another embodiment of the related art;

[0018] Figure 3 This is a schematic diagram of a target size ranging method according to an embodiment of the related art;

[0019] Figure 4 This is a flowchart illustrating a ranging method for a monocular camera according to an embodiment of this application.

[0020] Figure 5 This is a flowchart illustrating a ranging method for a monocular camera according to another embodiment of this application;

[0021] Figure 6 This is a flowchart illustrating a ranging method for a monocular camera according to another embodiment of this application; Detailed Implementation

[0022] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0023] In related technologies, monocular cameras typically use inverse projection algorithms or target size ranging algorithms for ranging.

[0024] Each vehicle has its own ideal coordinate system. In this system, the X and Y axes are parallel to the road surface, and the Z axis is perpendicular to the road surface. The X-axis points towards the front and rear of the vehicle, while the Y-axis points towards the side doors. A monocular camera also has its own ideal coordinate system. The ideal coordinate system of a monocular camera... c Axis and Z c The axis is parallel to the vehicle chassis, Y c The axis is perpendicular to the vehicle chassis, where X c The axis points in the direction of the front and rear of the vehicle, Z. c The axis represents the direction the vehicle points towards the side doors. Images captured by a monocular camera also have a corresponding image pixel coordinate system (u, v). The vehicle's ideal coordinate system, the monocular camera's ideal coordinate system, and the image pixel coordinate system can be transformed based on the camera's internal parameters and real-time calibration parameters.

[0025] For the inverse projection algorithm, the inverse projection algorithm is usually based on an image captured by a monocular camera. The image contains the target object to be measured. The target object can be a vehicle in front, a person to the front or side, or an object to the front or side. The detection box of the target object is obtained from the image captured by the monocular camera. The detection box of the target object can be identified by the trained image recognition model. The detection box of the target object can be a box that defines the outline of the image, or the largest bounding rectangle that defines the outline of the image, or the largest bounding circle that defines the outline of the image, etc. Taking the target object as a vehicle in front as an example, the key points of the target object in the image captured by the monocular camera are extracted. The key points can also be called grounding points. The distance between the current vehicle and the target object in front is measured using the key points. The distance measurement can be achieved by formulas (1)-(3).

[0026] like Figure 1As shown, in formulas (1) and (2), the road where the current vehicle and the target object in front are located has no slope, that is, the current vehicle and the target object in front are on the same plane. At this time, the pitch angle θ of the monocular camera is 0, and the current camera's attitude is in the ideal camera coordinate system (the optical axis is parallel to the ground):

[0027]

[0028]

[0029] Where d is the range measured by the monocular camera, and F c H is the focal length of a monocular camera. c v is the height of the monocular camera above the ground. b v represents the coordinates of the vehicle's ground contact point. v Let v be the v-coordinate of the vanishing line, and z be the z-coordinate. c Let y be the Z-axis coordinate of the grounding point in the monocular camera coordinate system. c Let z be the Y-axis coordinate of the grounding point in the monocular camera coordinate system, where z c =1.

[0030] like Figure 2 As shown in formula (3), the road where the current vehicle and the target object are located has a slope, meaning that the current vehicle and the target object are not on the same plane. At this time, the pitch angle θ of the monocular camera is not 0, and the real-time calibration parameters of the monocular camera can be obtained directly from the monocular camera. The pitch angle θ of the monocular camera can be determined from the real-time calibration parameters of the monocular camera.

[0031]

[0032] Where d is the ranging value of the monocular camera, and F c H is the focal length of a monocular camera. c v is the height of the monocular camera above the ground. b v represents the coordinates of the vehicle's ground contact point. v Let v be the v-coordinate of the vanishing line, and θ be the pitch angle.

[0033] The ranging value of a monocular camera can be calculated using formulas (1), (2) and (3). The advantage of using the inverse projection algorithm is that it can obtain a more accurate estimate for nearby objects. However, it has the assumption that it is based on the same plane, and the camera's pitch angle has a significant impact on the ranging, which can easily lead to inaccurate ranging values.

[0034] For target size ranging algorithms, such as Figure 3As shown, the target size ranging algorithm mainly estimates the distance between the current vehicle and the target object, i.e., the ranging value, based on the known physical dimensions of the object (the height or width of the object) and the focal length of the monocular camera, using similar triangles.

[0035] If the true width (or height) of the target object is known, it can be determined by a trained deep learning model. Based on the imaging pixels of the target object's width (or height) in the image, the longitudinal distance between the target object and the vehicle can be calculated using similar triangles, as shown in formula (4) or formula (5).

[0036]

[0037]

[0038] Where d is the ranging value of the monocular camera, and F c Let W be the focal length of the monocular camera, H be the true width of the target object, Δu be the width of the detection box of the target object in the image captured by the monocular camera, and Δv be the height of the detection box of the target object in the image captured by the monocular camera. The width and height of the detection box of the target object can be determined by a trained deep learning model or directly by pixels.

[0039] The target size ranging algorithm can also be used to measure distances using a monocular camera, obtaining the corresponding distance values. Furthermore, the distance values ​​obtained from the inverse projection algorithm can be adjusted based on the distance values ​​obtained from the target size ranging algorithm. The advantage of using the target size ranging algorithm is that the ranging results are insensitive to pitch angles. However, the ranging results depend on the variables used, such as the estimation of the target's physical size and the target's pixel size in the image. If the variables used have errors, the measured distance values ​​from the monocular camera can easily be inaccurate.

[0040] Both of the above methods have their own advantages and disadvantages. However, in actual use, they may both have the problem of inaccurate ranging values. While using millimeter-wave radar, lidar, and binocular cameras for ranging improves the accuracy of the ranging values, it also significantly increases the cost, provides less environmental information, and increases the computational complexity.

[0041] Figure 4 A flowchart illustrating a ranging method using a monocular camera according to an embodiment of this application is shown. Figure 4 As shown, the ranging method of this monocular camera may include:

[0042] Step S410: Obtain the detection image and the camera's internal parameters.

[0043] Step S420: Based on the detected image, determine the initial ranging value, the detection box of the target object, and the true size of the target object.

[0044] Step S430: Determine the orientation angle of the target object based on the initial ranging value and the detection image.

[0045] Step S440: Determine the three-dimensional coordinates of the target object's outline based on the actual size, initial distance measurement value, and orientation angle.

[0046] Step S450: Generate the projection area of ​​the target object based on the three-dimensional coordinates and internal parameters of the target object's outline.

[0047] Step S460: Determine the final distance measurement value based on the projection area and the detection frame.

[0048] In this embodiment, an initial ranging value is first determined by detecting the image, and simultaneously, the detection box and the actual size of the target object are determined based on the detection image. Since the orientation angle of the target object affects the three-dimensional coordinates of its outline, the orientation angle needs to be determined first. After determining the orientation angle, the three-dimensional coordinates of the target object's outline are determined by combining the actual size of the target object and the initial ranging value. After determining the three-dimensional coordinates of the target object's outline, the three-dimensional coordinates of the target object's outline are converted into two-dimensional coordinates of the image, i.e., the projection area of ​​the target object, using the camera's internal parameters. The detection box of the target object is determined by detecting the image. Using the detection box as a reference, the projection area of ​​the target object determined by the initial ranging value can be evaluated and adjusted, thereby effectively determining an accurate ranging value as the final ranging value. That is, the ranging method of this embodiment does not require additional equipment and effectively obtains an accurate final ranging value from a monocular camera in the same scene. The process of determining the final ranging value of the monocular camera can be performed in real time during vehicle movement, and the monocular camera can be adjusted in real time based on the final ranging value, which is convenient, fast, and highly reliable, improving vehicle driving safety.

[0049] The monocular camera ranging method in this embodiment can be executed on the vehicle's domain controller or electronic control unit (ECU), or on the vehicle's ranging processor. All of these executions are vehicle-side, allowing for more convenient and faster evaluation of the monocular camera's ranging accuracy. Furthermore, the monocular camera can be adjusted more promptly and effectively based on the accuracy, thus improving its ranging process and overall accuracy. In some embodiments, the ranging method can also be executed on a server by collecting data such as the detected image and the camera's internal parameters from the vehicle. The server can be a cloud-based system, roadside equipment, or other terminal device. The server's enhanced processing power allows for ranging evaluation without consuming vehicle memory, while also being more efficient and accurate. The following embodiments use a vehicle-side example; server-side execution can be derived similarly.

[0050] In step S410, the vehicle acquires the detected image and the camera's internal parameters.

[0051] The detected image can be obtained by capturing images using a monocular camera mounted on the vehicle. The camera's internal parameters, such as its intrinsic parameter matrix and ideal coordinate system, are fixed at the factory and can be directly obtained through the camera itself. Specifically, when the vehicle and target object are on the same plane, the monocular camera's pitch angle is 0. When they are not on the same plane, the camera's pitch angle is adjusted by the camera's mount, and this pitch angle is a parameter that needs to be calibrated in real-time. When the vehicle and target object are on the same plane, since the pitch angle is 0, as described in the inverse projection algorithm, the pitch angle can be disregarded, allowing the inverse projection algorithm to calculate the distance between the vehicle and target object, i.e., the ranging value.

[0052] In step S420, the vehicle determines the initial ranging value, the detection box of the target object, and the actual size of the target object based on the detected image.

[0053] Based on the detection image obtained from the monocular camera, the vehicle-side can calculate the initial distance between the vehicle and the target object using either an inverse projection algorithm or a target size ranging algorithm. This initial distance value is the distance value whose accuracy needs to be evaluated. This initial distance value may be accurate or inaccurate, and therefore needs to be verified through evaluation. In the inverse projection algorithm, when the pitch angle is 0, formulas (1) and (2) are used for calculation; when the pitch angle is not 0, formula (3) is used for calculation.

[0054] Based on the detection image obtained from the monocular camera, the vehicle can determine the detection box of the target object through a trained deep learning model. The trained deep learning model, such as an image detection model, identifies the target object and determines its contour. The target object is bounded based on its contour. The target object's contour can be used directly as the detection box, or the maximum bounding rectangle or the maximum bounding circle of the target object's contour can be used as the detection box.

[0055] Based on the detection images acquired from the monocular camera, the vehicle-mounted system can determine the true size of the target object using a trained deep learning model. This model can directly identify the true size of the target object from the detection images. The deep learning model can be pre-trained using training and testing sets to compare the true size of the target object with the detection images, allowing for direct determination of the target object's true size from the detection images. The true size of the target object includes parameters such as its length, width, and height.

[0056] In step S430, the vehicle determines the orientation angle of the target object based on the initial ranging value and the detected image.

[0057] Based on the initial ranging value of the detected image, determine at least two key points (grounding points) of the target object. These key points are all on the same straight line. Taking two key points as an example, the line connecting the two key points points in the front-back direction of the target object. That is, the two key points are on the X-axis of the ideal coordinate system of the target object itself. One of the two key points can be located at the head of the vehicle and the other at the tail of the vehicle, which is convenient for selection and calculation.

[0058] The planar coordinates of two key points can be calculated and determined based on the inverse projection algorithm and the initial distance measurement value. In the process of the inverse projection algorithm, when the pitch angle is 0, formulas (1) and (2) are used for calculation, and when the pitch angle is not 0, formula (3) is used for calculation.

[0059] Point R is the critical point (grounding point) at the tail of the target object. y Let R be the Y-axis coordinate value of R projected onto the ideal coordinate system of the current vehicle. x Let R be the X-axis coordinate value of R in the ideal coordinate system of the current vehicle after inverse projection.

[0060] Point B is the key point (grounding point) at the head of the target object. y Let B be the Y-axis coordinate value of B projected onto the vehicle body coordinate system. x Let B be the X-axis coordinate value of B in the ideal coordinate system of the current vehicle after inverse projection.

[0061] Among them, R y R x B y and B x All of these can be determined by inverse projection algorithm formulas (1), (2), or (3) given the initial ranging value and the detection image.

[0062]

[0063] Where α is the orientation angle of the target object.

[0064] If the target object is a vehicle, then the direction angle of the target object is the heading angle of the vehicle.

[0065] In step S440, the vehicle end determines the three-dimensional coordinates of the target object's outline based on the actual size, initial distance measurement value, and direction angle;

[0066] Based on the actual dimensions of the target object obtained above, the vehicle-side system can determine the three-dimensional coordinates of the target object's baseline contour. These three-dimensional coordinates are based on the ideal coordinate system of the current vehicle. For example, taking the center coordinates of the target object as the origin of the ideal coordinate system of the current vehicle, the three-dimensional coordinates of the eight vertices of the target object can be determined using its actual dimensions. These eight vertices can form the contour of the target object. The number of vertices can be selected according to the actual situation, but it must be at least four; fewer than four vertices cannot form the contour of the most basic cube.

[0067] After determining the three-dimensional coordinates of the reference contour of the target object, the adjusted three-dimensional coordinates are obtained by summing the initial distance measurement value with respect to the direction angle of the target object relative to the reference contour of the target object. This is the acquisition of the three-dimensional coordinates of the target object contour.

[0068] In step 450, the vehicle generates the projection area of ​​the target object based on the three-dimensional coordinates and internal parameters of the target object's outline.

[0069] The vehicle-mounted system transforms the three-dimensional coordinates of the target object's outline based on the intrinsic parameters of the monocular camera, which are the camera's intrinsic parameter matrix. This transforms the three-dimensional coordinates of the target object's outline into two-dimensional coordinates in the image captured by the monocular camera. The lines connecting the transformed two-dimensional coordinates of the image captured by the monocular camera form the projection area of ​​the target object.

[0070] One approach is to project directly onto the detection image, or to determine the outline of the projection area, i.e., the maximum bounding box of the projection area, on a blank detection image.

[0071] In step S460, the vehicle determines the final distance measurement value based on the projection area and the detection frame.

[0072] The above method determines the projection area and detection box of the target object. Since the detection box is directly determined from the detected image, it is not calculated using an initial ranging value as a comparison benchmark. The projection area, however, is determined by the initial ranging value. If the initial ranging value is inaccurate, there will be a difference in size between the projection area and the detection box; for example, the projection area might be larger or smaller than the detection box. In other words, this method effectively determines the accuracy of the measured initial ranging value. If the maximum bounding box of the projection area matches the detection box, the initial ranging value is considered accurate and is thus determined as the final ranging value. If the maximum bounding box of the projection area does not match the detection box, the initial ranging value is considered inaccurate. The initial ranging value can then be adjusted and re-evaluated until the maximum bounding box of the projection area matches the detection box, at which point the adjusted initial ranging value is determined as the final ranging value.

[0073] The evaluation criteria can be set for comparing the maximum bounding box of the target object's projected area and the target object's detection box. Since there may be errors in the calculated projected area, if the difference between the maximum bounding box and the detection box is within a certain range, the initial ranging value can be considered accurate. If the difference exceeds a certain range, the initial ranging value can be considered inaccurate, thus allowing for the evaluation of the initial ranging value and the determination of the final ranging value. This certain range can be set according to the actual situation.

[0074] The monocular camera ranging method of this embodiment eliminates the need for additional equipment such as binocular cameras or lidar, effectively determining an accurate final ranging value. This allows for adjustments to the vehicle based on the final ranging value, improving vehicle safety. Compared to related technologies, using a monocular camera effectively reduces equipment costs and acquires a wider and richer image, thereby reducing the influence of a single variable in target size ranging on the ranging value, minimizing ranging errors in monocular ranging, and improving ranging robustness.

[0075] In one implementation, evaluating the accuracy of the initial ranging value based on the projection area and the detection frame includes:

[0076] If the maximum bounding box of the projection area and the detection box are consistent, the initial distance measurement value is determined to be accurate.

[0077] The accuracy of the initial ranging value is determined by whether the maximum bounding box of the target object's projected area and the target object's detection box are consistent. Specifically, it's determined whether the maximum bounding box of the target object's projected area and the target object's detection box can overlap. If they overlap, the initial ranging value is considered accurate. If they do not overlap, the initial ranging value is inaccurate. In this case, the initial ranging value needs to be adjusted, the 3D coordinates of the target object's contour need to be re-determined, and the maximum bounding box of the target object's projected area needs to be re-determined. The evaluation and comparison between the maximum bounding box of the target object's projected area and the target object's detection box is repeated. After multiple iterations, if the maximum bounding box of the target object's projected area and the target object's detection box are re-established to be consistent, then the adjusted ranging value is the final ranging value. In other words, by determining whether the maximum bounding box of the target object's projected area and the target object's detection box are consistent, the initial ranging value can be effectively adjusted to obtain a more accurate final ranging value from the monocular camera. This ensures that the vehicle can accurately determine the distance to the target object during driving, thus ensuring vehicle safety.

[0078] like Figure 5 As shown, in one embodiment, determining the final ranging value based on the projection area and the detection frame includes:

[0079] Step S510: If the maximum bounding box of the projection area and the detection box are inconsistent, adjust the initial distance measurement value.

[0080] Step S520: Based on the actual size of the target object, the orientation angle of the target object, and the adjusted initial distance measurement value, redetermine the three-dimensional coordinates of the target object's outline.

[0081] Step S530: Generate the adjusted projection area of ​​the target object based on the camera's internal parameters and the three-dimensional coordinates of the redefined target object contour.

[0082] Step S540: If the projection area of ​​the adjusted target object is consistent with the detection box, determine the adjusted initial ranging value as the final ranging value.

[0083] By comparing the maximum bounding box and the detection box of the projection area in this application, if the maximum bounding box and the detection box of the projection area are the same, the initial distance measurement value is directly determined as the final distance measurement value, and no distance measurement value adjustment is required.

[0084] If the maximum bounding box of the projection area and the detection box are inconsistent, the initial ranging value needs to be readjusted so that the initial ranging value measured by the monocular camera can be adjusted to obtain an accurate final ranging value.

[0085] Adjusting the initial ranging value primarily involves increasing or decreasing it. After adjustment, it's necessary to re-evaluate the accuracy of the adjusted initial ranging value. The adjusted initial ranging value significantly impacts the 3D coordinates of the target object's contour. The actual size and orientation angle of the target object change little, or even not at all, after changing the initial ranging value. Therefore, in determining the 3D coordinates of the target object's contour based on its actual size, initial ranging value, and orientation angle, the initial ranging value is used as the sole variable. By adjusting the initial ranging value, the 3D coordinates of different target object contours are redefined. Then, using the redefined 3D coordinates of the target object's contour, the 3D coordinates are converted into 2D coordinates in the image coordinate system based on the camera's internal parameters and projected onto the image. This projection can be directly onto the detection image. The maximum bounding box of the projected area and the detection box are then re-evaluated for consistency. If they are consistent, the adjusted initial ranging value is considered accurate. If they are inconsistent, the adjusted ranging value is considered inaccurate, and the initial ranging value is readjusted. After multiple iterations, an accurate initial ranging value can be determined.

[0086] The monocular camera ranging method presented in this example can effectively improve the accuracy of monocular camera ranging. Compared with millimeter-wave radar, lidar and binocular cameras, its equipment is simpler and cheaper, and it can obtain a wider range of scene information. At the same time, it has a lower computational load, reduces ranging error, obtains accurate monocular camera ranging values, and improves ranging robustness.

[0087] In one implementation, adjusting the initial ranging value includes:

[0088] Adjust the initial distance measurement value based on the comparison between the maximum bounding box of the projection area and the detection box.

[0089] The initial distance measurement value can be adjusted systematically. This is based on a comparison between the maximum bounding box of the projection area and the detection box. For example, if the maximum bounding box of the projection area is greater than the detection box, the initial distance measurement value is determined to be greater than the final distance measurement value. In this case, the initial distance measurement value is adjusted by decreasing it. Similarly, if the maximum bounding box of the projection area is less than the detection box, the initial distance measurement value is determined to be less than the final distance measurement value. In this case, the initial distance measurement value is adjusted by increasing it.

[0090] It can also collect the comparison results (e.g., area difference) between the maximum bounding box and the detection box of the projection area, as well as the numerical values ​​of increasing or decreasing the initial ranging value, as a training set. The deep learning model is then trained using this training set to obtain a well-trained deep model. This model can directly determine the value for increasing or decreasing the initial ranging value by analyzing the comparison results between the maximum bounding box and the detection box of the projection area. This allows for a quick and efficient search for the final ranging value, reducing computational complexity.

[0091] In one implementation, determining the orientation angle of the target object based on the initial ranging value and the detected image includes:

[0092] Obtain the camera's real-time calibration parameters;

[0093] The orientation angle of the target object is determined based on the initial ranging value, the detected image, and the real-time calibration parameters.

[0094] When the vehicle and the target object are not on the same plane, it is necessary to obtain the pitch angle of the monocular camera. The real-time calibration parameters of the camera include the real-time pitch angle of the monocular camera. The real-time calibration parameters of the camera (i.e., the pitch angle) can be obtained through real-time dynamic calibration of the camera. Specifically, the monocular camera or the vehicle end calculates the pose change of the monocular camera based on the pixel changes of the previous and next frames of the image, and outputs the real-time calibration parameters of the camera, i.e., the pitch angle of the monocular camera.

[0095] As shown in Formula (3) above, the pitch angle of the monocular camera will affect the inverse projection algorithm. In other words, the initial ranging value or the direction angle of the target object determined by the inverse projection algorithm will be affected by the pitch angle of the monocular camera. Therefore, the pitch angle of the monocular camera needs to be added to the calculation of the direction angle of the target object in order to obtain the accurate direction angle of the target object.

[0096] In this embodiment, by incorporating the camera's real-time calibration parameters into the inverse projection algorithm, the applicability of the monocular camera evaluation method can be effectively broadened. It can effectively evaluate the initial ranging value even when the current vehicle and the target object are not on the same plane, making it convenient, fast, and widely applicable.

[0097] like Figure 6 As shown, in one embodiment, determining the orientation angle of the target object based on the initial ranging value and the detected image includes:

[0098] Step S610: Based on the detected image, determine the first key point and the second key point of the target object, wherein the line connecting the first key point and the second key point points points in the front-back direction of the target object;

[0099] Step S620: Determine the coordinates of the first key point and the second key point based on the inverse projection algorithm and the initial ranging value;

[0100] Step S630: Determine the orientation angle of the target object based on the coordinates of the first key point and the second key point.

[0101] The orientation angle of the target object is actually the angle formed by the X-axis of the target object's ideal coordinate system and the X-axis of the current vehicle's ideal coordinate system. The target object's orientation angle is determined in the current vehicle's ideal coordinate system. It is determined by the coordinates of the first and second key points of the target object. The line connecting the first and second key points points to the X-axis of the target object's ideal coordinate system points in the forward / backward direction of the target object. To facilitate the use of the inverse projection algorithm, both the first and second key points determined on the detection image are considered to be the target object's ground points.

[0102] By using the inverse projection algorithm and the initial distance measurement, the coordinates of the first and second key points can be calculated. These coordinates can be three-dimensional or two-dimensional coordinates in a plane. Since the orientation angle of the target object is not substantially related to the Z-axis, it can be determined directly by determining the two-dimensional coordinates in the plane. Specifically, by placing the coordinates of the first and second key points into the ideal coordinate system of the current vehicle, the angle between the line connecting the coordinates of the first and second key points and the ideal coordinate system of the current vehicle is the orientation angle of the target object. The orientation angle of the target object has a significant impact on reconstructing its contour; by determining the orientation angle, the three-dimensional coordinates of the target object's contour can be effectively determined.

[0103] In one implementation, determining the detection bounding box and the true size of the target object based on the detected image includes:

[0104] The trained deep learning model is used to identify target objects in the detected image, and the detection bounding box and the true size of the target object are obtained.

[0105] Deep learning models can be trained from the cloud or on-board devices. The training set can be trained using data collected on-board devices. This can effectively determine the detection box and the true size of the target object from the detected image, with high accuracy and convenience.

[0106] In one implementation, determining the initial ranging value based on the detected image includes:

[0107] The initial distance value is determined by using an inverse projection algorithm or a target size ranging algorithm to perform distance measurement calculations on the detected image.

[0108] The vehicle-mounted system can directly use either the inverse projection algorithm or the target size ranging algorithm to measure the distance to the detection image captured by the monocular camera. However, when using the inverse projection algorithm, there are cases where the pitch angle calibrated by the monocular camera in real-time may be 0 or not.

[0109] Both inverse projection algorithms and target size ranging algorithms can effectively determine the initial ranging value, which is then used as the ranging value to be evaluated.

[0110] In one implementation, determining the three-dimensional coordinates of the target object's outline based on its actual size, initial distance measurement, and orientation angle includes:

[0111] Determine the three-dimensional coordinates of the reference contour of the target object based on its actual dimensions;

[0112] The three-dimensional coordinates of the reference profile are adjusted based on the initial distance measurement value and direction angle to determine the three-dimensional coordinates of the target object profile.

[0113] Based on the actual dimensions of the target object, the vehicle-side system can construct the outline of the target object using the ideal coordinate system of the current vehicle, and determine the three-dimensional coordinates of the target object's baseline outline. For example, with the center coordinates of the target object as the origin of the current vehicle's ideal coordinate system, the three-dimensional coordinates of the target object's eight vertices can be determined using the target object's actual dimensions. These eight vertices can constitute the outline of the target object. The number of vertices can be selected according to the actual situation, but it must be at least four; fewer than four vertices cannot form the outline of the most basic cube.

[0114] After determining the 3D coordinates of the target object's baseline contour, the adjusted 3D coordinates are obtained by summing the target object's orientation angle with the initial distance measurement value. This is the process of acquiring the target object's 3D coordinates. For example, after determining the target object's baseline contour's 3D coordinates, the baseline contour's 3D coordinates are translated using the initial distance measurement value. The translated baseline contour's 3D coordinates are then adjusted according to the target object's orientation angle, resulting in the final adjusted 3D coordinates, which represent the acquired 3D coordinates of the target object's contour.

[0115] In this embodiment, the actual size, initial distance measurement value, and orientation angle can redetermine the three-dimensional coordinates of the target object's outline. The target object's outline can be determined in the current vehicle's ideal coordinate system, thereby generating a projection area for evaluating the initial distance measurement value.

[0116] In one implementation, generating the projection region of the target object based on the three-dimensional coordinates and internal parameters of the target object's contour includes:

[0117] The three-dimensional coordinates of the target object's outline are projected onto the detection image using internal parameters to generate the target object's projection area.

[0118] The vehicle-mounted system obtains the intrinsic parameters of the monocular camera, which may include the camera's intrinsic parameter matrix K, an inherent property of the camera lens, which can be directly obtained from the monocular camera. The camera's intrinsic parameter matrix K is as follows.

[0119]

[0120] Among them, f x f is the focal length of the camera along the u-axis in the camera's image coordinate system. y Let c be the focal length of the camera along the v-axis in the camera's image coordinate system. x Let c be the u-axis coordinate of the origin (optical center) of the camera's image coordinate system in the pixel coordinate system. y Let c be the v-axis coordinate of the origin (optical center) of the camera's image coordinate system in the pixel coordinate system. x c represents the coordinates of the optical center along the u-axis in the camera's image coordinate system. y This represents the coordinates of the light spot along the v-axis in the camera's image coordinate system.

[0121] The 3D coordinates of the target object's contour are converted into 2D coordinates in the image using the camera's intrinsic parameter matrix K. The contour enclosed by all the 2D coordinates is then defined as the projection region of the target object. To facilitate comparison with the detection bounding box of the target object in the detection image, the contour enclosed by all the 2D coordinates can be directly projected onto the detection image. This allows for evaluation and comparison of the projected region and the detection bounding box of the target object, thereby determining the accuracy of the initial ranging value.

[0122] The embodiments of this application also provide a computer program product, which implements any of the methods provided in the embodiments of this application when the computer program / instructions are executed by a single processor or by multiple processors in cooperation.

[0123] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0124] The aforementioned computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions may be transmitted from one website, computer, training equipment, or data center to another website, computer, training equipment, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0125] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other physical classes of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage media, or any other non-transferable media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0126] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] It should be understood that the various processes shown above can be used, with steps rearranged, added, or deleted. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein. The above embodiments are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A ranging method for a monocular camera, characterized in that, include: Acquire the detection image and the camera's internal parameters; Based on the detected image, determine the initial ranging value, the detection box of the target object, and the true size of the target object; Based on the initial ranging value and the detected image, the orientation angle of the target object is determined, specifically including: Based on the detected image, a first key point and a second key point of the target object are determined, wherein the line connecting the first key point and the second key point points points in the front-back direction of the target object. Based on the inverse projection algorithm and the initial ranging value, determine the coordinates of the first key point and the coordinates of the second key point; Determine the orientation angle of the target object based on the coordinates of the first key point and the second key point; Based on the actual dimensions, the initial distance measurement value, and the orientation angle, determine the three-dimensional coordinates of the target object's outline; Based on the three-dimensional coordinates of the target object's outline and the internal parameters, generate the projection area of ​​the target object; The final distance measurement value is determined based on the projection area and the detection frame; The step of determining the final ranging value based on the projection area and the detection frame includes: If the maximum bounding box of the projection area and the detection box are the same, the initial ranging value is determined to be the final ranging value. If the maximum bounding box of the projection area and the detection box are inconsistent, adjust the initial distance measurement value; Based on the actual size of the target object, the orientation angle of the target object, and the adjusted initial distance measurement value, the three-dimensional coordinates of the target object's outline are redefined; Based on the camera's internal parameters and the redefined 3D coordinates of the target object's outline, an adjusted projection area of ​​the target object is generated. If the projection area of ​​the adjusted target object is consistent with the detection box, the adjusted initial ranging value is determined to be the final ranging value.

2. The method according to claim 1, characterized in that, The adjustment of the initial ranging value includes: The initial ranging value is adjusted based on the comparison between the maximum bounding box of the projection area and the detection box.

3. The method according to claim 1, characterized in that, Determining the orientation angle of the target object based on the initial ranging value and the detected image includes: Obtain the camera's real-time calibration parameters; The orientation angle of the target object is determined based on the initial ranging value, the detected image, and the real-time calibration parameters.

4. The method according to claim 1, characterized in that, Determining the detection bounding box and the true size of the target object based on the detected image includes: The target object is identified in the detected image based on the trained deep learning model, and the detection box and the true size of the target object are obtained.

5. The method according to claim 1, characterized in that, Determining the initial ranging value based on the detected image includes: The detection image is measured using an inverse projection algorithm or a target size ranging algorithm to determine the initial ranging value.

6. The method according to claim 1, characterized in that, Determining the three-dimensional coordinates of the target object's outline based on the actual size, the initial distance measurement value, and the orientation angle includes: The three-dimensional coordinates of the reference contour of the target object are determined based on the actual dimensions; The three-dimensional coordinates of the reference contour are adjusted based on the initial distance measurement value and the direction angle to determine the three-dimensional coordinates of the target object contour.

7. The method according to claim 1, characterized in that, The step of generating the projection region of the target object based on the three-dimensional coordinates of the target object's contour and the internal parameters includes: The three-dimensional coordinates of the target object's outline are projected onto the detection image using the internal parameters to generate the target object's projection area.

8. A computer program product comprising a computer program / instructions, characterized in that, The computer program / instructions implement the method described in any one of claims 1 to 7 when executed by a single processor or by multiple processors in cooperation.

Images