METHODS AND APPARATUS FOR ESTIMATION OF THE DISTANCE OF THE OBJECTS IN EUCLIDEAN SPACE USING FUSION OF CAMERA POSE INFORMATION and SCENE PRIOR INFORMATION'S AND THEREOF

The fusion of camera pose and scene prior information allows for real-time, accurate distance estimation in autonomous vehicles, addressing the limitations of conventional methods by eliminating the need for specific regions of interest and reducing hardware requirements.

KR103043885B1Active Publication Date: 2026-11-06HL KLEMOVE CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
KR1020230093810
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-08-12
Filing Date
2023-07-19
Publication Date
2026-11-06
Estimated Expiration
2043-07-19

AI Technical Summary

Technical Problem

Conventional distance estimation methods for autonomous vehicles are challenging due to their reliance on regression-based techniques, requiring significant pre-deployment work and resources, and are limited to specific regions of interest, making them unsuitable for real-time applications in systems like autonomous vehicles.

Method used

A method and apparatus for estimating distances in Euclidean space by fusing camera pose information and scene prior information, using a mesh grid in a real-world coordinate system, marking pixel coordinates, and performing alpha blending to estimate distances without requiring a specific region of interest.

Benefits of technology

This approach enables real-time, accurate distance estimation suitable for autonomous vehicles, overcoming speed and accuracy issues of prior art methods, and can operate on various camera types without hardware dependencies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112023079578331-PAT00113_ABST
    Figure 112023079578331-PAT00113_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time distance estimation method and apparatus (100) for a fisheye lens camera as well as a pinhole camera. The present invention utilizes camera pose information as well as scene prior information to estimate the distance of an object from a camera (160). The camera pose information is estimated with the help of extrinsic camera correction, which essentially utilizes intrinsic camera correction information. Fisheye lens cameras and any other camera models introduce distortion into the acquired input image. Therefore, an additional step of image non-distortion or pixel point non-distortion is required. The above additional step is necessary to ensure that the input image or its pixel points follow the pinhole camera model before applying distance estimation to any input image or its random pixel points.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to the estimation of the distance of objects in Euclidean space. More specifically, the present invention relates to a method and apparatus for estimating the distance between objects by fusing camera pose information and previous information of a scene. Background Technology

[0002] With the recent rise in interest in autonomous vehicles, there is growing interest in real-time and precise distance estimation of objects in semantically segmented images. Accurate and rapid distance estimation aids in proper path planning and obstacle avoidance for autonomous vehicles and driver assistance systems. The accuracy and latency of distance estimation methods significantly impact the decisions of other active and passive sensors in autonomous vehicles.

[0003] KR20180121259A relates to a distance detection device and method for a camera-equipped computer that detects the distance from a single camera to a target object using image information of a target object captured by a single camera in an all-in-one computer equipped with a camera. The distance detection device of the camera-equipped computer comprises: a web camera installed on a computer monitor to capture an image of a target object positioned in front and acquire an image; an image processing unit that receives an image from the web camera and generates image information; an internal parameter setting unit that detects and stores internal parameters corresponding to focal length and refractive index in the image captured by the web camera; an external parameter setting unit that measures and stores the rotational translation and translational translation transformation relationship between the camera coordinate system and the world coordinate system; an external parameter input unit that receives horizontal distance information from the web camera to the target object and stores the received information in an external parameter storage unit; and a position information calculation unit that recognizes the target object in the image captured by the web camera and calculates the distance between the position coordinates of a feature point at a specific height of the target object and the position coordinates of the feature point using parameters input through the internal parameter setting unit, the external parameter setting unit, and the external parameter input unit.

[0004] The paper titled “Front vehicle detection and distance estimation using single-lens video camera” (2015) by Chen, CH, Chen, TY, Huang, DY, and Feng, KW discusses front vehicle detection and distance estimation using a single-lens automotive video camera on urban and suburban roads. The proposed method mainly consists of road area detection, front vehicle detection, and distance calculation. First, lines are detected using the Hough transform, where appropriate straight lines are selected and their intersections are utilized to obtain vanishing points. Subsequently, strong right and left edges are extracted and connected to vanishing points to divide the road area, and the vehicle's position is located using the lower shadow of the front vehicle within these areas. Finally, the distance between the ego vehicle and the front vehicle is calculated based on the vehicle's position and vanishing points.

[0005] Most conventional algorithms use regression-based techniques to estimate the distance of an object from a camera to a specific, predefined region of interest in an input image.

[0006] In addition, conventional technology relies heavily on the marking of points on a straight line based on regression-based techniques and the subsequent estimation of polynomial coefficients using existing or machine / deep learning-based methods.

[0007] Therefore, deploying existing distance estimation methods in real time is a relatively difficult task because these methods require significant pre-deployment work and resources, and due to their atypical nature, they will likely only operate within specific regions of interest.

[0008] The aforementioned points make the placement of algorithms of the conventional technology undesirable in systems such as autonomous vehicles. The problem to be solved

[0009] The objective of the present invention is to provide a real-time distance estimation method for fisheye lens cameras as well as conventional pinhole cameras.

[0010] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art to which the present invention pertains from the description below. means of solving the problem

[0011] The present invention provides a method and apparatus for estimating the distance between objects in Euclidean space by fusion of camera pose information and scene prior information.

[0012] In one embodiment, a method for estimating distances of objects in Euclidean space comprises: forming a mesh grid in a real-world coordinate system using ground measurement information knowledge of internal intersections of a rectangular grid structure; marking corresponding pixel coordinate values ​​of the mesh grid in a 2D image captured by a camera during a pre-placement period; storing the marked pixel coordinate values ​​for the mapping; estimating horizontal and vertical grid IDs for an input pixel point by fitting horizontal and vertical lines; back-mapping in the real-world coordinate system based on the grid ID values—for the back-mapping, existing ground measurement information values ​​and linear interpolation are used—; and performing a final fusion of individual longitudinal and lateral object distances estimated using scene prior knowledge with the help of camera pose information and alpha blending.

[0013] If the camera used to capture the above 2D image is not a pinhole camera, the pixel coordinates of each object are not distorted.

[0014] The above pixel world coordinates are estimated using the following formula:

[0015]

[0016] Here,

[0017] u, v: pixel coordinates;

[0018] X, Y, Z: world-coordinates;

[0019] s: Scale factor;

[0020] K: Camera intrinsic matrix;

[0021] R: Camera rotation matrix; and

[0022] T: Camera translation vector.

[0023] The above scale factor s is calculated using the camera's intrinsic parameters, its rotation matrix, and translation vector using the following formulas:

[0024]

[0025]

[0026] Here,

[0027] u, v: pixel coordinates;

[0028] X, Y, Z: world-coordinates;

[0029] s: Scale factor;

[0030] K: Camera intrinsic matrix;

[0031] R: Camera rotation matrix; and

[0032] T: Camera translation vector.

[0033] If the above estimated world coordinates are close to [0.0, 0.0, 0.0]mm, that is, if they satisfy the following criteria 0.0mm <= [|XP|, |YP|, |ZP|] < 50.0mm, the above coordinates are stored as translation vector T_pre and pinhole camera focal length f_pin.

[0034] The above Euclidean distance is calculated using the following formula:

[0035]

[0036] The above longitudinal distance is calculated using the following formula:

[0037]

[0038] The above lateral distance is calculated using the following formula:

[0039]

[0040]

[0041] Here,

[0042] : Euclidean distance from the origin of the camera coordinate system to the point currently under consideration with respect to the world coordinate system (using camera pose information);

[0043] : World coordinates of pixel points corresponding to the world coordinate system;

[0044] : Each camera pose X, Y, Z coordinates in the world coordinate system;

[0045] : Longitudinal distance of the point currently under consideration from the origin of the camera coordinate system with respect to the world coordinate system (using camera pose information); and

[0046] : Lateral distance of the point currently under consideration from the origin of the camera coordinate system relative to the world coordinate system (using camera pose information).

[0047] The fusion of the estimated distances above using two methods (i.e., using camera pose information and scene prior information (grid mapping)) is performed using alpha blending; and is calculated using the following formulas:

[0048]

[0049]

[0050] Here,

[0051] : Longitudinal distance estimated using camera pose information;

[0052] : Longitudinal distance estimated using pre-scene information (grid mapping);

[0053] : Lateral distance estimated using camera pose information;

[0054] : Lateral distance estimated using pre-scene information (grid mapping); and

[0055] : Alpha blending parameter.

[0056] In summary, the distance estimation approach of the present invention is based on the fusion of distances estimated using two methods: namely, Euclidean distance in Euclidean space using camera pose information, and distance estimated using prior knowledge information acquired during the estimate camera calibration process.

[0057] The method of the present invention is practically general. It does not require any region of interest in the input image and has no limitations on the minimum and maximum distances that can be estimated.

[0058] In another embodiment, an apparatus for estimating distances between objects in Euclidean space is provided. The apparatus comprises at least one processor; a camera communically coupled to the at least one processor; and a memory communically coupled to the at least one processor, wherein, when executed by the at least one processor, the apparatus: forms a mesh grid in a real-world coordinate system using ground measurement information knowledge of internal intersections of a rectangular grid structure; marks corresponding pixel coordinate values ​​of the mesh grid in a 2D image captured by the camera during the pre-placement period; stores the pixel coordinate values ​​marked for the mapping; estimates horizontal and vertical grid IDs for input pixel points by fitting horizontal and vertical lines; backmaps in the real-world coordinate system based on the grid ID values—for the backmap, existing ground measurement information values ​​and linear interpolation are used—; and estimates longitudinal and lateral object distances using linear interpolation; And store commands configured to perform the final fusion of estimated individual longitudinal and lateral object distances using camera pose information and scene prior knowledge with the help of alpha blending.

[0059] The present invention overcomes the real-time speed and accuracy problems of existing distance estimation methods of the prior art.

[0060] The foregoing is a summary and, therefore, includes, as necessary, simplification, generalization, and omission of details; consequently, those skilled in the art will recognize that the summary is merely illustrative and is not intended to limit the invention in any way. Furthermore, as will be apparent to those skilled in the art, the operations disclosed herein may be implemented in a number of ways, and such changes and modifications may be made without departing from the invention and its broader aspects. Other aspects of the invention, features of the invention, and advantages will become apparent from the following non-limiting detailed description, as defined only by the claims. Brief explanation of the drawing

[0061] The above and other objects and features of the present invention will become apparent from the following description of preferred embodiments taken together with the accompanying drawings, wherein similar reference numerals indicate similar features. Figure 1 illustrates an extrinsic camera calibration setup for distance estimation and a corresponding world coordinate system. FIG. 2 illustrates a schematic diagram for calculating the longitudinal distance, lateral distance, and Euclidean distance of an object from the camera-coordinate system origin, i.e., the camera world position. FIG. 3 illustrates a flowchart of the steps to be followed before the deployment of the distance estimation method on edge devices. FIG. 3a illustrates a 2D image of a distance estimation grid acquired by a camera (wherein the grid intersections are numbered in red and left-right order). In this case, 'Longitudinal' and 'Lateral' indicated in FIG. 3a mean 'longitudinal' and 'lateral', respectively. Figure 3b illustrates a grid structure of GT (Ground Measurement Information) coordinate values ​​for estimating longitudinal and lateral distances corresponding to the image in Figure 3a. In this case, 'Longitudinal' and 'Lateral' indicated in Figure 3b mean 'longitudinal' and 'lateral', respectively. FIG. 3c illustrates a flowchart of a method for acquiring undistorted image pixel points during the pre- and post-batch periods. Figure 3d illustrates the contents of longitudinal and lateral GT files. FIG. 3e illustrates the corresponding pixel values ​​of GT coordinate points as shown in FIG. 3a and 3b (the numbering is row-based, and each row follows the same point ordering as shown in FIG. 3a). FIG. 3f illustrates the camera coordinate system axes (Xc, Yc, Zc), where the image of world point P is p, which is p' by the pinhole camera, the angle θ between the principal axis and the incident ray, the distance r between the image point and the principal point, the focal length f, and the angle φ with respect to the x-axis of the sensor plane formed by the image point vector derived from the principal point and other related parameters in the coordinate system. FIGS. 4a and 4b illustrate a flowchart of the steps to be followed after the arrangement of the method for distance estimation on edge devices. Figure 4c illustrates a sample quadrilateral in a grid structure of GT coordinate values ​​of a 2D image. In this case, 'Longitudinal' and 'Lateral' indicated in Figure 4c mean 'longitudinal' and 'lateral', respectively, and 'Sample Quadrilateral' indicated in Figure 4c means 'sample quadrilateral'. Figure 5 illustrates marked points in an input image captured for an external camera calibration operation. Figure 6 illustrates the simulation results for the marked points as shown in Figure 3a. In this case, 'Only pose', 'Prior Info', 'Fusion', 'Longitudinal distance from car', and 'Lateral distance from car' indicated in Figure 6 mean 'Pose only', 'Prior Info', 'Fusion', 'Longitudinal distance from car', and 'Lateral distance from car', respectively. FIG. 7 illustrates an apparatus for estimating the distance between objects in Euclidean space according to one embodiment of the present invention. Specific details for implementing the invention

[0062] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings, some of which are illustrated. The following description is intended to be illustrative of embodiments of the present invention and should not be construed as restrictive.

[0063] The words “comprising,” “having,” “containing,” and “including,” and other forms thereof are intended to have an equivalent meaning and are considered open forms in that the item or items following any one of these words do not imply that such item or items are a comprehensive list of such item or items, or that they are limited only to the enumerated item or items.

[0064] It should also be noted that, as used in this application and the appended claims, singular forms (“a,” “an,” and “the”) include plural objects unless the context clearly indicates otherwise. Although any systems and methods similar or equivalent to those described herein may be used in the practice or testing of embodiments of the invention, preferred systems and methods are now described.

[0065] The disclosed embodiments are merely examples of the invention and may be implemented in various forms.

[0066] The elements illustrated in the drawings interact as described in more detail below. However, before presenting the detailed description, it should be noted that all discussions below are illustrative rather than restrictive, regardless of the specific implementation being described.

[0067] The present invention discloses a real-time distance estimation method for a fisheye lens camera as well as a pinhole camera. The present invention utilizes camera pose information as well as scene prior information to estimate the distance of an object from a camera. The camera pose information is estimated with the help of extrinsic camera calibration, which essentially utilizes intrinsic camera calibration information.

[0068] Fisheye lens cameras and any other camera models introduce distortion to the acquired input image. Therefore, an additional step of image distortion or pixel point distortion is required.

[0069] The above additional step is necessary to ensure that the input image or its random pixel points follow the pinhole camera model before applying distance estimation to any input image or its random pixel points.

[0070] The distance estimation method according to an embodiment of the present invention is practically general, does not require any region of interest in the input image, and has no limitations on the minimum and maximum distances that can be estimated.

[0071] This method overcomes the real-time speed and accuracy issues of existing distance estimation methods of the prior art.

[0072] Apart from dependence on camera intrinsic and extrinsic parameters, this method has no additional dependencies. It relies on highly qualified mathematical calculations that can be deployed on edge devices for real-time applications. Due to low latency and high accuracy, the invention is friendly to edge devices and port preparation.

[0073] Figure 1 illustrates an extrinsic camera calibration setup for distance estimation and a corresponding world coordinate system.

[0074] FIG. 2 illustrates a schematic diagram for calculating the longitudinal distance, lateral distance, and Euclidean distance of an object from the camera-coordinate system origin, i.e., the camera world position.

[0075] Referring to FIGS. 1 and 2, the coordinates (Xw, Yw, Zw) represent world coordinates originating at Ow, and the coordinates (Xc, Yc, Zc) represent camera coordinates originating at Oc. The coordinates (Xc→w, Yc→w, Zc→w) represent camera pose coordinates for a world coordinate system originating at Oc→w.

[0076] More specifically, a detailed description of the coordinates (Xw, Yw, Zw), coordinates (Xc, Yc, Zc), and coordinates (Xc→w, Yc→w, Zc→w) is as follows.

[0077] Oc: Camera coordinate origin, Xc: Camera coordinate X-axis, Yc: Camera coordinate Y-axis, Zc: Camera coordinate Z-axis, Oc->w: Position of camera coordinate origin relative to the world coordinate system origin, Xc->w: Camera pose X-coordinate relative to the world coordinate system, Yc->w: Camera pose Y-coordinate relative to the world coordinate system, Zc->w: Camera pose Z-coordinate relative to the world coordinate system, Ow: World coordinate origin, Xw: World coordinate X-axis, Yw: World coordinate Y-axis, Zw: World coordinate Z-axis (wherein the coordinates (Xw, Yw, Zw), negative Z-axis values ​​point upward from the plane)

[0078] In the proposed distance estimation method, a specific camera model was considered for all calibration parameter calculations, and these calibration parameters are an essential part of the proposed method.

[0079] FIG. 3 illustrates a flowchart of the steps to be followed before the deployment of the distance estimation method on edge devices.

[0080] Referring to FIG. 3, during the pre-batch operation, only camera intrinsic and extrinsic parameters were considered, and based on these, the camera pose was estimated without explicit knowledge from any other additional sensors.

[0081] In step (301), pixel coordinates of the world coordinate system origin (Ow) are obtained from the 2D captured image. The reference origin of the world coordinate system for external camera calibration is located at [0.0, 0.0, 0.0] mm.

[0082] In step (302), if the camera used to capture the 2D captured image is a fisheye lens camera, each object pixel coordinate must not be distorted. This step can be skipped for a pinhole camera. The target pinhole camera focal length is set (starting at 0.004150).

[0083] In step (303), the scale factor s of the planner homography matrix is ​​calculated using the camera intrinsic parameters, its rotation matrix and translation vector, i.e., the fisheye lens camera extrinsic parameters, using formulas (1) and (2) as shown below:

[0084] … (1)

[0085] … (2)

[0086] Here,

[0087] u, v: pixel coordinates;

[0088] X, Y, Z: world-coordinates;

[0089] s: Scale factor;

[0090] K: Camera intrinsic matrix;

[0091] R: Camera rotation matrix; and

[0092] T: Camera translation vector.

[0093] In step (304), the world coordinates of the corresponding pixel coordinates (of the world coordinate system origin Ow) are estimated using Equation (3) as shown below:

[0094] … (3)

[0095] Here,

[0096] u, v: pixel coordinates;

[0097] X, Y, Z: world-coordinates;

[0098] s: Scale factor;

[0099] K: Camera intrinsic matrix;

[0100] R: Camera rotation matrix; and

[0101] T: Camera translation vector.

[0102] In step (305), if the estimated world coordinates are close to [0.0, 0.0, 0.0] mm (i.e., satisfy the next range criterion: 0.0 mm ≤ [X, Y, Z] < 50.0 mm), the world coordinates are stored as a translation vector T_pre and a corresponding pinhole camera focal length f_pin (i.e., satisfy the next range criterion: 0.0 mm ≤ [X, Y, Z] < 50.0 mm). Otherwise, after heuristically adjusting the value of the pinhole camera focal length, steps (302 to 304) are repeated until the estimated world coordinates are close to [0.0, 0.0, 0.0] (i.e., satisfy the next range criterion: 0.0 mm ≤ [X, Y, Z] < 50.0 mm).

[0103] The target pinhole camera focal length of the geometric structure that the undistorted fisheye lens image pixel points follow after distortion is not known in advance. This leads to some error during the distortion process (the pinhole focal length must be heuristically refined). The accuracy of implicit correction also affects the accuracy of the undistorted input pixel points of the fisheye lens image. All these factors affect the accuracy of the image's world coordinate transformation (inverse perspective transformation). To minimize deviations in distances estimated using the proposed approach and actual ground measurement data, the target pinhole camera focal length must be heuristically estimated so that the inverse perspective transformation of the undistorted pixel coordinates at the world coordinate system origin maps to [0.0, 0.0, 0.0] mm (in the ideal case). However, in actual scenarios, it never maps to [0.0, 0.0, 0.0] mm, and there is always some deviation. After several experiments, it was found that an acceptable deviation is 50 mm on each axis of the world coordinate system.

[0104] The target pinhole camera focal length value is heuristically adjusted until the above deviation does not become less than 50 mm on each axis of the world coordinate system.

[0105] Therefore, the proposed method attempts to reduce the error in the estimated distance value by heuristically estimating the optimal value of the target pinhole camera focal length during the pre-placement phase itself. This mitigates error propagation during the post-placement period.

[0106] In the real-world coordinate system, camera pose information is calculated using Equation (4) as shown below:

[0107] … (4)

[0108] Here,

[0109] R: Camera rotation matrix; and

[0110] T: Camera translation vector.

[0111] FIG. 3a shows a 2D image of a distance estimation grid acquired by a camera, where the grid intersections are numbered in red and left-right order.

[0112] FIG. 3b illustrates a grid structure of GT coordinate values ​​for longitudinal and lateral distance estimation corresponding to the image shown in FIG. 3a.

[0113] As shown in Fig. 3, a rectangular grid type structure with known ground measurement information of internal intersections was used. From this knowledge of ground measurement information, a mesh grid was formed in a real-world coordinate system.

[0114] The corresponding pixel coordinate values ​​of the above mesh grid points were marked on the 2D image during the pre-placement period (as illustrated in Fig. 3a) and subsequently saved for mapping. Next, horizontal and vertical grid IDs were estimated for the input pixel points by fitting horizontal and vertical lines. Based on the grid ID values, inverse mapping was performed in the real-world coordinate system. For this inverse mapping, existing ground measurement data values ​​and linear interpolation were used. With the help of linear interpolation, the final longitudinal and lateral object distances were estimated. Subsequently, the distance values ​​estimated using both approaches were fused together.

[0115] More specifically, in step (401), a reference distance estimation grid-like structure is established to measure distance ground actual measurement information values.

[0116] In step (402), the longitudinal and lateral ground measurement coordinate values ​​measured in mm are stored in two separate text files, “gt_logitudinal_coordinates.txt” and “gt_lateral_coordinates.txt” (as illustrated in FIG. 3d).

[0117] In step (403), intersections within the 2D image of the distance estimation grid-like structure are marked. These intersections are obtained using the same camera that is used for the input image for implicit and extrinsic camera calibration.

[0118] The marking order starts from the visible intersection located in the lower-left part of the image, moves toward the right, and then proceeds to the left to mark the intersections located in the row above.

[0119] In step (404), the marked point pixel coordinate values ​​are saved in a text file “gt_PixelValues.txt” (as shown in FIG. 3e).

[0120] FIG. 3c illustrates a flowchart of a method for acquiring undistorted image pixel points during the pre- and post-batch periods.

[0121] Now, referring to FIG. 3c, in step (501), an inverse projection is obtained using the fisheye lens camera intrinsic parameters and the pinhole camera focal length (f_pin) (as estimated in step number 305).

[0122] In step (502), the perspective projection value is the direction of the incoming ray (angle, as shown in FIG. 3f) using the formula (5) as shown below. & It is estimated using ):

[0123] r = tan( ) / f_pin … (5)

[0124] Here,

[0125] : Angle between the principal axis and the incoming beam;

[0126] r: distance between the image point and the bar; and

[0127] f_pin: Pinhole camera focal length (as estimated in step number 305).

[0128] In step (503), the undistorted pixel points are calculated using the formula (6) as shown below:

[0129]

[0130] Here,

[0131] : Undistorted pixel point coordinates;

[0132] r: distance between the image point and the bar;

[0133] : Angle with respect to the x-axis of the sensor plane created by the image point vector starting from the bar;

[0134] : Image center value in the x-direction; and

[0135] : Image center value in the y-direction.

[0136] FIG. 3e illustrates the corresponding pixel values ​​of GT coordinate points as shown in FIG. 3a and FIG. 3b. The numbering is row-based, and each row follows the same point sequence as shown in FIG. 3a.

[0137] Figure 3f illustrates the camera coordinate system axes (Xc, Yc, Zc), where the image of world point P is p, whereas this is p' by the pinhole camera, the angle between the principal axis and the incident ray. , distance r between the image point and the principal point, focal length f, angle with respect to the x-axis of the sensor plane formed by the image point vector derived from the principal point and other related parameters in the coordinate system would.

[0138] FIGS. 4a and 4b illustrate a flowchart of the steps to be followed after the arrangement of the method for distance estimation on edge devices.

[0139] In step (601), the pixel coordinates of the objects are obtained from semantically segmented input images or from any other module that generates similar information from 2D captured input images.

[0140] In step (602), if the camera used to capture the 2D input image is a fisheye lens camera, the pinhole camera focal length f_pin (estimated during the batch-previous period) is used, camera intrinsic parameters are used, and an undistorted one is applied to each pixel coordinate. This step can be skipped for a pinhole camera.

[0141] In step (603), the scale factor s of the planner homography matrix is ​​calculated using the camera intrinsic parameters, its rotation matrix, and translation vector according to formulas (1) and (2) as shown below:

[0142] … (1)

[0143] … (2)

[0144] Here,

[0145] u, v: pixel coordinates;

[0146] X, Y, Z: world-coordinates;

[0147] s: Scale factor;

[0148] K: Camera intrinsic matrix;

[0149] R: Camera rotation matrix; and

[0150] T: Camera translation vector.

[0151] In step (604), the world coordinates (X_p, Y_p, Z_p) of the corresponding pixel coordinates are estimated using Equation (3), which is reproduced as follows:

[0152] … (3)

[0153] Here,

[0154] u, v: pixel coordinates;

[0155] X, Y, Z: world-coordinates;

[0156] s: Scale factor;

[0157] K: Camera intrinsic matrix;

[0158] R: Camera rotation matrix; and

[0159] T: Camera translation vector.

[0160] In step (605), linear translation is applied to each calculated world coordinate using the translation vector (T_pre) calculated during the batch-pre-step number 305. Then, the estimated world coordinates are updated after the linear translation operation.

[0161] In step (606), camera pose information in the real-world coordinate system is calculated using Equation (4) as reproduced below:

[0162] … (4)

[0163] Here,

[0164] R: Camera rotation matrix; and

[0165] T: Camera translation vector.

[0166] In step (607), the Euclidean, longitudinal, and lateral distances are calculated using formulas (5), (6), (7) and (8), respectively, as shown below:

[0167] … (5)

[0168] … (6)

[0169] … (7)

[0170] … (8)

[0171] Here,

[0172] : Euclidean distance from the origin of the camera coordinate system to the point currently under consideration with respect to the world coordinate system;

[0173] : World coordinates of the pixel points corresponding to the world coordinate system as estimated in step number 605;

[0174] : Camera pose X, Y, Z coordinates for the world coordinate system, respectively, as estimated in step number 606;

[0175] : Longitudinal distance of the point currently under consideration from the origin of the camera coordinate system with respect to the world coordinate system; and

[0176] : Lateral distance of the point currently under consideration from the origin of the camera coordinate system relative to the world coordinate system.

[0177] Additionally, for input pixel points, horizontal and vertical grid IDs are estimated by fitting horizontal and vertical lines. Based on the grid ID values, inverse mapping is performed in the real-world coordinate system. For this inverse mapping, existing ground measurement data values ​​and linear interpolation are used. With the help of linear interpolation, the final longitudinal and lateral object distances are estimated. Finally, the distance values ​​estimated using the two approaches (i.e., using camera pose information and pre-scene information (grid mapping)) are fused together.

[0178] More specifically, in step (701), longitudinal and lateral GT grid point coordinate values ​​are read from their corresponding text files (“gt_logitudinal_coordinates.txt” and “gt_lateral_coordinates.txt” (as illustrated in FIG. 3d)) stored during the pre-placement step. For example:

[0179]

[0180]

[0181] Marked / detected 2D pixel-point coordinate values ​​are read from a text file (“gt_PixelValues.txt” (as shown in FIG. 3e)) stored during the batch-pre-stage.

[0182] In step (702), if the 2D captured input image is a fisheye lens camera image, the distortion is applied to the corresponding pixel-point coordinate values ​​of each GT point. This step can be skipped for a pinhole camera.

[0183] In step (703), a mesh-grid structure (of the world coordinate system) is generated from real-world GT coordinate values.

[0184] In step (704), four nearest neighbor coordinate GT point values ​​(bottom-left, bottom-right, top-left, top-right), and their corresponding 2D pixel values ​​(these may form a continuous quadrilateral) are selected.

[0185] In step (705), the undistorted input pixel coordinates are mapped to one of the pixel-point quadrilaterals by calculating the area of ​​four sub-triangles and comparing the sum of their areas with the total area of ​​the quadrilateral in order to estimate the grid ID number.

[0186] In step (706), LongitudinalDistanceGridMapping is estimated by following the following sub-steps:

[0187] In step (7061), the value of delta is estimated by taking the ratio of the y-coordinate differences of the (upper-left, lower-left) and (upper-right, lower-right) quadrilateral vertex groups.

[0188] In step (7062), an array of y-coordinate values ​​is generated by using delta values ​​for the left and right sides of the quadrilateral. Next, an array of x-coordinate values ​​is generated by utilizing estimated tangent and slope values ​​for the left and right sides of the quadrilateral.

[0189] In step (7063), an array of horizontally fitted lines is generated by using arrays of x- and y-coordinate values ​​for the left and right sides of the quadrilateral.

[0190] In step (7064), the vertical distance of each horizontally fitted line is estimated using the input pixel point coordinate values.

[0191] In step (7065), it is checked whether the estimated distance is within an allowed threshold. The allowed threshold is It is the pixel value. Then, the absolute values ​​of all eligible distance values ​​are stored in an array and sorted.

[0192] In step (7066), the y-coordinate grid ID number of the minimum distance value is taken, and the horizontal grid resolution value, which is the minimum value of the difference between the y-coordinate values ​​of the left and right sides of the quadrilateral, is stored.

[0193] In step (7067), a delta step value for real-world longitudinal GT coordinate values ​​is calculated based on the horizontal grid resolution value.

[0194] In step (7068), the real-world longitudinal distance (LongitudinalDistanceGridMapping) value is estimated using the longitudinal GT coordinates and the estimated delta step value.

[0195] In step (707), LateralDistanceGridMapping is estimated by following the following sub-steps:

[0196] In step (7071), the value of delta is estimated by taking the ratio of the x-coordinate differences of the (upper-left, upper-right) and (lower-left, lower-right) quadrilateral vertex groups.

[0197] In step (7072), an array of x-coordinate values ​​is generated by using delta values ​​for the upper and lower parts of the quadrilateral. Next, an array of y-coordinate values ​​is generated by utilizing estimated tangent and slope values ​​for the upper and lower parts of the quadrilateral.

[0198] In step (7073), an array of vertically fitted lines is generated by using arrays of x- and y-coordinate values ​​for the upper and lower sides of the quadrilateral.

[0199] In step (7074), the vertical distance of each vertically fitted line is estimated using the input pixel point coordinate values.

[0200] In step (7075), it is checked whether the estimated distance is within an allowed threshold. The allowed threshold is It is the pixel value. Then, the absolute values ​​of all eligible distance values ​​are stored in an array and sorted.

[0201] In step (7076), the x-coordinate grid ID number of the minimum distance value is taken, and the vertical grid resolution value, which is the minimum value of the difference between the x-coordinate values ​​of the upper and lower sides of the quadrilateral, is stored.

[0202] In step (7077), a delta step value for real-world lateral GT coordinate values ​​is calculated based on the vertical grid resolution value.

[0203] In step (7078), the real-world lateral distance (LateralDistanceGridMapping) value is estimated using the lateral coordinates and the estimated delta step value.

[0204] In step 708, the fusion of distances estimated using both methods (i.e., using camera pose information and scene prior information (grid mapping)) is performed by utilizing alpha blending (where α=0.8) as shown in equations (9) and (10) below to obtain refined estimated longitudinal and lateral distance values.

[0205] … (9)

[0206] … (10)

[0207] Here,

[0208] : Longitudinal distance estimated from formula (6) as shown in step number 607;

[0209] : Lateral distance estimated from formula (7) / (8) (depending on the case) as illustrated in step number 607; and

[0210] : Alpha blending parameter.

[0211] None of the other solutions in the prior art adequately solve the real-time processing problems of distance estimation and simultaneously fail to produce the accuracy required for safety-critical autonomous vehicles.

[0212] The present invention does not require any special machine / deep learning training methods to estimate various parameters as required by existing methods of the prior art.

[0213] This method can deliver distance estimation results in real time due to its inherently simple mathematical steps and the requirement of very limited hardware resources.

[0214] In summary, the distance estimation approach of the present invention is based on the fusion of distances estimated using two methods, namely, i. Euclidean distance in Euclidean space using camera pose information, and ii. distances estimated using prior knowledge information acquired during camera calibration operations.

[0215] The present invention is practically general and can operate on any type of camera, whether it is a normal pinhole camera or a fisheye lens camera.

[0216] In the present invention, a specific camera model was considered for all calibration parameter calculations, and these calibration parameters are an essential part of the proposed approach.

[0217] In the present invention, during the pre-batch operation, only camera intrinsic and extrinsic parameters were considered, and based on these, the camera pose was estimated without explicit knowledge from any other additional sensors. There is no requirement for selecting a specific region of interest from the input image, and the present invention operates on the entire image resolution.

[0218] FIG. 7 illustrates an apparatus (100) for estimating the distance between objects in Euclidean space according to an embodiment of the present invention. The apparatus (100) for estimating the distance between objects in Euclidean space comprises at least one processor (110), a camera (160) communicably coupled to at least one processor (110), and a memory (120) communicably coupled to at least one processor (110). The memory (120) stores commands configured to allow the apparatus (100) to estimate the distance between objects in Euclidean space when executed by at least one processor (110). The apparatus also includes a display (140), a storage system (150), and an I / O interface (130) that allow a user to interact with the apparatus. It should be understood that other hardware and / or software components, although not illustrated, may be used with the apparatus.

[0219] The method and apparatus of the present invention reduce errors in estimated distance values ​​by heuristically estimating the optimal value of the target pinhole camera focal length during the pre-placement stage itself, which mitigates error propagation during the pre-placement period. The present invention provides a unique method of distance estimation in which prior scene knowledge acquired during camera calibration is taken into account.

[0220] Although the foregoing description of the present invention enables a person skilled in the art to make and use what is currently considered the optimal method, those skilled in the art will understand and recognize the existence of variations, combinations, and equivalents of the methods and details of the specific embodiments thereof.

[0221] Although preferred embodiments of the present invention have been described and illustrated, it will be understood that various modifications may be made without departing from the scope of the invention.

Claims

Claim 1 A method for estimating the distances of objects in Euclidean space comprises: a step of forming a mesh grid in a real-world coordinate system using ground measurement information knowledge of internal intersection points of a rectangular grid structure by a processor (110); a step of marking corresponding pixel coordinate values ​​of the mesh grid in a 2D image captured by a camera (160) during a pre-deployment phase to pre-establish a correspondence between the pixel coordinates of the mesh grid and the real-world coordinates by the processor (110); a step of storing the marked pixel coordinate values ​​for mapping in a memory (120); a step of estimating horizontal and vertical grid IDs for an input pixel point by fitting horizontal and vertical lines by the processor (110); a step of back-mapping in the real-world coordinate system based on the grid ID values ​​by the processor (110)—for back-mapping, existing ground measurement information values ​​and linear interpolation are used—; and a step of using linear interpolation by the processor (110) to final longitudinal A method comprising: a step of estimating longitudinal and lateral object distances; and a step of performing a fusion of the final longitudinal and lateral object distances using alpha blending by the processor (110); wherein the fusion of the estimated distances is performed using alpha blending by using two methods (i.e., using camera pose information and scene prior information (grid mapping)); and wherein the fusion is calculated using the following formulas: Here, : Longitudinal distance estimated using camera pose information; : Longitudinal distance estimated using pre-scene information (grid mapping); : Lateral distance estimated using camera pose information; : Lateral distance estimated using pre-scene information (grid mapping); and : Alpha blending parameter. Claim 2 A method according to claim 1, characterized in that if the camera (160) used to capture the 2D image is not a pinhole camera, each object pixel coordinate is not distorted. Claim 3 A method according to claim 1, characterized in that the real-world coordinates of the pixel coordinates are estimated using the following formula: Here, u, v: pixel coordinates; X, Y, Z: world coordinates; s: scale factor; K: camera intrinsic matrix; R: camera rotation matrix; and T: camera translation vector. Claim 4 A method according to claim 3, characterized in that the scale factor s is calculated using the camera intrinsic parameters, its rotation matrix, and translation vector using the following formulas: Here, u, v: pixel coordinates; X, Y, Z: world coordinates; s: scale factor; K: camera intrinsic matrix; R: camera rotation matrix; and T: camera translation vector. Claim 5 A method according to claim 1, wherein if the estimated world coordinates are close to [0.0, 0.0, 0.0], that is, if the world coordinates satisfy the following criterion 0.0 mm <= [|XP|, |YP|, |ZP|] < 50.0 mm (where [|XP|, |YP|, |ZP|] are the estimated world coordinates of the corresponding pixel coordinates in the 2D captured image of the real-world coordinate system origin OW), the coordinates are stored as a translation vector T_pre and a pinhole camera focal length f_pin. Claim 6 A method according to claim 1, characterized in that the Euclidean distance is calculated using the following formula: Here, : Euclidean distance from the origin of the camera coordinate system to the point currently under consideration with respect to the world coordinate system (using camera pose information); : Camera pose X coordinate with respect to the world coordinate system; : Camera pose Y coordinate in the world coordinate system; : Camera pose Z coordinate with respect to the world coordinate system; : X coordinate of the point currently under consideration in the world coordinate system; : The Y coordinate of the point currently under consideration in the world coordinate system; and : The Z coordinate of the point currently under consideration in the world coordinate system. Claim 7 A method according to claim 1, characterized in that the longitudinal distance is calculated using the following formula: Here, : Longitudinal distance of the point currently under consideration from the origin of the camera coordinate system relative to the world coordinate system (using camera pose information); : Camera pose X coordinate with respect to the world coordinate system; and : The X coordinate of the point currently under consideration in the world coordinate system. Claim 8 A method according to claim 1, characterized in that the lateral distance is calculated using the following formulas: Here, : Lateral distance of the point currently under consideration from the origin of the camera coordinate system relative to the world coordinate system (using camera pose information); : Camera pose Y coordinate for the world coordinate system; and : The Y coordinate of the point currently under consideration in the world coordinate system. Claim 9 delete Claim 10 In a device (100) for estimating the distance between objects in Euclidean space, the device (100) comprises: at least one processor (110); and a camera (160) communicateably coupled to the at least one processor (110). and includes a memory (120) communicably coupled to the at least one processor (110), wherein, when executed by the at least one processor (110), the device (100) comprises: forming a mesh grid in a real-world coordinate system using ground actual measurement information knowledge of internal intersection points of a rectangular grid structure; marking corresponding pixel coordinate values ​​of the mesh grid in a 2D image captured by the camera (160) during a pre-deployment phase to pre-establish a correspondence between the pixel coordinates of the mesh grid and the real-world coordinates; storing the marked pixel coordinate values ​​for mapping; estimating horizontal and vertical grid IDs for input pixel points by fitting horizontal and vertical lines; back-mapping in the real-world coordinate system based on the grid ID values—for the back-mapping, existing ground actual measurement information values ​​and linear interpolation are used—; and estimating longitudinal and lateral object distances using linear interpolation; A device characterized by storing commands configured to perform a final fusion of individual longitudinal and lateral object distances estimated using camera pose information and scene prior knowledge with the help of alpha blending, wherein the fusion of the estimated distances is performed using two methods (i.e., using camera pose information and scene prior information (grid mapping)) utilizing alpha blending; and wherein the fusion is calculated using the following formulas: Here, : Longitudinal distance estimated using camera pose information; : Longitudinal distance estimated using pre-scene information (grid mapping); : Lateral distance estimated using camera pose information; : Lateral distance estimated using pre-scene information (grid mapping); and : Alpha blending parameter. Claim 11 A device according to claim 10, characterized in that, if the camera used to capture the 2D image is not a pinhole camera, each object pixel coordinate is not distorted. Claim 12 A device according to claim 10, characterized in that the world coordinates of the above-mentioned pixels are estimated using the following formula: Here, u, v: pixel coordinates; X, Y, Z: world coordinates; s: scale factor; K: camera intrinsic matrix; R: camera rotation matrix; and T: camera translation vector. Claim 13 In claim 12, the device is characterized in that the scale factor s is calculated using the camera intrinsic parameters, its rotation matrix, and translation vector using the following formulas: Here, u, v: pixel coordinates; X, Y, Z: world coordinates; s: scale factor; K: camera intrinsic matrix; R: camera rotation matrix; and T: camera translation vector. Claim 14 An apparatus according to claim 10, wherein if the estimated world coordinates are close to [0.0, 0.0, 0.0], i.e., if the following criterion 0.0 mm <= [|XP|, |YP|, |ZP|] < 50.0 mm is satisfied (where [|XP|, |YP|, |ZP|] are the estimated world coordinates of the corresponding pixel coordinates in the 2D captured image of the real-world coordinate system origin OW), the coordinates are stored as a translation vector T_pre and a pinhole camera focal length f_pin. Claim 15 In claim 10, a device characterized in that the Euclidean distance is calculated using the following formula: Here, : Euclidean distance from the origin of the camera coordinate system to the point currently under consideration with respect to the world coordinate system (using camera pose information); : Camera pose X coordinate with respect to the world coordinate system; : Camera pose Y coordinate in the world coordinate system; : Camera pose Z coordinate with respect to the world coordinate system; : X coordinate of the point currently under consideration in the world coordinate system; : The Y coordinate of the point currently under consideration in the world coordinate system; and : The Z coordinate of the point currently under consideration in the world coordinate system. Claim 16 In claim 10, the device is characterized in that the above longitudinal distance is calculated using the following formula: Here, : Longitudinal distance of the point currently under consideration from the origin of the camera coordinate system relative to the world coordinate system (using camera pose information); : Camera pose X coordinate with respect to the world coordinate system; and : The X coordinate of the point currently under consideration in the world coordinate system. Claim 17 In claim 10, the device is characterized in that the lateral distance is calculated using the following formulas: Here, : Lateral distance of the point currently under consideration from the origin of the camera coordinate system relative to the world coordinate system (using camera pose information); : Camera pose Y coordinate for the world coordinate system; and : The Y coordinate of the point currently under consideration in the world coordinate system. Claim 18 delete

Citation Information

Patent Citations

  • Object detection device, object detection system, moving object and object detection method

    JP2021051476A

  • Correction distance calculation device, program for correction distance calculation and correction distance calculation method

    JP2021085855A

  • Device and method for obtaining distance information from a view

    KR1020190065432A

  • Image generation apparatus and image generation method

    US20190158809A1

  • Monocular distance measuring device

    CN111047639A