Method and apparatus for estimating pose of multiple camera

Abstract

A method of estimating a pose of a plurality of cameras mounted on a vehicle includes detecting a vanishing line from each of a first camera and a second camera. The method further includes transmitting first vanishing line information of the first camera to each camera hierarchically connected to the first camera, and transmitting second vanishing line information including a vanishing line of the second camera to each camera hierarchically connected to the second camera. The method further includes calculating a difference between the two pieces of vanishing line information in at least one camera receiving the first vanishing line information and the second vanishing line information. The method further includes obtaining final vanishing line information, where a difference between two pieces of vanishing line information converges within a predetermined reference error value, and estimating a pose of each of the plurality of cameras.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority and benefit of Korean Patent Application No. 10 - 2024 - 0005460, filed with the Korean Intellectual Property Office on January 12, 2024, the entire content of which is hereby incorporated by reference. Technical field

[0003] The present disclosure relates to a technique for estimating the poses of multiple cameras, and more particularly, to an apparatus and method for estimating the poses of multiple cameras based on vanishing line information detected from two cameras among multiple cameras mounted on a vehicle. Background art

[0004] Various image recognition techniques are generally employed to operate autonomous vehicles. For example, recognizing lanes and vanishing points during driving are important factors.

[0005] A vanishing point is the infinite extension of parallel lines in three - dimensional space, which intersects at a point on a two - dimensional plane when projected onto the two - dimensional plane. As an example of using vanishing point detection, the architectural structure can be analyzed by obtaining vanishing points and vanishing lines in three orthogonal directions, thereby re - interpreting the building. In the 3D transformation of a 2D image including artificial structures, a depth map can be generated by detecting vanishing points. The reason is that when a 3D space is converted into a 2D image, since the part where the vanishing point is located usually corresponds to the farthest part in the image, the relative depth can be estimated.

[0006] In addition, vanishing point information is an important basis for position information analysis in autonomous driving systems such as lane detection in autonomous vehicles or robots. This is because the road can be detected by connecting the main edges connected from the vanishing point.

[0007] A vehicle may be equipped with multiple cameras, including, for example, a front camera, a rear camera, a left - front camera, a right - front camera, a wide - angle front camera, a left - rear camera, a right - rear camera, a wide - angle rear camera, a wide - angle left camera, and a wide - angle right camera. The vehicle may adopt techniques to estimate the poses of multiple cameras.

[0008] Conventional techniques can estimate the pose of an unaligned camera by monitoring the relationship between three or more cameras. However, there may be camera combinations for which it is difficult to estimate the camera - camera relationship. In addition, if two or more cameras are unaligned, pose estimation may be difficult, and information about the camera - road relationship may be lost as calibration is repeated. Summary of the invention

[0009] The present disclosure has been made to solve the above problems that occur in the prior art while maintaining intact the advantages achieved by the prior art.

[0010] Aspects of the present disclosure provide an apparatus and method for estimating the poses of a plurality of cameras mounted on a vehicle. The apparatus and method are capable of estimating the poses of the plurality of cameras based on vanishing line information detected from two of the plurality of cameras mounted on the vehicle.

[0011] Aspects of the present disclosure provide an apparatus and method for estimating the poses of a plurality of cameras. The apparatus and method can estimate the poses of the plurality of cameras by transmitting the matching vanishing line information of a front camera and a rear camera to other cameras hierarchically connected to the front camera and the rear camera, and using the vanishing points of each camera and the transmitted vanishing line information.

[0012] Aspects of the present disclosure provide an apparatus and method for estimating the poses of a plurality of cameras. The apparatus and method are capable of estimating the poses of the plurality of cameras based on the received vanishing line information and vanishing points. The apparatus and method can also obtain the translation and / or rotation of each of the plurality of cameras based on the geometric relationship structure between the plurality of cameras to avoid offline calibration of the vehicle.

[0013] The technical problems to be solved by the present disclosure are not limited to the above problems. Other technical problems not mentioned herein should be more clearly understood by those of ordinary skill in the art to which the present disclosure pertains from the following description.

[0014] According to one aspect of the present disclosure, there is provided an apparatus for estimating the poses of a plurality of cameras mounted on a vehicle. The apparatus includes a memory configured to store computer-executable instructions. The apparatus further includes at least one processor configured to access the memory and execute the instructions. The at least one processor is configured to detect a vanishing line from each of a first and a second camera among the plurality of cameras mounted on the vehicle that are capable of detecting the vanishing line. The at least one processor is further configured to send first vanishing line information including the vanishing line of the first camera to each camera hierarchically connected to the first camera. The at least one processor is further configured to send second vanishing line information including the vanishing line of the second camera to each camera hierarchically connected to the second camera. The at least one processor is further configured to calculate a difference between the first vanishing line information and the second vanishing line information in at least one camera that receives the first vanishing line information of the first camera and the second vanishing line information of the second camera. The at least one processor is additionally configured to obtain final vanishing line information in which the difference between the first vanishing line information and the second vanishing line information converges within a predetermined reference error value. The at least one processor is further configured to estimate the pose of each of the plurality of cameras based on the vanishing points detected by the plurality of cameras and the final vanishing line information.

[0015] According to one embodiment, the at least one processor may be configured to, when the difference between the first vanishing line information and the second vanishing line information is greater than a predetermined reference error value, re-detect the vanishing line from each of the first camera and the second camera based on the difference between the first vanishing line information and the second vanishing line information. The at least one processor may further be configured to re-calculate the difference between first re-calculated vanishing line information including the re-detected vanishing line of the first camera and second re-calculated vanishing line information including the re-detected vanishing line of the second camera. The at least one processor may further be configured to obtain final vanishing line information when the difference between the first re-calculated vanishing line information and the second re-calculated vanishing line information converges within a predetermined reference error value.

[0016] According to one embodiment, the at least one processor may be configured to re-detect the vanishing line by removing edge points corresponding to outliers based on i) the difference between the first vanishing line information and the second vanishing line information and ii) a previous vanishing line slope.

[0017] According to one embodiment, the at least one processor may be configured to obtain the translation of each of the plurality of cameras based on the geometric relationship structure between the plurality of cameras and design data including a three-dimensional map of the vehicle.

[0018] According to one embodiment, after obtaining the orientations between multiple cameras through epipolar geometry and obtaining the included angles between the multiple cameras, a geometric relationship structure can be obtained based on the included angles between the multiple cameras.

[0019] According to one embodiment, at least one processor may be configured to obtain the translation of each of the multiple cameras by i) scaling the geometric relationship structure between the multiple cameras to match design data including a three-dimensional map of the vehicle, and ii) fitting the scaled geometric relationship structure between the multiple cameras to the wheel rim of the vehicle.

[0020] According to one embodiment, the at least one processor may be configured to repeatedly execute the following process: i) adjusting the translation such that any one camera overlaps with the wheel rim of the vehicle in the scaled geometric relationship structure between the multiple cameras; ii) performing scaling such that the distance between each of the multiple cameras and the wheel rim of the vehicle is minimized, so as to obtain the translation of each of the multiple cameras.

[0021] According to one embodiment, at least one processor may be configured to adjust the translation such that any one camera overlaps with the wheel rim of the vehicle, and adjust the rotation such that the remaining cameras are aligned with the wheel rim of the vehicle, and then perform scaling such that the distance between each of the multiple cameras and the wheel rim of the vehicle is minimized.

[0022] According to one embodiment, the multiple cameras may include a front camera, a rear camera, a left front camera, a right front camera, a wide-angle front camera, a left rear camera, a right rear camera, a wide-angle rear camera, a wide-angle left camera, and a wide-angle right camera.

[0023] According to one aspect of the present disclosure, a method for estimating the poses of multiple cameras mounted on a vehicle is provided. The method includes detecting a vanishing line from each of a first camera and a second camera among the multiple cameras mounted on the vehicle that are capable of detecting the vanishing line. The method further includes transmitting first vanishing line information including the vanishing line of the first camera to each camera hierarchically connected to the first camera. The method also includes sending second vanishing line information including the vanishing line of the second camera to each camera hierarchically connected to the second camera. The method further includes calculating a difference between the first vanishing line information and the second vanishing line information in at least one camera that receives the first vanishing line information of the first camera and the second vanishing line information of the second camera. The method also includes obtaining final vanishing line information in which the difference between the first vanishing line information and the second vanishing line information converges within a predetermined reference error value. The method additionally includes estimating the pose of each of the multiple cameras based on the vanishing points detected by the multiple cameras and the final vanishing line information.

[0024] According to one embodiment, detecting a vanishing line may include re-detecting the vanishing line from each of a first camera and a second camera based on a difference between first vanishing line information and second vanishing line information when the difference between the first vanishing line information and the second vanishing line information is greater than a reference error value. Obtaining final vanishing line information may include re-calculating a difference between first re-calculated vanishing line information including the re-detected vanishing line of the first camera and second re-calculated vanishing line information including the re-detected vanishing line of the second camera. Obtaining final vanishing line information may further include: when the difference between the first re-calculated vanishing line information and the second re-calculated vanishing line information converges within a predetermined reference error value, obtaining the final vanishing line information.

[0025] According to one embodiment, detecting a vanishing line may include re-detecting the vanishing line by removing edge points corresponding to outliers based on i) a difference between first vanishing line information and second vanishing line information and ii) a previous vanishing line slope.

[0026] According to one embodiment, the method may further include obtaining a translation of each of a plurality of cameras based on a geometric relationship structure between the plurality of cameras and design data including a three-dimensional map of a vehicle.

[0027] According to one embodiment, after obtaining an orientation between a plurality of cameras through epipolar geometry and obtaining an included angle between the plurality of cameras, a geometric relationship structure may be obtained based on the included angle between the plurality of cameras.

[0028] According to one embodiment, obtaining a translation of each of a plurality of cameras may include scaling a geometric relationship structure between the plurality of cameras to match design data including a three-dimensional map of a vehicle. Obtaining a translation of each of a plurality of cameras may further include fitting the scaled geometric relationship structure between the plurality of cameras to a wheel rim of a vehicle.

[0029] According to one embodiment, obtaining a translation of each of a plurality of cameras may include repeatedly performing the following processes: i) adjusting the translation such that any one camera overlaps with a wheel rim of a vehicle in a scaled geometric relationship structure between the plurality of cameras; ii) performing scaling such that a distance between each of the plurality of cameras and the wheel rim of the vehicle is minimized so that all cameras can obtain a translation of each of the plurality of cameras.

[0030] According to one embodiment, obtaining a translation of each of a plurality of cameras may include adjusting the translation such that any one camera overlaps with a wheel rim of a vehicle, and adjusting the rotation such that the remaining cameras are aligned with the wheel rim of the vehicle. Obtaining a translation of each of a plurality of cameras may further include performing scaling such that a distance between each of the plurality of cameras and the wheel rim of the vehicle is minimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features, and advantages of the present disclosure should become more apparent from the following detailed description in conjunction with the accompanying drawings, wherein:

[0032] Figure 1 A flowchart showing a method for estimating the poses of multiple cameras according to an embodiment of the present disclosure;

[0033] Figure 2 An example diagram showing a method for describing the detection of vanishing lines according to an embodiment of the present disclosure;

[0034] Figure 3 An example diagram showing a path for transmitting vanishing line information according to an embodiment of the present disclosure;

[0035] Figure 4 An example diagram showing a process for obtaining final vanishing line information according to an embodiment of the present disclosure;

[0036] Figure 5 An example diagram showing a geometric relationship structure between cameras according to an embodiment of the present disclosure;

[0037] Figure 6 An example diagram showing a process for obtaining the translation of each camera according to an embodiment of the present disclosure;

[0038] Figure 7 A configuration of an apparatus for estimating the poses of multiple cameras according to an embodiment of the present disclosure; and

[0039] Figure 8 A block diagram of a computing system for performing a method for estimating the poses of multiple cameras according to an embodiment of the present disclosure. Detailed Embodiments

[0040] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present disclosure. In addition, the present disclosure can be embodied in several different forms. The present disclosure is not limited to the embodiments described herein.

[0041] When describing the embodiments of the present disclosure, if a detailed description of a known configuration or function may obscure the gist of the present disclosure, its detailed description is omitted. In the drawings, parts irrelevant to the description are omitted. Throughout the drawings and the entire specification, the same reference numerals refer to the same elements.

[0042] In the present disclosure, if an element is referred to as "connected to", "coupled to", or "combined with" another element, the element can be directly connected or coupled to the other element, can be combined with the other element, or there can be one or more intervening elements between the element and the other element. When terms such as "comprising", "including", or "having" are used in the present disclosure, the presence of the specified element is designated. These terms do not exclude the presence or addition of one or more other elements.

[0043] In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one element from other elements. Unless otherwise specifically mentioned, these terms do not limit the order or importance of the elements. Thus, within the scope of the present disclosure, the first element in one embodiment can be referred to as the second element in another embodiment. Similarly, the second element in one embodiment can be referred to as the first element in another embodiment.

[0044] In the present disclosure, different elements are only used to clearly describe their features and do not mean that these elements must be separate. For example, multiple elements can be integrated to form a single hardware or software unit, or a single element can be distributed to form multiple hardware or software units. Accordingly, even if not otherwise stated, such integrated or distributed embodiments are included within the scope of the present disclosure.

[0045] In the present disclosure, the elements described in various embodiments are not necessarily essential elements. Some elements may be optional. Thus, embodiments including a subset of the elements described in one embodiment are also included within the scope of the present disclosure. In addition, embodiments including other elements in addition to the elements described in various embodiments are also within the scope of the present disclosure.

[0046] In the present disclosure, for ease of description, expressions of positional relationships used in the specification, such as top, bottom, left, or right, are described. It should be understood that if the drawings are viewed in reverse, the positional relationships described in the specification can also be interpreted in the opposite manner.

[0047] In the present disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one item listed together with the relevant phrase, or any possible combination thereof.

[0048] When a component, device, element, etc. of the present disclosure is described as having a certain purpose or performing an operation, function, etc., the component, device, or element should be regarded herein as "configured to" meet that purpose or perform that operation or function.

[0049] To estimate the pose of each of a plurality of cameras mounted on a vehicle, embodiments of the present disclosure can use at least two cameras capable of detecting vanishing lines to detect the vanishing lines, and can repeatedly perform the process of detecting the vanishing lines such that the two vanishing line information matches each other, or the difference between the two vanishing line information is minimized. Embodiments of the present disclosure can send the vanishing line information to each other camera to estimate the pose of each of the plurality of cameras based on the vanishing line information and the vanishing points detected by the plurality of cameras. According to embodiments of the present disclosure, the pose of a camera can include the position and orientation of the camera.

[0050] Embodiments of the present disclosure can obtain the geometric relationship structure between a plurality of cameras through epipolar geometry. Embodiments of the present disclosure can also obtain the translation and / or rotation of each of the plurality of cameras based on the geometric relationship structure between the plurality of cameras and design data including a three-dimensional map of the vehicle.

[0051] Embodiments of the present disclosure can obtain the translation of each of the plurality of cameras by i) scaling the geometric relationship structure between the plurality of cameras to match the design data including the three-dimensional map of the vehicle, and ii) fitting the scaled geometric relationship structure between the plurality of cameras to the rim of the vehicle.

[0052] For example, embodiments of the present disclosure can repeatedly perform the following process: i) adjusting the rotation such that in the scaled geometric relationship structure between the plurality of cameras, any one camera is aligned with the rim of the vehicle; ii) performing scaling such that the distance between each of the plurality of cameras and the rim of the vehicle is minimized so that all cameras can obtain the translation of each of the plurality of cameras.

[0053] The plurality of cameras in embodiments of the present disclosure can refer to cameras mounted on a vehicle, and can include a front camera, a rear camera, a left front camera, a right front camera, a wide-angle front camera, a left rear camera, a right rear camera, a wide-angle rear camera, a wide-angle left camera, and a wide-angle right camera. Of course, the plurality of cameras is not limited to the above cameras and can include any camera capable of being mounted on a vehicle.

[0054] Reference is made below Figure 1-7 to describe in more detail a method and apparatus for estimating the pose of a plurality of cameras according to embodiments of the present disclosure.

[0055] Figure 1 is an operation flowchart of a method for estimating the pose of a plurality of cameras according to embodiments of the present disclosure. Figure 1 An operation flowchart of a method for estimating the pose of each of a plurality of cameras mounted on a vehicle is shown.

[0056] In operation S110, a vanishing line can be detected in each of a first camera (e.g., a front L4 camera) and a second camera (e.g., a rear L4 camera) among a plurality of cameras mounted on a vehicle, where the vanishing line is detectable.

[0057] For example, if the cameras mounted on the vehicle include a front camera, a rear camera, a left front camera, a right front camera, a wide-angle front camera, a left rear camera, a right rear camera, a wide-angle rear camera, a wide-angle left camera, and a wide-angle right camera, a vanishing line may be detected in the front camera (e.g., the front L4 camera) and the rear camera (e.g., the rear L4 camera) due to the field of view related to the ego-lane. Hereinafter, for illustrative purposes, the vanishing line is described as being detected by the front L4 camera and the rear L4 camera.

[0058] In addition, each of the plurality of cameras can obtain a vanishing point based on the image information acquired from each camera. Here, each camera can use any method capable of obtaining a vanishing point from an image to obtain the vanishing point. For example, each camera can detect the vanishing point as the focus of expansion (FOE) in straight driving by creating a region of interest (such as a static object region) from the image, extracting the optical flow in the static object region, and estimating the focus of expansion (FOE) through the intersection of the extracted optical flow vectors.

[0059] According to one embodiment, in operation S110, if an image as shown in (a) is captured by the front L4 camera while the vehicle is moving, the vehicle included in the image can be detected, and the edge points of the vehicle on the ego-lane can be detected. Thereafter, the horizontal and vertical components as shown in (b) can be estimated by principal component analysis (PCA) to detect the vanishing line 210 of the front L4 camera. The method for detecting the vanishing line of the front L4 camera or the rear L4 camera in operation S110 is not limited to the above method, but any method capable of detecting a vanishing line from an image can be applied. Figure 2 Figure 2

[0060] In an embodiment, the method according to an embodiment of the present disclosure can apply a vanishing point detection method and a vanishing line detection method. Therefore, the method can use the detected vanishing point and vanishing line as described above to estimate the pose of each of the plurality of cameras.

[0061] ​​If vanishing lines of the front L4 camera and the rear L4 camera are detected in operation S110, then in operation S120, vanishing line information including the vanishing lines of the front L4 camera and the rear L4 camera can be sent to each of the other cameras hierarchically connected to the front L4 camera and the rear L4 camera. For example, first vanishing line information including the vanishing line of the front L4 camera can be sent to each of the other cameras hierarchically connected to the front L4 camera. In addition, second vanishing line information including the vanishing line of the rear L4 camera can be sent to each of the other cameras hierarchically connected to the rear L4 camera.

[0062] For example, as Figure 3 shown, in operation S120, the vanishing line information of the front L4 camera can be sent to the left front camera, the wide-angle front camera, the right front camera, the wide-angle left camera 310, and the wide-angle right camera 320 hierarchically connected to the front L4 camera. In addition, the vanishing line information of the rear L4 camera can be sent to the left rear camera, the wide-angle rear camera, the right rear camera, the wide-angle left camera 310, and the wide-angle right camera 320 hierarchically connected to the rear L4 camera. The wide-angle left camera 310 and the wide-angle right camera 320 can receive the vanishing line information of the front L4 camera via the wide-angle front camera and receive the vanishing line information of the rear L4 camera via the wide-angle rear camera.

[0063] If the vanishing line information of the front L4 camera and the rear L4 camera is sent to each of the other cameras in operation S120, then in operation S130, at least one camera that receives two pieces of vanishing line information (for example, the first vanishing line information including the vanishing line of the front L4 camera and the second vanishing line information including the vanishing line of the rear L4 camera), such as Figure 3 at least one of the wide-angle left camera 310 and the wide-angle right camera 320, can calculate the difference (α) between the two pieces of vanishing line information. In operation S140, at least one camera can determine whether the calculated difference between the two pieces of vanishing line information converges within a predetermined reference error value.

[0064] Here, the reference error value can be a preset very small difference to obtain the final vanishing line information, or can be the value "0" indicating that the two pieces of vanishing line information are the same. It should be noted that the reference error value can be determined by a commercial operator or an individual providing the technology of the present disclosure.

[0065] As a result of the determination in operation S140, if the difference between the two pieces of vanishing line information does not converge to the reference error value, then in operation S150, the difference between the two vanishing lines and the vanishing line slope can be used to remove the edge points corresponding to the outliers. In operation S160, the vanishing lines of the front L4 camera and the rear L4 camera can be redetected.

[0066] For example, in operation S150, the edge points corresponding to outliers can be removed by removing the remaining edge points except the edge points between the vanishing line slope +α and the vanishing line slope -α shown in (b) of Figure 2 Thus, in operation S160, the front L4 camera can re-detect the vanishing line information of the front L4 camera by removing the remaining edge points except the edge points between the previous vanishing line slope +α and the previous vanishing line slope -α of the front L4 camera. In addition, in operation S160, the rear L4 camera can re-detect the vanishing line information of the rear L4 camera by removing the remaining edge points except the edge points between the previous vanishing line slope +α and the previous vanishing line slope -α of the rear L4 camera.

[0067] If the vanishing line information of the front L4 camera and the rear L4 camera is re-detected through the processes of operations S150 and S160, then in operation S120, the re-detected vanishing line information of the front L4 camera and the rear L4 camera can be respectively sent to each of the other cameras hierarchically connected to the front L4 camera and the rear L4 camera. In operation S130, at least one of the wide-angle left camera 310 and the wide-angle right camera 320 can recalculate the difference (α) between the two pieces of vanishing line information. In operation S140, at least one of the wide-angle left camera 310 and the wide-angle right camera 320 can determine whether the calculated difference between the two pieces of vanishing line information converges within a reference error value. The process including operations S120, S130, and S140 can be repeated until the difference between the two pieces of vanishing line information converges within the reference error value.

[0068] According to one embodiment, if the difference between the two pieces of vanishing line information converges within the reference error value, this may mean that the difference between the two vanishing lines is close to "0", which may indicate that the two pieces of vanishing line information are the same. The above-mentioned vanishing line information can be referred to as the final vanishing line information. In other words, if the difference between the two pieces of vanishing line information converges within a reference error value close to "0", then the vanishing line information may be the final vanishing line information.

[0069] The process of obtaining the final vanishing line information according to one embodiment will be described in more detail below with reference to Figure 4 FIG. is a schematic diagram for describing the process of obtaining the final vanishing line information according to one embodiment, assuming that the wide-angle right camera calculates the difference between the two pieces of vanishing line information.

[0070] Figure 4 FIG. is a schematic diagram for describing the process of obtaining the final vanishing line information according to one embodiment, assuming that the wide-angle right camera calculates the difference between the two pieces of vanishing line information.

[0071] As shown in Figure 4As shown, the process of obtaining the final vanishing line information may include, in operation ①, when the vanishing line information of the front L4 camera and the vanishing line information of the rear L4 camera are received via the vanishing line transmission path, calculating the difference (α) between the vanishing line information of the front L4 camera and the vanishing line information of the rear L4 camera by the wide-angle right camera.

[0072] When calculating the difference (α) between the two vanishing line information by the wide-angle right camera, the difference (α) between the two vanishing line information can be sent to the front L4 camera and the rear L4 camera. In operations ② and ③, the front L4 camera and the rear L4 camera can remove the edge points corresponding to the outliers by sequentially removing the edge points other than the edge points between the vanishing line slope +α and the vanishing line slope -α from the edge points with a small r value, and then re-detect the vanishing line information of the front L4 camera and the vanishing line information of the rear L4 camera.

[0073] When the vanishing line information of the front L4 camera and the rear L4 camera is re-detected, the difference (α') between the two re-detected vanishing line information can be recalculated in operation ④.

[0074] In operation ⑤, the process of operations ② - ④ can be repeated until the difference (α') between the two re-detected vanishing line information approaches '0', that is, converges within the reference error value.

[0075] If through this process, the difference (α') between the two vanishing line information converges within the reference error value, the vanishing line information of the front L4 camera and the rear L4 camera can be obtained as the final vanishing line information.

[0076] Refer again to Figure 1 , when the final vanishing line information is obtained through the above process, the pose of each camera can be estimated based on the vanishing point and the final vanishing line information of each camera in operation S170.

[0077] When estimating the pose of each camera in operation S170, the translation and / or rotation of each camera can be obtained based on the geometric relationship structure between the cameras and the design data of the vehicle in operation S180.

[0078] According to an embodiment, after obtaining the orientation between multiple cameras through epipolar geometry and obtaining the included angle between multiple cameras, the geometric relationship structure between the cameras can be obtained based on the included angle between the multiple cameras.

[0079] According to an embodiment, in operation S180, the translation of each of the multiple cameras can be obtained by i) scaling the geometric relationship structure between the multiple cameras to match the design data including the three-dimensional map of the vehicle, and ii) fitting the scaled geometric relationship structure between the multiple cameras to the wheel rim of the vehicle.

[0080] According to one embodiment, in operation S180, the following process may be repeatedly performed for all cameras: i) adjusting the translation such that in the scaling geometric relationship structure among the multiple cameras, any one camera overlaps with the rim of the vehicle, and ii) performing scaling such that the distance between each of the multiple cameras and the vehicle rim is minimized to obtain the translation of each camera.

[0081] According to one embodiment, in operation S180, the following process may be repeatedly performed for all cameras: i) adjusting the translation to overlap any one camera with the rim of the vehicle, ii) adjusting the rotation to align the remaining cameras with the vehicle rim, and iii) performing scaling to minimize the distance between each of the multiple cameras and the vehicle rim to obtain the translation of each of the multiple cameras.

[0082] Reference Figure 5 and 6 A process performed in operation S180 according to one embodiment will be described in more detail. In Figure 5 it, a wide-angle front camera, a wide-angle left camera, a wide-angle right camera, and a wide-angle rear camera are shown as examples of cameras.

[0083] As Figure 5 shown in (a) of it, according to the epipolar geometry, the wide-angle front camera can know in which directions the wide-angle right camera and the wide-angle left camera are located and know their included angle, although the wide-angle front camera does not know their lengths.

[0084] In this way, the included angles of all cameras can be known, and a square as shown in (b) of Figure 5 it can be obtained through the included angles of the wide-angle front camera, the wide-angle right camera, the wide-angle left camera, and the wide-angle rear camera. Since the orientations of the left and right cameras relative to a reference camera (such as the front L4 camera) are known, the included angle of each camera relative to the reference camera can be obtained.

[0085] In Figure 5 the square shown in (b) of it, the distances between the cameras may be unknown, but the ratios of the distances between the cameras may be known. For example, the ratios of the distances between the wide-angle front camera and the wide-angle left camera, between the wide-angle front camera and the wide-angle right camera, between the wide-angle left camera and the wide-angle rear camera, and between the wide-angle rear camera and the wide-angle right camera may be 1:0.6:1:0.7.

[0086] Since the distance ratio between cameras can be known from the square according to the orientation of the cameras, but the area and size of the square are unknown, it may be difficult to know which of the squares formed by the four wide-angle cameras has the actual area, as shown in Figure 5 an example of a square formed by four wide-angle cameras shown in (c) below. Therefore, the geometric relationship structure between the cameras can be obtained by finding the ratio value between the cameras and adjusting the ratio value.

[0087] In an embodiment, the geometric relationship structure between the cameras is a relationship structure formed by the included angles of the cameras, which may mean Figure 5 the structure formed by four wide-angle cameras shown in (b) below.

[0088] To find the ratio value between the cameras in the geometric relationship structure between the cameras, the ratio value between the cameras in the geometric shape relationship structure between the cameras can be obtained by adjusting the ratio value between the cameras, so that the geometric shape relationship structure between the cameras matches the design data including the three-dimensional view of the corresponding vehicle.

[0089] For example, the method according to an embodiment of the present disclosure may include scaling the geometric relationship structure such that the volume of the geometric relationship structure between the multiple cameras (where only the ratio between the cameras is known, as shown in Figure 6 (a) below) is the same as the volume of the design data of the actual vehicle to obtain the translation of each of the multiple cameras.

[0090] Then, as shown in Figure 6 (b) below, the translation can be adjusted such that one of the multiple cameras, for example, the wide-angle front camera 610, overlaps at a position corresponding to the design value (or design data) of the wide-angle front camera of the actual vehicle.

[0091] Then, as shown in Figure 6 (c) below, the rotation can be adjusted so that the remaining cameras are aligned with the rims of the design data of the actual vehicle. As can be seen from Figure 6 (c) below, the geometric relationship structure between the cameras after the rotation adjustment deviates from the rims of the actual vehicle.

[0092] For this purpose, the method according to an embodiment of the present disclosure may include performing scaling such that the distance between each camera and the vehicle rim is minimized, as shown in Figure 6 (d) below.

[0093] The processes shown in (b)-(d) above can be repeatedly executed for all cameras until no scaling is required, thereby obtaining the translation and / or rotation of each camera. Figure 6

[0094] ​Thus, in an embodiment, the process of obtaining the translation of each camera may include: scaling the geometric relationship structure between the cameras to match the actual design data of the vehicle; adjusting the translation of one camera to overlap with the design data, and then adjusting the rotation of the remaining cameras to align the remaining cameras with the vehicle rim of the design data; performing scaling such that the distance between each camera and the vehicle rim of the design data is minimized; and repeating the above process for all cameras until convergence to a preset condition.

[0095] As described above, the method of estimating the poses of multiple cameras according to an embodiment of the present disclosure may estimate the poses of multiple cameras based on the vanishing line information detected from two cameras mounted on a vehicle.

[0096] In addition, the method of estimating the poses of multiple cameras according to an embodiment of the present disclosure may estimate the poses of multiple cameras by transmitting the matching vanishing line information of the front camera and the rear camera to other cameras hierarchically connected to the front camera and the rear camera, and using the vanishing points of each camera and the transmitted vanishing line information.

[0097] In addition, the method of estimating the poses of multiple cameras according to an embodiment of the present disclosure may estimate the poses of multiple cameras based on the received vanishing line information and vanishing points, and may obtain the translation and / or rotation of each of the multiple cameras based on the geometric relationship structure between the multiple cameras to omit the offline calibration of the vehicle, thereby reducing the cost and time of calibration. In other words, the method of estimating the poses of multiple cameras according to an embodiment of the present disclosure may perform automatic calibration after the vehicle leaves the factory without performing pre-calibration.

[0098] Figure 7 The configuration of an apparatus for estimating the poses of multiple cameras according to another embodiment of the present disclosure is shown. Figure 7 is a block diagram of the configuration of an apparatus that can execute Figure 1-6 the method according to an embodiment.

[0099] Referring to Figure 7 , according to an embodiment of the present disclosure, an apparatus 700 for estimating the poses of multiple cameras may include a detector 710, an information transmitter 720, a calculator 730, a pose estimator 740, and an acquisition device 750.

[0100] The detector 710 may detect the vanishing lines in each of a first camera (e.g., a front L4 camera) and a second camera (e.g., a rear L4 camera) among the multiple cameras mounted on the vehicle, where the vanishing lines are detectable.

[0101] In one embodiment, the detector 710 may detect the vanishing points based on the image information from each camera.

[0102] According to one embodiment, if the difference between two vanishing line information is greater than a reference error value, the detector 710 may re-detect the vanishing line in each of the first camera and the second camera based on the difference between the two vanishing lines.

[0103] According to one embodiment, the detector 710 may re-detect the vanishing line by removing edge points corresponding to outliers based on the difference between two vanishing line information and the slope of the previous vanishing line.

[0104] The detector 710 may detect the vanishing point and the vanishing line by using various methods or schemes. The present disclosure is not limited to or restricted by the methods of detecting the vanishing point and the vanishing line described herein.

[0105] The information transmitter 720 may send the vanishing line information including the vanishing line of the first camera to each camera hierarchically connected to the first camera. The information transmitter 720 may also send the vanishing line information including the vanishing line of the second camera to each camera hierarchically connected to the second camera.

[0106] The calculator 730 may calculate the difference between the vanishing line information from the first camera and the vanishing line information from the second camera in at least one camera that receives two vanishing line information simultaneously.

[0107] The pose estimator 740 may obtain the final vanishing line information, where the difference between the two vanishing line information converges within a predetermined reference error value. The pose estimator 740 may estimate the pose of each of the multiple cameras based on the vanishing point detected in the multiple cameras and the final vanishing line information.

[0108] According to one embodiment, the pose estimator 740 may re-calculate the difference between the two vanishing line information including the re-detected vanishing line. If the difference between the two re-calculated vanishing line information converges within the reference error value, the pose estimator 740 may obtain the final vanishing line information.

[0109] The acquisition device 750 may acquire the translation of each of the multiple cameras based on the design data including the geometric relationship structure between the multiple cameras and the three-dimensional map of the vehicle.

[0110] After obtaining the orientation between the multiple cameras through epipolar geometry and obtaining the included angle between the multiple cameras, the geometric relationship structure between the multiple cameras may be obtained based on the included angle between the multiple cameras.

[0111] According to one embodiment, the acquisition device 750 can acquire the translation of each of the plurality of cameras by i) scaling the geometric relationship structure between the plurality of cameras to match the design data including the three-dimensional map of the vehicle, and ii) fitting the scaled geometric relationship structure between the plurality of cameras to the rim of the vehicle.

[0112] According to one embodiment, the acquisition device 750 can repeatedly perform the following processes: i) adjusting the translation so that any one camera overlaps with the rim of the vehicle in the scaled geometric relationship structure between the plurality of cameras, and ii) performing scaling so that the distance between each of the plurality of cameras and the rim of the vehicle is minimized, so that all cameras can acquire the translation of each of the plurality of cameras.

[0113] According to one embodiment, the acquisition device 750 can adjust the translation so that any one camera overlaps with the rim of the vehicle. The acquisition device 750 can also adjust the rotation so that the remaining cameras are aligned with the rim of the vehicle. The acquisition device 750 can perform scaling so that the distance between each of the plurality of cameras and the rim of the vehicle is minimized.

[0114] In various embodiments, even if the description of the device 700 with reference Figure 7 is omitted, the device 700 can include and / or be configured to perform all the content and / or operations described with reference Figure 1-6 which will be obvious to those of ordinary skill in the art.

[0115] Figure 8 A block diagram of a computing system for performing a method of estimating the poses of a plurality of cameras according to an embodiment of the present disclosure is shown.

[0116] Reference Figure 8 , a method of estimating the poses of a plurality of cameras according to an embodiment of the present disclosure can be implemented by a computing system 1000. The computing system 1000 can include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage 1600, and a network interface 1700, which are connected to each other via a bus 1200.

[0117] The processor 1100 can be a central processing unit (CPU) or a semiconductor device for processing instructions stored in the memory 1300 and / or the storage 1600. The memory 1300 and the storage 1600 can include various types of volatile or non-volatile storage media. For example, the memory 1300 can include a ROM (read-only memory) 1310 and a RAM (random access memory) 1320.

[0118] Accordingly, the operations of the methods or algorithms described in connection with the embodiments of the present disclosure may be embodied directly in hardware or in a software module executed by processor 1100, or in a combination thereof. The software module may reside on a storage medium (e.g., memory 1300 and / or storage 1600), such as RAM, flash memory, ROM, EPROM, EEPROM, registers, a hard disk, a removable disk, or a CD-ROM. The storage medium may be coupled to processor 1100, and processor 1100 may read information from the storage medium and may record information in the storage medium. Alternatively, the storage medium may be integrated with processor 1100. Processor 1100 and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside within a user terminal. Alternatively, processor 1100 and the storage medium may reside as separate components within the user terminal.

[0119] The embodiments described herein may be implemented with hardware elements and software components and / or a combination of hardware elements and software components. For example, the apparatus, methods, and components according to the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the OS. The processing device may also access, store, manipulate, process, and create data in response to the execution of the software. For ease of understanding, one processing device is described as being used. However, those of ordinary skill in the art should understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a single processor and a single controller. Additionally, different processing configurations, such as parallel processors, are possible.

[0120] The software may include a computer program, a piece of code, an instruction, or some combination thereof, for independently or jointly instructing or configuring the processing device to operate as desired. The software and data may be permanently or temporarily embodied in any type of machine, component, physical or virtual device, computer storage medium or device, or contained in a propagated signal wave capable of providing instructions or data to or being interpreted by the processing device. The software may also be distributed over network-coupled computer systems so that the software is stored and executed in a distributed manner. In one embodiment, the software and data may be stored by one or more computer-readable recording media.

[0121] The above method can be embodied in the form of program instructions, which can be executed by various computer devices and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be those specifically designed and constructed for the purpose of the inventive concept of the present invention, or they may be of the kind well-known and available to those of ordinary skill in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks, magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROMs, RAMs, flash memories, etc. Examples of program instructions include not only machine code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter, etc. The above hardware devices may be configured to operate as one or more software modules to perform the operations of the present disclosure, and vice versa.

[0122] Although the technical concept of the present disclosure has been described with reference to exemplary embodiments and the drawings, it should be understood that those of ordinary skill in the art can make various modifications and changes based on the above description. For example, the described technology can be executed in an order different from the described method, and / or the components of the described system, structure, device, circuit, etc. can be combined or combined in a form different from the described method or other components, or even if replaced or substituted by equivalents, appropriate results can be achieved.

[0123] The above description is only an illustration of the technical idea of the present disclosure, and those skilled in the art to which the present disclosure pertains can make various modifications and changes without departing from the basic features of the present disclosure. Therefore, the embodiments disclosed in the present disclosure are not intended to limit the technical idea of the present disclosure, but to describe the present disclosure. The scope of the technical idea of the present disclosure is not limited by the embodiments. The protection scope of the present disclosure should be interpreted by the appended claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included within the scope of the present disclosure.

[0124] According to an embodiment of the present disclosure, the poses of multiple cameras can be estimated based on vanishing line information detected from two cameras installed on a vehicle.

[0125] In addition, according to an embodiment of the present disclosure, by sending the vanishing line information of the matched front camera and rear camera to other cameras hierarchically connected to the front camera and the rear camera, the poses of multiple cameras can be estimated using the vanishing points of each camera and the sent vanishing line information.

[0126] In addition, according to an embodiment of the present disclosure, the poses of multiple cameras can be estimated based on the received vanishing line information and vanishing points, and the translation and / or rotation of each of the multiple cameras can be obtained based on the geometric relationship structure between the multiple cameras, so as to omit the offline calibration of the vehicle, thereby reducing the cost and time of calibration.

[0127] The effects obtainable in the present disclosure are not limited to the above effects. Those of ordinary skill in the art to which the present disclosure pertains should clearly understand any other effects not mentioned herein from the above description.

[0128] In the foregoing, although the present disclosure has been described with reference to exemplary embodiments and the accompanying drawings, the present disclosure is not limited thereto. The present disclosure can be variously modified and changed by those of ordinary skill in the art to which the present disclosure pertains without departing from the spirit and scope of the present disclosure claimed in the appended claims.

Claims

1. An apparatus for estimating the poses of multiple cameras mounted on a vehicle, the apparatus comprising: a memory configured to store computer-executable instructions; and at least one processor configured to access the memory and execute the computer-executable instructions, wherein the at least one processor is configured to: detect a vanishing line from each of a first camera and a second camera among the multiple cameras mounted on the vehicle, the first camera and the second camera being capable of detecting the vanishing line; send first vanishing line information including the vanishing line of the first camera to each camera hierarchically connected to the first camera; send second vanishing line information including the vanishing line of the second camera to each camera hierarchically connected to the second camera; calculate a difference between the first vanishing line information and the second vanishing line information in at least one camera that receives the first vanishing line information of the first camera and the second vanishing line information of the second camera; obtain final vanishing line information in which the difference between the first vanishing line information and the second vanishing line information converges within a predetermined reference error value; and estimate the pose of each of the multiple cameras based on the vanishing points detected by the multiple cameras and the final vanishing line information.

2. The device according to claim 1, wherein The at least one processor is configured to: when the difference between the first vanishing line information and the second vanishing line information is greater than a predetermined reference error value, re-detect the vanishing line from each of the first camera and the second camera based on the difference between the first vanishing line information and the second vanishing line information; recalculate the difference between first re-calculated vanishing line information including the re-detected vanishing line of the first camera and second re-calculated vanishing line information including the re-detected vanishing line of the second camera; and when the difference between the first re-calculated vanishing line information and the second re-calculated vanishing line information converges within a predetermined reference error value, obtain the final vanishing line information.

3. The device according to claim 2, wherein The at least one processor is configured to: re-detect the vanishing line by removing edge points corresponding to outliers based on i) the difference between the first vanishing line information and the second vanishing line information and ii) a previous vanishing line slope.

4. The apparatus according to claim 1, wherein, The at least one processor is configured to: obtain the translation of each of the multiple cameras based on the geometric relationship structure between the multiple cameras and design data including a three-dimensional map of the vehicle.

5. The device according to claim 4, wherein After obtaining the orientation between the multiple cameras through epipolar geometry and obtaining the included angle between the multiple cameras, obtain the geometric relationship structure based on the included angle between the multiple cameras.

6. The apparatus according to claim 5, wherein The at least one processor is configured to obtain the translation of each of the plurality of cameras by: i) scaling the geometric relationship structure between the plurality of cameras to match design data including a three-dimensional map of the vehicle, and ii) fitting the scaled geometric relationship structure between the plurality of cameras to the rim of the vehicle.

7. The apparatus according to claim 6, wherein, The at least one processor is configured to repeatedly perform the following process: i) adjusting the translation such that any one of the cameras overlaps with the rim of the vehicle in the scaled geometric relationship structure between the plurality of cameras, and ii) performing scaling such that the distance between each of the plurality of cameras and the rim of the vehicle is minimized, so that all the cameras obtain the translation of each camera.

8. The apparatus according to claim 7, wherein The at least one processor is configured to: Adjust the translation such that any one of the cameras overlaps with the rim of the vehicle, and adjust the rotation such that the remaining cameras are aligned with the rim of the vehicle; And Perform scaling such that the distance between each of the plurality of cameras and the rim of the vehicle is minimized.

9. The device according to claim 1, wherein The plurality of cameras include a front camera, a rear camera, a left front camera, a right front camera, a wide-angle front camera, a left rear camera, a right rear camera, a wide-angle rear camera, a wide-angle left camera, and a wide-angle right camera.

10. A method for estimating the poses of a plurality of cameras mounted on a vehicle, the method comprising the steps of: Detecting a vanishing line from each of a first camera and a second camera among the plurality of cameras mounted on the vehicle that are capable of detecting the vanishing line; Sending first vanishing line information including the vanishing line of the first camera to each camera hierarchically connected to the first camera; Sending second vanishing line information including the vanishing line of the second camera to each camera hierarchically connected to the second camera; Calculating a difference between the first vanishing line information and the second vanishing line information in at least one camera that receives the first vanishing line information of the first camera and the second vanishing line information of the second camera; Obtaining final vanishing line information in which the difference between the first vanishing line information and the second vanishing line information converges within a predetermined reference error value; And estimating the pose of each of the plurality of cameras based on the vanishing points detected by the plurality of cameras and the final vanishing line information.

11. The method according to claim 10, wherein: Detecting the vanishing line includes: when the difference between the first vanishing line information and the second vanishing line information is greater than a predetermined reference error value, re-detecting the vanishing line from each of the first camera and the second camera based on the difference between the first vanishing line information and the second vanishing line information; and Obtaining the final vanishing line information includes: the difference between first recalculated vanishing line information including a recalculated vanishing line of the first camera and second recalculated vanishing line information including a recalculated vanishing line of the second camera, and obtaining the final vanishing line information when the difference between the first recalculated vanishing line information and the second recalculated vanishing line information converges within the predetermined reference error value.

12. The method according to claim 11, wherein, Recalculating the vanishing line includes: recalculating the vanishing line by removing edge points corresponding to outliers based on i) the difference between the first vanishing line information and the second vanishing line information and ii) a previous vanishing line slope.

13. The method according to claim 10, further comprising the step of obtaining a translation of each of the cameras based on a geometric relationship structure between the plurality of cameras and design data including a three-dimensional map of the vehicle.

14. The method according to claim 13, wherein, After obtaining an orientation between the plurality of cameras through epipolar geometry and obtaining an included angle between the plurality of cameras, the geometric relationship structure is obtained based on the included angle between the plurality of cameras.

15. The method according to claim 14, wherein, Obtaining a translation of each of the plurality of cameras includes: i) obtaining a translation of each camera by scaling the geometric relationship structure between the plurality of cameras to match design data including a three-dimensional map of the vehicle, and ii) fitting the scaled geometric relationship structure between the plurality of cameras to a rim of the vehicle.

16. The method according to claim 15, wherein, Obtaining the translation of each of the plurality of cameras includes repeatedly performing the following process: i) adjusting the translation such that any one of the cameras overlaps with the rim of the vehicle in the scaled geometric relationship structure between the plurality of cameras, and ii) performing scaling such that the distance between each of the plurality of cameras and the rim of the vehicle is minimized, so that each of the plurality of cameras obtains the translation of each of the plurality of cameras.

17. The method according to claim 16, wherein, Obtaining the translation of each of the plurality of cameras includes: adjusting the translation such that any one of the cameras overlaps with the rim of the vehicle, adjusting the rotation such that the remaining cameras are aligned with the rim of the vehicle; and performing scaling such that the distance between each of the plurality of cameras and the rim of the vehicle is minimized.

Images