Device and method for obtaining coordinate transformation information

Abstract

According to an aspect of the present disclosure, a method for obtaining coordinate system transformation information is provided. The method includes: obtaining three-dimensional information including first lane information corresponding to a lane adjacent to a vehicle by a lidar mounted on the vehicle, and obtaining a surrounding image including second lane information corresponding to the lane by an imaging device mounted on the vehicle; and obtaining first coordinate system transformation information regarding the lidar and the imaging device by matching the second lane information with the first lane information; and obtaining second coordinate system transformation information regarding the vehicle and the imaging device by using top view image transformation information obtained based on the surrounding image and the traveling direction of the vehicle.

Description

Technical Field

[0001] The present disclosure relates to an apparatus for obtaining coordinate transformation information between a vehicle, a lidar mounted on the vehicle, and a camera device mounted on the vehicle, and a method for obtaining the coordinate transformation information. As a reference, this application claims priority based on a Korean patent application (No. 10-2018-0096210) filed on August 17, 2018, the entire content of which is incorporated herein by reference. Background Art

[0002] A vehicle generally refers to a transportation machine powered by fossil fuel or electricity and running on roads or tracks.

[0003] With the development of technology, vehicles have been developed to provide various functions to drivers. In particular, in line with the electrification trend of vehicles, vehicles with an active safety system (ASS) have emerged, which is activated to avoid accidents.

[0004] In addition, active research is currently being carried out on vehicles equipped with an advanced driver assistance system (ADAS), which actively provides information about the driving environment, such as the state of the vehicle, the state of the driver, and the surrounding environment of the vehicle, to reduce the driver's burden and improve convenience for the driver.

[0005] The advanced driver assistance system may have a detection tool for detecting the driving environment. For example, the detection tool may include a camera device and a lidar (LiDAR). The camera device and the lidar are mounted on the outside or inside of the vehicle, and they can detect the driving environment by acquiring images or point clouds corresponding to their installation positions and attitude angles. Summary of the Invention

[0006] The problem to be solved by the present disclosure is to provide a coordinate transformation information acquisition apparatus and method for obtaining coordinate transformation information between an in-vehicle camera device, a lidar, and a moving vehicle.

[0007] The problems to be solved by the present disclosure are not limited to the above-mentioned problems, and other problems clearly understood by those skilled in the art not mentioned herein will also be included in the problems to be solved by the present disclosure.

[0008] According to one aspect of the present disclosure, there is provided a method for obtaining coordinate system transformation information, the method comprising: obtaining three-dimensional information including first lane information corresponding to a lane adjacent to a vehicle by a lidar mounted on the vehicle, and obtaining a surrounding image including second lane information corresponding to the lane by an imaging device mounted on the vehicle; obtaining first coordinate system transformation information regarding the lidar and the imaging device by matching the second lane information with the first lane information; and obtaining second coordinate system transformation information regarding the vehicle and the imaging device by using top view image transformation information obtained based on the surrounding image and the traveling direction of the vehicle.

[0009] According to another aspect of the present disclosure, there is provided a device for obtaining coordinate system transformation information, the device comprising: a horizontal ground recognition unit configured to recognize whether the vehicle is traveling on a horizontal ground based on first lane information regarding a lane adjacent to the vehicle obtained by a lidar mounted on the vehicle; a first coordinate system transformation information obtaining unit configured to match second lane information regarding a lane in a surrounding image of the vehicle obtained by an imaging device mounted on the vehicle with the first lane information to obtain first coordinate system transformation information regarding the lidar and the imaging device; and a second coordinate system transformation information obtaining unit configured to obtain second coordinate system transformation information regarding the vehicle and the imaging device by using top view image transformation information obtained based on the surrounding image and the traveling direction of the vehicle.

[0010] The device and method for obtaining coordinate system transformation information according to an embodiment of the present disclosure can obtain coordinate system transformation information of a lidar and an imaging device for a traveling vehicle without the need for equipment or manual operation, thereby reducing the cost and time required for calibration. Description of the Drawings

[0011] Figure 1 and Figure 2 shows a functional block diagram of a coordinate system transformation information acquisition system according to various embodiments.

[0012] Figure 3 is a view for explaining the coordinate systems of a vehicle, an imaging device, and a lidar according to an embodiment.

[0013] Figure 4 is a flowchart of a method for identifying whether a vehicle is traveling on a horizontal ground in a method for obtaining coordinate system transformation information according to an embodiment.

[0014] Figure 5 is a view for explaining a method for extracting first lane information from three-dimensional information obtained by a lidar of a vehicle according to an embodiment.

[0015] Figure 6 It is a flowchart of a method for obtaining first coordinate system transformation information in a method for obtaining coordinate system transformation information according to an embodiment.

[0016] Figure 7 It is a view for explaining a method of extracting second lane information from a surrounding image obtained by a camera device of a vehicle according to an embodiment.

[0017] Figure 8 It is a flowchart of a method for obtaining second coordinate system transformation information in a method for obtaining coordinate system transformation information according to an embodiment.

[0018] Figure 9 It is a view for explaining a method of extracting an extended focus from a surrounding image obtained by a camera device of a vehicle according to an embodiment.

[0019] Figure 10 It is a view of a top view image of a surrounding image obtained by a camera device of a vehicle passing through according to an embodiment. Detailed Embodiments

[0020] Advantages and features of embodiments of the present disclosure and methods for implementing them will be clearly understood from the descriptions of the embodiments made below in conjunction with the accompanying drawings. However, the present disclosure is not limited to these embodiments, but is implemented in various forms. Note that the embodiments are provided for a full disclosure and also enable those skilled in the art to know the entire scope of the present disclosure.

[0021] In the following description, if well-known functions and / or configurations will unnecessarily obscure the features of the present disclosure, they will not be described in detail. In addition, the terms to be described below are defined in consideration of their functions in the embodiments of the present disclosure and vary according to the intention or practice of the user or operator. Therefore, they are defined based on the entire content of the present disclosure.

[0022] Figure 1 and Figure 2 shows a functional block diagram of a coordinate system transformation information acquisition system according to each embodiment. Figure 3 It is a view for explaining the coordinate systems of a vehicle, a camera device, and a lidar according to an embodiment.

[0023] Referring to Figure 1 , a coordinate system transformation information acquisition system 1 according to an embodiment may include a vehicle V and a coordinate system transformation information acquisition device 100.

[0024] The vehicle V may refer to a means of transportation that allows people, objects, or animals to move from one place to another when traveling along a road or track. According to one embodiment, the vehicle V may include a three-wheeled vehicle or a four-wheeled vehicle, a two-wheeled vehicle such as a motorcycle, construction machinery, motorized equipment, a bicycle, and a train operating on a track.

[0025] Figure 1 The vehicle V may pre-store an accurate map. Here, the accurate map may refer to the following map: the map has high accuracy for safe and accurate control of the vehicle V and contains information about the height, slope, curvature, etc. of the road and the planar position of the road.

[0026] In addition, the accurate map is a map in which at least the lanes are marked, and may additionally include road facilities such as road signs, traffic lights, and guardrails.

[0027] The accurate map includes point clouds. Each point cloud is a set of points obtained by scanning the road with a laser scanner or the like, and each point in the point cloud may have three-dimensional spatial coordinates in a base coordinate system. The accurate map can be constructed by filtering meaningful data from the acquired point clouds with a noise filter and then marking landmarks in each point cloud.

[0028] Here, the base coordinate system refers to an orthogonal coordinate system that does not depend on any device and may include a world coordinate system.

[0029] In addition, the accurate map can also be stored in the coordinate system transformation information acquisition device 100 and the vehicle V.

[0030] In addition, Figure 1 The vehicle V may be equipped with an advanced driver assistance system (ADAS). Here, the advanced driver assistance system may refer to a system that provides driving environment information such as the state of the vehicle V, the state of the driver, and surrounding environment information or actively controls the vehicle V. For example, the vehicle V may be equipped with a lane departure warning system (LDWS), a lane keeping assist system (LKAS), etc. It should be noted that the advanced driver assistance system installed in the vehicle V is not limited to the above.

[0031] Since the advanced driver assistance system operates in response to the driving environment information of the vehicle V, the vehicle V may include a detection tool for detecting the driving environment information provided to the advanced driver assistance system. The detection tool according to one embodiment may include a radar and / or an ultrasonic sensor. The radar detects the driving environment by emitting pulses around the vehicle V and receiving echo pulses reflected from an object in that direction. The ultrasonic sensor emits ultrasonic waves around the vehicle V and receives echo ultrasonic waves reflected from an object in that direction.

[0032] In addition, the vehicle V may include a camera device C as a detection tool. The camera device C may be configured to face the front, side, and / or rear of the vehicle V and capture images in the corresponding directions. The captured images may serve as a basis for obtaining information such as lanes, road signs, and objects around the vehicle V through an image processing process.

[0033] Hereinafter, an image captured by the camera device C mounted on the vehicle V is referred to as a surrounding image of the vehicle, and the surrounding image may include a front image captured by the camera device C configured to face the front of the vehicle V, a rear image captured by the camera device C configured to face the rear of the vehicle V, and a side image captured by the camera device C configured to face the side of the vehicle V.

[0034] In addition, the vehicle V may further include a lidar L as a detection tool. The lidar L may be configured to face the front, side, and / or rear of the vehicle V and emit laser light in the corresponding directions. The lidar L can detect three-dimensional information of the surrounding environment of the vehicle V by receiving the laser light reflected from an object located in the emission direction of the laser, as driving environment information of the vehicle V.

[0035] In this case, the surrounding image obtained by the camera device C and the three-dimensional information detected by the lidar L may include information about at least two identical lanes. This will be described later.

[0036] Meanwhile, the vehicle V may incorporate CAN (Controller Area Network) DATA, for example, steering angle information and yaw rate information transmitted via CAN communication (i.e., a method of communication between the camera device C, the lidar L, and modules inside the vehicle V), and use the CAN DATA to control the vehicle V. In this case, the image obtained by the camera device C may conform to the camera device coordinate system, the point cloud obtained by the lidar L may conform to the lidar coordinate system, and the CAN DATA may conform to the vehicle coordinate system.

[0037] Figure 3 is a schematic plan view of a vehicle V according to an embodiment, which shows the coordinate system of the vehicle V, the coordinate system of the lidar L installed in the vehicle V, the coordinate system of the camera device C installed in the vehicle V, and the coordinate system of the earth's surface R based on these coordinate systems. Refer to Figure 3 , the vehicle V may have the following vehicle coordinate system: the vehicle coordinate system includes an X v axis along the traveling direction of the vehicle V, a Z v axis along the direction perpendicular to the earth's surface, and a Y v axis perpendicular to the X v axis and the Z v axis, where O vis the origin. On the other hand, the lidar L installed in the vehicle V may have an X l axis, a Y l axis, and a Z l axis, and a lidar coordinate system with O l as the origin. The X l axis, Y l axis, and Z l axis are determined by the installation position and attitude angle of the lidar L. In addition, the imaging device C installed in the vehicle V may have an X c axis, a Y c axis, a Z c axis, and an imaging device coordinate system with O c as the origin. The X c axis, Y c axis, and Z c axis are determined by the installation position and attitude angle of the imaging device C. Additionally, the coordinate system of the earth's surface R refers to the coordinate system of the top-view image transformed from the image obtained by the imaging device C, where O r is used as the origin, the X r axis and Y r axis exist on the earth's surface, and the Z r axis is defined to be in the opposite direction of the Z v axis in the vehicle coordinate system. The coordinate systems need to be unified to incorporate information in different coordinate systems, which is called calibration.

[0038] To this end, calibration of the imaging device C and the lidar L can be performed before the vehicle V is driven. Specifically, the vehicle V can be parked at a predetermined position, and then images and point clouds of calibration points pre-perceived from that position can be acquired. Next, the coordinates of the calibration points with respect to the imaging device coordinate system, the coordinates of the calibration points with respect to the lidar coordinate system, and the coordinates of the calibration points with respect to the vehicle coordinate system are compared to obtain the coordinate system transformation information between them.

[0039] However, the above method requires the vehicle V to be parked correctly at the predetermined position and also requires precise perception of the positions of the calibration points. This preliminary work is performed manually by a person, which causes difficulties in terms of correctness and will take a large amount of time and money to achieve high precision. In addition, if the imaging device and / or the lidar are replaced or their positions are changed, the preliminary calibration process needs to be performed before driving.

[0040] To solve this problem, the coordinate system transformation information acquisition device 100 according to one embodiment can perform calibration in real time on a moving vehicle. Refer back to Figure 1, the coordinate system transformation information acquisition device 100 according to an embodiment can acquire the coordinate system transformation information between the vehicle V, the imaging device C, and the lidar L by using the information received from the traveling vehicle V.

[0041] In order to receive the surrounding environment information detected by the traveling vehicle V, the coordinate system transformation information acquisition device 100 can exchange information by communicating with the vehicle V according to various known communication methods. The coordinate system transformation information acquisition device 100 according to an embodiment can adopt known communication methods such as CDMA, GSM, W-CDMA, TD-SCDMA, WiBro, LTE, or EPC to communicate with the vehicle V via a base station. On the contrary, the coordinate system transformation information acquisition device 100 according to another embodiment can adopt communication methods such as wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), UWB (Ultra-Wideband), IrDA (Infrared Data Association), BLE (Bluetooth Low Energy), and NFC (Near Field Communication) to communicate with the vehicle V within a predetermined distance. However, the method for the coordinate system transformation information acquisition device 100 to communicate with the vehicle V is not limited to the above embodiments.

[0042] The coordinate system transformation information acquisition device 100 can acquire the coordinate system transformation information based on the surrounding images acquired by the imaging device C mounted on the vehicle V and the point cloud acquired by the lidar L. To this end, the coordinate system transformation information acquisition device 100 according to an embodiment can include: a lane information acquisition unit 140; a horizontal ground recognition unit 110; a first coordinate system transformation information acquisition unit 120; and a second coordinate system transformation information acquisition unit 130.

[0043] The lane information acquisition unit 140 can acquire first lane information corresponding to the lane adjacent to the vehicle V through the lidar L mounted at the vehicle V. Specifically, the lane information acquisition unit 140 according to an embodiment can receive the three-dimensional information of the surrounding environment of the vehicle V acquired by the lidar L of the vehicle V. Next, the lane information acquisition unit 140 according to an embodiment can extract the first lane information about the lane on the three-dimensional information of the surrounding environment of the vehicle V.

[0044] The above-described embodiments have been described by taking as an example that the lane information acquisition unit 140 directly receives the three-dimensional information obtained by the lidar L from the own vehicle V. On the contrary, according to another embodiment, the lane information acquisition unit 140 may receive only the first lane information from the vehicle V. That is, once the vehicle V extracts the first lane information from the three-dimensional information obtained by the lidar L and then transmits the extracted first lane information to the horizontal ground recognition unit 110, when receiving the first lane information, the horizontal ground recognition unit 110 may determine whether the vehicle V is traveling on a horizontal ground by using the first lane information.

[0045] In addition, the lane information acquisition unit 140 may acquire a surrounding image including second lane information corresponding to a lane through the imaging device C of the vehicle V. Further, the lane information acquisition unit 140 may receive the second lane information extracted from the surrounding image of the vehicle V. On the contrary, the lane information acquisition unit 140 may directly extract the second lane information from the received surrounding image.

[0046] In this case, the first lane information extracted from the three-dimensional information and the second lane information extracted from the surrounding image may need to include information on at least two corresponding lanes. For this purpose, the lidar L and the imaging device C may be installed in the vehicle V such that they acquire the three-dimensional information and the surrounding image both including at least two corresponding lanes.

[0047] In addition, the lane information acquisition unit 140 may receive the three-dimensional information and the surrounding image acquired at the same time point by the lidar L and the imaging device C of the vehicle V, respectively. As a result, the three-dimensional information obtained by the lidar L and the surrounding image obtained by the imaging device C may include information on the lanes existing adjacent to the vehicle V. The horizontal ground recognition unit 110 may identify whether the vehicle V is traveling on a horizontal ground based on the first lane information of the surrounding environment of the vehicle V obtained by the lidar L of the vehicle V. When the vehicle V is traveling on a horizontal ground, the lane information on the point cloud and the image of the surrounding environment may be used to achieve calibration accuracy. Therefore, the coordinate system transformation information acquisition device 100 may identify whether the vehicle V is traveling on a horizontal ground by the horizontal ground recognition unit 110 before acquiring the coordinate system transformation information.

[0048] Specifically, the horizontal ground recognition unit 110 may fit a plane based on the first lane information acquired by the lane information acquisition unit 140, and if the fitting error in the fitted plane is equal to or less than a predetermined reference error, it may be determined that the vehicle V is traveling on a horizontal ground.

[0049] The first coordinate system transformation information acquisition unit 120 can compare the second lane information in the surrounding image of the vehicle V with the first lane information to acquire the first coordinate system transformation information regarding the lidar L and the imaging device C. As described above, in order to improve the accuracy of the acquired coordinate system transformation information, the first coordinate system transformation information acquisition unit 120 can perform the operation for acquiring the first coordinate system transformation information only when the vehicle V is traveling on a horizontal ground.

[0050] Specifically, once the vehicle V is recognized by the horizontal ground recognition unit 110 as traveling on a horizontal ground, the first coordinate system transformation information acquisition unit 120 according to one embodiment can extract the second lane information regarding the lanes existing in the surrounding image received by the lane information acquisition unit 140. Next, the first coordinate system transformation information acquisition unit 120 can acquire the first coordinate system transformation information by matching the extracted second lane information with the previously extracted first lane information.

[0051] The above embodiment has been described by taking the example that the first coordinate system transformation information acquisition unit 120 is only provided with the surrounding image received by the lane information acquisition unit 140. On the contrary, the first coordinate system transformation information acquisition unit 120 according to another embodiment can receive the second lane information extracted from the surrounding image from the lane information acquisition unit 140. That is, once the vehicle V extracts the second lane information from the surrounding image acquired by the imaging device C and then sends the extracted second lane information to the lane information acquisition unit 140, the lane information acquisition unit 140 can provide the second lane information and the surrounding image to the first coordinate system transformation information acquisition unit 120, and the first coordinate system transformation information acquisition unit 120 can determine whether the vehicle V is traveling on a horizontal ground by using the second lane information.

[0052] The second coordinate system transformation information acquisition unit 130 can acquire the second coordinate system transformation information regarding the vehicle V and the imaging device C by using the top view image transformation information of the surrounding image and the traveling direction of the vehicle. Similar to the first coordinate system transformation information acquisition unit 120, the second coordinate system transformation information acquisition unit 130 can perform the operation for acquiring the second coordinate system transformation information only when the vehicle V is traveling on a horizontal ground.

[0053] Specifically, the second coordinate system transformation information acquisition unit 130 according to one embodiment can obtain the focus of expansion from the surrounding image acquired by the imaging device C mounted on the vehicle V and provided by the lane information acquisition unit 140, and obtain the traveling direction of the vehicle V based on the acquired focus of expansion. After obtaining the traveling direction of the vehicle V, the second coordinate system transformation information acquisition unit 130 can obtain the top view image transformation information by using the width and direction of the lane acquired from the first lane information. Finally, the second coordinate system transformation information acquisition unit 130 obtains the second coordinate system transformation information based on the inverse information of the top view image transformation information and the traveling direction of the vehicle V.

[0054] The above embodiment has been described by taking as an example that the second coordinate system transformation information acquisition unit 130 receives the surrounding image acquired by the imaging device C from the vehicle V through the lane information acquisition unit 140. On the contrary, the second coordinate system transformation information acquisition unit 130 according to another embodiment can receive the position of the focus of expansion and the surrounding image from the vehicle V through the lane information acquisition unit 140. That is, once the vehicle V identifies the position of the focus of expansion in the surrounding image acquired by the imaging device C and then sends the position of the focus of expansion together with the surrounding image to the lane information acquisition unit 140, the lane information acquisition unit 140 provides the received surrounding image and the position of the focus of expansion to the second coordinate system transformation information acquisition unit 130, and when receiving the surrounding image and the position of the focus of expansion, the second coordinate system transformation information acquisition unit 130 can obtain the traveling direction of the vehicle V by using the focus of expansion.

[0055] Meanwhile, although Figure 1 the coordinate system transformation information acquisition device 100 is shown as being separately configured from the vehicle V and constituting the coordinate system transformation information acquisition system 1, the coordinate system transformation information acquisition device 100 can also be included as a component of the vehicle V.

[0056] Referring to Figure 2 , the coordinate system transformation information acquisition system 1 according to another embodiment can be configured as the vehicle V including the coordinate system transformation information acquisition device 100. It should be noted that except for the manner in which the coordinate system transformation information acquisition device 100 is configured, Figure 1 the coordinate system transformation information acquisition system 1 of Figure 2 and

[0057] According to Figure 1 and Figure 2Each of the components of the coordinate system transformation information acquisition device 100 according to an embodiment can be implemented as a computing device including a microprocessor - for example, at least one of a central processing unit (CPU) and a graphics processing unit (GPU). Conversely, at least two of the components of the coordinate system transformation information acquisition device 100 can be implemented as a system on a chip (SOC).

[0058] So far, the components of the coordinate system transformation information acquisition system 1 have been described. Referring to Figures 4 to 10 , the coordinate system transformation information acquisition method executed by the coordinate system transformation information acquisition system 1 will be described below.

[0059] First, the vehicle V can acquire three-dimensional information of the surrounding environment of the vehicle V through the lidar L.

[0060] The coordinate system transformation information acquisition system 1 can identify whether the vehicle V is traveling on a horizontal ground based on the first lane information of the surrounding environment of the vehicle V acquired through the lidar L of the vehicle V. This will be described with reference to Figure 4 and Figure 5 hereinafter.

[0061] Figure 4 is a flowchart of a method for identifying whether a vehicle is traveling on a horizontal ground in the coordinate system transformation information acquisition method according to an embodiment. Figure 5 is a view for explaining a method of extracting first lane information from three-dimensional information acquired through a lidar of a vehicle according to an embodiment.

[0062] Referring to Figure 4 , first, the horizontal ground recognition unit 110 of the coordinate system transformation information acquisition device 100 can extract first lane information from the three-dimensional information of the surrounding environment of the vehicle V (S100). Here, the three-dimensional information may refer to a point cloud generated by laser light reflected by an object around the vehicle V. Specifically, the lidar L can receive laser light reflected by an object existing within the radar emission area in the surrounding environment and generate a point cloud, which is a set of points whose brightness values vary with the intensity of the received laser light. For example, for an object with a high laser reflection ratio, the lidar L can increase the brightness value of the point corresponding to the position of the object.

[0063] In addition, the first lane information may include the curvature derivative, curvature, direction, offset, etc. of the speculated lane area on the point cloud constituting the three-dimensional information.

[0064] Since the laser reflection ratio of the lane area on the road around the vehicle V is higher than that of its surrounding area, in the three-dimensional information obtained by the lidar L, the brightness value of the points corresponding to the lane area can be higher than that of the surrounding area. Therefore, according to one embodiment, the horizontal ground recognition unit 110 can extract first lane information about the lane from the three-dimensional information based on the brightness value patterns of the points corresponding to the lane area and its surrounding area on the road.

[0065] Figure 5 is a plan view of the three-dimensional information obtained by the lidar L disposed in the vehicle V corresponding to the central rectangular area. In Figure 5 it, point clouds extending left and right can be seen, and the horizontal ground recognition unit 110 can recognize the point clouds as lanes and extract first lane information about the lanes.

[0066] To extract the first lane information, the horizontal ground recognition unit 110 according to one embodiment can use one of the well-known pattern recognition techniques, or can use a machine learning method such as deep learning.

[0067] Although the above embodiment assumes that the horizontal ground recognition unit 110 of the coordinate system transformation information acquisition device 100 performs step S100 by obtaining information through coordinate system transformation, step S100 can also be performed by the vehicle V, and the vehicle V can send the first lane information to the coordinate system transformation information acquisition device 100. Conversely, the vehicle V can send the three-dimensional information to the coordinate system transformation information acquisition device 100, and then the lane information acquisition unit 140 of the coordinate system transformation information acquisition device 100 can extract the first lane information from the received three-dimensional information and provide the first lane information to the horizontal ground recognition unit 110.

[0068] Next, the horizontal ground recognition unit 110 of the coordinate system transformation information acquisition device 100 can fit a plane based on the extracted first lane information (S110). Since the lane is set on the driving road, the driving road can be fitted to the plane by obtaining the equation of the plane formed by the points on the first lane information corresponding to the lane. Specifically, the horizontal ground recognition unit 110 can obtain the coefficients a, b, c, and d of the plane equation ax + by + cz = d by using at least four points (assuming they have coordinates (x, y, z)) that make up the first lane information.

[0069] After fitting the plane, the horizontal ground recognition unit 110 of the coordinate system transformation information acquisition device 100 can compare whether the error in the fitted plane is equal to or less than a reference error (S120). Here, the reference error may refer to the maximum value of the error that a reliable fitted plane equation has for the first lane information. Specifically, the horizontal ground recognition unit 110 can obtain a fitting error according to Equation 1:

[0070] [Equation 1]

[0071]

[0072] where (x i , y i , z i ) represents the coordinates of the points constituting the i-th first lane information.

[0073] If the error in the fitted plane exceeds the reference error, the horizontal ground recognition unit 110 of the coordinate system transformation information acquisition device 100 can extract the first lane information again. On the other hand, if the error in the fitted plane is equal to or less than the reference error, the horizontal ground recognition unit 110 of the coordinate system transformation information acquisition device 100 can determine that the vehicle V is traveling on a horizontal ground (S130).

[0074] Therefore, once it is determined based on the first lane information that the vehicle V is traveling on a horizontal ground, the coordinate system transformation information acquisition device 100 can acquire first coordinate system transformation information regarding the lidar L and the imaging device C. This will be described with reference to Figure 6 and Figure 7 .

[0075] Figure 6 is a flowchart of a method for acquiring first coordinate system transformation information in a coordinate system transformation information acquisition method according to an embodiment. Figure 7 is a view for explaining a method of extracting second lane information from surrounding images acquired by an imaging device of a vehicle according to an embodiment.

[0076] First, the first coordinate system transformation information acquisition unit 120 of the coordinate system transformation information acquisition device 100 according to an embodiment can identify whether the vehicle V is traveling on a horizontal ground (S200). If the vehicle V is not traveling on a horizontal ground, the first coordinate system transformation information acquisition unit 120 can repeatedly identify whether the vehicle V is traveling on a horizontal ground.

[0077] On the other hand, if the vehicle V is traveling on a horizontal ground, the first coordinate system transformation information acquisition unit 120 according to an embodiment can extract second lane information from the surrounding image (S210). Here, the second lane information may include the curvature derivative, curvature, direction, offset, etc. of a hypothetical lane area in the surrounding image.

[0078] Figure 7 is an example diagram of a surrounding image acquired by the imaging device C of the vehicle V traveling on a horizontal ground. In Figure 7 the surrounding image, white lines on a black road are seen, and the first coordinate system transformation information acquisition unit 120 can recognize the white lines as lanes and extract second lane information about the lanes. Figure 7 The lines extending in the same direction as the lanes are shown in thick lines.

[0079] To extract the second lane information, the first coordinate system transformation information acquisition unit 120 according to an embodiment can use one of well-known pattern recognition techniques or can use a machine learning method such as deep learning.

[0080] Although the above embodiment assumes that step S210 is performed by the first coordinate system transformation information acquisition unit 120 of the coordinate system transformation information acquisition device 100, step S210 can be performed by the vehicle V, and the vehicle V can send the obtained second lane information to the coordinate system transformation information acquisition device 100. Conversely, the vehicle V can send the surrounding image to the coordinate system transformation information acquisition device 100, and then, the lane information acquisition unit 140 of the coordinate system transformation information acquisition device 100 can extract the second lane information from the received surrounding image and provide the second lane information to the first coordinate system transformation information acquisition unit 120.

[0081] Once the second lane information is extracted, the first coordinate system transformation information acquisition unit 120 can obtain the first coordinate system transformation information by matching the extracted second lane information with the first lane information (S220). As described above, the first lane information and the second lane information include information about at least two corresponding lanes. Therefore, the first coordinate system transformation information acquisition unit 120 can match the first lane information and the second lane information of the corresponding lanes.

[0082] Specifically, the first coordinate system transformation information acquisition unit 120 can obtain the first coordinate system transformation information between the imaging device C and the lidar L according to Mathematical Formula 2:

[0083] [Mathematical Formula 2]

[0084]

[0085] where the solution T* of Mathematical Formula 2(c,l) represents a three-dimensional transformation matrix, which represents the attitude angles of the lidar L with respect to the coordinate system of the imaging device C, and serves as the first coordinate system transformation information, Z k represents the coordinates of the second lane information extracted from the surrounding image, P k represents the point corresponding to Z k on the first lane information, C zk and C pk respectively represent the covariance of the errors of Z k and P k , and H represents the Jacobian determinant of the function h(). The function h() can be the following function: the function transforms P (c,l) into the coordinate values in the coordinate system of the imaging device C by means of the three-dimensional transformation matrix T k , and projects it into a two-dimensional image by means of the intrinsic parameters of the imaging device C.

[0086] In order to obtain T* by using Mathematical Formula 2 (c,l) , the first coordinate system transformation information acquisition unit 120 can perform the following steps. In the first step, the first coordinate system transformation information acquisition unit 120 can transform the coordinates of the points on the first lane information into the coordinate values in the coordinate system of the imaging device C by using T (c,l) , then search for the pixels corresponding to the second lane information in the surrounding image, and then obtain T (c,l) representing the attitude angles of the lidar L with respect to the coordinate system of the imaging device C by using Mathematical Formula 2. In the second step, the solution T* of Mathematical Formula 2 can be found by repeating the first step until the difference between the previous T (c,l) and the current T (c,l) becomes equal to or less than the threshold. (c,l) .

[0087] In order to find the solution of Mathematical Formula 2, the first coordinate system transformation information acquisition unit 120 can select at least one of the well-known algorithms, for example, the Gauss Newton algorithm or the Levenberg-Marquardt algorithm.

[0088] After obtaining the first coordinate system transformation information by the above method, the coordinate system transformation information acquisition device 100 can obtain the second coordinate system transformation information about the vehicle V and the imaging device C. This will be described with reference to Figures 8 to 10 .

[0089] Figure 8 is a flowchart of the method for obtaining the second coordinate system transformation information in the coordinate system transformation information acquisition method according to an embodiment. Figure 9It is a view for explaining a method of extracting an extended focus from a surrounding image obtained by a vehicle's imaging device according to an embodiment. Figure 10 It is a view of a top - view image of a surrounding image obtained by a vehicle's imaging device according to an embodiment.

[0090] Referring to Figure 8 , the second coordinate system transformation information acquisition unit 130 of the coordinate system transformation information acquisition device 100 can obtain the traveling direction of the vehicle V based on the extended focus in the surrounding image (S300). To this end, the second coordinate system transformation information acquisition unit 130 can first determine whether the vehicle V is traveling straight forward. Specifically, the second coordinate system transformation information acquisition unit 130 can determine whether the vehicle V is traveling straight forward based on at least one of the steering angle and yaw rate of the vehicle V. If the absolute values of the steering angle and yaw rate of the vehicle V are less than a predetermined threshold, the second coordinate system transformation information acquisition unit 130 can determine that the vehicle V is traveling straight forward.

[0091] Once it is determined that the vehicle V is traveling straight forward, the second coordinate system transformation information acquisition unit 130 can extract a plurality of feature points from a plurality of surrounding images obtained at different time points. The second coordinate system transformation information acquisition unit 130 according to an embodiment can extract feature points in the surrounding image by using optical flow based on the Lucas - Kanade method, and the result is as Figure 9 shown. However, this is only an embodiment of the method of extracting feature points in the surrounding image, and the feature point extraction method is not limited to the above - mentioned embodiment.

[0092] Next, the second coordinate system transformation information acquisition unit 130 can obtain the motion vectors of the same feature points in a plurality of surrounding images. After obtaining the motion vectors, the second coordinate system transformation information acquisition unit 130 can find the intersection point of the motion vectors. When the vehicle V is traveling straight forward, the motion vectors meet at a point, which is called the extended focus. Assuming that the direction perpendicular to the traveling direction of the vehicle is called the reference side direction, when the imaging device C is installed to face the traveling direction with respect to the reference side direction, the intersection point of the motion vectors of the plurality of feature points extracted from the plurality of surrounding images can be formed in front of the vehicle. In this case, the second coordinate system transformation information acquisition unit 130 can determine that the intersection point is the extended focus. On the other hand, when the imaging device C is installed to face the opposite direction of the traveling direction with respect to the reference side direction of the vehicle V, the intersection point of the motion vectors of the plurality of feature points extracted from the plurality of surrounding images can be formed behind the vehicle. In this case, the second coordinate system transformation information acquisition unit 130 can determine that the 180 - degree rotation position of the Z - coordinate of the intersection point is the extended focus.

[0093] Meanwhile, when performing optical flow, there may be errors generated due to the actual road environment. Therefore, the second coordinate system transformation information acquisition unit 130 can find the extended focus through the following steps.

[0094] In the first step, the second coordinate system transformation information acquisition unit 130 can find the intersection point of the motion vectors of the feature point group including k feature points in the surrounding image, and obtain this intersection point as the candidate extended focus. In the second step, the second coordinate system transformation information acquisition unit 130 can find the number of motion vectors of the feature points in the surrounding image passing through the candidate extended focus obtained in the first step. In the third step, the second coordinate system transformation information acquisition unit 130 can repeatedly execute the first step and the second step. In the fourth step, the second coordinate system transformation information acquisition unit 130 can determine that the candidate extended focus through which the motion vectors of the maximum number of feature points pass is the extended focus.

[0095] On the contrary, in addition to the above method, the second coordinate system transformation information acquisition unit 130 according to another embodiment can also execute a fifth step: find another intersection point by using all the motion vectors of the feature points passing through the extended focus determined in the fourth step, and select this other intersection point as the final extended focus. Thereby, the accuracy of extended focus determination can be improved.

[0096] As a result of finding the Figure 9 extended focus through the above process, point P is determined as the extended focus.

[0097] After determining the extended focus based on the surrounding image, the second coordinate system transformation information acquisition unit 130 can obtain the driving direction of the vehicle V relative to the coordinate system of the imaging device C based on the extended focus. Here, the driving direction of the vehicle V can be represented by Figure 3 the X v axis in the coordinate system of the vehicle V. Specifically, the second coordinate system transformation information acquisition unit 130 can obtain the driving direction X (c,v) of the vehicle V relative to the coordinate system of the imaging device C according to the following mathematical formula 3:

[0098] [Mathematical formula 3]

[0099]

[0100] where the vector X (c,v) represents the X v axis in the coordinate system of the vehicle V relative to the coordinate system of the imaging device C, K represents a 3x3 matrix of the intrinsic parameters of the imaging device C, and m FOE represents the coordinates (u, v, l) of the extended focus. The vector X (c,v) obtained through the mathematical formula 3 is represented as a unit vector.

[0101] After obtaining the traveling direction of the vehicle V, the second coordinate system transformation information acquisition unit 130 acquires top view image transformation information by using the width and direction of the lane acquired from the first lane information. Here, the top view image transformation information may indicate a transformation matrix representing the attitude angle of the coordinate system of the imaging device C with respect to the earth's surface, and the transformation matrix is used to convert the surrounding image into a top view image, which can be defined by the following mathematical formula 4:

[0102] [Mathematical formula 4]

[0103] m topview = KR (r，c) K -1 Xm original

[0104] where m topview represents the pixel coordinates in the top view image, K represents a 3x3 matrix of the intrinsic parameters of the imaging device C, and m original represents the pixel coordinates in the surrounding image.

[0105] The second coordinate system transformation information acquisition unit 130 may transform the surrounding image into a top view image based on the initial value of the transformation matrix R (r,c) , and the lane in the top view image may be defined by the linear equation of the following mathematical formula 5:

[0106] [Mathematical formula 5]

[0107] xcosθ i + yysinθ i = r i

[0108] where (x, y) represents the pixel coordinates in the top view image with respect to the coordinate system of the earth's surface, and i represents an index for distinguishing multiple lanes of the lane.

[0109] Referring to Figure 10 , there is a white lane in the top view image, and the black straight line extending along the lane direction can be given by the linear equation defined by the mathematical formula 5.

[0110] After solving the linear equation for the lane, the second coordinate system transformation information acquisition unit 130 may acquire the top view image transformation information by using the lane width and lane direction acquired from the first lane information (S310). The first lane information in the three-dimensional information acquired by the lidar L includes the actual width information of the lane, and the second coordinate system transformation information acquisition unit 130 may obtain the transformation matrix R by using the parallel structure of the lane in the surrounding image according to the following mathematical formula 6 (r,c) :

[0111] [Mathematical formula 6]

[0112]

[0113] where i and j represent the indices of the lanes, and d i,j represents the width between the i-th lane and the j-th lane.

[0114] The second coordinate system transformation information acquisition unit 130 can iteratively obtain the transformation matrix R* that is the solution of the mathematical formula 6 by substituting the transformation matrix R obtained from the mathematical formula 6 (r,c) into the mathematical formula 4 and repeating the above process. (r,c) That is, the second coordinate system transformation information acquisition unit 130 can obtain the transformation matrix R* obtained when multiple lanes in the top view image are parallel to each other and the width between multiple lanes is very similar to the actual lane distance (r,c) as the solution of the mathematical formula 6.

[0115] After obtaining the top view image transformation information according to the above method, the second coordinate system transformation information acquisition unit can finally obtain the second coordinate system transformation information based on the inverse information of the top view image transformation information and the traveling direction of the vehicle V (S320). Specifically, the second coordinate system transformation information acquisition unit 130 can obtain the attitude angle R of the vehicle V with respect to the coordinate system of the imaging device C as the second coordinate system transformation information according to the following mathematical formula 7 (c,v) :

[0116] [Mathematical formula 7]

[0117]

[0118]

[0119]

[0120]

[0121] where R (c,r) represents the transformation matrix R (r,c) i.e., the inverse of the top view image transformation information, X (c,v) represents the X v axis in the coordinate system of the vehicle V with respect to the coordinate system of the imaging device C, as the traveling direction of the vehicle V, and the symbol 'x' represents the cross product of two vectors. Referring to Figure 7 , the vector Y(c, v) and the vector Z(c, v) are obtained by the cross product of the given vectors and can be represented as unit vectors.

[0122] According to the above process, once the first coordinate system transformation information and the second coordinate system transformation information are obtained, the coordinate system transformation information acquisition device 100 can obtain the third coordinate system transformation information between the vehicle V and the lidar L by using the first coordinate system transformation information and the second coordinate system transformation information. Thereby, the calibration among the vehicle V, the imaging device C, and the lidar L can be completed.

[0123] The above-described device and method for obtaining coordinate system transformation information allow obtaining the coordinate system transformation information of the imaging device and the lidar of a moving vehicle without equipment or manual operation, thereby reducing the cost and time required for calibration and obtaining accurate coordinate system transformation information.

[0124] Meanwhile, each step included in the above-described coordinate system transformation information acquisition method according to an embodiment can be implemented by a computer-readable recording medium for storing a computer program programmed to execute these steps.

[0125] The above description is only an exemplary description of the technical scope of the present disclosure, and those skilled in the art will understand that various changes and modifications can be made without departing from the original features of the present disclosure. Therefore, the embodiments disclosed in the present disclosure are intended to illustrate rather than limit the technical scope of the present disclosure, and the technical scope of the present disclosure is not limited by the embodiments. The protection scope of the present disclosure should be interpreted based on the appended claims, and it should be understood that all technical scopes within the scope equivalent thereto are included in the protection scope of the present disclosure.

[0126] According to an embodiment, since the above-described device and method for obtaining coordinate system transformation information are used in various fields such as home or industry, they are industrially applicable.

Claims

1. A method for obtaining coordinate system transformation information, the method comprises: obtaining three-dimensional information including first lane information corresponding to a lane adjacent to the vehicle by a lidar mounted on the vehicle, and obtaining a surrounding image including second lane information corresponding to the lane by a camera device mounted on the vehicle; extracting the first lane information from the three-dimensional information; fitting a plane based on the extracted first lane information; responsive to an error of fitting the plane being equal to or less than a predetermined reference error, determining that the vehicle is traveling on a horizontal ground, and extracting the second lane information from the surrounding image; and obtaining first coordinate system transformation information about the lidar and the camera device by matching the second lane information extracted from the surrounding image in response to the error of fitting the plane being equal to or less than the predetermined reference error with the first lane information.

2. The method according to claim 1, further comprises: obtaining second coordinate system transformation information about the vehicle and the camera device by using top view image transformation information obtained based on the surrounding image and the traveling direction of the vehicle.

3. The method according to claim 2, wherein obtaining the second coordinate system transformation information comprises: obtaining the traveling direction of the vehicle by using an extended focus obtained based on the surrounding image; obtaining the top view image transformation information by using the width and the direction of the lane obtained from the first lane information; and obtaining the second coordinate system transformation information based on the inverse information of the top view image transformation information and the traveling direction of the vehicle.

4. The method according to claim 3, wherein obtaining the traveling direction of the vehicle comprises: obtaining motion vectors of corresponding feature points in a plurality of surrounding images obtained at different times; determining the extended focus based on intersections of the obtained motion vectors; and obtaining the traveling direction of the vehicle by using the determined extended focus and the intrinsic parameters of the camera device.

5. The method according to claim 2, further comprises: determining whether the vehicle is traveling straight forward based on at least one of a steering angle and a yaw rate of the vehicle, wherein when it is determined that the vehicle is traveling straight forward, obtaining the second coordinate system transformation information is performed.

6. The method according to claim 4, wherein the determining the extended focus comprises: obtaining a plurality of candidate extended foci based on intersections of the obtained motion vectors of respective different feature point groups among a plurality of feature points; and determining, as the extended focus in the surrounding image, the following candidate extended focus among the plurality of candidate extended foci: the candidate extended focus having the largest number of motion vectors passing through the candidate extended focus.

7. The method according to claim 3, wherein the top view image transformation information includes a transformation matrix for transforming the surrounding image into a top view image, and Among them, the transformation matrix minimizes the difference between the width of the lane in the top view image and the width of the corresponding lane in the surrounding image, and the difference between the direction of the lane in the top view image and the direction of the corresponding lane in the surrounding image.

8. The method according to claim 2, further comprising obtaining third coordinate system transformation information about the vehicle and the lidar based on the first coordinate system transformation information and the second coordinate system transformation information.

9. An apparatus for obtaining coordinate system transformation information, the apparatus comprising: A lane information acquisition unit configured to extract the first lane information from three-dimensional information including the first lane information corresponding to a lane adjacent to the vehicle obtained by a lidar mounted on the vehicle; A horizontal ground recognition unit configured to recognize whether the vehicle is traveling on a horizontal ground based on the extracted first lane information, wherein a plane is fitted based on the extracted first lane information; and A first coordinate system transformation information acquisition unit configured to match second lane information about the lane with the first lane information to obtain first coordinate system transformation information about the lidar and the imaging device, wherein the second lane information is extracted from a surrounding image of the vehicle obtained by an imaging device mounted on the vehicle in response to an error of fitting the plane being equal to or less than a predetermined reference error, wherein, in response to an error of fitting the plane being equal to or less than the predetermined reference error, it is determined that the vehicle is traveling on a horizontal ground.

10. A computer-readable recording medium for storing instructions, the instructions, when executed by a processor, cause the processor to execute a method for obtaining coordinate system transformation information, the method comprising: Obtaining three-dimensional information including first lane information corresponding to a lane adjacent to the vehicle by a lidar mounted on the vehicle, and obtaining a surrounding image including second lane information corresponding to the lane by an imaging device mounted on the vehicle; Extracting the first lane information from the three-dimensional information; Fitting a plane based on the extracted first lane information; In response to an error of fitting the plane being equal to or less than a predetermined reference error, determining that the vehicle is traveling on a horizontal ground and extracting the second lane information from the surrounding image; and Obtaining first coordinate system transformation information about the lidar and the imaging device by matching the second lane information extracted from the surrounding image in response to an error of fitting the plane being equal to or less than the predetermined reference error and the first lane information.

Images