Apparatus and method for providing on-line camera calibration by using map matching

Abstract

A calibration method for a vehicle camera according to one embodiment includes the steps of: matching a landmark recognized from a captured image of a camera during vehicle operation with a landmark existing in an electronic map to obtain camera position and attitude angle information based on an absolute coordinate system; and obtaining camera rotation information based on a vehicle coordinate system based on changes in camera position and attitude angle based on an absolute coordinate system obtained as the vehicle moves.

Description

Technology Field

[0001] The present invention relates to a camera calibration device and method for performing calibration of a vehicle camera online using map matching. Background Technology

[0002] Generally, a vehicle refers to a transportation device that travels on roads or tracks using fossil fuels, electricity, or other sources as a power source.

[0003] Vehicles have evolved to provide drivers with a variety of functions in line with technological advancements. In particular, driven by the trend toward vehicle electrification, vehicles equipped with Active Safety Systems (ASS) that operate to prevent accidents immediately before or at the moment of impact have emerged.

[0004] Furthermore, in recent years, active research has been conducted on vehicles equipped with Advanced Driver Assistance Systems (ADAS) that actively provide information regarding the driving environment, such as vehicle status, driver status, and surrounding conditions, in order to reduce the burden on drivers and enhance convenience.

[0005] An advanced driver assistance system may be equipped with a sensing means to detect the driving environment, and may include a camera as an example of the sensing means. The camera may be installed outside or inside the vehicle and may detect the driving environment by acquiring an image corresponding to the position and attitude angle at which it is installed. Prior art literature

[0006] Korean Registered Patent Publication, No. 10-1584693 (Published Jan. 14, 2016) The problem to be solved

[0007] As mentioned above, since the camera acquires images corresponding to its installation position and attitude angle, it is necessary to recognize the camera's coordinate system to control the vehicle based on the acquired images. To this end, the camera's mounting position and attitude angle can be obtained from the vehicle by using a calibration recognition pattern or by recognizing the vanishing point location based on lanes in the driving environment. However, calibration pattern recognition requires an additional pre-calibration process and cannot reflect changes in the camera's position during driving, while using the vanishing point location results in reduced accuracy.

[0008] Accordingly, the problem to be solved by the present invention is to enable highly precise online camera calibration by performing the calibration of a vehicle camera online using map matching.

[0009] Furthermore, online camera calibration can be performed using only accurate information confirmed through the verification of the validity and / or reliability of the information obtained during the calibration process.

[0010] However, the problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below. means of solving the problem

[0011] A camera calibration method for a vehicle camera according to the first aspect includes the step of obtaining camera position and attitude angle information based on an absolute coordinate system by matching a landmark recognized from a captured image of the camera with a landmark existing in an electronic map, and the step of obtaining camera rotation information based on a vehicle coordinate system based on the change in camera position and attitude angle based on the absolute coordinate system obtained as the vehicle moves.

[0012] A camera calibration device according to a second aspect includes an acquisition unit that acquires a captured image of a camera and a control unit that performs calibration of the camera based on the captured image and an electronic map, wherein the control unit acquires camera position and attitude angle information based on an absolute coordinate system by matching a landmark recognized from the captured image with a landmark existing in the electronic map, and acquires camera rotation information based on a vehicle coordinate system based on changes in camera position and attitude angle based on the absolute coordinate system acquired according to the movement of the vehicle.

[0013] A computer-readable recording medium storing a computer program according to a third aspect includes instructions for a processor to perform the camera calibration method.

[0014] A computer program stored on a computer-readable recording medium according to the fourth aspect includes instructions for a processor to perform the camera calibration method. Effects of the invention

[0015] A camera calibration device and method according to one embodiment can perform online camera calibration with great precision by performing calibration of a vehicle camera online using map matching.

[0016] Furthermore, since online camera calibration can be performed using only accurate information confirmed through the validation and / or reliability verification of information acquired during calibration, there is an effect of improving the accuracy of camera calibration. Brief explanation of the drawing

[0017] FIG. 1 is a configuration diagram of an online camera calibration device according to one embodiment. FIG. 2 is a diagram illustrating the coordinate system of a vehicle and a camera according to one embodiment. FIGS. 3 to 6 are flowcharts for explaining an online camera calibration method according to one embodiment. FIG. 7 is a diagram illustrating an example of determining the straightness of a vehicle according to an online camera calibration method according to one embodiment. FIG. 8 is a diagram showing an example in which a rotation matrix is ​​expressed as an axis-angle to be used for reliability verification according to an online camera calibration method according to one embodiment. FIGS. 9 and FIGS. 10 are drawings for explaining landmark matching results according to an online camera calibration method according to one embodiment. Specific details for implementing the invention

[0018] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0019] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions in the embodiments of the present invention, and these definitions may vary depending on the intentions or conventions of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0020] In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as 'comprising' or 'composing' are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0021] Furthermore, in the embodiments of the present invention, when it is stated that a part is connected to another part, this includes not only a direct connection but also an indirect connection through another medium. Additionally, the meaning that a part includes a certain component implies that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0022] FIG. 1 is a configuration diagram of an online camera calibration device according to one embodiment, and FIG. 2 is a diagram for explaining the coordinate system of a vehicle and a camera according to one embodiment.

[0023] A vehicle camera calibration system (1) according to one embodiment may include a vehicle (V) and an online camera calibration device (100).

[0024] A vehicle (V) is a type of means of transportation capable of moving humans, objects, or animals from one location to another while traveling along a road or track. A vehicle (V) according to one embodiment may include a three-wheeled or four-wheeled automobile, a two-wheeled automobile such as a motorcycle, construction machinery, a motorized bicycle, a bicycle, and a train traveling on a track.

[0025] In the embodiment shown in FIG. 1, the vehicle (V) and the online camera calibration device (100) are implemented independently and connected to each other through communication, but the online camera calibration device (100) may be implemented integrally with the vehicle (V) or mounted on the vehicle (V).

[0026] The vehicle (V) of FIG. 1 may include a Global Positioning System (GPS) module and may receive satellite signals containing navigation data from at least one GPS satellite. Based on the satellite signals, the vehicle (V) may obtain the current location of the vehicle (V) and the direction of travel of the vehicle (V), etc.

[0027] Additionally, the vehicle (V) or online calibration device (100) of FIG. 1 may store a precise electronic map in advance. Here, the electronic map may refer to an electronic map that has high accuracy for safe and precise vehicle (V) control and includes information regarding the planar position of the road as well as elevation, slope, curvature, etc. Additionally, the electronic map may further include information regarding landmarks on the road, such as lanes, signs, traffic lights, and guardrails.

[0028] In addition, the vehicle (V) of FIG. 1 may be equipped with an Advanced Driver Assistance System (ADAS). Here, an Advanced Driver Assistance System may refer to a system that provides driving environment information, such as the vehicle (V) state, driver state, and surrounding environment information, or actively controls the vehicle (V) in order to reduce the burden on the driver and enhance convenience.

[0029] An advanced driver assistance system mounted on a vehicle (V) may include a sensing means for detecting the driving environment of the vehicle (V). A sensing means according to one embodiment may include a radar that detects the driving environment by irradiating a pulse around the vehicle (V) and receiving an echo pulse reflected from an object located in that direction, a LiDAR that irradiates a laser around the vehicle (V) and receives a laser reflected from an object located in that direction, and / or an ultrasonic sensor that irradiates an ultrasonic wave around the vehicle (V) and receives an echo ultrasonic wave reflected from an object located in that direction.

[0030] Additionally, the advanced driver assistance system may include a camera (C) as a detection means. The camera (C) is positioned to face the front, side, and / or rear of the vehicle (V) to acquire images in those directions. The acquired images can serve as a basis for acquiring information such as lanes or signs, as well as objects around the vehicle (V), through an image processing process.

[0031] Meanwhile, the vehicle (V) can use the fusion of image information acquired by the camera (C) and CAN DATA, such as wheel rotation information and yaw rate information transmitted via the CAN (Controller Area Network) communication method, which is a communication method between internal modules of the vehicle (V), for vehicle (V) control. At this time, the image acquired by the camera (C) follows the camera coordinate system, whereas the CAN DATA may follow the vehicle coordinate system.

[0032] FIG. 2 is a schematic plan view of a vehicle (V) according to one embodiment, illustrating the coordinate system of the vehicle (V) and the coordinate system of a camera (C) installed on the vehicle (V). Referring to FIG. 2, the vehicle (V) is O V With as the origin, X in the direction of travel of the vehicle (V) V Axis, Z in the direction perpendicular to the ground V Axis, and X VAxis and Z V Y perpendicular to the axis V It can have a vehicle coordinate system composed of axes. On the other hand, the camera (C) installed on the vehicle (V) is O C X, determined by the installation position and attitude angle, with as the origin C Axis, Y C Axis, Z C It can have a camera coordinate system composed of axes. In order to fuse two pieces of information on such different coordinate systems, the unification of the coordinate systems is required, and this is called camera (C) calibration.

[0033] To this end, camera (C) calibration can be performed in advance prior to the full-scale autonomous driving of the vehicle (V). Specifically, an image of a recognition pattern for correction can be acquired using a camera (C) installed on the vehicle (V), and the angle and position of the mounted camera (C) can be manually acquired using this. In this case, there is the inconvenience of having to perform camera (C) calibration as a preliminary step, and it is difficult to reflect changes in the position and attitude angle of the camera (C) that may occur during driving.

[0034] Alternatively, the attitude angle of the camera (C) can be obtained by recognizing the lane through the camera (C) while the vehicle (V) is driving and confirming the location of the vanishing point based on the recognized lane. However, in situations where the vanishing point of the lane is not accurately extracted, such as on a curved road, it is difficult to apply this method, and the accuracy is lower compared to a method performed manually.

[0035] To solve this, a vehicle camera calibration system (1) according to one embodiment of the disclosed invention can provide mutual compatibility by converting the coordinate systems between the vehicle (V) and the camera (C) using a satellite signal received by the vehicle (V), an image acquired by the camera (C), and an electronic map.

[0036] Referring to FIG. 1, an online camera calibration device (100) according to one embodiment acquires satellite navigation signals, images captured by a camera, etc. from a vehicle (V), and acquires camera position and attitude angle information based on an absolute coordinate system by matching landmarks recognized from images captured by a camera (C) according to the operation of the vehicle (V) with landmarks existing on an electronic map, and acquires camera rotation information based on a vehicle coordinate system based on changes in camera position and attitude angle based on an absolute coordinate system acquired according to the movement of the vehicle (V). For example, a position matrix can be calculated based on changes in camera position and attitude angle based on an absolute coordinate system acquired according to the straight movement of the vehicle (V), and a camera rotation matrix based on a vehicle coordinate system can be acquired as camera rotation information using the calculated position matrix. Such an online camera calibration device (100) may include an acquisition unit (110) and a control unit (120) as shown in FIG. 1.

[0037] The acquisition unit (110) of the online camera calibration device (100) can acquire images captured by the camera (C) during the operation of the vehicle (V). Here, the acquisition unit (110) may include a communication modem that communicates with the vehicle (V) or a communication interface that can input and output various information with the vehicle (V). For example, the acquisition unit (110) can exchange information by communicating with the vehicle (V) using various known communication methods. For example, the communication modem may adopt known communication methods such as CDMA, GSM, W-CDMA, TD-SCDMA, WiBro, LTE, EPC, etc., and communicate with the vehicle (V) via a base station. Alternatively, the communication modem may communicate with the vehicle (V) within a predetermined distance by adopting communication methods such as Wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), UWB (Ultra wideband), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), and NFC (Near Field Communication).

[0038] The control unit (120) receives images, etc., acquired by the camera (C) from the vehicle (V) by the acquisition unit (110), and can use this to acquire position and attitude angle information of the camera (C) and provide it to the vehicle (V). The installation position of the camera (C) can be determined by inputting actual measured values ​​from the outside or by internal calculations of the control unit (120). Accordingly, the following describes in detail how the control unit (120) acquires attitude angle information of the camera (C) and uses this to provide camera rotation information based on the vehicle coordinate system, such as a camera rotation matrix, to the vehicle (V) as coordinate system transformation information.

[0039] The control unit (120) can obtain camera position and attitude angle information based on an absolute coordinate system by matching a landmark recognized from a captured image of a camera (C) according to the operation of the vehicle (V) with a landmark existing in an electronic map, and can calculate a position matrix based on the change in camera position and attitude angle based on an absolute coordinate system obtained according to the straight movement of the vehicle (V), and can obtain a camera rotation matrix based on a vehicle coordinate system using the calculated position matrix.

[0040] When the control unit (120) obtains camera position and attitude angle information based on an absolute coordinate system, it can estimate the vehicle position and direction of travel based on an absolute coordinate system using satellite navigation signals or vehicle wheel speeds, recognize landmarks existing on an electronic map from captured images, and determine the camera position and attitude angle information based on an absolute coordinate system by matching the recognized landmarks on the electronic map using the information on the estimated vehicle position and direction of travel based on the absolute coordinate system.

[0041] When the control unit (120) obtains a camera rotation matrix based on the vehicle coordinate system, it can calculate a vehicle first axis direction vector based on the camera coordinate system from the position matrix, and can calculate a vehicle second axis direction vector based on the camera coordinate system using the absolute coordinate system camera position and attitude angle information used in calculating the vehicle first axis direction vector based on the camera coordinate system and landmarks recognized in the electronic map, and based on the calculated vehicle first axis direction vector based on the camera coordinate system and the calculated vehicle second axis direction vector based on the camera coordinate system, it can calculate a camera rotation matrix based on the vehicle coordinate system that reflects a vehicle third axis direction vector based on the camera coordinate system.

[0042] When calculating the vehicle first axis direction vector based on the camera coordinate system, the control unit (120) can verify the validity of the acquired camera position and attitude angle information based on the absolute coordinate system based on the matching result of the landmark recognized from the captured image and the landmark existing in the electronic map, and calculate the vehicle first axis direction vector based on the camera coordinate system based on the verified camera position change and attitude angle change based on the absolute coordinate system, and then calculate the vehicle first axis direction vector based on the camera coordinate system from the position matrix.

[0043] When verifying validity, the control unit (120) may determine validity based on the result of comparing the matching error or matching ratio of a landmark recognized in a captured image with a landmark existing in an electronic map with a preset threshold. The landmark may include continuous landmarks and discrete landmarks. In this case, when verifying validity, the control unit (120) may determine validity based on the result of comparing the matching error or matching ratio of a continuous landmark with a preset first threshold and the result of comparing the matching error or matching ratio of a discrete landmark with a preset second threshold. Additionally, the control unit (120) may determine validity by further considering the result of comparing the number of continuous landmarks recognized or the number of discrete landmarks recognized with a preset threshold.

[0044] When calculating the vehicle first axis direction vector based on the camera coordinate system, the control unit (120) can verify the reliability of the camera position and attitude angle information based on the absolute coordinate system based on the obtained camera position and attitude angle information based on the absolute coordinate system based on the error in the distance traveled corresponding to the movement component due to the change in the camera position based on the absolute coordinate system and the distance traveled due to the vehicle wheel speed, and calculate the vehicle first axis direction vector based on the camera coordinate system based on the verified reliability of the camera position change based on the absolute coordinate system and then calculate the vehicle first axis direction vector based on the camera coordinate system based on this position matrix. When verifying reliability, the control unit (120) can determine that reliability exists if there is no rotation component due to the change in the camera attitude angle based on the absolute coordinate system.

[0045] When the control unit (120) calculates the vehicle's first axis direction vector based on the camera coordinate system, it can determine the vehicle's straight movement based on the vehicle's left wheel travel distance, right wheel travel distance, and left and right wheel distances.

[0046] Meanwhile, when the vehicle's first axis is the vehicle's longitudinal axis and the vehicle's second axis is the vehicle's vertical axis, the control unit (120) can calculate a direction vector perpendicular to the ground using camera position and attitude angle information based on the absolute coordinate system and landmarks recognized in the electronic map when calculating the vehicle's second axis direction vector based on the camera coordinate system, and can calculate a vehicle's vertical axis direction vector based on the camera coordinate system using the direction vector perpendicular to the ground.

[0047] The control unit (120) can calculate the vehicle second axis direction vector based on the camera coordinate system, and then recalculate the vehicle first axis direction vector based on the camera coordinate system based on the result of comparing the standard deviation of the vehicle second axis direction vector based on the camera coordinate system with a preset threshold, and can use the recalculated vehicle first axis direction vector based on the camera coordinate system when calculating the camera rotation matrix based on the vehicle coordinate system.

[0048] When the control unit (120) calculates a camera rotation matrix based on the vehicle coordinate system, it can calculate a vehicle rotation matrix based on the camera coordinate system that reflects the vehicle third axis direction vector based on the camera coordinate system based on the calculated vehicle first axis direction vector and the calculated vehicle second axis direction vector based on the camera coordinate system, and can calculate the inverse matrix of the calculated vehicle rotation matrix based on the camera coordinate system as the camera rotation matrix based on the vehicle coordinate system.

[0049] FIGS. 3 to 6 are flowcharts for explaining an online camera calibration method according to one embodiment, FIG. 7 is a diagram for explaining an example of determining the straightness of a vehicle according to an online camera calibration method according to one embodiment, FIG. 8 is a diagram showing an example of expressing a rotation matrix as an axis-angle for use in reliability verification according to an online camera calibration method according to one embodiment, and FIGS. 9 and 10 are diagrams for explaining landmark matching results according to an online camera calibration method according to one embodiment.

[0050] Hereinafter, with reference to FIGS. 1 to 10, we will examine in detail the online camera calibration process of a vehicle camera performed in a vehicle camera calibration system (1) according to one embodiment.

[0051] Referring to FIG. 3, the acquisition unit (110) acquires a captured image of a camera (C) as the vehicle (V) moves and provides it to the control unit (120), and the control unit (120) matches a landmark recognized from the captured image with a landmark existing in a previously stored electronic map to acquire camera position and attitude angle information based on an absolute coordinate system (S310).

[0052] Referring to Figure 4, step S310 can be examined. When the control unit (120) obtains camera position and attitude angle information based on the absolute coordinate system, it can estimate the vehicle position and direction of travel based on the absolute coordinate system using a GPS signal or vehicle wheel speed (S410).

[0053] In step S410, the control unit (120) can estimate vehicle position and direction of travel information based on an absolute coordinate system using a known position measurement method using GPS signals. When at least two sets of position information based on GPS signals are received consecutively, the direction of travel of the vehicle based on an absolute coordinate system can be easily obtained.

[0054] In sections where GPS signals are not received, the control unit (120) can estimate the position and direction of travel information of the vehicle based on the absolute coordinate system using the vehicle wheel speed.

[0055] Referring to FIG. 7, the travel distance S is based on the wheel speed of the vehicle's left wheel. r This is calculated, and the travel distance S is based on the wheel speed of the vehicle's right wheel. l Once this is calculated, the vehicle position (x', y') can be estimated using mathematical formulas 1 to 3.

[0056]

[0057]

[0058]

[0059]

[0060] And, vehicle position and direction of travel at time k [x k , y k , θ k Given ], the vehicle position and direction of travel at time k+1 estimated by wheel speed [x k +1 , y k +1 , θ k +1] is equal to mathematical formula 5.

[0061]

[0062] As described above, the control unit (120) can estimate the vehicle position and direction of travel based on the absolute coordinate system using a GPS signal or vehicle wheel speed, and updates information regarding the vehicle position and direction of travel based on the absolute coordinate system whenever a GPS signal or vehicle wheel speed is received.

[0063] And, the control unit (120) can recognize landmarks existing in the electronic map from the captured image (S420). For example, the control unit (120) can be trained in advance through deep learning, etc., to recognize lanes, stop lines, traffic lights, signs, etc., existing as landmarks in the electronic map, and can recognize landmarks included in the captured image through a deep learning algorithm. Landmarks can be classified into continuous landmarks and discrete landmarks, continuous landmarks may include lanes, stop lines, etc., and discrete landmarks may include traffic lights, signs, etc. (S420).

[0064] In addition, the control unit (120) can determine the camera position and attitude angle information based on the absolute coordinate system by matching the recognized landmarks on the electronic map using information about the vehicle position and direction of travel based on the estimated absolute coordinate system (S430).

[0065] Referring again to FIG. 3, the control unit (120) calculates a position matrix based on the change in camera position and the change in attitude angle based on the absolute coordinate system obtained as the vehicle (V) moves straight (S320).

[0066] In step S320, the control unit (120) can determine whether the vehicle is moving straight based on the distance traveled by the left wheel and the distance traveled by the right wheel, and the distances between the left and right wheels of the vehicle. For example, the control unit (120) can determine that the vehicle is moving straight when the absolute value of θ, which can be obtained through Equation 2, is below a threshold value.

[0067] Additionally, the control unit (120) can obtain a position matrix T representing the camera position and attitude angle based on the absolute coordinate system in the electronic map by using an ICP (Iterative Closest Point) algorithm that obtains the solution to Equation 6. However, the solution obtained by the ICP algorithm may be inaccurate due to incorrect matching.

[0068]

[0069] Here, Z cl(i) represents the i-th image coordinate among multiple points of continuous landmarks recognized in the image, and P cl(i) represents 3D coordinates matched with an electronic map, and Cz cl(i) and Cp cl(i) are respectively Z cl(i) and P cl(i) It is the covariance matrix of Z. dl(j) represents the j-th image coordinate among the discrete landmarks recognized in the image, and P dl(j) represents 3D coordinates matched with an electronic map, and Cz dl(j) and Cp dl(j) are respectively Z dl(j) and P dl(j) It is the covariance matrix of.

[0070] As such, in order to compensate for the inaccuracy when calculating the position matrix representing the camera position and attitude angle based on the absolute coordinate system, it is necessary to determine whether the camera position and attitude angle information based on the absolute coordinate system identified in step S430 is valid, and to perform online camera calibration using only valid information.

[0071] Referring to step S320 with reference to FIG. 5, when the control unit (120) calculates the position matrix, it can verify the validity of the camera position and attitude angle information based on the acquired absolute coordinate system based on the matching result between the landmark recognized from the captured image and the landmark existing in the electronic map (S510).

[0072] For example, when verifying validity in step S510, the control unit (120) may determine validity based on the result of comparing the matching error or matching ratio of a landmark recognized in the captured image and a landmark existing in the electronic map with a preset threshold. For example, if the matching error is less than or equal to the preset threshold and the matching ratio is greater than or equal to the preset threshold, the camera position and attitude angle information based on the corresponding absolute coordinate system may be determined to be valid.

[0073] Here, the matching error may refer to the average value of the positional error in the image coordinate system between the landmarks recognized in the image and the corresponding landmarks in the electronic map, which are finally validly matched by the ICP algorithm. The matching ratio may refer to the result of dividing the number of landmarks for which the matching error is below a threshold by the number of landmarks recognized in the image. Since continuous landmarks, such as a single lane or a stop line, consist of multiple points, the matching error and matching ratio can be calculated using these multiple points. For example, validation conditions can be set such as when the matching error of a discrete landmark is below threshold 1, when the matching error of a continuous landmark is below threshold 2, when the matching ratio of a discrete landmark is above threshold 3, or when the matching ratio of a continuous landmark is above threshold 4.

[0074] Additionally, when verifying validity in step S510, the control unit (120) may determine validity by further considering the result of comparing the number of continuous landmarks recognized or the number of discrete landmarks recognized with a preset threshold number. For example, it may be determined that validity exists when there is at least one discrete landmark and the sum of continuous landmarks, such as lanes or stop lines recognized in the image, is at least two.

[0075] In addition, the control unit (120) can further verify the reliability of the camera position and attitude angle information based on the absolute coordinate system, the validity of which has been verified through step S510. Here, the control unit (120) can verify the reliability of the camera position and attitude angle information based on the absolute coordinate system based on the error in the distance traveled corresponding to the movement component caused by the change in the camera position based on the absolute coordinate system and the distance traveled due to the vehicle wheel speed (S520).

[0076] For example, in step S520, the control unit (120) has a matrix T representing the k-th camera position and attitude angle. k and matrix T representing the second camera position and attitude angle k +1 Given, T k and T k +1 Matrix T regarding camera movement between k,k +1 = T k - 1 T k + 1 and T k,k +1 The rotation of can be expressed by the axis-angle in Fig. 8, and in the situation where the vehicle is moving straight, T k,k +1 Rotation value (θ) expressed as an axis-angle in the matrix k,k + 1) This must be less than or equal to the critical value. T k,k +1Calculated using the positional components [tx, ty, tz] must be equal to the vehicle travel distance S calculated by the wheel speed (see FIG. 7). Depending on the embodiment, |Sd accounting for errors k,k +1 When the value is below the threshold, it can be determined in step S520 that it has reliability.

[0077] Next, the control unit (120) can calculate a position matrix based on camera position changes in an absolute coordinate system whose validity and reliability have been verified through steps S510 and S520 (S530). For example, the control unit (120) can calculate camera position and attitude angle data (T1, …, T) that satisfy the above conditions in a straight driving situation. n T from ) and 2 adjacent camera positions k,k+1 You can find the matrix.

[0078] FIGS. 9 and FIGS. 10 are drawings illustrating landmark matching results according to an online camera calibration method according to one embodiment. As shown in FIGS. 9 and FIGS. 10, although the driving sections are adjacent, there may be valid cases and invalid cases. This is because an ICP-based map matching algorithm may converge to an incorrect solution depending on errors in the initial camera position and attitude angle, and the results may become inaccurate due to incorrect matching between landmarks recognized in the image and 3D landmarks in the electronic map. According to the embodiment, the accuracy of the online camera calibration can be improved by verifying the accuracy of the map matching results. Since an angle error of about 1 degree in the rotation component of the camera external parameters significantly affects ADAS or autonomous driving performance, a process of verifying the accuracy of the map matching results is necessary. FIG. 9 is an example of a map matching result determined to be invalid, and FIG. 10 is an example of a map matching result determined to be valid. When a lane (white) recognized from a captured image is matched with a lane (sky blue) existing on an electronic map, there may be a discrepancy as in the example of Fig. 9, or a matched result as in the example of Fig. 10.

[0079] Referring again to FIG. 3, the control unit (120) can obtain a camera rotation matrix based on the vehicle coordinate system using the position matrix captured in step S320 (S330).

[0080] Referring to Fig. 6, step S330 can be examined. When the control unit (120) obtains a camera rotation matrix based on the vehicle coordinate system, it can calculate a vehicle first axis direction vector based on the camera coordinate system from the position matrix (S610).

[0081] Referring to FIG. 2, camera position and attitude angle data (T1, …, T) based on an absolute coordinate system validly obtained when the vehicle is driving straight n T obtained from 2 adjacent camera positions among ) k,k+1 In the matrix, the unit vector of [tx, ty, tz] is the vector X representing the direction of the vehicle's longitudinal axis (Xv-axis) relative to the camera coordinate system. c,v It means direction vector X c,v Since multiple can be obtained in a straight-line driving situation, there are m direction vectors X c,v When data exists, the average vector as in Equation 7 You can improve accuracy by calculating .

[0082]

[0083] And, the control unit (120) can calculate a direction vector perpendicular to the ground using camera position and attitude angle information based on the absolute coordinate system and landmarks recognized in the electronic map (S620).

[0084] Referring to Fig. 2, a direction vector perpendicular to the ground is calculated using lane data (3D points) around the camera position relative to the absolute coordinate system. Covariance is calculated using the 3D points on the lane corresponding to the ground, and new mutually orthogonal bases (axes) are found through Eigen Decomposition while preserving the variance of the 3D data as much as possible. Subsequently, Principal Component Analysis (PCA) can be used to identify the principal components of the distributed data. Through Equation 8, eigenvectors are obtained from the 3x3 covariance matrix through Eigen Decomposition. We can find , and the vector Z represents the direction perpendicular to the ground where e3 is . w,r It is a direction vector perpendicular to the ground in the absolute coordinate system.

[0085]

[0086] Here, e1, e2, and e3 represent mutually orthogonal eigenvectors (3x1 vectors). Cov3 *3 represents the covariance matrix for 3-dimensional data, where ∑ is a 3x3 diagonal matrix and the diagonal elements represent eigenvalues. Additionally, e1 represents the direction with the largest variance, e2 represents the direction perpendicular to e1 with the next largest variance, and e3 represents the direction perpendicular to e1 and e2 with the next largest variance.

[0087] Referring again to FIG. 6, the control unit (120) can calculate the vehicle second axis direction vector based on the camera coordinate system using the camera position and attitude angle information based on the absolute coordinate system used to calculate the vehicle first axis direction vector based on the camera coordinate system and the landmark recognized in the electronic map (S630).

[0088] The direction vector Z perpendicular to the ground, calculated using Equation 8 under the assumption that “the ground around the vehicle is currently flat” while the vehicle is traveling straight. w,r It can be determined that it is parallel to the vehicle's vertical direction vector Zv. Under these conditions, in a situation where multiple vehicles are driving straight, the vertical direction vector Z w,r We can find the direction vector Z perpendicular to the ground relative to the camera coordinate system. c,r Formula to express as Uses. (Here, R k T represents the k-th camera position and attitude angle. k It refers to the rotation matrix of a matrix.) Multiple Z c,r When data exists, the average vector as in Equation 9 You can improve accuracy by calculating .

[0089]

[0090] Additionally, m direction vectors Z c,rSince there is a high probability that the data is incorrect if the standard deviation exceeds a threshold, the data is judged as unreliable, and the average vector is repeatedly calculated until data with a standard deviation less than the threshold is obtained. can calculate.

[0091] Referring again to FIG. 6, the control unit (120) can recalculate the vehicle first axis direction vector based on the camera coordinate system based on the result of comparing the standard deviation of the vehicle second axis direction vector based on the camera coordinate system with a preset threshold (S640). m direction vectors X c,v Since there is a high probability that the data is incorrect if the standard deviation exceeds a threshold, the data is judged as unreliable, and the average vector is repeatedly calculated until data with a standard deviation less than the threshold is obtained. can calculate.

[0092] Next, the control unit (120) can calculate a vehicle rotation matrix based on the camera coordinate system that reflects the vehicle third axis direction vector based on the camera coordinate system, based on the vehicle first axis direction vector based on the camera coordinate system and the calculated vehicle second axis direction vector based on the camera coordinate system (S650). Vector and vector Given this, the rotation matrix representing the vehicle's longitudinal axis (X-axis), transverse axis (Y-axis), and vertical axis (Z-axis) relative to the camera coordinate system can be obtained through Equation 10.

[0093]

[0094] Here, “X” represents the cross product of vectors, and vector X c,v , vector Z c,r , vector Y c,v , vector Z c,v is represented as a unit vector.

[0095] In addition, the control unit (120) can calculate the inverse matrix of the calculated vehicle rotation matrix based on the camera coordinate system as the camera rotation matrix based on the vehicle coordinate system (S660). R obtained through Equation 10 c,v The inverse of is the camera rotation matrix R relative to the vehicle coordinate system that we ultimately want to find for online camera calibration. v,c It means.

[0096] Meanwhile, each step included in the online camera calibration method according to the above-described embodiment may be implemented in a computer-readable recording medium that records a computer program programmed to perform such steps.

[0097] Additionally, each step included in the online camera calibration method according to the above-described embodiment may be implemented in a computer-readable recording medium that records a computer program programmed to perform such steps.

[0098] As explained above, according to the embodiments of the present invention, by performing calibration of a vehicle camera online using map matching, online camera calibration can be performed with great precision.

[0099] Furthermore, the accuracy of camera calibration is improved because online camera calibration can be performed using only accurate information confirmed through the validation and / or reliability verification of the information acquired during calibration.

[0100] Combinations of each step of each flowchart attached to the present invention may be performed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a computer for special purposes, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in each step of the flowchart. Since these computer program instructions may also be stored in a computer-available or computer-readable recording medium that can be oriented toward the computer or other programmable data processing equipment to implement the function in a specific manner, the instructions stored in the computer-available or computer-readable recording medium may also produce a manufactured item containing instruction means for performing the function described in each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each step of the flowchart.

[0101] Additionally, each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the steps may occur out of order. For example, two steps described in succession may actually be performed substantially simultaneously, or the steps may sometimes be performed in reverse order according to the corresponding function.

[0102] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Industrial applicability

[0103] According to one embodiment, online camera calibration can be performed with high precision by performing the calibration of a vehicle camera online using map matching. This online camera calibration technology can be utilized in various technical fields where online calibration of a vehicle camera that acquires images while the vehicle is in operation is required. Explanation of the symbols

[0104] 1: Automotive Camera Calibration System V: Vehicle 100: Online Camera Calibration Device 110: Acquisition Department 120: Control unit

Claims

Claim 1 A calibration method for a vehicle camera comprises: a step of obtaining camera position and attitude angle information based on an absolute coordinate system by matching a landmark recognized from an image captured by the vehicle camera with a landmark existing in an electronic map; a step of calculating a position matrix based on a change in camera position and attitude angle based on the absolute coordinate system obtained according to the movement of the vehicle; and a step of obtaining a camera rotation matrix based on a vehicle coordinate system that reflects a vehicle first axis direction vector, a vehicle second axis direction vector, and a vehicle third axis direction vector calculated using the calculated position matrix. The step of obtaining the camera rotation matrix based on the vehicle coordinate system comprises: a step of calculating the vehicle first axis direction vector based on the camera coordinate system from the position matrix; a step of calculating the vehicle second axis direction vector based on the camera coordinate system using the camera position and attitude angle information based on the absolute coordinate system used in calculating the vehicle first axis direction vector based on the camera coordinate system and the landmark recognized in the electronic map; and the calculated vehicle first axis direction vector based on the camera coordinate system and the calculated vehicle second axis based on the camera coordinate system A method for calibrating a vehicle camera, comprising the step of calculating a camera rotation matrix based on the vehicle coordinate system, which reflects the vehicle third axis direction vector based on the camera coordinate system, based on the direction vector. Claim 2 A method for calibrating a vehicle camera according to claim 1, wherein the step of acquiring camera position and attitude angle information based on the absolute coordinate system comprises: a step of estimating the vehicle position and direction of travel based on the absolute coordinate system using satellite navigation signals or vehicle wheel speed; a step of recognizing a landmark existing in the electronic map from the captured image; and a step of determining the camera position and attitude angle information based on the absolute coordinate system by matching the recognized landmark to the electronic map using the information regarding the estimated vehicle position and direction of travel based on the absolute coordinate system. Claim 3 A method for calibrating a vehicle camera according to claim 1, wherein, in the step of calculating the position matrix, the change in camera position and the change in attitude angle based on the absolute coordinate system are obtained as the vehicle moves in a straight line. Claim 4 delete Claim 5 A method for calibrating a vehicle camera according to claim 1, wherein the step of calculating a vehicle first axis direction vector based on the camera coordinate system comprises: a step of verifying validity based on the matching result of a landmark recognized from the captured image and a landmark existing in the electronic map with respect to the acquired camera position and attitude angle information based on the absolute coordinate system; and a step of calculating a position matrix based on the verified camera position change and attitude angle change based on the absolute coordinate system, and then calculating a vehicle first axis direction vector based on the camera coordinate system from the position matrix. Claim 6 A calibration method for a vehicle camera according to claim 5, wherein the step of verifying the validity determines the validity based on the result of comparing the matching error or matching ratio of a landmark recognized in the captured image and a landmark existing in the electronic map with a preset threshold. Claim 7 A calibration method for a vehicle camera according to claim 5, wherein the landmark includes a continuous landmark and a discrete landmark, and the step of verifying the validity determines the validity based on the result of comparing the matching error or matching ratio of the continuous landmark with a preset first threshold and the result of comparing the matching error or matching ratio of the discrete landmark with a preset second threshold. Claim 8 A calibration method for a vehicle camera according to claim 7, wherein the validity is determined by further considering the result of comparing the number of continuous landmarks recognized or the number of discrete landmarks recognized with a preset threshold number. Claim 9 A method for calibrating a vehicle camera according to claim 1, wherein the step of calculating a vehicle first axis direction vector based on the camera coordinate system comprises: a step of verifying the reliability of the camera position and attitude angle information based on the absolute coordinate system based on the acquired camera position and attitude angle information based on the distance traveled corresponding to the change in the camera position based on the absolute coordinate system and the error in the distance traveled due to the vehicle wheel speed; and a step of calculating a position matrix based on the camera position change based on the verified reliability of the absolute coordinate system and then calculating the vehicle first axis direction vector based on the camera coordinate system from the position matrix. Claim 10 In claim 9, the step of verifying reliability is a calibration method for a vehicle camera in which reliability is determined when there is no rotation component caused by a change in the camera attitude angle based on the absolute coordinate system. Claim 11 In claim 1, the step of calculating the vehicle first axis direction vector based on the camera coordinate system is a calibration method for a vehicle camera that determines the straightness of the vehicle based on the left wheel travel distance, the right wheel travel distance, and the left and right wheel distances of the vehicle. Claim 12 A method for calibrating a vehicle camera according to claim 1, wherein the first axis is the longitudinal axis of the vehicle and the second axis is the vertical axis of the vehicle, and the step of calculating the vehicle second axis direction vector based on the camera coordinate system comprises: a step of calculating a direction vector perpendicular to the ground using camera position and attitude angle information based on the absolute coordinate system and landmarks recognized in the electronic map, and a step of calculating the vehicle vertical axis direction vector based on the camera coordinate system using the direction vector perpendicular to the ground. Claim 13 A method for calibrating a vehicle camera according to claim 1, further comprising the step of calculating a vehicle second axis direction vector based on the camera coordinate system, and then recalculating a vehicle first axis direction vector based on the camera coordinate system based on the result of comparing the standard deviation of the vehicle second axis direction vector based on the camera coordinate system with a preset threshold, wherein the step of calculating a camera rotation matrix based on the vehicle coordinate system uses the recalculated vehicle first axis direction vector based on the camera coordinate system. Claim 14 A method for calibrating a vehicle camera according to claim 1, wherein the step of calculating a camera rotation matrix based on the vehicle coordinate system comprises: a step of calculating a vehicle rotation matrix based on the camera coordinate system that reflects a vehicle third axis direction vector based on the camera coordinate system, based on the calculated vehicle first axis direction vector based on the camera coordinate system and the calculated vehicle second axis direction vector based on the camera coordinate system, and a step of calculating the inverse matrix of the calculated vehicle rotation matrix based on the camera coordinate system as the camera rotation matrix based on the vehicle coordinate system. Claim 15 The method includes an acquisition unit for acquiring a captured image of a vehicle camera and a control unit for performing calibration of the vehicle camera based on the captured image and an electronic map. The control unit acquires camera position and attitude angle information based on an absolute coordinate system by matching a landmark recognized from the captured image with a landmark existing in the electronic map. It calculates a position matrix based on the change in camera position and attitude angle based on the absolute coordinate system acquired according to the movement of the vehicle. Using the position matrix, it acquires a camera rotation matrix based on a vehicle coordinate system that reflects a vehicle first axis direction vector, a vehicle second axis direction vector, and a vehicle third axis direction vector based on the camera coordinate system. When acquiring the camera rotation matrix based on the vehicle coordinate system, it calculates the vehicle first axis direction vector based on the camera coordinate system from the position matrix. It also calculates the vehicle second axis direction vector based on the camera coordinate system using the camera position and attitude angle information based on the absolute coordinate system and the landmark recognized in the electronic map used in calculating the vehicle first axis direction vector based on the camera coordinate system. The calculated vehicle first axis direction vector based on the camera coordinate system and the calculated A calibration device for a vehicle camera that calculates a camera rotation matrix based on the vehicle coordinate system, in which the vehicle third axis direction vector based on the camera coordinate system is reflected, based on the vehicle second axis direction vector based on the camera coordinate system. Claim 16 A computer-readable recording medium storing a computer program, comprising instructions for a processor to perform any one of the methods of claims 1 to 3 and claims 5 to 14. Claim 17 A computer program stored on a computer-readable recording medium, comprising instructions for a processor to perform any one of the methods of claims 1 to 3 and claims 5 to 14.

Images