Camera parameter estimation device and program

Abstract

To appropriately perform an estimation of a camera parameter.SOLUTION: An ECU 20 includes: a feature point detection part 21 that detects a plurality of feature points in each target marker in a captured image in which a plurality of target markers are captured; a coordinate calculation part 22 that performs a projective transformation to convert the plurality of feature points into true values on a marker front-view coordinate for each target marker, and calculates feature point coordinates in a coordinate system on which the projective transformation has been performed; a marker specifying part 23 that, for each target marker, determines detection accuracy of the feature points based on residuals remaining as differences from true value coordinates in the feature point coordinates, and specifies a target marker determined to have low detection accuracy of the feature points; and a parameter estimation part 24 that estimates camera parameters using the coordinates of the feature points of remaining target markers excluding an identified target marker when any of the plurality of target markers is identified as a target marker having low detection accuracy of the feature points.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The disclosure in this specification relates to a camera parameter estimation device and a program. [Background technology]

[0002] For example, there is known a technology for monitoring the surroundings of a vehicle and providing driving assistance based on images captured by a camera mounted on the vehicle. In order to properly monitor the surroundings of a vehicle and provide driving assistance, it is necessary to ensure that the correlation between the coordinate system of the image captured by the camera and the coordinate system of the image used for image processing is consistent. To ensure this accuracy, a technology has been proposed for estimating camera parameters that indicate the mounting state of the camera.

[0003] For example, the technology described in Patent Document 1 corrects distortion in an image captured by a camera, detects feature points of a calibration pattern (i.e., a target marker) in the image, and calculates the position and orientation of the camera from the feature points. In addition, when a series of processes consisting of image distortion correction, feature point detection, and camera position and orientation calculation are repeated, the process is repeated based on the number of repetitions or the reliability value determined in the most recent repetition, thereby estimating parameters with high accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 1,134,1681 Summary of the Invention [Problem to be solved by the invention]

[0005] However, for example, if the camera mounted on the vehicle has poor image quality or if there is an error in the camera's installation, the accuracy of detecting feature points in the calibration pattern may decrease due to the camera's image quality or installation error. In this case, the accuracy of parameters estimated using feature points with low accuracy will be significantly reduced, and the accuracy cannot be improved by simply performing iterative processing. Therefore, there is a concern that the parameters may not be estimated appropriately.

[0006] The present invention has been made in view of the above circumstances, and has an object to provide a camera parameter estimation device and a program that can properly estimate camera parameters. [Means for solving the problem]

[0007] The present disclosure provides: A camera parameter estimation device that acquires, from a camera mounted on a mounting target, a photographed image in which a plurality of target markers, each having a plurality of feature points defined thereon and arranged at a plurality of different locations, is photographed, and estimates camera parameters indicating a mounting state of the camera based on the photographed image, a feature point detection unit that detects the plurality of feature points of each of the target markers in the captured image; a coordinate calculation unit that performs projective transformation for each of the target markers to convert the plurality of feature points detected by the feature point detection unit into true values ​​on a marker front view coordinate system, and calculates feature point coordinates that are coordinates of each of the feature points in the projectively transformed coordinate system; a marker identifying unit that determines, for each of the target markers, the detection accuracy of the feature point based on a residual that remains as a difference between the feature point coordinates calculated by the coordinate calculating unit and true coordinates, and identifies the target marker that is determined to have low detection accuracy of the feature point; a parameter estimation unit that, when the marker identification unit identifies any one of the plurality of target markers as a target marker with low detection accuracy of the feature points, estimates the camera parameters using coordinates of the feature points of the remaining target markers excluding the identified target marker; Equipped with.

[0008] When multiple target markers placed at multiple locations are photographed by a camera mounted on the target, the detection accuracy of feature points for one of the target markers may decrease, and if camera parameters are estimated using feature points with low detection accuracy, there is a concern that the accuracy of the camera parameters may decrease.

[0009] In this regard, the above configuration performs a projective transformation to convert multiple feature points of each target marker detected in the captured image into true values ​​in the marker's front-view coordinate system, and calculates feature point coordinates, which are the coordinates of each feature point for each target marker, in the projectively transformed coordinate system. Furthermore, for each target marker, the feature point detection accuracy is determined based on the residual difference between the feature point coordinates in the coordinate system after projective transformation and the true coordinates, and target markers determined to have low feature point detection accuracy are identified. In this case, if the feature points contained in each target marker are correctly detected in the captured image, the feature point coordinates in the coordinate system after projective transformation will approximately match the true coordinates, whereas if a feature point detection error occurs, the residual difference between the feature point coordinates and the true coordinates will be large. This allows the accuracy of the feature points for each target marker to be appropriately determined.

[0010] Then, of the multiple target markers, the camera parameters are estimated using the feature point coordinates of the remaining target markers, excluding target markers with low feature point detection accuracy. In this case, the suitability of each target marker for use in parameter estimation is determined, and only the feature point information of target markers suitable for parameter estimation is used for parameter estimation, thereby improving the accuracy of parameter estimation. Furthermore, even if it is determined that the feature point detection accuracy of some target markers is low, parameter estimation can be performed using the feature points of the remaining target markers, thereby improving the efficiency of parameter estimation work. As a result, camera parameters can be estimated appropriately. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a plan view showing a vehicle equipped with a camera and a target marker. [Figure 2] FIG. 10 is a diagram showing the configuration of a target marker. [Figure 3] FIG. 2 is a block diagram showing a configuration related to a parameter estimation function in the ECU. [Figure 4] FIG. 10 is a diagram showing detected feature points in an image coordinate system. [Figure 5] FIG. 10 is a diagram showing feature points after normalization. [Figure 6] FIG. 10 is a diagram showing feature points after projective transformation. [Figure 7] FIG. 10 is a diagram showing an error amount for feature points after projective transformation. [Figure 8] FIG. 10 is a diagram showing the relationship between the marker size and the threshold coefficient. [Figure 9] 10 is a flowchart showing a processing procedure for parameter estimation. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of the present invention will be described below with reference to the drawings. This embodiment is a camera parameter estimation device that estimates camera parameters that indicate the mounting state of a camera in an on-board camera system having a camera mounted on a vehicle when the camera is calibrated during vehicle manufacture. Estimating the camera parameters makes it possible to align the coordinate system of an image captured by the camera with the coordinate system of an image used for image processing.

[0013] 1 shows a vehicle 10 equipped with a camera 11 for parameter estimation, and a target marker 12. The vehicle 10 is equipped with multiple cameras 11 for capturing images of the area around the vehicle. The cameras 11 are installed, for example, at the front, rear, left, and right sides of the vehicle 10. Each camera 11 has a CMOS image sensor or a CCD image sensor as an imaging element, and captures an image of an area extending over a predetermined angular range in the forward direction of the camera.

[0014] When calibrating the camera 11, a plurality of target markers 12 are placed at a plurality of different locations in front of the camera 11 in the shooting direction. FIG. 1 shows, as an example, three target markers 12 placed in front of the vehicle 10, and the three target markers 12 are photographed by the camera 11 (front camera) on the front side of the vehicle. The target markers 12 are placed side by side at a predetermined distance from the vehicle 10. The target markers 12 are provided upright on the floor of a factory where the camera shooting is performed, for example.

[0015] The target marker 12 is a calibration marker used for camera calibration, and has a graphic pattern on the front side, for example, as shown in FIG. 2. In FIG. 2, the target marker 12 has a graphic pattern of alternating black and white checkered patterns and an X pattern in the center of the marker, and a plurality of feature points F are defined in this graphic pattern to serve as landmarks. In this example, a total of five points, including intersections of the checkered patterns and X patterns, are defined as the feature points F. The graphic pattern of the target marker 12 may be any pattern as long as it has a plurality of feature points F defined. It is preferable that the target marker 12 has four or more feature points F defined.

[0016] The vehicle 10 is equipped with an ECU (Electronic Control Unit) 20 that functions as a camera parameter estimation device. The ECU 20 is a computer (more specifically, a processor such as a microcomputer) equipped with a CPU, ROM, RAM, an input / output interface, etc. The CPU realizes these functions by executing programs stored in the ROM and RAM as storage media. The ECU 20 is capable of communicating with the front, rear, left, and right cameras 11 via wired or wireless communication. The ECU 20 acquires images captured by each camera 11 and estimates camera parameters based on the captured images.

[0017] Next, the parameter estimation function of the ECU 20 will be described with reference to Fig. 3. The ECU 20 includes a feature point detection unit 21, a coordinate calculation unit 22, a marker identification unit 23, a parameter estimation unit 24, and a notification unit 25.

[0018] The feature point detection unit 21 detects multiple feature points F for each target marker 12 in the captured image captured by the camera 11. For example, the feature point detection unit 21 detects five feature points F for each target marker 12 based on the luminance information of the captured image. At this time, the five feature points F are detected in an image coordinate system on the captured image. Figure 4 shows the five feature points F detected in the image coordinate system.

[0019] The coordinate calculation unit 22 performs projective transformation for each target marker 12 to convert the multiple feature points F detected by the feature point detection unit 21 into true values ​​on the marker front view coordinate system, and calculates feature point coordinates, which are the coordinates of each feature point F, for each target marker 12 in the projectively transformed coordinate system. The marker front view coordinates are planar view coordinates when the target marker 12 is viewed from directly in front. The projective transformation determines the coordinate position of each feature point F on the front view coordinate system.

[0020] In this embodiment, the coordinate calculation unit 22 performs projective transformation using a homography matrix. In this case, first, to perform homography transformation with high accuracy, the coordinates of the five feature points F detected by the feature point detection unit 21 are normalized. Specifically, as shown in Fig. 5, the center and scale of the coordinates of the five feature points F are changed so that the average is 0 and the standard deviation is 1 in the image coordinate system.

[0021] In addition, a homography matrix used for projective transformation is estimated. At this time, a matrix for projective transformation is calculated to calibrate the deviation in the viewing direction from the camera 11 in order to determine the positional relationship of each target marker 12 when viewed from the front. Known methods for estimating a homography matrix include, for example, an optimization method using iterative calculations and an estimation method using the least squares method. Here, calculation of a homography matrix using the least squares method is described.

[0022] First, the homography transformation formula from the normalized feature point coordinates to the true coordinates is as follows:

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027] Then, the coordinate calculation unit 22 multiplies the coordinates of the five feature points F normalized in the image coordinate system (FIG. 5) by the homography matrix to obtain the five feature point coordinates after the projective transformation. That is, the coordinate calculation unit 22 calculates the feature point coordinates, which are the coordinates of each feature point F for each target marker 12, in the projectively transformed coordinate system. At this time, by transforming into a projective image, the screen coordinates of the image captured by the camera 11 are transformed into front-view coordinates when each target marker 12 is viewed from the front. Then, the positional relationship of the target markers 12 when viewed from the front is obtained as the positional relationship of each feature point F. FIG. 6 shows the five feature point coordinates after the projective transformation.

[0028] The marker identification unit 23 calculates, for each target marker 12, a residual that remains as the difference between the feature point coordinates calculated by the coordinate calculation unit 22 and the true coordinates, and determines the detection accuracy of the feature point F based on the residual. Then, it identifies target markers 12 for which the detection accuracy of the feature point F is determined to be low. When each feature point F of the target marker 12 is transformed into a positional relationship as seen from the front by homography transformation, the position coordinates that make this positional relationship accurate are the true coordinates. In this embodiment, for each feature point F of the target marker 12, the residual, which is the difference between the feature point coordinates after projective transformation and the known true coordinates, is compared with a threshold, and if the residual is greater than the threshold, it is determined that the detection accuracy of the feature point F is low.

[0029] 7, if there is an error in the feature point coordinates after projective transformation, indicated by black dots, with respect to the true coordinates, indicated by white dots, even if optimal homography transformation is performed, the coordinates will be transformed to positions that are displaced from the true coordinates, resulting in a large residual. Therefore, if this residual exceeds a predetermined value, it is determined that the accuracy of the feature point coordinates is low.

[0030] More specifically, the marker identification unit 23 calculates the root mean square of the error between the true coordinates and the feature point coordinates after projective transformation of the five points as the guard index value. The guard index value is calculated for each target marker 12. Furthermore, the smaller the size of the target marker 12 on the captured image, the more likely the guard index value is to deteriorate due to a decrease in resolution. Therefore, in this embodiment, in order to level out the guard sensitivity, a threshold is set for each target marker 12 based on the size of the target marker 12 on the captured image (threshold setting unit 26).

[0031] In this case, it is preferable to calculate the threshold coefficient based on the marker size using, for example, the relationship in Fig. 8. In Fig. 8, a relationship is defined such that the smaller the marker size, the smaller the threshold coefficient. Then, the threshold is set by multiplying a base threshold defined as an allowable deviation amount from the true coordinates in the feature point coordinates by the threshold coefficient calculated using the relationship in Fig. 8.

[0032] The marker identification unit 23 compares a guard index value, which corresponds to the residual between the feature point coordinates after projective transformation of the five points and the true coordinates, with a threshold value for each target marker 12. If the guard index value is smaller than the threshold value, it determines that the detection accuracy of the feature point F is not low, and if the guard index value is larger than the threshold value, it determines that the detection accuracy of the feature point F is low. It also identifies target markers 12 that include feature points F with low detection accuracy. That is, for each target marker 12, it determines whether or not the marker recognition by the camera 11 is correct based on the accuracy determination result of the feature point F, and distinguishes between target markers 12 for which the marker recognition is correct and target markers 12 for which the marker recognition is unsuccessful.

[0033] The parameter estimation unit 24 estimates optimal camera parameters for the camera 11 based on the coordinates of each feature point F (for example, the coordinates of five feature points after projective transformation). Specifically, the difference between the coordinates obtained by converting the coordinates of the feature point F into world coordinates (coordinates in real space) and the design coordinates is used as an evaluation value, and parameters that minimize this evaluation value are estimated. The camera parameters are parameters that indicate, for example, the mounting position of the camera 11 on the vehicle 10, the direction of the optical axis of the camera 11, etc., and more specifically, the position (X coordinate, Y coordinate, Z coordinate) and angle (roll, pitch, yaw) of the camera 11, and distortion of the camera lens.

[0034] If any of the multiple target markers 12 is identified as a target marker 12 with low detection accuracy of the feature point F, the parameter estimation unit 24 estimates the camera parameters using the feature point coordinates of the remaining target markers 12 excluding the identified target marker 12 (i.e., the target marker 12 for which marker recognition failed).

[0035] When the marker identifying unit 23 identifies any one of the multiple target markers 12 included in the captured image as a target marker 12 with low detection accuracy of the feature point F, the notification unit 25 notifies the external device of the identification information. In this case, the success or failure of marker recognition for each of the multiple (e.g., three) target markers 12 positioned in front of the camera 11 in the shooting direction is notified to an external device. The external device is, for example, a management device that manages production in a manufacturing factory.

[0036] The notification content from the notification unit 25 may include information about the cause of the failure of marker recognition for the target marker 12. In this case, it is possible to encourage improvements in camera recognition of the target marker 12. The notification may also include suggestions for improvements to the operator, such as the appropriateness of the placement of the target marker 12 or repositioning. Information quantifying the detectability index and false detection risk associated with the vehicle specifications and marker placement may be calculated and notified. Information quantifying the reliability of parameter estimation (for example, the determinant of the homography matrix as a measure) may also be calculated and notified.

[0037] 9 is a flowchart showing the procedure for parameter estimation. This process is repeatedly executed by the ECU 20 at a predetermined interval during the camera calibration process at the time of vehicle manufacture. In this embodiment, parameter estimation using the feature points F of each target marker 12 is performed in a multiple loop.

[0038] 9, in step S11, an image captured by the camera 11 is acquired. In the following step S12, viewpoint conversion is performed for each target marker 12 in the captured image so that the target marker 12 is displayed in a position directly in front of the camera 11. At this time, the target marker image is enlarged, reduced, and rotated using the mounting information of the camera 11 and the placement information of the target marker 12. The camera mounting information is, for example, the position and angle of the camera 11, and the target marker placement information is, for example, the position and angle of the target marker 12 relative to the camera 11.

[0039] Then, in step S13, a plurality of feature points F are detected in each target marker 12 in the captured image (see FIG. 4). In step S14, the coordinates of each feature point F detected in step S13 are normalized (see FIG. 5).

[0040] Then, in step S15, a homography matrix used for projective transformation is estimated. In step S16, the coordinates of the five feature points F normalized in the image coordinate system are multiplied by the homography matrix to obtain the coordinates of the five feature points after projective transformation (see FIG. 6).

[0041] In step S17, a guard index value corresponding to the residual between the true coordinates and the feature point coordinates after the projective transformation is calculated. At this time, for each target marker 12, the guard index value is calculated as the mean square of the error between the true coordinates and the feature point coordinates after the projective transformation of five points.

[0042] In step S18, a threshold is set for comparing with the residual error of the feature point coordinates after projective transformation for each feature point F. At this time, the threshold is preferably set based on the size of the target marker 12 on the captured image.

[0043] In step S19, the guard index value calculated in step S17 is compared with the threshold value set in step S18 to identify target markers 12 with low detection accuracy of the feature point F.

[0044] In step S20, the camera parameters of the camera 11 are estimated based on the coordinates of each feature point F. At this time, if any of the multiple target markers 12 is identified in step S19 as a target marker 12 with low detection accuracy of the feature point F, the camera parameters are estimated using the feature point coordinates of the remaining target markers 12 excluding the identified target marker 12.

[0045] Thereafter, in step S21, it is determined whether or not the current parameter estimation was the final loop. If it is not the final loop, the process returns to step S12, and if it is the final loop, the process proceeds to the subsequent step S22.

[0046] In step S22, the ECU 20 notifies the outside of the plurality of target markers 12 used in the current parameter estimation of whether or not any target markers 12 have failed to be recognized, i.e., the identification information of the target markers 12. At this time, the identification information of the target markers 12 can also be stored in a backup memory within the ECU 20.

[0047] According to the present embodiment described above in detail, the following excellent effects can be obtained.

[0048] For each target marker 12, the feature point coordinates, which are the coordinates after projective transformation for a plurality of feature points F, are compared with the true coordinates to determine the detection accuracy of the feature points F, and target markers 12 determined to have low detection accuracy of the feature points F are identified. Then, camera parameters are estimated using the feature point coordinates of the remaining target markers 12, excluding the target markers 12 with low detection accuracy of the feature points F. In this case, the suitability for use in parameter estimation is determined for each target marker 12, and only the feature point information of target markers 12 suitable for parameter estimation is used for parameter estimation, thereby improving the accuracy of parameter estimation. Furthermore, even if the feature point detection accuracy of some target markers 12 is determined to be low, parameter estimation can be performed using the feature points of the remaining target markers 12, thereby improving the efficiency of parameter estimation. As a result, camera parameters can be estimated appropriately.

[0049] Because the projective transformation is performed using a homography matrix, it is possible to accurately grasp the positional relationship of each feature point F when viewing the target marker 12 from the front. This allows the detection error of each feature point F to be accurately determined, thereby improving the accuracy of parameter estimation.

[0050] In the image captured by the camera 11, the resolution changes depending on the size of the target marker 12. Taking this into consideration, a threshold value is set for each target marker 12 based on the size of the target marker 12 in the captured image, against which the residual error of the feature point coordinates after projective transformation is compared. This makes it possible to level the sensitivity of the accuracy judgment for each feature point, regardless of the size of the target marker 12.

[0051] When the detection accuracy of the feature points F is low for some of the target markers 12 due to camera image quality or installation errors of the camera 11, it is possible to estimate the camera parameters using the coordinates of the feature points F of the remaining target markers 12, while notifying the outside of marker identification information for the target markers 12 with low detection accuracy of the feature points F. This makes it possible to encourage improvements to the marker positions during camera calibration work.

[0052] The above embodiment may be modified as follows, for example.

[0053] In the above embodiment, the projective transformation is performed by homography transformation, but the projective transformation method can be changed. The point is that it is sufficient to be able to transform the image captured by the camera 11 into a positional relationship where each feature point of the target marker is viewed from the front.

[0054] In the above embodiment, the vehicle 10 is configured to estimate the parameters of each camera 11 mounted thereon, but this may be modified. For example, the system may estimate the parameters of a camera mounted on a moving body other than a vehicle, such as a ship or an aircraft. Furthermore, the system may estimate the parameters of a camera mounted on a stationary device, not limited to a moving body.

[0055] The control unit and method described herein may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the control unit and method described herein may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and method described herein may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to perform one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer. [Explanation of symbols]

[0056] 11...camera, 12...target marker, 20...ECU.

Claims

1. A camera parameter estimation device (20) acquires, from a camera (11) mounted on a mounting target, a photographed image of a plurality of target markers (12) having a plurality of feature points and arranged at a plurality of different locations, and estimates camera parameters indicating the mounting state of the camera based on the photographed image, the device comprising: a feature point detection unit that detects the plurality of feature points of each of the target markers in the captured image; a coordinate calculation unit that performs projective transformation for each of the target markers to convert the plurality of feature points detected by the feature point detection unit into true values ​​on a marker front view coordinate system, and calculates feature point coordinates that are coordinates of each of the feature points in the projectively transformed coordinate system; a marker identifying unit that determines, for each of the target markers, the detection accuracy of the feature point based on a residual that remains as a difference between the feature point coordinates calculated by the coordinate calculating unit and true coordinates, and identifies the target marker that is determined to have low detection accuracy of the feature point; a parameter estimation unit that, when the marker identification unit identifies any one of the plurality of target markers as a target marker with low detection accuracy of the feature points, estimates the camera parameters using coordinates of the feature points of the remaining target markers excluding the identified target marker; A camera parameter estimation device comprising:

2. The camera parameter estimation device according to claim 1 , wherein the coordinate calculation unit performs the projective transformation using a homography matrix.

3. the marker identification unit compares the residual with a threshold, and determines that the feature point detection accuracy is low when the residual is greater than the threshold; The camera parameter estimation device according to claim 1 , further comprising a threshold setting unit that sets the threshold for each of the target markers based on the size of the target marker on the captured image.

4. The camera parameter estimation device according to any one of claims 1 to 3, further comprising a notification unit that, when the marker identification unit identifies any one of the plurality of target markers as a target marker with low detection accuracy of the feature points, notifies an external party of the identification information.

5. A parameter estimation program for acquiring, from a camera (11) mounted on a mounting target, a photographed image in which a plurality of target markers (12) having a plurality of feature points and arranged at a plurality of different locations is photographed, and estimating camera parameters indicating the mounting state of the camera based on the photographed image, comprising: a feature point detection process for detecting the plurality of feature points of each of the target markers in the captured image; a coordinate calculation process for performing a projective transformation for converting the plurality of feature points detected by the feature point detection process into true values ​​on a marker front view coordinate system for each of the target markers, and calculating feature point coordinates, which are coordinates of each of the feature points in the projectively transformed coordinate system; a marker identification process for determining, for each of the target markers, the detection accuracy of the feature point based on a residual that remains as a difference between the feature point coordinates calculated by the coordinate calculation process and true coordinates, and identifying the target markers for which the detection accuracy of the feature point is determined to be low; a parameter estimation process for estimating the camera parameters using coordinates of the feature points of the remaining target markers excluding the identified target marker when the marker identification process identifies any one of the plurality of target markers as a target marker with low detection accuracy of the feature points; A program that causes a computer to execute the above.

Images