Camera parameter evaluation device and program

Abstract

To provide a camera parameter evaluation device that can properly evaluate camera parameters. [Solution] The ECU 20 evaluates camera parameters indicating the camera's mounting status based on a camera image in which a target marker containing feature points is captured. The ECU 20 includes a position information acquisition unit 22 that acquires position information including the distance from the camera to the feature points, a coordinate position calculation unit 23 that calculates the coordinate position of the feature points on the coordinates of the captured image, a projection plane setting unit 24 that sets a spherical projection plane in real space based on the distance from the camera to the feature points, a feature point projection unit 25 that projects the coordinate position calculated by the coordinate position calculation unit 23 onto the projection plane set by the projection plane setting unit 24, and a parameter evaluation unit 26 that evaluates camera parameters based on the position information of the feature points acquired by the position information acquisition unit 22 and the projected position of the feature points projected onto the projection plane.

Description

Technical Field

[0001] The disclosure in this specification relates to a camera parameter evaluation apparatus and a program. Background Art

[0002] For example, there is known a technology in which vehicle surroundings monitoring and driving assistance are performed based on images captured by a camera mounted on a vehicle. Further, in order to appropriately perform vehicle surroundings monitoring and driving assistance, it is necessary to appropriately evaluate camera parameters indicating the mounting state of the camera on the vehicle, such as the position and orientation of the camera. As a method for evaluating camera parameters, for example, there is known a technology in which a target marker placed in front of a vehicle is captured by a camera, feature points of the target marker are detected in the camera image, and camera parameters are estimated from the feature points. Further, there is known a technology in which feature points in real space and feature points detected on a camera image are projected onto the same projection plane, and camera parameters are evaluated on the projection plane (see, for example, Patent Document 1). Prior Art Literature Patent Literature

[0003] Patent Document 1 Japanese Unexamined Patent Publication No. 2011-215082 Summary of the Invention Problem to be Solved by the Invention

[0004] By the way, when setting a projection plane on a camera image, there is a concern that feature points cannot be projected appropriately depending on lens characteristics of the camera or the like. For example, when a plurality of target markers are captured by a camera using a wide-viewing-angle lens such as a fisheye lens, the feature points of all target markers cannot be projected into the camera image, and there is a concern that some feature points end up outside the camera image.

[0005] Furthermore, when setting a projection plane in real space, if that projection plane is uniform, projection may become difficult depending on the positional relationship between the camera and the feature points of the target marker. For example, if the projection plane is set parallel to the ground and at a predetermined height from the ground, and the camera's height is lower than the projection plane and the camera's shooting direction is downwards from the projection plane, it will be difficult to project the feature points onto the projection plane. Also, if the projection plane extends in the left-right direction of the vehicle, it will be difficult to project the feature points onto the projection plane using a camera (side camera) located on the side of the vehicle.

[0006] As described above, if the projection plane is not set properly, there is a concern that the camera parameters cannot be properly evaluated using feature points on the projection plane.

[0007] This disclosure is made in view of the above circumstances and aims to provide a camera parameter evaluation device and program that can properly evaluate camera parameters. [Means for solving the problem]

[0008] The camera parameter evaluation device of this disclosure is A camera parameter evaluation device that acquires a captured image of a target marker including feature points from a camera mounted on an object, and evaluates camera parameters indicating the mounting status of the camera based on the captured image, A position information acquisition unit acquires position information including the distance from the camera to the feature point for the aforementioned feature point, A coordinate position calculation unit that calculates the coordinate position of the feature point on the coordinates of the captured image, A projection surface setting unit sets the projection surface in real space as a sphere based on the distance between the camera and the feature point, A feature point projection unit projects the coordinate position calculated by the coordinate position calculation unit onto the projection plane set by the projection plane setting unit, A parameter evaluation unit evaluates the camera parameters based on the position information of the feature points acquired by the position information acquisition unit and the projected position of the feature points projected onto the projection surface. It is equipped with.

[0009] According to the above configuration, the projection plane in real space is set as a sphere based on the distance between the camera and the feature point, and the coordinate position of the feature point on the coordinate system of the captured image is projected onto the spherical projection plane. Then, the camera parameters are evaluated based on the position information of the feature point and the projected position of the feature point on the projection plane. As described above, with a projection plane set as a sphere, the feature point can be suitably projected onto the projection plane regardless of the positional relationship between the camera and the feature point of the target marker. Similarly, even when the target marker is captured with a wide-angle lens, the feature point can be suitably projected onto the projection plane. As a result, the camera parameters can be properly evaluated. [Brief explanation of the drawing]

[0010] [Figure 1] A plan view showing a vehicle equipped with a camera and a target marker. [Figure 2] A block diagram showing the configuration of the parameter evaluation function in the ECU. [Figure 3] A diagram showing an image captured by a camera. [Figure 4] A diagram showing the spherical projection plane for each feature point. [Figure 5] A diagram showing a projection plane with an increased radius of a sphere. [Figure 6] A flowchart illustrating the parameter evaluation process. [Modes for carrying out the invention]

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. This embodiment embodies a camera parameter evaluation device for an in-vehicle camera system having a camera mounted on a vehicle, which evaluates camera parameters indicating the mounting state of the camera during camera calibration at the time of vehicle manufacturing. By evaluating the camera parameters, it is possible to align the coordinate system of the image captured by the camera with the coordinate system of the image used for image processing.

[0012] Figure 1 shows a vehicle 10 equipped with a camera 11 to be evaluated for parameter evaluation, and a target marker 12. The vehicle 10 is equipped with multiple cameras 11 that capture images of the area around the vehicle. The cameras 11 are installed, for example, at the front, rear, left, and right sides of the vehicle 10. Each camera 11 has a CMOS image sensor or a CCD image sensor as its image sensor and captures an area that extends within a predetermined angular range in front of its respective shooting direction.

[0013] When calibrating camera 11, multiple target markers 12 are placed at different locations in front of the camera 11 in the direction of shooting. Figure 1 shows, as an example, a state in which three target markers 12 are placed in front of vehicle 10, and the three target markers 12 are photographed by the camera 11 on the front side of the vehicle (front camera). Each target marker 12 is placed at a predetermined distance from vehicle 10. The target markers 12 are installed in an upright position on the floor of a factory where camera shooting takes place.

[0014] The target marker 12 is a calibration marker used for camera calibration and has a graphic pattern for camera recognition on its front side. The graphic pattern of the target marker 12 is, for example, an alternating black and white checkerboard pattern, and feature points F are defined in this graphic pattern. In this example, one feature point F is defined at the intersection of the checkerboard pattern. However, the graphic pattern of the target marker 12 is arbitrary.

[0015] A vehicle 10 includes an ECU (Electronic Control Unit) 20 that functions as a camera parameter evaluation device. The ECU 20 is a computer (more specifically, a processor such as a microcomputer) including a CPU, ROM, RAM, input / output interface and the like. The CPU implements these respective functions by executing programs stored in a ROM or RAM serving as a storage medium. The ECU 20 is capable of mutual communication with each of the front-rear and left-right cameras 11 via wired or wireless communication. The ECU 20 acquires captured images from each camera 11 and evaluates camera parameters based on the captured images.

[0016] The camera parameters are, for example, parameters indicating the mounting position of the camera 11 on the vehicle 10, the direction of the optical axis of the camera 11, and the like. More specifically, the camera parameters are the position (X coordinate, Y coordinate, Z coordinate), angle (roll, pitch, yaw) of the camera 11, and distortion of the camera lens. The ECU 30 calculates evaluation values of camera parameters for each camera 11 mounted on the vehicle 10, and appropriately updates the camera parameters based on the evaluation values.

[0017] Next, the parameter evaluation function in the ECU 20 will be described with reference to FIG. 2. The ECU 20 includes a feature point detection unit 21, a position information acquisition unit 22, a coordinate position calculation unit 23, a projection plane setting unit 24, a feature point projection unit 25, and a parameter evaluation unit 26.

[0018] The feature point detection unit 21 detects the feature point F of each target marker 12 using an image captured by the camera 11. For example, the feature point detection unit 21 detects a feature point F for each target marker 12 based on luminance information in the captured image. FIG. 3 shows an image captured by the camera 11. In this captured image, three target markers 12 are captured, and a feature point F is detected for each target marker 12.

[0019] The position information acquisition unit 22 acquires predetermined position information for the feature point F of each target marker 12. The positional relationship between the vehicle 10 and the feature point F of each target marker 12 is predetermined, and the ECU 20 acquires, as known position information, position data of each feature point F referenced to the camera 11 of the vehicle 10, and the separation distance D from the camera 11 of the vehicle 10 to each feature point F. Note that the separation distance D from the camera 11 to each feature point F is the actual distance from the camera 11 to each feature point F, and may be measured by a distance measuring device such as a radar sensor in the vehicle 10. In this case, the distance measuring device measures the distance to the target marker 12 (feature point F), and the ECU 20 acquires the measured distance from the distance measuring device as the separation distance D.

[0020] The coordinate position calculation unit 23 calculates the coordinate position of each feature point F detected on the coordinates of the captured image. In the following description, the position of each feature point F on the captured image is also referred to as the "detection position of the feature point F". In the captured image shown in FIG. 3, detection positions are calculated respectively for three feature points F having different separation distances D from the camera 11.

[0021] The projection plane setting unit 24 sets the projection plane PP as a spherical surface in real space, which is three-dimensional world coordinates, based on the separation distance D from the camera 11 to the feature point F. Specifically, as shown in FIG. 4, the projection plane setting unit 24 sets the projection plane PP in real space by a spherical surface centered at the position of the camera 11, having a radius equal to the separation distance D to the feature point F, and passing through the feature point F. In FIG. 4, a known feature point F existing at a predetermined position is denoted as "feature point Fa" and marked with a black circle. The projection plane PP is set as a three-dimensional spherical surface passing through the known feature point Fa in real space.

[0022] The projection plane setting unit 24 sets a projection plane PP on a sphere for each feature point F of each target marker 12, based on the distance D from the camera 11 to the feature point F. The three feature points Fa shown in Figure 4 each have different distances D from the camera 11, and the projection plane PP is set by three spheres corresponding to each feature point Fa. In Figure 4, each projection plane PP is set by a sphere passing through each known feature point Fa. Here, the projection planes PP are designated as projection plane P1, projection plane P2, and projection plane P3, starting from the side closest to the camera 11.

[0023] The feature point projection unit 25 projects each feature point F detected on the image captured by the camera 11 onto the projection surface PP set by the projection surface setting unit 24. At this time, the detected position of each feature point F calculated by the coordinate position calculation unit 23 is converted into the corresponding coordinate position on the projection surface PP and projected onto the projection surface PP. In Figure 4, the projected points Fp on the projection surface PP where each feature point F is projected are shown as white circles. At this time, if there is an error in the detected position of each feature point F, a discrepancy will occur between the known feature point Fa and the projected point Fp.

[0024] The parameter evaluation unit 26 calculates the position error of feature point F based on the known feature point Fa and the projection point Fp on the projection plane PP, and evaluates the camera parameters based on that position error. At this time, the positional difference between the known feature point Fa and the projection point Fp is used as the evaluation value, and it is preferable to update the camera parameters to minimize this evaluation value.

[0025] The sum of the positional errors of all feature points detected by the camera 11 being calibrated should be calculated as the evaluation value (Cost). The Cost is calculated using the following formula (1).

[0026]

number

[0027] According to the cost function described above, the positional error between a known feature point Fa and a projection point Fp in the camera 11 being calibrated is calculated by summing all feature points. The parameter evaluation unit 26 updates the camera parameters to minimize the evaluation value described above.

[0028] Furthermore, when multiple feature points F (target markers 12) are captured by the camera 11, the degree of influence due to positional error may differ for each feature point F. Taking this into consideration, it is possible to set a contribution level for each feature point F when calculating the sum of positional errors. In this case, the parameter evaluation unit 26 may include a contribution level setting unit 27 that sets the contribution level for each feature point F when calculating the sum of positional errors.

[0029] The contribution setting unit 27 changes the contribution of each feature point F by changing the weight coefficient k of each feature point F from 1 in equation (1). In this case, changing the radius of the spherical projection plane PP onto which the feature point F is projected is equivalent to changing the contribution of each feature point F. Therefore, in this embodiment, as a setting of the contribution, the radius of the projection plane PP onto which any of the multiple feature points F is projected is changed.

[0030] Figure 5 shows the correspondence when the radius of the projection plane PP onto which the feature point F is projected is increased. In Figure 5, r is the radius of the sphere. θ is the angle between the vector from camera 11 to feature point Fa and the vector to projection point Fp. In this case, the positional error between feature point Fa and projection point Fp can be defined as shown in equation (2) below. Here, |vF| represents the magnitude of the position vector vF, i.e., the distance between camera 11 and feature point Fa.

[0031]

number

[0032]

number

[0033] Here, in object recognition using images captured by camera 11, the accuracy of object recognition is likely to decrease as the object is farther from vehicle 10. Taking this into consideration, the contribution setting unit 27 should set the contribution of feature points F that are farther from vehicle D to be greater than the contribution of feature points F that are closer to vehicle D. In other words, for the feature points F that are farther from vehicle 10 among the multiple feature points F, the radius of the projection plane PP is set to be greater than the separation distance D.

[0034] Figure 6 is a flowchart showing the parameter evaluation process. This process is repeatedly performed at predetermined intervals by the ECU20 during the camera calibration process in vehicle manufacturing.

[0035] In Figure 6, step S11 acquires an image captured by camera 11. In the subsequent step S12, the image captured by camera 11 is used to detect the feature points F of each target marker 12 (see Figure 3).

[0036] In step S13, predetermined positional information is acquired for each feature point F of each target marker 12. At this time, known positional information is acquired, which includes the positional data of each feature point F and the distance D from the camera 11 of the vehicle 10 to each feature point F.

[0037] In step S14, the coordinate positions (detection positions) of each feature point F detected on the coordinates of the captured image are calculated.

[0038] In step S15, the projection plane PP is set by a sphere in real space based on the distance D between the camera 11 and the feature point F (see Figure 4). At this time, for each feature point F of each target marker 12, the projection plane PP is set by a sphere passing through a known feature point Fa. At this time, it is possible to make the radius of the projection plane PP larger than the distance D for any of the multiple feature points F (see Figure 5).

[0039] In step S16, each feature point F detected on the image captured by the camera 11 is projected onto the projection surface PP set by the projection surface setting unit 24.

[0040] In step S17, the position error of feature point F is calculated based on the known feature point Fa and the projection point Fp of each feature point F projected onto the projection plane PP, and the camera parameters are evaluated based on this position error. In this parameter evaluation, the sum of the position errors of the camera parameters for each feature point at multiple target markers 12 is calculated using the evaluation value (Cost) described above.

[0041] According to the embodiment described in detail above, the following excellent effects can be obtained.

[0042] The projection plane PP in real space is set as a sphere based on the separation distance D from the camera 11 to the feature point F, and camera parameters are evaluated based on the known positional information of the feature point and the projected position of the feature point on the projection plane PP. This allows the feature point F to be suitably projected onto the projection plane regardless of the positional relationship between the camera 11 and the feature point F of the target marker 12. Similarly, even when the target marker 12 is captured with a wide-angle lens, the feature point F can be suitably projected onto the projection plane PP. As a result, camera parameters can be properly evaluated.

[0043] The projection plane PP in real space is set as a sphere whose radius is the distance D from the camera 11 to the feature point F, and which passes through the known feature point Fa. In this case, the feature point F of the target marker 12 captured by the camera 11 can be properly projected onto the projection plane PP, thereby improving the accuracy of camera parameter evaluation.

[0044] When multiple target markers 12 with different distances D from the camera 11 are captured by the camera 11, a projection plane PP is set for each feature point F of each target marker 12 using a sphere passing through a known feature point Fa. Furthermore, for each feature point F, the coordinate position is projected onto the corresponding projection plane PP, and the camera parameters are evaluated. In this case, a projection plane PP that considers the positional relationship with the camera 11 is set for each feature point F, improving the accuracy of parameter evaluation at each feature point F. Additionally, relative evaluation can be performed at each feature point F according to the distance between the camera 11 and the feature point F. This makes it possible to improve the accuracy of object recognition performance over a wide range, from nearby to distant locations, in an in-vehicle camera.

[0045] When multiple feature points F (target markers 12) are captured, the degree of influence due to positional error may differ for each feature point F. Considering this, a contribution level is set for each feature point F when calculating the sum of positional errors. This allows the camera 11 to perform a proper overall evaluation of camera parameters.

[0046] The contribution of feature points F located far from vehicle 10 is made greater than the contribution of feature points F located close to vehicle 10. This allows for optimal object recognition using camera images over a wide range of distances, from near to far, around the vehicle.

[0047] For feature points F, which have a relatively large contribution, the radius of the spherical projection plane PP was made larger than the distance between camera 11 and feature point F. Increasing the radius of the projection plane (sphere) increases the sensitivity to positional errors and increases the contribution to the sum of positional errors. This allows for a proper overall evaluation of camera parameters.

[0048] The above embodiment may be modified as follows, for example.

[0049] In areas close to the vehicle 10, the required accuracy of object recognition may be higher than in areas farther from the vehicle 10, in order to avoid contact with objects near the vehicle. Taking this into consideration, the contribution setting unit 27 may set the contribution of feature points F that are close to the vehicle 10 to be greater than the contribution of feature points F that are farther from the vehicle 10. In other words, for the feature points F that are closer to the vehicle 10 among the multiple feature points F, the radius of the projection plane PP is set to be greater than the separation distance D. This allows the vehicle 10 to perform object recognition using camera images effectively, and consequently, to properly avoid contact with objects near the vehicle.

[0050] Multiple feature points F may be defined for a single target marker 12. In this case, a spherical projection plane may be set for each feature point F.

[0051] In the above embodiment, the vehicle 10 is designated as the "mounting target," and the configuration is set up to evaluate the parameters of each camera 11 mounted on the vehicle 10. However, this can be changed. For example, the system may evaluate the parameters of cameras mounted on moving objects other than vehicles, such as ships or aircraft. Furthermore, it may evaluate the parameters of cameras mounted on stationary devices, not limited to moving objects.

[0052] The control unit and method described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the control unit and method described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the control unit and method described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. [Explanation of Symbols]

[0053] 11...Camera, 12...Target marker, 20...ECU, 22...Position information acquisition unit, 23...Coordinate position calculation unit, 24...Projection plane setting unit, 25...Feature point projection unit, 26...Parameter evaluation unit.

Claims

1. A camera parameter evaluation device (20) that acquires an image of a target marker (12) including feature points from a camera (11) mounted on an object to be mounted, and evaluates camera parameters indicating the mounting status of the camera based on the captured image, A position information acquisition unit (22) acquires position information including the distance from the camera to the feature point for the aforementioned feature point, A coordinate position calculation unit (23) calculates the coordinate position of the feature point on the coordinates of the captured image, A projection plane setting unit (24) sets the projection plane in real space as a sphere based on the distance between the camera and the feature point, A feature point projection unit (25) projects the coordinate position calculated by the coordinate position calculation unit onto the projection plane set by the projection plane setting unit, A parameter evaluation unit (26) evaluates the camera parameters based on the position information of the feature points acquired by the position information acquisition unit and the projected position of the feature points projected onto the projection surface, A camera parameter evaluation device equipped with the following features.

2. The camera parameter evaluation device according to claim 1, wherein the projection plane setting unit sets the projection plane in real space by a sphere whose radius is the distance from the camera to the feature point and which passes through the feature point.

3. Multiple feature points with different distances from the camera are captured by the camera, and the camera parameters are evaluated based on each of these feature points. The projection plane setting unit sets the projection plane on a spherical surface for each feature point based on the distance from the camera to the feature point. The feature point projection unit projects the coordinate position onto the corresponding projection surface for each feature point. The camera parameter evaluation device according to claim 1 or 2, wherein the parameter evaluation unit evaluates the camera parameter for each feature point based on the positional information of the feature point and the projected position of the feature point projected onto the projection surface.

4. The parameter evaluation unit, This method calculates the sum of the positional errors of the camera parameters for multiple feature points. The camera parameter evaluation device according to claim 3, further comprising a contribution setting unit (27) for setting the contribution of each of the aforementioned feature points when calculating the sum of the position errors.

5. The aforementioned mounting target is a vehicle, and the multiple aforementioned feature points include those with different distances from the vehicle. The camera parameter evaluation device according to claim 4, wherein the contribution setting unit makes the contribution of feature points that are farther away from the vehicle greater than the contribution of feature points that are closer to the vehicle.

6. The aforementioned mounting target is a vehicle, and the multiple aforementioned feature points include those with different distances from the vehicle. The camera parameter evaluation device according to claim 4, wherein the contribution setting unit makes the contribution of feature points that are close in distance from the vehicle greater than the contribution of feature points that are far from the vehicle.

7. The camera parameter evaluation device according to claim 4, wherein the projection plane setting unit makes the radius of the spherical projection plane larger than the distance from the camera to the feature point for the feature point that has a relatively large contribution.

8. A program applied to a camera parameter evaluation device that acquires an image of a target marker (12) including characteristic points from a camera (11) mounted on an object, and evaluates camera parameters indicating the mounting status of the camera based on the captured image, The processor, A position information acquisition process that acquires position information including the distance from the camera to the feature point for the aforementioned feature point, A coordinate position calculation process that calculates the coordinate position of the feature point on the coordinates of the captured image, A projection plane setting process that sets the projection plane in real space as a sphere based on the distance between the camera and the feature point, A feature point projection process is performed to project the coordinate position calculated by the above coordinate position calculation process onto the projection plane set by the above projection plane setting process, A parameter evaluation process that evaluates the camera parameters based on the position information of the feature points obtained by the position information acquisition process and the projected position of the feature points projected onto the projection surface, A program that executes the command.

Images