Camera calibration device and its program

Abstract

To provide a camera calibration apparatus capable of calibrating each of positions of feature points and camera parameters separately, for camera calibration.SOLUTION: A camera calibration apparatus 1 includes: a projection unit 11 which projects feature point global coordinates on image coordinates, using camera parameters; a projection error calculation unit 12 which calculates an error between the feature point image coordinates and projection coordinates projected by the projection unit 11; a camera parameter update unit 13 which updates the camera parameters so as to reduce the error; a feature point global coordinate update unit 14 which updates the feature point global coordinates so as to reduce the error; an operation control unit 10 which controls operation by switching between the camera parameter update unit 13 and the feature point global coordinate update unit 14; and an image synthesis unit 16 which synthesizes the feature point image coordinates and information indicating a position of the projection coordinates on a captured image to be displayed.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a camera calibration device and a program for calibrating a camera. [Background technology]

[0002] Conventionally, a method for calibrating a camera uses an image with a known pattern shape or the like, and estimates parameters (camera parameters) related to the camera's attitude, position, and angle of view based on the image and geometric features (vanishing point, etc.) obtained from the image (see, for example, Patent Document 1). Pattern shapes or the like include chessboard patterns, AR (augmented reality) markers, two-dimensional barcodes, lines on sports stadiums, and three-dimensional charts (see, for example, Patent Document 2). There is also a SLAM (Simultaneous Localization and Mapping) technology that simultaneously obtains the three-dimensional arrangement of feature points and camera parameters by associating feature points contained in natural images captured by multiple cameras (see, for example, Patent Document 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4743538 [Patent Document 2] Patent No. 4590780 [Patent Document 3] Patent Publication No. 2021-81989 Summary of the Invention [Problem to be solved by the invention]

[0004] There are two types of methods for calibrating a camera using an image of a known pattern shape (hereinafter simply referred to as a pattern): one in which the pattern is detected manually by visual confirmation, and one in which the pattern is detected automatically. The method of manually detecting patterns and calibrating the camera has the problem that pattern detection is time-consuming. On the other hand, the method of automatically detecting patterns and calibrating the camera may contain unintended errors in the pattern detection. Furthermore, since this method involves an automatic process of deriving camera parameters from the pattern detection results, it is difficult to adjust for errors in the pattern detection.

[0005] In particular, with SLAM technology, the processes of extracting feature points, eliminating errors, and deriving camera parameters are all performed automatically. Therefore, although SLAM technology can intervene in the number of feature points and parameters for eliminating errors, the pattern detection results are not necessarily correct, and accurate camera parameters cannot necessarily be derived. As described above, it is difficult to perform highly accurate camera calibration using conventional techniques.

[0006] The present invention has been made in consideration of the above problems, and an object of the present invention is to provide a camera calibration device and a program therefor that are capable of performing highly accurate camera calibration by individually correcting the positions of feature points and camera parameters. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the camera calibration device according to the present invention is a camera calibration device that estimates camera parameters of a camera, and is configured to include a projection unit, a projection error calculation unit, a camera parameter update unit, a feature point world coordinate update unit, an operation control unit, and an image synthesis unit.

[0008] In such a configuration, the camera calibration device uses the camera parameters to project, by the projection unit, the feature point world coordinates, which are the positions of the feature points in world coordinates, onto the image coordinates. The camera calibration device then uses a projection error calculation unit to calculate the error between the projection coordinates projected by the projection unit and the feature point image coordinates, which are the positions of the feature points on the captured image taken by the camera. This allows the error on the image of the feature points, which occurs due to a deviation of either or both of the camera parameters and the feature point world coordinates, to be calculated.

[0009] Then, the camera calibration device updates the camera parameters in a direction that reduces the error using the camera parameter updater. Here, the camera parameter update unit updates the camera parameters based on the error and a Jacobian that represents the amount of change in each component of the error relative to the amount of change in each component of the camera parameter. Furthermore, the camera calibration device updates the feature point world coordinates in a direction that reduces the error using the feature point world coordinate update unit. Here, the feature point world coordinate updating unit updates the feature point world coordinates based on the error and a Jacobian that represents the amount of change in each component of the error relative to the amount of change in each component of the feature point world coordinates. The camera calibration device switches between operating the camera parameter update unit and the feature point world coordinate update unit using the operation control unit, thereby allowing the camera calibration device to update the camera parameters and the world coordinates of the feature points independently.

[0010] Furthermore, the camera calibration device uses an image synthesis unit to synthesize and display information indicating the positions of the feature point image coordinates and projection coordinates on the captured image, thereby allowing the user to visually check the positions of the feature points on the captured image and errors in the feature point image coordinates and projection coordinates. Then, if the user does not reduce the error in the position of the feature point by updating using the camera parameter updating unit, the user performs an update using the feature point world coordinate updating unit, and if the user does not reduce the error in the position of the feature point by updating using the feature point world coordinate updating unit, the user performs an update using the camera parameter updating unit, and the camera calibration device can efficiently calibrate the camera parameters and the feature point world coordinates with high accuracy. The camera calibration device can be operated by a camera calibration program that causes a computer to function as each of the above-mentioned units. [Effects of the Invention]

[0011] According to the present invention, the camera parameters and the world coordinates of the feature points can be selectively updated individually, thereby enabling efficient camera calibration. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an overall configuration diagram showing the configuration of a camera calibration system including a camera calibration device according to an embodiment of the present invention. [Figure 2] FIG. 10 is an explanatory diagram for explaining the relationship between the world coordinates of a feature point and the image coordinates and camera parameters of a camera. [Figure 3] 1 is a block diagram showing the configuration of a camera calibration device according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram showing an example of a control screen displayed by the camera calibration device according to the embodiment of the present invention. [Figure 5] 3 is a flowchart showing the overall operation of the camera calibration device according to the embodiment of the present invention. [Figure 6] 6 is a flowchart showing the operation of the camera designation process in FIG. 5. [Figure 7] 6 is a flowchart showing the operation of the feature point designation process of FIG. 5. [Figure 8] 6 is a flowchart showing the operation of the one-camera optimization process in FIG. 5. [Figure 9] 6 is a flowchart showing the operation of the all-camera optimization process of FIG. 5. [Figure 10] 6 is a flowchart showing the operation of the one feature point optimization process in FIG. 5. [Figure 11] 6 is a flowchart showing the operation of the all feature point optimization process of FIG. 5. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Camera calibration system] The configuration of a camera calibration system including a camera calibration device according to an embodiment of the present invention will be described with reference to FIG.

[0014] The camera calibration system S performs camera calibration of the camera 2. Here, camera calibration is a process of estimating the camera parameters of the camera 2. As shown in FIG. 1, the camera calibration system S is composed of a camera calibration device 1 and one or more cameras 2 (21, 22, . . . ).

[0015] The camera calibration device 1 estimates the camera parameters of the camera 2 based on the feature point image coordinates, which are the positions of the feature points on the image captured by the camera 2, and the feature point world coordinates, which are the positions of the feature points in world coordinates. The configuration and operation of this camera calibration device 1 will be explained in detail later.

[0016] The camera 2 captures the image of the space. The camera 2 captures an image of a space including a subject to be photographed, from which feature points such as an object O and a pattern M can be selected and extracted. The subject O is a subject including feature points, such as an object, a person, etc. For example, if the subject O is a person, any position, such as the position of the fingers or the position of the eyes, becomes a feature point that can be selected and extracted. The pattern M is a pattern whose shape, etc., is known. For example, the pattern M may be a general pattern such as a pattern to which identification information, such as a QR code (registered trademark) or an ArUco marker, or a chessboard pattern (checkered grid). The camera 2 outputs a captured image to the camera calibration device 1 online or offline.

[0017] Now, with reference to FIG. 2, an example of camera parameters used in the present invention will be described. FIG. 2 shows the relationship between the coordinate systems, with one camera 2 modeled as a pinhole camera model. As shown in Figure 2, all cameras 2 share a common coordinate system (X W ,Y W ,Z W ) are world coordinates, and each camera has its own coordinate system (XC ,Y C ,Z C ) are the camera coordinates. Here, the point that passes through the optical principal point T and intersects perpendicularly with the imaging surface (image surface) I of the camera 2 is the image coordinate at the center of the image coordinate system (u, v). The distance between the optical principal point T and the imaging surface I at this time is the focal length f. The optical principal point T is located in the world coordinate system (T X ,T Y ,T Z ) Also, when the world coordinates are used as the reference, the camera coordinates are X and W δ (elevation angle: tilt) with respect to the axis, Y W α (azimuth angle: pan) and Z W It rotates by φ (roll angle) around the axis.

[0018] In this context, the camera parameters include the camera position, camera attitude, and field of view information of the camera 2. The camera position is the world coordinate of the optical principal point T. The camera attitude is the three-axis attitude angle (tilt, pan, roll) or rotation matrix that defines the attitude of the imaging surface I relative to the world coordinate. The angle of view information is the focal length f of the lens or pinhole attached to the camera 2, or the angle of view determined by the focal length and the size of the imaging element. The camera parameters may also include a parameter representing distortion.

[0019] 2, Q denotes the world coordinate of the feature point (feature point world coordinate), and q denotes the projection coordinate of the feature point of world coordinate Q projected onto the imaging plane (image plane) I of the camera 2. Here, let c∈{1, 2, ..., C} be the identifiers (camera identifiers) that identify C cameras 2 (C is an integer equal to or greater than 1). The pixel value at the projection coordinate q of the image captured by the camera identifier c is expressed as I (c) The pixel values ​​may be scalar values ​​(for example, in the case of a monochrome image) or vector values ​​(for example, in the case of a color image or a multispectral image).

[0020] Also, the camera parameters of camera 2 with camera identifier c are p(c) It is written as follows. This camera parameter p (c) As shown in the following equation (1), the feature point world coordinate Q is converted to the projected coordinate q on the image plane of camera 2 with camera identifier c using the function h for projective transformation. (c) It is assumed that correspondence can be made between the following:

[0021]

number

[0022] In the case of perspective projection, this function h may be formulated as in the following equation (2).

[0023]

number

[0024] Note that R(α,δ,φ) represents a rotation matrix for the attitude angles of the three axes specified by α (pan), δ (tilt), and φ (roll). The configuration and operation of the camera calibration device 1 will be described below.

[0025] [Configuration of camera calibration device] The configuration of a camera calibration device 1 according to an embodiment of the present invention will be described with reference to FIG. Here, the camera calibration device 1 is connected to a display device 3 and an input device 4. The display device 3 and the input device 4 may be integrated with the camera calibration device 1.

[0026] The display device 3 displays an image obtained by combining the captured image taken by the camera 2 to be calibrated with information indicating the positions of the feature point image coordinates indicating the positions of the feature points on the captured image and the projection coordinates obtained by projecting the feature points on the world coordinates onto the image plane of the camera 2 to be calibrated. The display device 3 may be any type, such as a liquid crystal display, a cathode ray tube, an organic electroluminescence display, a plasma display, a light emitting diode display, a projector, or a tactile display.

[0027] The input device 4 is used by the user to input coordinates of feature points and instructions to the camera calibration device 1. The input device 4 may be of any type or combination, such as a switch, keyboard, mouse, pen tablet, touch panel, trackball, light pen, digitizer, joystick, pointing device, data glove, inertial sensor, voice input device, etc. Also, a liquid crystal pen tablet or a display with a touch panel may be used as both the display device 3 and the input device 4. Alternatively, buttons, switches, menus, etc. may be arranged as images on the screen of the display device 3, and the coordinates of feature points or instructions may be given by virtually pressing and operating these using a device of the input device 4. As shown in FIG. 3, the camera calibration device 1 includes a control unit CL and a storage unit ME.

[0028] The control unit CL controls the overall operation of the camera calibration device 1. The control unit CL operates when a camera calibration program stored in a storage medium such as a ROM (not shown) is loaded into a RAM, for example. The control unit CL includes, as functional blocks, an operation control unit 10, a projection unit 11, a projection error calculation unit 12, a camera parameter update unit 13, a feature point world coordinate update unit 14, a feature point extraction unit 15, and an image synthesis unit 16.

[0029] The operation control unit 10 displays a control screen on the display device 3 and provides a user interface for performing camera calibration based on a user instruction input via the input device 4. The operation control unit 10 includes a control screen generation unit 100, a camera selection unit 101, a feature point selection unit 102, a camera parameter update instruction unit 103, and a feature point world coordinate update instruction unit 104.

[0030] The control screen generating unit 100 generates a control screen that provides a user interface for performing camera calibration based on a user instruction input via the input device 4. Here, the control screen includes an area for inputting instructions to the camera calibration device 1 and an area for displaying an image captured by the camera 2 to be calibrated. The area for displaying the image on the control screen is generated by the image synthesis unit 16, which will be described later. Here, an example of the control screen SC generated by the control screen generating unit 100 will be described with reference to FIG. The control screen SC is made up of a toolbar 110 and a viewer 120 .

[0031] The toolbar 110 is an area for inputting instructions to the camera calibration device 1. Here, the toolbar 110 is made up of a camera specification field 111, a feature point specification field 112, buttons for instructing optimization of camera parameters (optimize camera parameters button 113, optimize all camera parameters button 114), and buttons for instructing optimization of the world coordinates of feature points (optimize feature point world coordinates button 115, optimize all feature points world coordinates button 116).

[0032] The camera specification field 111 is an input field for specifying the camera 2 to be calibrated to optimize the camera parameters. In this camera specification field 111, a predetermined identifier (camera identifier) ​​for individually identifying each of the multiple cameras 2 is input. The feature point specification field 112 is an input field for specifying feature points to be calibrated for optimizing the feature point world coordinates. Predetermined identifiers (feature point identifiers) for identifying multiple feature points are input into this feature point specification field 112.

[0033] The camera parameter optimization button 113 is a button for receiving an instruction to optimize the camera parameters of the camera 2 designated in the camera designation field 111 . The all camera parameter optimization button 114 is a button for receiving an instruction to optimize the camera parameters for all cameras 2 (21, 22, . . . ).

[0034] The feature point world coordinate optimization button 115 is a button for receiving an instruction to optimize the world coordinates of the feature points designated in the feature point designation field 112 . The all feature point world coordinate optimization button 116 is a button for receiving an instruction to optimize the world coordinates of all feature points.

[0035] The viewer 120 is an area that displays images taken by the camera 2 specified in the camera specification field 111 . The viewer 120 displays a targeting graphic (first targeting graphic) superimposed on the image coordinate position of the feature point of the camera 2 to be calibrated, which is stored in the feature point image coordinate storage unit 18 described below, as information indicating the position. In addition, the viewer 120 superimposes an aiming figure (second aiming figure) as information indicating the position at the projection coordinate position where the projection unit 11 projects the world coordinates of the feature points stored in the feature point world coordinate memory unit 20 described below onto the image coordinates of the camera 2 to be calibrated.

[0036] The aiming figures (first aiming figure, second aiming figure) may be any figure that allows for identifying a position. For example, a crosshair, an X mark, etc. may be used. In addition, an identifier (feature point identifier) ​​for distinguishing between multiple feature points may be presented in the vicinity of the aiming figure using visual information such as a figure, letter, symbol, color, etc. Note that the visual information of the feature point identifier is preferably visualized using a specific figure, letter, symbol, color, etc., to make it clear that it is an aiming figure and not a captured image.

[0037] In the example of Fig. 4, crosshairs 121 are a symbol for aiming to indicate image coordinates. Also, for example, identification mark 122 (a ◇ mark with "12" displayed inside) indicates a feature point identifier by its number, and the surrounding symbol (◇ mark) indicates that the first aiming symbol (crosshairs) visualizes the image coordinates. Additionally, the cross mark 123 is a symbol for aiming to indicate the projected coordinates. For example, the identification mark 124 (a circle with the number "12" displayed inside) indicates the feature point identifier by its number, and the surrounding symbol (circle) indicates that the second aiming symbol (cross mark) visualizes the projected coordinates. This viewer 120 allows the user to visually check the error between the image coordinates and the projection coordinates for the same feature point. Returning to Fig. 3, we will continue to explain the configuration of the camera calibration device 1. Note that Fig. 4 will be referred to as appropriate for the control screen.

[0038] The control screen generating unit 100 instructs the image combining unit 16 to display the control screen SC (excluding the contents of the viewer 120). The camera selection unit 101 accepts the selection of the camera 2 to be calibrated. Here, the camera selection unit 101 acquires, as a camera identifier, a numerical value inputted into the camera specification field 111 of the control screen SC via the input device 4. The camera selection unit 101 outputs the acquired camera identifier to the camera parameter update instruction unit 103 .

[0039] The feature point selection unit 102 receives the selection of feature points to be calibrated. Here, the feature point selection unit 102 acquires, as a feature point identifier, a numerical value inputted into the feature point specification field 112 of the control screen SC via the input device 4. The feature point selection unit 102 outputs the acquired feature point identifier to the feature point world coordinate update instruction unit 104 .

[0040] The camera parameter update instruction unit 103 receives an instruction to update the camera parameters. Here, the camera parameter update instruction unit 103 receives an instruction to update the camera parameters when the camera parameter optimization button 113 or the all camera parameter optimization button 114 on the control screen SC is pressed. When the camera parameter optimization button 113 is pressed, the camera parameter update instruction unit 103 instructs the camera parameter update unit 13 to update the camera parameters of the camera 2 having the camera identifier selected by the camera selection unit 101.

[0041] When the optimize all camera parameters button 114 is pressed, the camera parameter update instruction unit 103 instructs the camera parameter update unit 13 to optimize the camera parameters for all cameras 2. In this case, the camera parameter update instruction unit 103 is free to decide which cameras 2 to update and in what order, and may operate the camera parameter update units 13 in parallel for multiple cameras 2.

[0042] The feature point world coordinate update instruction unit 104 receives an instruction to update the feature point world coordinates. Here, the feature point world coordinate update instruction unit 104 receives an instruction to update the feature point world coordinates when the feature point world coordinate optimization button 115 or the all feature point world coordinate optimization button 116 on the control screen SC is pressed. When the feature point world coordinate optimization button 115 is pressed, the feature point world coordinate update instruction unit 104 instructs the feature point world coordinate update unit 14 to update the feature point world coordinates of the feature point identifier selected by the feature point selection unit 102.

[0043] When the all feature point world coordinate optimization button 116 is pressed, the feature point world coordinate update instruction unit 104 instructs the feature point world coordinate updating unit 14 to update the feature point world coordinates for all feature points. In this case, the feature point world coordinate update instruction unit 104 is free to decide which feature point world coordinates to update and in what order, and the feature point world coordinate update instruction unit 104 may operate in parallel for a plurality of feature point world coordinates.

[0044] In this way, when the operation control unit 10 is instructed to update the camera parameters of the selected camera 2, the camera parameter update unit 13 updates the camera parameters of only the selected camera. Furthermore, when the operation control unit 10 is instructed to update the camera parameters of all the cameras 2, the camera parameter update unit 13 updates the camera parameters of all the cameras 2. Furthermore, when the operation control unit 10 is instructed to update the feature point world coordinates of the selected feature points, the feature point world coordinate update unit 14 updates the feature point world coordinates of only the selected feature points. Furthermore, when the operation control unit 10 is instructed to update the world feature point coordinates of all feature points, the world feature point coordinate update unit 14 updates the world feature point coordinates of all feature points.

[0045] The operation control unit 10 may instruct the feature point extraction unit 15 to detect feature points at the timing when a user gives an instruction via the input device 4. For example, when a position on an image is selected by mouse operation on the input device 4, the operation control unit 10 instructs the feature point extraction unit 15 to detect feature points along with the selected image coordinates.

[0046] The projection unit 11 projects the feature point world coordinates onto the image coordinates using the camera parameters. Here, the projection unit 11 calculates projected coordinates obtained by projecting the feature point world coordinates stored in the feature point world coordinate storage unit 20 onto the image plane of the camera 2 using the camera parameters for each camera 2 stored in the camera parameter storage unit 19. Specifically, the projection unit 11 projects the camera parameters p (c) Using this, we can calculate the world coordinates Q of N feature point identifiers n (n∈{1,2,…,N}). n For example, the projection coordinates h(p (c) ,Q n ) is calculated.

[0047] When the camera parameter update instruction unit 103 instructs the projection unit 11 to specify the camera 2 and calculate the projection coordinates, the projection unit 11 calculates the projection coordinates h(p (c) ,Q n) is calculated. Furthermore, when the feature point world coordinate update instruction unit 104 instructs the projection unit 11 to specify a feature point and calculate projected coordinates, the projection unit 11 calculates projected coordinates h(p (c) ,Q n ) is calculated. The projection unit 11 outputs the calculated projection coordinates to the projection error calculation unit 12 and the image synthesis unit 16.

[0048] The projection error calculation unit 12 calculates the error between the projection coordinates projected by the projection unit 11 and the image coordinates (feature point image coordinates) of the feature points for each camera 2 stored in the feature point image coordinate storage unit 18 as an error evaluation value. The projection error calculation unit 12 calculates the projection coordinates h(p (c) ,Q n ) and image coordinates q of the feature point having the feature point identifier n in the image captured by the camera 2 having the camera identifier c stored in the feature point image coordinate storage unit 18. n (c) The difference between n (c) It is calculated as follows.

[0049]

number

[0050] The projection error calculation unit 12 outputs an error evaluation value E to the camera parameter update unit 13 or the feature point world coordinate update unit 14 that has instructed the projection unit 11 to calculate the projection coordinates. n (c) Output.

[0051] The camera parameter update unit 13 updates the camera parameters in a direction that reduces the error calculated by the projection error calculation unit 12 . The camera parameter update unit 13 updates the camera parameters based on an instruction from the camera parameter update instruction unit 103 of the operation control unit 10. Here, the camera parameter update unit 13 includes a gradient calculation unit 130 and a camera parameter optimization unit 131.

[0052] The gradient calculation unit 130 calculates the camera parameters p (c) (camera parameters of the camera with the camera identifier c) and the feature point world coordinates Q stored in the feature point world coordinate storage unit 20 n (world coordinates of the feature point of feature point identifier n), the gradient of the error evaluation value of the feature point world coordinates corresponding to one camera 2 (camera identifier c) is calculated. Specifically, the gradient calculation unit 130 calculates the gradient as a Jacobian G using the following equation (4).

[0053]

number

[0054] The error evaluation value E n (c) When (cε{1, 2, . . . , C}, nε{1, 2, . . . , N}) is the above-mentioned formula (3), the gradient calculation unit 130 can analytically obtain the Jacobian G by the following formula (5).

[0055]

number

[0056] The gradient calculation unit 130 also calculates a sufficiently small real constant δ k (For example, camera parameters p (c) The Jacobian G may be approximately calculated by the following equation (6) using a non-zero constant (which is equal to or smaller than 1 / 100 of the k-th component in

[0057]

number

[0058] In this equation (6), the camera parameter p(c) Let e ​​be K-dimensional (K is a natural number), where k is an integer satisfying k∈{1, 2, …, K}. k is a K-dimensional column vector with the k-th component being "1" and the other components being "0". Furthermore, it is assumed that the real numbers a and b are a ≠ b. For example, the real numbers a and b can be (a, b) = (1, 0), (a, b) = (0, -1), (a, b) = (0.5, -0.5), etc. The gradient calculation unit 130 outputs the gradient (Jacobian) of the error evaluation value of the feature point identifier corresponding to one camera 2 (camera identifier c) to the camera parameter optimization unit 131.

[0059] The camera parameter optimization unit 131 calculates the error evaluation value E calculated by the projection error calculation unit 12. n (c) the Jacobian G calculated by the gradient calculation unit 130 and the camera parameters p (c) The camera parameter optimization unit 131 updates the camera parameters based on the above. The camera parameter optimization unit 131 updates the updated camera parameters as camera parameters p (c) and repeating this a predetermined number of times to update the camera parameters.

[0060] For updating the camera parameters, for example, the Gauss-Newton method or the Levenberg-Marquardt method can be used. When the Gauss-Newton method is used, the camera parameter optimization unit 131 calculates the updated camera parameters p (c)′ can be obtained.

[0061]

number

[0062] Furthermore, when the Levenberg-Marquardt method is used, the camera parameter optimization unit 131 calculates the updated camera parameters p (c)′ can be obtained.

[0063]

number

[0064] I denotes the identity matrix. Also, λ is a real number equal to or greater than 0, and the value of λ at the mth update (m is an integer equal to or greater than 0) is λ m Then, the camera parameter optimization unit 131 divides the cases into two categories using the following equation (9) to determine λ m Determine.

[0065]

number

[0066] Here, an arbitrary λ0≧0 is set in advance as an initial value, and v is an arbitrary real constant v>1. Also, Φ(λ) and f(p (c) ) are defined as the following equations (10) and (11), respectively.

[0067]

number

[0068]

number

[0069] The camera parameter optimization unit 131 calculates λ by the above equation (9) when m≧1. m is determined, and the updated camera parameters p (c)′ Ask for. This allows the camera parameter optimization unit 131 to obtain camera parameters optimized for the feature point world coordinates. The camera parameter optimization unit 131 stores the updated camera parameters in the camera parameter storage unit 19 .

[0070] The feature point world coordinate updating unit 14 updates the feature point world coordinates in a direction that reduces the error calculated by the projection error calculating unit 12 . The feature point world coordinate update unit 14 updates the feature point world coordinates based on an instruction from the feature point world coordinate update instruction unit 104 of the operation control unit 10. Here, the feature point world coordinate updating unit 14 includes a gradient calculation unit 140 and a feature point world coordinate optimization unit 141.

[0071] The gradient calculation unit 140 calculates the camera parameters p (c) (camera parameters of the camera with the camera identifier c) and the feature point world coordinates Q stored in the feature point world coordinate storage unit 20 n (world coordinates of the feature point of feature point identifier n), the gradient of the error evaluation value for each camera 2 corresponding to one feature point is calculated. Specifically, the gradient calculation unit 140 calculates the gradient as a Jacobian Γ using the following equation (12).

[0072]

number

[0073] The error evaluation value E n (c) When (c∈{1, 2, . . . , C}, n∈{1, 2, . . . , N}) is the above-mentioned formula (3), the gradient calculation unit 140 can analytically obtain the Jacobian Γ by the following formula (13).

[0074]

number

[0075] The gradient calculation unit 140 also calculates a sufficiently small real constant ε j (For example, the feature point world coordinate Q n The Jacobian Γ may be calculated approximately by the following equation (14) using a non-zero constant (1 / 100 or less of the j-th component of

[0076]

number

[0077] In this equation (14), the feature point world coordinate Q n is J-dimensional (J is a natural number (usually J=3)), and the integer j satisfies j∈{1,2,…,J}. Also, e j is a J-dimensional column vector in which the j-th component is "1" and the other components are "0." The real numbers a and b are the same as those explained in the above equation (6). The gradient calculation unit 140 outputs the gradient (Jacobian) of the error evaluation value for each camera 2 corresponding to one feature point to the feature point world coordinate optimization unit 141.

[0078] The feature point world coordinate optimization unit 141 calculates the error evaluation value E n (c) the Jacobian Γ calculated by the gradient calculation unit 140 and the feature point world coordinates Q stored in the feature point world coordinate storage unit 20. n The feature point world coordinates are updated based on the above. The feature point world coordinates optimization unit 141 updates the feature point world coordinates after the update as the feature point world coordinates Q n and repeating this a predetermined number of times to update the feature point world coordinates.

[0079] For updating the camera parameters, the Gauss-Newton method or the Levenberg-Marquardt method can be used, as in the case of updating the camera parameters. When the Gauss-Newton method is used, the feature point world coordinate optimization unit 141 calculates the updated feature point world coordinate Q by the following equation (15): n ' can be obtained.

[0080]

number

[0081] Furthermore, when the Levenberg-Marquardt method is used, the feature point world coordinate optimization unit 141 calculates the updated feature point world coordinate Q by the following equation (16): n ' can be obtained.

[0082]

number

[0083] I denotes the identity matrix. Also, μ is a real number equal to or greater than 0, and the value of μ at the rth update (r is an integer equal to or greater than 0) is μ r Then, the feature point world coordinate optimization unit 141 divides the cases into two groups according to the following equation (17): r Determine.

[0084]

number

[0085] Here, an arbitrary μ0≧0 is set in advance as an initial value, and τ is an arbitrary real constant τ>1. Also, Ψ(μ) and φ(Q n ) are defined as the following equations (18) and (19), respectively.

[0086]

number

[0087]

number

[0088] The feature point world coordinate optimization unit 141 calculates μ by the above formula (17) for r≧1. r is determined, and the updated feature point world coordinate Q is calculated using the above equation (16). n ' is sought. This allows the feature point world coordinate optimization unit 141 to obtain feature point world coordinates optimized for the camera parameters. The feature point world coordinate optimization unit 141 stores the updated feature point world coordinates in the feature point world coordinate storage unit 20 .

[0089] The feature point extraction unit 15 extracts feature points from the images captured by the camera 2 and stored in the captured image storage unit 17 . The feature point extraction unit 15 may detect feature points in response to an instruction from the operation control unit 10, or may detect feature points by detecting that a captured image has been stored in the captured image storage unit 17. The feature points detected by the feature point extraction unit 15 may be predetermined positions of a predetermined image pattern, or may be positions of light-emitting bodies that emit light with a predetermined time change (for example, blinking or color change) in the sequentially input photographed images. Alternatively, the feature points may be image features such as corner points. When detecting a pattern to which identification information has been added, the feature point extraction unit 15 can acquire the image position and the identification information by using an existing general method.

[0090] Furthermore, when detecting a predetermined pattern that does not have identification information, such as a chessboard pattern, the feature point extraction unit 15 may obtain the image coordinates by a semi-automatic method, such as extracting a predetermined pattern that is closest to the image coordinates specified by operating the mouse of the input device 4 as the image coordinates input from the operation control unit 10. For example, when detecting a chessboard pattern, the feature point extraction unit 15 searches for a location where the local Gaussian curvature of the two-dimensional luminance value sequence of the image is minimum in the vicinity of the mouse press coordinates (for example, an area within a predetermined radius), and detects the point where two colored quadrangles that make up the pattern meet as the position of the image coordinates of the feature point.

[0091] The feature point extraction unit 15 stores the positions (image coordinates) of the extracted feature points in the feature point image coordinate storage unit 18. When storing image coordinates in the feature point image coordinate storage unit 18, the feature point extraction unit 15 may add and store them as new feature points, or may overwrite and store them as new image coordinates of existing feature points. When overwriting and storing them, the feature point extraction unit 15 may determine that feature points with the same identification information are to be overwritten, or that feature points with close image coordinates are to be overwritten.

[0092] The image synthesis unit 16 synthesizes and displays information indicating the positions of the feature point image coordinates and projection coordinates on the captured image. Here, the image synthesis unit 16 reads out the captured image corresponding to the camera identifier of the camera 2 selected by the camera selection unit 101 from the captured image storage unit 17, reads out the feature point image coordinates extracted from the captured image from the feature point image coordinate storage unit 18, and superimposes the first aiming figure at the position of the feature point image coordinates of the captured image. Furthermore, the image synthesis unit 16 superimposes a second aiming figure at the position of the projection coordinates where the feature point world coordinates are projected onto the image coordinates via the projection unit 11.

[0093] The image synthesis unit 16 outputs a control screen including a captured image on which a targeting graphic is superimposed to the display device 3. Of course, the image synthesis unit 16 may also display on the display device 3 a mouse cursor for operation controlled via the operation control unit 10, or the operating state of a user interface (for example, the state of a button being pressed).

[0094] The storage unit ME stores various data used in the camera calibration device 1. This storage unit ME is a rewritable memory or medium that stores data, and the method of realizing it does not matter. Here, the memory unit ME includes a captured image memory unit 17, a feature point image coordinate memory unit 18, a camera parameter memory unit 19, and a feature point world coordinate memory unit 20. Note that the memory units (17, 18, 19, 20) may store data in separate areas on the same storage medium, or may store data in different storage media.

[0095] The captured image storage unit 17 stores the captured images captured by the camera 2. The captured image storage unit 17 stores the captured images in association with the camera identifier of the camera 2. The feature point image coordinate storage unit 18 stores image coordinates of feature points extracted by the feature point extraction unit 15 in correspondence with the captured image stored in the captured image storage unit 17. Note that in addition to the feature point image coordinates extracted by the feature point extraction unit 15, the feature point image coordinates and feature point identifiers may be input and stored in the feature point image coordinate storage unit 18 by operating the input device 4 (for example, pressing the mouse, entering from the keyboard, etc.).

[0096] The camera parameter storage unit 19 stores the camera parameters of the camera 2 when the captured image stored in the captured image storage unit 17 was captured. The camera parameters may be omitted if a constant parameter is used, such as when a lens with a fixed focal length is used. The camera parameters may also include a parameter representing distortion, but if the distortion is sufficiently small, the value of that parameter may be omitted. The camera parameters to be stored in the camera parameter storage unit 19 may be set in advance as initial values ​​for each camera 2, or may be directly input and stored by operating the input device 4 (for example, by pressing the mouse, entering data from the keyboard, etc.).

[0097] The feature point world coordinate storage unit 20 stores the world coordinates of feature points (feature point world coordinates). The feature point world coordinates to be stored in the feature point world coordinate storage unit 20 may have initial values ​​set in advance for each feature point identifier, or the feature point world coordinates and feature point identifiers may be directly input and stored by operating the input device 4 (for example, by pressing the mouse or entering data from the keyboard). As the initial value of the feature point world coordinate, set an appropriate coordinate value (e.g., [0,0,0] T ) may be associated with all feature point identifiers. With the above-described configuration, the camera calibration device 1 allows the user to independently optimize the positions of feature points and the camera parameters.

[0098] As a result, when the first aiming figure (crosshairs 121) indicating the image coordinates of the feature points and the second aiming figure (x mark 123) indicating the projected coordinates of the feature points projected onto the world coordinates in the viewer 120 of the control screen SC shown in FIG. 4 sufficiently match (for example, when the error between the image coordinates and the projected coordinates for a predetermined number of feature points or more is within a predetermined value), it can be determined that the camera calibration is complete.

[0099] The user can determine whether camera calibration is complete while viewing the viewer 120. Alternatively, the completion of camera calibration may be determined automatically by determining whether the errors between the image coordinates and the projection coordinates for a predetermined number or more of feature points are within a predetermined value. In this case, for example, when optimizing all the feature point world coordinates, the feature point world coordinate update instruction unit 104 acquires error evaluation values ​​from the feature point world coordinate update unit 14, and determines whether the error evaluation values ​​for a predetermined number or more of feature points are within the predetermined value.

[0100] In this way, the camera calibration device 1 can independently optimize the positions of feature points when deriving camera parameters, and therefore can perform highly accurate camera calibration in a shorter time than conventional methods. The camera calibration device 1 can be operated by a program (camera calibration program) that causes a computer to function as each of the above-mentioned units.

[0101] [Camera calibration device in action] Next, with reference to FIGS. 5 to 11 (and also with reference to FIGS. 3 and 4 as appropriate), the main operations of the camera calibration device 1 according to the embodiment of the present invention will be described. It is assumed that the captured image storage unit 17 stores captured images captured by the camera 2 in association with camera identifiers c (c∈{1, 2, ..., C}). It is also assumed that the feature point image coordinate storage unit 18 stores image coordinates of feature points in the captured images in association with feature point identifiers n (n∈{1, 2, ..., N}). It is also assumed that the camera parameter storage unit 19 stores camera parameters of the camera 2 when the captured images were captured in association with camera identifiers c (c∈{1, 2, ..., C}). It is also assumed that the feature point world coordinate storage unit 20 has an initial value of world coordinates set for each feature point identifier n (n∈{1, 2, ..., N}).

[0102] In step S1, the operation control unit 10 determines the user's instruction input from the input device 4 and branches the process. When the camera specification field 111 on the control screen SC is selected (camera specification), the camera calibration device 1 advances the operation to step S2. When the feature point specification field 112 on the control screen SC is selected (feature point specification), the camera calibration device 1 advances the operation to step S3. When the camera parameter optimization button 113 is pressed on the control screen SC (one camera optimization), the camera calibration device 1 advances the operation to step S4. When the optimize all camera parameters button 114 is pressed on the control screen SC (optimize all cameras), the camera calibration device 1 advances the operation to step S5. When the feature point world coordinate optimization button 115 is pressed on the control screen SC (one feature point optimization), the camera calibration device 1 advances the operation to step S6. When the all feature point world coordinate optimization button 116 is pressed on the control screen SC (all feature point optimization), the camera calibration device 1 advances the operation to step S7. The operation of each step will be explained below.

[0103] (Camera designation processing) As shown in FIG. 6, in the camera designation process of step S2, in step S21, a camera identifier c that designates the camera 2 whose camera parameters are to be calibrated and is input into the camera designation field 111 is read and stored in a memory not shown.

[0104] In step S22, the image synthesis unit 16 reads out the captured image of the camera identifier c input in step S22 from the captured image storage unit 17 and displays it on the viewer 120. In step S23, the image synthesis unit 16 superimposes the first aiming figure at the position of the feature point image coordinates of the feature point image stored in the feature point image coordinate storage unit 18 corresponding to the camera identifier c. In step S24, the projection unit 11 projects all of the feature point world coordinates stored in the feature point world coordinate storage unit 20 onto image coordinates, using the camera parameters stored in the camera parameter storage unit 19 corresponding to the camera identifier c input in step S22.

[0105] In step S25, the image synthesis unit 16 superimposes the second aiming figure at the position of the coordinates (projected coordinates) projected in step S24. This allows the camera calibration device 1 to superimpose the aiming figure on the image captured by the designated camera 2, and visually present the deviation between the first aiming figure and the second aiming figure. After this, the camera calibration device 1 returns to the operation at step S1 (FIG. 5).

[0106] (Feature point specification processing) As shown in FIG. 7, in the feature point designation process of step S3, in step S31, a feature point identifier n that designates a feature point to be calibrated in the feature point world coordinates, which is input into the feature point designation field 112, is read and stored in a memory (not shown). After this, the camera calibration device 1 returns to the operation at step S1 (FIG. 5).

[0107] (1 camera optimization process) As shown in FIG. 8, in the one-camera optimization process of step S4, in step S41, the projection unit 11 projects all of the feature point world coordinates stored in the feature point world coordinate storage unit 20 onto image coordinates, using the camera parameters of the camera identifier c designated in step S2 (FIG. 5) that are stored in the camera parameter storage unit 19.

[0108] In step S42, the projection error calculation unit 12 calculates an error evaluation value from the feature point image coordinates corresponding to the camera identifier c stored in the feature point image coordinate storage unit 18 and the projection coordinates projected in step S41. In step S43, the gradient calculation unit 130 of the camera parameter update unit 13 calculates the gradients (Jacobian) of the error evaluation values ​​of all the feature point world coordinates calculated in step S42 corresponding to the camera 2 with the camera identifier c, based on the camera parameters of the camera identifier c stored in the camera parameter storage unit 19 and the feature point world coordinates stored in the feature point world coordinate storage unit 20.

[0109] In step S44, the camera parameter optimization unit 131 updates the camera parameters of the camera identifier c stored in the camera parameter storage unit 19, and stores the updated camera parameters in the camera parameter storage unit 19. Here, the camera parameter optimization unit 131 updates the camera parameters using the Gauss-Newton method, the Levenberg-Marquardt method, or the like, based on the error evaluation value calculated in step S42, the gradient calculated in step S43, and the camera parameters stored in the camera parameter storage unit 19. Note that steps S41 to S44 may be repeated a predetermined number of times to update the camera parameters.

[0110] In step S45, the image composition unit 16 superimposes and displays the aiming figures (first aiming figure, second aiming figure) of the feature points on the captured image taken by the camera with the specified camera identifier c. Note that the operation of this step S45 is the same as the operation of steps S22 to S25 in FIG. 6. This allows the camera calibration device 1 to determine camera parameters optimized for the feature point world coordinates, and to superimpose an aiming figure on the image captured by the specified camera 2, thereby visually presenting the deviation between the first aiming figure and the second aiming figure. After this, the camera calibration device 1 returns to the operation at step S1 (FIG. 5).

[0111] (All camera optimization processing) As shown in FIG. 9, in the all-camera optimization process of step S5, in step S51, the camera parameter update instruction unit 103 saves the camera identifier c specified in step S2 (FIG. 5) in a variable c0 and sets the initial value of c to “1.” The processing in steps S52 to S55 is the same as the processing in steps S41 to S44 described with reference to FIG.

[0112] In step S56, the camera parameter update instruction unit 103 determines whether the camera identifier c is the maximum value (C) or not, thereby determining whether the camera parameters of all cameras 2 have been updated. Here, if the camera identifier c has not reached the maximum value C (No in step S56), in step S57 the camera parameter update instruction unit 103 adds "1" to c, and returns to step S52 to continue the operation.

[0113] On the other hand, if the camera identifier c reaches the maximum value C (No in step S56), the image composition unit 16 superimposes and displays the aiming figures (first aiming figure, second aiming figure) of the feature points on the captured image taken by the camera with the camera identifier c0 that was retracted in step S51. Note that the operation of this step S58 is the same as the operation of steps S22 to S25 in FIG. 6, except that the camera identifier c is c0. This allows the camera calibration device 1 to determine all camera parameters optimized for the feature point world coordinates, and to superimpose an aiming figure on the image captured by the specified camera 2, thereby visually presenting the deviation between the first aiming figure and the second aiming figure. After this, the camera calibration device 1 returns to the operation at step S1 (FIG. 5).

[0114] (1 feature point optimization process) As shown in FIG. 10, in the one feature point optimization process of step S6, in step S61, the projection unit 11 projects the feature point world coordinates of the feature point identifier n designated in step S3 (FIG. 5) and stored in the feature point world coordinate storage unit 20 onto image coordinates for each camera 2, using the camera parameters stored in the camera parameter storage unit 19.

[0115] In step S62, the projection error calculation unit 12 calculates an error evaluation value from the feature point image coordinates corresponding to all camera identifiers stored in the feature point image coordinate storage unit 18 and the projection coordinates for each camera 2 projected in step S61. In step S63, the gradient calculation unit 140 of the feature point world coordinate update unit 14 calculates the gradient (Jacobian) of the error evaluation value for each camera 2 corresponding to the feature point identifier n calculated in step S62, based on the camera parameters corresponding to all camera identifiers stored in the camera parameter storage unit 19 and the feature point world coordinates corresponding to the feature point identifier n stored in the feature point world coordinate storage unit 20.

[0116] In step S64, the feature point world coordinate optimization unit 141 updates the feature point world coordinates of the feature point identifier n stored in the feature point world coordinate storage unit 20, and stores the updated feature point world coordinates in the feature point world coordinate storage unit 20. Here, the feature point world coordinate optimization unit 141 updates the feature point world coordinates using the Gauss-Newton method, the Levenberg-Marquardt method, or the like, based on the error evaluation value calculated in step S62, the gradient calculated in step S63, and the feature point world coordinates stored in the feature point world coordinate storage unit 20. Note that steps S61 to S64 may be repeated a predetermined number of times to update the feature point world coordinates.

[0117] In step S65, the image composition unit 16 superimposes and displays the aiming figures (first aiming figure, second aiming figure) of the feature points on the captured image taken by the camera with the specified camera identifier c. Note that the operation of this step S65 is the same as the operation of steps S22 to S25 in FIG. 6. This allows the camera calibration device 1 to determine the feature point world coordinates optimized for the camera parameters, and to superimpose an aiming figure on the image captured by the specified camera 2, thereby visually presenting the deviation between the first aiming figure and the second aiming figure. After this, the camera calibration device 1 returns to the operation at step S1 (FIG. 5).

[0118] (All feature point optimization processing) As shown in FIG. 11, in the all feature point optimization process of step S7, in step S71, the feature point world coordinate update instruction unit 104 sets the initial value “1” to the feature point identifier n specified in step S3 (FIG. 5). The processing in steps S72 to S75 is the same as the processing in steps S61 to S64 described with reference to FIG.

[0119] In step S76, the feature point world coordinate update instruction unit 104 determines whether the feature point identifier n is the maximum value (N) or not, thereby determining whether the world coordinates of all feature points have been updated. Here, if the feature point identifier n has not reached the maximum value N (No in step S76), in step S77 the feature point world coordinate update instruction unit 104 adds "1" to n, and returns to step S72 to continue the operation.

[0120] On the other hand, if the feature point identifier n reaches the maximum value N (No in step S76), the image composition unit 16 superimposes and displays the feature point aiming figures (first aiming figure, second aiming figure) on the image captured by the camera with the specified camera identifier c. Note that the operation in step S78 is the same as the operation in steps S22 to S25 in FIG. 6. This allows the camera calibration device 1 to determine the feature point world coordinates optimized for the camera parameters, and to superimpose an aiming figure on the image captured by the specified camera 2, thereby visually presenting the deviation between the first aiming figure and the second aiming figure. After this, the camera calibration device 1 returns to the operation at step S1 (FIG. 5). Through the above operations, the camera calibration device 1 allows the user to independently optimize the positions of feature points and the camera parameters.

[0121] Although the embodiment of the present invention has been described above, the present invention is not limited to this embodiment. Here, the captured images used are images captured from different viewpoints by a plurality of different cameras 2. However, the captured images may also be images captured at different times by a single camera 2. In that case, the camera identifiers should be ones that identify the different times. This allows the camera calibration device 1 to estimate camera parameters for images captured at different times.

[0122] In addition, here, as the aiming figures to be displayed on the viewer 120 of the control screen SC (Figure 4), a first aiming figure is used for the image coordinates (feature point image coordinates) of the feature points detected in the captured image, and a second aiming figure is used for the position (projection coordinates) where the projection unit 11 projects the feature point world coordinates onto the image coordinates. However, the feature point image coordinates may be configured to distinguish between feature points extracted by the feature point extraction unit 15 and feature points manually set via the input device 4, and a first aiming figure may be used for the coordinates of the former, and a third aiming figure different from the first and second aiming figures may be used for the coordinates of the latter. [Explanation of symbols]

[0123] S Camera Calibration System 1. Camera calibration device 10. Operation control section 100 Control screen generation unit 101 Camera selection section 102 Feature point selection unit 103 Camera parameter update instruction unit 104 Feature point world coordinate update instruction unit 11 Projection section 12 Projection error calculation section 13 Camera parameter update section 130 Gradient calculation unit 131 Camera Parameter Optimization Unit 14. Feature point world coordinate update unit 140 Gradient calculation unit 141 Feature point world coordinate optimization unit 15 Feature point extraction unit 16 Image synthesis unit 17. Photographed image storage unit 18 Feature point image coordinate storage unit 19 Camera parameter memory section 20 Feature point world coordinate memory 2 Cameras

Claims

1. A camera calibration device for estimating camera parameters of a camera, comprising: a projection unit that projects feature point world coordinates, which are positions of the feature points in world coordinates, onto image coordinates using the camera parameters; a projection error calculation unit that calculates an error between the projection coordinates projected by the projection unit and feature point image coordinates that are the positions of the feature points on the captured image captured by the camera; a camera parameter update unit that updates the camera parameters in a direction that reduces the error; a feature point world coordinate updating unit that updates the feature point world coordinates in a direction that reduces the error; an operation control unit that switches between operating the camera parameter update unit and the feature point world coordinate update unit; an image synthesis unit that synthesizes and displays information indicating the positions of the feature point image coordinates and the projection coordinates on the captured image, the camera parameter update unit updates the camera parameters based on a Jacobian that represents a change in each component of the error relative to a change in each component of the camera parameters, and the error; the feature point world coordinate update unit updates the feature point world coordinates based on a Jacobian that represents a change in each component of the error with respect to a change in each component of the feature point world coordinates, and the error.

2. 2. The camera calibration device according to claim 1, wherein, when instructed to update the feature point world coordinates of selected feature points, the operation control unit updates the feature point world coordinates of only the selected feature points using the feature point world coordinate update unit.

3. 3. The camera calibration device according to claim 1, wherein when the operation control unit is instructed to update the feature point world coordinates of all the feature points, the feature point world coordinate update unit updates the feature point world coordinates of all the feature points.

4. 4. The camera calibration device according to claim 1, wherein when the operation control unit is instructed to update the camera parameters of a selected camera, the camera parameter update unit updates the camera parameters of only the selected camera.

5. 5. The camera calibration device according to claim 1, wherein when the operation control unit is instructed to update the camera parameters of all the cameras, the camera parameter update unit updates the camera parameters of all the cameras.

6. 6. The camera calibration device according to claim 1, further comprising a feature point extraction unit that extracts the feature points from the captured image and sets the feature point image coordinates.

7. 7. The camera calibration device according to claim 6, wherein the feature point extracting unit extracts the feature points based on a specific pattern or an image feature.

8. A camera calibration program for causing a computer to function as the camera calibration device according to any one of claims 1 to 7.

Images