Camera calibration device, camera calibration method, and camera calibration program

Abstract

To improve the accuracy of calibrating a camera and work efficiency.SOLUTION: A camera calibration device 10 comprises an image acquisition circuit 16A, a first attitude information acquisition circuit 16B, a second attitude information acquisition circuit 16C, a calculation circuit 16D, and a correction circuit 16E. The image acquisition circuit 16A acquires a photographic image, which is obtained by a camera 20, of a marker for calibration 40 provided on a display unit 30 (information apparatus) comprising an attitude sensor 34. The first attitude information acquisition circuit 16B acquires first attitude information on the marker for calibration during photographing of the photographic image. The second attitude information acquisition circuit 16C acquires second attitude information on a device mounted with the camera (vehicle 2). The calculation circuit 16D calculates the amount of calibration according to camera attitude information on the camera 20 derived based on the marker for calibration included in the photographic image. The correction circuit 16E corrects the amount of calibration calculated by the calculation circuit 16D based on the difference between the first attitude information and the second attitude information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a camera calibration device, a camera calibration method, and a camera calibration program. [Background technology]

[0002] Conventionally, installation errors of cameras mounted on camera-mounted devices such as vehicles and measuring instruments have been calibrated by their distributors, manufacturers, etc. For example, a method has been disclosed in which a calibration marker placed on the floor at a precise angle relative to the vehicle is photographed by an on-board camera, and the amount of deviation in the camera's posture is calculated using feature points of the marker included in the photographed image (see, for example, Patent Documents 1 to 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-118414 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-30554 [Patent Document 3] Japanese Patent Application Laid-Open No. 2012-123750 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, calibration accuracy can be reduced when it is difficult to place calibration markers in accurate positions relative to the mounted equipment around the camera mounted on a vehicle, etc. Furthermore, placing calibration markers in accurate positions often requires a significant amount of work time.

[0005] Non-limiting embodiments of the present disclosure contribute to providing a camera calibration device, a camera calibration method, and a camera calibration program that can improve the accuracy of camera calibration. [Means for solving the problem]

[0006] A camera calibration device according to the present disclosure includes an image acquisition circuit, a first orientation information acquisition circuit, a second orientation information acquisition circuit, a calculation circuit, and a correction circuit. The image acquisition circuit acquires an image captured by a camera of a calibration marker provided in an information device equipped with an orientation sensor. The first orientation information acquisition circuit acquires first orientation information of the calibration marker at the time of capturing the captured image. The second orientation information acquisition circuit acquires second orientation information of a device mounted on the camera. The calculation circuit calculates a calibration amount corresponding to the camera orientation information of the camera derived based on the calibration marker included in the captured image. The correction circuit corrects the calibration amount based on the difference between the first orientation information and the second orientation information. The calibration marker includes a plurality of parallel lines arranged in directions that intersect with each other, and the image acquisition circuit acquires the captured image of the calibration marker, in which at least one of the parallel lines included in the calibration marker is arranged parallel to the shooting direction of the camera, and the calculation circuit calculates the camera attitude information based on a vanishing point that is a convergence point of extensions of the parallel lines formed by the calibration markers included in the captured image and a vanishing line that is a straight line passing through the plurality of vanishing points. [Effects of the Invention]

[0007] According to the camera calibration device, camera calibration method, and camera calibration program disclosed herein, it is possible to improve the accuracy of camera calibration and work efficiency. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing an example of a calibration system according to this embodiment. [Figure 2A] FIG. 2A is a schematic diagram of an example of a calibration marker displayed on a display unit. [Figure 2B] FIG. 2B is a schematic diagram of an example of a calibration marker displayed on the display unit. [Figure 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of a camera calibration device and a display device. [Figure 4] FIG. 4 is a block diagram of an example of the functional configuration of the calibration system. [Figure 5A] FIG. 5A is an explanatory diagram of an example of the positional relationship between a vehicle and a display device. [Figure 5B] FIG. 5B is an explanatory diagram of an example of the positional relationship between the vehicle and the display device. [Figure 6] FIG. 6 is a schematic diagram of an example of a captured image. [Figure 7A] FIG. 7A is a schematic diagram showing an example of a display form of an image representing first posture information. [Figure 7B] FIG. 7B is a schematic diagram showing an example of a display form of an image representing the first attitude information. [Figure 8] FIG. 8 is a diagram illustrating an example of calculation of camera attitude information. [Figure 9A] FIG. 9A is a diagram illustrating an example of calculation of the difference between the first attitude information and the second attitude information. [Figure 9B] FIG. 9B is an explanatory diagram of an example of calculation of the difference between the first attitude information and the second attitude information. [Figure 9C] FIG. 9C is a diagram illustrating an example of calculation of the difference between the first attitude information and the second attitude information. [Figure 10] FIG. 10 is a flowchart showing an example of the flow of information processing executed by the display device of this embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the flow of information processing executed by the camera calibration device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of a camera calibration device, a camera calibration method, and a camera calibration program according to the present disclosure will be described with reference to the accompanying drawings.

[0010] The configuration of a calibration system 1 of this embodiment will be described with reference to FIG.

[0011] The calibration system 1 is a system that calibrates the installation error of a camera 20 mounted on an on-board device. The on-board device may be any device that can mount a camera. More specifically, the on-board device is a device that executes various processes using images captured by the mounted camera. Specific examples of the on-board device include a vehicle and a measuring device. In this embodiment, a form in which the on-board device is a vehicle 2 will be described as an example.

[0012] The calibration system 1 includes a camera calibration device 10 , a camera 20 , an attitude sensor 22 , and a display device 30 .

[0013] The camera calibration device 10 is an information processing device for calibrating an installation error of the camera 20 mounted on the vehicle 2. For example, the camera calibration device 10 is realized by an ECU (Electronic Control Unit) mounted on the vehicle 2.

[0014] The camera 20 is a device to be calibrated, and captures captured image data by capturing images. In this embodiment, the captured image data will be simply referred to as a captured image.

[0015] In this embodiment, the camera 20 is mounted on the vehicle 2. For example, a plurality of cameras 20 with different shooting directions are attached to the vehicle 2. FIG. 1 shows an example in which the vehicle 2 is provided with a camera 20A that shoots an image in front of the vehicle 2, a camera 20D that shoots an image behind the vehicle 2, and cameras 20C and 20D that shoot an image in the vehicle width direction around the vehicle 2. Note that the number and positions of the cameras 2 mounted on the vehicle 2 are not limited to the example shown in FIG. 1.

[0016] The attitude sensor 22 is a sensor that detects second attitude information, which is attitude information of the vehicle 2.

[0017] Attitude information is expressed by roll angle, pitch angle, and yaw angle. The roll angle is the rotation angle around the Z axis, which is the roll axis. The pitch angle is the rotation angle around the X axis, which is the pitch axis. The yaw angle is the rotation angle around the Y axis, which is the yaw axis.

[0018] Examples of the attitude sensor 22 include a three-axis angle sensor and a geomagnetic sensor. In this embodiment, the attitude sensor 22 detects second attitude information including a roll angle, a pitch angle, and a yaw angle, which are tilts around each of the X, Y, and Z axes detected by the three-axis angle sensor, based on the directions obtained by the geomagnetic sensor and the horizontal plane.

[0019] The display device 30 is an information processing device that displays the calibration marker 40. The display device 30 is a mobile terminal, a tablet terminal, a personal computer, etc. In this embodiment, an example in which the display device 30 is a tablet terminal will be described.

[0020] The display device 30 includes a display unit 32 and an attitude sensor 34. The display unit 32 displays various types of information on a display surface that is a two-dimensional plane. The display unit 32 is a display or monitor that displays various types of information.

[0021] The attitude sensor 34 is a sensor that detects first attitude information, which is attitude information of the calibration marker 40 displayed on the display unit 32. In this embodiment, the attitude sensor 34 detects attitude information of the display, which is the display unit 32 that displays the calibration marker 40, as the first attitude information of the calibration marker 40.

[0022] A three-axis angle sensor and a geomagnetic sensor are examples of the attitude sensor 34. In this embodiment, the attitude sensor 34 detects second attitude information including a roll angle, a pitch angle, and a yaw angle, which are tilts around each of the X, Y, and Z axes detected by the three-axis angle sensor, based on the directions obtained by the geomagnetic sensor and the horizontal plane.

[0023] The calibration marker 40 is an index used to calibrate the camera 20. For example, the calibration marker 40 includes one or more patterns represented by dots, multiple straight lines, etc. In this embodiment, an example will be described in which the calibration marker 40 includes parallel lines arranged in directions that intersect with each other. The parallel lines are made up of multiple parallel straight lines.

[0024] 2A is a schematic diagram of an example of a grid-shaped calibration marker 40. For example, the calibration marker 40 includes a plurality of parallel lines L1 and a plurality of parallel lines L2 arranged in a direction intersecting the parallel lines L1. The parallel lines L1 and L2 are examples of parallel lines L formed by the calibration marker 40. FIG. 2A shows an example in which the parallel lines L1 and L2 intersect in perpendicular directions.

[0025] 2B is a schematic diagram of an example of a parallelogram-shaped calibration marker 40. For example, the calibration marker 40 is represented by one or more parallelogram patterns made up of parallel lines L1 and L2.

[0026] In this embodiment, an example will be described in which the calibration marker 40 has a grid pattern as shown in Fig. 2A. In this embodiment, an example will be described in which the calibration marker 40 has a pattern including a plurality of parallel lines L1 and a plurality of parallel lines L2 arranged in a direction perpendicular to the parallel lines L1.

[0027] Returning to Figure 1, the explanation will continue. When calibrating camera 20, display device 30 is placed by the user within the shooting angle of view of camera 20 to be calibrated. Figure 1 shows an example in which camera 20A is the camera 20 to be calibrated. Camera 20 captures an image of calibration marker 40 displayed on display unit 32 of display device 30, and the captured image is used in processing by camera calibration device 10, which will be described later.

[0028] The configurations of the camera calibration device 10 and the display device 30 will be described with reference to FIG.

[0029] The camera calibration device 10 and the display device 30 have a hardware configuration that utilizes a conventional computer, with a CPU (Central Processing Unit) 90A, a ROM (Read Only Memory) 90B, a RAM 90C, an I / F 90D, etc. connected to each other via a bus 90E.

[0030] The CPU 90A is a computing device that controls the camera calibration device 10 and the display device 30 of this embodiment. The ROM 90B stores programs and the like that realize various processes by the CPU 90A. The RAM 90C stores data used for various processes by the CPU 90A. The I / F 90D is an interface for sending and receiving data.

[0031] The programs for executing information processing executed by the camera calibration device 10 and display device 30 of this embodiment are provided by being pre-installed in ROM 90B etc. The programs executed by the camera calibration device 10 and display device 30 of this embodiment may be provided by being recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk) in a format that can be installed or executed on the camera calibration device 10 and display device 30.

[0032] Next, the functional configuration of the calibration system 1 will be described in detail with reference to FIG.

[0033] As described above, the calibration system 1 includes the camera calibration device 10, the camera 20, the attitude sensor 22, and the display device 30. The camera calibration device 10, the camera 20, the attitude sensor 22, and the display device 30 are communicatively connected to each other.

[0034] The display device 30 includes a display unit 32, an input unit 33, an attitude sensor 34, a memory unit 35, a communication unit 36, and a processing unit 37. The display unit 32, the input unit 33, the attitude sensor 34, the memory unit 35, the communication unit 36, and the processing unit 37 are communicatively connected via a bus 38 or the like.

[0035] The input unit 33 receives operation inputs from a user. The input unit 33 is, for example, a pointing device such as a mouse, a keyboard, etc. The display unit 32 and the input unit 33 may be an integrated touch panel.

[0036] The storage unit 35 stores various types of data. In this embodiment, the storage unit 35 stores image data of the calibration marker 40 to be displayed on the display unit 32 in advance.

[0037] The communication unit 36 ​​is a communication interface for communicating with an external information processing device such as the camera calibration device 10. The processing unit 37 executes various types of information processing.

[0038] The camera calibration device 10 includes a communication unit 12, a storage unit 14, and a processing unit 16. The communication unit 12, the storage unit 14, and the processing unit 16 are communicatively connected via a bus 18 or the like.

[0039] The communication unit 12 is a communication interface that communicates with the display device 30, the camera 20, the attitude sensor 22, etc. The storage unit 14 stores various types of information.

[0040] The processing unit 16 executes various types of information processing and includes an image acquisition circuit 16A, a first attitude information acquisition circuit 16B, a second attitude information acquisition circuit 16C, a calculation circuit 16D, and a correction circuit 16E.

[0041] At least one of the image acquisition circuit 16A, the first attitude information acquisition circuit 16B, the second attitude information acquisition circuit 16C, the calculation circuit 16D, and the correction circuit 16E may be realized by, for example, causing a processing device such as the CPU 90A to execute a program, for example, by software, or by hardware such as an IC (Integrated Circuit), or by a combination of software and hardware. Also, at least one of the first attitude information acquisition circuit 16B, the second attitude information acquisition circuit 16C, the calculation circuit 16D, and the correction circuit 16E may be mounted on an external information processing device communicatively connected to the camera calibration device 10 via a network or the like.

[0042] The image acquisition circuit 16A acquires an image of a calibration marker 40 provided on the display device 30 captured by the camera 20.

[0043] The user sets up the display device 30 within the shooting angle of view of the camera 20 to be calibrated. In this embodiment, a situation in which the display device 30 is set up within the shooting angle of view of the camera 20A will be described as an example. Then, the user positions the display device 30 so that at least one parallel line L included in the calibration marker 40 displayed on the display unit 32 of the display device 30 is parallel to the shooting direction of the camera 20A.

[0044] 2A, the user places the display device 30 within the angle of view of the camera 20A so that the parallel line L1 of the calibration marker 40 displayed on the display unit 32 is parallel to the shooting direction SD of the camera 20A. Note that the parallel line L2 included in the calibration marker 40 does not have to be positioned so as to be completely parallel to the shooting direction SD.

[0045] For example, while viewing the calibration marker 40 displayed on the display unit 32, the user positions the display device 30 within the shooting angle of view of the camera 20 so that the parallel line L1 of the calibration marker 40 is approximately parallel to the shooting direction SD of the camera 20A to be calibrated.

[0046] As shown in FIG. 5A, for example, a user places the display device 30 directly on an installation surface G of a vehicle 2 equipped with a camera 20. The installation surface G is, for example, the ground, a floor, or the like. Alternatively, as shown in FIG. 5B, the user may place the display device 30 on a support member 39 placed on the installation surface G. In this way, the display device 30 may be installed directly on the installation surface G, or may be placed on a support member 39 installed on the installation surface G.

[0047] Then, the user places the display device 30 within the shooting angle of view of the camera 20A so that the shooting direction SD of the camera 20A is approximately parallel to the parallel line L1 of the calibration marker 40 displayed on the display unit 32. At this time, as described above, the parallel line L1 or the parallel line L2 of the calibration marker 40 displayed on the display unit 32 and the shooting direction SD of the camera 20 do not have to be completely parallel.

[0048] In this embodiment, a form in which the calibration marker 40 is displayed on the display unit 32 of the display device 30 will be described as an example. However, the calibration marker 40 may be provided on a two-dimensional plane of a device equipped with an attitude sensor 34, such as the display device 30, and is not limited to a form in which it is displayed on the display unit 32. For example, the display device 30 may be configured to have a flat, plate-like surface, and the calibration marker 40 may be recorded, written, or attached to the plate-like surface. In this case, the attitude sensor 34 may be configured to be able to detect attitude information of the plate-like surface as first attitude information of the calibration marker 40.

[0049] Returning to Figure 4, the explanation will be continued. For example, when the processing unit 37 of the display device 30 receives input of a calibration start request by a user operating the input unit 33, the processing unit 37 reads image data of the calibration marker 40 stored in the storage unit 35 and displays it on the display unit 32. While visually checking the calibration marker 40 displayed on the display unit 32, the user places the display device 30 within the shooting angle of view of the camera 20A so that the parallel line L1 of the calibration marker 40 and the shooting direction SD of the camera 20A are approximately parallel to each other.

[0050] The camera 20A captures an image of the calibration marker 40 displayed on the display unit 32 of the display device 30 placed within the camera 20A's angle of view, thereby acquiring a captured image of the calibration marker 40 displayed on the display unit 32. The image acquisition circuit 16A receives the captured image captured by the camera 20A from the camera 20A, thereby acquiring the captured image of the calibration marker 40 placed in the real space S by the camera 20A.

[0051] 6 is a schematic diagram of an example of a captured image 50 captured by the camera 20A. The captured image 50 includes an image of the calibration marker 40. In the following description, the captured image of the calibration marker 40 will be simply referred to as the calibration marker 40.

[0052] Continuing the explanation, returning to Fig. 4, the first attitude information acquisition circuit 16B acquires first attitude information of the calibration marker 40 when the photographed image 50 is captured.

[0053] In this embodiment, the first orientation information acquisition circuit 16B acquires orientation information of the display unit 32 at the time of capturing the captured image 50 as the first orientation information of the calibration marker 40. For example, the first orientation information is orientation information of the calibration marker 40 in the real space S at the time of capturing the captured image 50 by the camera 20A.

[0054] For example, the first attitude information acquisition circuit 16B receives the first attitude information from the display device 30 via the communication unit 12, thereby acquiring the first attitude information.

[0055] In this case, for example, the processing unit 37 of the display device 30 sequentially transmits the first attitude information of the display unit 32 detected by the attitude sensor 34 to the camera calibration device 10 together with detection time information indicating the detection time of the first attitude information. The image acquisition circuit 16A identifies the first attitude information received from the camera 20A together with detection time information that matches the shooting time of the captured image 50 as the first attitude information of the calibration marker 40 included in the captured image 50 at the time of shooting. Then, the image acquisition circuit 16A may acquire the identified first attitude information as the first attitude information at the time of shooting the captured image 50.

[0056] Furthermore, for example, the processing unit 37 of the display device 30 may display an image representing the first orientation information detected by the orientation sensor 34 on the display unit 32 while the calibration marker 40 is being displayed on the display unit 32.

[0057] 7A, for example, the processing unit 37 of the display device 30 displays the calibration marker 40 and an image 42 representing the first attitude information on the display unit 32. In FIG. 7A, a character image 42A representing the first attitude information is shown as an example of the image 42 representing the first attitude information. Rdx represents the pitch angle of the display unit 32, Rdy represents the yaw angle of the display unit 32, and Rdz represents the roll angle of the display unit 32.

[0058] Furthermore, as shown in FIG. 7B, the processing unit 37 of the display device 30 may display a QR (Quick Response) (registered trademark) code 42B representing the first attitude information on the display unit 32 as the image 42 representing the first attitude information.

[0059] It should be noted that the processing unit 37 may display the image 42 representing the first orientation information, which is the latest detection result, on the display unit 32 every time new first orientation information is detected by the orientation sensor 34. Therefore, the image 42 representing the current first orientation information of the calibration marker 40 included in the display unit 32 is displayed on the display unit 32.

[0060] 4, the description will be continued. When the image 42 representing the first attitude information is displayed on the display unit 32, the first attitude information acquisition circuit 16B acquires the first attitude information of the calibration marker 40 included in the captured image 50 at the time of capturing the image 50 by performing image analysis of the image 42 representing the first attitude information included in the captured image 50 using a known method.

[0061] The second attitude information acquisition circuit 16C acquires second attitude information of the vehicle 2, which is an on-board device of the camera 20. The second attitude information acquisition circuit 16C acquires the second attitude information of the vehicle 2 by receiving the second attitude information detected by the attitude sensor 22. Note that the second attitude information acquisition circuit 16C may acquire the second attitude information detected by the attitude sensor 22 at a detection time that coincides with the shooting time of the captured image 50.

[0062] The calculation circuit 16D derives camera attitude information of the camera 20 based on the calibration markers 40 included in the captured image 50. Then, the calculation circuit 16D calculates the amount of calibration according to the derived camera attitude information.

[0063] The calibration amount is information used to calibrate the installation error of the camera 20. In this embodiment, a case where the calibration amount is expressed by a roll angle, a pitch angle, and a yaw angle will be described as an example.

[0064] The calculation circuit 16D derives camera attitude information of the camera 20 by a known method using the calibration markers 40 included in the captured image 50. For example, the calculation circuit 16D calculates the camera attitude information based on vanishing points that are convergence points of extensions of parallel lines L formed by the calibration markers 40 included in the captured image 50, and vanishing lines that are straight lines passing through multiple vanishing points.

[0065] The calculation of the camera attitude information will be described with reference to FIG.

[0066] The calculation circuit 16D identifies the convergence point P of at least one of the parallel lines L along the shooting direction SD and the parallel lines L along a direction intersecting the shooting direction SD, which are formed by the calibration marker 40 included in the captured image 50.

[0067] In detail, the calculation circuit 16D identifies parallel lines L3 and L4 that connect the intersections of parallel lines L1 and L2 that make up the grid-shaped calibration marker 40. Then, the calculation circuit 16D identifies the convergence point of parallel line L3 as vanishing point P3. Furthermore, the calculation circuit 16D identifies the convergence point of parallel line L4 as vanishing point P2. Similarly, the calculation circuit 16D identifies the convergence point of parallel line L1, which is a parallel line L that runs along the shooting direction SD of the calibration marker 40 placed in the real space S, as deep vanishing point P1. The deep vanishing point P1, vanishing point P2, and vanishing point P3 are examples of vanishing point P.

[0068] Furthermore, the calculation circuit 16D specifies a straight line passing through the two specified vanishing points P, for example, the vanishing points P2 and P3, as a vanishing line VL.

[0069] Then, the calculation circuit 16D calculates the roll angle and pitch angle included in the camera attitude information based on the vanishing point P2, the vanishing point P3, and the vanishing line VL using a known method.

[0070] For example, the calculation circuit 16D calculates the roll angle by a known method using the vanishing line VL. In detail, for example, the calculation circuit 16D calculates the angle of the vanishing line VL with respect to the horizontal direction in the real space S as the roll angle included in the camera posture information.

[0071] Furthermore, the calculation circuit 16D uses the calculated roll angle and one vanishing point P (for example, vanishing point P3) to calculate the pitch angle of the camera 20 using a known method. In detail, for example, the calculation circuit 16D calculates the movement angle around the pitch axis of the vanishing point P relative to the vanishing line VL as the pitch angle included in the camera posture information.

[0072] Furthermore, the calculation circuit 16D uses the calculated pitch angle and the deep vanishing point P1 to calculate the yaw angle of the camera 20 by a known method. In detail, the calculation circuit 16D calculates the angle corresponding to the movement distance of the deep vanishing point P1 along the vanishing line VL as the yaw angle included in the camera attitude information.

[0073] The calculation circuit 16D calculates, as the calibration amount, camera attitude information including the roll angle, pitch angle, and yaw angle calculated by these calculation processes. Note that the calculation circuit 16D may calculate, as the calibration amount, a value with positive and negative signs (±) reversed by multiplying each of the roll angle, pitch angle, and yaw angle included in the camera attitude information by "-1." In this embodiment, a case will be described as an example in which the calculation circuit 16D calculates, as the calibration amount, camera attitude information calculated from a calibration marker 40 included in a captured image 50.

[0074] Continuing the description, returning to Fig. 4, the correction circuit 16E calculates a corrected calibration amount by correcting the calibration amount calculated by the calculation circuit 16D based on the difference between the first attitude information and the second attitude information.

[0075] For example, the correction circuit 16E corrects the calibration amount calculated by the calculation circuit 16D based on the calibration marker 40 included in the captured image 50, based on first attitude information which is attitude information of the calibration marker 40 at the time of capturing the captured image 50, and second attitude information which is attitude information of the vehicle 2 at the time of capturing the captured image 50. Then, the correction circuit 16E stores the corrected calibration amount obtained by the correction in the memory unit 35 as the official calibration amount of the camera 20.

[0076] First, the correction circuit 16E calculates the difference between first attitude information, which is attitude information of the display unit 32 when the captured image 50 was captured, and second attitude information, which is attitude information of the vehicle 2 when the captured image 50 was captured.

[0077] Calculation of the difference between the first attitude information and the second attitude information will be described with reference to FIGS. 9A to 9C.

[0078] As described above, the first attitude information and the second attitude information are examples of attitude information, and are expressed by a roll angle, a pitch angle, and a yaw angle. dx represents the pitch angle of the display unit 32. R dy represents the yaw angle of the display unit 32. dz represents the roll angle of the display unit 32. cx represents the pitch angle of vehicle 2. R cy represents the yaw angle of vehicle 2. R cz represents the roll angle of the vehicle 2. In FIG. 9C, N represents the north direction.

[0079] The correction circuit 16E calculates the difference between the first attitude information and the second attitude information by calculating the differences in the roll angle, pitch angle, and yaw angle between the first attitude information and the second attitude information using the following equations (1) to (3).

[0080] Pitch angle difference = (R dx -R cx )...Equation (1) Yaw angle difference = (R dy -R cy )...Equation (2) Roll angle difference = (R dz -R cz )...Equation (3)

[0081] Then, the correction circuit 16E calculates, as the corrected calibration amount, the result of subtracting the difference between the first attitude information and the second attitude information from the calibration amount calculated by the calculation circuit 16D based on the calibration marker 40. In detail, the correction circuit 16E calculates the corrected calibration amount using the following equations (4) to (6).

[0082] Pitch angle correction calibration amount = Pitch angle correction amount - (R dx -R cx )...Equation (4) Yaw angle correction calibration amount = Yaw angle correction amount -(R dy -R cy )...Equation (5) Roll angle correction calibration amount = Roll angle correction amount - (R dz -R cz )...Equation (6)

[0083] For example, in the camera calibration device 10 of this embodiment, the difference between the first attitude information and the second attitude information expressed by the above formulas (1) to (3) is regarded as an installation error of the calibration marker 40 as seen from the camera 20A to be calibrated. Then, the correction circuit 16E of the camera calibration device 10 calculates a corrected calibration amount, which is the correct calibration amount, by removing an influence component due to the installation error of the calibration marker 40 as seen from the camera 20A from the calibration amount calculated by the calculation circuit 16D based on the calibration marker 40 included in the captured image 50.

[0084] Then, the correction circuit 16E stores the correction calibration amount obtained by the correction in the storage unit 14 as the official calibration amount of the camera 20A.

[0085] The correction circuit 16E may store the correction calibration amount obtained by the correction in the storage unit 14 in association with the identification information of the camera 20A to be calibrated. The correction circuit 16E may also correct the installation error of the camera 20A by a known method using the calculated correction calibration amount.

[0086] In addition, the camera calibration device 10 of this embodiment performs similar processing for the other cameras 20 to be calibrated (cameras 20B to 20D), thereby calculating the correction calibration amount for each of the cameras 20 mounted on the vehicle 2.

[0087] Next, an example of the flow of information processing executed by the proofreading system 1 of this embodiment will be described with reference to FIG.

[0088] When the processing unit 37 of the display device 30 receives a signal representing a calibration start request from the input unit 33 in response to an operation instruction from the user on the input unit 33 (step S100), the process proceeds to step S102.

[0089] In step S102, the processing unit 37 reads image data of the calibration marker 40 from the storage unit 35 (step S102), and displays it on the display unit 32 (step S104).

[0090] While viewing the calibration marker 40 displayed on the display unit 32, the user positions the display device 30 within the shooting angle of view of the camera 20 so that the parallel line L1 of the calibration marker 40 is approximately parallel to the shooting direction SD of the camera 20A to be calibrated.

[0091] The processing unit 37 acquires first orientation information of the display unit 32 from the orientation sensor 34 (step S106), and displays an image 42 representing the first orientation information on the display unit 32 (step S108). Therefore, the calibration marker 40 and the image 42 representing the first orientation information are displayed on the display unit 32. Note that the processing unit 37 may transmit the first orientation information detected by the orientation sensor 34 and the detection time of the first orientation information to the camera calibration device 10.

[0092] Next, the processing unit 37 determines whether a calibration end request has been received (step S110). For example, the processing unit 37 makes the determination in step S110 by determining whether a signal representing a calibration end request has been received from the input unit 33 due to a user's operation instruction on the input unit 33, etc. If the determination in step S110 is negative (step S110: No), the process returns to step S106. If the determination in step S110 is positive (step S110: Yes), the process ends this routine.

[0093] Next, an example of the flow of information processing executed by the camera calibration device 10 of this embodiment will be described with reference to FIG.

[0094] When the processing unit 16 of the camera calibration device 10 receives a calibration start request by an operation instruction from a user or the like (step S200), the processing unit 16 acquires the captured image 50 from the camera 20 (step S202).

[0095] The calculation circuit 16D derives camera attitude information of the camera 20 based on the calibration marker 40 included in the photographed image 50 acquired in step S202, and calculates the amount of calibration according to the camera attitude information (steps S204 to S222).

[0096] In detail, the calculation circuit 16D detects the parallel lines L1 and L2 that intersect with each other and are included in the calibration marker 40 included in the captured image 50 (step S204).

[0097] If the detection of the parallel lines L1 and L2 fails (step S202: No), the process proceeds to step S238, which will be described later. If the detection of the parallel lines L1 and L2 succeeds (step S202: Yes), the process proceeds to step S208.

[0098] In step S208, the calculation circuit 16D identifies the convergence point of at least one of the parallel lines L along the shooting direction SD and the parallel lines L along a direction intersecting the shooting direction SD as a vanishing point P (step S212). For example, as shown in FIG. 8, the calculation circuit 16D identifies parallel lines L3 and L4 connecting the intersections of parallel lines L1 and L2 that constitute the grid-shaped calibration marker 40. The calculation circuit 16D then identifies the convergence point of the parallel line L3 as a vanishing point P3. The calculation circuit 16D also identifies the convergence point of the parallel line L4 as a vanishing point P2. The calculation circuit 16D also identifies a vanishing point P that is the convergence point of the parallel line L1 included in the captured image 50, which is the parallel line L along the shooting direction SD of the calibration marker 40 placed in the real space S.

[0099] The calculation circuit 16D determines whether or not two or more vanishing points P have been identified by the processing of step S208 (step S210). If it is difficult to identify two or more vanishing points P (step S210: No), the process proceeds to step S238, which will be described later. If two or more vanishing points P have been identified (step S210: Yes), the process proceeds to step S212.

[0100] In step S212, the calculation circuit 16D specifies a line passing through the two vanishing points P specified in step S208 as a vanishing line VL (step S212). In addition, the calculation circuit 16D specifies, among the vanishing points P specified in step S208, the vanishing point P that is the convergence point of parallel lines L1, which are parallel lines L along the shooting direction SD, as a deep vanishing point P1 (step S214).

[0101] Next, the calculation circuit 16D calculates the angle of the vanishing line VL identified in step S212 with respect to the horizontal direction in the real space S as the roll angle of the camera 20 (step S216).

[0102] Next, the calculation circuit 16D calculates the pitch angle of the camera 20 using the roll angle calculated in step S216 and one vanishing point P (for example, vanishing point P3, see FIG. 8) identified in step S212 (step S218).

[0103] Next, the calculation circuit 16D calculates the yaw angle of the camera 20 using the pitch angle calculated in step S218 and the deep vanishing point P1 identified in step S214 (step S220).

[0104] The calculation circuit 16D calculates a calibration amount of the camera 20 according to the camera attitude information represented by the roll angle, pitch angle, and yaw angle of the camera 20 calculated in steps S216 to S220 (step S222). For example, the calculation circuit 16D calculates the camera attitude information represented by the roll angle, pitch angle, and yaw angle of the camera 20 calculated in steps S216 to S220 as the calibration amount of the camera 20.

[0105] Next, the calculation circuit 16D determines whether the calibration amount calculated in step S222 is normal (step S224). For example, the calculation circuit 16D stores in advance in the storage unit 14 the range of camera posture information that the camera 20 can take, in association with the identification information of the camera 20. The range of camera posture information that the camera 20 can take may be information provided by the manufacturer of the camera 20 or the like. The calculation circuit 16D makes the determination in step S224 by determining whether the calibration amount calculated in step S222 is a value within the range of camera posture information that corresponds to the camera identification information of the camera 20 to be calibrated.

[0106] If a negative decision is made in step S224 (step S224: No), the process proceeds to step S238, which will be described later. If a positive decision is made in step S224 (step S224: Yes), the process proceeds to step S226.

[0107] In step S226, the first attitude information acquisition circuit 16B acquires first attitude information of the calibration marker 40 at the time of capturing the captured image 50 acquired in step S202 (step S230). If the first attitude information acquisition circuit 16B fails to acquire the first attitude information (step S228: No), the process proceeds to step S238, which will be described later. If the first attitude information acquisition circuit 16B succeeds in acquiring the first attitude information (step S228: Yes), the process proceeds to step S230.

[0108] In step S230, the second attitude information acquisition circuit 16C acquires second attitude information of the vehicle 2 at the time of capturing the captured image 50 acquired in step S202 (step S230).

[0109] The correction circuit 16E calculates the difference between the first attitude information acquired in step S226 and the second attitude information acquired in step S230 (step S232).

[0110] Next, the correction circuit 16E calculates the corrected calibration amount by subtracting the difference between the first attitude information and the second attitude information calculated in step S232 from the calibration amount calculated in step S222 (step S234). Then, the correction circuit 16E stores the corrected calibration amount calculated in step S234 as the official calibration amount of the camera 20 to be calibrated in the storage unit 14 in association with the identification information of the camera 20 (step S236). Then, this routine ends.

[0111] On the other hand, in step S238, the processing unit 16 outputs information indicating the calibration error to a display unit or the like (not shown) of the camera calibration device 10 (step S238), and ends this routine.

[0112] As described above, the camera calibration device 10 of this embodiment includes an image acquisition circuit 16A, a first attitude information acquisition circuit 16B, a second attitude information acquisition circuit 16C, a calculation circuit 16D, and a correction circuit 16E. The image acquisition circuit 16A acquires a captured image 50 of a calibration marker 40 provided on a display device 30 (information device) equipped with an attitude sensor 34, captured by the camera 20. The first attitude information acquisition circuit 16B acquires first attitude information of the calibration marker 40 at the time of capturing the captured image 50. The second attitude information acquisition circuit 16C acquires second attitude information of the device (vehicle 2) on which the camera 20 is mounted. The calculation circuit 16D calculates a calibration amount corresponding to the camera attitude information of the camera 20 derived based on the calibration marker 40 included in the captured image 50. The correction circuit 16E corrects the calibration amount calculated by the calculation circuit 16D based on the difference between the first attitude information and the second attitude information.

[0113] Here, the prior art discloses a method of capturing an image of a calibration marker placed on the floor at a precise angle relative to the vehicle 2 with the camera 20 mounted on the vehicle 2, and determining the calibration amount of the camera 20 using feature points of the marker included in the captured image. However, in the prior art, it is necessary to place the markers with high precision around the mounted equipment of the camera 20 of the vehicle 2, etc., at a precise angle relative to the mounted equipment, and the calibration accuracy may be reduced depending on the installation environment.

[0114] For example, when a dealer or the like that sells the vehicle 2 performs calibration work on the camera 20, there may not be enough space around the vehicle 2 to place markers. Furthermore, unlike a vehicle production factory where markers are precisely placed, calibration personnel manually place markers each time, which makes it difficult for the calibration personnel to place the markers around the vehicle 2 at an accurate angle relative to the vehicle 2, and in addition, this may take a great deal of time.

[0115] However, the conventional technology uses markers that are positioned with high precision at an accurate angle relative to the vehicle 2, which can result in a decrease in calibration accuracy.

[0116] On the other hand, in the camera calibration device 10 of this embodiment, the calibration amount of the camera 20 derived based on the calibration marker 40 captured in the captured image 50 is corrected using first attitude information of the calibration marker 40 at the time of capturing the captured image 50 and second attitude information of the vehicle 2 at the time of capturing the captured image 50. Then, in the camera calibration device 10 of this embodiment, the corrected calibration amount obtained by correcting the calibration amount is treated as the official calibration amount of the camera 20.

[0117] For example, the camera calibration device 10 of this embodiment regards the difference between the first attitude information and the second attitude information as an installation error of the calibration marker 40 as seen from the camera 20 to be calibrated. Then, the camera calibration device 10 calculates, as the correct calibration amount of the camera 20, a corrected calibration amount obtained by removing an influence component due to the installation error of the calibration marker 40 as seen from the camera 20 from the calibration amount calculated based on the calibration marker 40 included in the captured image 50.

[0118] In this way, the camera calibration device 10 of this embodiment calculates a corrected calibration amount that eliminates the influence of installation error of the calibration marker 40 placed in the real space S. Therefore, the camera calibration device 10 of this embodiment can calculate the calibration amount of the camera 20 with high accuracy even if the installation accuracy of the calibration marker 40 with respect to the camera 20 mounted on the vehicle 2 is low.

[0119] Therefore, the camera calibration device 10 of this embodiment can improve the calibration accuracy of the camera 20 and the work efficiency.

[0120] Furthermore, the camera calibration device 10 of this embodiment can improve the calibration accuracy and work efficiency of the camera 20 even in an environment where there is insufficient space to install the calibration marker 40 around the onboard equipment, such as a vehicle 2 equipped with the camera 20 to be calibrated.

[0121] Furthermore, the camera calibration device 10 of this embodiment can improve the calibration accuracy and work efficiency of the camera 20 regardless of the level of skill of the calibration technician who places the calibration markers 40 around the vehicle 2. Furthermore, since it has a function to correct the difference between the calibration markers and the vehicle attitude, it is not necessary to place the calibration markers accurately, and work time can be reduced.

[0122] In the above-described embodiments, the notation "... part" used for each component may be replaced with other notations such as "... circuitry," "... assembly," "... device," "... unit," or "... module."

[0123] Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims. It is understood that such modifications or alterations also fall within the technical scope of the present disclosure. Furthermore, the components in the embodiments may be combined in any manner without departing from the spirit of the present disclosure.

[0124] The present disclosure can be realized by software, hardware, or software in cooperation with hardware. Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by a single LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of a single chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be referred to as an IC, system LSI, super LSI, or ultra LSI.

[0125] The integrated circuit method is not limited to LSI, but may be realized by dedicated circuits, general-purpose processors, or dedicated processors. Also, FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow the connections and settings of circuit cells within LSIs to be reconfigured, may be used. The present disclosure may be realized as digital processing or analog processing.

[0126] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology can be used to integrate functional blocks. The application of biotechnology is also a possibility.

[0127] Although the embodiments have been described above, they are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The above embodiments are included within the scope and spirit of the invention, and are also included in the scope of the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0128] 2 vehicles 10 Camera Calibration Device 16A Image acquisition circuit 16B First attitude information acquisition circuit 16C 2nd attitude information acquisition circuit 16D calculation circuit 16E Correction circuit 20, 20A, 20B, 20C, 20D Camera 22 Attitude Sensor 30 Display device 32 Display section 34 Attitude Sensor

Claims

1. an image acquisition circuit that acquires an image of a calibration marker provided in an information device having an attitude sensor, the image being captured by a camera; a first orientation information acquisition circuit for acquiring first orientation information of the calibration marker at the time of capturing the captured image; a second attitude information acquisition circuit that acquires second attitude information of the mounted device of the camera; a calculation circuit for calculating a calibration amount according to camera attitude information of the camera derived based on the calibration marker included in the captured image; a correction circuit that corrects the calibration amount based on a difference between the first attitude information and the second attitude information; Equipped with The calibration marker is It includes a plurality of parallel lines arranged in directions that intersect with each other, The image acquisition circuit acquiring the captured image of the calibration marker arranged so that at least one parallel line included in the calibration marker is parallel to the imaging direction of the camera; The calculation circuit calculating the camera posture information based on a vanishing point that is a convergence point of extensions of parallel lines formed by the calibration markers included in the captured image and a vanishing line that is a straight line passing through a plurality of the vanishing points; Camera calibration equipment.

2. The correction circuit calculating a corrected calibration amount by subtracting the difference between the first attitude information and the second attitude information from the calibration amount; The camera calibration device of claim 1 .

3. The image acquisition circuit acquiring the captured image of the calibration marker displayed on a monitor of the information device by the camera; The first attitude information acquisition circuit acquiring the first attitude information of the monitor on which the calibration marker is displayed; The camera calibration device of claim 1 .

4. The first attitude information acquisition circuit acquiring the first attitude information from the information device or the captured image; The camera calibration device of claim 1 .

5. an image acquisition step of acquiring, by a camera, an image of a calibration marker provided on an information device equipped with an attitude sensor; a first orientation information acquisition step of acquiring first orientation information of the calibration marker at the time of capturing the captured image; a second attitude information acquisition step of acquiring second attitude information of the mounted device of the camera; a calibration amount calculation step of calculating a calibration amount according to camera attitude information of the camera derived based on the calibration marker included in the captured image; a correcting step of correcting the calibration amount based on a difference between the first attitude information and the second attitude information, The calibration marker is It includes a plurality of parallel lines arranged in directions that intersect with each other, The image acquisition step includes: acquiring the captured image of the calibration marker arranged so that at least one parallel line included in the calibration marker is parallel to the imaging direction of the camera; The calibration amount calculation step calculating the camera posture information based on a vanishing point that is a convergence point of extensions of parallel lines formed by the calibration markers included in the captured image and a vanishing line that is a straight line passing through a plurality of the vanishing points; Camera calibration method.

6. The correction step calculating a corrected calibration amount by subtracting the difference between the first attitude information and the second attitude information from the calibration amount; The camera calibration method according to claim 5 .

7. The image acquisition step includes: acquiring the captured image of the calibration marker displayed on a monitor of the information device by the camera; The first attitude information acquisition step includes: acquiring the first attitude information of the monitor on which the calibration marker is displayed; The camera calibration method according to claim 5 .

8. The first attitude information acquisition step includes: acquiring the first attitude information from the information device or the captured image; The camera calibration method according to claim 5 .

9. acquiring, by a camera, an image of a calibration marker provided on an information device equipped with an attitude sensor; acquiring first orientation information of the calibration marker at the time of capturing the captured image; acquiring second attitude information of the camera's mounting device; calculating a calibration amount according to camera attitude information of the camera derived based on the calibration marker included in the captured image; correcting the calibration amount based on a difference between the first attitude information and the second attitude information; A camera calibration program for causing a computer to execute the above, The calibration marker is It includes a plurality of parallel lines arranged in directions that intersect with each other, The step of acquiring the photographed image includes: acquiring the captured image of the calibration marker arranged so that at least one parallel line included in the calibration marker is parallel to the imaging direction of the camera; The step of calculating the calibration amount includes: calculating the camera posture information based on a vanishing point that is a convergence point of extensions of parallel lines formed by the calibration markers included in the captured image and a vanishing line that is a straight line passing through a plurality of the vanishing points; Camera calibration program.

10. The correction step comprises: calculating a corrected calibration amount by subtracting the difference between the first attitude information and the second attitude information from the calibration amount; 10. The camera calibration program according to claim 9.

11. The step of acquiring the photographed image comprises: acquiring the captured image of the calibration marker displayed on a monitor of the information device by the camera; The step of acquiring first attitude information includes: acquiring the first attitude information of the monitor on which the calibration marker is displayed; 10. The camera calibration program according to claim 9.

Images