Calibration system, calibration device, and calibration program

Abstract

To enable autonomous calibration of in-vehicle cameras. [Solution] A calibration system comprising an imaging device provided on a moving body and a calibration device for calibrating the position and orientation of the imaging device, wherein the calibration device includes a storage unit for storing position information of a marker that serves as a calibration reference for the imaging device, an acquisition unit for acquiring an image of a subject including the marker from the imaging device, and a calculation unit for calculating the amount of change in the position and orientation of the imaging device based on the position information of the marker and the image of the marker acquired by the acquisition unit.

Description

Technical Field

[0001] The present invention relates to a calibration system, a calibration device, and a calibration program.

Background Art

[0002] The following Patent Document 1 discloses a vehicle equipped with an omnidirectional camera that captures a monitoring target within the imaging range.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

[0004] The calibration system of the first disclosed technology is a calibration system including an imaging device provided on a moving body and a calibration device for calibrating the position and orientation of the imaging device. The calibration device includes a storage unit that stores position information of a marker serving as a reference for calibration of the imaging device, an acquisition unit that acquires an image of a subject including the marker from the imaging device, and a calculation unit that calculates a change amount in the position and orientation of the imaging device based on the position information of the marker and the image of the marker acquired by the acquisition unit.

[0005] The calibration device of the second disclosed technology is a calibration device for calibrating the position and orientation of an imaging device provided on a moving body. The calibration device includes a storage unit that stores position information of a marker serving as a reference for calibration of the imaging device, an acquisition unit that acquires an image of a subject including the marker from the imaging device, and a calculation unit that calculates a change amount in the position and orientation of the imaging device based on the position information of the marker and the image of the marker acquired by the acquisition unit.

[0006] The calibration program of the third disclosed technology is a calibration program that causes a processor to perform calibration of the position and orientation of an imaging device installed on a moving object, and causes the processor to perform an acquisition process to acquire an image of a subject including a marker that serves as a calibration reference for the imaging device from the imaging device, and a calculation process to calculate the amount of change in the position and orientation of the imaging device based on the position information of the marker and the image of the marker acquired by the acquisition process. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is an explanatory diagram showing an example of camera placement. [Figure 2] Figure 2 is an explanatory diagram showing an example of camera structure. [Figure 3] Figure 3 is a block diagram showing an example of the configuration of a calibration device. [Figure 4] Figure 4 is an explanatory diagram showing an example of image changes due to changes in camera position. [Figure 5] Figure 5 is an explanatory diagram showing an example of image changes due to changes in the camera's orientation. [Figure 6] Figure 6 is a flowchart showing an example of the camera calibration process using a calibration device. [Figure 7] Figure 7 is a flowchart showing a detailed example of the calibration information calculation process (steps S603, S606). [Figure 8] Figure 8 is an explanatory diagram showing an example of conversion from a widescreen image to a perspective projection image. [Figure 9] Figure 9 is an explanatory diagram showing an example of an image displayed on a display device. [Modes for carrying out the invention]

[0008] <Figure 1: Example of camera placement> Figure 1 is an explanatory diagram showing an example of camera placement. Figure 1 shows an example of camera placement mounted on truck 101. In this embodiment, the configuration is applicable not only to truck 101 but also to other mobile vehicles such as passenger cars, buses, and motorcycles.

[0009] The global coordinate system 100 is a three-dimensional coordinate system in which the direction of movement of the truck 101 is the X-axis, the width direction of the truck 101 is the Y-axis, and the vertical direction is the Z-axis. The direction in which the arrowheads of the X, Y, and Z axes are located is considered the positive direction, and the opposite direction is considered the negative direction. The origin O is set to a predetermined fixed position.

[0010] Truck 101 has a tractor 102 and a trailer 103. The trailer 103 is connected to the tractor 102.

[0011] Tractor 102 has a right side mirror 110R and a left side mirror 110L. When the right side mirror 110R and the left side mirror 110L are not distinguished, they are referred to as side mirror 110.

[0012] The right side mirror 110R is equipped with a right front camera 111RF and a right rear camera 111RB. The right front camera 111RF is positioned in the positive direction of the X-axis so that its optical axis is parallel to the X-axis, and it images the right front of the track 101. The right front camera 111RF has a right front camera marker 121RF. The right front camera marker 121RF is positioned on the surface of the right front camera 111RF within the field of view (indicated by the dashed line) of the respective wide-angle lenses of the left front camera 111LF and the left rear camera 111LB.

[0013] The right rear camera 111RB is positioned in the negative direction of the X-axis so that its optical axis is parallel to the X-axis, and it images the right rear of track 101. The right rear camera 111RB has a right rear camera marker 121RB. The right rear camera marker 121RB is positioned on the surface of the right rear camera 111RB within the field of view (indicated by the dashed line) of the respective wide-angle lenses of the left front camera 111LF and the left rear camera 111LB.

[0014] The left side mirror 110L is equipped with a left front camera 111LF and a left rear camera 111LB. The left front camera 111LF is arranged in the positive direction of the X-axis so that its optical axis is parallel to the X-axis, images the left front of the track 101, and generates a left front wide image 131LF.

[0015] The left front camera 111LF has a left front camera marker 121LF. The left front camera marker 121LF is arranged on the surface of the left front camera 111LF at a position within the range of the field of view of the wide-angle lens of each of the right front camera 111RF and the right rear camera 111RB (the range indicated by the dashed line).

[0016] The left rear camera 111LB is arranged in the negative direction of the X-axis so that its optical axis is parallel to the X-axis, and images the left rear of the track 101. The left rear camera 111LB has a left rear camera marker 121LB. The left rear camera marker 121LB is arranged on the surface of the left rear camera 111LB at a position within the range of the field of view of the wide-angle lens of each of the right front camera 111RF and the right rear camera 111RB (the range indicated by the dashed line).

[0017] When the right front camera 111RF and the left front camera 111LF are not distinguished, they are denoted as the front camera 111F. When the right rear camera 111RB and the left rear camera 111LB are not distinguished, they are denoted as the rear camera 111B.

[0018] When the right front camera marker 121RF and the left front camera marker 121LF are not distinguished, they are denoted as the front camera marker 121F. When the right rear camera marker 121RB and the left rear camera marker 121LB are not distinguished, they are denoted as the rear camera marker 121B. When the front camera marker 121F and the rear camera marker 121B are not distinguished, they are denoted as the camera marker 121.

[0019] The camera marker 121 has a shape that can be distinguished when translated and rotated (for example, a polygon with sides of different lengths). The camera marker 121 may be displayed on the surfaces of the front camera 111F and the rear camera 111B in any form of formation, pasting, description, or emission. Also, the camera marker 121 may be a characteristic shape on the surfaces of the front camera 111F and the rear camera 111B. The coordinate values of the feature points (for example, 4 vertices if it is a rectangle) of the camera marker 121 on the global coordinate system 100 serve as reference points when calibrating the front camera 111F and the rear camera 111B that image the camera marker 121.

[0020] The rear camera 111MB is disposed at the rear of the trailer 103 in the negative direction of the X-axis such that its optical axis is parallel to the X-axis, and images the area behind the truck 101.

[0021] When not distinguishing between the right front camera 111RF, the right rear camera 111RB, the left front camera 111LF, the left rear camera 111LB, and the rear camera 111MB, they are denoted as the camera 111.

[0022] The camera 111 is an imaging device with a telephoto lens and a wide-angle lens arranged on its optical axis, and can simultaneously capture tele images and wide images.

[0023] On the front of the tractor 102, a right projector 112R and a left projector 112L are provided. The right projector 112R projects a right projection marker 122R with a predetermined pattern in front of the tractor 102. The left projector 112L projects a left projection marker

[0023] with a predetermined pattern in front of the tractor 102.

[0024] When not distinguishing between the right projector 112R and the left projector 112L, they are denoted as the projector 112. When not distinguishing between the right projection marker 122R and the left projection marker 122L, they are denoted as the projection marker 122. The installation position and projection direction of the projector 112 are fixed.

[0025] The projection marker 122 is positioned within the field of view of the wide-angle lens of the front camera 111F (the area indicated by the dashed line). The coordinate values ​​of the feature points (for example, four vertices if it is a rectangle) of the projection marker 122 on the global coordinate system 100 serve as reference points when calibrating the front camera 111F. The projector 112 may project the projection marker 122 using infrared light that is invisible to the naked eye.

[0026] Trailer 103 is equipped with a right rear marker 123R and a left rear marker 123L at both rear ends. The right rear marker 123R is positioned within the field of view (indicated by the dashed line) of the wide-angle lenses of the right rear camera 111RB and the rear camera 111MB, respectively. The left rear marker 123L is positioned within the field of view (indicated by the dashed line) of the wide-angle lenses of the left rear camera 111LB and the rear camera 111MB, respectively.

[0027] If the right rear marker 123R and the left rear marker 123L are not distinguished, they are referred to as rear marker 123. The rear marker 123, like the camera marker 121, can have any shape that is distinguishable when translated and rotated (for example, a polygon or ellipse where not all sides are of the same length).

[0028] The rear markers 123 can be formed, affixed, written, or illuminated, as long as they are displayed on both rear ends of the trailer 103. Alternatively, the rear markers 123 may be the distinctive shapes on the surfaces of the front camera 111F and the rear camera 111B. In this example, the rear markers 123 are LED (Light Emitting Diode) light sources. The rear markers 123 may be constantly illuminated or illuminated only during calibration.

[0029] <Figure 2: Example of Camera 111 Structure> Figure 2 is an explanatory diagram showing an example of the structure of camera 111. The local coordinate system 200 is a three-dimensional coordinate system composed of mutually orthogonal x, y, and z axes at the origin o. The local coordinate system 200 may be a three-dimensional coordinate system common to each camera 111, or it may be a different three-dimensional coordinate system for each camera 111. In this example, the local coordinate system 200 is a different three-dimensional coordinate system for each camera 111.

[0030] Camera 111 comprises a housing 201, a condensing lens 202, a prism 203, a telephoto lens 204, an image sensor 205, a wide-angle lens 206, and an image sensor 207. The optical axis of camera 111 is parallel to the x-axis. The optical axis is split into the x-axis and y-axis by the prism 203.

[0031] The housing 201 houses the prism 203, the telephoto lens 204, the image sensor 205, the wide-angle lens 206, and the image sensor 207. A condensing lens 202 is provided on the front surface 201F of the camera 111. The optical axis of the condensing lens 202 is parallel to the x-axis. The condensing lens 202 emits incident light from the outside into the housing 201.

[0032] If camera 111 is a right front camera 111RF, a right front camera marker 121RF is provided on the left side 201L of the housing 201. If camera 111 is a right rear camera 111RB, a right rear camera marker 121RB is provided on the right side 201R of the housing 201.

[0033] If camera 111 is a left front camera 111LF, a left front camera marker 121LF is provided on the right side 201R of the housing 201. If camera 111 is a left rear camera 111LB, a left rear camera marker 121LB is provided on the left side 201L of the housing 201.

[0034] The prism 203 splits the incident light from the condensing lens 202 into transmitted light and reflected light. The prism 203 emits the transmitted light to the telephoto lens 204 and the image sensor 205. The prism 203 emits the reflected light to the wide-angle lens 206 and the image sensor 207.

[0035] The telephoto lens 204 has a field of view 210 (for example, 40 degrees) that is narrower than the field of view 220 (for example, 190 degrees) of the wide-angle lens 206. The image sensor 205 receives the transmitted light from the prism 203 through the telephoto lens 204, converts it into photoelectric light, and outputs the image signal of the telephoto image.

[0036] The wide-angle lens 206 has a field of view 220 that is wider than the field of view 210 of the telephoto lens 204. The image sensor 207 receives the reflected light from the prism 203 through the wide-angle lens 206, converts it into photoelectric light, and outputs an image signal for a wide image. Note that the positions of the telephoto lens 204 and the wide-angle lens 206 may be reversed.

[0037] <Figure 3: Example of Calibration Device Configuration> Figure 3 is a block diagram showing an example configuration of a calibration device. First, let's describe the hardware configuration example. The calibration device 300 includes a processor 301, a storage device 302, an operating device 303, a display device 304, an LSI (Large Scale Integration) 305, and a communication IF (Interface) 306. These are connected by a bus 307.

[0038] The processor 301 controls the calibration device 300. The memory device 302 serves as the work area for the processor 301.

[0039] The storage device 302 is a non-temporary or temporary recording medium for storing various programs and data. Examples of storage devices 302 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory.

[0040] Specifically, for example, the memory device 302 stores the global coordinate values ​​of the camera 111 in the global coordinate system 100, the global coordinate values ​​of the feature points of the camera marker 121, the global coordinate values ​​of the feature points of the projection marker 122, the global coordinate values ​​of the feature points of the projection marker 122, and the global coordinate values ​​of the feature points of the rear marker 123. These global coordinate values ​​are referred to as global reference position information. The memory device 302 also stores the roll angle, pitch angle, and yaw angle, which indicate the rotation direction when the optical axis of the camera 111 in the global coordinate system 100 is the X-axis direction, as global reference attitude information.

[0041] Furthermore, the memory device 302 stores the local coordinate values ​​of the camera 111 in the local coordinate system 200, the local coordinate values ​​of the feature points of the camera marker 121, the local coordinate values ​​of the feature points of the projection marker 122, the local coordinate values ​​of the feature points of the projection marker 122, and the local coordinate values ​​of the feature points of the rear marker 123. These local coordinate values ​​are referred to as local reference position information. In addition, the memory device 302 stores the roll angle α, yaw angle β, and pitch angle γ, which indicate the rotation direction when the optical axis of the camera 111 in the local coordinate system 200 is the X-axis direction, as local reference attitude information.

[0042] The memory device 302 stores transformation matrices that perform translation and rotation of coordinate values ​​between the global coordinate system 100 and the local coordinate system 200. The memory device 302 also stores the intrinsic parameters of the camera 111 (intrinsic parameter matrix K, described later) that define the focal length, image center coordinate values, and lens distortion coefficient.

[0043] The operating device 303 accepts operation input. Examples of the operating device 303 include buttons, switches, and touch panels.

[0044] The display device 304 displays image data captured and processed by the camera 111. Specifically, for example, the display device 304 displays an image in which a processed telephoto image is embedded in the center of a processed wide image. A display device 304 may be provided for each camera 111.

[0045] The LSI305 is an integrated circuit that performs specific image processing tasks such as color interpolation, white balance adjustment, edge enhancement, gamma correction, tone conversion, debayering, positive-negative cylindrical transformation, and parallel projection, as well as encoding, decoding, and compression / decompression.

[0046] The communication IF306 communicates with the camera 111, the projector 112, and the ECU (Electronic Control Unit) 310 located in the track 101.

[0047] Next, an example of a functional configuration will be described. The calibration device 300 is composed of an acquisition unit 311, a conversion unit 312, an extraction unit 313, a calculation unit 314, and an image processing unit 315. Specifically, the acquisition unit 311, the conversion unit 312, the extraction unit 313, the calculation unit 314, and the image processing unit 315 are functions realized, for example, by having a processor execute a program stored in a storage device 302, or by an LSI 305.

[0048] The acquisition unit 311 acquires the image to be calibrated from the camera 111. Specifically, for example, the acquisition unit 311 acquires a wide image captured by the image sensor 207 of the camera 111 from the storage device 302. The acquisition unit 311 may also acquire a telephoto image captured by the image sensor 205 of the camera 111 from the storage device 302 as the image to be calibrated. In this embodiment, the wide image will be described as the image to be calibrated.

[0049] The conversion unit 312 converts the wide image acquired by the acquisition unit 311 into a perspective projection image. The extraction unit 313 extracts the feature points of the marker from the perspective projection image from the conversion unit 312. The calculation unit 314 calculates the rotation matrix and translation vector of the marker as seen from the camera 111 as calibration information.

[0050] The marker's rotation matrix represents the change in attitude (Δα, Δβ, Δγ) from the camera 111's reference attitude information. The marker's translation vector represents the change in position (Δx, Δy, Δz) from the camera 111's reference position information.

[0051] The image processing unit 315 performs image processing using the calibration information. Specifically, for example, after calculating the calibration information, the image processing unit 315 corrects the perspective projection image obtained from the conversion unit 312 to a perspective projection image taken with the camera 111 in a reference orientation by substituting the rotation matrix into a known distortion correction alignment function. The distortion correction alignment function performs a transformation that combines a distortion correction transformation and a parallelization transformation, and outputs a map corresponding to the pixel position of each pixel in the perspective projection image obtained from the conversion unit 312. The image processing unit 315 refers to this map and corrects the perspective projection image obtained from the conversion unit 312 to a perspective projection image taken with the reference orientation.

[0052] Furthermore, the image processing unit 315 corrects the coordinate values ​​of objects in the perspective projection image after the above correction using translation vectors. The image processing unit 315 may be implemented in the ECU 310 instead of the calibration device 300.

[0053] In this way, the image processing unit 315 corrects the image obtained from the camera 111 to cancel out the amount of attitude change (Δα, Δβ, Δγ), thereby generating a perspective projection image taken at a reference attitude, as if the attitude of the camera 111 had actually been calibrated. Furthermore, the image processing unit 315 identifies the subject and its distance within the perspective projection image using known image recognition, and corrects the distance of the subject using the amount of position change (Δx, Δy, Δz). This makes it possible to measure the distance to the subject as if the position of the camera 111 had actually been calibrated.

[0054] <Figure 4: Example of image change due to change in the position of camera 111> Figure 4 is an explanatory diagram showing an example of image change due to a change in the position of camera 111. Camera 111 captures images of the subject at the origin o, which is the reference position (x, y, z).

[0055] (A) shows the change in the subject image 400 captured by camera 111 when camera 111 moves along the x-axis. Subject image 400 is the image when camera 111 is located at the origin o. When camera 111 moves Δx in the positive x-axis direction from the origin o, the subject image 400 captured by camera 111 changes to a larger subject image 401. When camera 111 moves Δx in the negative x-axis direction from the origin o, the subject image 400 captured by camera 111 changes to a smaller subject image 402.

[0056] (B) shows the change in the subject image 400 captured by camera 111 when camera 111 moves in the z-axis direction. When camera 111 moves Δz in the positive z-axis direction from the origin o, the subject image 400 captured by camera 111 changes to a trapezoidal subject image 411 with a longer upper base in the x-axis direction. When camera 111 moves Δz in the negative z-axis direction from the origin o, the subject image 400 captured by camera 111 changes to a trapezoidal subject image 412 with a longer lower base in the x-axis direction.

[0057] (C) shows the change in the subject image 400 captured by camera 111 when camera 111 moves in the y-axis direction. When camera 111 moves Δy in the positive y-axis direction from the origin o, the subject image 400 captured by camera 111 changes to a trapezoidal subject image 411 with a longer upper base in the y-axis direction. When camera 111 moves Δy in the negative y-axis direction from the origin o, the subject image 400 captured by camera 111 changes to a trapezoidal subject image 412 with a longer lower base in the y-axis direction.

[0058] <Figure 5: Example of image change due to change in the posture of camera 111> Figure 5 is an explanatory diagram showing an example of image change due to a change in the attitude of camera 111. The rotation angle around the x-axis is called the roll angle α. The rotation angle around the z-axis is called the yaw angle β. The rotation around the y-axis is called the pitch angle γ.

[0059] (A) shows the change in the subject image 400 captured by the camera 111 when the camera 111 rotates around the x-axis. When the camera 111 rotates Δα around the x-axis, the subject image 400 also rotates Δα and becomes the subject image 501.

[0060] (B) shows the change in the subject image 400 captured by the camera 111 when the camera 111 rotates around the z axis. When the camera 111 rotates Δβ around the z axis, the subject image 400 moves in the y-axis direction within the imaging area 500 to become the subject image 502.

[0061] (C) shows the change in the subject image 400 captured by the camera 111 when the camera 111 rotates around the y-axis. When the camera 111 rotates Δγ around the y-axis, the subject image 400 moves in the z-axis direction within the imaging area 500 to become the subject image 503.

[0062] <Figure 6 Camera Calibration Procedure> Figure 6 is a flowchart showing an example of the camera calibration process procedure using the calibration device 300.

[0063] (Step S601) The calibration device 300 determines whether or not it has received a system start instruction from the ECU 310. The system start instruction is sent from the ECU 310 to the calibration device 300 when the ECU 310 detects engine start based on the driver's operation. Alternatively, the system start instruction may also be sent when the calibration device 300 is connected to the camera 111 and the ECU 310 and becomes able to communicate, for example, when the calibration device 300 is installed on the truck 101. If the calibration device 300 has not received a system start instruction (step S601: No), it waits for a system start instruction from the ECU 310. If the calibration device 300 has received a system start instruction (step S601: Yes), it proceeds to step S602.

[0064] (Step S602) The calibration device 300 reads reference information that serves as the basis for calibration. Specifically, for example, the calibration device 300 reads, for each camera 111, the local reference position information and local reference attitude information of camera 111, the local reference position information of camera marker 121, the local reference position information of projection marker 122, and the local reference position information of rear marker 123. The calibration device 300 also reads, for each camera 111, the internal parameters of camera 111.

[0065] (Step S603) The calibration device 300 performs calibration information calculation processing for each camera 111. Details of the calibration information calculation processing (step S603) will be described later in Figure 7.

[0066] (Step S604) The calibration device 300 determines whether or not it is time for calibration. The calibration timing may be a predetermined time after the previous calibration information calculation process (step S603), or it may be the timing when a predetermined variation occurs in the track 101. A variation signal indicating a predetermined variation in the track 101 is notified to the calibration device 300 from the ECU 310. The predetermined variation is, for example, a change in the load weight of the track 101 detected by a weight sensor, or the detection of an acceleration above a certain level detected by an acceleration sensor. Note that the calibration timing may be different for each camera 111.

[0067] Furthermore, the projection of the projection marker 122 by the projector 112 may be limited to specific timings, such as when the track 101 is stopped.

[0068] (Step S605) The calibration device 300 determines whether or not it has received a system termination instruction from the ECU 310. The system termination instruction is sent from the ECU 310 to the calibration device 300 when the ECU 310 detects that the engine has stopped due to the driver's operation. If the system termination instruction has not been received (step S605: No), the process returns to step S604. If the system termination instruction has been received (step S605: Yes), the process proceeds to step S606.

[0069] (Step S606) The calibration device 300 performs calibration information calculation processing for each camera 111. The calibration information calculation processing (step S605) is the same process as the calibration information calculation processing (step S603). Details of the calibration information calculation processing (step S605) will be described later in Figure 7. This completes the camera calibration process. When the camera 111 takes an image after the camera calibration process is completed, the image processing unit 315 performs calibration by correcting the image obtained from the camera 111 to cancel out the calibration information, as if the position and orientation of the camera 111 had actually been calibrated.

[0070] <Figure 7 Calibration information calculation process (steps S603, S606)> Figure 7 is a flowchart showing a detailed example of the calibration information calculation process (steps S603, S606).

[0071] (Step S701) The calibration device 300 acquires a wide-angle image including the marker captured by the camera 111 using the acquisition unit 311. For example, if the right front camera 111RF is used, the calibration device 300 projects a projection marker 122 using the projector 112. The calibration device 300 then captures a wide image including the left front camera marker 121LF, the left rear camera marker 121LB, and the projection marker 122.

[0072] Furthermore, if the right rear camera 111RB is selected, for example, the calibration device 300 will illuminate the right rear marker 123R. The calibration device 300 will capture a wide image including the left rear camera marker 121LB and the right rear marker 123R.

[0073] Furthermore, if the left front camera 111LF is used, for example, the calibration device 300 projects a projection marker 122 using the projector 112. The calibration device 300 then captures a left front wide image 131LF that includes the right front camera marker 121RF, the right rear camera marker 121RB, and the projection marker 122.

[0074] Furthermore, if the left rear camera 111LB is selected, for example, the calibration device 300 will illuminate the left rear marker 123L. The calibration device 300 will capture a wide image including the right rear camera marker 121RB and the left rear marker 123L.

[0075] Furthermore, if the rear camera is 111MB, the calibration device 300 will illuminate the rear marker 123. The calibration device 300 will capture a wide image including the rear marker 123.

[0076] (Step S702) The calibration device 300 converts the wide image captured in step S701 into a perspective projection image using the conversion unit 312. An example of conversion from a wide image to a perspective projection image will now be described.

[0077] [Figure 8: Example of conversion from widescreen image to perspective projection image] Figure 8 is an explanatory diagram showing an example of conversion from a wide image to a perspective projection image. In Figure 8, the case where the left front camera 111LF captures the right front camera marker 121RF and the right rear camera marker 121RB is used as an example. (A) shows the state in which transmitted light from the prism 203 is imaged by the wide-angle lens 206.

[0078] (B) shows the left front wide image 131LF input from the image sensor 207 and debayered by the imaging in (A). The left front wide image 131LF includes a wide image 801 of the right front camera marker 121RF and a wide image 802 of the right rear camera marker 121RB.

[0079] (C) shows the equirectangular transformation of the left front wide image 131LF. The left front equirectangular transformed image 800LF is the image obtained by equirectangular transformation from the left front wide image 131LF. The left front equirectangular transformed image 800LF includes the equirectangular transformed image 811 of the right front camera marker 121RF and the equirectangular transformed image 812 of the right rear camera marker 121RB.

[0080] (D) shows the process of pasting the left-front equirectangular transformed image 800LF onto the virtual spherical surface 804 of the virtual sphere 803. (E) shows parallel projection onto the virtual spherical surface 804. The left-front equirectangular transformed image 800LF is pasted onto the virtual spherical surface 804. By performing parallel projection onto the left-front equirectangular transformed image 800LF from the projection direction (for example, the direction of the optical axis), the left-front perspective projection image 810LF is extracted from the virtual spherical surface 804 within a predetermined extraction range 805.

[0081] The left front perspective projection image 810LF includes the perspective projection image 821 of the right front camera marker 121RF and the perspective projection image 822 of the right rear camera marker 121RB. In this way, the conversion from the left front wide image 131LF to the left front perspective projection image 810LF is performed. The left front perspective projection image 810LF is an image-processed wide image.

[0082] (Step S703) Returning to Figure 7, the calibration device 300, using the extraction unit 313, extracts feature points of the perspective projection image of each marker on the two-dimensional perspective projection image using an existing feature point extraction algorithm. Let p1 to pn be the n feature points of the perspective projection image of the markers on the two-dimensional perspective projection image. n is an integer greater than or equal to 4. Let pi be an arbitrary feature point (where i is an integer satisfying 1 ≤ i ≤ n).

[0083] (Step S704) The calibration device 300 calculates the rotation matrix R and translation vector t of the marker as seen from the camera 111. Specifically, for example, the calibration device 300 calculates the rotation matrix R and translation vector t by minimizing the objective function E in equation (1) below.

[0084]

number

[0085] In equation (1) above, the function proj() on the right-hand side is a function that projects a 3D coordinate point onto a 2D image plane. K is the intrinsic parameter matrix of camera 111. Pwi is the 3D coordinate value of the marker. If Pwi is the 3D coordinate value in global coordinate system 100, the resulting rotation matrix R and translation vector t will be the rotation matrix R and translation vector t in global coordinate system 100. If Pwi is the 3D coordinate value in local coordinate system 200, the resulting rotation matrix R and translation vector t will be the rotation matrix R and translation vector t in local coordinate system 200.

[0086] The calibration device 300 searches for the rotation matrix R and translation vector t that minimize the value of the objective function E, and outputs the searched rotation matrix R and translation vector t to the image processing unit 315. The output rotation matrix R represents the changes in roll angle α, yaw angle β, and pitch angle γ (Δα, Δβ, Δγ) in the local reference attitude information of the camera 111. The output translation vector t represents the changes in the three-dimensional coordinate values ​​(x, y, z) of the camera (Δx, Δy, Δz) in the local reference position information of the camera 111.

[0087] Furthermore, a single camera 111 may capture images of multiple markers. For example, the left front camera 111LF captures a left front wide image 131LF that includes the right front camera marker 121RF and the right rear camera marker 121RB. In such cases, the calibration device 300 calculates a rotation matrix R and a translation vector t for each marker.

[0088] In this case, the calibration device 300 calculates the average value of the rotation matrix R and the average value of the translation vector t.

[0089] Furthermore, if one camera 111 captures images of multiple markers, the calibration device 300 may search for the rotation matrix R and translation vector t that minimize the value of the objective function E by substituting the feature points p1 to pn of each of the multiple markers and the 3D coordinate values ​​of each of the multiple markers into equation (1) above. This makes it possible to obtain calibration information that shows the balanced amount of position change and attitude change across the multiple markers.

[0090] <Figure 9 Display Image> Figure 9 is an explanatory diagram showing an example of a display image on the display device 304. Figure 9 shows the left front display image 900LF when captured by the left front camera 111LF. The left front display image 900LF has the left front perspective projection image 810LF described above and the left front telephoto image 901LF which is synthesized by image processing. In this way, the telephoto image and the wide image are displayed on a single screen.

[0091] As described above, according to this embodiment, the position and orientation of the cameras 111 mounted on the truck 101 can be autonomously calibrated. Specifically, for example, real-time calibration is possible while the truck 101 is in operation. Therefore, it is possible to respond even if the position and orientation of individual cameras 111 change during operation due to the presence or absence of cargo or the aging of the truck 101.

[0092] Furthermore, calibration using a dedicated chart requires specialized equipment, making periodic calibration difficult. However, in this embodiment, real-time calibration during operation is possible, eliminating the need for such equipment.

[0093] Furthermore, in camera 111, a telephoto lens 204 and a wide-angle lens 206 are positioned on the optical axis. Therefore, calibration information obtained using the wide-angle lens 206 can be directly applied to the image captured by the image sensor 205 via the telephoto lens 204. Consequently, there is no need to calculate different calibration information for each image from image sensors 205 and 207. Therefore, the computational load of calibration information can be reduced.

[0094] Furthermore, the calibration device 300 described above may also perform the removal of abnormal data. Specifically, for example, the calibration device 300 may determine whether the rotation matrix R and translation vector t calculated by the calculation unit 314 are abnormal data, and if they are abnormal data, it may discard them without outputting them to the image processing unit 315. In this case, for example, the calibration device 300 determines that the rotation matrix R is abnormal data if at least one of the elements Δα, Δβ, and Δγ of the rotation matrix R is greater than or equal to a threshold value. Also, for example, the calibration device 300 determines that the rotation matrix R is abnormal data if at least one of the elements Δx, Δy, and Δz of the translation vector t is greater than or equal to a threshold value.

[0095] If the calibration device 300 determines that at least one of the rotation matrix R and the translation vector t is abnormal data, it may discard both the rotation matrix R and the translation vector t as abnormal data. This makes it possible to suppress adverse effects on image data correction and coordinate value measurement due to abnormal data.

[0096] It should be noted that the present invention is not limited to the above, and may be combined in any way. Furthermore, other embodiments that can be conceivable within the scope of the technical idea of ​​the present invention are also included in the scope of the present invention. [Explanation of Symbols]

[0097] 100 Global coordinate system, 101 Truck, 102 Tractor, 103 Trailer, 110 Side mirror, 111 Camera, 112 Projector, 121 Camera marker, 122 Projection marker, 123 Rear marker, 200 Local coordinate system, 201 Housing, 202 Focusing lens, 203 Prism, 204 Telephoto lens, 205, 207 Image sensor, 206 Wide-angle lens, 210, 220 Field of view, 300 Calibration device, 301 Processor, 302 Memory device, 311 Acquisition unit, 312 Transformation unit, 313 Extraction unit, 314 Calculation unit, 315 Image processing unit

Claims

1. A calibration system comprising an imaging device mounted on a moving body, and a calibration device for calibrating the position and orientation of the imaging device, The aforementioned calibration device is A storage unit that stores position information of a marker that serves as a calibration reference for the imaging device, An acquisition unit that acquires an image of a subject including the marker from the imaging device, A calculation unit calculates the amount of change in the position and orientation of the imaging device based on the position information of the marker and the image of the marker acquired by the acquisition unit, A calibration system having the following features.

2. A calibration system according to claim 1, The marker is provided on another imaging device provided on the moving body. Calibration system.

3. A calibration system according to claim 1, The marker is projected from the moving body. Calibration system.

4. A calibration system according to claim 1, The marker is provided on the moving body, Calibration system.

5. A calibration system according to any one of claims 1 to 4, It has an extraction unit that extracts feature points from the image of the marker, The calculation unit calculates the amount of change in the position and orientation of the imaging device based on the position information of the marker and the feature points of the image of the marker acquired by the extraction unit. Calibration system.

6. A calibration system according to claim 1, The imaging device captures images of a plurality of the markers, The storage unit stores the position information of each of the multiple markers, The acquisition unit acquires an image of the subject including a plurality of markers from the imaging device. The calculation unit calculates the amount of change in the position and orientation of the imaging device based on the position information of each of the plurality of markers and the images of the plurality of markers acquired by the acquisition unit. Calibration system.

7. A calibration system according to any one of claims 1 to 6, The imaging device comprises a plurality of lenses with different angles of view arranged coaxially, and a plurality of image sensors that receive light from each of the plurality of lenses. Calibration system.

8. A calibration device for calibrating the position and orientation of an imaging device installed on a moving object, A storage unit that stores position information of a marker that serves as a calibration reference for the imaging device, An acquisition unit that acquires an image of a subject including the marker from the imaging device, A calculation unit calculates the amount of change in the position and orientation of the imaging device based on the position information of the marker and the image of the marker acquired by the acquisition unit, A calibration device having the following features.

9. A calibration program that causes a processor to perform calibration of the position and orientation of an imaging device installed on a moving object, The aforementioned processor, An acquisition process to acquire an image of a subject containing a marker that serves as a calibration standard for the imaging device from the imaging device, A calculation process that calculates the amount of change in the position and orientation of the imaging device based on the position information of the marker and the image of the marker obtained by the acquisition process, A calibration program that executes the following:

Images