Camera Calibration Device, Overhead Line Height Measurement Device, and Camera Calibration Method

Abstract

To enable proper calibration to be performed under a high-speed environment having no line sensor camera.SOLUTION: A camera calibration device is provided, comprising: an image coordinate calculation unit which calculates checker image coordinate data being intersection coordinates of 4 squares of a checker board on the basis of an image of the checker board in which each square has a known value; an internal parameter calculation unit which calculates an internal parameter of an area camera on the basis of the checker image coordinate data, a lens initial parameter, and checker true value coordinates; an external parameter calculation unit which calculates an external parameter of the area camera on the basis of the checker image coordinate data, the internal parameter, and the checker true value coordinates; an ROI range calculation unit which sets an ROI range in such a manner that an ideal imaging line in the acquired image is superimposed on a rectangular diagonal line indicating the ROI range; and a line image conversion unit which converts into a line image the ROI image being the image of the ROI range.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a camera calibration device, an overhead line height measurement device using the camera calibration device, and a camera calibration method in the field of image processing.

Background Art

[0002] Conventionally, calibration techniques have been proposed that enable measurement in a high-speed environment using a camera installed in an environment where camera installation is limited, such as on the roof of a railway vehicle, or a camera installed in a high-speed train or a commercial vehicle during high-speed travel. Note that calibration of the internal parameters of a camera regarding the focal length, principal point coordinates, and distortion may be referred to as lens calibration.

[0003] Non-Patent Document 1, which is an example of the prior art, discloses a technique for obtaining external parameters for each photographing and internal parameters of a camera by photographing a planar marker at various positions and postures using an area camera. Patent Document 1, which is an example of the prior art, discloses a technique that enables calibration using a line sensor camera by applying an L-shaped marker to the technique of Non-Patent Document 1. Patent Document 2, which is an example of the prior art, discloses a technique that enables stereo calibration of a line sensor camera by applying the technique of Patent Document 1.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Patent Document

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the above prior art has the following problems. According to the technique disclosed in Non-Patent Document 1, lens calibration of an area camera is possible, but the line sensor camera has not been discussed. Also, according to the techniques disclosed in Patent Documents 1 and 2, calibration of a line sensor camera in a high-speed environment is possible, but the information on the X-axis, Z-axis, and pan angle obtained by the line sensor camera can only be simplified.

[0007] The present invention has been made in view of the above, and an object thereof is to provide a technique that enables appropriate camera calibration in a high-speed environment without providing a line sensor camera.

Means for Solving the Problems

[0008] One aspect of the present invention that solves the above problems and achieves the object is, in order to convert from camera coordinates to world coordinates, based on a plurality of images captured by an area camera of a checkerboard in which each square with different colors arranged alternately is installed at known world coordinates serving as a reference, a camera calibration device that calculates internal parameters and external parameters for calibration of the area camera, comprising: a storage unit; an image input unit to which an image acquired by the area camera is input; an image coordinate calculation unit that calculates checker image coordinate data, which is the intersection coordinates of four squares of the checkerboard, based on the image of the checkerboard input to the image input unit; an internal parameter calculation unit that calculates the internal parameters of the area camera based on the checker image coordinate data, the lens initial parameters pre-stored in the storage unit, and the checker true coordinates; an external parameter calculation unit that calculates the external parameters of the area camera based on the checker image coordinate data, the internal parameters, and the checker true coordinates; an ROI range calculation unit that sets the ROI range so that an ideal imaging line in the image input to the image input unit overlaps with the diagonal of a rectangle indicating the ROI range; and a line image conversion unit that converts an ROI image, which is an image of the ROI range, into a line image.

[0009] In the camera calibration device having the above configuration, it is preferable that the ROI range calculation unit includes a filter that removes noise in the image acquired by the area camera, and the ROI range is set by being enlarged for the filter.

[0010] In the camera calibration device having the above configuration, two area cameras are provided, and the ROI range calculation unit can perform stereo calibration by making the ideal imaging lines of the two area cameras common.

[0011] Alternatively, one aspect of the present invention is an overhead line height measuring device that calibrates the area camera by the camera calibration device having the above configuration and measures the height of an overhead line connected to a vehicle on which the area camera is installed based on an image acquired by the area camera.

[0012] Alternatively, one aspect of the present invention is a camera calibration method for calculating internal parameters and external parameters for calibration of the area camera based on a plurality of images obtained by imaging, with an area camera, a checkerboard in which squares of different colors are alternately arranged and each square has a known value, and which is installed at a reference known world coordinate for converting from camera coordinates to world coordinates. The method includes: inputting an image acquired by the area camera; calculating checker image coordinate data, which are intersection coordinates of four squares of the checkerboard, based on the input image of the checkerboard; calculating the internal parameters of the area camera based on the checker image coordinate data, lens initial parameters stored in advance, and checker true coordinates; calculating the external parameters of the area camera based on the checker image coordinate data, the internal parameters, and the checker true coordinates; setting the ROI range so that an ideal imaging line in the input image overlaps a diagonal line of a rectangle indicating the ROI range; and converting an ROI image, which is an image of the ROI range, into a line image.

Advantages of the Invention

[0013] According to the present invention, appropriate camera calibration can be achieved in a high-speed environment without providing a line sensor camera.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0015] Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the description of the following embodiments.

[0016] <Embodiment 1> Figure 1 is a diagram showing a vehicle 1 equipped with the calibration device 4 according to the present embodiment. On the roof of the vehicle 1 shown in FIG. 1, one area camera 2 and lighting 3 are installed, and the vehicle 1 is equipped with a calibration device 4. The area camera 2 acquires an image of a checkerboard, which is a reference device during calibration, and acquires an image of the overhead line when measuring the height of the overhead line connected to the vehicle 1. The lighting 3 projects light onto the imaging area of the area camera 2. In FIG. 1, the number of installed lighting 3 is one, but the number of installations is not limited to this. The calibration device 4 calibrates the area camera 2. The overhead line height measuring device 5 calibrates the area camera 2 by the calibration device 4, and measures the height of the overhead line connected to the vehicle 1 on which the area camera 2 is installed based on the image acquired by the area camera 2.

[0017] Figure 2 is a diagram showing the camera axis of the area camera 2. The camera axis shown in FIG. 2 is defined in a right-handed coordinate system. The X-axis direction is the sleeper direction (right direction of the traveling direction), the Y-axis direction is the track direction (traveling direction), and the Z-axis direction is the height direction (upward direction). The rotation angle with the X-axis as the rotation axis is defined as tilt, the rotation angle with the Y-axis as the rotation axis is defined as pan, and the rotation angle with the Z-axis as the rotation axis, that is, the angle around the optical axis of the camera, is defined as roll.

[0018] Figure 3 is a diagram showing the area camera 2 and the checkerboard 6 used for calibration. The checkerboard 6 shown in FIG. 3 is a reference device with known values, in which white squares and black squares are alternately arranged, and is installed in a known world coordinate as a reference for converting from camera coordinates to world coordinates. Here, a checkerboard 6 with white and black squares arranged alternately is shown, but the present invention is not limited to this. The checkerboard 6 only needs to have squares of different colors that can be distinguished by the area camera 2 arranged alternately.

[0019] FIG. 4 is a functional block diagram showing the configuration of the calibration device 4 according to the present embodiment. The calibration device 4 shown in FIG. 4 includes a storage unit 40, an image input unit 41, an image coordinate calculation unit 42, an internal parameter calculation unit (nonlinear least squares unit) 43, an external parameter calculation unit 44, an ROI range calculation unit 45, and a line image conversion unit 46.

[0020] FIG. 5 is a flowchart showing the calibration method of the calibration device 4 according to the present embodiment.

[0021] The storage unit 40 stores data output from each component in the calibration device 4. Note that the storage unit 40 stores in advance the lens initial parameters and checker true coordinates [mm] used in the internal parameter calculation unit 43 or the external parameter calculation unit 44.

[0022] The image input unit 41 outputs the image acquired by the area camera 2 and input to the storage unit 40 (S1). Here, the image at the time of calibration is an image of the checkerboard 6 (checkerboard image).

[0023] The image coordinate calculation unit 42 calculates checker image coordinate data [pix], which are the intersection coordinates [pix] of the four white and black squares of the checkerboard 6 where each square has a known value and are necessary for camera calibration, based on the checkerboard image stored in the storage unit 40, and outputs it to the storage unit 40 (S2). The checker image coordinate data [pix] can be calculated by the corner detection method, which is a known feature detection technique. Examples of corner detection methods include the Moravec corner detection method, Harris corner detection method, Shi-Tomasi corner detection method, and FAST corner detection method.

[0024] Based on the checker image coordinate data [pix], the lens initial parameters pre-stored in the storage unit 40, and the checker true value coordinates [mm], the internal parameter calculation unit 43 calculates the internal parameters of the area camera 2 and outputs them to the storage unit 40 (S3). Here, the internal parameters are calculated by a known technique disclosed in Non-Patent Document 1. Note that the processes of S1 to S3 are performed for the number of photographed images.

[0025] Based on the checker image coordinate data [pix], the internal parameters, and the checker true value coordinates [mm], the external parameter calculation unit 44 calculates the external parameters of the area camera 2 and outputs them to the storage unit 40 (S4). Here, the external parameters are calculated with the internal parameters calculated by a known technique disclosed in Non-Patent Document 1 as the initial values. Since the calculated external parameters are the external parameters in the camera coordinate system, they are converted into the external parameters in the world coordinate system of the checkerboard 6 installed in the reference world coordinates.

[0026] The ROI range calculation unit 45 calculates and sets the ROI range using the internal parameters and external parameters stored in the storage unit 40, and outputs them to the storage unit 40 (S5). Here, the ROI range calculation unit 45 sets the ROI range so that even if the deviation from the ideal imaging line of the actual imaging line is considered, the imaging can be performed, and the ideal imaging line in the image overlaps with the diagonal line of the rectangle indicating the ROI range. Note that in the ideal imaging line, the actual imaging line of the camera (sleeper direction) and the traveling direction of the vehicle 1 (rail direction) are perpendicular. However, due to the design system of fixtures (not shown), the actual imaging line of the installed camera has a slight angular deviation of about several degrees and a slight positional deviation of about several millimeters with respect to the ideal imaging line, and it is necessary to adjust this deviation.

[0027] FIG. 6 is a diagram showing an ideal imaging line and an actual imaging line. On the roof (plane) of the vehicle 1 in FIG. 6, an area camera 2 and two illuminations 3 arranged with the area camera 2 in between are installed. The dotted line shown in FIG. 6 indicates the actual imaging line, and the solid line indicates the ideal imaging line.

[0028] The conversion from the camera coordinate system of the area camera 2 to the world coordinate system is performed by the following formula (1).

[0029]

Equation

[0030] Here, the parameters in the above formula (1) are as follows. s: Scaling coefficient (u, v): Image position [pix] f x , f y : Focal length [pix] c x , c y : Pixel center position [pix] t1, t2, t3: Camera coordinates [mm] r 11 , r 12 , r 13 , r 21 , r 22 , r 23 , r 31 , r 32 , r 33 : Camera rotation matrix X: Target offset [mm] Y: Target depth [mm] Z: Target height [mm] Here, the focal length, the image center position, the camera coordinates, and the camera rotation matrix are known by the internal parameter calculation unit 43 and the external parameter calculation unit 44. Also, if the target depth Y is set as the world coordinate value (Ystandard) of the ideal imaging line and Z = 0, since the variables are only the target deviation X and the horizontal coordinate v of the image position, it is possible to calculate v when u = 0 and v when u = MAX. u = MAX is the maximum value of the horizontal pixels in the image.

[0031] FIG. 7 is a diagram showing an image that is deviated from the ideal imaging line and the ROI range in the image. In FIG. 7, the horizontal axis of the image that is deviated from the ideal imaging line is set as the u-axis, the vertical axis is set as the v-axis, and the ROI range is set so that the ideal imaging line overlaps with the diagonal line of the rectangle indicating the ROI range. The ideal imaging line extends from the position where u = 0 to the position where u = MAX within the rectangle indicating the ROI range. According to the above formula (1), as shown in FIG. 7, the ROI range can be set.

[0032] Next, the line image conversion process will be described. FIG. 8 is a flowchart showing the line image conversion process in the line image conversion unit 46 of the calibration device 4 according to the present embodiment. The image input unit 41 outputs an ROI image, which is an image of the ROI range acquired by the area camera 2 and input, to the storage unit 40 (S11).

[0033] The line image conversion unit 46 converts the ROI image stored in the storage unit 40 into a line image necessary for measurement (S12). The processes of S11 and S12 are performed for the number of photographed images.

[0034] FIG. 9 is a diagram showing the angle of deviation of the imaging line in the ROI image. As shown in Fig. 9, assuming that the angle of deviation of the imaging line is θ, the image acquired by the area camera 2 can be converted into the pixel information of the ideal imaging line according to the following formula (2).

[0035]

Equation

[0036] In this embodiment, as shown in Fig. 9, the imaging line is arranged obliquely with respect to the pixels, so that the resolution can be improved.

[0037] In the prior art, since the information acquired by the line sensor camera is only 1 pixel, it is difficult to adjust the area camera 2. According to this embodiment, by using the ROI function of the area camera, the area camera 2 can be adjusted, and appropriate camera calibration can be achieved in a high-speed environment without providing a line sensor camera.

[0038] Note that the area camera 2 can also be a camera capable of color photography. In the conventional line sensor camera, when performing color photography, the prism method is generally used, which leads to an increase in the cost and size of the device. Or, when performing color photography with a line sensor camera using a method other than the prism method, there are problems such as color bleeding in the outdoor environment. By using the area camera 2 as a camera capable of color photography, it is possible to obtain information equivalent to that of a color line sensor camera without causing an increase in the cost and size of the device.

[0039] <Embodiment 2> In this embodiment, a form in which the calibration device 4 according to Embodiment 1 is made more accurate will be described.

[0040] The calibration device according to this embodiment has the same configuration as that shown in Fig. 4. Also, the calibration method according to this embodiment is the same as the flowchart shown in FIG. 5. The difference between this embodiment and Embodiment 1 is that the ROI range calculation unit 45 is provided with a filter. The filter provided in the ROI range calculation unit 45 has an arbitrary filter size. The ROI range calculation unit 45 sets an expanded ROI range in consideration of this filter. The filter provided in the ROI range calculation unit 45 is not limited to a specific one as long as it is a filter capable of removing noise, and examples thereof include a smoothing filter, a Gaussian filter, a median filter, and a Sobel filter. In this embodiment, since it becomes possible to incorporate processing as an area image from the line image by expanding the ROI range on the v side, it becomes possible to use line information obtained by performing noise processing using information on the v side as an area camera.

[0041] According to this embodiment, in addition to Embodiment 1, it becomes possible to convert to pixel information with noise removed when converting from the pixel information of the area camera to line information. Therefore, according to this embodiment, high-precision pixels can be obtained.

[0042] <Embodiment 3> In this embodiment, a form enabling stereo calibration will be described.

[0043] FIG. 10 is a diagram showing a vehicle 1 equipped with a calibration device 4a according to this embodiment. On the roof of the vehicle 1 shown in FIG. 10, a first area camera 2a, a second area camera 2b, and a lighting 3 are installed, and the vehicle 1 is equipped with a calibration device 4a. The first area camera 2a and the second area camera 2b acquire an image of a checkerboard, which is a reference device, during calibration, and acquire an image of the overhead line when measuring the height of the overhead line connected to the vehicle 1. The illumination 3 projects light onto the imaging portions of the first area camera 2a and the second area camera 2b. In FIG. 10, the number of installed illuminations 3 is one, but the number of installations is not limited to this. The calibration device 4a calibrates the first area camera 2a and the second area camera 2b. The first area camera 2a and the second area camera 2b correspond to the area camera 2 in FIG. 1, and the calibration device 4a corresponds to the calibration device 4 in FIG. 1. Note that the vehicle 1 is also equipped with an overhead line height measurement device (not shown). The overhead line height measurement device calibrates the first area camera 2a and the second area camera 2b by the calibration device 4a, and measures the height of the overhead line connected to the vehicle 1 on which the first area camera 2a and the second area camera 2b are installed, based on the images acquired by the first area camera 2a and the second area camera 2b.

[0044] FIG. 11 is a functional block diagram showing the configuration of the calibration device 4a according to the present embodiment. The calibration device 4a shown in FIG. 11 includes a storage unit 40, an image input unit 41, an image coordinate calculation unit 42, an internal parameter calculation unit 43, an external parameter calculation unit 44a, an ROI range calculation unit 45, and a line image conversion unit 46. The calibration device 4a shown in FIG. 11 is different from the calibration device 4 shown in FIG. 4 in that it includes an external parameter calculation unit 44a instead of the external parameter calculation unit 44, and the other configurations are the same.

[0045] FIG. 12 is a flowchart showing the calibration method in the calibration device 4a according to the present embodiment. The flowchart shown in FIG. 12 is different from the flowchart shown in FIG. 5 in that it includes an external parameter calculation process S4a instead of the external parameter calculation process S4, and the other processes are the same.

[0046] The image input unit 41 outputs the images acquired and input by the first area cameras 2a and 2b to the storage unit 40 (S1). Then, in the same manner as in the first embodiment, image coordinate calculation processing (S2) and internal parameter calculation processing (S3) are performed. The external parameter calculation unit 44a calculates the external parameters of the first area cameras 2a and 2b and the external parameters between the cameras based on the checker image coordinate data [pix], the internal parameters, and the checker true value coordinates [mm], and outputs them to the storage unit 40 (S4a). The ROI range calculation unit 45 calculates and sets the ROI range using the internal parameters and external parameters stored in the storage unit 40, and outputs them to the storage unit 40 (S5). Here, the ROI range calculation unit 45 sets the ROI range so that even considering the deviation from the ideal imaging line of the actual imaging line, the ideal imaging line in the image overlaps with the diagonal line of the rectangle indicating the ROI range. The ideal imaging line in the first area camera 2a and the ideal imaging line in the second area camera 2b are the same, and measurements can be made on the common ideal imaging line for the two area cameras.

[0047] FIG. 13 is a flowchart showing the line image conversion process in the line image conversion unit 46 of the calibration device 4a according to the present embodiment. The flowchart shown in FIG. 13 is different from the flowchart shown in FIG. 8 in that the line image conversion process is performed for the number of cameras, and the other processes are the same.

[0048] According to the present embodiment, in addition to the first embodiment, by making the ideal imaging lines of the two area cameras a common line, stereo calibration can be performed.

[0049] Note that in the first to third embodiments, the camera calibration device and the camera calibration method have been described, but the present invention is not limited thereto. The calibration of one or more area cameras is performed by the camera calibration device described in Embodiments 1 to 3, and based on the images acquired by the area cameras, a catenary height measurement device that measures the height of the catenary connected to the vehicle on which the area cameras are installed is also an aspect of the present invention.

[0050] Note that the present invention is not limited to the above-described embodiments, and also includes various modified examples in which components are added, deleted, or converted with respect to the above-described configuration.

Explanation of Reference Numerals

[0051] 1 Vehicle 2 Area camera 2a First area camera 2b Second area camera 3 Lighting 4, 4a Calibration device 40 Storage unit 41 Image input unit 42 Image coordinate calculation unit 43 Internal parameter calculation unit 44, 44a External parameter calculation unit 45 ROI range calculation unit 46 Line image conversion unit 5 Catenary height measurement device 6 Checkerboard

Claims

1. In order to convert from camera coordinates to world coordinates, based on a plurality of images obtained by imaging, with an area camera, a checkerboard in which squares of different colors are alternately arranged and which is installed at a known reference world coordinate, each cell being a known value, an internal parameter and an external parameter for calibration of the area camera are calculated. A camera calibration device provided in a vehicle, a storage unit; an image input unit into which an image acquired by the area camera is input; an image coordinate calculation unit that calculates checker image coordinate data, which is the intersection coordinates of four cells of the checkerboard, based on the image of the checkerboard input to the image input unit; an internal parameter calculation unit that calculates the internal parameter of the area camera based on the checker image coordinate data, the lens initial parameters stored in advance in the storage unit, and the checker true coordinates; an external parameter calculation unit that calculates the external parameter of the area camera based on the checker image coordinate data, the internal parameter, and the checker true coordinates; an ROI range calculation unit that sets the ROI range so that an ideal imaging line, which is a sleeper direction perpendicular to the traveling direction of the vehicle, overlaps with the diagonal of a rectangle indicating the ROI range in the image input to the image input unit; A camera calibration device comprising a line image conversion unit that converts an ROI image, which is an image of the ROI range, into a line image.

2. The ROI range calculation unit includes a filter that removes noise in an image acquired by the area camera, The camera calibration device according to claim 1, wherein the ROI range is set by being enlarged for the filter.

3. Two area cameras are provided, The camera calibration device according to claim 1, wherein the ROI range calculation unit makes the ideal imaging lines of the two area cameras common.

4. A camera calibration device according to any one of claims 1 to 3; an area camera that is calibrated by the camera calibration device; A vehicle comprising an overhead line height measurement device configured to measure the height of an overhead line connected to the vehicle on which the area camera is installed, based on an image acquired by the area camera.

5. In order to convert from camera coordinates to world coordinates, a checkerboard with known values, in which squares of different colors are alternately arranged, is installed at known reference world coordinates. Based on a plurality of images captured by an area camera provided on a vehicle, a camera calibration method for calculating internal parameters and external parameters for calibration of the area camera, comprising: inputting an image acquired by the area camera; calculating checker image coordinate data, which are intersection coordinates of four squares of the checkerboard, based on the input image of the checkerboard; calculating internal parameters of the area camera based on the checker image coordinate data, lens initial parameters stored in advance, and checker true coordinates; calculating external parameters of the area camera based on the checker image coordinate data, the internal parameters, and the checker true coordinates; setting an ROI range in the input image such that an ideal imaging line, which is a sleeper direction perpendicular to the traveling direction of the vehicle, overlaps a diagonal line of a rectangle indicating the ROI range; converting an ROI image, which is an image of the ROI range, into a line image. A camera calibration method comprising the above steps is provided.

Images