Image capture device, correction device, correction method, and computer program

Abstract

The present invention provides an image capture device that comprises an optical element and an arithmetic circuit and that is capable of using at least two rotation axes to change an image capture direction. The optical element is disposed at a position away from an intersection point of the at least two rotation axes. The arithmetic circuit captures a plurality of images in different image capture directions, uses an image capture direction at the time of image capture of each of the plurality of images to determine the distance from a focal point of each image defined by the optical element to the intersection point, uses the distance to calculate an internal parameter of the image capture device, determines a correction parameter on the basis of the internal parameter, and uses the correction parameter to correct internal distortion of the image capture device.

Description

Photographing device, correction device, correction method, and computer program

[0001] The present disclosure relates to an imaging device, a correction device, a correction method, and a computer program for correcting internal distortion of the imaging device.

[0002] Images captured by a camera such as an imaging device may be affected by internal distortion inherent to the camera. To address this, camera calibration may be performed before the camera captures an image. One camera calibration method uses multiple images of the same subject captured from different locations by moving the camera (see, for example, Patent Document 1).

[0003] JP 2009-17480 A

[0004] However, as described above, when the camera is moved to capture images for camera calibration, the calibration requires many steps and time.

[0005] The present disclosure provides an imaging device, a correction device, a correction method, and a computer program that can easily correct internal distortion of the imaging device.

[0006] The imaging device disclosed herein includes an optical element and an arithmetic circuit, and is capable of changing an imaging direction around at least two rotation axes. The optical element is positioned away from an intersection of the at least two rotation axes. The arithmetic circuit captures multiple images at different imaging directions, calculates a distance from a focal point of each image defined by the optical element to the intersection using the imaging direction at the time of capture of each of the multiple images, calculates internal parameters of the imaging device using the distance, calculates correction parameters based on the internal parameters, and corrects internal distortion of the imaging device using the correction parameters.

[0007] Such general and specific aspects may be realized by a system, a method, and a computer program, as well as a combination thereof.

[0008] The imaging device, correction device, correction method, and computer program of the present disclosure can easily correct internal distortion of the imaging device.

[0009] 5A is a schematic diagram illustrating epipolar geometry. FIG. 5B is a schematic diagram illustrating epipolar geometry following FIG. 1A. FIG. 5C is a schematic diagram illustrating epipolar constraints. FIG. 5D is a block diagram illustrating the configuration of an image capture device according to a first embodiment. FIG. 5E is a schematic diagram illustrating the positional relationship between the pan rotation axis and tilt rotation axis and the image capture element. FIG. 5F is a schematic diagram illustrating the positional relationship between the pan rotation axis and tilt rotation axis and the image capture element following FIG. 4A. FIG. 5G is a schematic diagram illustrating the distance from the pan rotation axis and tilt rotation axis to the image capture element and the optical element. FIG. 5G is a schematic diagram illustrating an example in which the pan angle is changed from the state of FIG. 5A. FIG. 5G shows an example of an image captured by the image capture device. FIG. 5H shows another example of an image captured by the image capture device. FIG. 5H shows an example of extracting feature points from the image of FIG. 6A. FIG. 5H shows an example of extracting feature points from the image of FIG. 6B. FIG. 5H shows the positional relationship of each coordinate used to estimate internal parameters. FIG. 5H shows the positional relationship of each coordinate used to estimate internal parameters together with FIG. 8A. FIG. 5H shows the positional relationship of each coordinate used to identify the measurable distance limit. FIG. 5H shows the positional relationship of each coordinate used to identify the measurable distance limit together with FIG. 9A. 10 is a flowchart illustrating an example of processing executed by the imaging device. FIG. 11 is a block diagram illustrating the configuration of a correction device and an imaging device connected thereto according to a second embodiment. FIG.

[0010] [Embodiments] Hereinafter, embodiments of the present disclosure will be described using the drawings, with appropriate reference to the drawings. However, in the detailed description, unnecessary parts of the description of the prior art and substantially identical configurations may be omitted. This is for the sake of simplicity. Furthermore, the following description and the accompanying drawings are disclosed to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

[0011] The imaging device, correction device, correction method, and computer program according to the present disclosure correct internal distortion of the imaging device. In the following description, "internal distortion" may refer to distortion that occurs in an image due to, for example, distortion of a camera lens. The following equation (1) is generally known as an equation that represents a captured image and spatial coordinates.

[0012]

[0013] The imaging device, correction device, correction method, and computer program according to the present disclosure use the above formula (1) to determine external parameters and internal parameters from image coordinate points and world coordinate points. Here, "world coordinate point" refers to any point in the space in which the imaging device exists that is photographed by the imaging device. "Image coordinate point" refers to the coordinates on the image of the any point photographed by the imaging device. "Internal parameters" are parameters resulting from the lens of the imaging device, etc. Furthermore, "external parameters" are parameters that specify the position of the imaging device, etc. The imaging device, correction device, correction method, and computer program according to the present disclosure correct internal distortion using the determined internal parameters.

[0014] The imaging device, correction device, correction method, and computer program disclosed herein use epipolar geometry, which is used in photogrammetry technology, to correct internal distortion. Specifically, the imaging device, correction device, correction method, and computer program disclosed herein analyze images containing target points captured by cameras at multiple different positions and can determine the three-dimensional coordinates of the target points, the imaging positions, etc. First, a brief description of epipolar geometry will be provided.

[0015] "Epipolar geometry" is a geometry used in the field of computer vision, which involves capturing images of three-dimensional space using cameras at different positions. Figure 1A shows an example of capturing an object point P using two cameras at different positions. In Figure 1A, the coordinates of the optical centers of the two cameras are denoted by A and B, respectively. In the following description, the camera whose optical center is coordinate A is referred to as the first camera, and the camera whose optical center is coordinate B is referred to as the second camera. The object point P can have any coordinates in space. However, the object point P and coordinates A and B exist on the same plane. In epipolar geometry, the plane represented by the object point P and coordinates A and B is referred to as the "epipolar plane." The line where the epipolar plane intersects with each image plane is referred to as the "epipolar line." Furthermore, when the same object point P is captured by the first and second cameras, the object points lie on each other's epipolar lines on the image plane, which is referred to as the "epipolar constraint." 1B, when all we know is that "the target point P is captured at point p on the image plane when viewed from coordinate A of the first camera," it is unknown where the target point P is located among coordinates P, P1, and P2. By utilizing the epipolar constraint, when we know that "the same target point P is captured at point p' on the image plane when viewed from coordinate B of the second camera," it is possible to identify that the target point P is located at coordinate P.

[0016] As shown in FIG. 2, based on the concepts of epipolar constraint and triangulation, the geometric relationship between feature points p and p′ corresponding to target point P in images captured by the first and second cameras is defined as a fundamental matrix F.

[0017] Furthermore, the equation (2.1) holds due to the epipolar constraint condition.

[0018] p' Ｔ Ep=0 (2.1) Here, T is a transposed matrix, and E is a fundamental matrix (a rotation component R and a translation component t) that represents an epipolar constraint between corresponding feature points.

[0019] Furthermore, let K be the internal parameter of the first camera and K' be the internal parameter of the second camera. In this case, if m is the coordinate on the image plane of the first camera and m' is the coordinate on the image plane of the second camera, then m = Kp and m' = K'p'.

[0020] The above formula (2.1) can be expressed in coordinates on the image as follows: m' Ｔ K' －Ｔ Em' Ｔ K －１ m=0, m' Ｔ Fm=0. Therefore, the fundamental matrix F is expressed by equation (2.2).

[0021] F=K' －Ｔ EK －１ (2.2) Furthermore, the fundamental matrix F is expressed by equation (2.3) where R is the rotation component and t is the translation component.

[0022] F=K' －Ｔ tRK －１ (2.3) K, t, and R are unknown. Therefore, a fundamental matrix F or a fundamental matrix E is obtained using multiple images, and K, t, and R are estimated.

[0023] To find the fundamental matrix F, for example, an 8-point algorithm is used. The 8-point algorithm is a method for estimating the fundamental matrix F using eight or more corresponding points. In the 8-point algorithm, the fundamental matrix F is estimated by the least squares method from eight or more corresponding points using SVD. In the 8-point algorithm, the fundamental matrix E can be found based on the internal parameters using the estimated fundamental matrix F.

[0024] Furthermore, for example, a five-point algorithm is used to find the fundamental matrix E. The five-point algorithm is a method for directly estimating the fundamental matrix E using five or more corresponding points.

[0025] In epipolar geometry, the fundamental matrix E obtained as described above can be used to obtain external parameters that indicate the positions and orientations between the cameras.

[0026] [First Embodiment] <Photographing Device> As shown in FIG. 3 , the photographing device 1 according to the first embodiment is a camera with a motorized head that can change the horizontal direction, vertical direction, and zoom. A camera with a motorized head is also called a pan-tilt camera. For example, as shown in FIG. 3 , the photographing device 1 includes an optical mechanism 11, a drive mechanism 12, a storage device 13, and an arithmetic circuit 14. The photographing device 1 may also include an input device 15, an output device 16, and a communication circuit 17. The following description will be given assuming that the photographing device 1 can be driven to change its pan and tilt angles but its installation position is unchanged. Therefore, this disclosure will be described under the assumption that the pan and tilt angles of the photographing device 1 are changeable, but the position of the photographing device 1 itself is unchanged. Furthermore, a camera with a motorized head may also perform roll control in addition to pan and tilt control. However, for the sake of simplicity, the following description will be given assuming that the roll is unchanged.

[0027] The optical mechanism 11 includes a series of mechanisms related to capturing an image. The optical mechanism 11 acquires an image under control of the arithmetic circuit 14. Specifically, the optical mechanism 11 includes an image sensor 111, an optical element 112, an A / D converter 113, and the like.

[0028] The drive mechanism 12 drives the optical mechanism 11. The drive mechanism 12 also drives the image capturing device 1 itself, thereby controlling pan, tilt, and zoom.

[0029] The storage device 13 is a recording medium for recording various information. The storage device 13 is realized by, for example, RAM, ROM, flash memory, SSD (Solid State Drive), hard disk drive, other storage devices, or an appropriate combination thereof. The storage device 13 stores various data obtained by the shooting and correction processes in the imaging device 1. The storage device 13 also stores the correction program P1.

[0030] The arithmetic circuit 14 is a controller that controls the entire photographing device 1. For example, the arithmetic circuit 14 reads and executes a correction program P1 stored in the storage device 13. This allows the photographing device 1 to realize various processes related to the correction of internal parameters. The arithmetic circuit 14 may be various processors such as a CPU, an MPU, a GPU, an FPGA, a DSP, an ASIC, or a dedicated hardware circuit.

[0031] The input device 15 may be an operation button, a touch panel, a microphone, etc., which are used for inputting operations and data. The output device 16 may be a display, a speaker, a lamp, etc., which are used for outputting processing results and data.

[0032] The communication circuit 17 performs data communication with an external device when the photographing device 1 is connected to the external device. The data communication may be performed wired and / or wirelessly, for example, in accordance with a known communication standard. For example, wired data communication may be performed by using a communication controller of a semiconductor integrated circuit that operates in accordance with the Ethernet (registered trademark) standard and / or the USB (registered trademark) standard as the communication circuit 17. Wireless data communication may be performed by using a communication controller of a semiconductor integrated circuit that operates in accordance with the IEEE 802.11 standard for wireless local area networks (LANs) and / or the fourth-, fifth-, and sixth-generation mobile communication systems, known as 4G, 5G, and 6G, for mobile communication.

[0033] The relationship between the image sensor 111 and the pan rotation axis Lp and tilt rotation axis Lt will be described using FIGS. 4A and 4B . FIG. 4A shows the relationship between the pan rotation axis Lp and the tilt rotation axis Lt when the image sensor 111 is viewed from above. FIG. 4B shows the relationship between the pan rotation axis Lp and the tilt rotation axis Lt when the image sensor 111 is viewed from the front. The drive mechanism 12 rotates the image capture device 1 around the pan rotation axis Lp in accordance with a pan control control signal. The drive mechanism 12 also rotates the image capture device 1 around the tilt rotation axis Lt in accordance with a tilt control control signal. As shown in FIGS. 4A and 4B , the pan rotation axis Lp and the tilt rotation axis Lt intersect. In the example shown in FIGS. 4A and 4B , the intersection L between the pan rotation axis Lp and the tilt rotation axis Lt is located at a position perpendicular to the center of the image sensor 111. However, the position of the intersection L is not limited to a position perpendicular to the center of the image sensor 111.

[0034] As described above, roll control is omitted in this embodiment, but if there is a roll axis, parallax occurs depending on the position of the roll axis, making processing more complicated. Specifically, if the roll axis passes through the focal point and the image center of the image sensor 111, that is, if the roll axis is positioned so that the perpendicular to the image sensor 111 intersects with the focal point, parallax does not occur due to the roll axis. On the other hand, if the roll axis is deviated from the focal point or does not pass through the image center of the imaging surface, parallax occurs due to the roll axis.

[0035] FIG. 5A illustrates the relationship between the lens (optical element 112), the image sensor 111, the pan rotation axis Lp, and the tilt rotation axis Lt. Similar to FIG. 4A , FIG. 5A illustrates the relationship between the image sensor 111, the optical element 112, the pan rotation axis Lp, and the tilt rotation axis Lt as viewed from above. As shown in FIG. 5A , a predetermined offset distance ds is provided between the image sensor 111 and the pan rotation axis Lp. Also, in FIG. 5A , the focal length from the position of the optical element 112 to the image sensor 111 is defined as df. As shown in FIG. 5A , the distance d from the optical element 112 to the pan rotation axis Lp is calculated by the sum of the offset distance ds and the focal length df. FIG. 5B illustrates an example in which the pan angle is changed by θ from the state shown in FIG. 5A . As shown in FIG. 5B , even if the pan angle is changed, the distance d from the optical element 112 to the pan rotation axis Lp and the tilt rotation axis Lt does not change. Similarly, even if the tilt angle is changed, the distance d from the optical element 112 to the pan rotation axis Lp and tilt rotation axis Lt does not change. The image capture device 1 utilizes the existence of this offset distance ds to correct internal distortion. Specifically, the image capture device 1 uses multiple images captured by changing only the pan angle and / or tilt angle to correct internal distortion. Therefore, there is no need to move the position of the image capture device 1 to correct internal distortion.

[0036] The processing executed in the arithmetic circuit 14 in relation to the correction of internal distortion will be described below.

[0037] (Pan-Tilt Control) The arithmetic circuit 14 generates a pan-tilt control signal for performing pan-tilt control in order to capture an image for internal distortion correction. The arithmetic circuit 14 outputs the generated pan-tilt control signal to the drive mechanism 12. The arithmetic circuit 14 also generates a shooting control signal for performing shooting. The arithmetic circuit 14 outputs the generated shooting control signal to the optical mechanism 11. Here, the arithmetic circuit 14 generates pan-tilt control signals for driving to a plurality of different pan angles and tilt angles. For example, the storage device 13 may store in advance a plurality of combinations of different pan angles and tilt angles as calibration data. Furthermore, the plurality of different pan angles and tilt angles are set within a range such that the same subject is included in at least a portion of the image to be captured.

[0038] (Capturing Images) When the pan angle and / or tilt angle is changed, the arithmetic circuit 14 sends a control signal to the optical mechanism 11 to capture an image. The arithmetic circuit 14 also stores image data obtained at each pan angle and tilt angle in the storage device 13. Here, the arithmetic circuit 14 stores the pan angle and tilt angle at the timing when each image data is captured in the storage device 13 as pan / tilt data.

[0039] (Extraction of Feature Points and Description of Feature Amounts) The arithmetic circuit 14 extracts feature points from each acquired image and calculates feature amounts for each feature point. For example, the arithmetic circuit 14 can extract feature points from each image and describe feature amounts using a commonly used SIFT (Scale Invariant Feature Transform) algorithm. For example, the arithmetic circuit 14 may also use a SURF (Speeded Up Robust Features) or ORB (Oriented Fast and Rotated Brief) algorithm to extract and describe feature points. Assume that the image capture device 1 acquires two images shown in FIGS. 6A and 6B at different pan and tilt angles. The arithmetic circuit 14 extracts feature points from the image shown in FIG. 6A as shown in FIG. 7A. Furthermore, the arithmetic circuit 14 extracts feature points from the image shown in FIG. 6B as shown in FIG. 7B. In the examples of FIGS. 7A and 7B, the extracted feature points are indicated by circular dashed lines. 7A and 7B show simplified examples in which 12 feature points are extracted, but the number of feature points actually extracted from an image is greater than 12.

[0040] (Feature Point Matching) After detecting feature points and describing feature amounts, the arithmetic circuit 14 performs feature point matching between different images. The arithmetic circuit 14 can use various existing algorithms for feature point matching. For example, the arithmetic circuit 14 can use A-KAZE (ACCELERATED-KAZE), K-nearest neighbors (KNN), or brute-force matcher for feature point matching. In this case, the arithmetic circuit 14 generates all combinations of two images (image pairs) from multiple images. Furthermore, the arithmetic circuit 14 performs feature point matching by identifying points with similar feature amounts for all combinations of two images. The arithmetic circuit 14 generates a matching list as a result of performing feature point matching. The matching list is a list that indicates the association of corresponding feature points between two images.

[0041] (Removal of Outliers) The arithmetic circuit 14 removes erroneous matches from each matching list obtained by feature point matching. To detect erroneous matches, for example, epipolar geometry and the RANSAC method are used. The arithmetic circuit 14 uses multiple matching lists from which outliers have been removed to select the image combination with the highest degree of match.

[0042] (Internal Parameter Estimation) The arithmetic circuit 14 uses the combination of images with the highest degree of match to estimate the position of the camera device 1 when each image was captured. Here, the two images were captured by the camera device 1, whose position is fixed. However, the camera device 1 captured the images at different pan and tilt angles. Therefore, the arithmetic circuit 14 estimates the internal parameters using the coordinate relationships of each part set in the camera device 1 and the pan and tilt angles when each image was captured.

[0043] As shown in Fig. 8A, the focal point that is the starting point of the image capture device 1 is designated as A, and the focal point after pan and / or tilt rotation is designated as B. The intersection of the pan rotation axis Lp and the tilt rotation axis Lt is designated as L. Fig. 8B shows the relationship in Fig. 8A in more detail. As shown in Fig. 8B, the coordinates of the focal points A, B, and the intersection point L in the world coordinate system (Xw, Yw, Zw) are respectively designated as C. A , C B , C L Also, the intersection point L and the coordinate C A , C B The distance between them is d and the rotation angle is θ. Note that the distance d is the distance in the world coordinate system.

[0044]

[0045] These values ​​can be determined using the coordinate relationships known from the design of the image capture device 1 and the pan and tilt angles when each image was captured.

[0046] C A C L Between, C B C L The distance between them is given by equation (3.1) using d. A -C L |-|C B -C L |=d...(3.1) C L Since it rotates around the center, C A , C B C L The equation showing the relationship when the object is translated to the position and rotated by θ is the rotation matrix R L is shown in equation (3.2) using C B -C L =R L (C A -C L ) C B =R L (C A -C L ) + C L ...(3.2) Here, the rotation matrix R L If the unit vector in the direction of the rotation axis is n = (n1, n2, n3), then equation (3.3) is given below.

[0047]

[0048] For example, when rotating around the Yw axis, n = (0, 1, 0). The rotation direction is a right-handed screw direction, that is, the direction from the Xw axis toward the Zw axis is the positive rotation direction. The arithmetic circuit 14 calculates the internal parameter K by adding the constraints of equations (3.1) and (3.2) to equation (2.3) described above for the epipolar constraint. Specifically, the arithmetic circuit 14 calculates the internal parameter K by solving simultaneous equations for multiple feature points using the epipolar constraint that holds for feature points in two images. This makes it possible to improve the calculation accuracy of the internal parameter K.

[0049] (Calculation of Correction Parameters) Once the internal parameter K is estimated, the arithmetic circuit 14 generates correction parameters for correcting internal distortion using the estimated internal parameter K. The arithmetic circuit 14 then uses the generated correction parameters to correct the internal distortion. For example, the calculated internal parameter K should be the same for every coordinate point in the image, but the internal parameter may differ for each coordinate due to internal distortion. The arithmetic circuit 14 generates correction parameters for correcting the variation in the internal parameter K for each coordinate point.

[0050] In this way, for calibration, the position of the image capture device 1 itself is fixed, and only the pan angle and / or tilt angle is changed to capture multiple images. At this time, there is an offset distance ds between the pan rotation axis Lp and the tilt rotation axis Lt and the image sensor. This allows the image capture device 1 to apply the epipolar constraint and find the internal parameter K without changing the position of the image capture device 1 itself.

[0051] Here, the conditions for the distance d and the angle θ will be explained using Figures 9A and 9B. As described above, the distance d is the distance between the focal point A and the intersection L of the pan rotation axis Lp and the tilt rotation axis Lt. Furthermore, the angle θ is the rotation angle of the pan angle and the tilt angle. In Figure 9A, the optical axes entering each focal point A and B are represented by dashed dotted lines. In the example shown in Figure 9A, the distance between focal points A and B corresponds to the parallax. This distance is expressed by the following formula (4.1).

[0052]

[0053] If the angle θ is fixed, the greater the distance d, the greater the distance between points A and B.

[0054] The distance d and the angle θ have a large number of associated parameters. Therefore, the conditions for the distance d and the angle θ are determined using the measurable limit distance D shown in FIG. 9B. FIG. 9B shows the relationship between the parallax required to measure the measurable limit distance D and the target point P. In FIG. 9B, the distance between points A' and B' indicates the parallax required to measure the measurable limit distance D. The distance between the parallax indicated by points A' and B' is given by the following formula (4.2).

[0055]

[0056] Normally, the ratio of the distance between points A' and B' to the measurable limit distance D is constant. Therefore, to see further, the distance between points A' and B' needs to be increased. However, in the imaging device 1, the distance d is fixed and a small value, so even if the angle θ is changed, there is a limit to the range in which the values ​​of points A and B can be increased. Therefore, if the distance d is large, position measurement is possible over a longer distance.

[0057] That is, since the actual parallax needs to be larger than the distance between the parallaxes of points A' and B', the measurable limit distance D satisfies the following relational expression (4.3).

[0058]

[0059] Therefore, if you want to measure a distance greater than the measurable limit distance D, you must satisfy equation (4.3).

[0060] The flow of processing in the correction device will be described using the flowchart shown in Fig. 10. Since the specific processing of each step in the flowchart has been described above, each step will be described in a simplified manner.

[0061] The arithmetic circuit 14 changes the pan angle and / or tilt angle to capture a plurality of images (S001).

[0062] The arithmetic circuit 14 extracts feature points and describes feature amounts for each image obtained in step S001 (S002).

[0063] The arithmetic circuit 14 performs feature point matching using the feature amounts of the multiple feature points of each image obtained in step S002 (S003). At this time, the arithmetic circuit 14 generates combinations of multiple images and performs feature point matching for each combination. As a result of the feature point matching, the arithmetic circuit 14 generates a matching list.

[0064] The arithmetic circuit 14 uses the matching list generated in step S003 to remove outliers (S004).

[0065] The arithmetic circuit 14 uses the feature points from which the outliers have been removed in step S004 to select an image pair with a high degree of matching (S005).

[0066] The arithmetic circuit 14 uses the selected image pair to determine the internal parameters (S006).

[0067] The arithmetic circuit 14 uses the internal parameters obtained in step S006 to obtain correction parameters (S007).

[0068] As described above, the image capturing device 1 according to the first embodiment captures multiple images by changing only the pan angle and / or tilt angle without changing the position of the image capturing device 1. The image capturing device 1 also uses the multiple images to determine correction parameters and correct internal distortion.

[0069] Second Embodiment A correction device 2 according to a second embodiment will be described with reference to FIG. 11 . The image capture device 1 according to the first embodiment described above with reference to FIG. 3 calculates correction parameters and corrects internal distortion. The correction device 2 according to the second embodiment is connected to the image capture device 1A via a network 3. The correction device 2 calculates correction parameters using images captured by the image capture device 1A at a plurality of different pan angles and tilt angles. The image capture device 1A corrects internal distortion using the correction parameters calculated by the correction device 2.

[0070] 11 , the photographing device 1A includes an optical mechanism 11, a drive mechanism 12, a storage device 13, an arithmetic circuit 14, an input device 15, an output device 16, and a communication circuit 17. The storage device 13 of the photographing device 1A does not need to store the correction program P1. The arithmetic circuit 14 controls the pan angle and tilt angle in accordance with a control signal received from the correction device 2 via the communication circuit 17. The arithmetic circuit 14 also captures an image in accordance with the control signal received from the correction device 2 via the communication circuit 17. The arithmetic circuit 14 transmits, to the correction device 2, multiple images captured at different pan angles and tilt angles, as well as the pan angles and tilt angles used when the images were captured. The arithmetic circuit 14 also receives correction parameters via the communication circuit 17. The arithmetic circuit 14 corrects internal distortion using the received correction parameters.

[0071] 11 , the correction device 2 is an information processing device such as a personal computer that includes an arithmetic circuit 21, an input device 22, an output device 23, a communication circuit 24, and a storage device 25. The arithmetic circuit 21, the output device 23, the communication circuit 24, and the storage device 25 are the same as the arithmetic circuit 14, the output device 16, the communication circuit 17, and the storage device 13 of the imaging device 1 described above with reference to FIG. 3. The input device 22 may be a keyboard or a mouse in addition to operation buttons, a touch panel, a microphone, etc.

[0072] The arithmetic circuit 21 reads and executes the correction program P2 stored in the storage device 25, thereby executing processes such as pan / tilt control, image capture control, feature extraction and feature description, feature matching, outlier removal, internal parameter estimation, and correction parameter calculation. Each process is similar to the process executed by the arithmetic circuit 14 of the imaging device 1 according to the first embodiment, and therefore detailed description thereof will be omitted.

[0073] (Pan-tilt control and image capturing) The arithmetic circuit 21 generates a pan-tilt control signal for executing pan-tilt control. The arithmetic circuit 21 transmits the generated pan-tilt control signal to the photographing device 1 via the communication circuit 24. The arithmetic circuit 21 also generates a photographing control signal for executing photographing. The arithmetic circuit 21 transmits the generated photographing control signal to the photographing device 1 via the communication circuit 24. When the arithmetic circuit 21 receives image data obtained at each pan angle and tilt angle, it stores the data in the storage device 13. The arithmetic circuit 21 also receives pan-tilt data indicating the pan angle and tilt angle at the time when each image data was captured, and stores the data in the storage device 13.

[0074] (Extraction of Feature Points and Description of Feature Amounts) The arithmetic circuit 21 extracts feature points from each acquired image and calculates the feature amount for each feature point.

[0075] (Feature Point Matching) After detecting feature points and describing feature amounts, the arithmetic circuit 21 executes feature point matching between different images.

[0076] (Exclusion of Outliers) The arithmetic circuit 21 removes erroneous matches from each matching list obtained by feature point matching. Furthermore, the arithmetic circuit 21 selects the combination of images with the highest degree of match using the multiple matching lists from which the outliers have been removed.

[0077] (Internal Parameter Estimation) The arithmetic circuit 21 estimates the position of the image capturing device 1 in the state in which each image was captured, using the combination of images with the highest degree of match. The arithmetic circuit 21 also estimates the internal parameters of the image capturing device 1 using the estimation results.

[0078] (Calculation of Correction Parameters) The arithmetic circuit 21 generates correction parameters for correcting the internal distortion using the estimated internal parameters. The arithmetic circuit 21 also transmits the generated correction parameters to the imaging device 1A via the communication circuit 24.

[0079] As described above, the correction device 2 according to the second embodiment can calibrate the connected image capture device 1A. At this time, the position of the image capture device 1A itself is not changed, and multiple images captured by changing only the pan angle and / or tilt angle are transmitted to the correction device 2. The correction device 2 also generates correction parameters for correcting internal distortion using the multiple received images.

[0080] Overview of the embodiments (1) The imaging device of the present disclosure is an imaging device that includes an optical element and an arithmetic circuit, and is capable of changing the imaging direction around at least two rotation axes, wherein the optical element is positioned at a position away from the intersection of the at least two rotation axes, and the arithmetic circuit: captures a plurality of images in different imaging directions; uses the imaging direction at the time of capturing each of the plurality of images to determine the distance from the focal point of each image defined by the optical element to the intersection; calculates internal parameters of the imaging device using the distance; determines correction parameters based on the internal parameters; and may use the correction parameters to correct internal distortion of the imaging device.

[0081] This allows internal distortion to be corrected without moving the imaging device itself.

[0082] (2) A correction device according to the present disclosure includes an arithmetic circuit and corrects internal distortion of an imaging device whose imaging direction can be changed around at least two rotation axes, wherein an optical element of the imaging device is positioned away from the intersection of the at least two rotation axes, and the arithmetic circuit acquires multiple images captured in different imaging directions from the imaging device, calculates the distance from the focal point of each image defined by the optical element to the intersection using the imaging direction at the time of capturing each of the multiple images, calculates internal parameters of the imaging device using the distance, calculates correction parameters based on the internal parameters, and outputs the correction parameters to the imaging device.

[0083] This allows internal distortion to be corrected without moving the imaging device itself.

[0084] (3) The correction method disclosed herein is a correction method executed by a correction device that corrects internal distortion of an imaging device whose imaging direction can be changed around at least two rotation axes, wherein an optical element of the imaging device is positioned at a position away from an intersection of the at least two rotation axes, and the correction device includes an arithmetic circuit, wherein the arithmetic circuit: acquires multiple images captured in different imaging directions from the imaging device; determines the distance from the focal point of each image defined by the optical element to the intersection using the imaging direction at the time of capturing each of the multiple images; calculates internal parameters of the imaging device using the distance; determines correction parameters based on the internal parameters; and outputs the correction parameters to the imaging device.

[0085] This allows internal distortion to be corrected without moving the imaging device itself.

[0086] (4) In the correction method of (3), the arithmetic circuit may detect feature points in each of the plurality of image pairs, each of which is an image pair consisting of two images, and calculate the internal parameters using the image pair having a high degree of matching selected using the feature points.

[0087] This allows for highly accurate correction of internal distortion using two images that are highly similar.

[0088] (5) In the correction method of (3) or (4), the arithmetic circuit may use epipolar constraints to calculate the internal parameters.

[0089] This allows for highly accurate correction of internal distortion using two images.

[0090] (6) The computer program of the present disclosure is a computer program executed by a correction device that has an arithmetic circuit and corrects internal distortion of an imaging device that can change the imaging direction around a rotation axis, and may cause an information processing device to execute the correction method described in any one of (3) to (5).

[0091] This allows internal distortion to be corrected without moving the imaging device itself.

[0092] The imaging system, control method, and computer program according to all claims of the present disclosure are realized by hardware resources, such as a processor, a memory, and cooperation with a computer program.

[0093] The imaging device, correction device, correction method, and computer program of the present disclosure are useful for calibrating internal distortion of an imaging device.

[0094] 1, 1A Photographing device 2 Correction device 11 Optical mechanism 111 Image pickup element 112 Optical element 113 A / D converter 12 Drive mechanism 13, 25 Storage device 14, 21 Arithmetic circuit 15, 22 Input device 16, 23 Output device 17, 24 Communication circuit d Distance ds Offset distance df Focal length K Internal parameter L Intersection point Lp Pan rotation axis Lt Tilt rotation axis P1, P2 Correction program

Claims

1. An imaging device equipped with an optical element and an arithmetic circuit, and capable of changing the imaging direction around at least two rotation axes, wherein the optical element is positioned at a position away from the intersection of the at least two rotation axes, and the arithmetic circuit: captures a plurality of images with different imaging directions; uses the imaging direction at the time of capture of each of the plurality of images to determine the distance from the focal point of each image defined by the optical element to the intersection; calculates internal parameters of the imaging device using the distance; determines correction parameters based on the internal parameters; and corrects internal distortion of the imaging device using the correction parameters.

2. A correction device having an arithmetic circuit and correcting internal distortion of an image capturing device capable of changing the shooting direction around at least two rotation axes, wherein an optical element of the image capturing device is positioned away from the intersection of the at least two rotation axes, and the arithmetic circuit: acquires from the image capturing device a plurality of images captured in different shooting directions; determines the distance from the focal point of each image defined by the optical element to the intersection using the shooting direction at the time of capturing each of the plurality of images; calculates internal parameters of the image capturing device using the distance; determines correction parameters based on the internal parameters; and outputs the correction parameters to the image capturing device.

3. A correction method executed by a correction device that corrects internal distortion of an image capturing device whose shooting direction can be changed around at least two rotation axes, wherein an optical element of the image capturing device is arranged at a position away from an intersection of the at least two rotation axes, and the correction device has an arithmetic circuit, and the arithmetic circuit: acquires a plurality of images captured in different shooting directions from the image capturing device; determines a distance from a focal point of each image defined by the optical element to the intersection using the shooting direction at the time of capturing each of the plurality of images; calculates internal parameters of the image capturing device using the distance; determines correction parameters based on the internal parameters; and outputs the correction parameters to the image capturing device.

4. The correction method according to claim 3, wherein in the arithmetic circuit, the plurality of images are image pairs each consisting of two images, feature points are detected in each of the plurality of image pairs, and the internal parameters are calculated using the image pairs with high matching selected using the feature points.

5. The correction method according to claim 3, wherein the arithmetic circuit uses epipolar constraints to calculate the internal parameters.

6. A computer program for causing the correction device to execute the correction method according to any one of claims 3 to 5.