Imaging system

Abstract

The purpose of the present invention is to provide an imaging system capable of applying highly accurate calibration with which it is possible to correct a non-parametric distortion component. An imaging system according to the present invention performs the following steps one or more times: a step for applying differential distortion to an image; a step for calculating the parametric distortion from the image to which the differential distortion has been applied; and a step for calculating the difference between the distortion of the image and the calculated parametric distortion, or calculating a value derived from said difference, as an updated differential distortion value (see fig. 5).

Description

Imaging system

[0001] The present invention relates to an imaging system.

[0002] In order to perform 3D reconstruction using RGB images acquired by cameras installed at multiple viewpoints, 3D reconstruction using depth images acquired by depth cameras such as ToF, obtaining distance information using stereo cameras, and obtaining distance information using machine learning with a monocular RGB camera with high accuracy, highly accurate distortion correction (calibration, image correction) of each image is required.

[0003] Non-Patent Document 1 discloses a technique for easily performing calibration based on a parametric optical model from the results of photographing a calibration board placed at an arbitrary position multiple times.

[0004] Patent Document 1 discloses a technology that uses the three-dimensional coordinates of a calibration board (calibration plate) determined by assuming a parametric optical model at the center of the image, and corrects image distortion throughout the image from the difference between the two-dimensional coordinates of the board feature points of the desired corrected image calculated from the three-dimensional coordinates and the two-dimensional coordinates of the board feature points that are actually captured.

[0005] JP 2013-036831 A

[0006] Z. Zhang, "A flexible new technique for camera calibration," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, Nov. 2000, doi: 10.1109 / 34.888718.

[0007] In the above-mentioned Non-Patent Document 1, a parametric optical model is assumed, and therefore the distortion that can be corrected by calibration is only the parametric distortion component included in the optical model, which is a problem.

[0008] In the aforementioned Patent Document 1, non-parametric distortion components are also taken into account by calculating the difference between a desired corrected image and an actual captured image. However, in the case of an optical system with large distortion, the absolute value of the difference becomes large, which creates a problem of susceptibility to measurement errors. Furthermore, because a parametric optical model is assumed when calculating the three-dimensional coordinates of the calibration board, if there are distortion components not included in the optical model, errors will occur in the calculation of the three-dimensional coordinates of the board and in the image distortion correction of the entire image.

[0009] The present invention has been made in view of the above circumstances, and has as its object to realize an imaging system to which highly accurate calibration that can also correct non-parametric distortion components can be applied.

[0010] The imaging system of the present invention performs, one or more times, the steps of applying differential distortion to an image, calculating the parametric distortion from the image to which the differential distortion has been applied, and calculating the difference between the distortion of the image and the calculated parametric distortion, or a value derived from the difference, as an updated value of the differential distortion.

[0011] According to the present invention, an imaging system can be realized that can apply high-precision calibration that can also correct non-parametric distortion components. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiments.

[0012] FIG. 1 is a block diagram showing an example of the configuration of an imaging system 100 according to a first embodiment. FIG. 2 shows a result of calibration using the above-mentioned conventional technology. FIG. 3 shows a result of calibration when a captured image 50 contains non-parametric distortion that is not included in a parametric optical model and cannot be represented by parameters. FIG. 4 is a diagram explaining the technology described in Patent Document 1. FIG. 5 is a diagram explaining processing in a distortion information processing unit 4. FIG. 6 is a flowchart showing processing in the distortion information processing unit 4. FIG. 7 is a block diagram showing the internal configuration of the distortion information processing unit 4. FIG. 8 is a block diagram of an imaging system 100 according to a second embodiment. FIG. 9 is a block diagram of an imaging system 100 according to a third embodiment.

[0013] In all drawings used to explain the following embodiments, the same components are generally designated by the same reference numerals, and repeated explanations thereof will be omitted. Furthermore, in the following embodiments, the components (including element steps, etc.) are not necessarily essential unless otherwise specified or considered to be clearly essential in principle. Furthermore, when the terms "consisting of A," "made of A," "having A," or "including A" are used, it goes without saying that they do not exclude other elements, unless otherwise specified to include only that element. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., these terms include those that are substantially similar or similar to the shape, etc., unless otherwise specified or considered to be clearly essential in principle.

[0014] <First Embodiment> Fig. 1 is a block diagram showing an example of the configuration of an imaging system 100 according to a first embodiment of the present invention. The imaging system 100 is a system that outputs an image whose distortion has been corrected with high accuracy, and is used, for example, to perform high-accuracy 3D reconstruction from the corrected image using photogrammetry. Alternatively, the imaging system 100 is used to construct a stereo camera using two or more imaging systems 100 and to obtain high-accuracy distance information through stereo vision. The imaging system 100 includes a camera 1, an image processing unit 2, and a distortion information processing unit 4.

[0015] Generally, an image captured by the camera 1 is transmitted to the image processing unit 2. The image processing unit 2 is a functional unit that performs noise reduction processing, distortion correction, etc., and is configured by a CPU (Central Processing Unit) etc. The camera 1 is communicably connected to the image processing unit 2 directly or via a communication network, and outputs information such as a captured image 50 to the image processing unit 2.

[0016] The camera 1 includes an imaging unit 10. The imaging unit 10 is, for example, a monocular camera. The imaging unit 10 generates a captured image 50 based on received light and outputs the captured image 50 to an input I / F (interface) 20 of the image processing unit 2. There are no particular limitations on the wavelengths that the imaging unit 10 targets, and the wavelengths may include visible light, near-infrared light, infrared light, ultraviolet light, and the like. The imaging unit 10 may also be, for example, a depth camera based on ToF (Time of Flight).

[0017] The image processing unit 2 is a unit that performs image processing on the captured image 50 output from the imaging unit 10. The image processing unit 2 includes an input I / F 20, a distortion correction unit 21, and an output I / F 22. Each component of the image processing unit 2 may be realized by a circuit, or at least a part of it may be realized by a processor such as a CPU that executes a program and a memory.

[0018] The input I / F 20 includes, for example, an A / D (analog / digital) converter, converts the captured image 50 output from the imaging unit 10 into a digital signal, and outputs it to the distortion correction unit 21 at the subsequent stage.

[0019] The distortion correction unit 21 performs various image processing on the captured image 50 output from the input I / F 20 and outputs the result to the output I / F 22. The image processing performed by the distortion correction unit 21 is, for example, a general-purpose geometric transformation, which allows the distorted captured image to be converted into a predetermined coordinate system. At this time, as will be described later, the distortion correction unit 21 receives the parametric distortion 310 and the differential distortion 320 output from the parametric distortion calculation unit 2100 and the differential distortion calculation unit 2110, respectively, and performs distortion correction based on these. The image processing performed by the distortion correction unit 21 may also include other processing, such as demosaicing.

[0020] The output I / F 22 outputs the corrected captured image 51 output from the distortion correction unit 21 to a control unit such as a CPU or ECU at a downstream stage.

[0021] Generally, the captured image 50 output from the image capturing unit 10 contains distortion of the radial direction component expressed by, for example, equation (1) due to the influence of the lens projection method and manufacturing error. x', x'', y', y'' satisfy equation (2), and k １, k ２ , k ３ is the radial distortion coefficient, X, Y, Z are the three-dimensional coordinates of the object in the world coordinate system, u, v are the coordinates projected on the sensor, R, t are the rotation and translation of the transformation from the world coordinate system to the camera coordinate system, f ｘ , f ｙ is the focal length, c ｘ , c ｙ indicates the center of distortion. In addition, higher-order radial distortion and circumferential distortion may also be included. Distortion expressed by parameters, such as in equation (1), is called parametric distortion.

[0022]

[0023]

[0024] In order to accurately perform 3D reconstruction using photogrammetry or obtain distance information using stereoscopic vision for this captured image 50, it is necessary to accurately correct distortion in advance. Non-Patent Document 1 is an example of a widely used conventional calibration technique for correcting distortion. This conventional technique calculates and corrects image distortion using multiple images of a calibration board (calibration board) captured while changing its position and orientation. This calibration board simply needs to have a specific pattern printed on a flat surface, and can be easily created, for example, by attaching paper with the pattern printed on it to a flat plate. Furthermore, the position and orientation of the board during capture can be freely arranged as long as the pattern can be sufficiently recognized, making capture easy.

[0025] In the distortion calculation, parameters such as the camera's focal length, distortion center, and distortion calculation are determined from the captured board image, as expressed by equations (1) and (2). By determining these parameters, the parametric distortion contained in the captured image 50 is determined. In addition to the above parameters, the three-dimensional relative coordinates of the board with respect to the camera (external parameters) are also calculated simultaneously. However, because this calculation assumes an optical model defined by parameters (parametric optical model), it can only consider distortion components contained in this optical model. Therefore, while this conventional technique makes it easy to create and photograph a board, it suffers from accuracy problems in that the distortion components that can be considered are limited to parametric distortion.

[0026] FIG. 2 shows the results of calibration using the above-described conventional technique. Pre-correction distortion 300 indicates the image distortion contained in the captured image 50, with arrows indicating the average within each grid, which is divided into appropriate sizes for ease of viewing. The starting point of this arrow is the position where it should be projected, and the end point is the actual position where it is projected. However, the original projection point is calculated, for example, assuming a distortion-free perspective projection. This figure shows a case where the captured image 50 contains only parametric distortion of the radial component expressed by, for example, Equation (1), particularly a case known as barrel distortion. Pre-correction distortion profile 301 shows distortion on a line passing horizontally through the center of the pre-correction distortion 300, for example. In other words, both the pre-correction distortion 300 and the pre-correction distortion profile 301 represent the distortion before correction. The parametric distortion 310 is distortion information obtained by performing calibration using the above-described conventional technique on the captured image 50 containing the pre-correction distortion 300, and is displayed in the same format as the pre-correction distortion 300. That is, the start point of the obtained parametric distortion 310 is displayed as the position where it should be projected, and the end point is displayed as the actual projected position. Similarly, the obtained parametric distortion 310 is displayed in the same format as the pre-correction distortion profile 301 to form a parametric distortion profile 311. The vector difference between the pre-correction distortion 300 and the parametric distortion 310, that is, the difference vector between the projected positions, is the post-correction distortion 330, and its profile is the post-correction distortion profile 331.

[0027] In this example, the image distortion contained in the captured image 50 is, for example, the parametric distortion of the radial component expressed by equation (1), and therefore, in an ideal situation free from the influence of noise, etc., the parametric distortion 310 calculated using the parametric optical model will be exactly the same as the pre-correction distortion 300. Therefore, the post-correction distortion 330 will be 0 at all points in the image, meaning that the distortion has been completely corrected.

[0028] FIG. 3 shows the calibration results for a captured image 50 that contains, in addition to the parametric distortion of the radial component expressed by Equation (1), nonparametric distortion that is not included in the parametric optical model and cannot be expressed by parameters. Causes of this nonparametric distortion include, for example, manufacturing errors in the camera cover glass thickness, lens manufacturing errors, and assembly errors. These factors cannot be fully accounted for in a parametric optical model, or would require an unrealistic number of parameters to account for them. When a conventional distortion calculation assuming a parametric optical model is performed on this captured image 50, which contains nonparametric distortion in the pre-correction distortion 300, the post-correction distortion 330 obtained is primarily composed of nonparametric distortion components that remain as distortion that cannot be fully corrected. Furthermore, the nonparametric distortion also causes errors in the parameters of the parametric optical model and external parameters.

[0029] Another calibration technique is described in Patent Document 1. Patent Document 1 uses a dense pattern in the center of the board and a sparse pattern in the peripheral area of ​​the board as a calibration board. This is to increase the amount of information by densely arranging feature points in the center of the board, which appears with less distortion in the image center, and to make feature points easier to detect in the peripheral area, which appears with greater distortion in the image peripheral area. The image center is, for example, a region in the center of the image that includes a rectangle with each side being 1 / 4 of the size of the entire image. The image peripheral area is a region that includes areas near the top and bottom edges of the image whose width is 1 / 4 of the image height and areas near the left and right edges whose width is 1 / 4 of the image width. Since the image center represents an area with less image distortion, it does not necessarily have to be located in the center of the image. For example, if the optical axis of the imaging optical system and the center of the image sensor are misaligned, the image center is near the optical axis. This technique first calculates the external parameters of the board using each parameter of a parametric optical model calculated in advance and feature points in the center of the board. Next, using these external parameters, we determine the points where the feature points around the board should appear on the image in an ideal, distortion-free state. Finally, by taking the difference between the points that should appear and the points that actually appear, we can calculate the distortion of the entire image, including non-parametric components.

[0030] FIG. 4 is a diagram illustrating the technology described in Patent Document 1. This figure illustrates a case in which a captured image 50 contains nonparametric distortion in addition to parametric distortion. In this case, the distortion information is a differential distortion 320 calculated from the difference between the feature points that should be captured and the actual feature points captured across the entire image. The corrected distortion 330, which is the result of correction using this differential distortion 320, removes the pre-correction distortion 300, including the nonparametric distortion, across the entire image. However, because the pre-correction distortion 300 is expressed entirely as differential distortion 320, the absolute value of the differential distortion 320 is large and the function shape is complex, especially for cameras with large distortions. Because the absolute value of the differential distortion 320 is large and the numerical value is complex, there is a problem in that it is easily affected by errors that occur in feature point detection, etc., resulting in reduced accuracy.

[0031] Furthermore, in actual calibration, there is often non-parametric distortion in the central part of the image, and errors also occur in the parameters of the pre-determined parametric optical model and the external parameters of the board. If errors occur in the external parameters, errors also occur in the points that should be captured as feature points for the entire board, which are calculated using this information, and as a result, errors also occur in the differential distortion 320. Therefore, there is a problem in that errors occur in the differential distortion 320 due to the assumption of a parametric optical model for the central part of the image.

[0032] Therefore, in this invention, a parametric optical model is used to obtain global parametric distortion information for the entire image, and nonparametric distortion information is obtained for the remaining distortion that cannot be expressed by the parametric optical model using the difference between the board feature points that should be captured and the points that are actually captured, thereby achieving high-precision calibration that also takes nonparametric distortion components into account for the entire image. Furthermore, by iteratively obtaining parametric distortion components and nonparametric distortion components while adding conditions such as requiring the nonparametric distortion components to be smooth in consideration of physical rationality, external parameters of the board, which were a problem in Patent Document 1, for example, can be converged to accurate values.

[0033] Returning to Fig. 1, distortion correction according to the present invention will now be described. When calibrating the camera 1, a captured image 50 captured by the imaging unit 10 is input to the distortion information processing unit 4 in order to calculate distortion information in advance. The distortion information processing unit 4 is, for example, a calculation system that exists outside the camera 1, such as a general-purpose personal computer. The camera 1 and the distortion information processing unit 4 are connected directly or via a communication network, and input and output the captured image 50. Alternatively, the captured image 50 may be input and output using an information storage medium.

[0034] The distortion information processing unit 4 calculates a parametric distortion 310 and a differential distortion 320 based on a parametric optical model from the input captured image 50. The parametric distortion 310 and the differential distortion 320 are, for example, image information having the same number of pixels as the captured image 50, and include, for example, two types of information: an x ​​component and a y component. The distortion information processing unit 4 compresses the parametric distortion 310 and the differential distortion 320, for example, and outputs them as parametric distortion information 3100 and differential distortion information 3200 to the parametric distortion information storage unit 2101 and the differential distortion information storage unit 2111 in the image processing unit 2 of the camera 1, respectively. As an example of capacity compression, the parametric distortion 310 is output to the parametric distortion information storage unit 2101 in the form of numerical values ​​of each parameter. The differential distortion 320 is output to the differential distortion information storage unit 2111 in, for example, a compressed image format. The above compression is just one example, and the parametric distortion 310 and the differential distortion 320 may be output as is, or other processing may be performed. The distortion information processing unit 4 and the parametric distortion information storage unit 2101 or the differential distortion information storage unit 2111 are connected directly or via a communication network or an information storage medium, as described above.

[0035] The parametric distortion information storage unit 2101 stores and saves the parametric distortion information 3100 output from the distortion information processing unit 4, and selects appropriate information to output to the parametric distortion calculation unit 2100. The saved parametric distortion information 3100 does not have to be a single set, and multiple sets of parametric distortion information may be stored for each combination of, for example, the temperature of the environment in which the camera 1 captures an image, the humidity of that environment, the cumulative usage time of the camera 1, and the like.

[0036] The parametric distortion calculation unit 2100 restores the parametric distortion 310 based on the parametric optical model from the parametric distortion information 3100 output from the parametric distortion information storage unit 2101 , and outputs the restored parametric distortion 310 to the distortion correction unit 21 .

[0037] The differential distortion calculation unit 2110 processes the differential distortion information 3200 output from the differential distortion information storage unit 2111 , restores the differential distortion 320 , and outputs it to the distortion correction unit 21 .

[0038] As described above, the distortion correction unit 21 calculates a corrected captured image 51 in which distortion correction has been performed on the captured image 50 output from the input I / F 20 based on the parametric distortion 310 output from the parametric distortion calculation unit 2100 and the differential distortion 320 output from the differential distortion calculation unit 2110, and outputs the corrected captured image 51 to the output I / F 22. In addition, other processes such as demosaicing may also be performed.

[0039] Next, a specific procedure for calculating distortion information for the captured image 50 in the distortion information processing unit 4 will be described.

[0040] The parametric distortion 310 and differential distortion 320, which are information necessary for distortion correction, are calculated separately in advance for a captured image 50 whose distortion is to be corrected. When calculating these in advance, an image of a calibration board on which periodic feature points are printed, such as that used in Non-Patent Document 1, is used. It is desirable to photograph the board multiple times while changing its position and orientation so that the feature points are distributed throughout the image. The feature points printed on the board may be, for example, a grid pattern, a circle, a ring, or a two-dimensional code. These images acquired by the imaging unit 10 are input to the distortion information processing unit 4 of the image processing unit 2, where the parametric distortion 310 and differential distortion 320 are calculated.

[0041] FIG. 5 is a diagram illustrating the processing in the distortion information processing unit 4. As described above, the pre-correction distortion 300 and the pre-correction distortion profile 301 represent the pre-correction distortion in the captured image 50 that includes the calibration board. The parametric distortion 310 and the parametric distortion profile 311 represent the distortion of the captured image 50 expressed using a parametric optical model when the parameters included in, for example, Equation (1) or Equation (2) are given. The differential distortion 320 and the differential distortion profile 321 represent the correction applied to the captured image 50 before correcting the captured image 50 using the parametric distortion 310. The pre-correction distortion 300, the parametric distortion 310, and the differential distortion 320 may be in any format that represents distortion in the captured image 50. One example of such a format is an image format having the same number of pixels as the captured image 50. Furthermore, for example, the format may be two images that represent distortion in the x and y directions, respectively.

[0042] 6 is a flowchart showing the processing in the distortion information processing unit 4. Each step in FIG. 6 will be described below.

[0043] S1: This is an initialization process, and first, all the differential distortions 320 are initialized to zero.

[0044] S2: The differential distortion 320 is applied to the captured image 50 for calibration. Specifically, the differential distortion 320 is used to perform a geometric transformation on the captured image 50, thereby correcting the non-parametric component of the captured image 50 represented by the differential distortion 320. However, in the first iterative calculation, the differential distortion 320 is initialized to all zeros, so no correction is actually performed on the captured image 50. In Figure 5, the first calculations from S2 to S4 correspond to the upper half of the figure, and the differential distortion 320 is represented as all zeros.

[0045] S3: For the captured image 50 that has been corrected in S2, a parametric distortion 310 is calculated based on a parametric optical model, for example, using the technique of Non-Patent Document 1. In the first calculation, the captured image 50, which includes non-parametric distortion components, is represented by a parametric optical model, and therefore errors occur in each parameter of the obtained optical model and in the external parameters that represent the board position.

[0046] S4: Calculate the post-correction distortion 330 and post-correction distortion profile 331. In this calculation, first, the reprojection points of the feature points of the calibration board are calculated. That is, the positions (reprojection points) where each feature point is projected on the image are calculated using each parameter of the optical model obtained in S3 and the external parameters of the board. Next, a vector is calculated, starting from the reprojection point and ending at the feature point actually detected in the captured image 50. The set of these vectors is the post-correction distortion 330. Note, however, that because each parameter of the optical model and the external parameters of the board contain errors, the post-correction distortion 330 also contains errors. Ignoring these errors, the post-correction distortion 330 is the difference between the parametric distortion 310 and the pre-correction distortion 300 (i.e., the true value of the distortion of the captured image 50). Therefore, by treating the post-correction distortion 330 obtained here as the differential distortion 320 and using it together with the parametric distortion 310 for correction during calibration, correction including non-parametric components is possible. The lower half of FIG. 5 shows the steps S2 to S4 of repeating the distortion calculation using the calculated non-zero corrected distortion 330 as the differential distortion 320.

[0047] S5: It is determined whether the calculated distortion is sufficiently accurate based on the magnitude of the corrected distortion 330. In other words, if the corrected distortion 330 is sufficiently small over the entire image, it means that the distortion of the captured image 50 has been represented sufficiently accurately by the parametric distortion 310 and the differential distortion 320. If the accuracy is sufficient, the distortion calculation is terminated; if not, the process returns to S2 and the calculation is repeated. If it is not desired to repeat the calculation, it is also possible to forcibly terminate the distortion calculation regardless of the accuracy.

[0048] In the second or subsequent iterations of processing after returning to S2, a differential distortion 320 is applied to the captured image 50 for calibration. Since the differential distortion 320 is the difference between the pre-correction distortion 300 and the parametric distortion 310, when the geometric transformation represented by the differential distortion 320 is subtracted from the captured image 50, the distortion remaining in the captured image 50 is a distortion that can be represented by the parametric distortion 310. By recalculating the parametric distortion 310 in S3 for the captured image 50 after this differential distortion correction, a more accurate parametric distortion 310 with fewer errors in the parameters of the optical model and the external parameters of the board can be obtained. The corrected distortion 330 calculated in S4 thereafter is expected to have a smaller absolute value than the one calculated previously. The differential distortion 320 calculated in S4 from the second or subsequent iterations is updated to a new differential distortion 320 by combining the current differential distortion 320 and the corrected distortion 330.

[0049] In this way, by repeatedly calculating the parametric distortion 310 for the calibration image 50 to which the differential distortion 320 has been applied while updating the differential distortion 320, it is possible to improve the accuracy of the distortion calculation, including the non-parametric component represented by the differential distortion 320.

[0050] FIG. 7 is a block diagram showing the internal configuration of the distortion information processing unit 4. The distortion information processing unit 4 includes a differential distortion smoothing unit 40. The differential distortion smoothing unit 40 performs a smoothing process on the differential distortion 320 calculated in S4, for example, to remove noise. Examples of smoothing processes include a Gaussian filter, a median filter, and a binning process. Other smoothing processes may also be performed that take physical rationale and manufacturing errors into account. For example, during the manufacturing of a camera cover glass, minute thickness variations such as stripes may occur due to the stretching process. These thickness variations may result in nonparametric distortion with a striped pattern in the captured image 50. In this case, an example of a smoothing process that takes manufacturing errors into account is to change the standard deviation of the Gaussian filter in the direction parallel to and perpendicular to the stripe pattern. In other words, smoothing is performed based on the spatial frequency of the image. This smoothing process removes noise from the differential distortion 320, thereby improving the overall accuracy of the distortion calculation. Furthermore, the iterative calculations from steps S2 to S5 become stable, and divergence in the iterative calculations can be suppressed.

[0051] When step S1 is first performed, the prior information about the manufacturing error of the camera 1 can be used as an initial value for initializing the differential distortion 320. For example, a distortion other than parametric distortion that is assumed based on the information about the manufacturing error can be used as the initial value of the differential distortion 320.

[0052] Differences from Patent Document 1 will now be described using FIGS. 4 and 5 . In Patent Document 1, the differential distortion 320 is calculated by calculating the difference between the point where a feature point, calculated assuming perspective projection, should be projected and the feature point (pre-correction distortion 300) that appears in the captured image 50. In other words, when the captured image 50 having the pre-correction distortion 300 is corrected using the differential distortion 320, a desired distortion-free image, for example, a perspective projection image, is obtained. Therefore, since all distortion in the captured image 50 is expressed by the differential distortion 320, the absolute value of the differential distortion 320 becomes large and the functional form becomes complex, as shown in FIG. 4 , especially when the distortion in the captured image 50 is large. When the functional form of the differential distortion 320 becomes complex and the absolute value becomes large, it becomes susceptible to error and noise removal becomes difficult. Furthermore, because the differential distortion 320 is calculated only once, errors due to the influence of non-parametric component distortion are introduced into the camera internal parameters and board external parameters, which are prerequisite information for the calculation.

[0053] On the other hand, the differential distortion 320 in the present invention is the difference between the parametric distortion 310 and the pre-correction distortion 300. That is, a captured image 50 with pre-correction distortion 300 is corrected with the differential distortion 320 in a first step, and the captured image after this difference correction is corrected with the parametric distortion 310 in a second step. In this way, a desired distortion-free image, for example, a perspective projection image, can be obtained. In this case, since the distortion of the captured image 50 is generally represented by the parametric distortion 310, even when the distortion in the captured image 50 is particularly large, the absolute value of the differential distortion 320 is small, as shown in FIG. 5 , and the functional form is simpler than that distributed around 0. Because the functional form of the differential distortion 320 is simpler and the absolute value is small, it is less susceptible to the influence of errors and also facilitates noise removal. Furthermore, when the calculation of the differential distortion 320 is repeated, the influence of non-parametric component distortion due to the camera's internal parameters and the board's external parameters can be reduced, and the calculated distortion can be more accurately calculated.

[0054] Furthermore, by storing distortion information divided into global parametric distortion 310 and differential distortion 320, it becomes possible to fix the parametric distortion 310 and update only the differential distortion 320 when recalculating distortion in the distortion information processing unit 4. In this way, for example, when it is expected that there will be little fluctuation in distortion, it becomes possible to reduce the number of captured images 50 containing the calibration board that are required for distortion recalculation.

[0055] Summary of First Embodiment The imaging system 100 according to the first embodiment makes it possible to realize a camera to which highly accurate calibration that can also correct non-parametric distortion components can be applied. High-accuracy calculation of distortion involves combining parametric component distortion due to a parametric optical model with additional non-parametric component distortion derived therefrom, and repeating these distortion calculations, thereby enabling highly accurate calculation of distortion that also takes non-parametric distortion components into consideration. High-accuracy calibration can be achieved by using this highly accurately calculated distortion.

[0056] <Embodiment 2> Fig. 8 is a block diagram of an imaging system 100 according to Embodiment 2 of the present invention. In Embodiment 1, it is assumed that the distortion information processing unit 4 exists outside the image processing unit 2, for example, on a general-purpose personal computer. In Embodiment 2, as shown in Fig. 8, a case will be described in which the distortion information processing unit 4 exists inside the image processing unit 2. The distortion information processing unit 4 in Embodiment 2 may be realized by a circuit as part of the image processing unit 2, or at least a part of it may be realized by a processor such as a CPU that executes a program and a memory. The captured image 50 received by the distortion information processing unit 4 may be the captured image 50 output from the imaging unit 10, or may be the captured image 50 converted into a digital signal output from the input I / F 20.

[0057] In the second embodiment, since the distortion information processing unit 4 is located inside the image processing unit 2, recalculation of the parametric distortion 310 and the differential distortion 320 can be performed by only the camera 1 and the image processing unit 2. When this recalculation becomes necessary, for example, the calibration board can be photographed multiple times. This recalculation is necessary, for example, when the actual distortion of the camera 1 fluctuates compared to the parametric distortion information 3100 and differential distortion information 3200 previously calculated and stored in the parametric distortion information storage unit 2101 and the differential distortion information storage unit 2111, thereby reducing the accuracy of calibration using the distortion information. While the first embodiment requires the preparation of, for example, an external general-purpose personal computer equipped with the distortion information processing unit 4 for recalculation, the second embodiment requires only the camera 1 and the image processing unit 2. Therefore, it is possible to reduce the interruption time required for recalculation even in a site where, for example, an external general-purpose personal computer equipped with the distortion information processing unit 4 cannot be prepared.

[0058] 9 is a block diagram of an imaging system 100 according to a third embodiment of the present invention. In the third embodiment, a parametric distortion calculator 2100, a differential distortion calculator 2110, a parametric distortion information storage unit 2101, a differential distortion information storage unit 2111, and a distortion information processing unit 4 are located outside the image processing unit 2. The parametric distortion calculator 2100, the differential distortion calculator 2110, the parametric distortion information storage unit 2101, the differential distortion information storage unit 2111, and the distortion information processing unit 4 are implemented on a computing system such as a general-purpose personal computer. The distortion correction unit 21, the parametric distortion calculator 2100, and the differential distortion calculator 2110 are connected to each other, for example, directly or via a communication network.

[0059] In the third embodiment, the parametric distortion calculation unit 2100 and other components exist outside the image processing unit 2, so the processing load in the image processing unit 2 can be reduced, and power consumption and costs can be reduced.

[0060] <Modifications of the Present Invention> The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

[0061] In the above embodiments, the layout, dimensions, shape, and other configurations of the components of the camera are not limited to the examples described or illustrated above, as long as the object of the present invention can be achieved. Furthermore, terms expressing the relationship between components are not limited to their strict literal meanings, and may include cases where the meaning is substantially the same as the literal meaning, as long as the object and effect of the present invention can be achieved.

[0062] The above-described embodiment is not limited to a camera, and can be provided in various forms such as a stereo camera with two cameras arranged in parallel, or a camera system with multiple cameras installed.

[0063] In the above embodiment, steps S2 to S4 are repeated, but if the calibration results are sufficiently accurate, the number of repetitions may be one. Furthermore, even if step S5 is omitted and steps S2 to S4 are each performed only once, the differential distortion 320 can still be suppressed by performing each of these steps once, and the effects of the present invention can be achieved to that extent.

[0064] In the above embodiment, it has been explained that the differential distortion 320 is the difference between the parametric distortion 310 and the pre-correction distortion 300. The difference between the parametric distortion 310 and the pre-correction distortion 300 itself does not necessarily have to be the differential distortion 320; a homogeneous value derived from this difference may be used as the differential distortion 320. For example, the difference between the parametric distortion 310 and the pre-correction distortion 300 may be multiplied by a coefficient of 1.0 or less to obtain the differential distortion 320. In this case, by reducing the value of the differential distortion 320, it is possible to prevent the convergence calculation from diverging.

[0065] In the above embodiment, all of the functional units other than the camera 1 may be arranged on a device such as a computer external to the camera 1, or only some of these functional units may be configured integrally with the camera 1. When configured integrally with the camera 1, for example, these functional units may be arranged on the image processing unit 2, and the image processing unit 2 may be configured by a circuit device or the like configured integrally with the camera 1.

[0066] In the above embodiments, each functional unit other than the camera 1 can be configured by hardware such as a circuit device that implements these functions, or can be configured by a computing device such as a CPU executing software that implements these functions.

[0067] REFERENCE SIGNS LIST 1 camera 10 imaging unit 2 image processing unit 20 input I / F 21 distortion correction unit 22 output I / F 2100 parametric distortion calculation unit 2110 differential distortion calculation unit 2101 parametric distortion information storage unit 2111 differential distortion information storage unit 300 pre-correction distortion 301 pre-correction distortion profile 310 parametric distortion 3100 parametric distortion information 311 parametric distortion profile 310 parametric distortion 320 differential distortion 3200 differential distortion information 321 differential distortion profile 330 post-correction distortion 331 post-correction distortion profile 4 distortion information processing unit 40 differential distortion smoothing unit 50 captured image 51 corrected captured image 52 differentially corrected captured image

Claims

1. An imaging system for photographing a subject, comprising: a distortion information processing unit that calculates, from an image photographed by a camera, a parametric distortion based on a parametric optical model represented by internal parameters of the camera, and a differential distortion representing a non-parametric distortion component; and a distortion correction unit that applies distortion correction for each of the parametric distortion and the differential distortion to the image, wherein the distortion information processing unit performs a step of applying the differential distortion to the image, the distortion information processing unit performs a step of calculating the parametric distortion from the image to which the differential distortion has been applied, and the distortion information processing unit performs a step of calculating a difference between a distortion of the image and the calculated parametric distortion or a value derived from the difference, as an updated value of the differential distortion, and the distortion information processing unit performs the steps of applying the differential distortion, calculating the parametric distortion, and calculating the updated value of the differential distortion one or more times until the updated value of the differential distortion becomes equal to or less than a threshold.

2. The imaging system according to claim 1, characterized in that: the distortion information processing unit calculates a reprojection point of the feature point of the image; the distortion information processing unit calculates a vector whose start point is the reprojection point and whose end point is the actual feature point on the image; and the distortion information processing unit uses the vector or a vector obtained by multiplying the vector by a coefficient less than 1.0 as the updated value of the differential distortion.

3. The imaging system according to claim 1, further comprising a differential distortion smoothing unit that smoothes the differential distortion, the differential distortion smoothing unit smoothing the differential distortion based on the spatial frequency of the image and a manufacturing error of the camera.

4. An imaging system as described in claim 1, comprising: a parametric distortion information storage unit for storing parametric distortion information converted from the parametric distortion; a differential distortion information storage unit for storing differential distortion information converted from the differential distortion; a parametric distortion calculation unit for restoring parametric distortion from the parametric distortion information; and a differential distortion calculation unit for restoring differential distortion from the differential distortion information, wherein the distortion correction unit applies to the image distortion correction for each of the parametric distortion restored by the parametric distortion calculation unit and the differential distortion restored by the differential distortion calculation unit.

5. The imaging system of claim 4, wherein the parametric distortion information storage unit stores the parametric distortion information for each combination of the temperature of the environment in which the image is taken, the humidity of the environment, and the accumulated usage time of the camera, and the differential distortion information storage unit stores the differential distortion information for each combination of the temperature, the humidity, and the accumulated usage time.

6. The imaging system according to claim 1, wherein the distortion information processing section sets an initial value of the differential distortion based on prior information on manufacturing errors.

7. The imaging system according to claim 1, characterized in that the distortion correction unit is implemented on an image processing unit that receives the image from the camera via an interface, and the distortion information processing unit is implemented on a computer external to the camera and external to the image processing unit.

8. The imaging system according to claim 1, wherein the distortion information processing section and the distortion correction section are implemented on an image processing unit that receives the image from the camera via an interface.

9. The imaging system according to claim 4, characterized in that the distortion information processing unit, the distortion correction unit, the parametric distortion information storage unit, the differential distortion information storage unit, the parametric distortion calculation unit and the differential distortion calculation unit are implemented on an image processing unit that receives the image from the camera via an interface.

10. The imaging system according to claim 4, characterized in that the distortion correction unit is implemented on an image processing unit that receives the image from the camera via an interface, and the distortion information processing unit, the parametric distortion information storage unit, the differential distortion information storage unit, the parametric distortion calculation unit and the differential distortion calculation unit are implemented on a computer external to the camera and external to the image processing unit.

Images