Imaging system

Abstract

To realize an imaging system that can apply highly accurate calibration that can also correct non-parametric distortion components.SOLUTION: An imaging system according to the present invention performs, one or more times, the steps of: applying differential distortion to an image; calculating a parametric distortion from the image to which the differential distortion has been applied; and calculating a difference between the distortion of the image and the calculated parametric distortion or a value derived from the difference as an updated value of the differential distortion.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an imaging system.

Background Art

[0002] In three-dimensional restoration using RGB images obtained by cameras installed at multiple viewpoints, three-dimensional restoration using depth images obtained by depth cameras such as ToF, acquisition of distance information by a stereo camera, acquisition of distance information using machine learning by a monocular RGB camera, etc., in order to perform them with high precision, high-precision distortion correction (calibration, image calibration) of each image is required.

[0003] Non-Patent Document 1 discloses a technique for easily performing calibration based on a parametric optical model from the results of photographing a calibration board placed at an arbitrary position multiple times.

[0004] Patent Document 1 discloses a technique for correcting the image distortion of the entire image from the difference between the two-dimensional coordinates of the board feature points of the desired corrected image calculated therefrom and the two-dimensional coordinates of the board feature points actually captured, using the three-dimensional coordinates of a calibration board (calibration plate) obtained by assuming a parametric optical model at the center of the image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] In the above-mentioned Non-Patent Document 1, since a parametric optical model is assumed, there is a problem that the distortion that can be corrected by calibration is only the parametric distortion component included in the optical model.

[0008] In the above-mentioned Patent Document 1, by calculating the difference between the desired corrected image and the actual captured image, non-parametric distortion components are also considered. However, in the case of an optical system with large distortion, there is a problem that it is easily affected by measurement errors due to the increase in the absolute value of the difference. Furthermore, since a parametric optical model is assumed when calculating the three-dimensional coordinates of the calibration board, if there are distortion components not included in the optical model, there are problems of errors in the calculation of the three-dimensional coordinates of the board and in the image distortion correction of the entire image.

[0009] The present invention has been made in view of such a situation, and an object thereof is to realize an imaging system to which highly accurate calibration capable of correcting non-parametric distortion components can be applied. Means for Solving the Problems

[0010] The imaging system according to the present invention performs, one or more times, the steps of applying differential distortion to an image, calculating the parametric distortion from the image to which the differential distortion has been applied, and calculating, as an updated value of the differential distortion, a difference between the distortion of the image and the calculated parametric distortion or a value derived from the difference.

Advantages of the Invention

[0011] According to the present invention, it is possible to realize an imaging system to which highly accurate calibration applicable to correcting even non-parametric distortion components can be applied. Problems, configurations, and advantages other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0013] In all the figures for explaining the following embodiments, the same members are generally denoted by the same reference numerals, and repeated explanations thereof are omitted. Further, in the following embodiments, it goes without saying that the components (including element steps, etc.) are not necessarily essential except in cases where it is particularly specified or considered to be clearly essential in principle. Also, when it is said that "comprising A", "consisting of A", "having A", or "including A", it goes without saying that other elements are not excluded except in cases where it is particularly specified that only that element is involved. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., it is assumed to include those that are substantially approximated or similar to the shape, etc., except in cases where it is particularly specified or considered to be clearly not so in principle.

[0014] <Embodiment 1> FIG. 1 is a block diagram showing a configuration example of an imaging system 100 according to Embodiment 1 of the present invention. The imaging system 100 is a system that outputs an image with high-precision distortion correction, and is, for example, for performing high-precision 3D reconstruction from the corrected image using photogrammetry. Alternatively, it is for constructing a stereo camera using two or more sets of imaging systems 100 and performing high-precision acquisition of distance information by stereo vision. The imaging system 100 includes a camera 1, an image processing unit 2, and a distortion information processing unit 4.

[0015] Generally, the image acquired by the camera 1 is transmitted to the image processing unit 2. The image processing unit 2 is a functional unit that executes functions such as noise reduction processing and distortion correction, and is configured by a CPU (Central Processing Unit) or the like. The camera 1 is communicably connected to the image processing unit 2 directly or via a communication network, and outputs information such as an imaging image 50 to the image processing unit 2.

[0016] Camera 1 includes an imaging unit 10. The imaging unit 10 is, for example, a monocular camera. The imaging unit 10 generates a captured image 50 based on the received light and outputs it to the input I / F (interface) 20 of the image processing unit 2. There is no particular limitation on the wavelength targeted by the imaging unit 10, and it may be visible light, near-infrared, infrared, ultraviolet, etc. Also, the imaging unit 10 may be, for example, a depth camera based on ToF (Time of Flight).

[0017] The image processing unit 2 is a unit that performs image processing on the captured image 50 output from the imaging unit 10. The image processing unit 2 includes an input I / F 20, a distortion correction unit 21, and an output I / F 22. Each component of the image processing unit 2 may be realized by a circuit, or at least a part of it may be realized by a processor such as a CPU that executes a program and a memory.

[0018] The input I / F 20 includes, for example, an A / D (analog / digital) converter, converts the captured image 50 output from the imaging unit 10 into a digital signal, and outputs it to the subsequent distortion correction unit 21.

[0019] The distortion correction unit 21 performs various image processes on the captured image 50 output from the input I / F 20 and outputs it to the output I / F 22. The image processing executed by the distortion correction unit 21 is, for example, a general-purpose geometric transformation, and thereby, a distorted captured image can be converted into a predetermined coordinate system. At this time, as will be described later, it receives the parametric distortion 310 and the differential distortion 320 respectively output from the parametric distortion calculation unit 2100 and the differential distortion calculation unit 2110, and performs distortion correction based on them. The image processing executed by the distortion correction unit 21 may also include, for example, a demosaicking process.

[0020] The output I / F 22 outputs the corrected captured image 51 output from the distortion correction unit 21 to a control unit such as a subsequent CPU or ECU.

[0021] Generally, the captured image 50 output from the imaging unit 10 includes distortion of the radial component represented by, for example, Equation (1) due to the projection method of the lens and manufacturing errors. x’, x’’, y’, y’’ satisfy Equation (2), and k 1 , k 2 , k 3 are the radial distortion coefficients, X, Y, Z are the three-dimensional coordinates of the world coordinate system of the subject, u, v are the coordinates projected onto the sensor, R, t are the rotation and translation of the conversion from the world coordinate system to the camera coordinate system, f x , f y is the focal length, c x , c y indicates the distortion center. In addition to this, higher-order radial distortion and circumferential distortion may also be included. The distortion represented by parameters as in Equation (1) is called parametric distortion.

[0022]

Equation

[0023]

Equation

[0024] In order to accurately perform three-dimensional reconstruction by photogrammetry and distance information acquisition by stereo vision on this captured image 50, it is necessary to correct the distortion with high precision in advance. As an example of a widely used conventional technique for calibration to correct distortion, Non-Patent Document 1 can be cited. This conventional technique is a technique for calculating and correcting the distortion of an image using a plurality of images of a calibration board (calibration plate) taken while changing the position and orientation. This calibration board only needs to have a specific pattern printed on a plane, and can be easily created, for example, by attaching a piece of paper with the pattern printed on it to a flat plate. Also, since the position and orientation of the board during shooting can be freely arranged as long as the pattern can be sufficiently recognized, shooting can also be easily performed.

[0025] In the distortion calculation, parameters expressed by, for example, Equation (1) or Equation (2), such as the focal length of the camera, the distortion center, and the distortion calculation, are determined from the captured board image. By determining these parameters, the parametric distortion included in the captured image 50 is obtained. Also, together with the above parameters, the three-dimensional relative coordinates (external parameters) of the board with respect to the camera are calculated simultaneously. On the other hand, since this calculation assumes an optical model (parametric optical model) defined by the parameters, only the distortion of the components included in this optical model can be considered. Therefore, although this conventional technique has high ease of board creation and shooting, there is a problem in accuracy in that the distortion components that can be considered are limited to parametric distortion.

[0026] Figure 2 shows the results of the calibration according to the above prior art. The pre-correction distortion 300 indicates the image distortion included in the captured image 50, and the average within each grid divided into an appropriate size for visibility is indicated by an arrow. This arrow sets the starting point as the position where it should originally be projected and the ending point as the position where it is actually projected. However, the point that should originally be projected is calculated assuming, for example, a perspective projection without distortion. Here, for example, the case where only the parametric distortion of the radial component represented by Equation (1) is included in the captured image 50 is shown, and in particular, the case called barrel distortion is illustrated. The pre-correction distortion profile 301 shows the distortion on a straight line passing horizontally through, for example, the center of the pre-correction distortion 300. That is, both the pre-correction distortion 300 and the pre-correction distortion profile 301 represent the distortion before correction. Regarding the distortion information obtained by performing the calibration according to the above prior art on the captured image 50 including the pre-correction distortion 300, what is shown in the same format as the pre-correction distortion 300 is the parametric distortion 310. That is, regarding the obtained parametric distortion 310, the starting point is set as the position where it should originally be projected and the ending point is set as the position where it is actually projected. Similarly, regarding the obtained parametric distortion 310, what is shown in the same format as the pre-correction distortion profile 301 is the parametric distortion profile 311. The vector difference between the pre-correction distortion 300 and the parametric distortion 310, that is, the difference vector between the projected positions, is the post-correction distortion 330, and its profile is the post-correction distortion profile 331.

[0027] In this example, since the image distortion included in the captured image 50 is only the parametric distortion of the radial component represented by Equation (1), for example, in an ideal situation without the influence of noise or the like, the parametric distortion 310 obtained by calculating the distortion using the parametric optical model would be exactly the same as the pre-correction distortion 300. Therefore, the post-correction distortion 330 would be 0 at all points within the image, indicating that the distortion has been completely corrected.

[0028] FIG. 3 shows a calibration result in a case where, in addition to the parametric distortion of the radial component represented by Expression (1) for the captured image 50, there is non-parametric distortion that is not included in the parametric optical model and cannot be represented by parameters. Examples of the causes of this non-parametric distortion include manufacturing errors in the cover glass thickness of the camera, manufacturing errors and assembly errors of the lens, and the like. None of them can be completely considered in the parametric optical model, or a non-realistic number of parameters is required to consider them. For this captured image 50 including non-parametric distortion in the pre-correction distortion 300, the post-correction distortion 330 obtained by performing distortion calculation of the prior art assuming a parametric optical model mainly remains as distortion in which the non-parametric distortion component cannot be completely corrected. Further, due to the influence of the non-parametric distortion, errors also occur in each parameter and external parameter of the parametric optical model.

[0029] As another calibration technique, there is the technique described in Patent Document 1. Patent Document 1 uses a dense pattern in the central part of the board and a sparse pattern in the peripheral part of the board as a calibration board. This is to increase the amount of information by making the feature points dense in the central part of the board that appears with less distortion in the central part of the image, and to make it easier to detect the feature points in the peripheral part of the board that appears greatly distorted in the peripheral part of the image. The central part of the image is, for example, a region in the center of the image that includes a rectangle with a size of one-fourth of each side with respect to the size of the entire image, and the peripheral part of the image is a region that includes a region with a width of one-fourth of the image height near the upper and lower ends of the image and a region with a width of one-fourth of the image width near the left and right ends. Since the central part of the image represents a region with less image distortion, it does not necessarily have to be in the center of the image. For example, when the optical axis of the imaging optical system and the center of the image sensor are displaced, the vicinity of the optical axis becomes the central part of the image. This technique uses each parameter of the parametric optical model obtained in advance and the feature points in the central part of the board to first obtain the external parameters of the board. Next, using these external parameters, the points where the feature points in the peripheral part of the board should appear in an ideal state without distortion are obtained. Finally, by taking the difference between the points that should appear and the points that actually appear, it is possible to calculate the distortion including the non-parametric component for the entire image.

[0030] Figure 4 is a diagram for explaining the technique described in Patent Document 1. It shows a case where the captured image 50 includes non-parametric distortion in addition to parametric distortion. At this time, as the distortion information, the differential distortion 320 obtained from the difference between the points where the feature points should appear and the points that actually appear in the entire image is obtained. The corrected distortion 330, which is the correction result using this differential distortion 320, has removed the pre-correction distortion 300 including non-parametric distortion in the entire image. However, since the pre-correction distortion 300 is all represented by the differential distortion 320, especially in the case of a camera with large distortion, the absolute value of the differential distortion 320 is large and the function shape becomes complicated. Since the absolute value of the differential distortion 320 is large and the numerical value becomes complicated, there is a problem that it is easily affected by errors generated in feature point detection and the accuracy decreases.

[0031] Furthermore, in actual calibration, there are often non-parametric distortions in the central part of the image, and errors occur in each parameter of the parametric optical model obtained in advance and in the external parameters of the board. When an error occurs in the external parameters, an error also occurs in the points where the feature points of the entire board calculated using this information should appear, and as a result, an error also occurs in the differential distortion 320. Therefore, there is a problem that an error occurs in the differential distortion 320 due to assuming a parametric optical model for the central part of the image.

[0032] Therefore, in the present invention, while obtaining global parametric distortion information using a parametric optical model for the entire image, for the remaining distortion that cannot be expressed by the parametric optical model, non-parametric distortion information is obtained using the difference between the point where the board feature point should appear and the point that actually appears, thereby realizing high-precision calibration considering the non-parametric distortion component for the entire image. Furthermore, while adding conditions such as requiring smoothness considering the physical rationality of the non-parametric distortion component, the parametric distortion component and the non-parametric distortion component are obtained iteratively, so that, for example, the external parameters of the board, which were a problem in Patent Document 1, also converge to accurate values.

[0033] Returning to FIG. 1, the distortion correction of the present invention will be described. When calibrating the camera 1, in order to calculate the distortion information in advance, the captured image 50 captured by the imaging unit 10 is input to the distortion information processing unit 4. The distortion information processing unit 4 is, for example, a calculation system existing outside the camera 1, such as a general-purpose personal computer. The camera 1 and the distortion information processing unit 4 are connected directly or via a communication network to input and output the captured image 50. Alternatively, the captured image 50 may be input and output using an information storage medium.

[0034] The distortion information processing unit 4 calculates the parametric distortion 310 and the differential distortion 320 based on the parametric optical model from the input captured image 50. The parametric distortion 310 and the differential distortion 320 are, for example, image information having the same number of pixels as the captured image 50, and there are, for example, two types: the x component and the y component. The distortion information processing unit 4 outputs the parametric distortion 310 and the differential distortion 320, for example, after capacity compression, as parametric distortion information 3100 and differential distortion information 3200 to the parametric distortion information storage unit 2101 and the differential distortion information storage unit 2111 in the image processing unit 2 of the camera 1 respectively. As an example of capacity compression, for the parametric distortion 310, it is output to the parametric distortion information storage unit 2101, for example, in the form of the numerical values of each parameter. For the differential distortion 320, it is output to the differential distortion information storage unit 2111, for example, in the compressed image format. The above compression is an example, and the parametric distortion 310 and the differential distortion 320 may be output as they are, or other processes may be performed. The distortion information processing unit 4 and the parametric distortion information storage unit 2101 or the differential distortion information storage unit 2111 are connected directly or via a communication network, an information storage medium, etc., in the same manner as above.

[0035] The parametric distortion information storage unit 2101 stores and saves the parametric distortion information 3100 output from the distortion information processing unit 4, selects an appropriate one, and outputs it to the parametric distortion calculation unit 2100. The parametric distortion information 3100 to be saved does not have to be a single set, and for example, a plurality of sets may be stored for each combination of the temperature of the environment in which the camera 1 captures an image, the humidity of the same environment, the cumulative usage time of the camera 1, etc.

[0036] The parametric distortion calculation unit 2100 restores the parametric distortion 310 based on the parametric optical model for the parametric distortion information 3100 output from the parametric distortion information storage unit 2101, and outputs it to the distortion correction unit 21.

[0037] The differential distortion calculation unit 2110 performs processing on the differential distortion information 3200 output from the differential distortion information storage unit 2111, restores the differential distortion 320, and outputs it to the distortion correction unit 21.

[0038] As described above, the distortion correction unit 21 calculates a corrected captured image 51 that has undergone distortion correction based on the parametric distortion 310 output from the parametric distortion calculation unit 2100 and the differential distortion 320 output from the differential distortion calculation unit 2110 for the captured image 50 output from the input I / F 20, and outputs it to the output I / F 22. Additionally, for example, demosaicking processing or the like may also be performed.

[0039] Hereinafter, a specific procedure for calculating the distortion information for the captured image 50 in the distortion information processing unit 4 will be described.

[0040] For the captured image 50 that is the target for which distortion correction is desired, the parametric distortion 310 and the differential distortion 320, which are information necessary for distortion correction, are calculated and obtained separately in advance, for example. When obtaining them in advance, for example, an image of a calibration board on which periodic feature points are printed, as used in Non-Patent Document 1, is used. It is desirable to take multiple images while changing the position and orientation of the board so that the feature points are distributed throughout the image. The feature points printed on the board are, for example, a grid pattern, circles, rings, two-dimensional codes, etc. These images acquired by the imaging unit 10 are input to the distortion information processing unit 4 of the image processing unit 2, where the parametric distortion 310 and the differential distortion 320 are calculated.

[0041] FIG. 5 is a diagram for explaining the processing in the distortion information processing unit 4. As described above, the pre-correction distortion 300 and the pre-correction distortion profile 301 represent the pre-correction distortion in the captured image 50 in which the calibration board appears. The parametric distortion 310 and the parametric distortion profile 311 represent the distortion of the captured image 50 represented using the parametric optical model when each parameter included in, for example, Equation (1) or Equation (2) is given. The differential distortion 320 and the differential distortion profile 321 represent the correction applied to the captured image 50 before correcting the captured image 50 using the parametric distortion 310. The pre-correction distortion 300, the parametric distortion 310, and the differential distortion 320 may be in any form that represents the distortion with respect to the captured image 50. One example of such a form is an image format having the same number of pixels as the captured image 50. Also, for example, it is two image formats representing the distortion in the x-direction and the y-direction, respectively.

[0042] FIG. 6 is a flowchart showing the processing in the distortion information processing unit 4. Each step in FIG. 6 will be described below.

[0043] S1: This is an initialization process. First, all of the differential distortion 320 is initialized to zero.

[0044] S2: The differential distortion 320 is applied to the captured image 50 for calibration. Specifically, by performing a geometric transformation on the captured image 50 using the differential distortion 320, the non-parametric component of the captured image 50 represented by the differential distortion 320 is corrected. However, in the first iteration of the iterative calculation, since the differential distortion 320 is initialized to all zeros, the correction to the captured image 50 is not substantially performed. In FIG. 5, the calculations from the first S2 to S4 correspond to the upper half of the figure, and the differential distortion 320 is represented as all zeros.

[0045] S3: For the captured image 50 corrected in S2, parametric distortion 310 based on a parametric optical model is calculated, for example, by the technique of Non-Patent Document 1. In the first calculation, since the captured image 50 including non-parametric distortion components is represented by a parametric optical model, errors occur in each parameter of the obtained optical model and the external parameters representing the board position.

[0046] S4: Calculate the corrected distortion 330 and the corrected distortion profile 331. In this calculation, first, the reprojection points of the feature points of the calibration board are calculated. That is, using each parameter of the optical model obtained in S3 and the external parameters of the board, the position (reprojection point) where each feature point is projected on the image is calculated. Next, a vector with the reprojection point as the starting point and the feature point actually detected from the captured image 50 as the ending point is calculated. The set of these vectors is the corrected distortion 330. However, note that since errors are included in each parameter of the optical model and the external parameters of the board, the corrected distortion 330 also includes errors. Ignoring this error, the corrected distortion 330 is the difference between the parametric distortion 310 and the pre-correction distortion 300 (that is, the true value of the distortion of the captured image 50). Therefore, by treating the corrected distortion 330 obtained here as the differential distortion 320 and using it together with the parametric distortion 310 for correction during calibration, correction including non-parametric components becomes possible. The processing from S2 to S4 when the distortion calculation is repeated with the non-zero corrected distortion 330 calculated here as the differential distortion 320 is shown in the lower half of FIG. 5.

[0047] S5: Determine whether the calculated distortion has sufficient accuracy based on the magnitude of the corrected distortion 330. That is, if the corrected distortion 330 is sufficiently small throughout the image, it means that the distortion of the captured image 50 can be represented accurately enough by the parametric distortion 310 and the differential distortion 320. If the accuracy is sufficient, the distortion calculation is terminated, and if it is not sufficient, return to S2 and repeat the calculation. If you do not want to repeat the calculation, it is also possible to forcibly terminate the distortion calculation regardless of the accuracy.

[0048] In the processing after the second time and later when returning to S2, differential distortion 320 is applied to the calibration imaging image 50. Since the differential distortion 320 is the difference between the pre-correction distortion 300 and the parametric distortion 310, subtracting the geometric transformation represented by the differential distortion 320 from the imaging image 50 results in a distortion that can be represented by the parametric distortion 310 remaining in the imaging image 50. By recalculating the parametric distortion 310 for the imaging image 50 after this differential distortion correction in S3, a more accurate parametric distortion 310 with less error in each parameter of the optical model and the external parameters of the board can be obtained. Subsequently, the corrected distortion 330 calculated in S4 is expected to have a smaller absolute value than the previously calculated one. The differential distortion 320 calculated in S4 after the second time and later is updated to a new differential distortion 320 by synthesizing the current differential distortion 320 and the corrected distortion 330.

[0049] In this way, by repeatedly calculating the parametric distortion 310 for the calibration imaging image 50 to which the differential distortion 320 is applied while updating the differential distortion 320, it becomes possible to improve the accuracy of the distortion calculation including the non-parametric component represented by the differential distortion 320.

[0050] FIG. 7 is a block diagram showing the internal configuration of the distortion information processing unit 4. The distortion information processing unit 4 includes a differential distortion smoothing unit 40. The differential distortion smoothing unit 40 performs a smoothing process on the differential distortion 320 calculated in S4, for example, for noise removal. Examples of the smoothing process include a Gaussian filter, a median filter, and binning processing. In addition, a smoothing process considering physical rationality and manufacturing errors may be performed. For example, when manufacturing the cover glass of the camera, minute variations in thickness may occur in a striped pattern due to the influence of the stretching process. Due to this variation in thickness, non-parametric distortion in the form of a striped pattern occurs in the captured image 50. In this case, as an example of a smoothing process considering manufacturing errors, it is conceivable to change the magnitude of the standard deviation of the Gaussian filter in a direction orthogonal to the direction parallel to the striped pattern. In other words, smoothing is performed based on the spatial frequency of the image. By this smoothing process, noise removal of the differential distortion 320 and, thus, higher accuracy of the entire distortion calculation can be obtained. Furthermore, the iterative calculation of steps S2 to S5 is stabilized, and an effect of suppressing divergence in the iterative calculation is also obtained.

[0051] Prior information about the manufacturing error of the camera 1 can be used as an initial value when initializing the differential distortion 320 when step S1 is first performed. For example, based on the information about the manufacturing error, the distortion assumed as a distortion other than the parametric distortion can be used as the initial value of the differential distortion 320.

[0052] Hereinafter, the differences from Patent Document 1 will be described with reference to FIGS. 4 and 5. In Patent Document 1, in the calculation of the differential distortion 320, for example, the difference between the point where the feature point calculated assuming perspective projection is to be projected and the feature point (pre-correction distortion 300) appearing in the captured image 50 is calculated. That is, when the captured image 50 with the pre-correction distortion 300 is corrected by the differential distortion 320, an image with no target distortion, for example, a perspective projection image, can be obtained. Therefore, in order to represent all the distortions of the captured image 50 by the differential distortion 320, especially when the distortion in the captured image 50 is large, as shown in FIG. 4, the absolute value of the differential distortion 320 becomes large, and the functional form also becomes complex. When the functional form of the differential distortion 320 is complex and the absolute value is large, it is easily affected by errors and it is also difficult to remove noise. Also, since the calculation of the differential distortion 320 is only performed once, errors due to the influence of non-parametric component distortions in the camera internal parameters and the board external parameters, which are the prerequisite information for the calculation, are added.

[0053] On the other hand, the differential distortion 320 in the present invention is the difference between the parametric distortion 310 and the pre-correction distortion 300. That is, for the captured image 50 with the pre-correction distortion 300, as a first step, it is corrected by the differential distortion 320, and for the captured image after the differential correction, as a second step, it is corrected by the parametric distortion 310. In this way, an image with no target distortion, for example, a perspective projection image, can be obtained. In this case, since the distortion of the captured image 50 is globally represented by the parametric distortion 310, even when the distortion in the captured image 50 is particularly large, as shown in FIG. 5, the absolute value of the differential distortion 320 becomes small, and the functional form also becomes simpler, distributed around 0. Since the functional form of the differential distortion 320 is simpler and the absolute value is smaller, it is less affected by errors and it is also easier to remove noise. Also, when the calculation of the differential distortion 320 is repeated, the influence of non-parametric component distortions on the camera internal parameters and the board external parameters can be reduced, and the calculated distortion can also be made more accurate.

[0054] Furthermore, by separately retaining the distortion information as the global parametric distortion 310 and the differential distortion 320, when the distortion is recalculated by the distortion information processing unit 4, for example, it becomes possible to fix the parametric distortion 310 and update only the differential distortion 320. By doing so, for example, when it is assumed that the variation in distortion is small, it becomes possible to reduce the number of captured images 50 in which the calibration board appears for distortion recalculation.

[0055] <Embodiment 1: Summary> According to the imaging system 100 according to Embodiment 1, it is possible to realize a camera to which high-precision calibration applicable to non-parametric distortion components can be applied. In the high-precision calculation of distortion, while combining the parametric distortion of the parametric optical model and the additional distortion of the non-parametric component therefrom, by repeating these distortion calculations, it becomes possible to perform high-precision distortion calculation considering non-parametric distortion components. By using the distortion calculated with such high precision, high-precision calibration can be realized.

[0056] <Embodiment 2> FIG. 8 is a block diagram of the imaging system 100 according to Embodiment 2 of the present invention. In Embodiment 1, it was assumed that the distortion information processing unit 4 exists, for example, on a general-purpose personal computer outside the image processing unit 2. In Embodiment 2, as shown in FIG. 8, the case where the distortion information processing unit 4 exists inside the image processing unit 2 will be described. The distortion information processing unit 4 in Embodiment 2 may be realized by a circuit as a part of the image processing unit 2, or at least a part thereof may be realized by a processor such as a CPU that executes a program and a memory. The captured image 50 received by the distortion information processing unit 4 may be the captured image 50 output from the imaging unit 10, or the captured image 50 converted into a digital signal output from the input I / F 20.

[0057] In Embodiment 2, since the distortion information processing unit 4 exists inside the image processing unit 2, recalculation of the parametric distortion 310 and the differential distortion 320 can be performed only by the camera 1 and the image processing unit 2. When this recalculation is necessary, for example, a calibration board may be photographed multiple times. This recalculation is performed, for example, when the actual distortion of the camera 1 fluctuates with respect to the parametric distortion information 3100 and the differential distortion information 3200 that were calculated previously and stored in the parametric distortion information storage unit 2101 and the differential distortion information storage unit 2111, and the accuracy of calibration using the distortion information decreases. When performing the recalculation, in Embodiment 1, it is necessary to prepare, for example, an external general-purpose personal computer in which the distortion information processing unit 4 is implemented, but in Embodiment 2, only the camera 1 and the image processing unit 2 are required. Therefore, even at a site where it is not possible to prepare, for example, an external general-purpose personal computer in which the distortion information processing unit 4 is implemented, it is possible to reduce the interruption time for recalculation.

[0058] <Embodiment 3> FIG. 9 is a block diagram of an imaging system 100 according to Embodiment 3 of the present invention. Embodiment 3 will describe a case where the parametric distortion calculation unit 2100, the differential distortion calculation unit 2110, the parametric distortion information storage unit 2101, the differential distortion information storage unit 2111, and the distortion information processing unit 4 exist outside the image processing unit 2. The parametric distortion calculation unit 2100, the differential distortion calculation unit 2110, the parametric distortion information storage unit 2101, the differential distortion information storage unit 2111, and the distortion information processing unit 4 are implemented, for example, on a computing system such as a general-purpose personal computer. The distortion correction unit 21, the parametric distortion calculation unit 2100, and the differential distortion calculation unit 2110 are connected, for example, directly or via a communication network or the like.

[0059] In Embodiment 3, since the parametric distortion calculation unit 2100 and the like exist outside the image processing unit 2, the processing in the image processing unit 2 can be lightened, and it is possible to reduce the power consumption and cost.

[0060] <Regarding a modification of the present invention> The above-described embodiments have been described in detail for the purpose of clearly explaining the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, or replace other configurations for a part of the configuration of each embodiment.

[0061] In the above embodiments, the configuration such as the arrangement, dimensions, and shape of each component of the camera is not limited to the examples described or illustrated above as long as the object of the present invention can be achieved. Also, the words expressing the relationship between components, etc. are not limited to the strict meaning as the words as long as the object and effects of the present invention can be achieved, and can include cases substantially the same as the meaning.

[0062] The above embodiments are not limited to the camera, and can be provided in various forms such as a stereo camera with two cameras arranged in parallel, or a camera system with multiple cameras installed.

[0063] In the above embodiments, it has been described that steps S2 to S4 are repeated. However, if the calibration result has sufficient accuracy, the number of repetitions may be one. Further, even when S5 is omitted and S2 to S4 are each implemented only once, since it is still possible to suppress the differential distortion 320 by that single implementation, the effects of the present invention can be exhibited within that limit.

[0064] In the above embodiments, it has been described that the differential distortion 320 is the difference between the parametric distortion 310 and the pre-correction distortion 300. It is not necessarily required that the differential distortion 320 be exactly the difference between the parametric distortion 310 and the pre-correction distortion 300 itself, and a value of the same nature derived from this difference may be used as the differential distortion 320. For example, a value obtained by multiplying the difference between the parametric distortion 310 and the pre-correction distortion 300 by a coefficient of 1.0 or less may be used as the differential distortion 320. In this case, by reducing the numerical value of the differential distortion 320, it is possible to suppress the divergence of the convergence calculation.

[0065] In the above embodiments, each functional unit other than the camera 1 may be arranged on a device such as an external computer of the camera 1, or only a part of these functional units may be integrally configured with the camera 1. When configured integrally with the camera 1, for example, these functional units may be arranged on the image processing unit 2, and the image processing unit 2 may be configured by a circuit device or the like that is integrally configured with the camera 1.

[0066] In the above embodiments, each functional unit other than the camera 1 can also be configured by hardware such as a circuit device that implements these functions, or can be configured by software that implements these functions being executed by an arithmetic device such as a CPU.

Explanation of Reference Numerals

[0067] 1... Camera 10... Imaging unit 2... Image processing unit 20... Input I / F 21... Distortion correction unit 22... Output I / F 2100... Parametric distortion calculation unit 2110... Differential distortion calculation unit 2101... Parametric distortion information storage unit 2111... Differential distortion information storage unit 300... Distortion before correction 301... Distortion before correction profile 310... Parametric distortion 3100... Parametric distortion information 311... Parametric distortion profile 310... Parametric distortion 320... Differential distortion 3200... Differential distortion information 321... Differential distortion profile 330... Distortion after correction 331... Distortion after correction profile 4... Distortion information processing unit 40... Differential distortion smoothing unit 50…Captured image 51…Corrected captured image 52…Differentially corrected captured image

Claims

1. An imaging system for photographing a subject, comprising: a distortion information processing unit that calculates parametric distortion based on a parametric optical model represented by internal parameters of the camera and differential distortion representing a non-parametric distortion component from an image captured by the camera; a distortion correction unit that applies distortion correction for each of the parametric distortion and the differential distortion to the image; wherein the distortion information processing unit performs a step of applying the differential distortion to the image; the distortion information processing unit performs a step of calculating the parametric distortion from the image to which the differential distortion has been applied; the distortion information processing unit performs a step of calculating, as an updated value of the differential distortion, a difference between the distortion of the image and the calculated parametric distortion or a value derived from the difference; the distortion information processing unit performs, one or more times, the step of applying the differential distortion, the step of calculating the parametric distortion, and the step of calculating the updated value of the differential distortion until the updated value of the differential distortion becomes equal to or less than a threshold value. An imaging system characterized by the above.

2. the distortion information processing unit calculates a reprojection point of a feature point of the image; the distortion information processing unit calculates a vector having the reprojection point as a starting point and an actual feature point on the image as an end point; the distortion information processing unit uses, as the updated value of the differential distortion, the vector or a vector obtained by multiplying the vector by a coefficient less than 1.

0. The imaging system according to claim 1, characterized by the above.

3. The imaging system further includes a differential distortion smoothing unit that smooths the differential distortion; the differential distortion smoothing unit smooths the differential distortion based on a spatial frequency of the image and a manufacturing error of the camera. The imaging system according to claim 1, characterized by the above.

4. a parametric distortion information storage unit that stores parametric distortion information converted from the parametric distortion; a differential distortion information storage unit that stores differential distortion information converted from the differential distortion; a parametric distortion calculation unit that restores the parametric distortion from the parametric distortion information; a differential distortion calculation unit that restores the differential distortion from the differential distortion information; wherein The distortion correction unit applies distortion correction to the image for each of the parametric distortion restored by the parametric distortion calculation unit and the differential distortion restored by the differential distortion calculation unit. The imaging system according to claim 1, characterized in that.

5. The parametric distortion information storage unit stores the parametric distortion information for each combination of the temperature of the environment in which the image is captured, the humidity of the environment, and the cumulative usage time of the camera. The differential distortion information storage unit stores the differential distortion information for each combination of the temperature, the humidity, and the cumulative usage time. The imaging system according to claim 4, characterized in that.

6. The distortion information processing unit sets an initial value of the differential distortion based on prior information of manufacturing errors. The imaging system according to claim 1, characterized in that.

7. The distortion correction unit is implemented on an image processing unit that receives the image from the camera via an interface. The distortion information processing unit is implemented on a computer outside the camera and outside the image processing unit. The imaging system according to claim 1, characterized in that.

8. The distortion information processing unit and the distortion correction unit are implemented on an image processing unit that receives the image from the camera via an interface. The imaging system according to claim 1, characterized in that.

9. The distortion information processing unit, the distortion correction unit, the parametric distortion information storage unit, the differential distortion information storage unit, the parametric distortion calculation unit, and the differential distortion calculation unit are implemented on an image processing unit that receives the image from the camera via an interface. The imaging system according to claim 4, characterized in that.

10. The distortion correction unit is implemented on an image processing unit that receives the image from the camera via an interface. The distortion information processing unit, the parametric distortion information storage unit, the differential distortion information storage unit, the parametric distortion calculation unit, and the differential distortion calculation unit are implemented on a computer outside the camera and outside the image processing unit. The imaging system according to claim 4, characterized in that.

Images