Image processing device, imaging apparatus, image processing method, and program

Abstract

To properly update calibration information in a case of the occurrence of vibration in a camera.SOLUTION: An image processing device for calibrating an imaging apparatus with an optical system and an image sensor comprises: acquisition means which acquires an image captured by the imaging apparatus; detection means which detects whether vibration has occurred in the imaging apparatus; and update means which when the detection means detects occurrence of the vibration, updates calibration information for correcting the image captured by the imaging apparatus, based on the image acquired by the acquisition means.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to calibration of imaging devices. [Background technology]

[0002] There may be a misalignment in the optical axis of the lens optical system of an imaging device. In order to correct the influence of the misalignment in the captured image obtained by the imaging device, there is a method of calculating calibration information and correcting the captured image by taking the calibration information into account.

[0003] Patent Document 1 describes a method of correcting stereo images by estimating camera parameters as calibration information when camera setting values ​​are changed by a user operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-135495 A Summary of the Invention [Problem to be solved by the invention]

[0005] When the camera comes into contact with a tripod or the like and vibrations occur in the camera, the lens attached to the camera may become misaligned. In this case, the calibration information generated before the camera vibrations become inappropriate. Therefore, in a method in which the calibration information is updated by a user's operation as in Patent Document 1, if the user does not operate the camera after vibrations occur, the captured image is corrected based on inappropriate calibration information. In this case, an unnatural-looking image may be generated. [Means for solving the problem]

[0006] The image processing device disclosed herein is an image processing device for calibrating an imaging device having an optical system and an image sensor, and is characterized in having an acquisition means for acquiring an image obtained by the imaging device, a detection means for detecting whether or not vibration has occurred in the imaging device, and an update means for updating, when the detection means detects the occurrence of vibration, calibration information for correcting the image obtained by the imaging device based on the image acquired by the acquisition means. Effect of the Invention

[0007] According to the technology disclosed herein, when vibration occurs in the camera, the calibration information can be appropriately updated. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating the internal configuration of the camera. [Diagram 2] FIG. 2 is a diagram for explaining optical axis deviation. [Diagram 3] FIG. 2 is a diagram for explaining the functional configuration of a camera. [Figure 4] 11 is a flowchart for explaining a process of converting stereo images based on calibration information. [Diagram 5] FIG. 1 is a diagram for explaining image conversion. [Figure 6] 6 is a flowchart for explaining a process of determining calibration values ​​corresponding to stereo images. [Figure 7] FIG. 2 is a diagram for explaining the hardware configuration of an image processing apparatus. [Figure 8] 11 is a flowchart for explaining a process of converting stereo images based on calibration information. [Figure 9] 11 is a flowchart for explaining a process of converting stereo images based on calibration information. [Figure 10] 11 is a flowchart for explaining a process of converting stereo images based on calibration information. [Figure 11] 10 is a flowchart for explaining details of thread processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the technology of the present disclosure will be described in detail based on the embodiments with reference to the accompanying drawings. The following embodiments merely show specific examples of implementing the technology of the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited. The technology of the present disclosure can be implemented in various forms without departing from its technical idea or main features.

[0010] <Embodiment 1> There is a method to display VR (Virtual Reality) images by detecting the tilt of the HMD (Head Mounted Display) and displaying in real time an image cropped from a fisheye image in the front direction based on the display angle of the HMD. The sense of realism can also be improved by displaying stereo images on the left and right displays of the HMD.

[0011] When capturing images using a camera equipped with left and right lens optical systems (stereo lens optical systems) to obtain stereo images, optical axis misalignment of the lens optical systems with respect to the image sensor may occur. This optical axis misalignment causes large vertical shifts of the same subject in the left image obtained through the left lens optical system and the right image obtained through the right lens optical system. When the left and right images with such optical axis misalignment are fisheye images, the left and right images obtained by converting the left and right fisheye images into equirectangular images may be viewed on an HMD (head mounted display). However, if vertical misalignment occurs, the subject cannot be fused with the human eye.

[0012] When a camera has an interchangeable lens optical system and the lens optical system is used by attaching it to a mount on the camera body, physical misalignment is likely to occur. The lens optical system is replaced when the camera is turned off. Therefore, if calibration can be performed at the stage where a focused image is obtained after the camera is turned on, regardless of whether an image has been captured or not, the calibration information obtained as a result of the calibration can be used to correct the captured image in subsequent images.

[0013] When a plurality of images captured by a plurality of image capturing devices are to be synthesized, it is also desirable to perform image conversion on the plurality of images taking into account calibration information.

[0014] However, vibrations may be transmitted to the lens optical system attached to the camera when the camera is mounted on a tripod, when a person touches the camera, or when the camera is subjected to vibrations from the floor. When vibrations are transmitted to the lens optical system, the optical axis shift described above may occur. In this case, the optical axis shift may not be able to be corrected using calibration information from a previous calibration. Therefore, in this embodiment, a method of detecting the occurrence of vibrations and calibrating the optical axis shift in response to the detection of the vibrations will be described.

[0015] [Configuration of imaging device] FIG. 1 is a diagram for explaining the internal configuration of a camera 100. A CPU 105 cooperates with other components based on computer programs such as an operating system (OS) and application programs to control the operation of the entire camera. Note that in this embodiment, the description will be given assuming that there is one CPU, but this is not limiting and a configuration having multiple CPUs may be used. In this case, each process can be performed in parallel by multi-thread processing. A bus 101 controls the flow of data within the camera.

[0016] The RAM 102 is a random access memory. The graphic processor 103 performs calculation processing required to display an image on the display 104. The graphic processor 103 is a processor that enables encoding and decoding of images in real time. The user I / F 106 is an interface (I / F) to which a touch panel, switches, buttons, etc. are integrally connected, and accepts user operations such as starting or stopping video capture.

[0017] The display 104 is a display device that displays commands input from the user I / F 106, responses thereto from a PC (personal computer), and the like. A UI (user interface) screen or an image obtained as a result of processing can be displayed on the display 104 via the graphic processor 103. The graphic processor 103 can perform geometric transformation of an input image, and can also input / output an image from the RAM 102 and output the image directly to the display 104.

[0018] The external storage 107 is a non-volatile storage unit that functions as a so-called memory card. The external data input / output I / F 108 exchanges data via a network.

[0019] The stereo lens optical system 111 is arranged so that incident light is imaged on the image sensor 110. The stereo lens optical system 111 of this embodiment has two lenses, one on the left and one on the right, each of which is a fisheye lens with an angle of view of 180 degrees. Images obtained from each lens are imaged on one image sensor 110, and left and right fisheye images (stereo fisheye images) corresponding to the two left and right lenses are obtained. The stereo lens optical system 111 is replaceable, and is fixed to the camera body by connecting to a mount 201 (see FIG. 2).

[0020] The stereo lens optical system 111 is controlled by the CPU 105 via the ISP 109. The CPU 105 controls the aperture and focus of the stereo lens optical system 111 to control image capture. The image sensor 110 will be described as being driven so as to acquire one frame of image data every 33.333 ms (hereinafter referred to as 33 ms), that is, at 30 fps. The captured image data is temporarily stored in the RAM 102.

[0021] The image sensor 110 converts the light obtained by the stereo lens optical system 111 into digital image data. In this embodiment, it is assumed that the light is converted into digital image data in a Bayer array. Unless otherwise specified, a captured image obtained by capturing an image based on light rays passing through the left lens of the stereo lens optical system 111 is called a left image, and a captured image obtained by capturing an image based on light rays passing through the right lens is called a right image. The right and left images are collectively called left and right images or stereo images.

[0022] The ISP 109 is an image signal processor, a dedicated processor for performing image processing. The ISP 109 processes digital image data in the Bayer format obtained by the image sensor 110 to generate an RGB image, and stores the image in the RAM 102. The internal image format in this embodiment is described as an RGB image, but is not limited to this and may be a YUV image or a monochrome luminance image.

[0023] The stored image data is subjected to image quality improvement processing and encoding processing by the CPU 105 or the graphic processor 103, and then recorded in the external storage 107. Note that obtaining an image (image data) means obtaining the image from the ISP 109 or the external storage 107.

[0024] The inertial sensor unit 112 is composed of a three-axis gyro sensor and a three-axis capacitance acceleration sensor. Normally, an inertial sensor unit is used to detect the direction of gravity to display a level on a display and to detect shaking for vibration isolation processing, but in this embodiment, it is used to detect impacts and vibrations. Details will be described later.

[0025] The inertial information obtained from the inertial sensor unit 112 is stored in the RAM 102 as a ring buffer capable of storing 10 seconds' worth of data after being corrected by the CPU 105, such as by removing noise. The inertial information will be described as corrected information as described above. The inertial information is recorded as information synchronized with the image captured by the image sensor 110 at the same time as it is stored in the ring buffer capable of storing 10 seconds' worth of data. The inertial information in this embodiment is information that allows the CPU 105 to determine whether the camera 100 has received a strong external force, that is, whether it has detected the occurrence of a strong vibration. The inertial information for image data being captured is acquired from the inertial sensor unit 112, and the inertial information for recorded image data is acquired at the same time as the image data is acquired, by adding inertial information corresponding to the image data as metadata in advance.

[0026] The acceleration sensor information included in the inertial information is made up of three corrected values, XYZ, with the gravity component cancelled out. The acceleration sensor information is made up of eight sets of information, each set consisting of XYZ, as described above, with 33 ms for one frame divided into eight, i.e., 4.167 ms, as one unit. The frame rate may be a frame rate other than 30 fps. For example, the frame rate may be 29.97 fps, 50 fps, 60 fps, etc. In this case, the unit time of the acceleration sensor information is a value close to 4 ms, which is obtained by dividing the reciprocal of the frame rate by an integer. For example, if the frame rate is 50 fps, the unit time is 4 ms, and if it is 60 fps, it is 4.167 ms.

[0027] In this embodiment, the programs and data to be executed are assumed to be recorded in the external storage 107. The programs and data are input and stored in the RAM 102, and then executed by the CPU 105. The programs and data are input and output via the bus 101. Furthermore, it is assumed that the processing data can be recorded in the external storage 107 or stored in the RAM 102 and shared with other programs.

[0028] [Optical axis misalignment] FIG. 2 is a diagram for explaining the shift of the optical axis (optical axis shift). FIG. 2(a) shows a camera 100 in which a stereo fisheye lens, which is the stereo lens optical system 111 of this embodiment, is attached to a mount 201. The solid arrows in FIG. 2 show the trajectory of the incident light ray. For example, in a state in which the optical axes of the left and right lenses are shifted, the light incident on the image sensor 110 from the center position of the lens (light incident from infinity) does not enter parallel and perpendicular to the sensor surface of the image sensor 110 after passing through the lens. This causes the shift when the image is viewed with the HMD. Therefore, it is desirable that the optical axis of the light ray passing through the left lens optical system is perpendicular to the sensor surface as much as possible as shown by the dotted arrow, and the light ray passing through the left lens optical system and the light ray passing through the right lens optical system (left and right light rays) are parallel. Therefore, information on the extent to which the offset of the optical axis center of the fisheye image and the tilt of the optical axis should be corrected so that the light ray becomes perpendicular to the sensor surface and the two light rays become parallel is generated as calibration information. Then, by converting the fisheye image into an equirectangular image while taking into account the offset shift of the optical axis center and the inclination of the optical axis, it is possible to obtain an equirectangular image in which the effects of the optical axis shift are suppressed. Note that correcting the inclination of the optical axis is synonymous with creating an equirectangular image in which the longitude and latitude on the equirectangular image are shifted.

[0029] In this embodiment, as shown in FIG. 3(a), the following description will be given taking as an example a camera 100 that is an imaging device that obtains a stereo image by focusing left and right light rays from a stereo lens optical system 111 on a single image sensor 110, but the configuration of the imaging device is not limited to this configuration. In addition, as shown in FIG. 2(b), a camera having multiple optical systems may be configured by configuring multiple cameras in which one of the left and right lens optical systems and the image sensor 110 are paired. In this case, optical axis deviation means that a light ray on the optical axis incident from infinity is not incident at a right angle to the sensor surface. As long as there is a mechanism for detecting the occurrence of vibration and determining calibration information as in this embodiment, the configuration of the camera system to which the method of this embodiment can be applied is not limited.

[0030] [Function configuration] 3 is a diagram for explaining the configuration of a function for determining calibration information for stereo images in camera 100 and converting the images. Camera 100 of this embodiment has a calibration information acquisition unit 310, a vibration information acquisition unit 301, a vibration detection unit 302, a calibration information management unit 303, a calibration information update unit 304, an image acquisition unit 305, and an image conversion unit 306. Details of the processing executed by each functional unit in FIG. 3 will be described in conjunction with the description of the flowcharts described later.

[0031] 3 by using and controlling computer programs and data stored in external storage 107. Camera 100 may be configured to include one or more pieces of dedicated hardware other than CPU 105 and to execute at least a part of the processing by CPU 105. Examples of the dedicated hardware include a processor used for image processing and control, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0032] [flowchart] FIG. 4 is a flowchart for explaining a process of calculating calibration information in this embodiment and converting a stereo image while correcting it based on the calibration information. The series of processes shown in the flowchart in FIG. 4 are executed by the CPU 105 in the order of the arrows from "start" to "end" unless otherwise specified. That is, the CPU 105 loads the program code in the RAM 102 and executes it. In addition, some or all of the functions of the steps in FIG. 4 may be realized by hardware such as an ASIC or an electronic circuit. Note that the symbol "S" in the explanation of each process means a step in the flowchart. The flowchart in FIG. 4 is executed when a video capture start button provided on the camera 100 is pressed.

[0033] In S401, the calibration information acquisition unit 310 acquires calibration information managed by the calibration information management unit 303. The calibration information management unit 303 stores and manages, in a storage unit, calibration information used when the image conversion unit 306 performs image conversion involving correction of optical axis deviation.

[0034] In this embodiment, the calibration information is information that expresses the optical axis deviation of the right lens in the stereo lens optical system 111 with respect to the left lens as horizontal and vertical tilt and horizontal and vertical offset deviations.

[0035] At the start of this flowchart, the calibration information management unit 303 stores and manages pre-calculated calibration information in a storage unit. If S405, which will be described later, has not been executed, in S401 the calibration information acquisition unit 310 acquires initial calibration information. If S405 has been executed in the previous frame, in S401 the calibration information acquisition unit 310 acquires the calibration information updated in S405 in the processing of the previous frame.

[0036] In S402, the image acquisition unit 305 acquires stereo images of a processing target frame obtained by the image sensor 110 capturing left and right light beams that have passed through the stereo lens optical system 111. As described above, stereo images in which the left and right images are fisheye images are acquired. The image acquisition unit 305 outputs the acquired stereo images to the calibration information update unit 304 and the image conversion unit 306. In this embodiment, fisheye images in the RGB color space corresponding to the left and right lens optical systems for one frame stored in the RAM 102 are acquired as stereo images of the processing target frame in synchronization with the image sensor 110 every 33 ms. If the image data is encoded, the encoded data is sent to the graphic processor 103, where it is decoded and an image converted into an internal image format is acquired.

[0037] In S403, the vibration information acquisition unit 301 acquires vibration information and outputs the acquired vibration information to the vibration detection unit 302. In this embodiment, the vibration information will be described as information from the acceleration sensor of the inertial sensor unit 112. The vibration information acquisition unit 301 acquires vibration information for 33 ms before S403 is executed from the ring buffer. For example, acceleration sensor information for 33 ms of the frame preceding the currently processed frame is acquired. It is assumed that continuous acceleration sensor information, neither too much nor too little, is obtained every time S403 is executed.

[0038] In S404, the vibration detection unit 302 determines whether or not a predetermined vibration has occurred in the camera 100. If the vibration detection unit 302 detects the occurrence of the predetermined vibration based on the information of the acceleration sensor acquired in S403, it determines that the predetermined vibration has occurred in the camera 100. The vibration detection unit 302 outputs the determination result to the calibration information update unit 304.

[0039] The vibration detection unit 302 calculates norms for XYZ data, which is acceleration information in units of 4.167 ms included in the acceleration sensor information for 33 ms acquired in S403, for example. The vibration detection unit 302 detects the occurrence of a predetermined vibration if one or more norms indicate that an impact of 10 G or more has been applied to the camera 100. For example, an impact received by the camera 100 due to the camera 100 coming into contact with a tripod or the like is detected as the predetermined vibration. For example, if any one of eight pieces of information in units of 4.167 ms included in the acceleration sensor information for one frame indicates that an impact of 10 G or more has been applied, it is detected that a predetermined vibration has occurred in the camera 100.

[0040] In this embodiment, the information of the acceleration sensor is used to detect whether a predetermined vibration (shock) has occurred in the camera 100, but the method of detecting the occurrence of the predetermined vibration is not limited. Whether the camera 100 has received the predetermined vibration may be detected based on the information of the gyro. Usually, the information of the gyro is information of angular velocity of three axes. Therefore, when the angular velocity information of the three axes is expressed in a quaternion and the angular velocity obtained by slerp (spherical linear interpolation) exceeds 100 degrees / second, it is assumed that the predetermined vibration (shock) has occurred, and the predetermined vibration may be detected. In addition to the angular velocity, the predetermined vibration may be detected based on the conversion of the angular velocity per unit time. Note that the information of the inertial sensor unit 112 is not used to detect the positional deviation of the lens of the camera 100, but is used to detect the change in the position and attitude of the camera, and is used only to detect whether the vibration has occurred.

[0041] When the vibration detection unit 302 determines that a predetermined vibration has occurred (YES in S404), the process proceeds to S405 to operate the calibration information update unit 304, and operates the calibration information update unit 304. When the vibration detection unit 302 determines that a predetermined vibration has not occurred (NO in S404), the process proceeds to S407.

[0042] In S405, the calibration information update unit 304 determines the calibration values ​​corresponding to the stereo image of the current frame to be processed acquired in S402. The process of S405 is executed as a subroutine. The details of the process of S405 will be described later.

[0043] In S406, the proofreading information update unit 304 replaces the pre-update proofreading information currently stored and managed in the memory unit by the proofreading information management unit 303 with the proofreading value determined in S405, and updates the proofreading information used by the image conversion unit 306. In addition, the proofreading information update unit 304 outputs the updated proofreading information to the image conversion unit 306.

[0044] In S407, the image conversion unit 306 uses the input calibration information to convert the stereo images of the processing target frame acquired in S402 into left and right equirectangular images (stereo equirectangular images) in which the optical axis shift has been corrected. In this way, the image conversion unit 306 also functions as a correction unit that corrects the optical axis shift. Such conversion processing into equirectangular images with optical axis shift correction is performed one after another on the continuously input stereo images.

[0045] If it is determined that the predetermined vibration has occurred, updated calibration information, which is the calibration value determined in the previous S405, is input, and conversion processing to an equirectangular image with optical axis shift correction is performed based on the updated calibration information. If it is determined that the predetermined vibration has not occurred, the calibration information acquired by the calibration information acquisition unit 310 in S401 is input to the image conversion unit 306, and conversion processing to an equirectangular image with optical axis shift correction is performed based on the calibration information.

[0046] The image conversion by the image conversion unit 306 is performed by the graphic processor 103 under the control of the CPU 105. Alternatively, the ISP 109 may be provided with an image conversion function, and the ISP 109 may perform the image conversion of S206. Alternatively, the image to be processed may be buffered, and the buffered image may be subjected to image conversion. In other words, the image conversion may be performed asynchronously with the image capture, allowing for an output delay.

[0047] Since the image projection method in VR is often equirectangular images, the captured image is often converted to an equirectangular image and stored. Therefore, since it is efficient to reflect the calibration information when converting to an equirectangular image, the image conversion executed in S407 is described as a conversion to an equirectangular image accompanied by optical axis shift correction. However, the image conversion executed in S407 may be a geometric transformation accompanied by optical axis correction, and the method of image conversion is not limited.

[0048] FIG. 5 is a diagram for explaining the image conversion executed in S407. The image in FIG. 5 shows the right image constituting the stereo image. FIG. 5(a) is a diagram showing a fisheye image that does not require correction when the optical axis is in the position indicated by the solid arrow in FIG. 3. FIG. 5(b) is a diagram showing an image that requires correction by calibration information in which the optical axis of the right lens is offset to the upper left and the image of the optical system is also offset to the upper left. In reality, it is rare for the optical axis to be shifted to this extent, but it is illustrated for the purpose of explanation. FIG. 5(c) is an image after conversion processing to an equirectangular image accompanied by optical axis shift correction has been performed on the image in FIG. 5(b). If the calibration information set in this way is correct, conversion to an equirectangular image accompanied by deformation to correct the optical axis shift is performed.

[0049] In this embodiment, the calibration information is used to correct the optical axis shift during coordinate conversion, but the image itself may be converted based on the calibration information. By detecting the movement from the corrected image, the accuracy of the movement detection can be improved.

[0050] In S408, the image conversion unit 306 determines whether to end the process. In this embodiment, the method of determining whether to end the process is to determine that the process is to end when it is detected that the user has pressed the image capture stop button of the camera 100. The pressing of the button is detected based on information from the user I / F 106.

[0051] If it is determined that the process is to be ended (YES in S408), the flowchart in FIG. 4 is ended. If it is determined that the process is to be continued, the process returns to S401. In the next process of S401 to S407, the process is executed with the next frame as the frame to be processed. That is, in S402, a stereo image of the next frame is acquired, and in S403, new vibration information is acquired. In S407, image conversion is performed on the stereo image of the next frame. In this way, in the flowchart in FIG. 4, S402 to S407 are repeatedly executed every 33 ms in synchronization with the image sensor 110 until the process is ended, so that image conversion is performed in which the calibration information is reflected in the stereo image while the stereo image is continuously acquired.

[0052] [Proofreading information update process] Fig. 6 is a flowchart for explaining the details of a method for determining calibration values ​​corresponding to acquired stereo images. The details of the process of determining calibration information for correcting the optical axis shift in S405 will be explained using the flowchart in Fig. 6.

[0053] In S601, the calibration information update unit 304 acquires a stereo image consisting of a left image and a right image. The acquired image is the image specified by the caller of this flowchart. For example, the stereo image of the processing target frame acquired in S402 is acquired.

[0054] In S602, the calibration information update unit 304 detects feature points of the left and right images that constitute the stereo image. If the internal image format is an RGB color image or a YUV image, the acquired stereo image is converted into a luminance image and feature points are detected.

[0055] In this embodiment, the calibration information update unit 304 detects feature points of the left and right images using the ORB algorithm. In this embodiment, the feature points are determined by calculations performed by the CPU 105, but if the ISP 109 has a dedicated circuit, the ISP 109 may perform the calculations.

[0056] In S603, the proofreading information update unit 304 matches the feature points of the left and right images. In this embodiment, ORB feature amounts for the feature points of the left and right images are calculated, and the feature points of the left and right images with the smallest Hamming distance are matched. The processing of S603 is performed by the CPU 105, but if the ISP 109 has a dedicated circuit, it may be performed by the ISP 109.

[0057] Although the feature points are described as being detected by the ORB algorithm, the method of detecting the feature points is not limited thereto. For example, other feature point detection methods and matching methods such as the SIFT method may be used. Alternatively, the feature points may be associated using template matching.

[0058] In S604, the calibration information update unit 304 sets a provisional calibration value. In this embodiment, the elements of the calibration value are elements indicating the tilt of the horizontal and vertical optical axes and the offset deviation of the horizontal and vertical optical axes. Each time S604 is executed, the calibration information update unit 304 sets a different provisional calibration value so that the provisional calibration value converges in the form of so-called iterative calculation. Methods for converging values ​​by iterative calculation include the steepest descent method, Newton's method, Nelder Mead method, and Levenberg-Macquart method, and in this embodiment, the Nelder-Mead method, which is robust against comparative outliers, is used.

[0059] In S605, the calibration information update unit 304 calculates an evaluation value for evaluating the set provisional calibration value. Specifically, the calibration information update unit 304 identifies the ray direction from the coordinates of the matched left and right feature points using the provisional calibration value set in S603. Then, the calibration information update unit 304 calculates the closest distance of the identified ray pair, and calculates the sum of squares of the closest distances of all ray pairs corresponding to all feature point pairs as an evaluation value for the provisional calibration value.

[0060] The XY coordinate system of an ideal equidistant projection fisheye image that has no distortion and does not require calibration can be converted to polar coordinates. In this embodiment, the XY coordinate system is converted to polar coordinates by removing distortion and adding the inclination of the optical axis and offset deviation indicated by the provisional calibration value. Since the polar coordinates correspond to the direction of the light rays passing through the entrance pupil, if the center of the entrance pupil positions of the two lenses is taken as the origin, the light rays can be uniquely positioned in a three-dimensional space. The more correct the provisional calibration value is, the closer the two light rays corresponding to the feature point pair are to each other, and the more incorrect the provisional calibration value is, the farther apart they are. In other words, the more correct the provisional calibration value is, the smaller the evaluation value calculated based on the provisional calibration value is.

[0061] In S606, the calibration information update unit 304 determines whether the evaluation value calculated in S605 is the smallest value among the evaluation values ​​calculated since the start of the flowchart in Fig. 6. If the evaluation value calculated in S605 is the smallest evaluation value, the provisional calibration value set in S604 is updated as the provisional calibration value corresponding to the smallest evaluation value.

[0062] In this way, if the evaluation value of the provisional calibration value set in S604 this time is smaller than the minimum evaluation value already held, the minimum evaluation value and the provisional calibration value corresponding to the minimum evaluation value are updated. By repeating this process, the minimum evaluation value and an appropriate provisional calibration value corresponding to the minimum evaluation value are held. The method of the process executed in S606 is one of the so-called least squares methods, and the evaluation value is the sum of squares of the errors when calculated by the least squares method, but is not limited to the method of repeating iterations, and any method classified as the least squares method may be used.

[0063] In S607, the calibration information update unit 304 determines whether the provisional calibration value corresponding to the minimum evaluation value has converged. If it is determined that the provisional calibration value has converged (YES in S607), this flowchart ends, and the provisional calibration value corresponding to the currently held minimum evaluation value is output as the determined calibration value. If it is determined that the provisional calibration value has not converged (NO in S607), the process returns to S604. Then, the processes of S604 to S607 are repeated until the provisional calibration value converges. In this embodiment, it is determined that the provisional calibration value has converged when the minimum evaluation value becomes an evaluation value in which the root mean square value that can be calculated from the sum of the squares of the closest distances of all ray pairs is 0.01 degrees or less in terms of the angle of view.

[0064] 6 is executed, the calibration values ​​output are replaced with calibration information and stored in S406. The image is converted in S407 based on the updated calibration information, thereby obtaining a stereo equirectangular image in which the vertical deviation of the subjects in the left and right images is reduced.

[0065] In the above description, the calibration information is updated by determining the calibration value that minimizes the sum of squares of the closest distances of all ray pairs, but the method of updating the calibration information is not limited to this method. Alternatively, the feature points of the left image may be projected onto the right image via the midpoint of the closest point, and the difference between the corresponding feature point coordinates of the right image and the projected point may be used as an evaluation value to find a calibration value that minimizes the sum of squared errors of all feature point pairs, and the calibration information may be updated based on the calibration value obtained as a result. Furthermore, the evaluation value of the calibration value (calibration information) is not limited to the sum of squares, and may be the average sum of squares or the sum of absolute values. Furthermore, the evaluation value may be a value that increases as the reliability of the calibration information increases.

[0066] As described above, in this embodiment, the calibration information is updated when a certain vibration occurs. Therefore, when the optical axis is shifted due to vibration, even if no user operation is performed, the optical axis shift can be appropriately corrected to reduce the number of frames with degraded image quality.

[0067] Incidentally, a method of determining calibration information corresponding to each frame and applying the determined calibration information to an image to perform image conversion is also conceivable. However, the captured image is not exactly the same for multiple frames, and when the subject moves, slight fluctuations may occur in the calibration information for each frame. Therefore, a video obtained by applying calibration information updated for each frame to each frame will end up being a video with slight vibrations. In other words, it is preferable that the calibration information is updated when necessary and applied during image conversion, as in this embodiment, rather than changing it in a short time.

[0068] In addition, in this embodiment, the calibration information is updated when a predetermined vibration occurs, so that power consumption can be reduced compared to updating the calibration information every frame. In addition, it is possible to update the calibration information periodically, for example, once every five minutes in order to reduce power consumption, but in that case, if vibration occurs that causes optical axis misalignment, it may take time for the calibration information to be updated. On the other hand, according to this embodiment, the calibration information can be updated after optical axis misalignment occurs due to vibration.

[0069] In this embodiment, the stereo images are described as fisheye images (equidistant projection), but the images to be processed are not limited to fisheye images. A normal lens image (central projection image) may be acquired in S402.

[0070] In addition, in the present embodiment, the calibration information for correcting the optical axis inclination and offset deviation of the left and right lens optical systems is determined, but the calibration information is not limited to this. For example, if the magnification or focal length is different between the left and right lens optical systems, parameters for further correcting them may be determined as the calibration information. Also, calibration information regarding the optical axes of both the left and right lenses may be calculated and corrected.

[0071] In addition, in the present embodiment, a method of image conversion in which calibration information is reflected in a captured image in real time while the camera is capturing an image has been described, but calibration information may be determined for a pre-stored image as a processing target, and image conversion may be performed. That is, in S402, the image acquisition unit 305 may acquire an image to be processed from the external storage 107, not from the image sensor 110, and in S403, the vibration information acquisition unit 301 may acquire vibration information corresponding to a time before the capture time of the acquired image. In this case, there is no restriction that the processing of S401 to S407 for one frame must be completed within 33 ms, which is the time for one frame in this embodiment.

[0072] Also, although the present embodiment has been described assuming that the camera 100 is a stereo camera having left and right lens optical systems, the camera 100 may be a single-lens camera. For example, if a marker at a fixed position on the camera body can be captured through the lens optical system, the calibration information can be determined from the deviation of the marker, and therefore the calibration value can be determined and the calibration information updated by combining detection of a predetermined vibration as in the present embodiment. Also, even if a marker does not exist, the calibration information can be determined from the deviation of the vanishing point.

[0073] In this embodiment, the functional units of the image processing device shown in FIG. 3 have been described as being possessed by the camera 100, but the functional units of the image processing device shown in FIG. 3 may be possessed by a device other than the camera 100, such as a PC.

[0074] FIG. 7 is a block diagram for explaining the hardware configuration of a PC 700 having the functional units of FIG. 3. The same components as those of the camera 100 of FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted. The basic hardware configuration of the PC 700 of FIG. 7 is the same as that of the camera 100. However, the PC 700 does not have the stereo lens optical system 111, the image sensor 110, the inertial sensor unit 112, and the like. Also, the PC 700 differs in that the computing power of the CPU 105 is stronger than that of the camera 100, and the capacity of the RAM 102 is larger than that of the camera. When the PC 700 realizes the functions of the functional units of FIG. 3, the stereo images and vibration information output from the camera 100 are stored in the external storage 107 of FIG. 7. Then, for example, the image acquisition unit 305 acquires the stereo images of the processing target frame from the external storage 107 of FIG. 7.

[0075] <Embodiment 2> In the first embodiment, it has been described that the calibration information is updated when a predetermined vibration is detected during imaging. According to the method of the first embodiment, for example, when the camera comes into contact with a tripod or the like during imaging, causing the mount 201 to shift and the optical axis to shift, the calibration information is updated. However, in the method of the first embodiment, when the camera constantly vibrates, the calibration information may be updated for each frame. A video obtained by applying the calibration information updated for each frame to each frame may become a video that appears to have a slight vibration. Therefore, in this embodiment, a method of suppressing unnecessary updates of the calibration information by determining whether to update the calibration information based on the result of analyzing the image in addition to the vibration information will be described. This embodiment will be described mainly with respect to the differences from the first embodiment. The configuration and processing are the same as those of the first embodiment unless otherwise specified.

[0076] Fig. 8 is a flowchart for explaining the process of calculating calibration information and converting stereo images based on the calibration information in this embodiment. The processes of S801 to S804 are similar to S401 to S404 in Fig. 4, and therefore their explanations are omitted. Also, S809 to S810 are similar to S407 to S408, and therefore their explanations are omitted.

[0077] If the vibration detection unit 302 determines that a predetermined vibration has occurred (YES in S804), the process proceeds to S805, and operates the calibration information update unit 304. If the vibration detection unit 302 determines that a predetermined vibration has not occurred (NO in S804), the process proceeds to S809.

[0078] In S805, the calibration information update unit 304 calculates an evaluation value when the pre-update calibration information acquired in S801 is applied to the stereo image of the processing target frame acquired in S802. The method of calculating the evaluation value is the same as the calculation method described in S605 of Fig. 6, and the sum of squares of the closest distances of all ray pairs is used as the evaluation value.

[0079] In S806, the calibration information update unit 304 determines whether the evaluation value calculated in S805 is greater than a predetermined value. If it is determined that the evaluation value is greater than the predetermined value (YES in S806), the process proceeds to S807. The evaluation value is a value correlated with the optical axis shift, and the predetermined value is an evaluation value calculated when the optical axis shift occurs due to the misalignment of the stereo lens optical system 111 caused by the backlash of the mount 201.

[0080] In S801, calibration information is acquired that was set or updated before the occurrence of misalignment of the mount 201. Then, when misalignment of the mount 201 occurs, in S802, stereo images captured after the occurrence of misalignment of the mount 201 are acquired. When misalignment of the mount 201 occurs and the optical axis shifts, the calibration information acquired in S801, which is the calibration information before the occurrence of misalignment of the mount 201, is no longer appropriate calibration information for correction of the captured image captured after the occurrence of misalignment of the mount 201.

[0081] Also, as described above, the more erroneous the calibration information, the larger the calculated evaluation value. Therefore, if a backlash occurs in the mount 201, the evaluation value will be calculated to be high when the calibration information acquired in S801 is applied to the stereo image acquired in S802. Therefore, by comparing the evaluation value calculated in S805 with a predetermined value, it is possible to detect whether vibrations occurred in the camera 100 that can be considered to have caused a backlash in the mount 201. In this embodiment, the predetermined value is 0.5 degrees in terms of the angle of view.

[0082] 4, a calibration value corresponding to the stereo image of the processing target frame acquired in S802 is determined, and the calibration information is updated. If a stereo image is acquired in S802 after the mount 201 has shifted due to a backlash, the calibration information is updated to a calibration value that corrects the optical axis shift caused by the backlash of the mount 201.

[0083] On the other hand, if it is determined that the evaluation value is equal to or less than the predetermined value (NO in S806), the process proceeds to image conversion processing in S809. In this case, S807 to S808 are skipped, and the calibration information is not updated, and the process proceeds to S809. Then, in S809, image conversion is performed based on the calibration information acquired in S801.

[0084] As described above, in this embodiment, after a predetermined vibration is detected, a determination is made based on the stereo image and the calibration information as to whether an optical axis shift has occurred that can be regarded as a backlash of the mount 201. Then, when it is determined that an optical axis shift that can be regarded as a backlash of the mount 201 has occurred, the calibration information is updated. By processing in this manner, it is possible to perform necessary updates of the calibration information while suppressing unnecessary updates of the calibration information.

[0085] In this embodiment, the evaluation value is smaller as the reliability of the calibration information increases, but the present invention is not limited to this, and an evaluation value that increases as the reliability increases may be used. In this case, the direction of the conditional branch based on the comparison of the evaluation values ​​is reversed.

[0086] <Embodiment 3> In this embodiment, a method for suppressing unnecessary updates of calibration information will be described as in the second embodiment. The present embodiment will be described focusing on the differences from the first embodiment. The configuration and processing are the same as those of the first embodiment unless otherwise specified.

[0087] FIG. 9 is a flowchart for explaining the process of calculating the calibration information and converting the stereo images based on the calibration information in this embodiment.

[0088] In S901, the calibration information acquisition unit 310 acquires calibration information managed by the calibration information management unit 303. At the start of this flowchart, the calibration information management unit 303 stores calibration information of an initial value calculated in advance in the storage unit as initial calibration information and manages it. The initial calibration information is, for example, a value calibrated at the time of manufacture. Alternatively, the calibration information stored when the power was last turned off may be stored as the initial calibration information.

[0089] Furthermore, in the same manner as in the above-described embodiment, if step S908 (described later) has not been executed, in step S901 the calibration information acquisition unit 310 acquires initial calibration information as calibration information. If step S405 has been executed in the previous frame, in step S401 the calibration information acquisition unit 310 acquires the calibration information updated in step S405 in the processing of the previous frame.

[0090] The processes in S902 to S904 are similar to those in S402 to S404 in Fig. 4, and therefore their explanation will be omitted. Also, the processes in S911 to S912 are similar to those in S407 to S408, and therefore their explanation will be omitted.

[0091] If the vibration detection unit 302 determines that a predetermined vibration has occurred (YES in S904), the process proceeds to S905, and operates the calibration information update unit 304. If the vibration detection unit 302 determines that a predetermined vibration has not occurred (NO in S904), the process proceeds to S909.

[0092] In S905, the calibration information update unit 304 applies the initial calibration information to the stereo image of the processing target frame acquired in S902 to calculate an evaluation value. The calculated evaluation value is referred to as a first evaluation value.

[0093] In S906, the flow described in FIG. 4 is executed as a subroutine, a calibration value is determined from the stereo images acquired in S902, and an evaluation value corresponding to the determined calibration value is acquired as a second evaluation value.

[0094] In S907, the proofreading information update unit 304 determines whether the first evaluation value is greater than the second evaluation value. If it is determined that the first evaluation value is greater, the process proceeds to S908. If the first evaluation value is equal to or less than the second evaluation value, the process proceeds to S908.

[0095] In S908, the proofreading information update unit 304 replaces the proofreading information managed by the proofreading information management unit 303 with the proofreading value determined in S906, and updates the proofreading information used by the image conversion unit 306.

[0096] Since steps S909 to S910 are similar to steps S407 to S408, the description thereof will be omitted.

[0097] As described above, in this embodiment, as in the second embodiment, the calibration information is updated when a predetermined vibration is detected and a predetermined condition is satisfied. That is, it is determined whether or not there has been a backlash in the mount 201 after the predetermined vibration, and the calibration information is updated if it is deemed that there has been a backlash in the mount 201. Also, by updating the calibration information when the occurrence of vibration is detected, power consumption can be reduced compared to, for example, updating the calibration information every frame.

[0098] In this embodiment, an evaluation value that is determined to be smaller as the reliability of the calibration information is higher has been described, but the evaluation value used in this embodiment is not limited to this. An evaluation value that is determined to be larger as the reliability is higher may be used. In this case, the direction of conditional branching based on comparison of the magnitude of the evaluation value is reversed.

[0099] <Embodiment 4> In the first embodiment, a method for detecting the occurrence of vibration and updating the calibration information during 33 ms, which is the time for one frame, and then performing image conversion was described. In the present embodiment, a method for updating the calibration information asynchronously with the flow of image conversion will be described.

[0100] FIG. 10 is a flowchart for explaining the process of determining calibration information and converting stereo images based on the calibration information in this embodiment.

[0101] In S1001, the vibration detection unit 302 performs an initialization process to reset a vibration detection flag.

[0102] The processes in S1002 to S1004 are similar to those in S402 to S404 in FIG. 4, and therefore will not be described.

[0103] If the vibration detection unit 302 determines that a predetermined vibration has occurred (YES in S1004), the process proceeds to S1005. If the vibration detection unit 302 determines that a predetermined vibration has not occurred (NO in S1004), the process proceeds to S1006.

[0104] In S1005, the vibration detection unit 302 sets a vibration detection flag and stores the time when the occurrence of a predetermined vibration is detected (vibration detection time). Then, the process proceeds to S1006. If the vibration flag has already been set, a process of recording the vibration detection time may be performed. Since the acceleration sensor information, which is the vibration information in this embodiment, includes information in units of 4.167 ms, the time when an impact of 10 G or more is detected is stored in units of 4.167 ms.

[0105] In S1006, the vibration detection unit 302 determines whether or not the vibration detection flag is set and whether or not the vibration information includes a no-vibration state at a time after the vibration detection time.

[0106] The minimum accuracy of shock detection by the acceleration sensor is, for example, about 0.3G, so extremely small vibrations such as air vibrations are indicated as 0G. In this embodiment, a case where all of the XYZ components of the acceleration sensor information, which is vibration information, indicate 0G is considered to be a no-vibration state. After the vibration detection time is, for example, after the latest recorded vibration detection time.

[0107] If the vibration detection unit 302 determines that the vibration flag is set and that there is a no-vibration state after the vibration detection time (YES in S1006), the process proceeds to S1007. If the vibration detection unit 302 determines that the vibration flag is not set or that there is no no-vibration state (NO in S1006), the process proceeds to S1009.

[0108] In S1007, asynchronous execution of a thread shown in FIG. 11 is started, which determines calibration values ​​from stereo images and updates calibration information. When asynchronous execution of the thread is started, the process proceeds to S1008. If a previously started thread is in the middle of processing, the processing is interrupted and execution of the thread is started anew. Although details of the processing of the thread started in S1007 will be described later, processing similar to S405 to S406 in FIG. 4 is performed. That is, calibration values ​​suitable for the stereo images are determined, and the calibration information managed by the calibration information management unit 303 is replaced with the determined calibration values, thereby updating the calibration information managed by the calibration information management unit 303. For this reason, in this embodiment, the processing of S405 to S406 is performed asynchronously with the processing of the flowchart in FIG. 10. The updated calibration information is updated using an exclusive control mechanism.

[0109] In S1008, the vibration detection unit 302 resets the vibration detection flag, and the process proceeds to S1009.

[0110] In S1009, the calibration information acquisition unit 310 acquires the calibration information managed by the calibration information management unit 303. The calibration information updated by the processing of the threads shown in Fig. 11 is updated using an exclusive control mechanism. If the thread started in S1007 has not ended, the calibration information before the update is acquired. The calibration information before the update is the initial value of the calibration information or, if there is a previously executed thread, the calibration information updated by that thread.

[0111] The processes in S1010 and S1011 are similar to those in S407 and S408 in FIG. 4, and therefore will not be described.

[0112] FIG. 11 is a flowchart for explaining the details of the processing of the thread whose start is instructed in S1007.

[0113] In S1101, the calibration information update unit 304 determines a calibration value from the stereo images acquired in S1002. The processing in this step is executed as a subroutine according to the flowchart in FIG.

[0114] In S1102, the calibration information management unit 303 manages the calibration information shared with the main routine, and the calibration information update unit 304 replaces the managed calibration information with the calibration value determined in S1101, and updates the managed calibration information. The update is performed using an exclusive control mechanism.

[0115] As described above, in this embodiment, the process is performed so that the calibration information may be determined in 33 ms or more, which is the time for one frame. For example, the thread in FIG. 11 may be executed in about 3 seconds. In addition, in this embodiment, by not only detecting vibration but also determining that the vibration has converged, it is possible to perform calculations using a relatively weak CPU without being bound by the constraint of completing the process within 33 ms. In addition, the lack of constraints on the processing time leads to reduced power consumption. In this embodiment, it takes time from the detection of the occurrence of strong vibration until the correct calibration information is reflected. However, since the image of the captured image is blurred due to vibration and it takes time to recognize the image after the vibration ends, if appropriate optical axis correction is performed in a sufficiently short time, the user can continue viewing the stereo image without being aware that the optical axis has shifted.

[0116] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0117] The technology of the present disclosure includes the following configurations.

[0118] (Configuration 1) An image processing device for calibrating an imaging device having an optical system and an image sensor, An acquisition means for acquiring an image captured by the imaging device; a detection means for detecting whether or not a vibration has occurred in the imaging device; an update means for updating calibration information for correcting an image captured by the imaging device based on the captured image acquired by the acquisition means when the detection means detects the occurrence of the vibration; 13. An image processing device comprising:

[0119] (Configuration 2) When the detection means detects the occurrence of the vibration, the update means updates the calibration information based on a captured image captured by the imaging device after the occurrence of the vibration, among the captured images acquired by the acquisition means. 2. The image processing device according to claim 1,

[0120] (Configuration 3) The update means includes: When the detection means detects the occurrence of the vibration and a predetermined condition is satisfied, the calibration information is updated. 3. The image processing device according to configuration 1 or 2.

[0121] (Configuration 4) The update means includes: When the detection means detects the occurrence of the vibration and indicates that an evaluation value of the calibration information before updating for a captured image captured by the imaging device after the occurrence of the vibration satisfies the predetermined condition, the calibration information is updated. 4. The image processing device according to configuration 3.

[0122] (Configuration 5) The case where the predetermined condition is satisfied is a case where a deviation of the optical system occurs in the imaging device. 5. The image processing device according to configuration 3 or 4.

[0123] (Configuration 6) The update means updates the calibration information after determining that the vibration has converged when the detection means detects the occurrence of the vibration. 3. The image processing device according to configuration 1 or 2.

[0124] (Configuration 7) The update means includes: An evaluation value is determined when a provisional calibration value is applied to the captured image captured by the imaging device after the vibration occurs, and the provisional calibration value when the determined evaluation value becomes a predetermined value is regarded as the updated calibration information. 7. The image processing device according to any one of configurations 2 to 6,

[0125] (Configuration 8) The evaluation value is a value corresponding to the sum of squares of the error in the least squares method. 8. The image processing device according to configuration 7.

[0126] (Configuration 9) The imaging device further includes a correction unit that corrects the influence of deviation of the optical axis from the optical system in the captured image captured by the imaging device based on the calibration information. 9. The image processing device according to any one of configurations 1 to 8.

[0127] (Configuration 10) The detection means detects whether or not the vibration has occurred for each frame, when the detection means detects the occurrence of the vibration, the correction means performs correction based on the calibration information before the update means updates the calibration information for frames; For frames after the updating means has updated the calibration information, correction is performed based on the updated calibration information. 10. The image processing device according to configuration 9.

[0128] (Configuration 11) The imaging device includes a sensor; The detection means acquires a value of the sensor, and detects that the vibration has occurred when the acquired sensor value exceeds a predetermined value. 11. The image processing device according to any one of configurations 1 to 10,

[0129] (Configuration 12) The sensor is an acceleration sensor. 12. The image processing device according to claim 11,

[0130] (Configuration 13) 13. An imaging device comprising: each of the means of the image processing device according to any one of configurations 1 to 12; the optical system; and the image sensor.

[0131] (Configuration 14) The optical system is composed of a first optical system and a second optical system. 14. The imaging device according to configuration 13,

[0132] (Configuration 15) An image processing method for calibrating an imaging device having an optical system and an image sensor, comprising: an acquisition step of acquiring an image captured by the imaging device; a detection step of detecting whether or not a vibration occurs in the imaging device; an updating step of updating calibration information for correcting an image captured by the imaging device based on the acquired captured image when the occurrence of the vibration is detected; 13. An image processing method comprising:

[0133] (Configuration 16) 13. A program for causing a computer to execute each of the means of the image processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0134] 100 Cameras 105 CPU 302 Vibration detection unit 304 Calibration information update section 305 Image Acquisition Unit

Claims

1. An image processing device for calibrating an imaging device having an optical system and an image sensor, The imaging device acquires an image captured by the imaging device, A detection means for detecting whether a predetermined vibration has occurred in the imaging device, The imaging device provides a correction means for correcting the captured image obtained based on calibration information, When the detection means ceases to detect the predetermined vibration, an update means performs a process to update the calibration information based on the captured image acquired by the acquisition means, without synchronization with the correction by the correction means. An image processing apparatus characterized by having

2. The update means updates the calibration information based on the captured image obtained by the imaging device after it is detected that the predetermined vibration has stopped. The image processing apparatus according to feature 1.

3. The update means is, If the specified conditions are met, the calibration information will be updated. The image processing apparatus according to feature 1.

4. The update means is, If the evaluation value of the calibration information before updating, for the captured image obtained by the imaging device after the predetermined vibration occurs, indicates that the predetermined conditions are met, then update the calibration information. The image processing apparatus according to claim 3.

5. The case in which the above predetermined conditions are met is when a misalignment occurs in the optical system of the imaging device. The image processing apparatus according to feature 4.

6. The update means is, After the predetermined vibration occurs, the imaging device captures an image and applies a provisional calibration value to the resulting image to determine an evaluation value. If the determined evaluation value reaches a predetermined value, the provisional calibration value is used as the updated calibration information. The image processing apparatus according to claim 2.

7. The aforementioned evaluation value corresponds to the sum of the squares of the errors in the least squares method. The image processing apparatus according to claim 6.

8. The correction means corrects the effect of the misalignment of the optical axis from the optical system in the captured image obtained by the imaging device, based on the calibration information. The image processing apparatus according to feature 1.

9. The update means corrects frames up to the time the calibration information is updated based on the calibration information before the update, and corrects frames after the update means has updated the calibration information based on the calibration information after the update. The image processing apparatus according to feature 1.

10. The aforementioned imaging device includes a sensor, The detection means acquires the value of the sensor, and if the acquired value of the sensor exceeds a predetermined value, it detects that the predetermined vibration has occurred. The image processing apparatus according to feature 1.

11. The aforementioned sensor is an acceleration sensor. The image processing apparatus according to feature 10.

12. An imaging device having each means of the image processing apparatus according to any one of claims 1 to 11, the optical system, and the image sensor.

13. The optical system is composed of a first optical system and a second optical system. The imaging apparatus according to feature 12.

14. An image processing method for calibrating an imaging device having an optical system and an image sensor, The acquisition step involves acquiring the captured image obtained by the imaging device, A detection step of detecting whether a predetermined vibration has occurred in the imaging device, A correction step in which the image captured by the imaging device is corrected based on calibration information, If the predetermined vibration is no longer detected in the detection step, an update step is performed in which the calibration information is updated based on the acquired image, without synchronization with the correction in the correction step. An image processing method characterized by having the following features.

15. A program for causing a computer to execute each means of the image processing apparatus described in any one of claims 1 to 11.

Images