Image correction device, image correction method, and image correction program

Abstract

To perform defocus correction on a captured image with high accuracy.SOLUTION: An optical characteristic calculation unit 111 calculates optical characteristic data 123 indicating a calculation result of a point spread function of an optical system on the basis of defocus amount data 121 indicating a defocus amount for a subject and an aberration definition function 122 based on the characteristics of the optical system of an imaging device. A defocus correction unit 112 performs defocus correction on the captured image 131 using the optical characteristic data 123.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a technique for correcting an image with respect to defocus.

Background Art

[0002] A subject photographed by an imaging optical system has a minute spread in that light emitted from a single point does not converge to a single point due to the influence of aberration and diffraction generated inside the imaging optical system, defocus at the time of imaging, and the like. A distribution having such a minute spread is called a PSF. PSF is an abbreviation for Point Spread Function.

[0003] Due to the influence of such an imaging optical system, image degradation based on the PSF shape occurs in the captured image. As a result, the resolution decreases.

[0004] Techniques for correcting such image degradation by image processing have been proposed. The degradation conditions of a captured image due to defocus vary depending on the shooting distance from the subject or the image height. Therefore, there is a method of estimating the PSF for each of those imaging conditions and performing image correction by inverse calculation using the PSF and the captured image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Patent Document 1 discloses a technique for generating second optical characteristic data, which is optical characteristic data at the time of defocus, by adding defocus characteristic data to first optical characteristic data held in advance, and performing image correction processing using this.

[0007] When the first optical characteristic data, defocus characteristic data, and second optical characteristic data form a two-dimensional profile of wavefront aberration, the amount of computation required for the two-dimensional data calculation becomes large. This is because the first optical characteristic data needs to be sufficiently large compared to the optical characteristic data at the time of defocus. Furthermore, optical characteristic data during defocusing needs to be calculated discretely according to the shooting distance and image height. Therefore, there are challenges in the accuracy of defocus correction, especially in imaging scenes where the shooting distance varies.

[0008] There are various imaging scenes in which the shooting distance varies. For example, when imaging a scene with depth, or when imaging from above with the imaging device pointed rather than directly below it, the shooting distance will vary.

[0009] This disclosure aims to enable high-precision defocus correction for captured images. [Means for solving the problem]

[0010] The image correction device disclosed herein corrects captured images obtained by photographing a subject using an imaging device. The aforementioned image correction device is An optical characteristic calculation unit calculates optical characteristic data showing the result of calculating the point spread function of the optical system based on defocus amount data indicating the amount of defocus for the subject and an aberration definition function. The system includes a defocus correction unit that performs defocus correction on the captured image using the aforementioned optical characteristic data. [Effects of the Invention]

[0011] According to this disclosure, it becomes possible to perform defocus correction with high precision on captured images. [Brief explanation of the drawing]

[0012] [Figure 1]Configuration diagram of the image correction device 100 in Embodiment 1. [Figure 2] Functional configuration diagram of the image correction device 100 in Embodiment 1. [Figure 3] Flowchart of the image correction method in Embodiment 1. [Figure 4] Flowchart of step S120 in Embodiment 1. [Figure 5] Diagram showing an example of the functional configuration of the image correction device 100 in Embodiment 1. [Figure 6] Diagram showing the positional relationship between the subject and the optical system in Embodiment 1. [Figure 7] Functional configuration diagram of the image correction device 100 in Embodiment 2. [Figure 8] Flowchart of the image correction method in Embodiment 2. [Figure 9] Functional configuration diagram of the image correction device 100 in Embodiment 3. [Figure 10] Flowchart of the image correction method in Embodiment 3.

Mode for Carrying Out the Invention

[0013] In the embodiments and the drawings, the same elements or corresponding elements are denoted by the same reference numerals. The description of the elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. The arrows in the figures mainly indicate the flow of data or the flow of processing.

[0014] Embodiment 1. The image correction device 100 will be described based on FIGS. 1 to 6.

[0015] ***Description of the Configuration*** Based on FIG. 1, the configuration of the image correction device 100 will be described. The image correction device 100 is a computer including hardware such as a processor 101, a memory 102, an auxiliary storage device 103, and an input / output interface 104. These hardware components are connected to each other via signal lines.

[0016] The processor 101 is an IC that performs arithmetic operations and controls other hardware. For example, the processor 101 is a CPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit.

[0017] Memory 102 is a volatile or non-volatile storage device. Memory 102 is also called main memory. For example, memory 102 is RAM. Data stored in memory 102 is saved to auxiliary storage device 103 as needed. RAM is an abbreviation for Random Access Memory.

[0018] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, HDD, flash memory, or a combination thereof. Data stored in the auxiliary storage device 103 is loaded into memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0019] The input / output interface 104 is a port to which input and output devices are connected. For example, the input / output interface 104 is a USB terminal, the input devices are a keyboard and mouse, and the output device is a display. Input and output of the image correction device 100 are performed using the input / output interface 104. USB is an abbreviation for Universal Serial Bus.

[0020] The image correction device 100 includes elements such as an optical characteristics calculation unit 111 and a defocus correction unit 112. These elements are implemented by software.

[0021] The auxiliary storage device 103 stores an image correction program that allows the computer to function as an optical characteristics calculation unit 111 and a defocus correction unit 112. The image correction program is loaded into memory 102 and executed by the processor 101. The auxiliary storage device 103 also stores the operating system. At least a portion of the OS is loaded into memory 102 and executed by the processor 101. Processor 101 runs the image correction program while simultaneously running the OS. OS is an abbreviation for Operating System.

[0022] The input and output data for the image correction program are stored in the storage unit 120. The auxiliary storage device 103 functions as the storage unit 120. However, storage devices such as memory 102, registers in the processor 101, and cache memory in the processor 101 may function as the storage unit 120 instead of the auxiliary storage device 103, or together with the auxiliary storage device 103.

[0023] The image correction program can be recorded (stored) in a computer-readable format on a non-volatile recording medium such as an optical disc or flash memory.

[0024] Figure 2 shows the functional configuration of the image correction device 100. The captured image 131, corrected image 132, defocus amount data 121, aberration definition function 122, and optical characteristic data 123 will be described later.

[0025] ***Explanation of operation*** The operation procedure of the image correction device 100 corresponds to the image correction method. Furthermore, the operation procedure of the image correction device 100 corresponds to the processing procedure of the image correction program.

[0026] The image correction method will be explained based on Figure 3. In step S110, the user inputs the captured image 131 and the defocus amount data 121 to the image correction device 100. The defocus correction unit 112 receives the input captured image 131. The optical characteristics calculation unit 111 receives the input defocus amount data 121.

[0027] The captured image 131 is an image obtained by photographing a subject using an imaging device. Unless otherwise specified, "imaging device" refers to the imaging device used to obtain the captured image 131. For example, the imaging device is mounted on a flying object, and the captured image 131 is obtained by taking a picture from the flying object. Examples of flying objects include aircraft, drones, or satellites. A drone is an unmanned aerial vehicle (UAV).

[0028] The defocus amount data 121 is data indicating the amount of defocus relative to the subject.

[0029] In step S120, the optical characteristic calculation unit 111 calculates optical characteristic data 123 using the defocus amount data 121 and the aberration definition function 122.

[0030] Optical characteristic data 123 shows the calculation result of the point spread function of the optical system. Unless otherwise specified, "optical system" refers to the optical system of an imaging device.

[0031] The procedure for step S120 will be explained based on Figure 4. In step S121, the optical characteristic calculation unit 111 calculates the wavefront aberration W(ξ,η) using the aberration definition function func with the defocus amount data 121. The aberration definition function `func` defines the defocus characteristics of wavefront aberration. For example, it is a function that models defocus as the radius of curvature of the wavefront. Another example is a function that models defocus as a wavefront with a different radius of curvature depending on the direction, that is, a function that also includes astigmatism in its model.

[0032] The wavefront aberration W(ξ,η) is calculated by performing the calculation in equation (1). "ε ZThis represents the amount of defocus. "λ" is the wavelength of each channel image (e.g., the Red image) included in the captured image. (ξ,η) indicates the position in the pupil coordinate system.

[0033]

number

[0034] In step S122, the optical characteristic calculation unit 111 calculates the pupil function P(ξ,η) using the wavefront aberration W(ξ,η).

[0035] The pupil function P(ξ,η) is given by equation (2). "t" is the aperture mask. The aperture mask spatially indicates the region where the apertures of the optical system exist and is defined by 0 and 1. "exp" means exponential function.

[0036]

number

[0037] In step S123, the optical properties calculation unit 111 calculates the point spread function PSF(x,y) of the optical system using the pupil function P(ξ,η). The point spread function PSF(x,y) of the optical system is given by equation (3). (x,y) represents the position in the image plane coordinate system. "f" represents the focal length of the optical system. "F" stands for Fourier transform.

[0038]

number

[0039] Optical characteristic data 123 shows the calculation results of the point spread function PSF(x,y) of the optical system.

[0040] Returning to Figure 3, we will continue the explanation from step S130. In step S130, the defocus correction unit 112 performs defocus correction on the captured image 131 using the optical characteristic data 123 to generate a corrected image 132. Corrected image 132 is the captured image 131 after defocus correction.

[0041] In defocus correction, the optical characteristic data 123 is used as the filter kernel and an inverse convolution operation is performed. Various methods exist for correcting defocus. For example, Wiener deconvolution is a method that estimates the true image using the signal-to-noise ratio (S / N ratio) of the PSF, noise component, and true signal component. The S / N ratio refers to the signal-to-noise ratio.

[0042] In step S140, the defocus correction unit 112 outputs a corrected image 132. For example, the defocus correction unit 112 displays the corrected image 132 on the display.

[0043] ***Supplement to Embodiment 1*** I would like to add some information about the defocus amount data 121. The amount of defocus on the subject varies depending on the distance at the time of shooting. Therefore, it is desirable to calculate the defocus amount data 121 discretely according to the distance at the time of shooting. If the defocus amount data 121 is calculated discretely, the optical characteristics data 123 is calculated discretely according to the distance at the time of shooting, just like the defocus amount data 121.

[0044] Figure 5 shows an example of the functional configuration of the image correction device 100. The image correction device 100 may further include an element called a defocus amount calculation unit 119. The image correction program further utilizes a computer as a defocus amount calculation unit 119.

[0045] The defocus amount calculation unit 119 calculates the defocus amount data 121. The defocus amount data 121 can be calculated based on the imaging condition data or the captured image 131.

[0046] For example, the defocus amount data 121 is calculated as follows. The user inputs the imaging condition data into the image correction device 100. Image condition data is data that indicates the conditions (imaging conditions) under which the subject was imaged. The imaging conditions consist of the focal length of the optical system, the distance between the subject (ground surface) and the optical system's sensor, and information about the focal plane, or a combination thereof. For example, image condition data indicates the orientation of the optical system and the positional relationship between the subject and the optical system. The defocus amount calculation unit 119 receives the input imaging condition data and calculates the defocus amount data 121 based on the received imaging condition data.

[0047] Based on Figure 6, the method for calculating the defocus amount data 121 will be explained. Solid lines indicate elements when the light source from the subject is not in focus on the sensor surface (defocus). Dashed lines indicate elements when the light source from the subject is in focus on the sensor surface (focus). Note that areas where the light source in focus is projected onto the sensor surface and areas where it is not may exist simultaneously. Therefore, both focused and defocused states may exist at the same time. The subject plane is the plane on which the subject lies, and its shape is known. For example, the subject plane is the Earth's surface. The subject distance is the distance between the subject and the sensor of the optical system. Defocusing conditions refer to situations where the position of the light source image from the subject differs from the sensor surface. First, the defocus amount calculation unit 119 calculates the distance between the subject and the sensor of the optical system, and the focal plane of the subject, based on the orientation of the optical system and the positional relationship between the subject and the optical system. The defocus amount calculation unit 119 then calculates the amount of defocus relative to the subject based on the focal length of the optical system, the distance between the subject and the sensor of the optical system, and the focal plane of the subject. The data showing the calculated amount of defocus is the defocus amount data 121.

[0048] For example, the defocus amount calculation unit 119 analyzes the captured image 131 to calculate the defocus amount data 121. The defocus amount data 121 is calculated as follows. First, the defocus amount calculation unit 119 detects the edges of structures visible in the captured image 131. The structures may be subjects or objects other than subjects. It is desirable that subjects with a known shape be detected as structures. Next, the defocus amount calculation unit 119 obtains the LSF from the pixel value profile (image feature) around the detected edge. The pixel value profile shows the distribution of pixel values ​​in the direction of the edge normal. The LSF is the line spread function. It is an abbreviation for Function. The defocus amount calculation unit 119 then calculates the defocus amount for the subject using LSF. The defocus amount is calculated based on the steepness of the pixel values. The steepness is calculated, for example, using the Fourier transform. The data showing the calculated defocus amount is the defocus amount data 121.

[0049] ***Effects of Embodiment 1*** Embodiment 1 aims to perform defocus correction with high precision. In particular, Embodiment 1 aims to perform defocus correction with high precision in imaging scenes where there is variation in shooting distance. Embodiment 1 enables highly accurate defocus correction. The defocus correction according to Embodiment 1 is more accurate than the method of calculating PSF using (pre-held) optical characteristic data in imaging scenes with variations in shooting distance.

[0050] ***Summary of Embodiment 1*** The features of Embodiment 1 are described below. The symbols of the elements corresponding to the elements enclosed in parentheses are shown within the parentheses. Embodiment 1 relates to information processing that can perform highly accurate correction for fluctuations in defocus (focus shift) during image capture. The information processing device (100) includes an aberration definition function holding unit (120), a PSF calculation unit (111), and a PSF correction unit (112). The aberration definition function holder (120) holds the aberration definition function (122). The PSF calculation unit (111) generates second optical characteristic data (123) based on the aberration definition function (122) and the defocus amount (121). The PSF correction unit (112) performs recovery processing on the captured image (131) based on the second optical characteristic data (123).

[0051] Embodiment 2. The method for calculating the aberration definition function 122 will be explained, mainly based on the differences from Embodiment 1, using Figures 7 and 8.

[0052] ***Explanation of the structure*** The configuration of the image correction device 100 will be explained based on Figure 7. The image correction device 100 further includes elements such as a sensor data calculation unit 113 and an aberration definition function calculation unit 114. The image correction program further utilizes a computer as a sensor data calculation unit 113 and an aberration definition function calculation unit 114.

[0053] ***Explanation of operation*** The image correction method will be explained based on Figure 8. In step S210, the user inputs the captured image 131, the defocus amount data 121, and the reference image 133 to the image correction device 100. The sensor data calculation unit 113 receives the input reference image 133.

[0054] Reference image 133 is an image obtained by taking a photograph using an imaging device. For example, reference image 133 is an image of a light source, which is obtained by photographing a point light source using an imaging device. The imaging equipment used is the same model as the imaging equipment used to obtain the captured image 131. For example, the imaging equipment used to obtain the captured image 131 is used to obtain the reference image 133.

[0055] In step S220, the sensor data calculation unit 113 calculates sensor data 124 based on the reference image 133. Sensor data 124 shows the point spread function and wavefront aberration of the optical system.

[0056] First, the sensor data calculation unit 113 calculates the point spread function PSF of the optical system based on the point light source image (reference image 133) provided for calibration. The sensor data calculation unit 113 then calculates the wavefront aberration W(ξ,η) of the optical system based on the relationship between equations (2), (3), and (4) of Embodiment 1, using the point spread function PSF of the optical system.

[0057] In step S230, the aberration definition function calculation unit 114 calculates the aberration definition function 122 based on the sensor data 124.

[0058] Steps S240 to S260 are the same as steps S120 to S140 in Embodiment 1.

[0059] ***Effects of Embodiment 2*** Embodiment 2 makes it possible to maintain the accuracy of defocus correction using measured sensor data that corresponds to the aging deterioration of the optical system and alignment misalignment. Alignment misalignment occurs, for example, when a large impact is applied to the optical system.

[0060] ***Supplement to Embodiment 2*** The image correction device 100 may include a defocus amount calculation unit 119 to calculate defocus amount data 121.

[0061] Embodiment 3. The configuration for obtaining the optimal amount of defocus will be explained, primarily based on the differences from Embodiment 1, using Figures 9 and 10.

[0062] ***Explanation of the structure*** The configuration of the image correction device 100 will be explained based on Figure 9. The image correction device 100 further includes elements such as a cost calculation unit 115 and a defocus amount update unit 116. The image correction program further utilizes a computer as a cost calculation unit 115 and a defocus amount update unit 116.

[0063] ***Explanation of operation*** The image correction method will be explained based on Figure 10. In step S310, the captured image 131 and the defocus amount data 121 are input to the image correction device 100. Step S310 is the same as step S110 in Embodiment 1.

[0064] In step S320, the optical characteristic calculation unit 111 calculates optical characteristic data 123 using the defocus amount data 121 and the aberration definition function 122. Step S320 is the same as step S120 in Embodiment 1.

[0065] In step S330, the defocus correction unit 112 performs defocus correction on the captured image 131 using the optical characteristic data 123 to generate a provisional image 134. The provisional image 134 is the image 131 after defocus correction. The method for performing defocus correction is the same as the method in step S130 of Embodiment 1.

[0066] In step S340, the cost calculation unit 115 calculates cost data 125 using the provisional image 134, the captured image 131, and the optical characteristic data 123. Cost data 125 represents the cost of the degree of mismatch between the acquired image 131 and the optical characteristics data 123.

[0067] For example, cost data 125 is calculated as follows: First, the cost calculation unit 115 performs a blurring operation on the provisional image 134 using the optical characteristic data 123. The blurring operation is an operation that convolves the provisional image 134 with the optical characteristic data 123. In other words, the cost calculation unit 115 defocuses the provisional image 134 using the optical characteristic data 123. The cost calculation unit 115 then compares the provisional image 134 after the blur calculation with the captured image 131 and calculates the difference between the captured image 131 and the provisional image 134 after the blur calculation. For example, the difference is the simple difference in pixel values. The calculated difference is represented as cost data 125.

[0068] Cost data 125 may be calculated as follows: The cost calculation unit 115 divides both the captured image 131 and the provisional image 134 after blur calculation into multiple image regions and calculates the cost for each image region based on the amount of defocus. The calculated cost (cost volume) is shown in the cost data 125. By obtaining cost volume, it is possible to estimate the amount of defocus that minimizes cost (the optimal amount of defocus).

[0069] In step S350, the defocus amount update unit 116 determines whether to update the defocus amount data 121. If the optimal amount of defocus is not obtained, the defocus amount data 121 is updated.

[0070] For example, the defocus amount update unit 116 determines whether the cost shown in the cost data 125 satisfies the cost conditions. If the cost conditions are not met, the defocus amount data 121 is updated.

[0071] For example, the defocus amount update unit 116 compares the cost shown in the cost data 125 with a threshold. If the cost is greater than the threshold, the defocus amount data 121 is updated.

[0072] If the defocus amount data 121 is to be updated, the process proceeds to step S360.

[0073] In step S360, the defocus amount update unit 116 updates the defocus amount data 121 based on the cost data 125. The defocus amount data 121 is updated using an optimization method to reduce costs. The updated defocus amount data 121 is referred to as defocus amount data 126.

[0074] After step S360, the process proceeds to step S320. In step S320, the optical characteristic calculation unit 111 calculates new optical characteristic data 123 using the defocus amount data 126 and the aberration definition function 122. In step S330, the defocus correction unit 112 performs defocus correction on the captured image 131 using the new optical characteristic data 123 to generate a new provisional image 134. In step S340, the cost calculation unit 115 calculates new cost data 125 using the new provisional image 134, the captured image 131, and the optical characteristic data 123. In step S350, the defocus amount update unit 116 determines whether to update the defocus amount data 121. If the defocus amount data 121 is not updated, the process proceeds to step S370.

[0075] In step S370, the defocus correction unit 112 outputs the latest provisional image 134 as the corrected image 132.

[0076] ***Effects of Embodiment 3*** Embodiment 3 calculates the optimal defocus amount for defocus correction by repeatedly performing the processing of the cost calculation unit 115 and the processing of the defocus amount update unit 116. Embodiment 3 makes it possible to estimate the optimal defocus amount with finer granularity along the defocus amount axis, thus enabling more accurate defocus correction.

[0077] ***Supplement to Embodiment 3*** The image correction device 100 may include a defocus amount calculation unit 119 to calculate defocus amount data 121.

[0078] Embodiment 3 may be implemented in combination with Embodiment 2. That is, the image correction device 100 may include a sensor data calculation unit 113 and an aberration definition function calculation unit 114 to calculate the aberration definition function 122.

[0079] ***Supplementary Information on the Embodiment*** Each embodiment is an example of a preferred form and is not intended to limit the technical scope of this disclosure. Each embodiment may be implemented in part or in combination with other embodiments. Procedures described using flowcharts, etc., may be modified as appropriate.

[0080] Each element of the image correction device 100 may be implemented using software, hardware, firmware, or a combination thereof. The word "part" in each element of the image correction device 100 may be read as "processing," "process," "circuit," or "circuit."

[0081] The various aspects of this disclosure are described below as appendices. (Note 1) This is an image correction device that corrects captured images obtained by photographing a subject using an imaging device. An optical characteristic calculation unit calculates optical characteristic data showing the result of calculating the point spread function of the optical system of the imaging device, based on defocus amount data indicating the amount of defocus for the subject and an aberration definition function. A defocus correction unit that performs defocus correction on the captured image using the aforementioned optical characteristic data, An image correction device equipped with the following features.

[0082] (Note 2) A sensor data calculation unit calculates sensor data indicating the point spread function and wavefront aberration of the optical system based on a reference image obtained by imaging using the aforementioned imaging device, An aberration definition function calculation unit calculates the aberration definition function based on the sensor data, Equipped with The image correction device described in Appendix 1.

[0083] (Note 3) The image correction device comprises a cost calculation unit and a defocus amount update unit. The defocus correction unit uses the optical characteristic data to perform the defocus correction on the captured image and generates a provisional image. The cost calculation unit uses the provisional image, the captured image, and the optical characteristic data to calculate cost data that represents the degree of mismatch between the captured image and the optical characteristic data, The defocus amount update unit updates the defocus amount data based on the cost data, The optical characteristic calculation unit calculates new optical characteristic data based on the updated defocus amount data and the aberration definition function. The defocus correction unit performs the defocus correction on the captured image using the new optical characteristic data. The image correction device described in Appendix 1 or Appendix 2.

[0084] (Note 4) The aforementioned aberration definition function is a function that defines the defocus characteristics of wavefront aberration. An image correction device as described in any one of the appendices 1 to 3.

[0085] (Note 5) Defocus amount calculation unit that analyzes the captured image and calculates the defocus amount Equipped with An image correction device as described in any one of the appendices 1 through 4.

[0086] (Note 6) Defocus Amount Calculation Unit that calculates the defocus amount based on imaging condition data indicating the imaging conditions of the subject. Equipped with An image correction device as described in any one of the appendices 1 through 4.

[0087] (Note 7) The aforementioned imaging device is mounted on the flying object, The aforementioned captured image is an image obtained by taking a picture from the flying object. An image correction device as described in any one of the appendices 1 through 6.

[0088] (Note 8) This is an image correction method that corrects captured images obtained by photographing a subject using an imaging device. Based on the defocus amount data indicating the amount of defocus for the subject and the aberration definition function, optical characteristic data showing the calculation result of the point spread function of the optical system of the imaging device is calculated. Defocus correction is performed on the captured image using the aforementioned optical characteristic data. Image correction methods.

[0089] (Note 9) This is an image correction program for correcting captured images obtained by photographing a subject using an imaging device. An optical characteristic calculation process that calculates optical characteristic data showing the result of calculating the point spread function of the optical system of the imaging device, based on defocus amount data indicating the amount of defocus for the subject and an aberration definition function, A defocus correction process that performs defocus correction on the captured image using the aforementioned optical characteristic data, An image correction program that allows a computer to perform the following actions. [Explanation of symbols]

[0090] 100 Image correction device, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Input / output interface, 111 Optical characteristic calculation unit, 112 Defocus correction unit, 113 Sensor data calculation unit, 114 Aberration definition function calculation unit, 115 Cost calculation unit, 116 Defocus amount update unit, 119 Defocus amount calculation unit, 120 Storage unit, 121 Defocus amount data, 122 Aberration definition function, 123 Optical characteristic data, 124 Sensor data, 125 Cost data, 126 Defocus amount data, 131 Acquired image, 132 Corrected image, 133 Reference image, 134 Provisional image.

Claims

1. This is an image correction device that corrects captured images obtained by photographing a subject using an imaging device. An optical characteristic calculation unit calculates optical characteristic data showing the result of calculating the point spread function of the optical system of the imaging device, based on defocus amount data indicating the amount of defocus for the subject and an aberration definition function. A defocus correction unit generates a provisional image by performing defocus correction on the captured image using the aforementioned optical characteristic data through inverse convolution calculation. A cost calculation unit performs a blur operation on the provisional image by convolving the optical characteristic data to generate a re-blurred image, and calculates cost data that shows the difference in pixel values ​​between the captured image and the re-blurred image as the cost. A defocus amount update unit updates the defocus amount data so that the cost shown in the cost data becomes smaller if the cost data does not meet predetermined conditions, and repeatedly recalculates the optical characteristic data, regenerates the provisional image, and recalculates the cost data based on the updated defocus amount data, and outputs the provisional image as a corrected image when the cost data meets the predetermined conditions. An image correction device equipped with the following features.

2. A sensor data calculation unit calculates sensor data indicating the point spread function and wavefront aberration of the optical system based on a reference image obtained by imaging using the aforementioned imaging device, An aberration definition function calculation unit calculates the aberration definition function based on the sensor data, Equipped with The image correction device according to claim 1.

3. The aforementioned aberration definition function is a function that defines the defocus characteristics of wavefront aberration. The image correction device according to claim 1.

4. Defocus amount calculation unit that analyzes the captured image and calculates the defocus amount Equipped with The image correction device according to claim 1.

5. Defocus Amount Calculation Unit that calculates the defocus amount based on imaging condition data indicating the imaging conditions of the subject. Equipped with The image correction device according to claim 1.

6. The aforementioned imaging device is mounted on the flying object, The aforementioned captured image is an image obtained by taking a picture from the flying object. The image correction device according to any one of claims 1 to 5.

7. This is an image correction method that corrects captured images obtained by photographing a subject using an imaging device. Based on the defocus amount data indicating the amount of defocus for the subject and the aberration definition function, optical characteristic data showing the calculation result of the point spread function of the optical system of the imaging device is calculated. Using the aforementioned optical characteristic data, a defocus correction is performed on the captured image by inverse convolution to generate a provisional image. A blurring operation is performed on the provisional image by convolving the optical characteristic data to generate a re-blurred image, and cost data is calculated showing the difference in pixel values ​​between the captured image and the re-blurred image as the cost. If the cost data does not meet the predetermined conditions, the defocus amount data is updated so that the cost shown in the cost data becomes smaller, and the recalculation of the optical characteristics data, the regeneration of the provisional image, and the recalculation of the cost data are repeated based on the updated defocus amount data. If the cost data satisfies the predetermined conditions, the provisional image obtained when the cost data satisfies the predetermined conditions is output as a corrected image. Image correction methods.

8. This is an image correction program for correcting captured images obtained by photographing a subject using an imaging device. An optical characteristic calculation process that calculates optical characteristic data showing the result of calculating the point spread function of the optical system of the imaging device, based on defocus amount data indicating the amount of defocus for the subject and an aberration definition function, A defocus correction process is performed on the captured image using the aforementioned optical characteristic data by defocus correction using an inverse convolution operation to generate a provisional image. A cost calculation process is performed to generate a re-blurred image by convolving the optical characteristic data onto the provisional image, and to calculate cost data that shows the difference in pixel values ​​between the captured image and the re-blurred image as the cost. A defocus amount update process that updates the defocus amount data so that the cost shown in the cost data becomes smaller if the cost data does not meet predetermined conditions, and repeatedly recalculates the optical characteristic data, regenerates the provisional image, and recalculates the cost data based on the updated defocus amount data, and outputs the provisional image as a corrected image when the cost data meets the predetermined conditions, if the cost data does not meet predetermined conditions. An image correction program that allows a computer to perform the following actions.

Images