Image processing apparatus and image processing method

Abstract

Image processing apparatus and image processing method capable of appropriately handling high-resolution image generated by combining a plurality of frames of images SOLUTION: The image processing apparatus generates a reduced image obtained by reducing a second image having a resolution higher than that of a first image generated by combining a plurality of frames of first images imaged by an imaging apparatus. Before reducing the second image, the image processing apparatus applies a low-pass filter having a spatial frequency characteristic determined based on the spatial frequency characteristic of the captured image of the imaging apparatus and the Nyquist frequency of the reduced image to the second image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to an image scaling technique. [Background technology]

[0002] Imaging devices using image sensors are constantly being required to produce images with higher resolution. This can be achieved by increasing the number of pixels in the image sensor. However, increasing the number of pixels in the image sensor requires either increasing the size of the image sensor or reducing the size of the pixels, which is not easy.

[0003] On the other hand, Patent Document 1 proposes a technique for acquiring an image with a higher resolution than the resolution of an image sensor by synthesizing a plurality of frames of images captured by slightly shifting the position of the image sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-111776 Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Patent Document 1 is equivalent to artificially reducing the pixel pitch of the image sensor. As a result, the Nyquist frequency of the composite image is higher than the Nyquist frequency based on the pixel pitch of the image sensor. In general signal processing, if the sampling frequency is increased by n times, the Nyquist frequency also increases by n times, and if the sampling frequency is reduced to 1 / n, the Nyquist frequency also decreases by 1 / n times. However, when the Nyquist frequency of an image captured by an image capture device is treated in the same way as in general signal processing, expected results may not be obtained.

[0006] In order to improve upon the problems of the conventional technology, in one aspect, the present invention provides an image processing device and an image processing method that are capable of appropriately handling high-resolution images generated by combining multiple frame images. [Means for solving the problem]

[0007] In one aspect, the present invention provides an image processing device comprising: an acquisition means for acquiring a second image having a higher resolution than a first image, the second image being generated by synthesizing a plurality of frames of first images captured by an imaging device; and a generation means for generating a reduced image by reducing the second image, wherein the generation means calculates a Nyquist frequency of the reduced image, determines the spatial frequency characteristics of a low-pass filter to be applied to the second image based on the spatial frequency characteristics of the image captured by the imaging device and the Nyquist frequency of the reduced image, and reduces the second image to which the low-pass filter having the determined spatial frequency characteristics has been applied, to generate the reduced image. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device and an image processing method that are capable of appropriately handling a high-resolution image generated by combining images of a plurality of frames. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera as an image processing apparatus according to an embodiment; [Figure 2] FIG. 1 is a diagram for explaining a method for capturing a high-resolution image in an embodiment. [Figure 3] FIG. 3 is a diagram illustrating a method for combining images obtained by the photographing method shown in FIG. 2. [Figure 4] A diagram showing an example of an MTF curve for an imaging optical system [Figure 5] Flowchart for reduction processing in the first embodiment [Figure 6] FIG. 10 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following, the present invention will be described in terms of an embodiment using a digital camera as an example of an image processing device. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smart watches, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] ●(First embodiment) FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera 1 as an example of an image processing device according to the present invention.

[0013] The system control unit 5 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing programs. The system control unit 5 loads programs stored in the ROM 6 into the RAM 7 and executes them to control the operation of each unit of the digital camera 1 and realize the functions of the digital camera 1.

[0014] The ROM 6 is a rewritable non-volatile memory that stores programs executed by the system control unit 5, various setting values ​​for the digital camera 1, GUI data, and the like. The RAM 7 is used as a main memory for the system control unit 5, a buffer for temporarily storing captured image data, a work memory for temporarily storing data being processed by the image processing unit 204, etc. Furthermore, a part of the RAM 7 may be used as a video memory for storing image data to be displayed on the display unit 206.

[0015] The imaging optical system 3 forms an optical image of a subject on the imaging surface of the image sensor 2. The imaging optical system 3 is assumed to be fixed to the digital camera 1, but may be detachable. The imaging optical system 3 includes movable lenses such as a focus lens, a zoom lens, and an anti-vibration lens, as well as mechanisms for moving the lenses. The lens control unit 207 controls the mechanisms for moving the movable lenses in accordance with instructions from the system control unit 5.

[0016] The exposure mechanism 209 has aperture blades that control the diameter of the optical path and a drive mechanism for them. The aperture blades can fully close the optical path and also function as a mechanical shutter. The mechanical drive unit 208 controls the operation of the exposure mechanism 209 according to instructions from the system control unit 5, and achieves exposure of the image sensor 2 according to the exposure conditions (aperture and shutter speed) determined by the system control unit 5.

[0017] The image sensor 2 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor 2 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out signals from each pixel having voltages according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface can be obtained. The analog image signals read out from the image sensor 2 are supplied to an A / D conversion unit 203.

[0018] The moving stage 4 is a moving mechanism capable of moving the image sensor 2 in two-dimensional directions parallel to the imaging surface. The moving stage 4 has, for example, a motor or an actuator, and moves the image sensor 2 in a direction and by a distance specified by the system control unit 5. The minimum distance of movement of the moving stage 4 is smaller than the pixel pitch of the image sensor 2. Note that the moving stage 4 may also be capable of moving the image sensor 2 in elevation and depression directions when the optical axis direction of the imaging optical system 3 is set to the horizontal direction. Also, although the image sensor 2 is moved here, a configuration in which the digital camera 1 is moved may also be used.

[0019] The A / D conversion unit 203 performs correlated double sampling, gain adjustment, A / D conversion, etc. on the analog image signal output from the image sensor 2, and outputs it as a digital image signal (image data). The amount of gain adjustment increases as the ISO sensitivity increases. The amount of gain adjustment is set by the system control unit 5. The A / D conversion unit 203 supplies the image data to the image processing unit 204.

[0020] The image processing unit 204 applies predetermined image processing to the image data output by the A / D conversion unit 203 to generate signals and image data according to the application, and acquires and / or generates various information. The image processing unit 204 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 204 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing unit 204 outputs the acquired or generated information and data to the system control unit 5, the reduction processing unit 205, or the like according to the application.

[0021] The image processing applied by the image processing unit 204 can include, for example, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Color interpolation, also known as demosaicing, is performed when the image sensor is equipped with a color filter, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. The correction processing may include white balance adjustment, edge enhancement, noise reduction, gradation (gamma) correction, correction of the effect of optical aberrations of the imaging optical system 3 on the image, correction of the effect of peripheral light falloff of the imaging optical system 3, and color correction. The detection process may include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing can include processes such as cropping, compositing, signal format and color space conversion, encoding, and decoding. The generation of image data for recording is also included in the data processing. The data processing also includes the process of generating a high-resolution image by compositing multiple frames of images captured while moving the image sensor 2. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processes that can be applied by the image processing unit 204, and do not limit the processes that can be applied by the image processing unit 204.

[0022] In this embodiment, the image data for recording (including high-resolution image data) generated by the image processing unit 204 is supplied to the reduction processing unit 205. However, reduction processing is not essential.

[0023] The reduction processing unit 205 generates reduced image data by reducing the resolution of the image data for recording output by the image processing unit 204. Like the image processing unit 204, the reduction processing unit 205 may be a dedicated hardware circuit such as an ASIC, or may be configured to achieve a specific function by a processor such as a DSP or GPU executing software. In practice, the image processing unit 204 and the reduction processing unit 205 may be integrated.

[0024] The reduction processing unit 205 can generate multiple types of reduced image data depending on the application. For example, the reduction processing unit 205 can generate reduced image data for display on the display unit 206 and reduced image data for recording on the medium control unit 211. The reduction processing unit 205 stores the reduced image data for display on the display unit 206 in a video memory area of ​​the RAM 7. The reduction processing unit 205 also supplies the reduced image data for recording to the format conversion unit 210. Details of the operation of the reduction processing unit 205 will be described later.

[0025] The display unit 206 includes a display device such as an LCD, and performs display based on reduced image data for display stored in the video memory area of ​​the RAM 7. The display device included in the display unit 206 can function as an electronic viewfinder.

[0026] The format conversion unit 210 generates a data file that stores the image data supplied from the reduction processing unit 205. Specifically, the format conversion unit 210 generates metadata that conforms to specifications according to the format of the data file to be recorded, and encodes the image data according to the specifications. There are no particular restrictions on the file formats that the format conversion unit 210 can generate, and it may also generate a data file in a manufacturer-specific format.

[0027] The medium control unit 211 records the data file generated by the format conversion unit 210 in a predetermined recording destination, for example, a recording medium 213. The recording medium 213 may be, for example, a memory card attached to the digital camera 1 or a communicable external device. The medium control unit 211 also acquires the recorded data file from the recording medium 213 and supplies it to the image processing unit 204.

[0028] The digital camera 1 can take and record still images and moving images. Although not specifically described, when taking and recording moving images, audio is also processed in addition to images.

[0029] The operation unit 201 is a collective term for input devices (such as buttons, switches, and dials) that are provided for the user to input various instructions to the digital camera 1. The input devices that make up the operation unit 201 are named according to their assigned functions. For example, the operation unit 201 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a confirmation key. The release switch is a switch for recording still images, and the system control unit 5 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. The system control unit 5 also recognizes a press of the video recording switch in shooting standby mode as an instruction to start video recording, and a press of the video recording switch during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable. The input device may also be software buttons or keys using a touch display. The operation unit 201 may also include an input device that supports non-contact input methods such as voice input and eye-gaze input.

[0030] When the system control unit 5 detects an operation on the operation unit 201, it executes an operation corresponding to the detected operation. For example, when the system control unit 5 detects a half-press of the release switch in a shooting standby state, it executes AF and AE operations as a preparation operation for shooting a still image. In a shooting standby state, the image processing unit 204 continuously generates signals or evaluation values ​​used for AF and AE for the video (live view image) being shot so that the display unit 206 functions as an EVF. The system control unit 5 uses these signals or evaluation values ​​to execute AF processing that controls the position of the focus lens and AE processing that determines the exposure conditions (aperture value, shutter speed, ISO sensitivity). AF processing and AE processing can be executed by any known method, so further detailed explanation will be omitted.

[0031] Furthermore, when the system control unit 5 detects that the release switch has been fully pressed, it controls the operation of the related functional blocks so as to carry out a series of operations from taking a still image to recording a still image data file.

[0032] In addition, when a shooting mode is set in which a high-resolution image is generated by combining multiple frame images captured by moving the image sensor 2, the system control unit 5 controls the operation of the moving stage 4 to move the image sensor 2 to a predetermined position for each shot.

[0033] The system control unit 5 also notifies the image processing unit 204 and the reduction processing unit 205 of information such as the aperture value of the exposure mechanism 209 and the optical characteristics (aberration and vignetting) of the imaging optical system 3. The image processing unit 204 performs correction processing based on this information. The reduction processing unit 205 also determines the frequency characteristics of a filter to be used in the reduction processing based on this information.

[0034] (High resolution image capture operation) Next, a description will be given of the operation of the digital camera 1 when generating a high-resolution image. In this embodiment, as an example, a high-resolution image is generated when a high-resolution shooting mode is set through operation of the operation unit 201. However, the system control unit 5 may automatically set the high-resolution shooting mode in accordance with, for example, the image analysis results of the image processing unit 204.

[0035] In high-resolution shooting mode, digital camera 1 captures multiple frame images (first images) by moving image sensor 2 to different positions using moving stage 4. These multiple frame images have a shift less than the pixel pitch of image sensor 2. Digital camera 1 then synthesizes the captured multiple frame images to generate a high-resolution image (second image) having a pixel pitch smaller than the physical pixel pitch of image sensor 2.

[0036] 2 is a diagram showing an example of the position of the moving stage 4 (image sensor 2) when capturing each frame when the digital camera 1 synthesizes four images (frames) to generate a high-resolution image. Here, the pixel pitch in both the horizontal and vertical directions of the image sensor 2 is denoted by P. The position of the moving stage 4 when it is not moving is defined as the reference position.

[0037] In Figure 2, digital camera 1 (a) Reference position (b) Position moved horizontally by P / 2 from the reference position (c) Position moved vertically by P / 2 from the reference position (d) Position moved by P / 2 horizontally and P / 2 vertically from the reference position By combining the four frames captured at these positions, digital camera 1 can generate a high-resolution image with double the resolution in both the horizontal and vertical directions.

[0038] 2 are merely examples, and other values ​​may be used. However, the movement distance must be less than P or a value other than an integral multiple of P (for example, 3P / 2). The number of composite frames N (≧2) is determined by the Nyquist frequency of one frame of image before composition, f single , the Nyquist frequency of the synthesized high-resolution image is f sr Then, it may be determined by the following formula: N= (f sr / f single ) 2

[0039] However, the number of composite frames N may be greater or less than the value obtained by the above formula. Also, the number of composite frames may be increased or decreased depending on the resolution per frame.

[0040] It is preferable to suppress the effects of blurring when shooting in high-resolution shooting mode, so an electronic shutter may be used instead of a mechanical shutter (exposure mechanism 209).

[0041] The image data for N frames to be synthesized is processed by the A / D conversion unit 203 and then sequentially stored in, for example, the RAM 7. Since the image sensor 2 has color filters in a primary color Bayer array, the image data stored in the RAM 7 is RAW data in which each pixel has a single color component according to the primary color Bayer array.

[0042] The image processing unit 204 converts the image data for N frames into image data in which each pixel has RGB components (or YCbCr components) and then synthesizes the converted image data. A synthesis method for generating a high-resolution image will be described with reference to FIG.

[0043] FIG. 3 shows how the pixels of the four frame images captured at the positions shown in FIGS. 2(a) to (d) are arranged in the synthesized high-resolution image. P is the (actual) pixel pitch of the image sensor 2. As shown in FIG. 3, pixels of the second and subsequent frame images are arranged between the pixels (●) of the first frame image captured at the reference position, at positions corresponding to the offset between the reference position and the capturing position. For example, since the second frame was captured at a position offset by P / 2 horizontally to the right from the reference position, the pixel (■) of the second frame is arranged at the midpoint between the horizontally adjacent pixels in the first frame image. The same applies to the pixels (□) of the third frame and the pixels (×) of the fourth frame.

[0044] Here, the amount of movement is set to P / 2, but by synthesizing multiple frames taken by moving the camera horizontally, vertically, and diagonally in units of P / m (m≧3), a high-resolution image with horizontal and vertical resolution m times higher than before synthesis can be obtained. Theoretically, the Nyquist frequency of one frame of captured image before synthesis is f single , the Nyquist frequency of the synthesized high-resolution image is f sr Then, the following relationship holds: f sr =m×f single

[0045] FIG. 4 shows an example of an MTF curve for the imaging optical system 3. The vertical axis represents MTF (%), and the horizontal axis represents spatial frequency (lp / mm). FIG. 4 also shows the above-mentioned Nyquist frequency f single and f sr In addition, the spatial frequency f0 at which the MTF is 0% is shown. In the example shown in Figure 4, single , f0, f sr have the following relationship: f single < f0< f sr

[0046] In the example shown in Figure 4, sr The MTF in the spatial frequency region up to is 0. The subject image included in this spatial frequency region cannot be resolved by the imaging optical system 3 because MTF = 0. Therefore, the high-resolution image obtained by synthesis contains spatial frequency components that cannot be resolved by the imaging optical system 3.

[0047] The image processing unit 204 applies image processing such as white balance correction, optical correction, edge emphasis, noise removal, pixel interpolation, gamma correction, and color difference signal generation to the high-resolution image in the same manner as to a normal (non-composite) image. The high-resolution image data output by the image processing unit 204 is assumed to be in YUV format.

[0048] The high-resolution image data output from the image processing unit 204 is reduced by a reduction processing unit 205 and input to a format conversion unit 210. The format conversion unit 210 generates a recording data file that stores the reduced high-resolution image data, and outputs the file to a medium control unit 211. The medium control unit 211 records the recording data file on a recording medium 213.

[0049] (Reduction processing unit 205) The operation of the reduction processing unit 205 will be described with reference to the flowchart shown in FIG. In general signal processing, if the sampling frequency is increased by a factor of n, the Nyquist frequency also increases by a factor of n. However, the Nyquist frequency of a high-resolution image obtained by artificially reducing the pixel pitch to 1 / n tends to be smaller than n due to the effects of optical aberrations in the imaging optical system, etc.

[0050] Reducing an image (reducing resolution) is equivalent to lowering the sampling frequency, so the Nyquist frequency of the reduced image is lower than before reduction. Furthermore, to suppress aliasing noise, a filter process is applied to the image before reduction to remove frequencies above the Nyquist frequency of the reduced image. If the Nyquist frequency of the image before reduction is lower than expected, more high-frequency components of the reduced image will be removed than necessary, reducing the sharpness of the reduced image.

[0051] Therefore, in this embodiment, when reducing a high-resolution image whose Nyquist frequency has increased due to synthesis, the Nyquist frequency of the reduced image is calculated taking into account the frequency characteristics of the elements included in the optical system, thereby suppressing degradation in image quality of the reduced image.

[0052] In S601, the reduction processing unit 205 calculates a reduction ratio from the resolution (number of pixels in the horizontal direction × number of pixels in the vertical direction) of the original image (input image) to be reduced, which is supplied from the image processing unit 204, and the resolution of the reduced image. Here, the reduction ratio is the same in the horizontal and vertical directions of the image, and a reduction ratio of 1 / n means that the number of pixels is reduced to 1 / n in each direction. The reduction processing unit 205 can acquire the resolution of the reduced image, for example, from the system control unit 5 or RAM 7. The resolution of the reduced image may be changeable by the user via, for example, the operation unit 201. Note that if a reduction ratio has been set, the reduction processing unit 205 can acquire the set reduction ratio, for example, from the RAM 7 or the system control unit 5, instead of calculating the reduction ratio in S601.

[0053] In S602, the reduction processing unit 205 calculates the Nyquist frequency of the reduced image based on the reduction ratio calculated or acquired in S601. shrink is the Nyquist frequency of the original image (input image)in If the reduction ratio is 1 / n, it can be expressed by the following formula. f shrink =f in / n The Nyquist frequency of the input image is f if the input image is a high-resolution image obtained in high-resolution shooting mode. sr , and f for images taken in normal shooting mode. single The normal shooting mode is a shooting mode for acquiring an image (normal image) whose pixel pitch is equal to the pixel pitch of the image sensor 2. As described above, the Nyquist frequency f sr is a theoretical value based on the unit movement amount (pseudo pixel pitch) of the image sensor 2 when capturing each frame to be synthesized.

[0054] In S603, the reduction processing unit 205 acquires from the system control unit 5 whether the input image was captured in the normal shooting mode or the high-resolution shooting mode. Then, the image processing unit 204 determines whether the input image is a normal image or a high-resolution image based on the information acquired from the system control unit 5. Note that the reduction processing unit 205 may make this determination using another method. For example, the reduction processing unit 205 may determine that the input image is a high-resolution image if the resolution of the input image exceeds a threshold, and that the input image is a normal image if the resolution is equal to or less than the threshold. The threshold may be, for example, the number of effective pixels of the image sensor 2. Also, if the resolution of the input image is acquired when calculating the reduction ratio in S601, it may be used for the determination in S603. Alternatively, the reduction processing unit 205 may determine whether the input image is a normal image or a high-resolution image in S601, in which case S603 is unnecessary.

[0055] In step S604, the reduction processing unit 205 calculates the spatial frequency characteristic MTF of the image captured by the digital camera 1. total In this embodiment, the reduction processing unit 205 calculates the spatial frequency characteristic MTF (Modulation Transfer Function) of the image. total (f) is calculated based on the spatial frequency characteristics of the components of the optical system.

[0056] Specifically, the reduction processing unit 205 calculates the spatial frequency characteristic MTF of the captured image from the spatial frequency characteristics related to the imaging optical system 3, the optical low-pass filter of the imaging element 2, and the pixel aperture width of the imaging element 2. total Calculate (f).

[0057] Here, the spatial frequency characteristic of the imaging optical system 3 is expressed as MTF (Modulation Transfer Function) lens (f) The spatial frequency characteristics of the optical low-pass filter are shown as MTF. olpf (f) The spatial frequency characteristics of the pixel aperture width of the image sensor 2 are shown as MTF. sensor The reduction processing unit 205 reduces the spatial frequency characteristic MTF of the captured image. total (f) is calculated based on the following formula: MTF total (f) = MTF lens (f) × MTF sensor (f) × MTF olpf (f)

[0058] Spatial frequency characteristic MTF of the imaging optical system 3 lens (f) can be calculated based on the settings at the time of shooting (such as the aperture value) and information specific to the imaging optical system 3. The reduction processing unit 205 can acquire this information from the system control unit 5, or from the lens control unit 207 or the mechanical driving unit 208 via the system control unit 5.

[0059] Optical low-pass filter spatial frequency characteristics (MTF) olpf (f) and the spatial frequency characteristic MTF of the image sensor 2 based on the pixel aperture width sensor (f) is information specific to the image sensor 2. Therefore, it can be stored in the ROM 6 in advance, and the reduction processing unit 205 can obtain it through the system control unit 5.

[0060] The spatial frequency characteristics of the captured image according to the combination of dynamically changing parameters such as aperture value are also shown. total (f) may be calculated in advance and stored in the ROM 6. In this case, the reduction processing unit 205 can acquire, via the system control unit 5, the spatial frequency characteristics of the image based on the settings at the time of shooting.

[0061] For normal images with fewer high-frequency components than high-resolution images, the spatial frequency characteristics of the captured image are total (f)=1. Alternatively, MTF lens (f), MTF sensor (f), MTF olpf One or more of (f) may be set to 1.

[0062] In S605, the reduction processing unit 205 determines the parameters of a low-pass filter to be applied to the input image, using the Nyquist frequency of the reduced image calculated in S602 and the spatial frequency characteristics of the captured image calculated in S604. In this embodiment, the spatial frequency characteristics MTF (Module Transfer Function) are expressed by the following formula: dlpf A Gaussian low-pass filter with (f) is used, where σ is the dispersion parameter.

number

[0063] Spatial frequency characteristics of low-pass filters applied to suppress aliasing noise during image reduction: MTF dlpf (f) is MTF total (fshrink)×MTF dlpf It is necessary to determine (fshrink) so that it is close to 0. On the other hand, increasing the value of the dispersion parameter σ reduces the sharpness of the reduced image.

[0064] Therefore, the reduction processing unit 205 reduces the value of the dispersion parameter σ to the MTF total (fshrink)×MTF dlpf The spatial frequency characteristics of the low-pass filter are determined so that (fshrink) < T, where T is, for example, a threshold value (>0) that is experimentally determined in advance. In this way, the spatial frequency characteristics of the low-pass filter are determined so that the product of the spatial frequency characteristics of the captured image at the Nyquist frequency of the reduced image and the spatial frequency characteristics of the low-pass filter is less than the predetermined threshold.

[0065] In this way, the spatial frequency characteristics of the low-pass filter, MTF dlpfBy determining (f), it is possible to appropriately suppress spatial frequency components higher than the Nyquist frequency of the reduced image in consideration of the spatial frequency characteristics of the optical system during imaging. Therefore, it is possible to suppress the sharpness degradation of the reduced image caused by using a low-pass filter determined based on the theoretical Nyquist frequency that does not consider the spatial frequency characteristics of the optical system during imaging.

[0066] In addition, if the reduced image has the characteristic that the spatial frequency response (gain) above the Nyquist frequency is small and the spatial frequency response below the Nyquist frequency is large in consideration of the spatial frequency characteristics of the optical system during imaging, there is no limitation on the type of low-pass filter. Also, when the reduction processing unit 205 satisfies MTF total (fshrink) < T, it may execute S607 without applying a low-pass filter to the input image (skipping S606).

[0067] In S606, the reduction processing unit 205 applies a low-pass filter having the spatial frequency characteristics MTF dlpf (f) determined in S605 to the input image. When the low-pass filter is a two-dimensional spatial filter of size N×N, the relationship between the luminance value I(x, y) at the image coordinates (x, y) of the input image and the luminance value I'(x, y) at the image coordinates (x, y) after applying the low-pass filter is as follows.

Equation

[0068] In addition, the reduction processing unit 205 may use a low-pass filter having spatial frequency characteristics close to the MTF dlpf (f) calculated in S605 from among a plurality of low-pass filters stored in the ROM6 in advance.

[0069] In S607, the reduction processing unit 205 reduces the input image to which the low-pass filter was applied in S606. The reduction processing unit 205 outputs the generated reduced image to the format conversion unit 210. There are no limitations on the reduction method, and any known method can be used. Here, the nearest neighbor method is used as an example, but the bilinear method or bicubic method may also be used. When multiple pixel values ​​are used, as in the bilinear method or bicubic method, the spatial frequency characteristics according to the reduction method may also be taken into consideration when determining the spatial frequency characteristics of the low-pass filter in S606.

[0070] As described above, in this embodiment, the spatial frequency characteristics of the low-pass filter to be applied when reducing a high-resolution image generated by combining multiple frame images are determined based on the Nyquist frequency of the reduced image, taking into account the spatial frequency characteristics of the optical system of the imaging device. This makes it possible to suppress a decrease in perceived resolution of the reduced image that can occur when a low-pass filter based on the Nyquist frequency of the reduced image is applied, without taking into account the spatial frequency characteristics of the optical system of the imaging device.

[0071] ●(Second embodiment) Next, a second embodiment of the present invention will be described. In the first embodiment, it was assumed that a reduction process was performed on a high-resolution image when it was captured. In this embodiment, a case where a reduction process is performed on a high-resolution image that was recorded without being reduced will be described.

[0072] 6 is a block diagram showing an example of the functional configuration of an image processing device 700 according to this embodiment. The image processing device 700 may be any electronic device capable of executing a program, such as a computer device (personal computer, tablet computer, media player, PDA, etc.), a smartphone, or a game console. The reduction process according to this embodiment can also be performed in the digital camera 1 shown in FIG. 1. The system control unit 5 can perform the reduction process when the digital camera 1 is operating in, for example, playback mode.

[0073] In the first embodiment, a configuration was described in which a high-resolution image generated in high-resolution shooting mode is reduced by the reduction processing unit 205 before being recorded. In the present embodiment, it is assumed that an unreduced high-resolution image is recorded in high-resolution shooting mode (a reduced high-resolution image may or may not be recorded). It is also assumed that the format conversion unit 210 records metadata such as the model of the imaging device that generated the image and the shooting conditions (focal length, aperture value, shutter speed, ISO sensitivity, etc.) in association with the data file storing the unreduced high-resolution image. If the imaging optical system 3 is an interchangeable lens, the format conversion unit 210 also records the model of the interchangeable lens in the metadata. The format conversion unit 210 can record the metadata, for example, in the header of the data file, for example, as part of the Exif information.

[0074] The format conversion unit 210 may record, as metadata, the spatial frequency characteristics of the imaging optical system 3 according to the shooting conditions, the spatial frequency characteristics of the optical low-pass filter of the image sensor 2, and the spatial frequency characteristics of the image sensor 2 based on the pixel aperture width. The format conversion unit 210 can acquire these spatial frequency characteristics, for example, from the reduction processing unit 205 or the system control unit 5. Furthermore, if these spatial frequency characteristics are not included in the metadata, a program that implements the reduction processing can retain these spatial frequency characteristics for each model of digital camera (and interchangeable lens) and for each shooting condition.

[0075] In this way, the device (image processing device 700 or digital camera 1) that performs the reduction process on a high-resolution image is assumed to be able to acquire information required to calculate the Nyquist frequency of the reduced image from a data file or program.

[0076] In the image processing device 700, functional blocks with the same names as those in the digital camera 1 have the same functions, and their descriptions will be omitted. Note that the image processing unit 704, reduction processing unit 705, and format conversion unit 706 are described as independent functional blocks, but they may be realized by the system control unit 702 executing a program. Of course, some of the processing of these functional blocks may be performed using hardware circuits separate from the system control unit 702.

[0077] The image acquisition unit 703 acquires a data file storing a high-resolution image. The image acquisition unit 703 may read the data file from a memory card, for example, or may acquire the data file through communication with an external device.

[0078] The storage device 707 is a large-capacity storage device such as an HDD or SSD. Here, the reduced image is recorded in the storage device 707, but the reduced image may also be recorded in the source (memory card or external device) from which the high-resolution image before reduction was obtained.

[0079] The system control unit 702 displays, on the display unit 708, a GUI screen for selecting a data file storing an image to which reduction processing is to be applied, in response to a user instruction via the operation unit 701. The user operates the GUI screen via the operation unit 701 to select a desired data file and instructs the system control unit 702.

[0080] For example, when the enter button on the GUI screen is operated, the system control unit 702 instructs the image acquisition unit 703 to acquire the data file that is selected in the file list. The image acquisition unit 703 acquires the data file in response to the instruction from the system control unit 702 and temporarily stores it in, for example, the RAM 709.

[0081] The image processing unit 704 extracts a high-resolution image from a data file stored in the RAM 709. Then, the image processing unit 704 applies image processing to the extracted high-resolution image in the same manner as the image processing unit 204 does when capturing an image. If necessary for image processing, the image processing unit 704 extracts metadata from the data file and acquires information about the imaging device and imaging optical system that generated the image, information about the settings at the time of capturing an image, and the like.

[0082] The image processing unit 704 stores the processed high-resolution image in the RAM 709. It also notifies the reduction processing unit 705 of information necessary to calculate the Nyquist frequency of the reduced image. This information may include the spatial frequency characteristics of the imaging optical system 3 according to the shooting conditions, the spatial frequency characteristics of the optical low-pass filter of the image sensor 2, the spatial frequency characteristics of the image sensor 2 based on the pixel aperture width, and the shooting conditions.

[0083] The reduction processing unit 705 generates a reduced image by performing the same processing as in the first embodiment on the high-resolution image to which the image processing unit 704 has applied image processing. Specifically, the reduction processing unit 705 generates a reduced image by performing the processing described using the flowchart in Fig. 5.

[0084] The reduced image generated by the reduction processing unit 705 is stored in a data file by the format conversion unit 706 in the same manner as in the first embodiment. Thereafter, the format conversion unit 706 stores the data file in the storage device 707.

[0085] According to this embodiment, the same effect as in the first embodiment can be achieved even for high-resolution images that are recorded without being reduced.

[0086] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0087] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) an acquisition means for acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; generating means for generating a reduced image by reducing the second image; The generating means Calculating the Nyquist frequency of the reduced image; determining spatial frequency characteristics of a low-pass filter to be applied to the second image based on spatial frequency characteristics of an image captured by the imaging device and a Nyquist frequency of the reduced image; reducing the second image to which the low-pass filter having the determined spatial frequency characteristics has been applied to generate the reduced image; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the generation means determines the spatial frequency characteristics of the low-pass filter so that the product of the spatial frequency characteristics of the captured image at the Nyquist frequency of the reduced image and the spatial frequency characteristics of the low-pass filter is less than a predetermined threshold. (Item 3) 3. The image processing device according to item 1 or 2, wherein the generating means determines the spatial frequency characteristics of the captured image based on the spatial frequency characteristics of the optical system of the imaging device and the shooting conditions of the first image. (Item 4) 4. The image processing device according to item 3, wherein the spatial frequency characteristics of the optical system are based on the spatial frequency characteristics of an imaging optical system and the spatial frequency characteristics of an imaging element. (Item 5) 5. The image processing device according to item 4, wherein the spatial frequency characteristics related to the imaging element include the spatial frequency characteristics of an optical low-pass filter included in the imaging element and the spatial frequency characteristics based on the pixel aperture width of the imaging element. (Item 6) 6. The image processing device according to item 4 or 5, wherein the spatial frequency characteristics of the imaging optical system change depending on the imaging conditions. (Item 7) 7. The image processing device according to any one of items 1 to 6, wherein the Nyquist frequency of the reduced image is based on the Nyquist frequency of the second image and a reduction ratio when generating the reduced image from the second image. (Item 8) 8. The image processing device according to item 7, wherein the Nyquist frequency of the second image is based on the pixel pitch of the first image and the pixel pitch of the second image. (Item 9) The second image is stored in a data file, and the data file contains information necessary for the generating means to determine the spatial frequency characteristics of the image captured by the imaging device. (Item 10) 10. The image processing device according to any one of items 1 to 9, wherein the first images of the plurality of frames are images having a shift less than a pixel pitch of an imaging element. (Item 11) An image processing method executed by an image processing device, acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; generating a reduced image by reducing the second image; The generating step comprises: calculating a Nyquist frequency of the reduced image; determining spatial frequency characteristics of a low-pass filter to be applied to the second image based on spatial frequency characteristics of an image captured by the imaging device and a Nyquist frequency of the reduced image; applying a low-pass filter having the determined spatial frequency characteristics to the second image; reducing the second image to which the low-pass filter has been applied to generate the reduced image; An image processing method comprising: (Item 12) 11. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 10.

[0088] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0089] 1... digital camera, 2... image sensor, 3... image pickup optical system, 4... moving stage, 5... system control unit, 204... image processing unit, 205... reduction processing unit

Claims

1. an acquisition means for acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; generating means for generating a reduced image by reducing the second image; The generating means Calculating the Nyquist frequency of the reduced image; determining spatial frequency characteristics of a low-pass filter to be applied to the second image based on spatial frequency characteristics of an image captured by the imaging device and a Nyquist frequency of the reduced image; reducing the second image to which the low-pass filter having the determined spatial frequency characteristics has been applied to generate the reduced image; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the generation means determines the spatial frequency characteristics of the low-pass filter so that the product of the spatial frequency characteristics of the captured image at the Nyquist frequency of the reduced image and the spatial frequency characteristics of the low-pass filter is less than a predetermined threshold value.

3. 2. The image processing device according to claim 1, wherein the generating means determines the spatial frequency characteristics of the captured image based on the spatial frequency characteristics of an optical system of the image capturing device and the capturing conditions of the first image.

4. 4. The image processing apparatus according to claim 3, wherein the spatial frequency characteristics of the optical system are based on the spatial frequency characteristics of an imaging optical system and the spatial frequency characteristics of an imaging element.

5. 5. The image processing apparatus according to claim 4, wherein the spatial frequency characteristics relating to the image sensor include the spatial frequency characteristics of an optical low-pass filter included in the image sensor and the spatial frequency characteristics based on the pixel aperture width of the image sensor.

6. 5. The image processing apparatus according to claim 4, wherein the spatial frequency characteristics of the imaging optical system change depending on the photographing conditions.

7. 2. The image processing apparatus according to claim 1, wherein the Nyquist frequency of the reduced image is based on the Nyquist frequency of the second image and a reduction ratio used when generating the reduced image from the second image.

8. 8. The image processing apparatus according to claim 7, wherein the Nyquist frequency of the second image is based on the pixel pitch of the first image and the pixel pitch of the second image.

9. 2. The image processing device according to claim 1, wherein the second image is stored in a data file, and the data file contains information necessary for the generating means to determine the spatial frequency characteristics of the image captured by the imaging device.

10. The image processing device according to claim 1 , wherein the first images of the plurality of frames are images having a shift less than a pixel pitch of an image sensor.

11. An image processing method executed by an image processing device, acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; generating a reduced image by reducing the second image; The generating step comprises: calculating a Nyquist frequency of the reduced image; determining spatial frequency characteristics of a low-pass filter to be applied to the second image based on spatial frequency characteristics of an image captured by the imaging device and a Nyquist frequency of the reduced image; applying a low pass filter having the determined spatial frequency characteristics to the second image; reducing the second image to which the low-pass filter has been applied to generate the reduced image; An image processing method comprising:

12. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 10.

Images