Image processing device and image processing method

Abstract

To provide an image processing device and an image processing method capable of suppressing artifacts occurring in a high-resolution image obtained by combining images of multiple frames. An image processing device applies suppression processing to a second image having a higher resolution than a first image, generated by combining multiple frames of first images captured by an imaging device, to remove or suppress specific spatial frequency components. The specific spatial frequency components are spatial frequency components equal to or greater than a spatial frequency f0 at which the modulation transfer function (MTF) of the imaging optical system used to capture the multiple frames of the first images is 0%.

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Imaging devices using image sensors are constantly being required to produce images with higher resolution. This can be achieved by increasing the number of pixels in the image sensor. However, increasing the number of pixels in the image sensor requires either increasing the size of the image sensor or reducing the size of the pixels, which is not easy.

[0003] On the other hand, Patent Document 1 proposes a technique for acquiring an image with a higher resolution than the resolution of an image sensor by synthesizing a plurality of frames of images captured by slightly shifting the position of the image sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-111776 Summary of the Invention [Problem to be solved by the invention]

[0005] The high-resolution image generated by the method described in Patent Document 1 undergoes image processing aimed at improving image quality, such as noise reduction processing and processing to correct the effects of optical aberration. However, these image processing processes may result in images (artifacts) that do not actually exist. Furthermore, artifacts may also occur in the high-resolution image if the multiple frames of images used to generate the high-resolution image are affected by device shake or subject shake.

[0006] In view of the problems with the conventional techniques, one aspect of the present invention provides an image processing device and an image processing method that are capable of suppressing artifacts that occur in a high-resolution image obtained by combining images of multiple frames. [Means for solving the problem]

[0007] In one aspect, the present invention provides an image processing device comprising: an acquisition means for acquiring a second image having a higher resolution than a first image, the second image being generated by synthesizing a plurality of frames of first images captured by an imaging device; and a processing means for applying a suppression process to the second image to remove or suppress specific spatial frequency components, wherein the specific spatial frequency components are spatial frequency components equal to or greater than a spatial frequency f0 at which the modulation transfer function (MTF) of the imaging optical system used to capture the plurality of frames of the first images is 0%. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device and an image processing method that can suppress artifacts that occur in a high-resolution image obtained by combining images of multiple frames. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera as an image processing apparatus according to an embodiment; [Figure 2] FIG. 1 is a diagram for explaining a method for capturing a high-resolution image in an embodiment. [Figure 3] FIG. 3 is a diagram illustrating a method for combining images obtained by the photographing method shown in FIG. 2. [Figure 4] FIG. 1 is a diagram showing an example of an MTF curve of an imaging optical system and an application frequency component of AFR processing. [Figure 5] Block diagram showing an example of the functional configuration of the AFR processing unit [Figure 6] FIG. 10 is a diagram showing another example of the MTF curve of the imaging optical system and the frequency components to which AFR processing is applied. [Figure 7] Diagram showing an example of an artifact and its improvement [Figure 8] Another example of an artifact and its improvement. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following, the present invention will be described in terms of an embodiment using a digital camera as an example of an image processing device. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smart watches, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera 1 as an example of an image processing device according to the present invention.

[0013] The system control unit 5 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing programs. The system control unit 5 loads programs stored in the ROM 6 into the RAM 7 and executes them to control the operation of each unit of the digital camera 1 and realize the functions of the digital camera 1.

[0014] The ROM 6 is a rewritable non-volatile memory that stores programs executed by the system control unit 5, various setting values ​​for the digital camera 1, GUI data, and the like. The RAM 7 is used as a main memory for the system control unit 5, a buffer for temporarily storing captured image data, a work memory for temporarily storing data being processed by the image processing unit 104, etc. Furthermore, a part of the RAM 7 may be used as a video memory for storing image data to be displayed on the display unit 106.

[0015] The imaging optical system 3 forms an optical image of a subject on the imaging surface of the image sensor 2. The imaging optical system 3 is assumed to be fixed to the digital camera 1, but may be detachable. The imaging optical system 3 includes movable lenses such as a focus lens, a zoom lens, and an anti-vibration lens, as well as mechanisms for moving the lenses. The lens driver 107 controls the mechanisms for moving the movable lenses according to instructions from the system controller 5.

[0016] The exposure mechanism 109 has aperture blades that control the diameter of the optical path and a drive mechanism for them. The aperture blades can fully close the optical path and also function as a mechanical shutter. The mechanical drive unit 108 controls the operation of the exposure mechanism 109 according to instructions from the system control unit 5, and achieves exposure of the image sensor 2 according to the exposure conditions (aperture and shutter speed) determined by the system control unit 5.

[0017] The image sensor 2 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor 2 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out signals from each pixel having voltages according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface can be obtained. The analog image signals read out from the image sensor 2 are supplied to an A / D conversion unit 103.

[0018] The moving stage 4 is a moving mechanism capable of moving the image sensor 2 in two-dimensional directions parallel to the imaging surface. The moving stage 4 has, for example, a motor or an actuator, and moves the image sensor 2 in a direction and by a distance specified by the system control unit 5. The minimum distance of movement of the moving stage 4 is smaller than the pixel pitch of the image sensor 2. Note that the moving stage 4 may also be capable of moving the image sensor 2 in elevation and depression directions when the optical axis direction of the imaging optical system 3 is set to the horizontal direction. Also, although the image sensor 2 is moved here, a configuration in which the digital camera 1 is moved may also be used.

[0019] The A / D conversion unit 103 performs correlated double sampling, gain adjustment, A / D conversion, etc. on the analog image signal output from the image sensor 2, and outputs it as a digital image signal (image data). The amount of gain adjustment increases as the ISO sensitivity increases. The amount of gain adjustment is set by the system control unit 5. The A / D conversion unit 103 supplies the image data to the image processing unit 104.

[0020] The image processing unit 104 applies predetermined image processing to the image data output by the A / D conversion unit 103 to generate signals and image data according to the application, and acquires and / or generates various types of information. The image processing unit 104 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 104 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing unit 104 outputs the acquired or generated information and data to the system control unit 5, the AFR processing unit 105, or the like according to the application.

[0021] The image processing applied by the image processing unit 104 can include, for example, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Color interpolation, also known as demosaicing, is performed when the image sensor is equipped with a color filter, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. The correction processing may include white balance adjustment, edge enhancement, noise reduction, gradation (gamma) correction, correction of the effect of optical aberrations of the imaging optical system 3 on the image, correction of the effect of peripheral light falloff of the imaging optical system 3, and color correction. The detection process may include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing can include processes such as cropping, scaling, compositing, signal format and color space conversion, encoding, and decoding. Data processing also includes the generation of image data for display or recording. Data processing also includes the process of generating a high-resolution image by compositing multiple frames of images captured while moving the image sensor 2. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are examples of processes that the image processing unit 104 can apply, and do not limit the processes that the image processing unit 104 can apply.

[0022] In this embodiment, the recording image data (including high-resolution image data) generated by the image processing unit 104 is supplied to the AFR processing unit 105. However, when a setting is made not to perform artifact reduction processing (AFR (Artifact Reduction) processing), the image processing unit 104 may output the recording image data to the format conversion unit 110.

[0023] The AFR processing unit 105 analyzes the image data for recording output by the image processing unit 104 and determines whether or not an artifact exists. If the AFR processing unit 105 determines that an artifact exists, it applies AFR processing to the image data for recording. Details of the operation of the AFR processing unit 105 will be described later. Like the image processing unit 104, the AFR processing unit 105 may be a dedicated hardware circuit such as an ASIC, or may be configured to achieve a specific function by a processor such as a DSP or GPU executing software. In addition, the image processing unit 104 and the AFR processing unit 105 may actually be integrated.

[0024] The display unit 106 includes a display device such as an LCD, and performs display based on display image data stored in the video memory area of ​​the RAM 7. The display device included in the display unit 106 can function as an electronic viewfinder.

[0025] The format conversion unit 110 generates a data file that stores the image data supplied from the reduction processing unit 105. Specifically, the format conversion unit 110 generates metadata that conforms to specifications according to the format of the data file to be recorded, and encodes the image data according to the specifications. There are no particular restrictions on the file formats that the format conversion unit 110 can generate, and it may also generate a data file in a manufacturer-specific format.

[0026] The medium control unit 111 records the data file generated by the format conversion unit 110 in a predetermined recording destination, for example, a recording medium 113. The recording medium 113 may be, for example, a memory card attached to the digital camera 1 or a communicable external device. The medium control unit 111 also acquires the recorded data file from the recording medium 113 and supplies it to the image processing unit 104.

[0027] The digital camera 1 can take and record still images and moving images. Although not specifically described, when taking and recording moving images, audio is also processed in addition to images.

[0028] The operation unit 101 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the digital camera 1. The input devices constituting the operation unit 101 are named according to the functions assigned to them. For example, the operation unit 101 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, an enter key, etc. In this embodiment, the operation unit 101 can be used to set whether to always execute AFR processing, never execute AFR processing, or have the digital camera 1 automatically decide whether to execute AFR processing.

[0029] The release switch is a switch for recording still images, and the system control unit 5 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. Furthermore, the system control unit 5 recognizes a video recording switch pressed in the shooting standby state as an instruction to start video recording, and a video recording stop instruction when pressed during video recording. The functions assigned to the same input device may be variable. The input device may also be a software button or key using a touch display. The operation unit 101 may also include an input device compatible with non-contact input methods, such as voice input or eye-gaze input.

[0030] When the system control unit 5 detects an operation on the operation unit 101, it executes an operation corresponding to the detected operation. For example, when the system control unit 5 detects a half-press of the release switch in a shooting standby state, it executes AF and AE operations as a preparation operation for shooting a still image. In a shooting standby state, the image processing unit 104 continuously generates signals or evaluation values ​​used for AF and AE for the video (live view image) being shot so that the display unit 106 functions as an EVF. The system control unit 5 uses these signals or evaluation values ​​to execute AF processing, which controls the position of the focus lens, and AE processing, which determines the exposure conditions (aperture value, shutter speed, ISO sensitivity). Since the AF processing and AE processing can be executed by any known method, further detailed explanations are omitted.

[0031] Furthermore, when the system control unit 5 detects that the release switch has been fully pressed, it controls the operation of the related functional blocks so as to carry out a series of operations from taking a still image to recording a still image data file.

[0032] In addition, when a shooting mode is set in which a high-resolution image is generated by combining multiple frame images captured by moving the image sensor 2, the system control unit 5 controls the operation of the moving stage 4 to move the image sensor 2 to a predetermined position for each shot.

[0033] (High resolution image capture operation) Next, a description will be given of the operation of the digital camera 1 when generating a high-resolution image. In this embodiment, as an example, a high-resolution image is generated when a high-resolution shooting mode is set through operation of the operation unit 201. However, the system control unit 5 may automatically set the high-resolution shooting mode in accordance with, for example, the image analysis results of the image processing unit 204.

[0034] In high-resolution shooting mode, digital camera 1 captures multiple frame images (first images) by moving image sensor 2 to different positions using moving stage 4. These multiple frame images have a shift less than the pixel pitch of image sensor 2. Digital camera 1 then synthesizes the captured multiple frame images to generate a high-resolution image (second image) having a pixel pitch smaller than the physical pixel pitch of image sensor 2.

[0035] 2 is a diagram showing an example of the position of the moving stage 4 (image sensor 2) when capturing each frame when the digital camera 1 synthesizes four images (frames) to generate a high-resolution image. Here, the pixel pitch in both the horizontal and vertical directions of the image sensor 2 is denoted by P. The position of the moving stage 4 when it is not moving is defined as the reference position.

[0036] In Figure 2, digital camera 1 (a) Reference position (b) Position moved horizontally by P / 2 from the reference position (c) Position moved vertically by P / 2 from the reference position (d) Position moved by P / 2 horizontally and P / 2 vertically from the reference position By combining the four frames captured at these positions, digital camera 1 can generate a high-resolution image with double the resolution in both the horizontal and vertical directions.

[0037] 2 are merely examples, and other values ​​may be used. However, the movement distance must be less than P or a value other than an integral multiple of P (for example, 3P / 2). The number of synthesized frames N (≧2) is determined by the Nyquist frequency f(nq.) of one frame of image before synthesis and the Nyquist frequency f(nq.) of the high-resolution image obtained by synthesis. SR Based on (nq.), it may be determined using the following formula: N= (f SR (nq.) / f(nq.)) 2

[0038] However, the number of composite frames N may be greater or less than the value obtained by the above formula. Also, the number of composite frames may be increased or decreased depending on the resolution per frame.

[0039] It is preferable to suppress the effects of blurring when shooting in high-resolution shooting mode, so an electronic shutter may be used instead of a mechanical shutter (exposure mechanism 209).

[0040] The image data for N frames to be synthesized is processed by the A / D conversion unit 203 and then sequentially stored in, for example, the RAM 7. Since the image sensor 2 has color filters in a primary color Bayer array, the image data stored in the RAM 7 is RAW data in which each pixel has a single color component according to the primary color Bayer array.

[0041] The image processing unit 204 converts the image data for N frames into image data in which each pixel has RGB components (or YCbCr components) and then synthesizes the converted image data. A synthesis method for generating a high-resolution image will be described with reference to FIG.

[0042] FIG. 3 shows how the pixels of the four frame images captured at the positions shown in FIGS. 2(a) to (d) are arranged in the synthesized high-resolution image. P is the (actual) pixel pitch of the image sensor 2. As shown in FIG. 3, pixels of the second and subsequent frame images are arranged between the pixels (●) of the first frame image captured at the reference position, at positions corresponding to the offset between the reference position and the capturing position. For example, since the second frame was captured at a position offset by P / 2 horizontally to the right from the reference position, the pixel (■) of the second frame is arranged at the midpoint between the horizontally adjacent pixels in the first frame image. The same applies to the pixels (□) of the third frame and the pixels (×) of the fourth frame.

[0043] Here, the amount of movement is set to P / 2, but by synthesizing multiple frames captured by moving the image horizontally, vertically, and diagonally in units of P / m (m≧3), a high-resolution image with horizontal and vertical resolution m times higher than that before synthesis can be obtained. Theoretically, the Nyquist frequency of one frame of captured image before synthesis (determined according to the pixel pitch) is f(nq.), and the Nyquist frequency of the high-resolution image after synthesis is f SR (nq.), the following relationship holds: f SR (nq.)=m×f(nq.)

[0044] FIG. 4 shows an example of a modulation transfer function (MTF) curve of the imaging optical system 3. The vertical axis represents MTF (%), and the horizontal axis represents spatial frequency (lp / mm). FIG. 4 also shows the above-mentioned Nyquist frequency f (nq.) and f SR In addition to the (nq.), the spatial frequency f0 at which the MTF is 0% is also shown. In the example shown in Figure 4, f(nq.), f0, and f SR (nq.) have the following relationship: f(nq.) < f0< f SR (nq.)

[0045] In the example shown in Figure 4, SR The MTF in the spatial frequency region up to (nq.) is 0. The subject image included in this spatial frequency region cannot be resolved by the imaging optical system 3 because MTF=0. Therefore, the high-resolution image obtained by synthesis contains spatial frequency components that cannot be resolved (imaged) by the imaging optical system 3. As will be described later, the AFR processing unit 105 performs MTF analysis on the spatial frequency components above f0 where the MTF is 0% and above the Nyquist frequency f of the high-resolution image. SR AFR processing is applied to spatial frequency components below (nq.) to suppress artifacts.

[0046] The image processing unit 104 applies image processing such as white balance correction, optical correction, edge emphasis, noise removal, pixel interpolation, gamma correction, and color difference signal generation to the high-resolution image in the same manner as to a normal (non-composite) image. The high-resolution image data output by the image processing unit 104 is assumed to be in YUV format.

[0047] The high-resolution image data output from the image processing unit 104 is input to the AFR processing unit 105. The AFR processing unit 105 determines whether or not there are artifacts in the image-processed high-resolution image data. If it is determined that there are artifacts, the AFR processing unit 105 applies AFR processing to the high-resolution image data. Then, the AFR processing unit 105 outputs the high-resolution image data in which the artifacts have been suppressed to the format conversion unit 110. On the other hand, if it is determined that there are no artifacts, the AFR processing 105 outputs the high-resolution image data to the format conversion unit 110 without applying AFR processing to the high-resolution image data. Note that in the normal shooting mode instead of the high-resolution shooting mode, the AFR processing 105 also outputs the high-resolution image data to the format conversion unit 110 without applying AFR processing to the high-resolution image data. The normal shooting mode is a shooting mode in which one frame of image (normal image) is acquired, the pixel pitch of which is equal to the pixel pitch of the image sensor 2.

[0048] The format conversion unit 110 generates a recording data file for storing the recording image data, and outputs the file to the medium control unit 111. The medium control unit 111 records the recording data file on the recording medium 113.

[0049] (Artifacts in high resolution images) Here, we will explain the artifacts suppressed by the AFR processing unit 105. The AFR processing unit 105 suppresses artifacts by reducing the f0 to f SR The spatial frequency components (nq.) (Fig. 4) are considered artifacts and AFR processing is applied to suppress them.

[0050] Examples of artifacts in high-resolution images are shown in Figures 7(a) and 8(a). 7(a) shows an example of an artifact caused by noise reduction processing applied by the image processing unit 104. In the figure, a pixel-by-pixel grid pattern has occurred near the edge of the subject image in the area surrounded by a white circle. The pixel-by-pixel grid pattern in the high-resolution image has a spatial frequency that cannot be resolved by the imaging optical system 3, and is therefore an artifact caused by image processing of the high-resolution image.

[0051] Figure 8(a) shows an example of an artifact that occurs in a high-resolution image when subject or device shake occurs in one or more frames of multiple frames captured in high-resolution shooting mode. A minute pattern appears across the entire edge of the subject image. In high-resolution shooting mode, digital camera 1 must be fixed to a tripod or other device in order to capture multiple frames with the intended size of deviation. However, depending on the shooting environment, it is impossible to eliminate minute vibrations of digital camera 1, which can cause artifacts in the combined high-resolution image. The minute pattern caused by shake also has a spatial frequency that cannot be resolved by the imaging optical system 3.

[0052] Unlike noise that increases during high-sensitivity shooting, these artifacts generally exhibit contrast greater than the amplitude of the noise and often have contrast similar to that of the edge portions of the subject image. Therefore, noise reduction processing applied by the image processing unit 104, which primarily aims to reduce image noise while maintaining the contrast of edge portions, cannot sufficiently suppress these artifacts. Furthermore, if the noise reduction processing is strengthened to suppress the artifacts, the contrast of edge portions decreases, resulting in a problem of reduced image resolution.

[0053] 7A and 8A are examples of artifacts that can be effectively suppressed by the AFR processing unit 105. The AFR processing unit 105 performs a process of reducing the MTF of the imaging optical system 3 from f0, where the MTF is 0%, to the Nyquist frequency f of the high-resolution image. SR (nq.) can suppress all artifacts in the spatial frequency range.

[0054] (AFR processing unit 105) 5 is a block diagram showing an example of the functional configuration of the AFR processing unit 105 together with related peripheral blocks. The AFR processing unit 105 has an artifact detection unit 1051 and an AFR processing execution unit 1052. The high-resolution image output by the image processing unit 104 is input to both the artifact detection unit 1051 and the AFR processing execution unit 1052.

[0055] The artifact detection unit 1051 determines the presence or absence of artifacts in a high-resolution image generated in high-resolution shooting mode and notifies the AFR processing execution unit 1052 of the determination result. This determination is performed when the digital camera 1 is set to automatically determine whether to perform AFR processing. There are no particular limitations on the method for determining the presence or absence of artifacts. For example, the artifact detection unit 1051 can determine the presence or absence of artifacts using a trained machine learning model that has performed supervised learning using images with and without artifacts. Alternatively, the artifact detection unit 1051 can determine the presence or absence of artifacts using template matching using a template of an artifact image stored in advance in ROM 6. Specifically, the artifact detection unit 1051 can determine the presence of an artifact if an area with a correlation with the template equal to or greater than a threshold is detected in the high-resolution image, and determine the absence of an artifact if no artifact is detected.

[0056] When the setting is such that AFR processing is always performed, the artifact detection unit 1051 does not perform the determination process, and always notifies the AFR processing execution unit 1052 of the determination result that an artifact is present. On the other hand, when the setting is such that AFR processing is never performed, the artifact detection unit 1051 does not perform the determination process, and always notifies the AFR processing execution unit 1052 of the determination result that an artifact is not present.

[0057] The AFR processing execution unit 1052 determines its operation depending on the determination result of the presence or absence of artifacts notified by the artifact detection unit 1051. If the determination result is "no artifacts," the AFR processing execution unit 1052 inputs the image input from the image processing unit 104 to the formatting unit 110 without performing AFR processing on the image. On the other hand, if the determination result is "artifacts are present," the AFR processing execution unit 1052 performs AFR processing on the image input from the image processing unit 104 before inputting the image to the formatting unit 110.

[0058] The AFR processing will be described in detail below. As described above, the AFR processing is performed by adjusting the frequency of the high-resolution image from f0, at which the MTF of the imaging optical system 3 is 0%, to the Nyquist frequency f SR (nq.) are targeted. These spatial frequency components are components that cannot be imaged by the imaging optical system 3 and are artifacts that do not actually exist in the captured image, so removing or reducing them does not affect the perceived resolution of the image. In this way, the AFR processing execution unit 1052 processes the spatial frequency components from f0 to f SR The AFR processing unit 1052 applies AFR processing to the high-resolution image, removing or suppressing (blurring) spatial frequency components up to (nq.). Note that the AFR processing execution unit 1052 may change the lower limit of the spatial frequency band to be subjected to AFR processing to a spatial frequency lower than f0 that corresponds to the amplitude of the noise.

[0059] 6 is a diagram showing the relationship between the MTF (%) of the imaging optical system 3 and the spatial frequency (lp / mm) when the ISO sensitivity during shooting in the high-resolution shooting mode is equal to or greater than a threshold determined according to the characteristics of the image sensor 2. a has been added.

[0060] a is a value obtained by converting the amplitude of noise superimposed on a high-resolution image into the dimension of MTF. An edge of a subject image having a contrast equal to or less than the noise amplitude a cannot be distinguished from noise. In other words, the spatial frequency at which the MTF of the imaging optical system 3 becomes a is f a Then, the spatial frequency f a An image of a subject with a spatial frequency of (≦ f0) or more cannot be distinguished from noise. Therefore, when noise is taken into account, f a From f SR The spatial frequency region (nq.) can be the target of AFR processing.

[0061] The upper limit of the spatial frequency range to which AFR processing is applied is the Nyquist frequency f SR(nq.), but the upper limit does not have to be set. The AFR processing execution unit 1052 can apply AFR processing to a spatial frequency region at least equal to or higher than the spatial frequency f0. If an upper limit is necessary or is preferable depending on the AFR processing method or for the purpose of reducing the processing load, the AFR processing execution unit 1052 sets the upper limit of the spatial frequency region to which AFR processing is applied to f SR (nq.) can be used.

[0062] The AFR processing execution unit 1052 determines the spatial frequency region to which the AFR processing is applied, using spatial frequencies f0, f SR (nq.), f a can be obtained in a variety of ways.

[0063] The AFR processing execution unit 1052 performs the AFR process on f0 and f a The MTF of the imaging optical system 3 stored in the ROM 6 can be obtained through the system control unit 5, for example, and the spatial frequencies at which the MTF becomes 0 and a can be determined. The value of a varies depending on one or more of the characteristics of the imaging element 2, the ISO sensitivity at the time of shooting, and the brightness of the high-resolution image. Therefore, it is possible to determine in advance for the imaging element 2 the values ​​of a or f according to one or a combination of multiple ISO sensitivities and multiple brightnesses of high-resolution images. a can be stored in the ROM 6. Since f0 is an inherent value of the imaging optical system 3, it can also be stored in the ROM 6 in advance.

[0064] The AFR processing execution unit 1052 can acquire the ISO sensitivity at the time of capturing the multiple frame images that formed the high-resolution image from the system control unit 5, and information about the brightness of the high-resolution image from the image processing unit 104. The information about the brightness of the high-resolution image may be information indicating the overall brightness of the high-resolution image, such as an average luminance value. Then, the AFR processing execution unit 1052 selects a or f corresponding to the acquired information. aand f0 can be acquired from ROM 6 via the system control unit 5. If values ​​corresponding to the acquired ISO sensitivity or brightness information are not stored in ROM 6, the AFR process execution unit 1052 may obtain values ​​corresponding to the acquired ISO sensitivity or brightness information by, for example, interpolating the stored values.

[0065] The AFR process execution unit 1052 uses, for example, the acquired size of a or f a Based on the difference between a and f0, the AFR processing execution unit 1052 can determine the lower limit of the spatial frequency region to which the AFR processing is applied. Specifically, when the magnitude of a is equal to or greater than a first threshold, the AFR processing execution unit 1052 determines the lower limit of the spatial frequency region to which the AFR processing is applied. a If the first threshold is less than the first threshold, the AFR processing execution unit 1052 sets the lower limit of the spatial frequency region to which the AFR processing is applied as f a If the difference between f0 and f1 is equal to or greater than the second threshold, a Alternatively, the AFR processing execution unit 1052 may set the lower limit of the spatial frequency region to which the AFR processing is applied to f when the ISO sensitivity at the time of capturing the images of the multiple frames is equal to or greater than the third threshold. a When the first threshold value is less than the third threshold value, the value can be set to f0. The first to third threshold values ​​can be determined in advance.

[0066] Alternatively, the AFR processing execution unit 1052 may set the lower limit of the spatial frequency region to which the AFR processing is applied as f a and f0, whichever is set by the user.

[0067] In addition, the AFR processing execution unit 1052 performs f SR For (nq.), a value stored in advance in the ROM 6 can also be acquired. The Nyquist frequency f(nq.) of each frame before synthesis is a fixed value determined according to the pixel pitch of the image sensor 2. On the other hand, the Nyquist frequency f(nq.) of a high-resolution image is SR(nq.) changes depending on the number of composite frames or the amount of deviation between frames (the amount of movement of the image sensor 2 or the digital camera 1). Therefore, if the number of composite frames in high-resolution shooting mode is not fixed, f SR (nq.) can be stored in the ROM 6. The AFR processing execution unit 1052 executes the AFR processing according to the shooting settings of the high-resolution image to which the AFR processing is applied. SR (nq.) can be obtained from the ROM 6 through the system control unit 5.

[0068] The method described here is an example, and the AFR processing execution unit 1052 SR (nq.), f a may be obtained in other ways.

[0069] The AFR process may be a process that removes or reduces the target spatial frequency component by any known method. Examples of the AFR process include blurring, noise reduction, smoothing, interpolation, and low-pass filtering, which reduce the sharpness of an image. It is desirable for the AFR process to have as little effect as possible on spatial frequency components that are not included in the spatial frequency region to which the AFR process is applied.

[0070] The interpolation process is performed using the lower limit of the spatial frequency domain (f a The unit may be the number of pixels (>1) or less equivalent to the Nyquist frequency f of a high-resolution image. SR The blurring effect of the interpolation process is small when interpolating in units of one pixel, which corresponds to (nq.). On the other hand, the blurring effect of the interpolation process is small when the spatial frequency f a The blurring effect is also greater when the interpolation process is performed using a larger number of pixels as a unit, which corresponds to the above.

[0071] The interpolation process may be, for example, an image enlargement or reduction process (scaling process), and bilinear interpolation or bicubic interpolation may be used. For example, as the AFR process, a reduction process with a reduction ratio X that satisfies the following formula can be executed. The reduction ratio X is the ratio (<1) of the size after reduction when the size before reduction is 1. f a / f SR (nq.) <X<1

[0072] 7(b) and 8(b) show the results of the spatial frequency f a The results of applying AFR processing to blur the above are shown below. It can be seen that AFR processing effectively suppresses artifacts while maintaining image sharpness.

[0073] As described above, in this embodiment, a process for suppressing spatial frequency components equal to or higher than the spatial frequency at which the MTF of the imaging optical system becomes 0% is applied to a high-resolution image generated by combining multiple frame images. This makes it possible to effectively suppress artifacts caused by image processing applied to the high-resolution image.

[0074] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) an acquisition means for acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; a processing means for applying suppression processing to the second image to remove or suppress specific spatial frequency components, The image processing device is characterized in that the specific spatial frequency component is a spatial frequency component equal to or greater than a spatial frequency f0 at which a modulation transfer function (MTF) of an imaging optical system used to capture a first image of the plurality of frames is 0%. (Item 2) The specific spatial frequency component has a spatial frequency f lower than the spatial frequency f0. a These are the spatial frequency components, The spatial frequency f a is determined according to the amplitude of noise superimposed on the second image. (Item 3) The spatial frequency f a 3. The image processing device according to item 2, wherein the MTF value is a spatial frequency that is a value obtained by converting the amplitude of the noise into the dimension of the MTF. (Item 4) The processing means processes the spatial frequency f a The image processing device according to item 2 or 3, characterized in that the image processing device obtains the brightness of the second image in accordance with at least one of the ISO sensitivity used to capture the first image of the plurality of frames and the brightness of the second image. (Item 5) When the ISO sensitivity used to capture the first image of the plurality of frames or the brightness of the second image is equal to or greater than a threshold, the processing means converts the specific spatial frequency component into the spatial frequency f a 5. The image processing device according to any one of items 2 to 4, wherein the specific spatial frequency component is set to a component having a spatial frequency of f0 or more when the specific spatial frequency component is less than the threshold value. (Item 6) The specific spatial frequency component is the Nyquist frequency f of the second image. SR 6. The image processing device according to any one of items 1 to 5, characterized in that the spatial frequency components are (nq.) or less. (Item 7) Further, the image processing device has a determination unit that determines whether or not the suppression processing is to be applied to the second image by the processing unit. 7. The image processing device according to any one of items 1 to 6, wherein the processing means applies the suppression processing to the second image when the determination means determines that the suppression processing should be applied, and does not apply the suppression processing to the second image when the determination means determines that the suppression processing should not be applied. (Item 8) 8. The image processing device according to item 7, wherein the determination means determines to apply the suppression processing if it is determined that the second image contains an artifact caused by image processing applied to the second image, and determines not to apply the suppression processing if it is determined that the second image does not contain the artifact. (Item 9) 9. The image processing device according to item 8, wherein the spatial frequency of the artifact is equal to or greater than the spatial frequency f0. (Item 10) 10. The image processing device according to item 8 or 9, wherein the determination means determines the presence or absence of the artifact in the second image using a machine learning model trained using an image with the artifact and an image without the artifact. (Item 11) 10. The image processing device according to item 8 or 9, wherein the determination means determines the presence or absence of the artifact in the second image by template matching using the image with the artifact as a template. (Item 12) 12. The image processing device according to any one of items 1 to 11, wherein the suppression processing is blurring processing, scaling processing, or low-pass filtering processing of the second image. (Item 13) The scaling process is performed at a Nyquist frequency f of the second image. SR (nq.) and the spatial frequency f0, f a / f SR (nq.) <X<1 Item 13. The image processing device according to item 12, characterized in that the reduction process is performed at a reduction ratio X that satisfies the following. (Item 14) An image processing method executed by an image processing device, acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; applying a suppression process to the second image to remove or suppress specific spatial frequency components; An image processing method, characterized in that the specific spatial frequency component is a spatial frequency component equal to or greater than a spatial frequency f0 at which the modulation transfer function (MTF) of an imaging optical system used to capture the first image of the plurality of frames is 0%. (Item 15) 14. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 13.

[0076] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0077] 1... digital camera, 2... imaging element, 3... imaging optical system, 4... moving stage, 102... system control unit, 104... image processing unit, 105... AFR processing unit, 1051... artifact detection unit, 1052... AFR processing unit

Claims

1. an acquisition means for acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; a processing means for applying suppression processing to the second image to remove or suppress specific spatial frequency components, The specific spatial frequency component is a spatial frequency f 0 An image processing device characterized in that the spatial frequency components are as described above.

2. The specific spatial frequency component is the spatial frequency f 0 Lower spatial frequency f a These are the spatial frequency components, The spatial frequency f a 2. The image processing apparatus according to claim 1, wherein is determined according to the amplitude of noise superimposed on the second image.

3. The spatial frequency f a 3. The image processing apparatus according to claim 2, wherein the MTF value is a spatial frequency that is a value obtained by converting the amplitude of the noise into the dimension of the MTF.

4. The processing means processes the spatial frequency f a The image processing device according to claim 2 , characterized in that the brightness of the second image is acquired in accordance with at least one of the ISO sensitivity used in capturing the first image of the plurality of frames and the brightness of the second image.

5. When the ISO sensitivity used to capture the first images of the plurality of frames or the brightness of the second images is equal to or greater than a threshold, the processing means converts the specific spatial frequency component into the spatial frequency f a If the threshold value is less than the specified spatial frequency, the specified spatial frequency component is set to the spatial frequency f 0 3. The image processing device according to claim 2, wherein the components are as described above.

6. The specific spatial frequency component is the Nyquist frequency f of the second image. SR 2. The image processing device according to claim 1, wherein the spatial frequency components are equal to or less than nq.

7. Further, the image processing device has a determination unit that determines whether or not the suppression processing is to be applied to the second image by the processing unit.

2. The image processing device according to claim 1, wherein the processing means applies the suppression processing to the second image when the determination means determines that the suppression processing should be applied, and does not apply the suppression processing to the second image when the determination means determines that the suppression processing should not be applied.

8. 8. The image processing device according to claim 7, wherein the determination means determines to apply the suppression processing if it is determined that the second image contains an artifact caused by image processing applied to the second image, and determines not to apply the suppression processing if it is determined that the second image does not contain the artifact.

9. The spatial frequency of the artifact is the spatial frequency f 0 9. The image processing device according to claim 8, wherein:

10. 9. The image processing device according to claim 8, wherein the determining means determines the presence or absence of the artifact in the second image by using a machine learning model trained using an image with the artifact and an image without the artifact.

11. 9. The image processing apparatus according to claim 8, wherein the determining means determines the presence or absence of the artifact in the second image by template matching using the image having the artifact as a template.

12. 2. The image processing device according to claim 1, wherein the suppression processing is blurring processing, scaling processing, or low-pass filtering processing of the second image.

13. The scaling process is performed at a Nyquist frequency f of the second image. SR (nq.) and the spatial frequency f 0 In contrast, f a / f SR (nq.)<X<1 13. The image processing apparatus according to claim 12, wherein the reduction process is performed at a reduction ratio X that satisfies the following formula:

14. An image processing method executed by an image processing device, acquiring a second image having a higher resolution than the first image, the second image being generated by combining a plurality of frames of first images captured by an imaging device; applying a suppression process to the second image to remove or suppress specific spatial frequency components; The specific spatial frequency component is a spatial frequency f 0 An image processing method characterized in that the spatial frequency components are as described above.

15. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 13.

Images