Image processing device, imaging apparatus, image processing method, and program

Abstract

To provide a technique capable of acquiring a highly accurate differential, which is available for identifying a mobile region in a high-resolution image, from the high-resolution image and an original picked-up image.SOLUTION: An image processing device comprises: first acquisition means for acquiring a plurality of picked-up images picked up at a plurality of imaging positions which are located in such a manner that an interval between imaging positions which are adjacent in a first direction or a second direction which is orthogonal to the first direction is a non-integer multiple of a pixel pitch of an imaging element; generation means for generating a high-resolution image where the resolution is higher than that of the plurality of picked-up images by combination processing of the plurality of picked-up images; filter means for applying a filter which decreases a high frequency component caused by the combination processing to the high-resolution image; and second acquisition means which performs processing for acquiring a differential between the high-resolution image to which the filter is applied and one picked-up image in the plurality of picked-up images.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] In a digital camera in which the position of the image sensor can be changed, a technique is known in which an image (high-resolution image) with a resolution exceeding the number of pixels of the image sensor is generated by combining multiple images taken at shooting positions shifted by a distance smaller than the pixel pitch (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-226489 Summary of the Invention [Problem to be solved by the invention]

[0004] When there is a subject (moving subject) moving between multiple images to be synthesized, the synthesized image will have an area (moving object area) where components derived from images that include the moving subject and components derived from images that do not include the moving subject are mixed, making the synthesized image unnatural.

[0005] It is generally known that the area of a moving subject can be detected by obtaining the difference between images.

[0006] Here, in order to detect a moving object region in a high-resolution image, it is considered to obtain the difference between the high-resolution image and the original captured image. When a high-resolution image is generated using a technique such as that described in Patent Document 1, high-frequency components that are not included in the original captured image are generally generated. Therefore, if the high-resolution image and the original captured image are simply made the same size and the difference between the images is obtained, a relatively large difference may be obtained even in an area where a moving object does not actually exist. As a result, it is not possible to detect a moving subject with high accuracy.

[0007] The present invention has been made in consideration of this situation, and aims to provide a technology that makes it possible to obtain highly accurate differences between a high-resolution image and the original captured image that can be used to identify moving object areas in the high-resolution image. [Means for solving the problem]

[0008] In order to solve the above problem, the present invention provides an image processing device comprising: a first acquisition means for acquiring a plurality of captured images taken at a plurality of shooting positions positioned so that the distance between adjacent shooting positions in a first direction or a second direction perpendicular to the first direction is a non-integer multiple of the pixel pitch of an image sensor; a generation means for generating a high-resolution image having a higher resolution than the plurality of captured images by combining the plurality of captured images; a filter means for applying a filter to the high-resolution image to reduce high-frequency components generated by the combining process; and a second acquisition means for performing a process to acquire the difference between the high-resolution image after the filter has been applied and one of the plurality of captured images. [Effects of the Invention]

[0009] According to the present invention, it is possible to obtain a highly accurate difference between a high-resolution image and an original captured image, which can be used to identify a moving object region in the high-resolution image.

[0010] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiment of the present invention. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the configuration of a digital camera 100 (image capture device) including an image processing device. [Figure 2] 10 is a flowchart of a high-resolution image generation process. [Figure 3] A conceptual diagram of the movement of the imaging element. [Figure 4] 10 is a flowchart showing details of the mobile object area map generation process (S207). DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [First embodiment] FIG. 1 is a block diagram showing the configuration of a digital camera 100 (image capture device) including an image processing device. The digital camera 100 can capture still images. The digital camera 100 can also move the position of the image sensor included in the image capture unit 105 in units less than the pixel pitch. The digital camera 100 can also record information about the focus position, calculate the contrast value of an image, and combine multiple images. The digital camera 100 can also perform enlargement and reduction processes on images that it has captured and saved, or on images input from an external device.

[0014] In addition, although the present embodiment and other embodiments will be described using a digital camera, the embodiments are not limited to a digital camera. For example, instead of a digital camera, a mobile device with a built-in image sensor or a network camera capable of capturing images may be used.

[0015] The control unit 101 includes a processor such as a CPU or MPU, and controls each block of the digital camera 100 by reading and executing a program stored in advance in the ROM 107. For example, as will be described later, the control unit 101 issues commands to the imaging unit 105 regarding the start and end of imaging. The control unit 101 also issues commands to the image processing unit 109 for image processing based on the program stored in the ROM 107. Commands from the user are input to the digital camera 100 by the operation unit 112 and reach each block of the digital camera 100 via the control unit 101.

[0016] The driving unit 102 includes a motor and the like, and mechanically operates the optical system 103 under the command of the control unit 101. For example, the driving unit 102 adjusts the focal length of the optical system 103 by moving the position of a focus lens included in the optical system 103 based on the command of the control unit 101.

[0017] The optical system 103 includes a zoom lens, a focus lens, an aperture, etc. The aperture is a mechanism for adjusting the amount of light that passes through. The focal position can be changed by changing the position of the lens.

[0018] The communication unit 104 mainly transmits information between the control unit 101 and the optical system 103 under the command of the control unit 101 .

[0019] The imaging unit 105 includes an imaging element including a photoelectric conversion element, which performs photoelectric conversion to convert incident light into an electrical signal. For example, a CCD sensor or a CMOS sensor can be used as the imaging element of the imaging unit 105. The imaging element included in the imaging unit 105 is configured to be movable by a predetermined amount in the horizontal and vertical directions, and in the clockwise and counterclockwise directions around the optical axis, within a plane perpendicular to the optical axis of the optical system 103. The control unit 101 can control the position of the imaging element by controlling a driving member, such as a motor, included in the driving unit 102. The position of the imaging element is controlled to realize an optical image stabilization function and a high-resolution image generation function. The imaging unit 105 can also operate in a video capture mode, in which case it can capture multiple consecutive images as frames of a video.

[0020] The shake detection unit 106 detects shake (vibration) applied to the digital camera 100. Generally, a gyro sensor that detects the angular velocity of the shake is used as a sensor for detecting shake.

[0021] ROM 107 is a read-only nonvolatile memory that serves as a recording medium, and stores parameters and the like required for the operation of each block in addition to the operating programs of each block provided in digital camera 100. RAM 108 is a rewritable volatile memory that is used as a temporary storage area for data output during the operation of each block provided in digital camera 100.

[0022] The image processing unit 109 performs various image processing such as white balance adjustment, color interpolation, and filtering on the image output from the imaging unit 105 or the image recorded in the built-in memory 111. The image processing unit 109 also performs compression processing on the image captured by the imaging unit 105 in accordance with a standard such as JPEG.

[0023] Image processing unit 109 is configured as an application specific integrated circuit (ASIC) that is a collection of circuits that perform specific processing. Alternatively, control unit 101 may be configured to have some or all of the functions of image processing unit 109 by executing processing in accordance with a program read from ROM 107. If control unit 101 has all the functions of image processing unit 109, digital camera 100 does not need to have image processing unit 109 as hardware separate from control unit 101.

[0024] The display unit 110 is a liquid crystal display or an organic EL display, and displays images temporarily stored in the RAM 108, images stored in the internal memory 111, the setting screen of the digital camera 100, and the like.

[0025] The built-in memory 111 is a memory for recording images captured by the imaging unit 105, images processed by the image processing unit 109, and information on the focus position when capturing an image. A memory card or the like may be used instead of the built-in memory.

[0026] The operation unit 112 includes, for example, buttons, switches, keys, a mode dial, and the like attached to the digital camera 100. The operation unit 112 may also include a touch panel configured on the display unit 110. Commands from the user reach the control unit 101 via the operation unit 112.

[0027] Next, the high-resolution image generation process will be described with reference to FIG. 2. The high-resolution image generation process is executed by the image processing unit 109. Note that the high-resolution image generation process is executed, for example, when a shooting mode (super-resolution mode) for generating high-resolution images is set. When the super-resolution mode is set, the control unit 101 executes shooting of multiple frames of low-resolution images to be used for increasing the resolution. Here, a "low-resolution" image means an image with a relatively low resolution compared to the composite image (high-resolution image) that is ultimately generated. Therefore, the low-resolution image may be a still image with the highest resolution that can be captured by the imaging unit 105. The image processing unit 109 generates one frame of high-resolution image using the multiple frames of low-resolution images that have been shot.

[0028] 2, the image processing unit 109 captures multiple frames of low-resolution images and then generates a high-resolution image. However, the image processing unit 109 may temporarily stop processing after recording multiple frames of low-resolution images in the built-in memory 111, and then generate a high-resolution image based on the recorded multiple frames of low-resolution images at any timing thereafter.

[0029] FIG. 2 is a flowchart of the high-resolution image generation process. Unless otherwise specified, the process of each step in this flowchart is executed under the overall control of the control unit 101 according to a program. The process of this flowchart starts when an instruction to start shooting a still image is detected while the super-resolution mode is set. The instruction to start shooting a still image may be detected when a release switch included in the operation unit 112 is fully pressed, or when the waiting time of the self-timer expires. Note that, when a shooting preparation instruction is detected before the shooting start instruction is detected, the control unit 101 executes a process of determining shooting conditions (aperture value, shutter speed, shooting sensitivity) (AE process) and an autofocus detection process (AF process) for the optical system 103. Note that the shooting conditions may be values set by the user. The user may also manually set the focus distance of the optical system 103.

[0030] In S201, the control unit 101 determines the number of captured frames and the pixel shift amount of the low-resolution image that will be the basis for the high-resolution image. As described above, a "low-resolution" image means an image with a relatively low resolution compared to the composite image (high-resolution image) that will ultimately be generated. Therefore, the low-resolution image may be a still image with the highest resolution that can be captured by the imaging unit 105.

[0031] The number of captured frames may be a predetermined fixed value, or may be selectable by the user from multiple options. In the following description, the number of captured frames is assumed to be four. However, the number of captured frames is not limited to four and may be any value as long as it is two or more. The greater the number of captured frames, the higher the resolution of the high-resolution image can be, but even if the number of captured frames is only two, a high-resolution image can be generated from two low-resolution images.

[0032] The pixel shift amount for changing the shooting position (viewpoint) can be determined in various ways. For example, the pixel shift amount may be a fixed value or a value corresponding to the number of shooting frames. Here, the pixel shift amount is assumed to be a value smaller than the pixel pitch, and in the following description, the pixel shift amount is assumed to be 1 / 2 pixel (1 / 2 the pixel pitch). However, the pixel shift amount is not limited to a value smaller than the pixel pitch, and any value can be used as the pixel shift amount as long as it is a non-integer multiple of the pixel pitch. The pixel pitch is the distance between the centers of adjacent pixels. In the following description, the pixel pitch is assumed to be equal in the horizontal and vertical directions, but the pixel pitches in the horizontal and vertical directions may be different. Also, in the following description, the pixel shift amount is assumed to be 1 / 2 the pixel pitch in both the horizontal and vertical directions. However, the pixel shift amount in the horizontal and vertical directions may be different.

[0033] The method for determining at least one of the number of captured frames and the pixel shift amount can vary depending on the method used to synthesize a plurality of frames of low-resolution images to generate one frame of high-resolution image.

[0034] Here, it is assumed that digital camera 100 is fixed to a tripod or the like, and control unit 101 changes the shooting position of low-resolution images by moving image capture unit 105 (more precisely, the image sensor included in image capture unit 105). Also, it is assumed here that control unit 101 shoots multiple frames of low-resolution images using a continuous still image shooting function, but it is also possible to shoot multiple frames of video and use each frame of the video as a single low-resolution image. When shooting video, the frame rate can be determined taking into account the time required for moving image capture unit 105.

[0035] S202 to S205 are processes for capturing low-resolution images for the number of frames determined in S201. In S202, the control unit 101 controls the drive unit 102 to move the imaging unit 105 (image sensor) to achieve a pixel shift amount corresponding to the frame. It is assumed that the movement direction of the imaging unit 105 for each frame is predetermined according to the number of captured frames. Note that when capturing a reference frame (e.g., the first frame), the control unit 101 captures the image without moving the imaging unit 105 from the reference position. When capturing a total of four frames, for example, the control unit 101 moves the imaging unit 105 as shown in FIG. 3 when capturing each frame. FIG. 3 shows the position of the imaging unit 105 (image sensor) as viewed from the back side of the digital camera 100 (the side opposite the subject relative to the digital camera 100). As can be seen from FIG. 3, the movement amount of the imaging unit 105 relative to the reference position for each frame is as follows: 1st frame: No movement (reference position) 2nd frame: 1 / 2 pixel to the right from the reference position (1 / 2 of the pixel pitch) 3rd frame: 1 / 2 pixel downward from the reference position 4th frame: 1 / 2 pixel to the right and 1 / 2 pixel below the reference position

[0036] In this way, by moving the image capturing unit 105 to a different position for each frame, it is possible to obtain four frames of low-resolution images captured at different positions. When the movement of the image capturing unit 105 is completed, the process proceeds to S203.

[0037] In S203, the control unit 101 controls the optical system 103 and the imaging unit 105 to capture a still image of one frame (expose the imaging element).

[0038] In S204, the control unit 101 reads out an analog image signal from the imaging unit 105. The read out analog image signal is input to the image processing unit 109. The image processing unit 109 generates a digital image by A / D converting the analog image signal, and stores the digital image in the RAM 108 as a low-resolution image after performing various image processes such as development processing as necessary. At this point, the image processing unit 109 does not perform high-resolution processing.

[0039] In S205, the control unit 101 determines whether or not shooting of the number of frames determined in S201 has been completed. If shooting has been completed, the process proceeds to S206. If shooting has not been completed, the process returns to S202, and the processes of S202 to S204 are performed again to shoot the next frame.

[0040] In S206, the image processing unit 109 generates a high-resolution image by combining the multiple low-resolution images stored in the RAM 108 by executing a high-resolution process using pixel insertion, for example, as disclosed in Patent Document 1. Note that the method for generating a high-resolution image is not limited to the method disclosed in Patent Document 1, and other known methods may also be used.

[0041] In S207, the image processing unit 109 performs processing to generate a moving object area map.

[0042] FIG. 4 is a flowchart showing the details of the mobile object area map generation process (S207).

[0043] In S401, the image processing unit 109 performs a process to reduce high-frequency components of the high-resolution image in order to bring the frequency bands of the low-resolution image and the high-resolution image closer together. Specifically, the image processing unit 109 applies a filter (for example, a low-pass filter or a band-pass filter) to the high-resolution image to reduce high-frequency components generated by the process of generating the high-resolution image (the process of combining multiple low-resolution images). The specific frequency characteristics of the filter used here are not particularly limited, but for example, a filter having frequency characteristics determined by the following method can be used.

[0044] The first method is to store in advance in the ROM 107 (recording medium) information (filter information) indicating the frequency characteristics of a filter that reduces high-frequency components generated by the process of generating a high-resolution image (super-resolution processing). In this case, the image processing unit 109 reads the filter information from the ROM 107 and determines the frequency characteristics of the filter to be applied to the high-resolution image based on the read filter information.

[0045] The second method is to determine the frequency characteristics of the filter based on the positional relationship between the multiple shooting positions corresponding to the multiple low-resolution images to be synthesized. From the positional relationship between the multiple shooting positions (in other words, the number of frames and the pixel shift amount determined in S201), it is possible to predict to what extent frequency components will increase in the high-frequency band compared to the low-resolution image due to the super-resolution processing. Therefore, the image processing unit 109 uses a filter with frequency characteristics that reduce (or cut) frequency components in the high-frequency band that are predicted to increase.

[0046] The third method is to determine the frequency characteristics of the filter based on the positional relationship between the multiple shooting positions corresponding to the multiple low-resolution images to be combined, and the optical characteristics (optical data of the lens, such as the MTF curve and the number of resolving powers) of the optical system 103 used to capture the multiple low-resolution images. The limit of the frequency band of high-frequency components that increase due to super-resolution processing corresponds to the resolution performance of the lens of the optical system 103. Therefore, the high-frequency band in which frequency components are generated by super-resolution processing can be more accurately determined based on the positional relationship between the multiple shooting positions and the optical characteristics of the optical system 103. The image processing unit 109 uses a filter with frequency characteristics that reduce (or cut) frequency components in the high-frequency band thus determined.

[0047] The fourth method is to determine the frequency characteristics of a filter so as to pass a frequency band lower than the Nyquist frequency of a plurality of low-resolution images. The image processing unit 109 can determine the Nyquist frequency of the image captured by the imaging unit 105 from the number of pixels of the image sensor and the optical characteristics of the optical system 103 (optical data of the lens, such as the MTF curve and the number of lines of resolution). The image processing unit 109 uses a filter having frequency characteristics that pass a frequency band lower than the thus determined Nyquist frequency.

[0048] In S401, the image processing unit 109 may apply a filter that reduces high-frequency components generated by the super-resolution processing to one of the low-resolution images in addition to the high-resolution image, thereby making it possible to further approximate the frequency bands of the low-resolution image and the high-resolution image.

[0049] In S402, the image processing unit 109 performs processing to acquire the difference between the high-resolution image to which the filter has been applied in S401 and any one of the plurality of low-resolution images. In this case, the "one low-resolution image" refers to the low-resolution image to which the filter has been applied, if a filter has been applied to the low-resolution image in S401.

[0050] For example, the image processing unit 109 changes the magnification (enlarges or reduces) of at least one of the high-resolution image and the low-resolution image so as to make the sizes of the high-resolution image and the low-resolution image the same. Then, the image processing unit 109 calculates (obtains) the difference between the high-resolution image and the low-resolution image whose sizes have been made the same for each pixel, thereby generating a difference image.

[0051] It is not essential to make the high-resolution image and the low-resolution image the same size, and the image processing unit 109 may obtain the difference without performing a process of changing the size of at least one of the high-resolution image and the low-resolution image. In this case, for example, the image processing unit 109 can generate a difference image by identifying a position in the high-resolution image that corresponds to the position of each pixel in the low-resolution image, and calculating (obtaining) the difference between the corresponding positions for each pixel of the low-resolution image.

[0052] In S403, the image processing unit 109 generates a moving object region map based on the difference image acquired in S402. Regions with large differences in the difference image are likely to contain moving objects. Therefore, for example, the image processing unit 109 determines whether the absolute value of each pixel value in the difference image is equal to or greater than a threshold, and for pixels corresponding to absolute values equal to or greater than the threshold, sets "1" to the corresponding position in the moving object region map. Furthermore, for pixels corresponding to absolute values less than the threshold, the image processing unit 109 sets "0" to the corresponding position in the moving object region map. In this way, the moving object region map can be generated. Note that when generating the moving object region map, the image processing unit 109 may process the moving object region map so that it can be easily used in subsequent processing, for example by adjusting the threshold value or gain to take into account band errors that were not fully absorbed by the filter processing in S401.

[0053] Returning to FIG. 2, in S208, the image processing unit 109 corrects the moving object region of the high-resolution image generated in S206 based on the moving object region map generated in S403. For example, the image processing unit 109 identifies the moving object region in the high-resolution image based on the moving object region map. Then, the image processing unit 109 replaces the identified moving object region with a corresponding region from one of the multiple low-resolution images generated in S202 to S205. During the replacement, the image processing unit 109 enlarges the size of the corresponding region in the low-resolution image so as to match the size of the moving object region in the high-resolution image.

[0054] In addition, if the size of the identified moving object area is larger than a threshold value (for example, if the proportion of the moving object area in the high-resolution image exceeds a predetermined proportion), the image processing unit 109 may replace the high-resolution image itself with an image obtained by enlarging one of multiple low-resolution images to the size of the high-resolution image.

[0055] In S209, the image processing unit 109 saves the high-resolution image corrected (subjected to replacement processing) in S208. The save destination may be, for example, the built-in memory 111, a memory card, cloud storage, or the like. As mentioned above, in S208, the high-resolution image itself may be replaced with an image obtained by enlarging one of the multiple low-resolution images to the size of the high-resolution image. In this case, the image saved as the "high-resolution image" in S209 is "an image obtained by enlarging one of the multiple low-resolution images to the size of the high-resolution image."

[0056] As described above, according to this embodiment, the digital camera 100 acquires a plurality of captured images (a plurality of low-resolution images) captured at a plurality of capturing positions positioned such that the interval between adjacent capturing positions in a first direction (e.g., the horizontal direction) or a second direction (e.g., the vertical direction) perpendicular to the first direction is a non-integer multiple of the pixel pitch of the image sensor (e.g., ½ the pixel pitch in the example of FIG. 3 ) (S201 to S205). The digital camera 100 then performs a synthesis process on the plurality of captured images to generate a high-resolution image having a higher resolution than the plurality of captured images (S206). The digital camera 100 also applies a filter to the high-resolution image to reduce high-frequency components generated by the synthesis process (processing to generate a high-resolution image) (S401). The digital camera 100 then performs a process to acquire a difference between the high-resolution image to which the filter has been applied and one of the plurality of captured images (S402).

[0057] As described above, in this embodiment, a filter that reduces high-frequency components generated by the synthesis process (process for generating a high-resolution image) is applied to the high-resolution image. When this filter is applied to the high-resolution image, the possibility of a large difference occurring between the high-resolution image and the original captured image (low-resolution image) in an area where a moving object does not actually exist is reduced. Therefore, according to this embodiment, it is possible to obtain a highly accurate difference between the high-resolution image and the original captured image that can be used to identify a moving object area in the high-resolution image.

[0058] Although the above describes an example of a method for compensating an image of a moving object region in a high-resolution image, other methods may be used. For example, instead of replacing the image of the moving object region with an image of the corresponding region in a low-resolution image, a method may be used in which the image of the moving object region is interpolated from images of regions surrounding the moving object region in a high-resolution image.

[0059] Furthermore, the detection result of the moving object region may be used for processes other than compensating for the image of the moving object region, such as a process of notifying the user of the presence of the moving object region, a process of displaying the position of the moving object region, or a rating process of determining an evaluation value depending on whether or not there is a moving object region.

[0060] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0061] [summary] The above-described embodiment discloses at least the inventions shown in the following items, but is not limited to these inventions. [Item 1] a first acquisition means for acquiring a plurality of captured images captured at a plurality of capturing positions positioned such that an interval between adjacent capturing positions in a first direction or a second direction orthogonal to the first direction is a non-integer multiple of a pixel pitch of an image sensor; a generating means for generating a high-resolution image having a higher resolution than the plurality of captured images by combining the plurality of captured images; a filter means for applying a filter to the high-resolution image to reduce high-frequency components generated by the synthesis process; a second acquisition means for performing a process of acquiring a difference between the high-resolution image to which the filter has been applied and one of the plurality of captured images; An image processing device comprising: [Item 2] an identification means for identifying a moving object region in the high-resolution image based on the difference; a replacement means for replacing the moving object region of the high-resolution image with a corresponding region of any one of the plurality of captured images; 2. The image processing device according to item 1, further comprising: [Item 3] When the moving object region is larger than a threshold value, the replacement means replaces the high-resolution image with an image obtained by enlarging one of the plurality of captured images to a size of the high-resolution image. 3. The image processing device according to item 2, [Item 4] the process of acquiring the difference includes a process of scaling at least one of the high-resolution image to which the filter has been applied and the one captured image so as to make the sizes of the high-resolution image to which the filter has been applied and the one captured image uniform. 4. The image processing device according to any one of items 1 to 3, characterized in that: [Item 5] the filtering means reads filter information indicating the frequency characteristics of the filter from a recording medium, and applies the filter to the high-resolution image based on the filter information. 5. The image processing device according to any one of items 1 to 4, characterized in that: [Item 6] the filter has a frequency characteristic based on a positional relationship between the plurality of imaging positions. 5. The image processing device according to any one of items 1 to 4, characterized in that: [Item 7] the filter has frequency characteristics based on the positional relationship between the plurality of photographing positions and the optical characteristics of an optical system used to photograph the plurality of photographed images; 5. The image processing device according to any one of items 1 to 4, characterized in that: [Item 8] the filter has a frequency characteristic that passes a frequency band lower than the Nyquist frequency of the plurality of captured images. 5. The image processing device according to any one of items 1 to 4, characterized in that: [Item 9] the filtering means applies the filter to the one captured image; the second acquisition means acquires, as the difference, a difference between the high-resolution image to which the filter has been applied and the one captured image to which the filter has been applied. 9. The image processing device according to any one of items 1 to 8, characterized in that: [Item 10] The image processing device according to any one of items 1 to 9, the imaging element; An imaging device comprising: [Item 11] An image processing method executed by an image processing device, a first acquisition step of acquiring a plurality of captured images captured at a plurality of capturing positions positioned such that an interval between adjacent capturing positions in a first direction or a second direction orthogonal to the first direction is a non-integer multiple of a pixel pitch of an image sensor; a generating step of generating a high-resolution image having a higher resolution than the plurality of captured images by combining the plurality of captured images; a filtering step of applying a filter to the high-resolution image to reduce high-frequency components generated by the synthesis process; a second acquisition step of performing a process of acquiring a difference between the high-resolution image to which the filter has been applied and one of the plurality of captured images; An image processing method comprising: [Item 12] A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 9.

[0062] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0063] 100... digital camera, 101... control unit, 102... drive unit, 103... optical system, 104... communication unit, 105... imaging unit, 106... shake detection unit, 107... ROM, 108... RAM, 109... image processing unit, 110... display unit, 111... built-in memory, 112... operation unit

Claims

1. a first acquisition means for acquiring a plurality of captured images captured at a plurality of capturing positions positioned such that an interval between adjacent capturing positions in a first direction or a second direction orthogonal to the first direction is a non-integer multiple of a pixel pitch of an image sensor; a generating means for generating a high-resolution image having a higher resolution than the plurality of captured images by combining the plurality of captured images; a filter means for applying a filter to the high-resolution image to reduce high-frequency components generated by the synthesis process; a second acquisition means for performing a process of acquiring a difference between the high-resolution image to which the filter has been applied and one of the plurality of captured images; An image processing device comprising:

2. an identification means for identifying a moving object region in the high-resolution image based on the difference; a replacement means for replacing the moving object region of the high-resolution image with a corresponding region of any one of the plurality of captured images; 2. The image processing device according to claim 1, further comprising:

3. When the moving object region is larger than a threshold value, the replacement means replaces the high-resolution image with an image obtained by enlarging one of the plurality of captured images to a size of the high-resolution image.

3. The image processing device according to claim 2.

4. the process of acquiring the difference includes a process of scaling at least one of the high-resolution image to which the filter has been applied and the one captured image so as to make the sizes of the high-resolution image to which the filter has been applied and the one captured image uniform.

2. The image processing device according to claim 1, wherein:

5. the filtering means reads filter information indicating the frequency characteristics of the filter from a recording medium, and applies the filter to the high-resolution image based on the filter information.

2. The image processing device according to claim 1, wherein:

6. the filter has a frequency characteristic based on a positional relationship between the plurality of imaging positions.

2. The image processing device according to claim 1, wherein:

7. the filter has frequency characteristics based on the positional relationship between the plurality of photographing positions and the optical characteristics of an optical system used to photograph the plurality of photographed images; 2. The image processing device according to claim 1, wherein:

8. the filter has a frequency characteristic that passes a frequency band lower than the Nyquist frequency of the plurality of captured images.

2. The image processing device according to claim 1, wherein:

9. the filtering means applies the filter to the one captured image; the second acquisition means acquires, as the difference, a difference between the high-resolution image to which the filter has been applied and the one captured image to which the filter has been applied; 2. The image processing device according to claim 1, wherein:

10. An image processing device according to any one of claims 1 to 9; the imaging element; An imaging device comprising:

11. An image processing method executed by an image processing device, a first acquisition step of acquiring a plurality of captured images captured at a plurality of capturing positions positioned such that an interval between adjacent capturing positions in a first direction or a second direction orthogonal to the first direction is a non-integer multiple of a pixel pitch of an image sensor; a generating step of generating a high-resolution image having a higher resolution than the plurality of captured images by combining the plurality of captured images; a filtering step of applying a filter to the high-resolution image to reduce high-frequency components generated by the synthesis process; a second acquisition step of performing a process of acquiring a difference between the high-resolution image to which the filter has been applied and one of the plurality of captured images; An image processing method comprising:

12. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 9.

Images