Image processing device, image capturing device, image processing method, and program

Abstract

To provide a technique for acquiring information for highly accurately expressing difference between images photographed at photographing positions shifted from each other by a non-integer multiple of a pixel pitch of an imaging element.SOLUTION: An image processing device includes: first acquisition means configured to acquire two captured images captured at two capturing positions shifted from each other in a first direction by a non-integer multiple of a pixel pitch of an imaging element; first generation means configured to generate two enlarged images by enlarging the two captured images such that a pixel shift amount between the two enlarged images due to a shift between the two capturing positions is an integer in units of the number of pixels; and second acquisition means configured to acquire difference between the two enlarged images aligned based on the pixel shift amount between the two enlarged images.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program.

Background Art

[0002] In a digital camera capable of changing the position of an imaging element, a technique is known in which a plurality of images taken at imaging positions shifted by a distance smaller than the pixel pitch are combined to generate an image with a resolution exceeding the number of pixels of the imaging element (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When there is a moving subject (a moving object) that moves between a plurality of images to be combined, in the combined image, a region (a moving object region) in which components derived from an image including the moving object and components derived from an image not including the moving object are mixed occurs, so that the combined image becomes unnatural.

[0005] Generally, in a process of combining a plurality of images, it is known to detect a region of a moving subject by obtaining a difference between the images. In order to detect the region of the moving subject with high accuracy, it is necessary to obtain the difference between the images with high accuracy. However, when the shift in the imaging position between the images is a non-integer multiple of the pixel pitch (less than 1 multiple when smaller than the pixel pitch and a non-integer multiple greater than 1 when larger than the pixel pitch), even if the pixel positions of the images are moved, a shift smaller than the pixel pitch remains between the images, so it is difficult to obtain the difference between the images with high accuracy.

[0006] The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for acquiring information that accurately represents the difference between images captured at imaging positions shifted by a non-integer multiple of the pixel pitch of an imaging device.

Means for Solving the Problems

[0007] In order to solve the above problems, the present invention includes: a first acquisition means for acquiring two captured images captured at two imaging positions shifted in a first direction by a non-integer multiple of the pixel pitch of an imaging device; a first generation means for generating the two enlarged images by enlarging the two captured images so that the pixel shift amount between the two enlarged images derived from the shift between the two imaging positions becomes an integer in terms of the number of pixels; and a second acquisition means for acquiring the difference between the two enlarged images aligned based on the pixel shift amount between the two enlarged images. An image processing apparatus characterized by comprising the above is provided.

Effects of the Invention

[0008] According to the present invention, it becomes possible to acquire information that accurately represents the difference between images captured at imaging positions shifted by a non-integer multiple of the pixel pitch of an imaging device.

[0009] In addition, other features and advantages of the present invention will become more apparent from the accompanying drawings and the description in the following embodiments for carrying out the invention.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] [First Embodiment] FIG. 1 is a block diagram showing the configuration of a digital camera 100 (imaging device) including an image processing apparatus. The digital camera 100 can capture still images. Further, the digital camera 100 can move the position of the imaging element included in the imaging unit 105 in units smaller than the pixel pitch. Furthermore, the digital camera 100 can perform image processing such as enlargement processing, pixel shift processing, and synthesis processing on the captured and saved images or the images input from the outside.

[0013] In each of the embodiments including this embodiment, the description is made using a digital camera, but each embodiment is not limited to a digital camera. For example, instead of a digital camera, a portable device having an imaging element, a network camera capable of imaging an image, or the like may be used.

[0014] The control unit 101 includes a processor such as a CPU or an MPU, and controls each block of the digital camera 100 by reading and executing a program pre-stored in the ROM 107. For example, as will be described later, the control unit 101 issues commands to the imaging unit 105 regarding the start and end of imaging. Also, the control unit 101 issues an image processing command to the image processing unit 109 based on a program built into the ROM 107. Commands from the user are input into the digital camera 100 by the operation unit 112 and reach each block of the digital camera 100 through the control unit 101.

[0015] The drive unit 102 includes a motor or the like, and mechanically operates the optical system 103 under the command of the control unit 101. For example, the drive unit 102 adjusts the focal length of the optical system 103 by moving the position of the focus lens included in the optical system 103 based on a command from the control unit 101.

[0016] The optical system 103 includes a zoom lens, a focus lens, and a diaphragm. The diaphragm is a mechanism for adjusting the amount of transmitted light. By changing the position of the lens, the focusing position can be changed.

[0017] The communication unit 104 mainly performs information transmission between the control unit 101 and the optical system 103 under the command of the control unit 101.

[0018] The imaging unit 105 includes an imaging element including a photoelectric conversion element, and performs photoelectric conversion that converts incident light into an electrical signal. For example, as the imaging element of the imaging unit 105, a CCD sensor, a CMOS sensor, or the like can be applied. Also, the imaging element included in the imaging unit 105 is configured to be movable by a predetermined amount in the horizontal direction, the vertical direction, and the clockwise and counterclockwise directions around the optical axis within a plane orthogonal to the optical axis of the optical system 103. The control unit 101 can control the position of the imaging element by controlling a drive member such as a motor included in the drive unit 102. The position of the imaging element is controlled to realize an optical image blur correction function and a high-resolution image generation function.

[0019] The shake detection unit 106 detects the shake (vibration) applied to the digital camera 100. Generally, as a sensor for detecting shake, a gyro sensor that detects the angular velocity of the shake is used.

[0020] The ROM 107 is a read-only non-volatile memory as a recording medium, and stores operation programs for each block included in the digital camera 100, as well as parameters and the like necessary for the operation of each block. The RAM 108 is a rewritable volatile memory and is used as a temporary storage area for data output in the operation of each block included in the digital camera 100.

[0021] The image processing unit 109 performs various image processes such as white balance adjustment, color interpolation, and filtering on the image output from the imaging unit 105 or the image recorded in the built-in memory 111. In addition, the image processing unit 109 performs compression processing on the image captured by the imaging unit 105 in accordance with a standard such as JPEG.

[0022] The image processing unit 109 is composed of an integrated circuit (ASIC) that integrates circuits for performing specific processes. Alternatively, by executing processes according to the program read by the control unit 101 from the ROM 107, the control unit 101 may be configured to also have some or all of the functions of the image processing unit 109. When the control unit 101 has all the functions of the image processing unit 109, the digital camera 100 does not need to have the image processing unit 109 as hardware different from the control unit 101.

[0023] The display unit 110 is a liquid crystal display or an organic EL display, etc., and displays the image temporarily stored in the RAM 108, the image stored in the built-in memory 111, and the setting screen of the digital camera 100.

[0024] The built-in memory 111 is a memory for recording the image captured by the imaging unit 105, the image processed by the image processing unit 109, and information such as the focusing position at the time of image capture. Instead of the built-in memory, a memory card or the like may be used.

[0025] The operation unit 112 includes, for example, buttons, switches, keys, mode dials, etc. attached to the digital camera 100. Further, the operation unit 112 may include a touch panel configured on the display unit 110. A command from the user reaches the control unit 101 via the operation unit 112.

[0026] Next, with reference to FIG. 2, a high-resolution image generation process including detection of a moving subject region according to the first embodiment will be described. The high-resolution image generation process is executed by the image processing unit 109. Note that the high-resolution image generation process is executed, for example, when a shooting mode (super-resolution mode) for generating a high-resolution image is set. When the super-resolution mode is set, the control unit 101 executes shooting of a plurality of frames of low-resolution images used for high-resolution conversion. Here, the "low-resolution" image means an image having a relatively low resolution compared to the finally generated composite image (high-resolution image). Therefore, the low-resolution image may be a still image with the highest resolution that can be captured by the imaging unit 105. The image processing unit 109 generates one frame of high-resolution image using the plurality of frames of captured low-resolution images.

[0027] In the example of FIG. 2, the image processing unit 109 generates a high-resolution image continuously after shooting a plurality of frames of low-resolution images. However, the image processing unit 109 may record the plurality of frames of low-resolution images in the built-in memory 111, once stop the process, and then generate a high-resolution image based on the recorded plurality of frames of low-resolution images at an arbitrary timing.

[0028] FIG. 2 is a flowchart of a high-resolution image generation process including detection of a moving subject area according to the first embodiment. The processing of each step in this flowchart is executed under the overall control performed by the control unit 101 according to a program, unless otherwise specified. When a still image shooting start instruction is detected in a state where the super-resolution mode is set, the processing of this flowchart starts. The still image shooting start instruction may be detected in response to detecting that the release switch included in the operation unit 112 has been fully pressed, or may be detected in response to the expiration of the self-timer waiting time. Note that when a shooting preparation instruction is detected before the detection of the shooting start instruction, it is assumed that the control unit 101 has executed a determination process (AE process) of shooting conditions (aperture value, shutter speed, shooting sensitivity) and an automatic focus detection process (AF process) of the optical system 103. Note that the shooting conditions may be values set by the user. Also, the focal length of the optical system 103 may be manually set by the user.

[0029] In S201, the control unit 101 determines the number of shooting frames and the pixel shift amount of the low-resolution image that is the basis of the high-resolution image. As described above, the "low-resolution" image means an image with relatively low resolution compared to the composite image (high-resolution image) finally generated. Therefore, the low-resolution image may be a still image with the highest resolution that can be captured by the imaging unit 105.

[0030] The number of shooting frames may be a predetermined fixed value, or may be selectable by the user from a plurality of options. In the following description, it is assumed that the number of shooting frames is 4. However, the number of shooting frames is not limited to 4, and may be any value as long as it is 2 or more. The higher the number of shooting frames, the higher the resolution of the high-resolution image can be. However, even if the number of shooting frames is only 2, a high-resolution image can be generated from two low-resolution images.

[0031] The amount of pixel shift for varying the shooting position (viewpoint) can be determined by various methods. For example, the amount of pixel shift may be a fixed value or a value corresponding to the number of shooting frames. Here, it is assumed that the amount of pixel shift is smaller than the pixel pitch, and in the following description, the amount of pixel shift is 1 / 2 pixel (1 / 2 of the pixel pitch). However, the amount of pixel shift is not limited to a value smaller than the pixel pitch, and any value can be used as the amount of pixel shift as long as it is a non-integer multiple of the pixel pitch. The pixel pitch is the distance between the centers of adjacent pixels, and in the following description, it is assumed that the pixel pitches in the horizontal and vertical directions are equal, but the pixel pitches in the horizontal and vertical directions may be different. Also, in the following description, the amount of pixel shift in both the horizontal and vertical directions is set to pixel pitch / 2. However, the amount of pixel shift in the horizontal and vertical directions may be different.

[0032] Depending on how the low-resolution images of multiple frames are combined to generate a high-resolution image of one frame, the determination method of at least one of the number of shooting frames and the amount of pixel shift may change.

[0033] Here, the digital camera 100 is fixed to a tripod or the like, and the control unit 101 varies the shooting position of the low-resolution image by moving the imaging unit 105 (more precisely, the image sensor included in the imaging unit 105). Also, here, the control unit 101 shoots low-resolution images of multiple frames using the continuous shooting function for still images, but it is also possible to shoot a video of multiple frames and use each frame of the video as one low-resolution image. When shooting a video, the frame rate can be determined in consideration of the time required for the movement of the imaging unit 105.

[0034] S202 to S205 are processes for capturing low-resolution images of the number of frames determined in S201. In S202, the control unit 101 controls the drive unit 102 to move the imaging unit 105 (image sensor) in order to achieve a pixel shift amount corresponding to the frame. It is assumed that the moving direction of the imaging unit 105 in each frame is predetermined according to the number of captured frames. At the time of capturing the reference frame (for example, the first frame), the control unit 101 captures an image without moving the imaging unit 105 from the reference position. When a total of 4 frames are captured, as an example, when capturing each frame, the control unit 101 moves the imaging unit 105 as shown in FIG. 3. FIG. 3 shows the position of the imaging unit 105 (image sensor) as viewed from the back side of the digital camera 100 (the side opposite to the subject with respect to the digital camera 100). As can be understood from FIG. 3, the amount of movement of the imaging unit 105 with respect to the reference position in each frame is as follows. First frame: No movement (reference position) Second frame: 1 / 2 pixel to the right from the reference position (1 / 2 of the pixel pitch) Third frame: 1 / 2 pixel downward from the reference position Fourth frame: 1 / 2 pixel to the right and 1 / 2 pixel downward from the reference position

[0035] By moving the imaging unit 105 to different positions for each frame in this way, it becomes possible to obtain four low-resolution images with different shooting positions. When the movement of the imaging unit 105 is completed, the process proceeds to S203.

[0036] In S203, the control unit 101 controls the optical system 103 and the imaging unit 105 to execute still image shooting (exposure of the image sensor) for one frame.

[0037] In S204, the control unit 101 reads an analog image signal from the imaging unit 105. The read analog image signal is input to the image processing unit 109. The image processing unit 109 performs A / D conversion on the analog image signal and stores it in the RAM 108 as a low-resolution image. At this point, the image processing unit 109 does not execute the upscaling process.

[0038] In S205, the control unit 101 determines whether the shooting of the number of frames determined in S201 has been completed. If the shooting has been completed, the process proceeds to S206. If the shooting has not been completed, the process returns to S202, and the processes of S202 to S204 are performed again to shoot the next frame.

[0039] S206 to S210 are processes for aligning the positions between the low-resolution images and obtaining the differences. In the present embodiment, one of the plurality of low-resolution images stored in the RAM 108 is used as a reference image, the remaining low-resolution images are used as comparison images, and the differences between the reference image and each comparison image are obtained. The image processing unit 109 repeats the loop of S206 to S210 for the number of comparison images, and performs alignment with the reference image and acquisition of the difference for each comparison image.

[0040] FIG. 4 is a schematic diagram for explaining the alignment of the reference image and the comparison image by the processes of S206 to S208. In the example of FIG. 4, the reference image is the low-resolution image of the first frame, and the comparison image is the low-resolution image of the second frame. As shown in FIG. 4, the image processing unit 109 performs demosaicing processing and enlargement processing (S206 to S207) on the reference image. Further, the image processing unit 109 performs demosaicing processing, enlargement processing, and pixel shift (S206 to S208) on the comparison image. Although only one comparison image is shown in FIG. 4, the same processing is performed on the other comparison images (however, the direction of the pixel shift appropriately changes according to the relationship between the position of the imaging unit 105 at the time of shooting the comparison image and the position of the imaging unit 105 at the time of shooting the reference image).

[0041] Referring to FIG. 3 again, in S206, the image processing unit 109 executes demosaicing processing on the reference image and one of the plurality of comparison images. When the imaging element of the imaging unit 105 has a color filter of a primary color Bayer array, the image processing unit 109 generates an image (RGB image) in which each pixel has RGB components as shown in FIG. 4. Note that when the imaging element of the imaging unit 105 is a monochromatic imaging element not equipped with a color filter, the demosaicing processing does not need to be performed.

[0042] In S207, the image processing unit 109 performs an enlargement process on each color component of the RGB image generated in S206. The enlargement ratio is set so that the RGB image has the resolution of the high-resolution image (so that the resolutions of the enlarged reference image and comparison image are equal to the resolution of the high-resolution image). For example, as shown in FIG. 3, when four low-resolution images are captured by pixel shifting (movement of the imaging unit 105) in units of 1 / 2 pixels, as shown in FIG. 4, an enlargement ratio of 2 times is set in both the vertical and horizontal directions of the RGB image.

[0043] Note that the enlargement ratio used here is not limited to the value that makes the RGB image have the resolution of the high-resolution image. There is a pixel shift amount derived from the shift between the shooting positions between the enlarged reference image and the enlarged comparison image, but the image processing unit 109 may enlarge the reference image and the comparison image so that this pixel shift amount becomes an integer in pixel units. By performing such an enlargement process, in S208 described below, it becomes possible to accurately align the enlarged reference image and the enlarged comparison image.

[0044] In S208, the image processing unit 109 shifts the positions of the pixels of the comparison image enlarged in S207 so that the spatial phases (viewpoints) of the reference image and the comparison image enlarged in S207 match. For example, as shown in FIG. 3, when four low-resolution images are captured by pixel shifting in units of 1 / 2 pixels, the enlarged comparison image is shifted by 1 pixel in at least one of the vertical and horizontal directions with respect to the enlarged reference image. In the example of FIG. 4, as can be understood from the positional relationship between pixel 401 and pixel 402 corresponding to pixel 401, the enlarged comparison image is shifted 1 pixel to the left with respect to the enlarged reference image. Therefore, the image processing unit 109 shifts the enlarged comparison image 1 pixel to the right to align the spatial phase (viewpoint) of the enlarged comparison image with the enlarged reference image.

[0045] In S209, the image processing unit 109 generates a difference image by calculating (acquiring) the difference between the reference image obtained by executing the processes of S206 to S207 and the comparison image obtained by executing the processes of S206 to S208 for each pixel.

[0046] In S210, the control unit 101 determines whether or not the processing of all the comparison images has been completed. If the processing of all the comparison images has been completed, the process proceeds to S211. Otherwise, the process returns to S206, and the processes of S206 to S209 are performed on the next comparison image. Note that in the second and subsequent executions of S206 to S207, the processing on the reference image can be omitted. In this case, the image processing unit 109 reuses the reference image on which the demosaicing process and the enlargement process have been performed in the first execution of S206 to S207.

[0047] In S211, the image processing unit 109 generates a moving object region map by detecting a change region (a region with a large difference, that is, a region where the change in the image is large) based on the plurality of difference images generated in S209. For example, the image processing unit 109 generates a change region map of the R component based on the absolute value of the pixel value of each pixel in the R component of one difference image. For example, in the change region map of the R component, the image processing unit 109 sets "1" for the pixels corresponding to the absolute values equal to or greater than the threshold value, and sets "0" for the pixels corresponding to the absolute values less than the threshold value. In this case, the region where "1" is set in the change region map of the R component corresponds to the change region of the R component. Similarly, the image processing unit 109 also generates change region maps of the G component and the B component. Similarly, the image processing unit 109 generates change region maps of each color component for each of the remaining difference images as well. Then, the image processing unit 109 generates a moving object region map by adding the change region maps of each color component of each difference image. In this case, the region where a value of "1" or more is set in the moving object region map corresponds to the moving object region (the region where there is a movement of the subject between the reference image and at least one comparison image).

[0048] In the above description, the difference between the reference image and the comparison image is obtained for each pixel. However, a configuration may be adopted in which the reference image and the comparison image are each divided into blocks, and the difference is obtained for each block. Further, in the above description, the difference is obtained for each color component of the R component, the G component, and the B component. However, a configuration may be adopted in which the difference is obtained using the luminance component instead of the color component.

[0049] In S212, the image processing unit 109 generates a high-resolution image by synthesizing a plurality of low-resolution images stored in the RAM 108 by executing a resolution enhancement process using pixel insertion as disclosed in, for example, Patent Document 1. Note that the method for generating the high-resolution image is not limited to the method disclosed in Patent Document 1, and other known methods may be used.

[0050] In S213, the image processing unit 109 identifies the moving object region in the high-resolution image based on the moving object region map generated in S211. Then, the image processing unit 109 generates an image in which the moving object region has a natural depiction by replacing the moving object region with another image. For example, the image processing unit 109 uses the moving object region map as a mask, and cuts out the region corresponding to the moving object region from the high-resolution image. Then, the image processing unit 109 replaces the region cut out in the high-resolution image with a corresponding region of one of the plurality of low-resolution images.

[0051] As described above, according to the first embodiment, the digital camera 100 acquires two captured images (two low-resolution images) captured at two imaging positions shifted by a non-integer multiple of the pixel pitch of the imaging element (in the above example, 1 / 2 of the pixel pitch) in the first direction (for example, the horizontal direction) (201 to S205). Then, the digital camera 100 generates two enlarged images (the enlarged reference image and the comparison image) by enlarging the two captured images so that the pixel shift amount between the two enlarged images resulting from the shift between the two imaging positions becomes an integer in terms of the number of pixels (S207). Thereafter, the digital camera 100 obtains the difference between the two enlarged images that are aligned based on the pixel shift amount between the two enlarged images (S208 to S209).

[0052] In this way, in the present embodiment, the two captured images are enlarged such that the pixel shift amount between the two enlarged images becomes an integer in terms of the number of pixels. Therefore, it is possible to accurately align the two enlarged images and acquire the difference between the two enlarged images with high precision. The difference thus acquired represents the difference between the two captured images corresponding to the two enlarged images. Therefore, according to the present embodiment, it is possible to acquire information that accurately represents the difference between the images captured at the shooting positions shifted by a non-integer multiple of the pixel pitch of the image sensor.

[0053] [Second Embodiment] In the first embodiment, a configuration in which the replacement of the moving object region is performed after generating a high-resolution image using a plurality of captured low-resolution images has been described. In contrast, in the second embodiment, a configuration in which two or more low-resolution images used for generating a high-resolution image are selected from the plurality of captured low-resolution images, and then a high-resolution image is generated using the selected two or more low-resolution images will be described. In the present embodiment, the basic configuration of the digital camera 100 is the same as that of the first embodiment. Hereinafter, the configuration different from the first embodiment will be mainly described.

[0054] FIG. 5 is a flowchart of a high-resolution image generation process including region detection of a moving subject according to the second embodiment. The processing of each step of this flowchart is executed under the overall control performed by the control unit 101 according to a program unless otherwise specified.

[0055] The processing of S501 to S510 is the same as the processing of S201 to S210 in FIG. 2 described in the first embodiment. However, in the first embodiment, the number of shooting frames is two or more, while in the second embodiment, the number of shooting frames is three or more.

[0056] In S511, the image processing unit 109 selects a low-resolution image to be used for generating a high-resolution image based on the plurality of difference images generated in S509. The selection here is performed for each difference image. Specifically, the image processing unit 109 generates a change region map for each color component of a target difference image by the same method as S211 in FIG. 2, with one difference image as the target difference image. Then, the image processing unit 109 generates a difference region map by adding the change region maps of the respective color components. In the difference region map, the region where a value of "1" or more is set corresponds to the difference region (the region where there has been a movement of the subject between the reference image and the comparison image corresponding to the target difference image). The image processing unit 109 determines whether the size of the difference region is smaller than a threshold value. When the size of the difference region is smaller than the threshold value, the image processing unit 109 selects the low-resolution image corresponding to the comparison image corresponding to the target difference image as one of the low-resolution images to be used for generating the high-resolution image. Similarly, for each of the remaining difference images, the image processing unit 109 generates a difference region map and, when the size of the difference region is smaller than the threshold value, selects the corresponding low-resolution image as one of the low-resolution images to be used for generating the high-resolution image. Note that the low-resolution image corresponding to the reference image is selected as one of the low-resolution images to be used for generating the high-resolution image regardless of the content of each difference image.

[0057] In S512, the image processing unit 109 synthesizes two or more low-resolution images selected in S511 to generate a high-resolution image.

[0058] As described above, according to the second embodiment, the digital camera 100 acquires three or more captured images captured at three or more shooting positions (S501 to S505). Among the three or more shooting positions, a specific shooting position is shifted by a non-integer multiple of the pixel pitch in at least one of a first direction (for example, the horizontal direction) and a second direction (for example, the vertical direction) orthogonal to the first direction with respect to each of the two or more shooting positions other than the specific shooting position. The digital camera 100 generates three or more enlarged images (an enlarged reference image and two or more enlarged comparison images) by enlarging the three or more captured images so that the pixel shift amount between the three or more enlarged images resulting from the shift between the three or more shooting positions becomes an integer in terms of the number of pixels (S507). The digital camera 100 acquires, for each of the two or more comparison images, the difference between the aligned comparison image and the reference image based on the pixel shift amount between the comparison image and the reference image (S508 to S509). Then, for each of the two or more comparison images, when the difference is smaller than a predetermined reference (in the above example, when the size of the difference region is smaller than the threshold), the digital camera 100 selects the captured image (low-resolution image) corresponding to the comparison image as the captured image for synthesis. Then, the digital camera 100 generates a high-resolution image by synthesizing the captured image (low-resolution image) corresponding to the reference image and one or more captured images (low-resolution images) selected as the captured images for synthesis among the three or more captured images.

[0059] In this way, according to the second embodiment, when some low-resolution images are not selected in S511, the number of low-resolution images used for generating the high-resolution image may decrease. In this case, the resolution of the high-resolution image decreases. However, in the second embodiment, since the replacement of the moving object region does not occur, unlike the first embodiment, it is possible to align the resolution of the moving object region and the resolution of the non-moving object region.

[0060] In addition, in S511, if the difference for all of two or more comparison images is not smaller than a predetermined standard, the digital camera 100 may select, as a captured image for synthesis, the captured image corresponding to the comparison image corresponding to the smallest difference among the two or more comparison images. Thereby, even when the difference for all of two or more comparison images is not smaller than a predetermined standard, it becomes possible to generate a high-resolution image.

[0061] [Third Embodiment] In the third embodiment, a configuration for regenerating a captured image by performing photographing again at the same photographing position when the difference between the reference image and the comparison image is larger than a predetermined standard will be described.

[0062] FIG. 6 is a flowchart of a high-resolution image generation process including area detection of a moving subject according to the third embodiment. The processing of each step of this flowchart is executed under the overall control performed by the control unit 101 according to a program unless otherwise specified.

[0063] The processing of S601 to S604 is the same as the processing of S201 to S204 in FIG. 2 described in the first embodiment.

[0064] In S605, the control unit 101 determines whether the current photographing is the first-frame photographing. If the current photographing is the first-frame photographing, the process returns to S602, and the processes of S602 to S604 are performed again to photograph the next frame. If the current photographing is not the first-frame photographing, the process proceeds to S606.

[0065] The processing of S606 to S609 is the same as the processing of S206 to S209 in FIG. 2 described in the first embodiment. However, in the third embodiment, the comparison image is the most recently captured image, and the reference image is any one of the previously captured images. For example, when the processing of S606 to S609 is first executed, the comparison image is the second-frame low-resolution image, and the reference image is the first-frame low-resolution image.

[0066] In S610, the control unit 101 generates a differential region map based on the differential image in the same manner as S511 in FIG. 5, and determines whether the size of the differential region is larger than a threshold value. If the size of the differential region is not larger than the threshold value, the process proceeds to S611. If the size of the differential region is larger than the threshold value, the control unit 101 discards the captured image corresponding to the comparison image. Then, the control unit 101 returns the process to S603 to control so as to perform imaging again at the same imaging position as the discarded captured image and regenerate the captured image corresponding to this imaging position.

[0067] In S611, the control unit 101 determines whether the imaging of the number of frames determined in S601 has been completed. If the imaging has been completed, the process proceeds to S612. If the imaging has not been completed, the process returns to S602, and the processes of S602 to S610 are performed again to capture the next frame.

[0068] In S612, the control unit 101 generates a high-resolution image by the same process as S212 in FIG. 2.

[0069] As described above, according to the third embodiment, when the difference between the reference image and the comparison image is larger than a predetermined reference (in the above example, when the size of the differential region is larger than the threshold value), the digital camera 100 regenerates the captured image by performing imaging again at the same imaging position. Here, in the third embodiment, different from the first embodiment, the process of replacing the moving object region in the high-resolution image with a corresponding region of one of the plurality of low-resolution images is not performed. Therefore, according to the present embodiment, it is possible to suppress a decrease in resolution associated with the replacement of the moving object region.

[0070] Note that the above three embodiments describe different examples of methods for compensating the image of the moving object region in the high-resolution image, but other methods may also be used. For example, instead of the method of replacing with the image of the corresponding region of the low-resolution image in the first embodiment, a method of interpolating the image of the moving object region from the images of the regions around the moving object region in the high-resolution image may be used.

[0071] Further, the detection result of the moving object region may be used for processes other than compensating the image of the moving object region. For example, it may be used for a process of notifying the user that a moving object region exists, a process of performing a display indicating the position of the moving object region, a rating process of determining an evaluation value according to the presence or absence of the moving object region, and the like.

[0072] [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0073] [Summary] The above-described embodiments disclose the inventions shown in at least the following items, but are not limited to these inventions. [Item 1] A first acquisition means for acquiring two captured images captured at two imaging positions shifted in a first direction by a non-integer multiple of the pixel pitch of the imaging device; A first generation means for generating the two enlarged images by enlarging the two captured images so that the pixel shift amount between the two enlarged images derived from the shift between the two imaging positions becomes an integer in pixel units; A second acquisition means for acquiring a difference between the two enlarged images aligned based on the pixel shift amount between the two enlarged images; An image processing apparatus comprising the above. [Item 2] A second generation means for generating a high-resolution image having a higher resolution than the two captured images by synthesizing the two captured images; An identification means for identifying a moving object region in the high-resolution image based on the difference; A replacement means for replacing the moving object region in the high-resolution image with a corresponding region in any one of the two captured images; The image processing apparatus according to item 1, further comprising [Item 3] The resolution of the two enlarged images is equal to the resolution of the high-resolution image The image processing apparatus according to item 2, characterized in that [Item 4] The deviation between the two imaging positions is smaller than the pixel pitch The image processing apparatus according to any one of items 1 to 3, characterized in that [Item 5] The first acquisition means acquires three or more captured images captured at three or more imaging positions, Among the three or more imaging positions, a specific imaging position is shifted by a non-integer multiple of the pixel pitch in at least one of the first direction and a second direction orthogonal to the first direction with respect to each of the two or more imaging positions other than the specific imaging position, The first generation means generates the three or more enlarged images by enlarging the three or more captured images so that the pixel shift amount between the three or more enlarged images resulting from the shift between the three or more imaging positions becomes an integer in terms of the number of pixels, The second acquisition means uses any one of the three or more enlarged images as a reference image and the remaining two or more enlarged images as two or more comparison images, and for each of the two or more comparison images, obtains a difference between the comparison image and the reference image that are aligned based on the pixel shift amount between the comparison image and the reference image, The image processing apparatus Selection means for selecting, for each of the two or more comparison images, the captured image corresponding to the comparison image as a captured image for synthesis when the difference is smaller than a predetermined reference, Second generation means for generating a high-resolution image having a higher resolution than the three or more captured images by synthesizing the captured image corresponding to the reference image and one or more captured images selected as the captured images for synthesis among the three or more captured images, The image processing apparatus according to item 1, further comprising [Item 6] If the difference is not smaller than the predetermined standard for all of the two or more comparison images, the selection means selects, as the captured image for synthesis, the captured image corresponding to the comparison image corresponding to the minimum difference among the two or more comparison images. The image processing apparatus according to item 5, characterized in that. [Item 7] The image processing apparatus according to item 1, The image sensor, Control means for controlling to generate the two captured images by performing imaging at the two imaging positions using the image sensor, wherein when the difference between the two enlarged images generated from the two captured images is larger than a predetermined standard, the control means controls to re-image at the first imaging position among the two imaging positions so as to regenerate the captured image corresponding to the first imaging position among the two captured images. Second generation means for generating a high-resolution image having a higher resolution than the two captured images by synthesizing the two captured images. An imaging apparatus characterized by comprising. [Item 8] The image processing apparatus according to any one of items 1 to 6, The image sensor, An imaging apparatus characterized by comprising. [Item 9] An image processing method executed by an image processing apparatus, A first acquisition step of acquiring two captured images captured at two imaging positions shifted by a non-integer multiple of the pixel pitch of the image sensor in a first direction, A first generation step of generating the two enlarged images by enlarging the two captured images so that the pixel shift amount between the two enlarged images resulting from the shift between the two imaging positions becomes an integer in terms of the number of pixels, A second acquisition step of acquiring the difference between the two enlarged images aligned based on the pixel shift amount between the two enlarged images. An image processing method characterized by comprising. [Item 10] A program for causing a computer to function as each means of the image processing apparatus according to any one of Items 1 to 6.

[0074] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

Explanation of Signs

[0075] 100…Digital camera, 101…Control unit, 102…Drive unit, 103…Optical system, 104…Communication unit, 105…Imaging unit, 106…Shake detection unit, 107…ROM, 108…RAM, 109…Image processing unit, 110…Display unit, 111…Built-in memory, 112…Operation unit

Claims

1. First acquisition means for acquiring two captured images captured at two imaging positions shifted in a first direction by a non-integer multiple of the pixel pitch of the imaging element; First generation means for generating the two enlarged images by enlarging the two captured images so that the pixel shift amount between the two enlarged images resulting from the shift between the two imaging positions becomes an integer in terms of the number of pixels; Second acquisition means for acquiring the difference between the two enlarged images aligned based on the pixel shift amount between the two enlarged images; An image processing apparatus comprising the above.

2. Second generation means for generating a high-resolution image having a higher resolution than the two captured images by synthesizing the two captured images; Discrimination means for discriminating a moving object region in the high-resolution image based on the difference; Replacement means for replacing the moving object region of the high-resolution image with a corresponding region of either of the two captured images; The image processing apparatus according to claim 1, further comprising the above.

3. The resolution of the two enlarged images is equal to the resolution of the high-resolution image The image processing apparatus according to claim 2, characterized by the above.

4. The shift between the two imaging positions is smaller than the pixel pitch The image processing apparatus according to claim 1, characterized by the above.

5. The first acquisition means acquires three or more captured images captured at three or more imaging positions, Among the three or more imaging positions, a specific imaging position is shifted by a non-integer multiple of the pixel pitch in at least one of the first direction and a second direction orthogonal to the first direction with respect to each of the other two or more imaging positions, The first generation means generates the three or more enlarged images by enlarging the three or more captured images so that the pixel shift amount between the three or more enlarged images resulting from the shift between the three or more imaging positions becomes an integer in terms of the number of pixels, The second acquisition means uses any one of the three or more enlarged images as a reference image and the remaining two or more enlarged images as two or more comparison images, and for each of the two or more comparison images, acquires the difference between the comparison image and the reference image aligned based on the pixel shift amount between the comparison image and the reference image, The image processing apparatus is Selection means for selecting, for each of the two or more comparison images, a captured image corresponding to the comparison image as a captured image for synthesis when the difference is smaller than a predetermined criterion; Second generation means for generating a high-resolution image having a higher resolution than the three or more captured images by synthesizing the captured image corresponding to the reference image and one or more captured images selected as the captured images for synthesis among the three or more captured images; The image processing apparatus according to claim 1, further comprising the above.

6. When the difference is not smaller than the predetermined criterion for all of the two or more comparison images, the selection means selects, as the captured image for synthesis, a captured image corresponding to the comparison image corresponding to the minimum difference among the two or more comparison images The image processing apparatus according to claim 5, characterized in that.

7. The image processing apparatus according to claim 1, The image sensor, Control means for controlling to perform imaging at the two imaging positions using the image sensor to generate the two captured images, and when the difference between the two enlarged images generated from the two captured images is larger than a predetermined criterion, the control means re-performs imaging at a first imaging position among the two imaging positions, thereby controlling to re-generate the captured image corresponding to the first imaging position among the two captured images; Control means, Second generation means for generating a high-resolution image having a higher resolution than the two captured images by synthesizing the two captured images; An imaging apparatus comprising the above.

8. The image processing apparatus according to any one of claims 1 to 6, The image sensor, An imaging apparatus comprising the above.

9. An image processing method executed by an image processing apparatus, A first acquisition step of acquiring two captured images captured at two imaging positions shifted by a non-integer multiple of the pixel pitch of an image sensor in a first direction; A first generation step of generating the two enlarged images by enlarging the two captured images so that the amount of pixel shift between the two enlarged images resulting from the shift between the two imaging positions becomes an integer in terms of the number of pixels; A second acquisition step of acquiring a difference between the two enlarged images aligned based on the amount of pixel shift between the two enlarged images; An image processing method comprising the above.

10. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 6.

Images