Image processing device, method, and program

Abstract

The present technology relates to an image processing device and method, and a program which can enhance the image quality in sensor shift capture. When capturing an image while shifting a pixel phase of an image sensor having a two-dimensional pixel array, the image processing device adds, for each pixel phase, a plurality of frames captured at each of the pixel phases, generates additional frames for each of the pixel phases, and synthesizes the additional frames of each of the pixel phases. The present technology may be applied to a sensor shift capture system.

Description

[Technical field]

[0001] The present technology relates to an image processing device, method, and program, and in particular to an image processing device, method, and program that are capable of improving image quality in sensor shift imaging. [Background technology]

[0002] In recent years, cameras compatible with sensor shift imaging have appeared on the market. Sensor shift imaging is an imaging method in which images are captured while shifting the image sensor of a camera fixed to a tripod or the like, and the captured images are then synthesized. By performing sensor shift imaging, it is possible to obtain a resolution at least equivalent to that of a three-chip sensor, even with a single-chip sensor (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-171511 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in environments with many external disturbances, such as outdoors, the ground is often subject to minute vibrations, making it difficult to precisely fix the camera even when it is mounted on a tripod. As a result, when sensor shift imaging is performed outdoors, camera vibration can cause misalignment between images or blurring of the images, resulting in reduced image quality.

[0005] In addition, it is difficult from a system perspective to use optical image stabilization in conjunction with sensor shift imaging. Moreover, it is difficult to optically correct the minute vibrations of the camera that occur when sensor shift imaging is performed outdoors. Therefore, with conventional optical image stabilization, it is difficult to appropriately correct optical shake when sensor shift imaging is performed outdoors.

[0006] The present technology has been made in consideration of such circumstances, and makes it possible to improve image quality in sensor shift imaging. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present technology includes an adder unit that, when capturing images while shifting the pixel phase of an image sensor having a two-dimensional pixel array, adds up multiple frames captured at each pixel phase for each pixel phase to generate a summed frame for each pixel phase, and a synthesizer unit that synthesizes the summed frames for each of the pixel phases.

[0008] In one aspect of the present technology, when capturing an image while shifting the pixel phase of an image sensor having a two-dimensional pixel array, multiple frames captured at each pixel phase are added together for each pixel phase, an added frame is generated for each pixel phase, and the added frames for each pixel phase are synthesized. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a pixel array of an image sensor. [Diagram 2] 1A and 1B are diagrams illustrating the principle of sensor shift imaging. [Diagram 3] FIG. 1 is a diagram showing an example of conventional sensor shift imaging. [Figure 4] 1A and 1B are diagrams illustrating an example of sensor shift imaging according to the present technology. [Diagram 5] FIG. 1 is a diagram showing an overview of a processing flow of the present technology. [Figure 6] 1 is a block diagram showing a configuration of an embodiment of a sensor shift imaging system to which the present technology is applied. [Figure 7] 11 is a flowchart illustrating a sensor shift imaging process. [Figure 8] FIG. 2 is a diagram showing pixels acquired in normal imaging and high-speed imaging. [Figure 9] FIG. 11 is a diagram showing the relationship between sensitivity and resolution. [Figure 10] FIG. 11 is a diagram showing the size of the allowable range of positional deviation. [Figure 11] 11A and 11B are diagrams illustrating the relationship between the allowable range of positional deviation and image quality and processing speed. [Figure 12] FIG. 13 is a diagram illustrating a fixed control of a threshold value. [Figure 13] FIG. 13 is a diagram illustrating dynamic control of a threshold value. [Figure 14] FIG. 13 is a diagram showing a case where an upper limit is set for the assumed imaging time. [Figure 15] FIG. 2 is a block diagram showing an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Overview of this technology 2. System Configuration and Operation 3. Details of position deviation judgment 4.Other

[0011] 1. Overview of this technology (Example of pixel arrangement of an image sensor) FIG. 1 is a diagram showing an example of a pixel array of an image sensor.

[0012] In CCD (Charge Coupled Device) image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors, a pixel array in which multiple pixels that detect blue, green, and red light, respectively, are arranged on a plane using primary color filters is widely used.

[0013] For example, the image sensor section 1 shown in FIG. 1 uses a pixel array (Bayer array in the case of FIG. 1) in which a predetermined array pattern consisting of a plurality of pixels 2b, 2g, and 2r that detect blue, green, and red light, respectively, is repeated.

[0014] That is, within the image sensor unit 1, a plurality of pixels 2 corresponding to each color are arranged so that an arrangement pattern consisting of one blue pixel 2b, two green pixels 2g, and one red pixel 2r is repeated.

[0015] In recent years, sensor shift imaging has been proposed, which applies an anti-shake mechanism provided in an imaging device. Sensor shift imaging is an imaging method in which the image sensor unit 1 is shifted by one pixel at a time in a specified direction to acquire multiple images, and these acquired multiple images are then synthesized to generate a high-resolution image.

[0016] Here, shifting the image sensor unit 1 by one pixel means shifting the relative position of the image sensor unit 1 with respect to the subject image (or the optical axis of the lens) by one pixel.

[0017] In the following description, the "pixel phase" refers to the position of the image sensor unit 1 shifted relative to the subject image, expressed as a position within one period of the pixel array, assuming that the above-mentioned array pattern is one period of the pixel array. That is, in the present technology, the "pixel phase" relative to the subject image is shifted by shifting the image sensor unit 1, and imaging is performed.

[0018] Note that imaging can be performed by shifting the "pixel phase" relative to the subject image in the same manner by shifting the lens or the imaging device itself relative to the subject image, instead of shifting the image sensor unit 1. Therefore, the present technology is not limited to cases where the pixel phase is shifted by shifting the image sensor unit, but is also applicable to cases where the "pixel phase" is shifted as a result by shifting the lens or the imaging device itself.

[0019] (Principle of sensor shift imaging) FIG. 2 is a diagram illustrating the principle of sensor shift imaging.

[0020] In sensor shift imaging, the imaging device is fixed to a tripod or the like, and the image sensor unit 1 is shifted by one pixel at a time in the order of shot1 to shot4 as shown in Fig. 2 to take continuous shots, and the obtained images (shown in the foreground of Fig. 2) are synthesized. This makes it possible to obtain blue, green, and red light information from all valid pixels 2 (shown on the right side of Fig. 2).

[0021] In other words, sensor shift imaging ensures that there is no loss of light information of each color in all valid pixels 2 on the image sensor unit 1. Therefore, in sensor shift imaging, a high-resolution image can be generated by directly combining light information of each color without performing an interpolation process that interpolates light information of a missing color using information from surrounding pixels 2. As a result, sensor shift imaging minimizes the occurrence of color moiré (false colors) because no interpolation process is performed, making it possible to achieve higher definition and more faithful texture depiction.

[0022] (Example of conventional sensor shift imaging) FIG. 3 is a diagram showing an example of conventional sensor shift imaging.

[0023] In Fig. 3, the subscript numbers #0, #1, #2, and #3 of the frames of each pixel phase indicate the imaging order. Also, in Fig. 3, the arrow indicates the sensor shift direction, which is the shift direction of the image sensor unit 1.

[0024] In FIG. 3, the position of the pixel 2r of the image sensor unit 1 is indicated by hatching.

[0025] To explain in the order of imaging, first, frame #0 is imaged with one exposure (shutter) at the normal shutter speed when the image sensor unit 1 is in phase A. The normal shutter speed is a shutter speed that provides an appropriate amount of exposure in one exposure, and hereinafter, imaging an image with one exposure at the normal shutter speed is referred to as normal imaging. Frame #1 is imaged at the normal shutter speed in phase B, where the image sensor unit 1 is shifted one pixel to the right from the phase A state.

[0026] Frame #2 is captured at a normal shutter speed in phase C, where the image sensor unit 1 is shifted downward by one pixel from phase B. Furthermore, frame #3 is captured at a normal shutter speed in phase D, where the image sensor unit 1 is shifted left by one pixel from phase C.

[0027] It should be noted that by shifting the image sensor unit 1 upward from the phase D state by one pixel, the image sensor unit 1 returns to the phase A state.

[0028] Then, sensor shift synthesis is performed to synthesize frames of all pixel phases to generate one frame.

[0029] However, in an environment with many external disturbances such as outdoors, the ground often vibrates slightly, making it difficult to precisely fix the imaging device even when using a fixed tripod. Therefore, when performing sensor shift imaging outdoors, image position shifts and image blurring can occur due to vibrations of the imaging device, resulting in reduced image quality.

[0030] Therefore, in the present technology, multiple frames captured at each pixel phase are added together for each pixel phase, an added frame (hereinafter referred to as an added frame) is generated for each pixel phase, and the added frames for each pixel phase are synthesized.

[0031] (An example of sensor-shift imaging using this technology) FIG. 4 is a diagram showing an example of sensor shift imaging according to the present technology.

[0032] In FIG. 4, parts corresponding to those in FIG. 3 are given the same reference numerals, and their description will be omitted.

[0033] To explain in the order of imaging, first, frame #0 is generated by adding N frames acquired by high-speed imaging at a high shutter speed with the image sensor unit 1 in phase A. The high shutter speed is, for example, 1 / N of the normal shutter speed described above. Frames captured at this high shutter speed will be underexposed.

[0034] Frame #1 is generated by adding together N frames acquired by high-speed imaging at a high shutter speed when the image sensor unit 1 is in phase B.

[0035] Frame #2 is generated by adding together N frames acquired by high-speed imaging at a high shutter speed with the image sensor unit 1 in phase C. Furthermore, frame #3 is generated by adding together N frames acquired by high-speed imaging at a high shutter speed with the image sensor unit 1 in phase D.

[0036] After that, sensor shift synthesis is performed using frames of all pixel phases to generate one frame (hereinafter, referred to as a synthetic frame).

[0037] (Overview of the processing flow of this technology) FIG. 5 is a diagram showing an outline of the processing flow of the present technology.

[0038] In Fig. 5, the large arrow on the left indicates the overall processing flow. Also, parts corresponding to those in Fig. 3 are given the same reference numerals, and their explanation will be omitted.

[0039] In step S1, with the image sensor unit 1 in phase A, high-speed imaging is performed at a high shutter speed, and N frames are acquired.

[0040] In step S2, the N frames acquired in phase A are added together to generate frame #0 of phase A.

[0041] The processes of steps S1 and S2 are similarly performed for phases B to D, so that frame #1 for phase B, frame #2 for phase C, and frame #3 for phase D are generated.

[0042] In step S3, frame #0 of phase A, frame #1 of phase B, frame #2 of phase C, and frame #3 of phase D are used to perform sensor shift synthesis to generate one frame. Then, the process ends.

[0043] 5, some of the frames acquired by high-speed imaging in a micro-vibration environment are close to the target phase (target pixel position) that is set, and some are far from it. Therefore, in the present technology, only the frames close to the target phase among the frames acquired by high-speed imaging are selected and synthesized.

[0044] In this way, the occurrence of misalignment between images and blurring of images caused by vibration of the imaging device can be suppressed, and image quality can be improved.

[0045] <2. System configuration and operation> (Example of the configuration of a sensor shift imaging system) FIG. 6 is a block diagram showing a configuration of an embodiment of a sensor shift imaging system to which the present technology is applied.

[0046] The sensor shift imaging system 11 in FIG. 6 is configured with, for example, a single-lens reflex imaging device.

[0047] In FIG. 6, the sensor shift imaging system 11 includes a high-speed imaging calculation unit 21, a sensor shift synthesis unit 22, and an operation input unit .

[0048] The high-speed imaging calculation unit 21 adds up N frames acquired by performing high-speed imaging at a high shutter speed in each pixel phase, based on each setting information supplied from the operation input unit 23. The high-speed imaging calculation unit 21 outputs the added frames for all the added pixel phases to the sensor shift synthesis unit 22.

[0049] The high-speed imaging calculation unit 21 is composed of a high-speed imaging unit 31 including an image sensor unit 1, a sensor shift unit 32, a motion compensation unit 33, a frame addition unit 34, an IMU (Inertial Measurement Unit) measurement unit 36 ​​equipped with a gyro sensor 35, an accumulator 37, and a position deviation determination unit 38.

[0050] The sensor shift synthesis unit 22 performs sensor shift synthesis on frames for all pixel phases supplied from the high-speed imaging calculation unit 21, and outputs the synthesis frame to the subsequent stage.

[0051] The operation input unit 23 is composed of a button, a dial, a mouse, a touch panel, etc. The operation input unit 23 outputs a phase setting indicating a target phase (target pixel position) of each pixel phase to the sensor shift unit 32, the position deviation determination unit 38, and the sensor shift synthesis unit 22 in response to a user operation.

[0052] Furthermore, in response to a user's operation, the operation input unit 23 outputs an image quality / speed priority setting indicating whether image quality or the processing speed for acquiring N frames is to be prioritized to the positional deviation determination unit 38. Note that a threshold value for positional deviation determination may be directly input.

[0053] The high-speed imaging unit 31 is composed of an optical system such as a lens for collecting light or an aperture, the image sensor unit 1, an A / D conversion unit, etc. The high-speed imaging unit 31 performs high-speed imaging at each pixel phase corresponding to the relative position shifted by the sensor shift unit 32, at a shutter speed (high-speed shutter speed) that is 1 / N of the normal shutter speed for proper exposure. This high-speed imaging is performed until N frames are acquired for each pixel phase.

[0054] The high-speed imaging unit 31 outputs the acquired N frames to the motion compensation unit 33 for each pixel phase.

[0055] The sensor shift unit 32 shifts the relative position of the image sensor unit 1 with respect to the subject image, based on a phase setting input via the operation input unit 23.

[0056] The motion compensation unit 33 performs motion compensation only on frames to be added (hereinafter referred to as effective frames) according to the positional deviation determination result determined by the positional deviation determination unit 38 based on the angle information supplied from the accumulator 37, as necessary, and outputs the result to the frame adder 34. The angle information is acquired as a result of the angular velocity being accumulated by the accumulator 37, and is supplied.

[0057] The frame adder 34 adds only N effective frames for each pixel phase in accordance with the positional shift determination result supplied from the positional shift determination unit 38. The frame adder 34 outputs the added frames obtained by adding the N effective frames to the sensor shift synthesis unit 22 for all pixel phases.

[0058] The IMU measurement unit 36 ​​includes a gyro sensor 35 and sensors such as an accelerometer. The gyro sensor 35 is synchronized with the high-speed imaging unit 31. The IMU measurement unit 36 ​​outputs the angular velocity of the sensor-shift imaging system 11 measured by the gyro sensor 35 to an integrator 37.

[0059] The accumulator 37 accumulates the angular velocities, and outputs angle information acquired as a result of the accumulation to the motion compensation unit 33 and the positional deviation determination unit .

[0060] The positional deviation determination unit 38 determines the positional deviation of the frames captured at high speed, based on the angle information supplied from the integrating unit 37, using the phase setting (target phase) input via the operation input unit 23 as a reference value. The positional deviation determination unit 38 outputs the positional deviation determination result to the frame adder 34.

[0061] Moreover, the positional deviation determination unit 38 sets a threshold value for determining positional deviation, based on the image quality / speed priority setting supplied from the operation input unit 23 or the angle information supplied from the integrating unit 37. The positional deviation determination unit 38 performs positional deviation determination for frames captured at high speed, based on the set threshold value.

[0062] Although FIG. 6 shows an example in which the sensor-shift imaging system 11 includes the motion compensation section 33, the motion compensation section 33 is not essential to the sensor-shift imaging system 11.

[0063] (Operation of sensor shift imaging system) The sensor-shift imaging process of the sensor-shift imaging system 11 will be described with reference to FIG.

[0064] In step S11, the high-speed imaging unit 31 performs AE (auto exposure) and calculates an aperture and a normal shutter speed that provide proper exposure in normal imaging in which imaging is performed with one shutter. The high-speed imaging unit 31 sets the shutter speed to a speed (high-speed shutter speed) that is 1 / N of the calculated normal shutter speed.

[0065] In step S12, the sensor shift imaging system 11 performs a sensor shift. Specifically, the sensor shift unit 32 shifts the pixel phase of the image sensor unit 1 to a target phase (for example, phase A in FIG. 5) based on a phase setting input via the operation input unit 23.

[0066] In step S13, the high-speed imaging unit 31 performs high-speed imaging. Specifically, the high-speed imaging unit 31 performs high-speed imaging at the pixel phase set in step S12 at the high shutter speed set in step S11 to obtain one frame.

[0067] In step S14, the positional deviation determination unit 38 determines, based on the angle information supplied from the integrating unit 37, whether or not the positional deviation of the frame captured at high speed is within a predetermined threshold value.

[0068] If it is determined in step S14 that the positional deviation of the frame captured at high speed is not within the predetermined threshold, the process returns to step S13. The frame whose positional deviation is determined to be not within the predetermined threshold (hereinafter referred to as an invalid frame) is discarded without being added.

[0069] Thereafter, in step S14, the processes of steps S13 and S14 are repeatedly executed until it is determined that the positional deviation of the frames captured at high speed is within a predetermined threshold value.

[0070] On the other hand, if it is determined in step S14 that the positional deviation of the frames captured at high speed is within the predetermined threshold value, the process proceeds to step S15.

[0071] In step S15, the motion compensation unit 33 performs motion compensation as necessary for valid frames for which the position shift is determined to be within a predetermined threshold based on the angle information supplied from the integration unit 37, and outputs the resulting frames to the frame addition unit .

[0072] In step S16, the frame adder 34 performs frame addition. That is, the frame adder 34 adds the newly obtained effective frame to the added frame to which the effective frame in the current pixel phase has been added up to now.

[0073] In step S17, the frame adder 34 determines whether or not N frames have been added. If it is determined in step S17 that N frames have not been added, the process returns to step S13.

[0074] Thereafter, the processes of steps S13 to S17 are repeatedly executed until it is determined in step S17 that N frames have been added.

[0075] On the other hand, if it is determined in step S17 that N frames have been added, the process proceeds to step S18. At this time, the frame adder 34 outputs the added frame generated by adding the N effective frames to the sensor shift synthesis unit 22.

[0076] If it takes longer than expected for the number of valid frames to reach N, the process may proceed to step S18 before the number of valid frames reaches N.

[0077] In step S18, the sensor shift synthesis unit 22 determines whether or not the added frames of all pixel phases have been acquired. If it is determined in step S18 that the added frames of all pixel phases have not yet been acquired, the process returns to step S12.

[0078] Thereafter, in step S18, the processes of steps S12 to S18 are repeatedly executed until it is determined that the summed frames of all pixel phases have been acquired. As a result, the pixel phases of the image sensor unit 1 are shifted in order from phase A to phase D, and summed frames in which N frames are added together are obtained for all pixel phases.

[0079] On the other hand, in step S18, if the sensor shift synthesis unit 22 acquires an added frame in which N effective frames are added for all pixel phases from the frame adder 34, it determines that the added frames for all pixel phases have been acquired, and the process proceeds to step S19.

[0080] In step S19, the sensor shift synthesis unit 22 performs sensor shift synthesis on the added frames of the respective pixel phases, and outputs the synthesis frame after the sensor shift synthesis to the subsequent stage.

[0081] As a result, the occurrence of misalignment between frames and blurring of images caused by vibrations of the sensor-shift imaging system 11 is suppressed, and image quality can be improved in sensor-shift imaging.

[0082] <3. Details of displacement determination> The following will explain the three key points of this technology in order. a. Compatibility between sensitivity and resolution b. User settings for displacement determination c. Dynamic control of the threshold according to the vibration of the imaging device

[0083] <a. Compatibility between sensitivity and resolution> FIG. 8 is a diagram showing images obtained by normal imaging and high-speed imaging. The thick dashed rectangle indicates the target pixel position (the target phase described above), and the thick cross mark indicates the center of the pixel.

[0084] On the left side of FIG. 8, the state of the light exposed to the pixel in the conventional normal imaging is shown.

[0085] In the conventional case, due to the vibration of the image sensor unit 1 during exposure, the light in the range indicated by the non-clearly defined circle is exposed to the pixel, and the image is captured as if it is blurred.

[0086] On the right side of FIG. 8, the distribution of the pixel positions (exposed pixel positions) where the frames are captured in the high-speed imaging of this technology is shown. The dashed circle represents the range of the light exposed to the pixel in the conventional normal imaging as in the left side of FIG. 8. The solid rectangle represents the pixel positions to be added among the pixel positions where the frames are captured, and the dashed rectangle represents the pixel positions not to be added among the pixel positions where the frames are captured.

[0087] In the case of this technology, among the pixel positions where the frames are captured by high-speed imaging, only the pixel signals obtained at the pixel positions close to the target phase are subject to addition based on the angle information supplied from the integration unit 37. The pixel signals obtained at other pixel positions are not subject to addition.

[0088] FIG. 9 is a diagram showing a table summarizing the processes and performances of normal imaging and high-speed imaging.

[0089] In FIG. 9, the performance consists of sensitivity and resolution.

[0090] In the case of conventional normal imaging, the process is that imaging of one frame is performed at the normal shutter speed as described above. Therefore, although the sensitivity does not change, the image is blurred due to vibration and the resolution decreases.

[0091] On the other hand, in the case of high-speed imaging of the present technology, imaging of N frames is performed at a high shutter speed. In high-speed imaging, since imaging is performed at a high shutter speed that is 1 / N of the normal shutter speed that provides proper exposure in normal imaging, an underexposed (dark or noisy) frame (image) is acquired.

[0092] After that, frame high-speed imaging and addition are performed up to N frames to achieve the same sensitivity (S / N ratio) as in normal imaging, so the same sensitivity (S / N ratio) as in normal imaging is maintained. Also, since only the effective frames imaged at pixel positions close to the target phase are added, even if vibration occurs, the image is less likely to be blurred and a decrease in resolution is suppressed.

[0093] As described above, according to the present technology, since selection of high-speed imaged frames is performed, it is possible to achieve both improvement in sensitivity and resolution, and the image quality is improved.

[0094] <b. User setting for position shift determination> FIG. 10 is a diagram showing the size of the position shift tolerance range.

[0095] In FIG. 10, the same description is made for the part corresponding to FIG. 8. In FIG. 10, the thick-lined rectangle indicates the position shift tolerance range. The length of the side in the thick-lined rectangle is the threshold value a in the position shift determination.

[0096] On the left side of FIG. 10, an example of the distribution of pixel positions to be added and pixel positions not to be added when the position shift tolerance range is small is shown. On the right side of FIG. 10, an example of the distribution of pixel positions to be added and pixel positions not to be added when the position shift tolerance range is large is shown.

[0097] In either case, only pixel signals obtained at pixel positions whose centers are within the allowable positional deviation range when the frame is captured are added, and only pixel signals obtained at pixel positions whose centers are outside the allowable positional deviation range are excluded from addition.

[0098] Therefore, when the allowable range of positional deviation is small, the resolution improves, but it takes time to acquire N effective frames.

[0099] On the other hand, when the allowable range of positional deviation is large, the resolution decreases, but the time required to acquire N effective frames is shortened.

[0100] FIG. 11 is a diagram showing the relationship between the allowable range of positional deviation and the image quality and processing speed.

[0101] In Fig. 11, the relationship between image quality (resolution) and processing speed (calculation / imaging) according to the size of the positional deviation tolerance is shown. A circle indicates good or fast, a cross indicates bad or slow, and a triangle indicates medium.

[0102] When the misalignment tolerance is small, the image quality is good but the processing speed is slow. When the misalignment tolerance is medium, the image quality and processing speed are both medium. When the misalignment tolerance is large, the image quality is poor due to blurring and fuzzy images, but the processing speed is fast.

[0103] That is, as indicated by the arrow on the right side of FIG. 11, it can be seen that when the allowable range of positional deviation is narrowed, priority is given to image quality, and when the allowable range of positional deviation is widened, priority is given to processing speed.

[0104] As described above, there is a trade-off between image quality and processing speed depending on the setting of the allowable range of misregistration. Therefore, by setting the allowable range of misregistration, the user can select whether to give priority to image quality or to processing speed.

[0105] In other words, by allowing the user to select whether to prioritize image quality or processing speed, the positional deviation tolerance range (threshold value) can be set according to the selected priority.

[0106] <c. Dynamic control of the threshold value according to the vibration of the imaging device> FIG. 12 is a diagram showing the case where the positional deviation tolerance range (threshold value a for positional deviation determination) is fixed.

[0107] In FIG. 12, a graph showing the observed values of the vibration of the imaging device over time is shown. Note that the observed values of the vibration of the imaging device are obtained from the angle information supplied from the integration unit 37.

[0108] On the left side of FIG. 12, the relationship between the amplitude of the vibration of the imaging device (hereinafter referred to as the vibration amplitude) when the threshold value a is fixed to a large value and the imaging time (processing speed) until N effective frames are obtained is shown. On the right side of FIG. 12, the relationship between the vibration amplitude of the imaging device and the imaging time until N effective frames are obtained when the threshold value a is fixed to a small value is shown.

[0109] When the threshold value a is increased, the time for the vibration amplitude of the imaging device to fall within the positional deviation tolerance range becomes longer. Therefore, the probability that the captured frame is a valid frame increases, and the time required to capture N valid frames becomes shorter than when the threshold value a is decreased.

[0110] On the other hand, when the threshold value a is decreased, the time for the vibration amplitude of the imaging device to fall within the positional deviation tolerance range becomes shorter. Therefore, the probability that the captured frame is a valid frame decreases, and the time required to capture N valid frames becomes longer than when the threshold value a is increased.

[0111] That is, when the threshold value a is fixed to a large value, the imaging time becomes shorter than when the threshold value a is fixed to a small value. On the other hand, the blurring and blur of the image due to vibration become larger.

[0112] On the other hand, when the threshold value a is fixed to a small value, the blurring or shaking of the image caused by vibration is reduced compared to when the threshold value a is fixed to a large value, but the image capturing time becomes longer, which is inconvenient for the user.

[0113] FIG. 13 is a diagram showing a case where the positional deviation tolerance (threshold a for determining positional deviation) is dynamically controlled based on the result of FIG.

[0114] FIG. 13 shows the imaging time until N effective frames are acquired when the threshold value a is dynamically controlled in accordance with the observed value of the vibration of the imaging device.

[0115] When the threshold value a is dynamically controlled, for example, when the vibration amplitude of the imaging device is large, the threshold value a is increased, and when the vibration amplitude of the imaging device is small, the threshold value a is decreased.

[0116] Therefore, when the threshold a is dynamically controlled, it is possible to avoid the excessive degradation of image quality that occurs when the threshold a is fixed at a large value (left side of FIG. 12). Also, when the threshold a is dynamically controlled, it is possible to avoid the excessive increase in imaging time that occurs when the threshold a is fixed at a small value (right side of FIG. 12).

[0117] As described above, by dynamically controlling the threshold value a in response to vibration of the imaging device, it is possible to optimize image quality and imaging time.

[0118] FIG. 14 is a diagram showing a case where an upper limit is set on the imaging time.

[0119] In Fig. 14, the imaging time estimated based on the magnitude of the vibration amplitude of the imaging device and the magnitude of the allowable range of misalignment is shown in concrete numerical values. Note that the numerical values ​​are merely examples. Fig. 14 also shows that the smaller the allowable range of misalignment, the higher the image quality and the slower the processing speed, and that the larger the allowable range of misalignment, the lower the image quality and the faster the processing speed.

[0120] When the vibration amplitude of the imaging device is small, if the positional deviation tolerance is set to small, the expected imaging time is 1.00 sec. When the vibration amplitude of the imaging device is small, if the positional deviation tolerance is set to medium, the expected imaging time is 2.00 sec. When the vibration amplitude of the imaging device is small, if the positional deviation tolerance is set to large, the expected imaging time is 4.00 sec.

[0121] When the vibration amplitude of the imaging device is medium, if the positional deviation tolerance is set to small, the expected imaging time is 0.50 seconds. When the vibration amplitude of the imaging device is medium, if the positional deviation tolerance is set to medium, the expected imaging time is 1.00 seconds. When the vibration amplitude of the imaging device is medium, if the positional deviation tolerance is set to large, the expected imaging time is 2.00 seconds.

[0122] When the vibration amplitude of the imaging device is large, if the positional deviation tolerance is set to small, the expected imaging time is 0.25 seconds. When the vibration amplitude of the imaging device is large, if the positional deviation tolerance is set to medium, the expected imaging time is 0.50 seconds. When the vibration amplitude of the imaging device is large, if the positional deviation tolerance is set to large, the expected imaging time is 1.00 seconds.

[0123] In the case of the above configuration, when the highest image quality priority setting is set to minimize the allowable range of positional deviation, there arise cases where the expected imaging time does not fall within 1.00 sec, as shown in box a.

[0124] On the other hand, when the shortest speed priority setting is set, which maximizes the positional deviation tolerance, as shown in box b, the expected imaging time is always within 1.00 sec, but image quality is always sacrificed.

[0125] Therefore, an image quality priority setting with a speed upper limit may be provided, which sets an upper limit on the image capture time (processing speed), so that image quality is optimized within the upper limit of the image capture time.

[0126] For example, when the image quality priority setting with upper speed limit is set, as shown in box c, when the vibration amplitude of the imaging device is small, the positional deviation tolerance range is set to small, when the vibration amplitude of the imaging device is medium, the positional deviation tolerance range is set to medium, and when the vibration amplitude of the imaging device is large, the positional deviation tolerance range is set to large.

[0127] As described above, when it is desired to keep the expected image capture time to within 1 second from the viewpoints of system convenience and user convenience, the allowable range of positional deviation is dynamically changed according to the magnitude of vibration of the image capture device during image capture. This makes it possible to obtain an image of optimal image quality within the upper limit of the image capture time.

[0128] In the above description, the image sensor unit 1 is shifted by one pixel at a time (with pixel accuracy of one pixel), but may be shifted with pixel accuracy less than one pixel (for example, 0.5 pixel).

[0129] <4.Other> (Effects of this technology) In this technology, when capturing an image while shifting the pixel phase of an image sensor having a two-dimensional pixel array, multiple frames captured at each pixel phase are added together for each pixel phase, an added frame is generated for each pixel phase, and the added frames for each pixel phase are synthesized.

[0130] As a result, even if the imaging device is subjected to minute vibrations, image quality does not deteriorate and sensor shift imaging is possible.

[0131] In addition, in the present technology, frames are selected according to the amount of positional deviation from a target pixel phase, that is, imaging is performed until a predetermined number of valid frames are acquired, the amount of positional deviation being within a threshold.

[0132] This means that no motion blur occurs even when images are added, making it possible to achieve both sensitivity and resolution.

[0133] Furthermore, in this technology, the user can set the threshold value for determining positional deviation, so that it is possible to accommodate both users who prioritize image quality and users who prioritize speed.

[0134] Furthermore, in the present technology, the threshold value for determining the positional deviation is dynamically controlled in response to the vibration of the imaging device detected by the sensor.

[0135] This makes it possible to optimize image quality and processing speed in response to dynamic changes in vibration of the imaging device.

[0136] (Example of computer configuration) The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0137] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes according to a program.

[0138] A CPU 301 , a ROM (Read Only Memory) 302 , and a RAM 303 are connected to each other via a bus 304 .

[0139] An input / output interface 305 is further connected to the bus 304. An input unit 306 including a keyboard, a mouse, etc., and an output unit 307 including a display, a speaker, etc. are connected to the input / output interface 305. In addition, a storage unit 308 including a hard disk, a non-volatile memory, etc., a communication unit 309 including a network interface, etc., and a drive 310 that drives a removable medium 311 are connected to the input / output interface 305.

[0140] In the computer configured as above, the CPU 301 loads a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executes the program, thereby performing the above-mentioned series of processes.

[0141] The programs executed by the CPU 301 are installed in the storage unit 308 by being recorded on a removable medium 311, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0142] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0143] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0144] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0145] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0146] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0147] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0148] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0149] <Examples of configuration combinations> The present technology can also be configured as follows. (1) an adder that, when capturing an image while shifting a pixel phase of an image sensor having a two-dimensional pixel array, adds up a plurality of frames captured at each pixel phase for each pixel phase to generate a sum frame for each pixel phase; A synthesis unit that synthesizes the added frames of each of the pixel phases; An image processing device comprising: (2) The adder adds only valid frames captured in a state where the amount of positional shift of the image sensor with respect to each pixel phase is within a threshold value. The image processing device according to (1) above. (3) The image forming apparatus further includes a threshold setting unit that sets the threshold based on a user operation. The image processing device according to (2) above. (4) The threshold setting unit sets the threshold in accordance with a selection between image quality priority, which places priority on image quality, and speed priority, which places priority on a speed at which the effective frames are acquired. The image processing device according to (3) above. (5) The threshold setting unit sets the threshold to a value smaller when the image quality priority is selected out of the image quality priority and the speed priority than when the speed priority is selected. The image processing device according to (4) above. (6) When an upper limit value of the speed at which the valid frames are acquired is set, the threshold setting unit sets the threshold to a maximum value within a range in which the speed at which the valid frames are acquired is the upper limit value. The image processing device according to (3) above. (7) a sensor for detecting vibration of the image sensor; a threshold setting unit that sets the threshold in response to vibration of the image sensor detected by the sensor; The image processing device according to (1) or (2), further comprising: (8) The adder adds a predetermined number of frames for each pixel phase. The image processing device according to any one of (1) to (7). (9) the image sensor for capturing a frame at each of the pixel phases; a shift control unit that shifts the pixel phase; The image processing device according to any one of (1) to (8) further comprising: (10) The shift control unit shifts the pixel phase by shifting the image sensor. The image processing device according to (9) above. (11) The shift control unit shifts the pixel phase by shifting a lens. The image processing device according to (9) above. (12) The image sensor captures frames at a high shutter speed that is faster than a normal shutter speed that provides an appropriate amount of exposure in one exposure. The image processing device according to (9) above. (13) The image forming apparatus further includes a motion compensation unit that compensates for the motion of the frame captured at each pixel phase as necessary. The image processing device according to any one of (1) to (12). (14) The image processing device is an imaging device. The image processing device according to any one of (1) to (13). (15) The image processing device When capturing an image while shifting a pixel phase of an image sensor having a two-dimensional pixel array, a plurality of frames captured at each pixel phase are added for each pixel phase to generate an added frame for each pixel phase; The summed frames of each of the pixel phases are synthesized. Image processing methods. (16) an adder that, when capturing an image while shifting a pixel phase of an image sensor having a two-dimensional pixel array, adds up a plurality of frames captured at each pixel phase for each pixel phase to generate a sum frame for each pixel phase; A synthesis unit that synthesizes the added frames of each of the pixel phases; A program that allows a computer to function. [Explanation of symbols]

[0150] 1 image sensor unit, 2 pixel, 11 sensor shift imaging system, 21 high-speed imaging calculation unit, 22 sensor shift synthesis unit, 23 operation input unit, 31 high-speed imaging unit, 32 sensor shift unit, 33 motion compensation unit, 34 frame addition unit, 35 gyro sensor, 36 IMU measurement unit, 37 integration unit, 38 position deviation determination unit

Claims

1. an adder that, when capturing an image while shifting a pixel phase of an image sensor having a two-dimensional pixel array, adds up a plurality of frames captured at each pixel phase for each pixel phase to generate a sum frame for each pixel phase; A synthesis unit that synthesizes the added frames of each of the pixel phases; An image processing device comprising:

2. The adder adds only valid frames captured in a state where the amount of positional shift of the image sensor with respect to each pixel phase is within a threshold value. The image processing device according to claim 1 .

3. The image forming apparatus further includes a threshold setting unit that sets the threshold based on a user operation. The image processing device according to claim 2 .

4. The threshold setting unit sets the threshold in accordance with a selection between image quality priority, which places priority on image quality, and speed priority, which places priority on a speed at which the effective frames are acquired. The image processing device according to claim 3 .

5. The threshold setting unit sets the threshold to a value smaller when the image quality priority is selected out of the image quality priority and the speed priority than when the speed priority is selected. The image processing device according to claim 4.

6. When an upper limit value of the speed at which the valid frames are acquired is set, the threshold setting unit sets the threshold to a maximum value within a range in which the speed at which the valid frames are acquired is the upper limit value. The image processing device according to claim 3 .

7. a sensor for detecting vibration of the image sensor; a threshold setting unit that sets the threshold in response to vibration of the image sensor detected by the sensor; The image processing device according to claim 2 , further comprising:

8. The adder adds a predetermined number of frames for each pixel phase. The image processing device according to claim 1 .

9. the image sensor for capturing a frame at each of the pixel phases; a shift control unit that shifts the pixel phase; The image processing device according to claim 1 , further comprising:

10. The shift control unit shifts the pixel phase by shifting the image sensor. The image processing device according to claim 9 .

11. The shift control unit shifts the pixel phase by shifting a lens. The image processing device according to claim 9 .

12. The image sensor captures frames at a high shutter speed that is faster than a normal shutter speed that provides an appropriate amount of exposure in one exposure. The image processing device according to claim 9 .

13. The image forming apparatus further includes a motion compensation unit that compensates for the motion of the frame captured at each pixel phase as necessary. The image processing device according to claim 1 .

14. The image processing device is an imaging device. The image processing device according to claim 1 .

15. The image processing device When capturing an image while shifting a pixel phase of an image sensor having a two-dimensional pixel array, a plurality of frames captured at each pixel phase are added for each pixel phase to generate a sum frame for each pixel phase; The summed frames of each of the pixel phases are synthesized. Image processing methods.

16. an adder that, when capturing an image while shifting a pixel phase of an image sensor having a two-dimensional pixel array, adds up a plurality of frames captured at each pixel phase for each pixel phase to generate a sum frame for each pixel phase; A synthesis unit that synthesizes the added frames of each of the pixel phases; A program that allows a computer to function.

Images