Imaging device and imaging method

Abstract

To preferably correct an image shake.SOLUTION: An imaging device includes a first sensor for acquiring two-dimensional information having information in each coordinates on a two-dimensional plane, a second sensor for acquiring two-dimensional information of two-dimensional coordinates having information in each coordinates on the two-dimensional plane and having resolution lower than the two-dimensional information acquired by the first sensor, a camera shake detection part for detecting a degree of a camera shake by analyzing the information acquired by the first sensor, and a correction part for correcting the information acquired by the second sensor in accordance with the degree of a camera shake detected by the camera shake detection part.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an imaging device and an imaging method. [Background technology]

[0002] Some conventional image capturing devices have a function for correcting image blur caused by camera shake or the like. One example of a camera shake detection means for realizing the correction function is a technology using a gyro sensor. Patent Document 1, for example, can be cited as an example of a document disclosing such a technology. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-180283 Summary of the Invention [Problem to be solved by the invention]

[0004] Recently, imaging devices have become known that include multiple sensors and are capable of acquiring multiple pieces of information from the same subject. Such imaging devices may include, in addition to an image sensor, at least one of a ToF (Time of Flight) sensor, a polarization sensor, and an event sensor, allowing additional information to be added to image data captured by the image sensor. Applying image stabilization to such imaging devices requires a gyro sensor for each sensor, which poses problems in terms of power consumption and cost.

[0005] Furthermore, there is technology that calculates the amount of shake by detecting movement between time-lag frames of image data rather than using a gyro sensor as the detection means, but when using sensors other than image sensors, the amount of information is often insufficient due to resolution issues when calculating from the time-lag difference in sensor information, making it unsuitable for calculating the amount of shake.

[0006] The present invention has been made in view of the above circumstances, and has as its object to provide an imaging apparatus and an imaging method that are capable of correcting image blur in an appropriate manner. [Means for solving the problem]

[0007] [1] One aspect of the present invention is an imaging device comprising: a first sensor that acquires two-dimensional information having information at each coordinate on a two-dimensional plane; a second sensor that acquires two-dimensional information at two-dimensional coordinates having information at each coordinate on a two-dimensional plane and having lower resolution than the two-dimensional coordinates of the two-dimensional information acquired by the first sensor; a camera shake detection unit that detects the degree of camera shake by analyzing the information acquired by the first sensor; and a correction unit that corrects the information acquired by the second sensor according to the degree of camera shake detected by the camera shake detection unit.

[0008] [2] In addition, according to one aspect of the present invention, in the imaging device described in [1] above, the first sensor is an image sensor that acquires two-dimensional image information.

[0009] [3] In another aspect of the present invention, in the imaging device according to the above [1] or [2], the second sensor is any one of a ToF (Time Of Flight) sensor, a polarization sensor, and an event sensor.

[0010] [4] Furthermore, one aspect of the present invention is an imaging device according to any one of [1] to [3] above, wherein the correction unit converts two-dimensional coordinates of the two-dimensional information acquired by the first sensor into corresponding coordinates of the two-dimensional coordinates of the two-dimensional information acquired by the second sensor, and corrects the information acquired by the second sensor according to the degree of camera shake detected at each converted coordinate.

[0011] [5] Furthermore, one aspect of the present invention is an imaging device according to any one of [1] to [4] above, wherein the first sensor and the second sensor both capture successive frame images at a predetermined frame rate, the frame rate of the first sensor is higher than the frame rate of the second sensor, and the correction unit corrects the frame image of the second sensor based on the difference between multiple frames of the first sensor that correspond to one frame of the second sensor.

[0012] [6] Furthermore, in one aspect of the present invention, in the imaging device described in [5] above, the correction unit corrects the frame image of the second sensor based on the difference between the first frame image and the last frame image among multiple frames of the first sensor corresponding to one frame of the second sensor.

[0013] [7] Another aspect of the present invention is an imaging method including a first acquisition step of acquiring two-dimensional information having information at each coordinate on a two-dimensional plane, a second acquisition step of acquiring two-dimensional information having information at each coordinate on a two-dimensional plane and having two-dimensional coordinates with lower resolution than the two-dimensional coordinates of the two-dimensional information acquired by the first acquisition step, a camera shake detection step of detecting the degree of camera shake by analyzing the information acquired by the first acquisition step, and a correction step of correcting the information acquired by the second acquisition step in accordance with the degree of camera shake detected by the camera shake detection step. [Effects of the Invention]

[0014] According to the present invention, it is possible to provide an imaging apparatus and an imaging method that are capable of correcting image blur in an appropriate manner. [Brief explanation of the drawings]

[0015] [Figure 1] 2 is a schematic diagram showing a first example of a cross section of the imaging device according to the first embodiment. FIG. [Figure 2] 4 is a schematic diagram showing a second example of a cross section of the imaging device according to the first embodiment. FIG. [Figure 3]4 is a schematic diagram showing a third example of a cross section of the imaging device according to the first embodiment. FIG. [Figure 4] FIG. 4 is a schematic diagram showing a fourth example of a cross section of the imaging device according to the first embodiment. [Figure 5] FIG. 10 is a schematic diagram showing a fifth example of a cross section of the imaging device according to the first embodiment. [Figure 6] 1 is a functional configuration diagram showing an example of the functional configuration of an imaging device according to a first embodiment. [Figure 7] FIG. 10 is a functional configuration diagram showing an example of the functional configuration of an imaging device according to a second embodiment. [Figure 8] 10 is a flowchart showing a series of processing steps performed by an imaging device according to a second embodiment. [Figure 9] FIG. 10 is a functional configuration diagram showing a modified example of the functional configuration of the imaging device according to the second embodiment. [Figure 10] FIG. 10 is a functional configuration diagram showing an example of the functional configuration of an imaging device according to a third embodiment. [Figure 11] 10 is a flowchart showing a series of processing steps performed by an imaging device according to a third embodiment. [Figure 12] FIG. 2 is a block diagram showing an example of the internal configuration of the imaging device according to the embodiment. [Figure 13] 10A and 10B are diagrams for explaining problems with image blur correction when an imaging device according to a conventional technique is rotated. DETAILED DESCRIPTION OF THE INVENTION

[0016] [Prior art] An imaging device and an imaging method according to preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiments described below are merely examples, and the embodiments to which the present invention is applicable are not limited to the following embodiments. Furthermore, "based on XX" in this application means "based on at least XX" and includes cases where the method is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is directly used, but also includes cases where the method is based on XX after calculation or processing. "XX" is an arbitrary element (e.g., arbitrary information). Furthermore, in the following drawings, the scale and number of elements in each structure may differ from the scale and number of elements in the actual structure to make each configuration easier to understand.

[0017] First, problems in the prior art will be described with reference to FIG.

[0018] 13A and 13B are diagrams illustrating problems with image blur correction when an imaging device according to the related art is rotated. Both of FIGS. 13A and 13B are schematic diagrams of an imaging device 91 having two sensors, viewed from the front. Light incident on a lens 92 and light incident on a lens 93 enter the two sensors, respectively. In other words, the imaging device 91 can also be said to have a pair of lenses and sensors. The two sensors included in the imaging device 91 may be any two of an image sensor, a ToF (Time Of Flight) sensor, a polarization sensor, and an event sensor.

[0019] When attempting to correct image blur caused by camera shake or the like in an image captured by such an imaging device, it is conceivable to provide a gyro sensor corresponding to each sensor. However, providing a gyro sensor for each sensor would result in problems such as increased power consumption, increased costs, and an increase in the size of the device.

[0020] When considering image shake correction without a gyro sensor, it is conceivable to compare successively captured images, calculate the amount of movement, and correct the image shake. However, the performance of this image processing-based movement amount calculation depends on the resolution or number of pixels of the sensor. For example, it is known that the number of pixels of an image sensor is sufficient to calculate the amount of movement by image processing. However, the number of pixels of a ToF sensor, polarization sensor, etc. is currently not as large as that of an image sensor, and is not sufficient to calculate the amount of movement due to image shake.

[0021] Furthermore, in the case of an imaging device equipped with an image sensor with a large number of pixels and another sensor with a small number of pixels, it is conceivable to detect the degree of shake using information acquired by the image sensor and apply it to the other sensor. However, depending on the location of the center of rotation, the degree of shake detected using the image sensor cannot be directly applied. For example, in the case of the rotation shown in FIG. 13(A), the center of rotation is point 94. In this case, lenses 92 and 93 rotate in the direction of the arrows shown in FIG. 13(A). Also, in the case of the rotation shown in FIG. 13(B), the center of rotation is point 95. In this case, lenses 92 and 93 rotate in the direction of the arrows shown in FIG. 13(B). Thus, there has been a problem in that the degree of shake detected using the image sensor cannot be directly applied to the other sensor depending on the location of the center of rotation.

[0022] [Embodiment] Next, the present embodiment will be described with reference to Figures 1 to 12. First, the first embodiment will be described with reference to Figures 1 to 6.

[0023] [First embodiment] 1 is a schematic diagram showing a first example of a cross section of an imaging device according to an embodiment. An example of the configuration of the imaging device 10 will be described with reference to the same figure. The imaging device 10 includes an RGB sensor as an image sensor, a ToF sensor, and an event sensor. By including these three sensors, the imaging device 10 can acquire color image information, distance information to a subject, and brightness changes (events) of each pixel.

[0024] The imaging device 10 includes a lens 110, a laser diode 120, a distance measurement unit 140, and an image capturing unit 150. The distance measurement unit 140 performs distance measurement using infrared light, and the image capturing unit 150 captures an image using visible light.

[0025] The distance measuring unit 140 measures the distance to the subject using infrared light IL out of the light incident on the lens 110. The distance measuring unit 140 includes a visible light / infrared separating coat 141, a ToF sensor 143, and a reflecting surface 145.

[0026] The visible light / infrared separating coating 141 transmits visible light and reflects light with wavelengths in the near-infrared range or longer (i.e., infrared light). When light L enters the lens 110, the visible light / infrared separating coating 141 transmits visible light VL and reflects infrared light IL. The optical axis of the lens 110 is referred to as the optical axis OA. The infrared light IL transmitted by the visible light / infrared separating coating 141 is reflected by the reflecting surface 145 and enters the ToF sensor 143. The infrared light IL includes light that is emitted from the laser diode, reflected from the subject, and enters the lens 110.

[0027] Visible light VL and infrared light IL pass through approximately the same optical axis between lens 110 and visible light / infrared separating coat 141. The approximately same range may be, for example, a range in which an optical path is formed by a common lens.

[0028] The ToF sensor 143 detects the incident infrared light IL. Specifically, the ToF sensor 143 includes a plurality of pixels 144, and detects the time it takes for each pixel to receive the reflected light of the light emitted from the laser diode 120. Based on this time and the speed of light, the distance to the subject can be calculated.

[0029] The image capturing unit 150 includes a half mirror 130 , an event sensor 153 , and an RGB sensor 163 .

[0030] The visible light VL transmitted through the visible light / infrared separating coating 141 is split into two optical paths, transmitted light and reflected light, by the half mirror 130 in the image capturing section 150. The half mirror 130 may be, for example, a dielectric half mirror.

[0031] The half mirror 130 is provided on the optical path between the lens 110 and the event sensor 153, and on the optical path between the lens 110 and the RGB sensor 163. The first reflected light and the second reflected light separated by the half mirror 130 pass along approximately the same optical axis between the lens 110 and the half mirror 130. The "approximately the same range" may be, for example, a range in which an optical path is formed by a common lens. The half mirror 130 may be any optical member that transmits a portion of incident light and reflects the other portion of light.

[0032] The light split into two optical paths, transmitted light and reflected light, by the half mirror 130 is received by sensors arranged on each optical path. Specifically, the light transmitted through the half mirror 130 is received by the event sensor 153, and the light reflected by the half mirror 130 is received by the RGB sensor 163.

[0033] The event sensor 153 detects incident visible light VL. Specifically, the event sensor 153 includes a plurality of pixels 154, and detects the intensity of the visible light VL at each pixel. The event sensor 153 can detect the occurrence of an event by detecting a change in the intensity of the detected visible light VL.

[0034] The RGB sensor 163 detects the incident visible light VL. Specifically, the RGB sensor 163 has a plurality of pixels 164, and detects the intensity of the visible light VL in each pixel. A color filter array in a Bayer pattern may be provided between the RGB sensor 163 and the half mirror 130. The RGB sensor 163 generates image information based on the intensity of the detected visible light VL.

[0035] 2 is a schematic diagram showing a second example of a cross section of an imaging device according to an embodiment. An example of the configuration of an imaging device 10A will be described with reference to the same figure. The imaging device 10A is similar to the imaging device 10 in that it includes an RGB sensor, a ToF sensor, and an event sensor as image sensors. However, the imaging device 10A and the imaging device 10 differ from each other in the positions at which sensors are provided. In the description of the imaging device 10A, components similar to those of the imaging device 10 are denoted by similar reference numerals, and description thereof may be omitted.

[0036] The imaging device 10A includes a lens 110, a laser diode 120, a distance measuring unit 140A, and an image capturing unit 150A. The distance measuring unit 140A is a modified example of the distance measuring unit 140, and is provided at a different position from the distance measuring unit 140. The image capturing unit 150 is a modified example of the image capturing unit 150, and is provided at a different position from the image capturing unit 150.

[0037] The image capturing section 150A includes a half mirror 130A, a visible light / infrared separating coat 141A, a reflecting surface 145, an event sensor 153A, and an RGB sensor 163. The half mirror 130A is a modified example of the half mirror 130, and is provided at a different position than the half mirror 130. The visible light / infrared separating coat 141A is a modified example of the visible light / infrared separating coat 141, and is provided at a different position than the visible light / infrared separating coat 141. The event sensor 153A is a modified example of the event sensor 153, and is provided at a different position than the event sensor 153A.

[0038] The half mirror 130A is provided on the optical path between the lens 110 and the event sensor 153A, and on the optical path between the lens 110 and the RGB sensor 163. The first reflected light and the second reflected light separated by the half mirror 130A pass along approximately the same optical axis between the lens 110 and the half mirror 130A. The "approximately the same range" may be, for example, a range in which an optical path is formed by a common lens. The half mirror 130A may be any optical member that transmits a portion of incident light and reflects the other portion of light.

[0039] The light split into two optical paths, transmitted light and reflected light, by the half mirror 130A is received by sensors arranged on each optical path. Specifically, the light reflected by the half mirror 130A is reflected by the reflective surface 145 and received by the event sensor 153A. The light transmitted through the half mirror 130A is incident on the visible light / infrared separation coating 141A.

[0040] The visible light / infrared separating coating 141A transmits visible light VL and reflects infrared light IL. When light L enters the lens 110, the visible light / infrared separating coating 141 reflects the visible light VL and transmits the infrared light IL. The optical axis of the lens 110 is referred to as the optical axis OA. The infrared light IL that has passed through the visible light / infrared separating coating 141A enters the ToF sensor 143. Note that the infrared light IL includes light that is emitted from a laser diode, reflected from the subject, and entered the lens 110. The visible light VL reflected by the visible light / infrared separating coating 141A is received by the RGB sensor 163.

[0041] In the second example as well, the visible light VL and the infrared light IL pass along substantially the same optical axis between the lens 110 and the visible light / infrared separating coat 141A. The substantially same range may be, for example, a range in which an optical path is formed by a common lens.

[0042] 3 is a schematic diagram showing a third example of a cross section of an imaging device according to an embodiment. An example of the configuration of an imaging device 10B will be described with reference to the same drawing. The imaging device 10B is similar to the imaging devices 10 and 10A in that it includes an RGB sensor as an image sensor. On the other hand, the imaging device 10B differs from the imaging devices 10 and 10A in that it includes either a ToF sensor or an event sensor. In the description of the imaging device 10B, components similar to those of the imaging device 10 or 10A may be denoted by similar reference numerals, and description thereof may be omitted. As an example, the same drawing will describe an example in which the imaging device 10B includes an RGB sensor and a ToF sensor as image sensors.

[0043] The visible light / infrared separating coating 141B is a modified example of the visible light / infrared separating coating 141. The visible light / infrared separating coating 141B transmits visible light VL and reflects infrared light IL. When light L enters the lens 110, the visible light / infrared separating coating 141 reflects the visible light VL and transmits the infrared light IL. The optical axis of the lens 110 is referred to as the optical axis OA. The infrared light IL that has passed through the visible light / infrared separating coating 141B enters the ToF sensor 143B. Note that the infrared light IL includes light that is emitted from a laser diode, reflected from the subject, and entered the lens 110. The visible light VL reflected by the visible light / infrared separating coating 141B is received by the RGB sensor 163B.

[0044] 4 is a schematic diagram showing a fourth example of a cross section of an imaging device according to an embodiment. An example of the configuration of an imaging device 10C will be described with reference to the same figure. The imaging device 10C differs from the imaging device 10 in that it includes a polarization sensor instead of an event sensor. The imaging device 10C also differs from the imaging device 10 in that it includes a non-polarized visible light / infrared separating dichroic coat 170 instead of the visible light / infrared separating coat 141. In the description of the imaging device 10C, components similar to those of the imaging device 10 are denoted by similar reference numerals, and description thereof may be omitted.

[0045] The unpolarized visible light / infrared separating dichroic coat 170 separates the light L incident on the lens 110 into infrared light IL and visible light VL. This unpolarized visible light / infrared separating dichroic coat 170 may be designed to suppress changes in polarization state in the visible light range, taking wavelength dispersion into consideration. The separated infrared light IL is incident on the TOF sensor 143, as in the image capture device 10. The transmitted visible light VL is separated by the half mirror 130 and emitted to the RGB sensor 163 or the polarization sensor 173. In this case, the half mirror 130 may be designed using a half-coated metal film or a dielectric film.

[0046] 5 is a schematic diagram showing a fifth example of a cross section of an imaging device according to an embodiment. An example of the configuration of an imaging device 10D will be described with reference to the same figure. The imaging device 10D is similar to the imaging device 10C in that it includes an RGB sensor, a ToF sensor, and a polarization sensor as image sensors. On the other hand, the imaging devices 10B and 10C differ in the positions at which sensors are provided. Specifically, the positions of the ToF sensor and the polarization sensor are swapped between the imaging devices 10C and 10B. In the description of the imaging device 10C, components similar to those of the imaging device 10B are denoted by the same reference numerals, and description thereof may be omitted.

[0047] Furthermore, imaging device 10D includes half mirror 130D and unpolarized visible light / infrared separation dichroic coat 170D, where half mirror 130D is a modified example of half mirror 130, and unpolarized visible light / infrared separation dichroic coat 170D is a modified example of unpolarized visible light / infrared separation dichroic coat 170. The positional relationship between half mirror 130D and unpolarized visible light / infrared separation dichroic coat 170D in imaging device 10D is opposite to the positional relationship between half mirror 130 and unpolarized visible light / infrared separation dichroic coat 170 in imaging device 10C.

[0048] Furthermore, the imaging device 10D includes a depolarization plate 171. The light L incident on the lens 110 is separated into transmitted light and reflected light by the half mirror 130D. In this case, it is preferable to use a metal film half mirror for the half mirror 130D, which causes little change in the polarization state. The light reflected by the half mirror 130D is totally reflected by the reflecting surface 145 while maintaining its polarization state. At this time, the linearly polarized light is changed into elliptically polarized light by the total reflection. The elliptically polarized light is then returned by the depolarization plate 171 to a state close to the original linearly polarized light.

[0049] The coating of the half mirror 130D may be a half coating made of a metal film. However, the coating of the half mirror 130D is not limited to this example, and may be a dielectric multilayer film that separates visible light, transmits all infrared light, and maintains the polarization state to avoid attenuation of near-infrared light. The light that passes through the non-polarizing half mirror is separated into infrared light IL and visible light VL by the non-polarized visible light / infrared separation dichroic coating 170. The infrared light IL is incident on the TOF sensor 143D, and the visible light VL is incident on the RGB sensor 163.

[0050] Fig. 6 is a functional configuration diagram showing an example of the functional configuration of an imaging device according to an embodiment. With reference to Fig. 6, an example of the functional configuration when correcting image blur for information acquired by the imaging device described with reference to Figs. 1 to 5 will be described. The imaging device 10 shown in Fig. 6 is an example of the imaging device 10A, imaging device 10B, imaging device 10D, or imaging device 10E described above. By omitting the configuration related to a third sensor described below, a similar configuration can also be applied to imaging device 10C.

[0051] The imaging device 10 includes a first sensor 11, a second sensor 21, a third sensor 31, a first sensor processing unit 19, a second sensor processing unit 29, a third sensor processing unit 39, a camera shake detection unit 51, and a control unit 52. The first sensor processing unit 19, the second sensor processing unit 29, the third sensor processing unit 39, the camera shake detection unit 51, and the control unit 52 are implemented using, for example, electronic circuits. Each functional unit may include internal storage means such as a semiconductor memory or a magnetic hard disk drive, as necessary. Each function may be implemented using software and a computer having a CPU (Central Processing Unit). Some or all of the functional units may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). Some or all of the functional units may be implemented using a combination of software and hardware.

[0052] The first sensor 11 acquires two-dimensional information having information at each coordinate on a two-dimensional plane. The two-dimensional information is, for example, image information. That is, the first sensor 11 may be an image sensor that acquires two-dimensional image information.

[0053] The second sensor 21 has information at each coordinate on a two-dimensional plane, and acquires two-dimensional information of two-dimensional coordinates with a lower resolution than the two-dimensional coordinates of the two-dimensional information acquired by the first sensor 11. The two-dimensional information may be distance information, polarization information, event information, etc. In other words, the second sensor 21 may be any of a ToF sensor, a polarization sensor, and an event sensor.

[0054] The third sensor 31 has information at each coordinate on a two-dimensional plane, and acquires two-dimensional information of two-dimensional coordinates with lower resolution than the two-dimensional coordinates of the two-dimensional information acquired by the first sensor 11. The two-dimensional information may be distance information, polarization information, event information, etc. That is, the third sensor 31 may be any of a ToF sensor, a polarization sensor, and an event sensor. Note that the third sensor 31 is not a required component. For example, in a configuration having two sensors, like the image capture device 10B, the third sensor 31 and the components related to the third sensor 31 may be omitted.

[0055] The first sensor processing unit 19 converts the first analog data acquired by the first sensor 11 into digital data that has been subjected to image shake correction. The first sensor processing unit 19 includes a first AD conversion processing unit 12, a first sensor data processing unit 13, and a first correction unit 14. The first AD conversion processing unit 12 first converts the first analog data acquired by the first sensor 11 into first digital data. In other words, the first AD conversion processing unit 12 is an A / D converter. The first sensor data processing unit 13 converts the first digital data converted by the first AD conversion processing unit 12 into sensor data. Specifically, the sensor data may be two-dimensional information (e.g., an image). The first correction unit 14 corrects the first sensor data according to the degree of image shake detected by the hand shake detection unit 51 (described later). The first correction unit 14 outputs the corrected data to the control unit 52 as first corrected sensor data.

[0056] The configurations of the second sensor processing unit 29 and the third sensor processing unit 39 are similar to the configuration of the first sensor processing unit 19. Specifically, the configurations of the second AD conversion processing unit 22 and the third AD conversion processing unit 32 are similar to the configuration of the above-described first AD conversion processing unit 12, the configurations of the second sensor data processing unit 23 and the third sensor data processing unit 33 differ for each sensor but are similar to the above-described first sensor data processing unit 13 in that they ultimately output a two-dimensional image, and the configurations of the second correction unit 24 and the third correction unit 34 are similar to the configuration of the first correction unit 14.

[0057] Camera shake detection unit 51 detects the degree of image shake (e.g., camera shake) by analyzing information acquired by first sensor 11 (specifically, first sensor data). This detection process may use known techniques applied to electronic camera shake correction and the like. Examples of known techniques include a method of adjusting the cropping position of the entire screen, or a method of dividing an image into small blocks and changing which frame of time is used for each block. Any method may be used in this embodiment. Camera shake detection unit 51 outputs the detected degree of camera shake as a camera shake correction value to first correction unit 14, second correction unit 24, and third correction unit 34.

[0058] Here, the first correction unit 14, the second correction unit 24, and the third correction unit 34 all perform correction based on the degree of image shake detected based on the information acquired by the first sensor 11. It can also be said that the second correction unit 24 corrects the information acquired by the second sensor 21 in accordance with the degree of camera shake detected by the camera shake detection unit 51 based on the information acquired by the first sensor 11. Similarly, it can also be said that the third correction unit 34 corrects the information acquired by the third sensor 31 in accordance with the degree of camera shake detected by the camera shake detection unit 51 based on the information acquired by the first sensor 11. In the following description, the configuration of the first correction unit 14, the second correction unit 24, or the third correction unit 34 may be simply referred to as a correction unit.

[0059] Here, the number of pixels of the second sensor 21 and the number of pixels of the third sensor 31 may not match the number of pixels of the first sensor 11. In such a case, the correction unit converts the two-dimensional coordinates of the two-dimensional information acquired by the first sensor 11 into the corresponding two-dimensional coordinates of the two-dimensional information acquired by the second sensor 21 or the third sensor 31, and corrects the information acquired by the second sensor 21 or the information acquired by the third sensor 31 according to the degree of camera shake detected at each of the converted coordinates.

[0060] To explain this by taking a specific example, if the RGB sensor has a resolution four times higher than that of the ToF sensor, one pixel of the ToF sensor may be calculated as the average value of four corresponding pixels of the RGB sensor. Note that instead of the average value, other statistical calculation values such as the maximum value and the minimum value may be used.

[0061] The control unit 52 acquires the first corrected sensor data from the first sensor processing unit 19, acquires the second corrected sensor data from the second sensor processing unit 29, and acquires the third corrected sensor data from the third sensor processing unit 39. The control unit 52 transfers or stores the acquired data.

[0062] [Summary of the first embodiment] According to the embodiment described above, the imaging device 10 includes the first sensor 11 to acquire two-dimensional information (specifically, image information) having information at each coordinate on a two-dimensional plane, the second sensor 21 to acquire two-dimensional information of two-dimensional coordinates having lower resolution than the two-dimensional information (specifically, distance information, event information, polarization information, etc.) having information at each coordinate on a two-dimensional plane acquired by the first sensor 11, the camera shake detection unit 51 to detect the degree of camera shake by analyzing the information acquired by the first sensor 11, and the correction unit (e.g., either the second correction unit 24 or the third correction unit 34) to correct the information acquired by the second sensor 21 according to the degree of camera shake detected by the camera shake detection unit 51. That is, according to this embodiment, the degree of image shake is detected based on information acquired from the image sensor, and the detected degree of camera shake image shake is applied to the other sensor. By employing such a configuration, according to this embodiment, image blur can be corrected suitably without providing an expensive additional component such as a gyro sensor.

[0063] It is preferable that the first sensor 11, the second sensor 21, and the third sensor 31 are all arranged on the same optical axis. According to this embodiment, image shake can be preferably corrected by applying the image shake correction value of one of the sensors on the same optical axis to the other sensor.

[0064] Furthermore, according to the above-described embodiment, the first sensor 11 included in the imaging device 10 is an image sensor that acquires two-dimensional image information. More specifically, the first sensor 11 may be an image sensor capable of capturing RGB images or an image sensor capable of capturing monochrome images. The number of pixels of an image sensor is generally greater than that of a ToF sensor, polarization sensor, event sensor, or the like. That is, according to this embodiment, the degree of image shake is detected using information obtained by an image sensor with a large number of pixels. Therefore, according to this embodiment, image shake can be corrected with high accuracy.

[0065] Furthermore, according to the above-described embodiment, the second sensor 21 included in the imaging device 10 is any one of a ToF sensor, a polarization sensor, and an event sensor. Generally, the number of pixels in ToF sensors, polarization sensors, and event sensors is smaller than the number of pixels in an image sensor. That is, according to this embodiment, the ToF sensor, polarization sensor, and event sensor correct image shake by applying the degree of image shake detected by a sensor with a larger number of pixels. Therefore, according to this embodiment, image shake can be corrected with high accuracy.

[0066] Furthermore, according to the above-described embodiment, a correction unit (e.g., one of the second correction unit 24 and the third correction unit 34) included in the imaging device 10 converts the two-dimensional coordinates of the two-dimensional information acquired by the first sensor 11 into the corresponding two-dimensional coordinates of the two-dimensional information acquired by the second sensor 21, and corrects the information acquired by the second sensor 21 according to the degree of camera shake detected at each converted coordinate. Here, if the number of pixels of the first sensor 11 and the number of pixels of the second sensor 21 differ from each other, it may not be possible to simply apply the degree of image shake detected by the sensor with the larger number of pixels. According to this embodiment, by converting the two-dimensional coordinates of the two-dimensional information acquired by the first sensor 11 into the corresponding two-dimensional coordinates of the two-dimensional information acquired by the second sensor 21, the information acquired by the second sensor 21 can be easily corrected.

[0067] [Second embodiment] Next, a second embodiment will be described with reference to FIGS. 7 to 9. In the first embodiment described above, the processing according to this embodiment can be easily applied when the frame rates of the first sensor 11, the second sensor 21, and the third sensor 31 are the same, but the processing cannot be applied as is when the frame rates are different. However, there is a demand for applying the processing according to this embodiment even when the frame rates are different. The second embodiment describes the processing when the frame rates of the sensors are different.

[0068] 7 is a functional configuration diagram showing an example of the functional configuration of an imaging device according to the second embodiment. The functional configuration of an imaging device 10A according to the second embodiment will be described with reference to the same diagram. The imaging device 10A shown in the diagram is a modified example of the imaging device 10 described above. In the description of the imaging device 10A, components similar to those of the imaging device 10 are denoted by the same reference numerals as those of the imaging device 10, and description thereof may be omitted.

[0069] Image capture device 10A differs from image capture device 10 in that image capture device 10A further includes first correction vector calculation section 511, second hand shake vector calculation section 5121, second correction vector calculation section 5122, third hand shake vector calculation section 5131, and third correction vector calculation section 5132. Image capture device 10A also differs from image capture device 10 in that image capture device 10A includes first correction section 14A, second correction section 24A, and third correction section 34A instead of first correction section 14, second correction section 24, and third correction section 34.

[0070] First correction vector calculation unit 511 calculates a first correction vector based on a first hand shake vector indicating the degree of image shake detected by hand shake detection unit 51. The first hand shake vector is a vector indicating the amount and direction of hand shake shake for each frame image captured by first sensor 11. The first correction vector is a vector for canceling the first hand shake vector.

[0071] The first correction unit 14A corrects the first sensor data based on the first correction vector calculated by the first correction vector calculation unit 511.

[0072] Second camera shake vector calculation unit 5121 calculates a second camera shake vector based on a first camera shake vector that indicates the degree of multiple image shakes detected by camera shake detection unit 51. The second camera shake vector is similar to the first camera shake vector in that it is a vector that indicates the amount and direction of camera shake for each frame image captured by first sensor 11, but differs from the first camera shake vector in that it is a vector between frames that corresponds to the frame rate of second sensor 21.

[0073] For example, when the frame rate of the second sensor 21 is half that of the first sensor 11, the first camera shake vector is a vector indicating the difference between two frames, whereas the second camera shake vector is a vector indicating the difference between three frames, skipping one frame. In this case, the second camera shake vector calculation unit 5121 may calculate a vector indicating the difference between three frames by adding up the two first camera shake vectors detected by the camera shake detection unit 51.

[0074] Second correction vector calculation section 5122 calculates a second correction vector based on the second hand shake vector calculated by second hand shake vector calculation section 5121. The second correction vector is a vector for canceling the second hand shake vector.

[0075] The second correction unit 24A corrects the second sensor data based on the second correction vector calculated by the second correction vector calculation unit 5122.

[0076] The third camera shake vector calculation unit 5131 calculates a third camera shake vector based on the first camera shake vector, which indicates the degree of multiple image shakes, detected by the camera shake detection unit 51. The third camera shake vector is similar to the first camera shake vector in that it is a vector that indicates the amount and direction of camera shake for each frame image captured by the first sensor 11, but differs from the first camera shake vector in that it is a vector between frames according to the frame rate of the third sensor 31.

[0077] For example, if the frame rate of the third sensor 31 is one-third of the frame rate of the first sensor 11, the first camera shake vector is a vector indicating the difference between two frames, while the third camera shake vector is a vector indicating the difference between four frames, skipping two frames. In this case, the third camera shake vector calculation unit 5131 may calculate a vector indicating the difference between four frames by adding up the three first camera shake vectors detected by the camera shake detection unit 51.

[0078] Third correction vector calculation section 5132 calculates a third correction vector based on the third hand shake vector calculated by third hand shake vector calculation section 5131. The third correction vector is a vector for canceling the third hand shake vector.

[0079] The third correction unit 34A corrects the third sensor data based on the third correction vector calculated by the third correction vector calculation unit 5132.

[0080] FIG. 8 is a flowchart showing a series of processing steps performed by the imaging device according to the second embodiment. Next, a timing chart for processing using the imaging device 10A described above will be described. The example shown in FIG. 8 is an example in which the imaging device 10A includes a first sensor 11 that is an RGB sensor and a second sensor 21 that is a ToF sensor, a polarization sensor, or an event sensor. Thus, the imaging device 10A may include only two sensors, or may include three or more sensors. When three or more sensors are included, the processing of the third sensor 31 is the same as the processing of the second sensor 21, which will be described later, and processing can be performed at a timing according to the frame rate.

[0081] 8(A) shows the frame image capturing timing of the first sensor 11. In the example shown, the first sensor 11 captures six frame images, frame images i1 to i6. According to the drawing, the capturing cycle of the first sensor 11 is 1 / fr.

[0082] Figure 8(B) shows an example of a vector indicating the amount and direction of camera shake calculated from a frame image captured by the first sensor 11 (i.e., the first camera shake vector described above), and the timing for calculating the vector.

[0083] Figure 8(C) shows a vector for canceling the vector shown in Figure 8(B) and the timing for calculating this vector. The vector shown in Figure 8(C) can also be said to be the first correction vector described above.

[0084] 8(D) shows the frame image capturing timing of the second sensor 21. In the illustrated example, the second sensor 21 captures three frame images, frame images j1 to j3. That is, the frame rate of the second sensor 21 is lower than the frame rate of the first sensor 11, and can be said to be half that. According to the same figure, the capturing period of the second sensor 21 is 1 / fo. That is, it can also be said that fo = 2 × fr.

[0085] 8(E) is an example of a vector indicating the amount and direction of camera shake calculated from a frame image captured by the first sensor 11, and shows an example of a vector indicating the amount and direction of camera shake according to the frame rate of the second sensor 21 (i.e., the above-mentioned second camera shake vector), and the timing for calculating this vector. As shown in the figure, the second camera shake vector is calculated as the sum of multiple first camera shake vectors.

[0086] Fig. 8(F) shows a vector for canceling the vector shown in Fig. 8(E) and the timing for calculating this vector. The vector shown in Fig. 8(F) can also be said to be the second correction vector described above.

[0087] As shown in the figure, calculation of the first hand shake vector begins at time t2, which is the timing when capturing of frame images i1 and i2 ends. Calculation of the first correction vector begins at the timing when calculation of the first hand shake vector ends. From time t3 onwards, calculation processing of the first hand shake vector and the first correction vector continues for each frame.

[0088] Looking at the processing for the second sensor 21, after capturing the frame images for the second frame (frame image j2) is completed, a first hand shake vector is calculated for the frame image of the first sensor 11 corresponding to that frame image, and then a second hand shake vector is calculated. As shown in the figure, at time t3, after the first hand shake vector for frame image i3 corresponding to frame image j2 is calculated (between time t3 and time t4), calculation of the second hand shake vector begins. After calculation of the second hand shake vector is completed, calculation of the second correction vector begins. Similar processing is performed for the next frame and thereafter.

[0089] FIG. 9 is a functional configuration diagram showing a modified example of the functional configuration of the imaging device according to the second embodiment. The imaging device 10B shown in the figure is a modified example of the imaging device 10A described above. Here, the imaging device 10A calculates the hand shake vector for each sensor from the first hand shake vector of the first sensor 11, and calculates the correction vector for each sensor based on the hand shake vector for each sensor. However, the imaging device 10B differs from the example described with reference to FIGS. 7 and 8 in that it calculates the correction vector for each sensor by adding up the first correction vectors of the first sensors 11. It is also possible to adopt a configuration such as that shown in the imaging device 10B.

[0090] [Summary of the second embodiment] In the second embodiment described above, the first sensor 11, the second sensor 21, and the third sensor 31 all capture successive frame images at a predetermined frame rate. The frame rate of the first sensor 11 is higher than the frame rates of both the second sensor 21 and the third sensor 31. The second correction unit 24A corrects the frame image of the second sensor 21 based on the difference between multiple frames of the first sensor 11 that correspond to one frame of the second sensor 21. With this configuration, even if the frame rates of the sensors differ from each other, image blur can be corrected appropriately without the need for an expensive additional component such as a gyro sensor.

[0091] [Third embodiment] Next, a third embodiment will be described with reference to Fig. 10 and Fig. 11. The third embodiment differs from the second embodiment in that the hand shake vectors of the second sensor 21 and the third sensor 31 are calculated based on frame images captured by the first sensor 11.

[0092] 10 is a functional configuration diagram showing an example of the functional configuration of an imaging device according to the third embodiment. An example of the functional configuration of imaging device 10C will be described with reference to the same diagram. Imaging device 10C differs from imaging device 10A in that it includes a second hand shake vector calculation section 5121C instead of second hand shake vector calculation section 5121, and a third hand shake vector calculation section 5131C instead of third hand shake vector calculation section 5131.

[0093] The second hand shake vector calculation unit 5121C acquires first sensor data (RGB frame image data) from the first sensor data processing unit, and calculates the amount and direction of hand shake of the second sensor 21 from the difference between the acquired multiple frame image data. Here, the second hand shake vector calculation unit 5121C calculates the second hand shake vector based on the difference between the first frame image and the last frame image of the multiple frames of the first sensor 11 that correspond to one frame of the second sensor 21. The processing after calculating the second hand shake vector is the same as in the second embodiment, and therefore description thereof will be omitted.

[0094] The third hand shake vector calculation unit 5131C acquires first sensor data (RGB frame image data) from the first sensor data processing unit, and calculates the amount and direction of hand shake of the third sensor 31 from the difference between the acquired multiple frame image data. Here, the third hand shake vector calculation unit 5131C calculates the third hand shake vector based on the difference between the first frame image and the last frame image of the multiple frames of the first sensor 11 that correspond to one frame of the third sensor 31. The processing after calculating the third hand shake vector is the same as in the second embodiment, so a description thereof will be omitted.

[0095] FIG. 11 is a flowchart showing a series of processing steps performed by an imaging device according to a third embodiment. Next, a timing chart for processing using the imaging device 10C described above will be described. The example shown in FIG. 11 is an example in which the imaging device 10A includes a first sensor 11 that is an RGB sensor and a second sensor 21 that is a ToF sensor, a polarization sensor, or an event sensor. Thus, the imaging device 10A may include only two sensors, or may include three or more sensors. When three or more sensors are included, the processing of the third sensor 31 is similar to the processing of the second sensor 21, which will be described later, and processing can be performed at a timing according to the frame rate.

[0096] Figures 11(A) to 11(C) show the processing for the first sensor 11. Figures 11(A) to 11(C) are similar to Figures 8(A) to 8(C) described above, and therefore their explanation will be omitted.

[0097] 11(D) shows the frame image capturing timing of the second sensor 21. In the illustrated example, the second sensor 21 captures three frame images, frame images j1 to j3. That is, the frame rate of the second sensor 21 is lower than the frame rate of the first sensor 11, and can be said to be half that. According to the same figure, the capturing period of the second sensor 21 is 1 / fo. That is, it can also be said that fo = 2 × fr.

[0098] 11(E) is an example of a vector indicating the amount and direction of camera shake calculated from a frame image captured by first sensor 11, and shows an example of a vector indicating the amount and direction of camera shake according to the frame rate of second sensor 21 (i.e., the above-mentioned second camera shake vector), as well as the timing for calculating this vector. As shown in the figure, the second camera shake vector is the difference between the first frame image (frame image i1) and the last frame image (frame image i3) of multiple frames (frame image i1, frame image i2, and frame image i3) of first sensor 11, which correspond to one frame of second sensor 21 between frame image j1 and frame image j2.

[0099] Fig. 11(F) shows a vector for canceling the vector shown in Fig. 11(E) and the timing for calculating this vector. The vector shown in Fig. 11(F) can also be said to be the second correction vector described above.

[0100] As shown in the figure, after capturing the frame images for the second frame (frame image j2) has been completed, a second hand shake vector is calculated based on frame image i1, which is the first frame image of the first sensor 11 corresponding to that frame image, and frame image i3, which is the last frame image of the first sensor 11 corresponding to that frame image. As shown in the figure, calculation of the second hand shake vector begins after time t3 (between time t3 and time t4). After calculation of the second hand shake vector has been completed, calculation of the second correction vector begins. Similar processing is performed for the next frame and thereafter.

[0101] [Summary of the third embodiment] According to the third embodiment described above, the second correction unit 24A corrects the frame image of the second sensor 21 based on the difference between the first frame image and the last frame image among multiple frames of the first sensor 11 that correspond to one frame of the second sensor 21. By employing such a configuration, even if the frame rates of the sensors differ from each other, image blur can be suitably corrected without the need for an expensive additional component such as a gyro sensor.

[0102] [Handling when frame rate is not divisible] In the second and third embodiments described above, an example has been described in which the frame rate of the first sensor 11 is exactly divisible by the frame rate of the second sensor 21 or the third sensor 31. However, this embodiment is not limited to this example, and similar processing can be applied even when the frame rate is not divisible. Specifically, when the frame rate is not divisible, processing is performed based on the frame image of the first sensor 11 that corresponds to the frame image of the second sensor 21 or the third sensor 31 and is immediately preceding it.

[0103] As a specific example, a flag may be set for frame images from the first sensor 11 that are not used in the camera shake detection processing of the second sensor 21 or the third sensor 31, and the flag may be erased when the frame images are used for processing, and processing may be performed based on the frame images for which the flag is set.

[0104] As another example, the hand shake vector of the first sensor 11 may be summed up for each frame of the first sensor 11, and once the summed vector is used for the hand shake detection process of the second sensor 21 or the third sensor 31, it may be reset and summed up again.

[0105] [Internal configuration] FIG. 12 is a block diagram showing an example of the internal configuration of an imaging device according to an embodiment. At least some of the functions of the imaging device 10 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. All or part of the functional units included in the imaging device 10 may be realized using hardware such as an ASIC, a PLD, or an FPGA. All or part of the functional units may be realized by a combination of software and hardware.

[0106] Note that all or part of the functions of each device provided in the imaging device and imaging method in the above-described embodiments may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0107] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0108] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate.

[0109] 6, the first sensor 11 and the first AD conversion processing unit 12 are described separately, but the first sensor 11 may include the first AD conversion processing unit 12. The same applies to the second sensor 21 and the second AD conversion processing unit 22, and the third sensor 31 and the third AD conversion processing unit 32. [Explanation of symbols]

[0110] 10...imaging device, 11...first sensor, 12...first AD conversion processing unit, 13...first sensor data processing unit, 14...first correction unit, 21...second sensor, 22...second AD conversion processing unit, 23...second sensor data processing unit, 24...second correction unit, 31...third sensor, 32...third AD conversion processing unit, 33...third sensor data processing unit, 34...third correction unit, 51...hand shake detection unit, 52...control unit

Claims

1. a first sensor for acquiring two-dimensional information having information at each coordinate on a two-dimensional plane; a second sensor having information at each coordinate on a two-dimensional plane and configured to acquire two-dimensional information of two-dimensional coordinates having a lower resolution than the two-dimensional coordinates of the two-dimensional information acquired by the first sensor; a camera shake detection unit that detects the degree of camera shake by analyzing information acquired by the first sensor; a correction unit that corrects the information acquired by the second sensor in accordance with the degree of camera shake detected by the camera shake detection unit; An imaging device comprising:

2. the first sensor is an image sensor that acquires two-dimensional image information; The imaging device according to claim 1 .

3. the second sensor is any one of a Time Of Flight (ToF) sensor, a polarization sensor, and an event sensor; 3. The imaging device according to claim 1.

4. the correction unit converts two-dimensional coordinates of the two-dimensional information acquired by the first sensor into corresponding coordinates of the two-dimensional coordinates of the two-dimensional information acquired by the second sensor, and corrects the information acquired by the second sensor according to the degree of camera shake detected at each of the converted coordinates.

3. The imaging device according to claim 1.

5. the first sensor and the second sensor both capture successive frame images at a predetermined frame rate; the frame rate of the first sensor is greater than the frame rate of the second sensor; the correction unit corrects the frame image of the second sensor based on differences between a plurality of frames of the first sensor corresponding to one frame of the second sensor.

3. The imaging device according to claim 1.

6. the correction unit corrects the frame image of the second sensor based on a difference between a first frame image and a last frame image among a plurality of frames of the first sensor corresponding to one frame of the second sensor; The imaging device according to claim 5 .

7. a first acquisition step of acquiring two-dimensional information having information at each coordinate on a two-dimensional plane; a second acquisition step of acquiring two-dimensional information of two-dimensional coordinates having information at each coordinate on a two-dimensional plane and having a resolution lower than that of the two-dimensional coordinates of the two-dimensional information acquired by the first acquisition step; a camera shake detection step of detecting the degree of camera shake by analyzing the information acquired in the first acquisition step; a correction step of correcting the information acquired in the second acquisition step in accordance with the degree of camera shake detected in the camera shake detection step; An imaging method comprising:

Images