Photography device and photography method

Abstract

The present invention provides a photography device having a global shutter function and other photography functions, and a photography method. The photography device comprises: a plurality of light receiving portions, which convert light into electrical signals; a plurality of switches, which are respectively connected to the plurality of light receiving portions; a control portion, which is connected to the plurality of switches and controls the acquisition of the electrical signals so as to perform global shutter photography by means of the plurality of light receiving portions; and at least two memories, which are connected to the control portion, wherein by switching the types of the signals stored in the at least two memories, the control portion switches a photography function performed together with the global shutter photography.

Description

Photography device and photography method Technical Field

[0001] The present invention relates to a photographing device and a photographing method. Background Art

[0002] Cameras and other imaging devices installed in smartphones are required to have higher functionality. One such function is a global shutter function that captures the shape of a moving object with minimal distortion.

[0003] For example, the following Non-Patent Document 1 describes research on an image sensor that provides a memory element for each pixel and performs global shutter imaging.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent document 1: Seung-Sik Kim et al. "3-Layer Stacked Voltage-Domain Global Shutter CMOS Image Sensor with 1.8μm-Pixel-Pitch", 2022International Electron Devices Meeting (IEDM), 2022

[0007] Summary of the Invention

[0008] Problems to be solved by the invention

[0009] Camera shutter modes are divided into global shutter and rolling shutter. Global shutter means that the camera's sensor reads information from all pixels simultaneously, rather than row by row as with a rolling shutter. Therefore, global shutter eliminates the jello effect when capturing high-speed objects, eliminating the image distortion associated with a rolling shutter.

[0010] Using the image sensor described in Non-Patent Document 1, it is possible to provide an imaging device having a global shutter function. However, it is difficult to provide an imaging device having multiple functions as required for cameras mounted on smartphones and the like.

[0011] Therefore, the present invention provides a photographing device and a photographing method having both a global shutter function and other photographing functions.

[0012] Solutions for solving problems

[0013] A photographing device according to one embodiment of the present invention comprises: a plurality of light receiving units that convert light into electrical signals; a plurality of switches that are respectively connected to the plurality of light receiving units; a control unit that is connected to the plurality of switches and controls the acquisition of electrical signals to perform global shutter photography through the plurality of light receiving units; and at least two memories that are connected to the control unit, and the control unit switches the photography function performed together with the global shutter photography by switching the types of signals held in the at least two memories.

[0014] According to this aspect, by switching the type of signal held in the memory, it is possible to switch the imaging function to be executed together with the global shutter imaging, and it is possible to provide other imaging functions while providing the global shutter function.

[0015] In the above scheme, at least two memories may include a first memory and a second memory, the first memory stores the signal obtained when global shutter exposure is performed on multiple light receiving parts from the first moment to the second moment, and the second memory stores the signal obtained when global shutter exposure is performed on multiple light receiving parts from the first moment to the third moment.

[0016] According to this solution, global shutter photography can be performed using two different exposure times.

[0017] In the above scheme, at least two memories may include a first memory and a second memory. The control unit obtains signals obtained when global shutter exposure is performed on multiple light receiving parts at a predetermined exposure time at a first moment and a second moment respectively. The first memory maintains the signal obtained at the first moment, and the second memory maintains the signal obtained at the second moment.

[0018] According to this scheme, signals obtained when global shutter exposure is performed for a predetermined exposure time are acquired twice, and correlated multi-sampling is used, thereby reducing noise caused by reading.

[0019] In the above solution, the at least two memories may include a first memory and a second memory, the first memory storing a signal obtained by adding signals respectively obtained from the plurality of light receiving parts, and the second memory storing a signal obtained from a part of the plurality of light receiving parts.

[0020] According to this aspect, the spatial high-frequency component can be estimated based on the results of processing the signals held in the first memory and the second memory, and an image with a higher resolution than that obtained from the added signal can be obtained.

[0021] In the above scheme, at least two memories may include a first memory and a second memory, and the multiple light receiving parts may include a first light receiving part and a second light receiving part for each pixel. The first memory maintains the signal obtained from the first light receiving part, and the second memory maintains the signal obtained by adding the signal obtained from the first light receiving part and the signal obtained from the second light receiving part.

[0022] According to this solution, the phase difference between the signal stored in the first memory and the signal stored in the second memory is calculated, thereby providing global shutter shooting and image plane phase difference autofocus function.

[0023] In the above scheme, there may also be at least two memories including a first memory, a second memory, and a third memory, the first memory stores the signal obtained when the global shutter exposure is performed on multiple light receiving parts from the first moment to the second moment, the second memory stores the signal obtained when the global shutter exposure is performed on multiple light receiving parts from the first moment to the third moment, and the third memory stores the signal obtained when the global shutter exposure is performed on multiple light receiving parts from the first moment to the fourth moment.

[0024] According to this scheme, global shutter shooting can be performed using three different exposure times.

[0025] In the above scheme, at least two memories may include a first memory, a second memory, and a third memory, and the multiple light receiving parts include a first light receiving part and a second light receiving part for each pixel. The first memory maintains a signal obtained from the first light receiving part included in a part of the pixels, the second memory maintains a signal obtained by adding the signal obtained from the first light receiving part included in a part of the pixels and the signal obtained from the second light receiving part included in a part of the pixels, and the third memory maintains a signal obtained by adding the signal obtained from the first light receiving part and the signal obtained from the second light receiving part.

[0026] According to this solution, it is possible to provide both global shutter shooting and image plane phase difference autofocus functions while also estimating spatial high-frequency components, and to obtain an image with a higher resolution than an image obtained from the summed signal.

[0027] In the above scheme, there may also be a lateral overflow collective capacitor, at least two memories including a first memory, a second memory, and a third memory, the first memory maintains the signal obtained when the lateral overflow collective capacitor is initialized, the second memory maintains the signal obtained from multiple light receiving parts, and the third memory maintains the overflow signal accumulated in the lateral overflow collective capacitor.

[0028] According to this solution, global shutter photography with a wider dynamic range can be performed.

[0029] In the above scheme, there may also be at least two memories including a first memory, a second memory, a third memory, and a fourth memory. The control unit obtains the signals obtained when the multiple light receiving parts are initialized at the first moment and the second moment respectively, and obtains the signals obtained when the multiple light receiving parts are globally shutter exposed for a predetermined exposure time at the third moment and the fourth moment respectively. The first memory maintains the signal obtained at the first moment, the second memory maintains the signal obtained at the second moment, the third memory maintains the signal obtained at the third moment, and the fourth memory maintains the signal obtained at the fourth moment.

[0030] According to this scheme, a signal obtained when global shutter exposure is performed for a predetermined exposure time is acquired twice, and respective reset levels are also acquired twice, thereby making it possible to further reduce noise using correlated multi-sampling.

[0031] In the above scheme, there may also be at least two memories including a first memory, a second memory, a third memory, and a fourth memory, the first memory maintains a signal obtained with a first gain when the multiple light receiving parts are initialized, the second memory maintains a signal obtained with a second gain when the multiple light receiving parts are initialized, the third memory maintains a signal obtained when the signals obtained from the multiple light receiving parts are obtained with the first gain, and the fourth memory maintains a signal obtained when the signals obtained from the multiple light receiving parts are obtained with the second gain.

[0032] According to this solution, signals are acquired at two gains, thereby enabling global shutter photography with a wider dynamic range and less noise.

[0033] In the above aspect, the control unit may acquire the signal of each frame at a predetermined cycle regardless of the type of the signals held in the at least two memories.

[0034] According to this aspect, even when the imaging function used together with global shutter imaging is switched, it is possible to prevent image loss.

[0035] Another embodiment of the present invention is a shooting method which is a control method performed by a shooting device, wherein the shooting device comprises: a plurality of light receiving parts which convert light into electrical signals; a plurality of switches which are respectively connected to the plurality of light receiving parts; a control part which is connected to the plurality of switches; and at least two memories which are connected to the control part. The shooting method includes: controlling the acquisition of electrical signals to perform global shutter shooting through the plurality of light receiving parts; and switching the shooting function performed together with the global shutter shooting by switching the types of signals held in at least two memories.

[0036] According to this aspect, by switching the type of signal held in the memory, it is possible to switch the imaging function to be executed together with the global shutter imaging, and it is possible to provide other imaging functions while providing the global shutter function.

[0037] Effects of the Invention

[0038] According to the present invention, it is possible to provide an imaging device and an imaging method having both a global shutter function and other imaging functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG. 1 is a diagram schematically showing a circuit configuration of an imaging device according to a first embodiment of the present invention.

[0040] FIG. 2 is a flowchart of processing executed by the imaging device according to this embodiment.

[0041] FIG3 is a flowchart of a process executed by the imaging device according to the second embodiment of the present invention.

[0042] FIG. 4 is a flowchart of a process executed by the imaging device according to the third embodiment of the present invention.

[0043] FIG. 5 is a diagram schematically showing a circuit configuration of an imaging device according to a fourth embodiment of the present invention.

[0044] FIG6 is a flowchart of the process executed by the imaging device according to this embodiment.

[0045] FIG7 is a diagram schematically showing a circuit configuration of an imaging device according to a fifth embodiment of the present invention.

[0046] FIG8 is a flowchart of the process executed by the imaging device according to this embodiment.

[0047] FIG. 9 is a flowchart of processing executed by the imaging device according to the sixth embodiment of the present invention.

[0048] FIG. 10 is a diagram schematically showing a circuit configuration of an imaging device according to a seventh embodiment of the present invention.

[0049] FIG11 is a flowchart of the process executed by the imaging device according to this embodiment.

[0050] FIG. 12 is a diagram schematically showing a circuit configuration of an imaging device according to an eighth embodiment of the present invention.

[0051] FIG13 is a flowchart of the process executed by the imaging device according to this embodiment.

[0052] FIG. 14 is a diagram schematically showing a circuit configuration of an imaging device according to a ninth embodiment of the present invention.

[0053] FIG15 is a flowchart of the process executed by the imaging device according to this embodiment.

[0054] FIG. 16 is a timing chart showing a case where a plurality of functions are switched and executed by the imaging device according to the embodiment of the present invention. DETAILED DESCRIPTION

[0055] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same elements will be denoted by the same reference numerals, and duplicate descriptions will be omitted.

[0056] [First embodiment]

[0057] FIG1 is a diagram schematically illustrating the circuit configuration of an imaging device 10 according to a first embodiment of the present invention. The diagram schematically illustrates an image sensor 100 included in the imaging device 10. The image sensor 100 is, for example, a CMOS image sensor, comprising a plurality of pixels arranged two-dimensionally and having a plurality of light-receiving units that convert light into electrical signals. In this embodiment, the light-receiving units are formed by photodiodes. The diagram schematically illustrates a first photodiode PD1, a second photodiode PD2, a third photodiode PD3, and a fourth photodiode PD4 as the plurality of light-receiving units.

[0058] In the shooting device 10 of this embodiment, four pixels arranged in a 2×2 manner are grouped as a unit pixel group, but this is not limited to this. For example, three pixels arranged in a 3×1 manner, or eight pixels arranged in a 4×2 manner, or nine pixels arranged in a 3×3 manner, or sixteen pixels arranged in a 4×4 manner may be grouped as a unit pixel group.

[0059] The image sensor 100 includes a plurality of switches connected to the plurality of light-receiving elements. In this embodiment, the switches are formed by transfer transistors. In the figure, a first transfer transistor TX1, a second transfer transistor TX2, a third transfer transistor TX3, and a fourth transfer transistor TX4 are shown as the plurality of switches.

[0060] The image sensor 100 includes a control unit CTR, which is connected to multiple switches and controls the acquisition of electrical signals for global shutter photography via multiple photodetectors. The control unit CTR controls pixel initialization and signal reading, and transmits signals acquired by the photodetectors to multiple memories. The control unit CTR can also read the signals acquired by the photodetectors as charges, amplify voltage or current, or perform A / D conversion.

[0061] Image sensor 100 includes at least two memories connected to a control unit CTR. In this figure, the at least two memories include a first memory M1, a second memory M2, and a third memory M3. The memories may be charge-storing memories like CCDs, voltage-storing memories like dielectric capacitors, or digital memories such as SRAM or DRAM. The signals stored in the memories are read from the pixel array by a memory control unit MCTR. The memory control unit MCTR also initializes the memories to acquire the next signal.

[0062] The camera 10 of this embodiment includes multiple light receiving units, multiple switches, a control unit, and at least two memories as a pixel group. For example, if four pixels arranged in a 2×2 pattern are grouped as a unit pixel group, the camera 10 includes four light receiving units, four switches connected to the four light receiving units, a control unit connected to the four switches, and at least two memories connected to the control unit, forming a pixel group. The pixel groups can also be arranged in a two-dimensional array.

[0063] Based on a command from a main control unit included in the imaging device 10, the control unit CTR switches the type of signals stored in at least two memories, thereby switching the imaging functions performed in conjunction with global shutter imaging. Examples of imaging functions performed in conjunction with global shutter imaging include multiple-exposure imaging, noise reduction using correlated multisampling, super-resolution based on spatial high-frequency component estimation, image-plane phase difference autofocus, and dynamic range expansion using lateral overflow capacitors or multiple gains. The imaging device 10 of this embodiment switches the type of signals stored in the memories to switch the imaging functions performed in conjunction with global shutter imaging, and can provide other imaging functions simultaneously with the global shutter function. This makes it possible to provide an imaging device that has both global shutter imaging and multiple functions, such as those required in cameras installed in smartphones.

[0064] Fig. 2 is a flowchart of the processing executed by the imaging device 10 of this embodiment. This figure shows the processing performed in one frame. The imaging device 10 first resets the plurality of light receiving units (S10) and starts global shutter exposure at a first moment (S11).

[0065] The imaging device 10 stores the reset level signal of the control unit CTR in the third memory M3 (S12). Since the reset level of the control unit CTR is affected by thermal noise and varies between shots, storing the reset level signal and subtracting it from the captured signal can reduce noise caused by thermal noise.

[0066] The imaging device 10 then stores the signals obtained from the global shutter exposure performed from the first moment to the second moment in the first memory M1 (S13). Specifically, the first memory M1 stores the signals obtained from the global shutter exposure performed on the multiple light-receiving units from the first moment to the second moment. The second moment is later than the first moment.

[0067] Furthermore, the imaging device 10 stores the signals obtained from performing global shutter exposure from the first moment to the third moment in the second memory M2 (S14). Specifically, the second memory M2 stores the signals obtained from performing global shutter exposure on the plurality of light-receiving units from the first moment to the third moment. Here, the third moment is a moment later than the second moment.

[0068] The camera 10 performs global shutter photography at two exposure times (S15). Here, the camera 10 may also perform noise reduction processing based on the reset level signal stored in the third memory M3. According to the camera 10 of this embodiment, global shutter photography can be performed using two different exposure times. For example, the time from the first moment to the second moment is set to be longer than the time from the second moment to the third moment, thereby enabling the use of the global shutter to obtain a signal with a relatively long exposure time and a signal with a relatively short exposure time.

[0069] [Second embodiment]

[0070] FIG3 is a flowchart of the processing performed by the camera 10 according to the second embodiment of the present invention. The circuit structure of the camera 10 according to this embodiment can be the same as that of the first embodiment. This figure shows the processing performed within a single frame. The camera 10 first resets the multiple light receiving units ( S20 ) and performs a global shutter exposure for a predetermined exposure time ( S21 ).

[0071] The imaging device 10 stores the reset level signal of the control unit CTR in the third memory M3 (S22). Then, the control unit CTR obtains signals obtained when a global shutter exposure is performed on multiple light receiving units for a predetermined exposure time at a first moment and a second moment, respectively. The control unit CTR stores the signal read at the first moment in the first memory M1 (S23) and the signal read at the second moment in the second memory M2 (S24).

[0072] The camera 10 applies correlated multi-sampling to the signals acquired at the first and second moments to perform global shutter photography (S25). The camera 10 may further perform noise reduction processing based on the reset level signal stored in the third memory M3. According to this embodiment, the camera 10 acquires the signal twice, obtained when performing global shutter exposure for a predetermined exposure time, and utilizes correlated multi-sampling to reduce noise caused by reading.

[0073] [Third embodiment]

[0074] FIG4 is a flowchart of the processing performed by the camera 10 according to the third embodiment of the present invention. The circuit structure of the camera 10 according to this embodiment can be the same as that of the first embodiment. This figure illustrates the processing performed within a single frame. The camera 10 first resets the multiple light-receiving units ( S30 ) and performs a global shutter exposure ( S31 ).

[0075] The camera 10 stores the reset level signal of the control unit CTR in the third memory M3 (S32) and stores the signal obtained by adding the signals obtained from the multiple light receiving units in the first memory M1 (S33). Furthermore, the camera 10 stores the signal obtained from some of the multiple light receiving units in the second memory M2 (S34). For example, the camera 10 may store the signal obtained from one of the four pixels in the second memory M2. The camera 10 then performs global shutter photography for estimating spatial high-frequency components (S35).

[0076] Hereinafter, the signal maintained in the first memory M1 is referred to as a full-pixel binning signal, the signal maintained in the second memory M2 is referred to as a first partial pixel binning signal, and the signal obtained by subtracting the first partial pixel binning signal from the full-pixel binning signal is referred to as a second partial pixel binning signal. For example, the full-pixel binning signal may be a signal obtained by adding signals obtained from four pixels. In addition, the first partial pixel binning signal may be a signal obtained from one of the four pixels. In this case, the second partial pixel binning signal may be a signal obtained by subtracting the first partial pixel binning signal from the full-pixel binning signal, that is, a signal obtained by adding signals obtained from three of the four pixels.

[0077] The camera 10 analyzes the frequency characteristics of the image signal of the region formed by the unit pixel group based on the mutual correlation between the first partial pixel binning signal and the second partial pixel binning signal. For example, when the mutual correlation between the first partial pixel binning signal and the second partial pixel binning signal is below a predetermined threshold, the camera 10 can determine that the region formed by the unit pixel group contains a large number of high-frequency components. Here, the predetermined threshold can be pre-set based on, for example, the type and performance of the lens and image sensor, the subject, the surrounding environment, and other shooting conditions, or it can be changeable. In addition, the predetermined threshold can also be set using machine learning based on the type and performance of the lens and image sensor, the subject, the surrounding environment, and other shooting conditions. For example, regarding whether moiré fringes are generated, the first partial pixel binning signal, the second pixel binning signal, the full pixel binning signal, and the presence or absence of moiré fringes in the region formed by the unit pixel group can be used as training data for teacher learning to create a machine learning model, and the model can be used to determine the presence or absence of moiré fringes.

[0078] According to the imaging device 10 of this embodiment, the spatial high-frequency component can be estimated based on the results of processing the signals stored in the first memory M1 and the second memory M2, and an image with a higher resolution than an image obtained from the added signal can be obtained.

[0079] [Fourth embodiment]

[0080] FIG5 is a diagram showing an overview of the circuit structure of the imaging device 10 according to the fourth embodiment of the present invention. In this figure, an overview of the image sensor 100 included in the imaging device 10 is shown. The image sensor 100 includes a plurality of pixels arranged in two dimensions. In the imaging device 10 according to this embodiment, four pixels arranged in a 2×2 manner are used as a unit pixel group, and each pixel includes a first light receiving unit and a second light receiving unit. That is, the imaging device 10 according to this embodiment includes two light receiving units for each pixel that convert light into an electrical signal. In this embodiment, the light receiving unit is composed of photodiodes, including a first left photodiode PD1L, a first right photodiode PD1R, a second left photodiode PD2L, a second right photodiode PD2R, a third left photodiode PD3L, a third right photodiode PD3R, a fourth left photodiode PD4L, and a fourth right photodiode PD4R.

[0081] The image sensor 100 includes multiple switches connected to multiple photoreceiving units. The figure shows a first left transfer transistor TX1L connected to a first left photodiode PD1L, a first right transfer transistor TX1R connected to a first right photodiode PD1R, a second left transfer transistor TX2L connected to a second left photodiode PD2L, a second right transfer transistor TX2R connected to a second right photodiode PD2R, a third left transfer transistor TX3L connected to a third left photodiode PD3L, a third right transfer transistor TX3R connected to a third right photodiode PD3R, a fourth left transfer transistor TX4L connected to a fourth left photodiode PD4L, and a fourth right transfer transistor TX4R connected to a fourth right photodiode PD4R. The multiple switches are connected to a control unit CTR.

[0082] In the imaging device 10 of this embodiment, the first memory M1 stores the signal obtained from the first light receiving unit (the first left photodiode PD1L, the second left photodiode PD2L, the third left photodiode PD3L, and the fourth left photodiode PD4L). The second memory M2 stores the signal obtained by adding the signal obtained from the first light receiving unit and the signal obtained from the second light receiving unit (the first right photodiode PD1R, the second right photodiode PD2R, the third right photodiode PD3R, and the fourth right photodiode PD4R).

[0083] Fig. 6 is a flowchart of the process executed by the imaging device 10 of this embodiment. This figure shows the process performed in one frame. The imaging device 10 first resets the plurality of light receiving units (S40) and performs global shutter exposure (S41).

[0084] The camera 10 stores the reset level signal of the control unit CTR in the third memory M3 (S42) and stores the signal obtained from the left light receiving unit in the first memory M1 (S43). In the case of the unit pixel group shown in Figure 5, the left light receiving unit includes a first left photodiode PD1L, a second left photodiode PD2L, a third left photodiode PD3L, and a fourth left photodiode PD4L. The first memory M1 can store a signal obtained by adding the signals obtained by the first left photodiode PD1L, the second left photodiode PD2L, the third left photodiode PD3L, and the fourth left photodiode PD4L.

[0085] Furthermore, the camera 10 stores the signals obtained from the left and right light receiving sections in a second memory M2 (S44). In the case of the unit pixel group shown in FIG5 , the right light receiving section includes a first right photodiode PD1R, a second right photodiode PD2R, a third right photodiode PD3R, and a fourth right photodiode PD4R. The second memory M2 can store a signal obtained by adding the signals obtained by the first left photodiode PD1L, the first right photodiode PD1R, the second left photodiode PD2L, the second right photodiode PD2R, the third left photodiode PD3L, the third right photodiode PD3R, the fourth left photodiode PD4L, and the fourth right photodiode PD4R.

[0086] The camera 10 then performs image plane phase difference autofocus and global shutter photography (S45). The camera 10 subtracts the signal stored in the first memory M1 from the signal stored in the second memory M2, calculates the signal obtained from the right light receiving unit, and performs image plane phase difference autofocus based on the signal obtained from the left light receiving unit and the signal obtained from the right light receiving unit.

[0087] According to the shooting device 10 of this embodiment, by calculating the phase difference between the signal obtained by subtracting the signal held in the first memory from the signal held in the second memory and the signal held in the first memory, it is possible to provide an image plane phase difference automatic focusing function while providing global shutter shooting.

[0088] The camera 10 of this embodiment can also perform rolling shutter photography. In this case, the control unit CTR can independently read out each photodiode and output the readout signal to a circuit outside the pixel array, rather than storing it in memory. If the number of unit pixel groups is N, and these are formed by grouping four pixels in a 2×2 arrangement, the number of pixels used in rolling shutter photography becomes 4×N.

[0089] [Fifth embodiment]

[0090] FIG7 is a diagram schematically illustrating the circuit configuration of an imaging device 10 according to a fifth embodiment of the present invention. This diagram schematically illustrates the image sensor 100 included in the imaging device 10. The imaging device 10 of this embodiment includes a first memory M1, a second memory M2, a third memory M3, and a fourth memory M4.

[0091] Fig. 8 is a flowchart of the process executed by the imaging device 10 of this embodiment. This figure shows the process performed in one frame. The imaging device 10 first resets the plurality of light receiving units (S50).

[0092] The camera 10 begins global shutter exposure at a first moment (S51). Subsequently, the camera 10 stores the reset level signal of the control unit CTR in the fourth memory M4 (S52). Furthermore, the camera 10 stores the signal obtained from the global shutter exposure from the first moment to the second moment in the first memory M1 (S53). In other words, the first memory M1 stores the signal obtained from the global shutter exposure of multiple light-receiving units from the first moment to the second moment. The second moment is later than the first moment.

[0093] Furthermore, the imaging device 10 stores the signals obtained from the global shutter exposure performed from the first moment to the third moment in the second memory M2 (S54). Specifically, the second memory M2 stores the signals obtained from the global shutter exposure performed on the plurality of light receiving units from the first moment to the third moment. Here, the third moment is a moment later than the second moment.

[0094] Furthermore, the imaging device 10 stores the signals obtained from the global shutter exposure performed from the first moment to the fourth moment in the third memory M3 (S55). Specifically, the third memory M3 stores the signals obtained from the global shutter exposure performed on the plurality of light receiving units from the first moment to the fourth moment. Here, the fourth moment is a moment later than the third moment.

[0095] The photographing device 10 performs global shutter photography at three exposure times (S56). Here, the photographing device 10 may further perform noise reduction processing based on the reset level signal maintained in the fourth memory M4. According to the photographing device 10 of this embodiment, global shutter photography can be performed using three different exposure times. For example, the time from the first moment to the second moment is set to be longer than the time from the second moment to the third moment, and the time from the second moment to the third moment is set to be longer than the time from the third moment to the fourth moment, so that a relatively long exposure signal, a medium exposure signal, and a relatively short exposure signal can be obtained using the global shutter.

[0096] [Sixth embodiment]

[0097] Figure 9 is a flowchart of the processing performed by the camera 10 of the sixth embodiment of the present invention. The circuit structure of the camera 10 of this embodiment can be the same as the circuit structure of the fifth embodiment. In this figure, the processing performed in one frame is shown. The camera 10 first resets the multiple light receiving parts (S60) and performs global shutter exposure at a predetermined exposure time (S61). In addition, the camera 10 maintains the reset level signal of the control unit CTR read out at the first moment in the first memory M1 (S62), and maintains the reset level signal of the control unit CTR read out at the second moment in the second memory M2 (S63). Here, the second moment is a moment later than the first moment.

[0098] The imaging device 10 stores the signals read from the plurality of light receiving units at the third time in the third memory M3 (S64). Furthermore, the imaging device 10 stores the signals read from the plurality of light receiving units at the fourth time in the fourth memory M4 (S65). Here, the third time is a time later than the second time, and the fourth time is a time later than the third time.

[0099] In summary, the imaging device 10 acquires signals obtained when the multiple light receiving units are initialized at the first and second moments, respectively, and acquires signals obtained when the multiple light receiving units are subjected to global shutter exposure for a predetermined exposure time at the third and fourth moments, respectively. The first memory M1 stores the signal acquired at the first moment, the second memory M2 stores the signal acquired at the second moment, the third memory M3 stores the signal acquired at the third moment, and the fourth memory M4 stores the signal acquired at the fourth moment.

[0100] The imaging device 10 uses the reset level signals acquired at the first and second times for noise reduction processing and applies correlated multi-sampling to the signals acquired at the third and fourth times to perform global shutter photography (S66). According to the imaging device 10 of this embodiment, by acquiring the signals obtained by performing global shutter exposure for a predetermined exposure time twice and also acquiring the reset levels for each signal twice, the correlated multi-sampling technique can further reduce noise.

[0101] [Seventh embodiment]

[0102] Figure 10 is a diagram schematically illustrating the circuit configuration of a camera 10 according to a seventh embodiment of the present invention. This diagram schematically illustrates the image sensor 100 included in the camera 10. In the camera 10 of this embodiment, four pixels arranged in a 2×2 pattern are grouped as unit pixel groups, and each pixel includes a first light receiving unit and a second light receiving unit. In other words, the camera 10 of this embodiment includes two light receiving units per pixel that convert light into electrical signals. In this embodiment, the light receiving units are composed of photodiodes, including a first left photodiode PD1L, a first right photodiode PD1R, a second left photodiode PD2L, a second right photodiode PD2R, a third left photodiode PD3L, a third right photodiode PD3R, a fourth left photodiode PD4L, and a fourth right photodiode PD4R. Furthermore, the camera 10 of this embodiment includes a first memory M1, a second memory M2, a third memory M3, and a fourth memory M4.

[0103] In the imaging device 10 of this embodiment, the first memory M1 stores signals obtained from the first light receiving units (e.g., the first left photodiode PD1L, the second left photodiode PD2L, and the third left photodiode PD3L) included in a portion of pixels. The second memory M2 stores a signal obtained by adding the following two signals: the signals obtained from the first light receiving units (e.g., the first left photodiode PD1L, the second left photodiode PD2L, and the third left photodiode PD3L) included in a portion of pixels, and the signals obtained from the second light receiving units (e.g., the first right photodiode PD1R, the second right photodiode PD2R, and the third right photodiode PD3R) included in a portion of pixels. The third memory M3 maintains a signal obtained by adding the following two signals: a signal obtained from the first light receiving part (the first left photodiode PD1L, the second left photodiode PD2L, the third left photodiode PD3L, and the fourth left photodiode PD4L), and a signal obtained from the second light receiving part (the first right photodiode PD1R, the second right photodiode PD2R, the third right photodiode PD3R, and the fourth right photodiode PD4R).

[0104] Fig. 11 is a flowchart of the process executed by the imaging device 10 of this embodiment. This figure shows the process performed in one frame. The imaging device 10 first resets the plurality of light receiving units (S70) and performs global shutter exposure (S71).

[0105] The camera 10 stores the reset level signal of the control unit CTR in the fourth memory M4 (S72) and stores the signal obtained from the left light receiving portion included in a portion of the pixels in the first memory M1 (S73). In the case of the unit pixel group shown in FIG10 , the left light receiving portion included in the portion of the pixels may be the first left photodiode PD1L, the second left photodiode PD2L, and the third left photodiode PD3L. The first memory M1 may store a signal obtained by adding the signals obtained by the first left photodiode PD1L, the second left photodiode PD2L, and the third left photodiode PD3L.

[0106] Furthermore, the imaging device 10 stores the signals obtained from the left and right light-receiving portions included in a portion of the pixels in the second memory M2 (S74). In the case of the unit pixel group shown in FIG10 , the right light-receiving portion included in the portion of the pixels may be the first right photodiode PD1R, the second right photodiode PD2R, and the third right photodiode PD3R. The second memory M2 may store a signal obtained by adding the signals obtained from the first left photodiode PD1L, the first right photodiode PD1R, the second left photodiode PD2L, the second right photodiode PD2R, the third left photodiode PD3L, and the third right photodiode PD3R.

[0107] Then, the imaging device 10 stores the signals obtained from the left and right light receiving units in the third memory M3 (S75). The third memory M3 can store a signal obtained by adding the signals obtained by the first left photodiode PD1L, the first right photodiode PD1R, the second left photodiode PD2L, the second right photodiode PD2R, the third left photodiode PD3L, the third right photodiode PD3R, the fourth left photodiode PD4L, and the fourth right photodiode PD4R.

[0108] Thereafter, the imaging device 10 estimates the spatial high-frequency component, performs image plane phase difference autofocus, and performs global shutter imaging ( S76 ).

[0109] The shooting device 10 can subtract the signal maintained in the first memory M1 from the signal maintained in the second memory M2, calculate the signal obtained from the right light receiving part included in a part of the pixels, and perform image plane phase difference automatic focusing based on the signal obtained from the left light receiving part included in a part of the pixels and the signal obtained from the right light receiving part included in a part of the pixels.

[0110] Hereinafter, the signal held in the third memory M3 is referred to as a full-pixel-binning signal, the signal held in the second memory M2 is referred to as a first partial pixel-binning signal, and the signal obtained by subtracting the first partial pixel-binning signal from the full-pixel-binning signal is referred to as a second partial pixel-binning signal. For example, the full-pixel-binning signal may be a signal obtained by adding signals obtained from four pixels. Alternatively, the first partial pixel-binning signal may be a signal obtained by adding signals obtained from three of the four pixels. In this case, the second partial pixel-binning signal may be a signal obtained by subtracting the first partial pixel-binning signal from the full-pixel-binning signal, that is, a signal obtained from one of the four pixels.

[0111] The imaging device 10 analyzes the frequency characteristics of the image signal in the region consisting of the unit pixel group based on the correlation between the first partial pixel-binning signal and the second partial pixel-binning signal, and estimates the spatial high-frequency component as in the third embodiment.

[0112] According to the imaging device 10 of this embodiment, while providing both global shutter imaging and image plane phase difference autofocus functions, it is possible to further estimate spatial high-frequency components and obtain an image with a higher resolution than an image obtained from the summed signal.

[0113] [Eighth Embodiment]

[0114] FIG12 is a diagram schematically illustrating the circuit configuration of an imaging device 10 according to an eighth embodiment of the present invention. This diagram schematically illustrates an image sensor 100 included in the imaging device 10. The imaging device 10 of this embodiment includes a lateral overflow capacitance (LOFIC). The lateral overflow capacitance (LOFIC) accumulates electrons that overflow from a photodiode when exposed to strong light to a level exceeding the saturation capacitance of the photodiodes included in the unit pixel group.

[0115] The imaging device 10 of this embodiment includes a first memory M1 , a second memory M2 , a third memory M3 , and a fourth memory M4 .

[0116] Fig. 13 is a flowchart of the process executed by the imaging device 10 of this embodiment. This figure shows the process performed in one frame. The imaging device 10 first resets the plurality of light receiving units (S80) and performs global shutter exposure (S81).

[0117] The camera 10 stores the reset level signal of the control unit CTR in the fourth memory M4 (S82). The camera 10 stores the signal obtained by adding the signals obtained from the multiple light-receiving units through global shutter exposure in the second memory M2 (S83). Furthermore, the camera 10 stores the overflow signal accumulated in the lateral overflow integrated capacitor LOFIC in the third memory M3 (S84). The camera 10 then stores the signal obtained in the initialized state of the lateral overflow integrated capacitor LOFIC in the first memory M1 (S85).

[0118] Afterwards, the shooting device 10 reduces noise while performing global shutter shooting within a high dynamic range based on the reset level signal maintained in the fourth memory M4, the signal obtained when the lateral overflow collection capacitor LOFIC maintained in the first memory M1 is initialized, the signal obtained by global shutter exposure maintained in the second memory M2, and the overflow signal maintained in the third memory M3 (S86).

[0119] According to the imaging device 10 of this embodiment, it is possible to perform global shutter imaging with a wide dynamic range while suppressing noise to a low level.

[0120] [Ninth embodiment]

[0121] FIG14 is a diagram schematically illustrating the circuit configuration of an imaging device 10 according to a ninth embodiment of the present invention. This diagram schematically illustrates the image sensor 100 included in the imaging device 10. The imaging device 10 of this embodiment includes a gain control unit GC. The gain control unit GC changes the gain of the readout circuit when reading signals from the light-receiving portion of the pixel. The gain control unit GC controls the gain of the readout circuit so that the signal is read at at least two gains.

[0122] The imaging device 10 of this embodiment includes a first memory M1 , a second memory M2 , a third memory M3 , and a fourth memory M4 .

[0123] Fig. 15 is a flowchart of the process executed by the imaging device 10 of this embodiment. This figure shows the process performed in one frame. The imaging device 10 first resets the plurality of light receiving units (S90) and performs global shutter exposure (S91).

[0124] The imaging device 10 sets the control unit CTR to the second gain and stores the reset level signal of the control unit CTR in the second memory M2 ( S92 ). Here, the second gain may be a relatively low gain.

[0125] The camera 10 sets the control unit CTR to the first gain and stores the reset level signal of the control unit CTR in the first memory M1 (S93). The first gain may be a relatively high gain. The camera 10 then stores the signal obtained when the signal from the light receiving unit is acquired at the first gain in the third memory M3 (S94). Furthermore, the camera 10 stores the signal obtained when the signal from the light receiving unit is acquired at the second gain in the fourth memory M4 (S95).

[0126] Afterwards, the shooting device 10 performs global shutter shooting (S96) within a high dynamic range while reducing noise based on the reset level signal in the first gain maintained in the first memory M1, the reset level signal in the second gain maintained in the second memory M2, the light receiving signal in the first gain maintained in the third memory M3, and the light receiving signal in the second gain maintained in the fourth memory M4.

[0127] According to the imaging device 10 of this embodiment, by acquiring signals with two kinds of gains, global shutter imaging with a wider dynamic range and less noise can be performed.

[0128] FIG16 is a timing chart showing the switching and execution of multiple functions by the imaging device 10 according to an embodiment of the present invention. The multiple functions are those described in the first through ninth embodiments, and may include, for example, multiple-exposure shooting, noise reduction using correlated multisampling, super-resolution based on estimation of spatial high-frequency components, image plane phase difference autofocus, and dynamic range expansion using lateral overflow capacitors or multiple gains.

[0129] This figure shows the following operation: in successive frames, the function is switched from a first mode (MODE1) in which the first of a plurality of functions is operated to a second mode (MODE2) in which the second of a plurality of functions is operated. Before the start of the first frame (ST1), the imaging device 10 reads the signal of the N-1 pixel (#N-1) and the signal of the N-th pixel (#N), and initializes the photodiode (R1). Here, N is the total number of pixels. In addition, at the end of the first exposure time (EX1), the imaging device 10 turns on the transfer transistor and stores the signal in multiple memories (TR1).

[0130] When the first frame begins (ST1), the camera 10 sequentially reads the first pixel signal (#1), the second pixel signal (#2), the third pixel signal (#3), the N-1 pixel signal (#N-1), and the N-th pixel signal (#N). During this time, the camera 10 initializes the photodiode (R2) and performs global shutter exposure for the second exposure time (EX2). Thereafter, the camera 10 turns on the transfer transistor and stores the signal in multiple memories (TR2). The signals stored in the multiple memories in the second mode (MODE2) may differ from the signals stored in the multiple memories in the first mode (MODE1).

[0131] Similarly, when the second frame begins (ST2), the camera 10 sequentially reads the first pixel signal (#1), the second pixel signal (#2), the third pixel signal (#3), the (N-1)th pixel signal (#N-1), and the Nth pixel signal (#N). During this time, the camera 10 initializes the photodiode (R3), performs global shutter exposure, and turns on the transfer transistor to store the signal in multiple memories.

[0132] Here, the camera 10 can also acquire signals for each frame at a predetermined period, regardless of the type of signals stored in the at least two memories. That is, the frame period can be fixed between the first and second consecutive frames. By utilizing a predetermined frame period regardless of the capture function, even when switching between capture functions used in conjunction with global shutter capture, image loss, delays, and frame rate changes can be avoided.

[0133] Furthermore, the invention is not limited to the aforementioned embodiments and can be implemented in various other forms without departing from the spirit of the invention. Therefore, the aforementioned embodiments are merely illustrative in all respects and are not to be construed as limiting. For example, the aforementioned processing steps can be arbitrarily altered in order or performed in parallel without creating any inconsistency in the processing content.

[0134] Description of Reference Numerals: 10 ... imaging device, 100 ... image sensor, PD1 ... first photodiode, PD1L ... first left photodiode, PD1R ... First right photodiode, PD2…second photodiode, PD2L…second left photodiode, PD2R…second right photodiode, PD3…third photodiode, PD3L…third left photodiode, PD3R…third right photodiode, PD4…fourth photodiode, PD4L…fourth left photodiode, PD4R…fourth right photodiode, TX1…first transfer transistor, TX1L…first left transfer transistor, TX1R…first right transfer transistor, TX2…second transfer transistor, TX2L…second left transfer transistor, TX2R…second right transfer transistor, TX3…third transfer transistor, TX3L…third left transfer transistor, TX3R…third right transfer transistor, TX4…fourth transfer transistor, TX4L…fourth left transfer transistor, TX4R…fourth right transfer transistor, CTR…control unit, M1…first memory, M2…second memory, M3…third memory, M4…fourth memory, LOFIC…lateral overflow collection capacitor, GC…gain control unit.

Claims

1. A photographing device, characterized in that: have: a plurality of light receiving parts that convert light into electrical signals; a plurality of switches, each connected to the plurality of light receiving parts; a control unit connected to the plurality of switches and controlling acquisition of the electrical signal to perform global shutter photography through the plurality of light receiving units; as well as at least two memories connected to the control unit, The control unit switches the type of signals held in the at least two memories, thereby switching a photographing function to be executed together with the global shutter photographing.

2. The photographing device according to claim 1, wherein: The at least two memories include a first memory and a second memory, The first memory stores a signal obtained when the plurality of light receiving units are subjected to global shutter exposure from a first time to a second time. The second memory stores signals obtained when the plurality of light receiving units are subjected to global shutter exposure from the first time to a third time.

3. The photographing device according to claim 1, wherein: The at least two memories include a first memory and a second memory, The control unit acquires signals obtained when the plurality of light receiving units are subjected to global shutter exposure for a predetermined exposure time at a first moment and a second moment, respectively. The first memory stores the signal acquired at the first moment, The second memory stores the signal acquired at the second moment.

4. The photographing device according to claim 1, wherein: The at least two memories include a first memory and a second memory, The first memory stores a signal obtained by adding signals respectively obtained from the plurality of light receiving parts. The second memory holds signals obtained from some of the plurality of light receiving units.

5. The photographing device according to claim 1, wherein: The at least two memories include a first memory and a second memory, The plurality of light receiving units include a first light receiving unit and a second light receiving unit for each pixel. The first memory stores the signal obtained from the first light receiving unit. The second memory holds a signal obtained by adding a signal obtained from the first light receiving unit and a signal obtained from the second light receiving unit.

6. The photographing device according to claim 1, wherein: The at least two memories include a first memory, a second memory, and a third memory, The first memory stores a signal obtained when the plurality of light receiving units are subjected to global shutter exposure from a first time to a second time. The second memory stores signals obtained when the plurality of light receiving units are subjected to global shutter exposure from the first time to the third time. The third memory stores signals obtained when the plurality of light receiving units are subjected to global shutter exposure from a first time point to a fourth time point.

7. The photographing device according to claim 1, wherein: The at least two memories include a first memory, a second memory, and a third memory, The plurality of light receiving units include a first light receiving unit and a second light receiving unit for each pixel. The first memory stores a signal obtained from the first light receiving portion included in a part of the pixels. The second memory holds a signal obtained by adding a signal obtained from the first light receiving portion included in the part of pixels and a signal obtained from the second light receiving portion included in the part of pixels. The third memory holds a signal obtained by adding a signal obtained from the first light receiving unit and a signal obtained from the second light receiving unit.

8. The photographing device according to claim 1, wherein: The camera device also has a lateral overflow collection capacitor, The at least two memories include a first memory, a second memory, and a third memory, The first memory holds a signal obtained in a state where the lateral overflow collective capacitor is initialized. The second memory stores the signals obtained from the plurality of light receiving units. The third memory holds the overflow signal accumulated in the lateral overflow collection capacitor.

9. The photographing device according to claim 1, wherein: The at least two memories include a first memory, a second memory, a third memory, and a fourth memory, The control unit obtains signals obtained when the multiple light receiving units are initialized at a first moment and a second moment, and obtains signals obtained when the multiple light receiving units are exposed to global shutter for a predetermined exposure time at a third moment and a fourth moment, respectively. The first memory stores the signal acquired at the first moment, The second memory stores the signal acquired at the second moment, The third memory stores the signal acquired at the third moment, The fourth memory stores the signal acquired at the fourth moment.

10. The photographing device according to claim 1, wherein: The at least two memories include a first memory, a second memory, a third memory, and a fourth memory, The first memory holds a signal obtained with a first gain in a state where the plurality of light receiving units are initialized. The second memory holds a signal obtained with a second gain in a state where the plurality of light receiving units are initialized. The third memory stores a signal obtained when signals obtained from the plurality of light receiving units are obtained at the first gain. The fourth memory holds a signal obtained when signals obtained from the plurality of light receiving units are acquired at the second gain.

11. The photographing device according to any one of claims 1 to 10, characterized in that: The control unit acquires the signal of each frame at a predetermined cycle regardless of the type of the signals held in the at least two memories.

12. A shooting method, characterized in that: The shooting method is a control method executed by a shooting device, The imaging device includes: a plurality of light receiving units that convert light into electrical signals; a plurality of switches that are respectively connected to the plurality of light receiving units; a control unit connected to the plurality of switches; and at least two memories connected to the control unit, The shooting method includes: controlling acquisition of the electrical signal to perform global shutter photography through the plurality of light receiving parts; as well as By switching the types of signals held in the at least two memories, a photographing function performed together with the global shutter photographing is switched.

Images