Information processing device, information processing method, and program

Abstract

We provide technology that can suppress motion blur. [Solution] The information processing device is an information processing device that controls an imaging device having a photoelectric conversion element that counts pulse signals corresponding to the incidence of light and generates an image signal, and comprises: exposure period control means for controlling the exposure period of the main frame based on a main period which is the exposure period of the main frame; blur detection means for reading out the image signal of the exposed subframe for each unit period shorter than the main period, from the start of exposure to the end of the unit period, and detecting blur of the subject; and output means for outputting an output signal based on an image signal selected from either the main frame or the subframe based on the result of blur detection.

Description

Technical Field

[0001] The present invention relates to the control of an imaging device.

Background Art

[0002] In photography using an imaging device, it is necessary to appropriately set the exposure according to the subject and the shooting scene. Exposure is mainly set by at least one of the shutter speed, aperture, and sensitivity. When achieving proper exposure for the target subject, it is necessary to combine these elements well. Generally, when the shutter speed is slow, camera shake and subject blur are likely to occur. When the shutter speed is increased to prevent camera shake and subject blur, proper exposure can be maintained by increasing the sensitivity accordingly, but increasing the sensitivity will result in more noise. Since the shutter speed and sensitivity are in an inverse relationship between blur and noise, skilled techniques are required to appropriately set the exposure according to the subject and the shooting scene.

[0003] In recent imaging devices, a mode that automatically performs exposure setting according to the photometry result is installed, and appropriate exposure setting is possible for each shooting scene. However, since these controls perform exposure setting based on the photometry result, even if the subject suddenly starts moving, the photometry result does not change, and subject blur occurs without changing the exposure setting (especially the shutter speed).

[0004] Patent Document 1 describes a technique for calculating the shutter speed adjustment range according to the amount of subject blur from a preparatory shooting image and a main shooting image, and adjusting the shutter speed based on the calculation result to set the exposure so that subject blur does not occur.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] However, adjusting the shutter speed in Patent Document 1 requires both a preparatory image and the actual image, meaning that the shutter speed cannot be adjusted until motion blur occurs. In other words, in shooting scenes where the subject suddenly starts moving, motion blur cannot be suppressed.

[0007] Therefore, the present invention provides a technology that can suppress subject blur in the control of an imaging device. [Means for solving the problem]

[0008] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, An information processing device for controlling an imaging device having a photoelectric conversion element that counts pulse signals corresponding to the incidence of light and generates an image signal, Exposure period control means for controlling the exposure period of the main frame based on the main period, which is the exposure period of the main frame, A motion blur detection means reads out the image signal of the exposed subframe for each unit period shorter than the main period, from the start of exposure to the end of the unit period, and detects motion blur of the subject. An output means that outputs an output signal based on an image signal selected from either the main frame or the subframe, based on the results of blur detection, It is equipped with. [Effects of the Invention]

[0009] According to the present invention, subject blur can be suppressed in the control of the imaging device. [Brief explanation of the drawing]

[0010] [Figure 1] An exploded perspective view showing an example configuration of a photoelectric conversion element according to the first embodiment. [Figure 2]A diagram showing an example configuration of the sensor chip of the first embodiment. [Figure 3] A diagram showing an example of the configuration of a circuit chip according to the first embodiment. [Figure 4] An example of a block diagram of the equivalent pixel circuit and signal processing unit of the first embodiment. [Figure 5] A timing diagram illustrating the operation of the APD and waveform shaping unit in the first embodiment. [Figure 6] Functional block diagram of the imaging device according to the first embodiment. [Figure 7] Timing chart of the photoelectric conversion element of the first embodiment. [Figure 8] A flowchart of the imaging process of the imaging device according to the first embodiment. [Figure 9] A flowchart of the imaging process of the imaging device according to the second embodiment. [Figure 10] Functional block diagram of the imaging device according to the third embodiment. [Figure 11] A flowchart of the imaging process of the imaging device according to the third embodiment. [Figure 12] A block diagram showing the hardware configuration of the information processing device of the imaging device according to the embodiment. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] <First Embodiment> The first embodiment will be described below with reference to the drawings. The embodiment relates to technologies such as the control of an imaging device capable of suppressing subject blur. Figure 1 is an exploded perspective view showing an example of the configuration of a photoelectric conversion element of the first embodiment.

[0013] The photoelectric conversion element 100 has two chips, namely, a sensor chip 11 and a circuit chip 21. The sensor chip 11 and the circuit chip 21 are stacked and electrically connected to each other. The sensor chip 11 includes a pixel region 12. The circuit chip 21 includes a pixel circuit region 22 that processes the signals detected in the pixel region 12 and a readout circuit region 23 that reads out signals from the pixel circuit region 22.

[0014] FIG. 2 is a diagram showing a configuration example of the sensor chip 11 of the first embodiment.

[0015] The pixel region 12 of the sensor chip 11 includes a plurality of pixels 101 arranged two-dimensionally over a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 that includes an avalanche photodiode (hereinafter also referred to as APD). The photoelectric conversion unit 102 converts the incident light into an electrical signal and outputs it. In FIG. 2, symbols indicating the row number and column number together with "P" are assigned to 36 pixels 101 arranged in 6 rows from row 0 to row 5 and 6 columns from column 0 to column 5. For example, the pixel 101 arranged in row 1 and column 4 is assigned "P14". Note that the number of rows and columns of the pixel array forming the pixel region 12 is not particularly limited.

[0016] FIG. 3 is a diagram showing a configuration example of the circuit chip 21 of the first embodiment.

[0017] The circuit chip 21 includes a pixel circuit region 22 and a readout circuit region 23.

[0018] The pixel circuit region 22 includes multiple signal processing units 103 arranged in a two-dimensional manner across multiple rows and columns. Each signal processing unit 103 is associated with a pixel. In Figure 3, 36 signal processing units 103 arranged in six rows (rows 0 to 5) and six columns (columns 0 to 5) are assigned codes S00 to S55, which indicate the row and column numbers along with "S". For example, the signal processing unit 103 located in the first row and fourth column is assigned the code "S14". The number of rows and columns in the array of signal processing units 103 in the pixel circuit region 22 is not particularly limited.

[0019] The readout circuit area 23 includes a vertical scanning circuit 110, a column circuit 112, a horizontal scanning circuit 111, a control pulse generation unit 115, and an output circuit 114.

[0020] The control lines 116 extend in a first direction (horizontal direction in Figure 3). This first direction is sometimes referred to as the row direction or horizontal direction. The control lines 116 are located in each row of the signal processing array in the pixel circuit area 22. The control lines 116 are connected to each of the signal processing units 103 arranged in the first direction to supply a common control signal. In Figure 3, the control lines 116 are assigned the code PVSEL[0] to PVSEL[5], which indicates the row number along with "PVSEL". For example, the control line 116 of the first row is assigned the code "PVSEL[1]". The control lines 116 of each row are connected to the vertical scanning circuit 110.

[0021] The vertical scanning circuit 110 supplies control signals to the signal processing unit 103 via the control line 116 to drive the signal processing unit 103.

[0022] The signal line 113 extends in a second direction (vertical direction in Figure 3) that intersects the first direction. The second direction in which the signal line 113 extends may be referred to as the column direction or vertical direction. The signal line 113 is located in each column of the array of signal processing units 103 in the pixel circuit area 22. The signal line 113 is connected to each of the signal processing units 103 that are arranged in the second direction. In Figure 3, the signal line 113 is assigned the code POUT[0] to POUT[5] indicating the column number along with "POUT". For example, the signal line 113 in the fourth column is assigned the code "POUT4". Each of the signal lines 113 has n signal lines for outputting an n-bit digital signal. The signal line 113 is connected to the column circuit 112.

[0023] Each column circuit 112 is provided corresponding to each column of the signal processing array in the pixel circuit area 22 and is connected to the signal line 113 of the corresponding column. Each column circuit 112 has the function of holding the signal read from the signal processing unit 103 via the signal line 113 of the corresponding column.

[0024] The horizontal scanning circuit 111 supplies control signals to the column circuits 112 for reading signals from them. The horizontal scanning circuit 111 supplies control signals to the column circuits 112 of each column via control lines 117.

[0025] When the column circuit 112 receives a control signal from the horizontal scanning circuit 111, it outputs the held signal to the output circuit 114 via the horizontal output line 118.

[0026] The control line 117 connects the horizontal scanning circuit 111 and the column circuit 112. In Figure 3, the control line 117 is assigned the code PHSEL[0] to PHSEL[5], which indicates the column number along with "PHSEL". For example, the control line for the fourth column is assigned the code "PHSEL[4]".

[0027] The horizontal output line 118, indicated as “HSIG”, has n signal lines connected to the column circuit 112 to output an n-bit digital signal. The horizontal output line 118 is connected to the output circuit 114. The horizontal output line 118 outputs the digital signal output by the column circuit 112 to the output circuit 114.

[0028] The output circuit 114 outputs a signal corresponding to the pixel signal output by the column circuit 112 as the image signal SOUT of the photoelectric conversion element 100.

[0029] The control pulse generation unit 115 supplies control signals to control the operation and timing of the vertical scanning circuit 110, the horizontal scanning circuit 111, and the column circuit 112. At least a portion of the control signals that control the operation and timing of the vertical scanning circuit 110, the horizontal scanning circuit 111, and the column circuit 112 may be supplied from outside the photoelectric conversion element 100.

[0030] Figure 4 shows an example of an equivalent circuit for pixel 101 and a block diagram of the signal processing unit 103.

[0031] The pixel 101 of the sensor chip 11 includes an APD201 that functions as a photoelectric conversion unit 102. APD201 is an abbreviation for Avalanche Photo Diode. When light is incident on the APD201, the APD201 generates charge pairs corresponding to the incident light as electrical signals through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD201. In addition, a voltage VH (second voltage), which is higher than the voltage VL supplied to the anode, is supplied to the cathode of the APD201. A reverse bias voltage is supplied to the anode and cathode such that the APD201 performs avalanche multiplication. With this reverse bias voltage supplied, the APD201 undergoes avalanche multiplication using the charge generated by the incident light, and generates an avalanche current.

[0032] When a reverse bias voltage is supplied, there are two modes of operation: Geiger mode, where the potential difference between the anode and cathode is greater than the breakdown voltage, and linear mode, where the potential difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD (Single Photon Avalanche Diode). For example, the voltage VL is -30V and the voltage VH is 1V.

[0033] The signal processing unit 103 of the sensor chip 11 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212.

[0034] The quench element 202 is connected to the APD201 and a power supply that provides voltage VH. The quench element 202 has the function of converting the change in avalanche current generated in the APD201 into a voltage signal. When the signal is multiplied by avalanche multiplication, the quench element 202 functions as a load circuit (also called a quench circuit) and suppresses the voltage supplied to the APD201. In this way, the quench element 202 operates to suppress avalanche multiplication, also known as quench operation.

[0035] The waveform shaping unit 210 shapes the cathode potential change of the APD201 obtained during photon detection and outputs a pulse signal. The waveform shaping unit 210 may be, for example, at least one of an inverter circuit and a buffer circuit. Node A in Figure 4 indicates the input side of the waveform shaping unit 210. Node B indicates the output side of the waveform shaping unit 210.

[0036] The counter circuit 211 counts pulse signals corresponding to the incident light output from the waveform shaping unit 210. Furthermore, when the control signal PRES is supplied via the control line 213, the counter circuit 211 resets the count value.

[0037] The memory circuit 212 switches the electrical connection and disconnection state between the counter circuit 211 and the signal line 113 based on the control signal VSEL supplied from the vertical scanning circuit 110 in Figure 3 via the control line 214. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter circuit 211 and outputs the pixel count value from the counter circuit 211 as a signal to the signal line 113.

[0038] Figure 5 is a timing diagram illustrating the operation of the APD and waveform shaping unit in the first embodiment. Figure 5(a) shows the voltage change at node A in Figure 4. Figure 5(b) shows the voltage change at node B in Figure 4.

[0039] Between time t0 and time t1, a voltage of VH-VL is applied to the APD201 shown in Figure 4. At this time, the voltage at nodeB is at a low level. At time t1, when a photon is incident on the APD201 of the photoelectric conversion unit 102, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0040] At time t2, if the voltage of nodeA falls below a predetermined threshold, the waveform shaping unit 210 causes the voltage of nodeB to change from a low level to a high level.

[0041] At time t3, as the voltage drop increases further and the voltage applied to APD201 decreases, the avalanche multiplication of APD201 stops. As a result, the voltage level of nodeA no longer drops below a certain value. Subsequently, a current flows from voltage VL to nodeA to compensate for the voltage drop, and the voltage of nodeA rises.

[0042] At time t4, if the voltage of nodeA exceeds a predetermined threshold, the waveform shaping unit 210 causes the voltage of nodeB to change from a high level to a low level.

[0043] Subsequently, as shown at time t5, the voltage at nodeA gradually rises to a constant voltage.

[0044] Figure 6 is a functional block diagram of the imaging device according to the first embodiment.

[0045] The imaging device 600 includes a photoelectric conversion element 100, a lens 601, a photometer 602, a blur detection unit 603, a signal memory 604, an exposure period control unit 606, an exposure period memory 607, an output signal selection unit 608, and a signal processing unit 609. The output signal selection unit 608 and the signal processing unit 609 are examples of output means.

[0046] The photoelectric conversion element 100 includes the APD201 described in Figures 1 to 5. The photoelectric conversion element 100 receives light focused by the lens 601, counts pulse signals corresponding to the incident light, and generates an image signal. The photoelectric conversion element 100 is connected to the photometering unit 602, the blur detection unit 603, the signal memory 604, the exposure period control unit 606, and the output signal selection unit 608.

[0047] The photometering unit 602 calculates the exposure amount from the image signal obtained from the photoelectric conversion element 100, calculates the exposure period that results in proper exposure, and outputs it to the exposure period control unit 606. In the first embodiment, the exposure period calculated by the photometering unit 602 is also called the main period.

[0048] The blur detection unit 603 detects subject blur (hereinafter also referred to as blur) from the image signal obtained from the photoelectric conversion element 100. Specifically, the blur detection unit 603 reads and acquires the image signal at unit periods shorter than the main period. In the following explanation, the period from the start of exposure to the end of each unit period is also called a sub-period. The frames for the period from the start of exposure to the end of each unit period are also called subframes. The blur detection unit 603 calculates the difference in image signals between temporally adjacent subframes and may determine that subject blur exists if the subject has moved more than a predetermined threshold of pixels. The blur detection unit 603 may also perform motion vector calculation and determine that subject blur exists if the amount of motion vector is greater than or equal to a threshold. The blur detection unit 603 outputs blur information indicating the presence or absence of blur to the exposure period control unit 606 and the output signal selection unit 608. Furthermore, if the blur detection unit 603 does not detect blur in the read subframe, it saves the image signal of that subframe to the signal memory 604.

[0049] The exposure period memory 607 stores information related to the exposure period. For example, the exposure period memory 607 stores the main period calculated by the photometering unit 602, a sub-period for determining the presence or absence of blur by the blur detection unit 603 from the start of exposure, and a unit period for setting the sub-period. The exposure period memory 607 may store an exposure period set by the user as the main period, either in place of the main period calculated by the photometering unit 602 or in addition to the main period.

[0050] The exposure period control unit 606 controls the exposure period of the photoelectric conversion element 100. Specifically, the exposure period control unit 606 controls the exposure of the photoelectric conversion element 100 based on the main period calculated by the photometric unit 602, for example, to control the exposure period of the main frame. The main period may be the exposure period received from the user. Here, if the blur detection unit 603 detects blur, the exposure period control unit 606 stops exposure. On the other hand, if the blur detection unit 603 does not detect blur, the exposure period control unit 606 continues exposure until the next sub-period ends. Alternatively, the exposure period control unit 606 may continue exposure of the photoelectric conversion element 100 until the main period calculated by the photometric unit 602 ends. If the blur detection unit 603 detects blur, the image acquired in the sub-period before the blur was detected may be output. The exposure period control unit 606 may determine the exposure period of the next main frame based on the exposure period of the sub-period before the blur was detected. As a result, the exposure period control unit 606 can achieve exposure that suppresses blur while obtaining an image signal with proper exposure.

[0051] The signal memory 604 stores the image signal output by the photoelectric conversion element 100 after exposure during a sub-period.

[0052] The output signal selection unit 608 selects either the main frame image signal output from the photoelectric conversion element 100 or the subframe image signal stored in the signal memory 604, based on the blur information indicating the presence or absence of blur, which is the result of blur detection from the blur detection unit 603, and outputs it to the signal processing unit 609. For example, if blur is detected, the output signal selection unit 608 selects the subframe image signal. On the other hand, if blur is not detected, the output signal selection unit 608 selects the main frame image signal. That is, if no blur is detected in the subframe acquired during the exposure period until the main period is reached, the output signal selection unit 608 outputs the main frame image signal.

[0053] The signal processing unit 609 performs various image processing, such as digital gain processing, gamma processing, and white balance processing, on the image signal output by the output signal selection unit 608 to generate and output an output signal. The signal processing unit 609 may perform some of the above processing or other processing. Furthermore, a downstream processing unit (not shown) may perform recording to memory, and recording and displaying image and video signals using an external output device.

[0054] Figure 12 is a block diagram showing the hardware configuration of the information processing device 1200 of the imaging device 600. The information processing device 1200 is an example of a computer that controls the imaging device 600. The information processing device 1200 of the imaging device 600 includes a processor 1201, a memory 1202, a storage 1203, a communication IF 1204, an input IF 1205, an output IF 1206, and a bus 1207. The processor 1201, memory 1202, storage 1203, communication IF 1204, input IF 1205, and output IF 1206 are connected to each other via the bus 1207 so that they can send and receive information.

[0055] The processor 1201 is an arithmetic processing unit, for example, a CPU (Central Processing Unit). The information processing unit 1200 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of or in addition to the CPU. The processor 1201 implements various functions by reading programs stored in the storage 1203 and loading them into the memory 1202. For example, by reading a program, the processor 1201 implements some or all of the functions of the photometric unit 602, blur detection unit 603, exposure period control unit 606, output signal selection unit 608, and signal processing unit 609. Furthermore, some or all of the functions of the photometric unit 602, blur detection unit 603, exposure period control unit 606, output signal selection unit 608, and signal processing unit 609 may be implemented by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0056] Memory 1202 is a high-speed read / write storage device, such as RAM (Random Access Memory). Memory 1202 functions as a work area when the processor 1201 executes a program. Memory 1202 temporarily stores the program and parameters necessary for program execution.

[0057] Storage 1203 is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Storage 1203 retains programs, parameters necessary for program execution, and the results of program execution even when power is not supplied.

[0058] The communication interface 1204 is an interface for enabling communication with external devices via a wired or wireless network.

[0059] Input IF1205 is an interface for receiving information from input devices. Examples of input devices include shutter buttons, touch panels, mice, and keyboards.

[0060] Output IF1206 is an interface for outputting information to an external device. This external device could be, for example, a display device such as a monitor.

[0061] Figure 7 is a timing chart illustrating the photoelectric conversion method of the photoelectric conversion element 100 according to this embodiment. The blur detection unit 603 of this embodiment generates subframes 1_1, 1_2, 1_3, and 1_4 from the main frame MF1 based on a unit period shorter than the main period of the main frame. The unit period may be a period obtained by equally dividing (in this case, dividing into 4) the 33.3 ms of the main period, which is the exposure period of one main frame MF1 or MF2. The time from time Tn to time Tn+1, which is the unit period, may be, for example, 8.33 ms (≒ 33.3 / 4). As shown in Figure 7, the main period of the main frame MF1 and the sub-periods, which are the exposure periods of subframes 1_1, 1_2, 1_3, and 1_4, are integer multiples of the divided unit period. The blur detection unit 603 of this embodiment reads out the image signals of the subframes exposed during the sub-period from the start of exposure to the end of the unit period for each unit period, and detects blur of the subject for each subframe.

[0062] Subframe 1_1 has an exposure period equal to the unit period from the exposure start time T0 to time T1 of mainframe MF1. Subframe 1_2 has an exposure period twice the unit period from time T0 to time T2. Subframe 1_3 has an exposure period three times the unit period from time T0 to time T3. Subframe 1_4 has the same exposure period as the mainframe from time T0 to time T4, but with an exposure period four times the unit period. Therefore, subframe 1_4 is also the mainframe.

[0063] At time T0, the counter circuit 211 resets the count value and restarts counting the pulse signal. At times T1 to T4, the memory circuit 212 acquires the count values ​​C1_1, C1_2, C1_3, and C1_4 from the counter circuit 211. The memory circuit 212 also temporarily stores the count values ​​C1_1, C1_2, C1_3, and C1_4. The memory circuit 212 then sequentially outputs the temporarily stored image signal for one row from the photoelectric conversion element 100 via the buffer of the column circuit 112. Thus, according to this embodiment, the memory circuit 212 reads out and outputs the image signal accumulated during the subframe 1_1 period from time T1 to T2, and the blur detection unit 603 processes the image signal. Similarly, the memory circuit 212 sequentially reads and outputs the image signals accumulated during the periods of subframe 1_2, subframe 1_3, and subframe 1_4, respectively, over the time intervals T2-T3, T3-T4, and T4-T1, and the blur detection unit 603 processes these image signals.

[0064] As described above, the exposure period in this embodiment includes multiple sub-periods and a main period within a single main frame. Each sub-period is shorter than the main period. The photoelectric conversion element 100 outputs the image signal generated in each sub-period between the end of the sub-period and the end of the main period. In this embodiment, each sub-period overlaps with at least a portion of the main period. Multiple sub-periods and the main period begin exposure simultaneously. Furthermore, the end of the main period marks the end of the main frame. The sub-periods and main periods are integer multiples of a unit period.

[0065] Since this embodiment uses the APD201, unlike CMOS (Complementary Metal Oxide Semiconductor) sensors, there is no readout noise, allowing for overlapping exposure periods. In other words, the original signal does not degrade no matter how many times it is read out within a single main frame. This embodiment shows an example where the signal is read out four times within a single main frame, but the number of subframe readouts may be increased or decreased considering the processing time for blur detection, etc.

[0066] Figure 8 is a flowchart of the imaging process of the imaging device in the first embodiment. The detailed operation of this embodiment will be described in conjunction with the timing chart in Figure 7.

[0067] Before the start of the imaging process flowchart, the exposure period control unit 606 sets the main period for shooting as the exposure period. For example, the exposure period control unit 606 may set the main period by receiving input from the user via the input IF 1205 while the imaging device 600 is in shooting mode. The exposure period control unit 606 may also adopt the main period calculated by the photometering unit 602 to achieve proper exposure based on the photometering results. The exposure period control unit 606 may also accept changes to the composition by the user and may perform preparatory shooting operations to check the exposure before shooting.

[0068] In S801, the exposure period control unit 606 starts the shooting process. The exposure period control unit 606 may start the shooting process when the user presses the shutter button.

[0069] In S802, the exposure period control unit 606 starts exposure when it receives a press of the shutter button by the user. The start timing may be time T0 in Figure 7. A typical imaging device 600 controls the photoelectric conversion element 100 to output an image signal so that the exposure period is predetermined. However, the exposure period control unit 606 in this embodiment controls the exposure based on a sub-period and a main period, causing the photoelectric conversion element 100 to output an image signal. The main period here may be the exposure period calculated by the photometric unit 602 or the exposure period received from the user.

[0070] In S803, the exposure period control unit 606 determines whether a main frame has finished. Here, the exposure period control unit 606 determines the end of a main frame for each unit period, that is, for each subframe. In other words, the exposure period control unit 606 determines the end of a main frame for each timing T1, T2, T3, and T4 in Figure 7. If the exposure period control unit 606 determines that the main frame has finished at time T4, it proceeds to S810. On the other hand, if the exposure period control unit 606 determines that the main frame has not finished at the other timings T1, T2, and T3, it proceeds to S804.

[0071] In S804, the motion detection unit 603 reads out the subframe image signal from the photoelectric conversion element 100.

[0072] In S805, the blur detection unit 603 detects blur of the subject in a subframe based on the read image signal. The blur detection unit 603 may detect blur by calculating the difference between preceding and succeeding subframes. For example, the blur detection unit 603 may calculate the difference between subframe 1_1 and subframe 1_2 and detect blur based on whether the subject has moved by more than a threshold based on the difference.

[0073] In S806, the blur detection unit 603 determines whether or not blur has occurred in the subframe image signal. If the blur detection unit 603 determines that no blur has occurred, the process proceeds to S807. On the other hand, if the blur detection unit 603 determines that blur has occurred, it outputs blur information indicating that blur has been detected to the exposure period control unit 606 and the output signal selection unit 608, and proceeds to S808.

[0074] In S807, the blur detection unit 603 updates the image signal stored in the signal memory 604 based on the image signal acquired from the photoelectric conversion element 100. As a result, the image signal stored in the signal memory 604 is updated with the image signal of a newly acquired subframe that does not exhibit blur. The blur detection unit 603 may also store multiple subframe image signals in the signal memory 604. That is, the blur detection unit 603 may accumulate newly acquired image signals that do not exhibit blur in the signal memory 604. This makes it possible for the blur detection unit 603 to store image signals of subframes that do not exhibit blur and have different exposure periods in the signal memory 604. In this case, each time the exposure of the main period is completed, the blur detection unit 603 may delete the subframe image signals accumulated in the signal memory 604. Alternatively, the blur detection unit 603 may reset the signal memory 604 when the number of accumulated subframes reaches a predetermined number.

[0075] After this, the processes from S803 to S807 are executed until the exposure period of the main frame is reached, or until the blur detection unit 603 detects blur.

[0076] In S808, when the exposure period control unit 606 receives information from the blur detection unit 603 that blur has been detected, it stops the exposure of the photoelectric conversion element 100.

[0077] In S809, the output signal selection unit 608 selects an image signal based on the blur information from the blur detection unit 603 and outputs an output signal. Here, since the blur information indicates that blur has been detected, the output signal selection unit 608 selects the image signal of a subframe stored in the signal memory 604. The subframe stored in the signal memory 604 is the subframe immediately preceding the subframe in which blur was detected. Therefore, the output signal selection unit 608 can output an output signal of the image of a subframe without blur. The signal processing unit 609 may perform image processing on the image signal before outputting it as an output signal. Here, if multiple subframes are stored in the signal memory 604 as described above, the output signal selection unit 608 does not have to select the subframe immediately preceding the subframe in which blur was detected. That is, the output signal selection unit 608 may select an image of a subframe two or more frames prior. In this case, the output signal selection unit 608 may select which of the multiple stored subframes to select by determining whether the subframe satisfies the desired conditions. Alternatively, multiple subframes may be displayed side-by-side on an unillustrated display unit for the user to select from. The "desired conditions" described above may include conditions for at least one of the brightness of the image signal and the recognition rate of the subject. For example, the output signal selection unit 608 may, based on the desired conditions, select at least one subframe from among the multiple subframes in which the brightness of the image signal falls within a predetermined range. The predetermined range may be the range for proper exposure, or it may be a range specified by the user. Furthermore, the output signal selection unit 608 may select two or more subframes stored in the signal memory 604, perform image processing, and display them side-by-side on an unillustrated display unit. More preferably, the output signal selection unit 608 may also display the exposure period corresponding to each subframe on the display unit to provide the user with an indicator for selecting a subframe. This allows the imaging device 600 to not only produce blur-free images but also to perform appropriate exposure control.

[0078] In S810, the exposure period control unit 606, having determined in S803 that the main frame period has ended, stops the exposure and outputs a signal to the output signal selection unit 608 indicating that the main frame period has ended.

[0079] In S811, the output signal selection unit 608 acquires the image signal output by the photoelectric conversion element 100 because the mainframe period has ended, and the signal processing unit 609 processes the image signal before outputting it as an output signal.

[0080] As described above, the first embodiment outputs an output signal by selecting either the main frame or a subframe with a shorter exposure period than the main frame based on the results of blur detection, thereby enabling shooting with reduced subject blur.

[0081] In the first embodiment, when blur is detected, the image signal of the subframe before the blur was detected is selected, so an output signal of a blur-free image can be output.

[0082] In the first embodiment, if blur is not detected, the subframe image signal is saved, so if blur is detected, an output signal can be easily generated and output based on the image signal of the previous subframe.

[0083] In the first embodiment, the main period and sub-periods are controlled by integer multiples of the unit period, thereby reducing the processing burden.

[0084] <Second Embodiment> In the first embodiment, an example was described of obtaining an image signal without subject blur in the output of an image signal of a single frame. In the second embodiment, an example was described of obtaining an image without subject blur in the output of a continuous frame image signal, such as a burst of motion video and still images, and the adjustment of subframe images was described. The second embodiment has the same configuration as the first embodiment. Therefore, in the second embodiment, the description of the same configuration as the first embodiment will be omitted or simplified.

[0085] In the second embodiment, the exposure period control unit 606 compares the exposure period (hereinafter also referred to as the storage period) based on blur detection stored in the exposure period memory 607 with the determination exposure period. If it determines that the determination exposure period is longer, it sets the storage period as the main period for the next shot. The storage period may be the longest sub-period among the sub-periods in which no blur was detected in the previous frame. The previous frame is the main frame or sub-frame output in the previous main period in a series of frames in either continuous shooting of video or still images. Therefore, the previous frame is not the previous sub-frame in a series of sub-frames included in the current main frame. The determination exposure period may be the exposure period calculated by the photometering unit 602, or any exposure period set by the user in the shooting mode of the imaging device described in the first embodiment. In other words, the determination exposure period is not the storage period. This enhances the effectiveness of preventing subject blur in the next shot in this embodiment. Furthermore, if blur is not detected in the previous frame, or if blur is detected and the detection exposure period is not longer than the storage period, the exposure period for detection is set as the main period and the exposure for the next shot is controlled by the exposure period for detection. The exposure period control unit 606 in this embodiment controls the photoelectric conversion element 100 based on two exposure periods: a sub-period and a main period.

[0086] Figure 9 shows a flowchart of the imaging process of the imaging device in the second embodiment. Note that explanations of steps that overlap with the flowchart in Figure 8 are simplified or omitted.

[0087] In S901, the exposure period control unit 606 refers to the blur information of the blur detection unit 603 to determine whether or not blur was detected in the previous frame. If the exposure period control unit 606 determines that no blur was detected in the previous frame, or if it determines that there is no previous frame in the first frame in which the shutter button was pressed, the process proceeds to S902.

[0088] In S902, the exposure period control unit 606 starts exposure using the determination exposure period as the main period. For example, the exposure period control unit 606 starts exposure with the exposure period calculated by the photometric unit 602.

[0089] On the other hand, in S901, if the exposure period control unit 606 determines that blur was detected in the main frame of the previous main period, there is a high probability that blur will occur in the current main frame as well, so it proceeds to S903 and refers to the storage period stored in the exposure period memory 607.

[0090] In S903, the exposure period control unit 606 determines whether the exposure period for determination is longer than the storage period stored in the exposure period memory 607. If the exposure period control unit 606 determines that the exposure period for determination is not longer than the storage period, the process proceeds to S902. On the other hand, if the exposure period control unit 606 determines that the exposure period for determination is longer than the storage period, the process proceeds to S904.

[0091] In S904, the exposure period control unit 606 starts the exposure of the current main frame, using the storage period stored in the exposure period memory 607 as the main period.

[0092] Thereafter, the blur detection unit 603 and the exposure period control unit 606 read out subframes in the same manner as in S803 to S807 of the first embodiment, perform blur detection for each subframe, and update the image signal in the signal memory 604 if no blur is detected.

[0093] Meanwhile, the blur detection unit 603, the exposure period control unit 606, and the output signal selection unit 608 read out subframes in the same manner as in S803 to S809 of the first embodiment. If blur is detected in a subframe, the exposure period control unit 606 stops exposure, and the output signal selection unit 608 acquires and outputs an image signal from the signal memory 604. Subframes are exposed for a shorter sub-period than the main period. As a result, if the exposure period (i.e., the main period) is set so that the image signal of the main frame output during the main period is properly exposed, the image signal of the subframe output during the sub-period will be darker than the image of the main frame taken during the main period.

[0094] Therefore, in S905, the signal processing unit 609 performs sensitivity enhancement on the subframe image signal to generate and output an output signal. Specifically, the signal processing unit 609 performs sensitivity enhancement by multiplying the subframe image signal by a ratio (= main period / the longest subframe period among the subframe periods in which no blur was detected). The subframe period in the denominator is also the exposure period of the output subframe. In other words, the subframe period here is the subframe period of the subframe preceding the subframe in which blur was detected. As a result, the signal processing unit 609 compensates for the underexposure caused by the shortened exposure period and generates an output signal with proper exposure and no subject blur.

[0095] In S906, the exposure period control unit 606 saves the longest sub-period in which no blur occurred as a saved period in the exposure period memory 607, updating the stored saved period. In other words, the exposure period control unit 606 saves the sub-period prior to the sub-period in which blur occurred as a saved period in the exposure period memory 607. The exposure period control unit 606 uses the newly saved saved period in the exposure period memory 607 as one of the options for the exposure period of the next main frame. If there is movement in the subject and blur is detected, setting the same exposure period as the current exposure period as the main period would suggest that there will be similar movement in the next main frame. In such cases, the exposure period control unit 606 can suppress subject blur in the next main frame by using the saved period held in the exposure period memory 607 as the exposure period (i.e., main period) of the next main frame.

[0096] If the blur detection unit 603 does not detect subject blur before the end of the main frame, the exposure period control unit 606 stops exposure, similar to steps S803 to S811 in the first embodiment. Then, the output signal selection unit 608 outputs an output signal based on the image signal from the photoelectric conversion element 100, and proceeds to S907.

[0097] In S907, the photometering unit 602 measures light from the image signal for the next main frame, calculates the exposure period that will result in proper exposure, and outputs it to the exposure period control unit 606 as the determination exposure period. The exposure period control unit 606 may adjust the exposure period for the next main frame based on the determination exposure period.

[0098] In the second embodiment, if blur is detected in the previous frame and the exposure period for determination is longer than the storage period, exposure is performed based on the storage period, which is an exposure period in which no blur was detected. As a result, in the second embodiment, even if blur is detected in the previous frame, blur can be suppressed more reliably in the next frame.

[0099] In the second embodiment, if blur is detected in the previous frame and the detection exposure period is shorter than the storage period, the detection exposure period is used as the main exposure period, so that blur can be suppressed while exposure is performed for an appropriate exposure period or an exposure period desired by the user.

[0100] In the second embodiment, since the subframe image signal is sensitized, a high-quality output signal can be produced even with a shorter exposure period.

[0101] <Third Embodiment> Figure 10 is a functional block diagram of the imaging device of the third embodiment. As shown in Figure 10, the imaging device 600 of the third embodiment further includes a subject blur amount calculation unit 1001 compared to the block diagram in Figure 6.

[0102] The subject blur amount calculation unit 1001 calculates the subject blur amount based on the image signal output by the photoelectric conversion element 100, and then calculates the exposure period during which subject blur can be suppressed as the allowable exposure period and outputs it to the exposure period control unit 606.

[0103] The subject blur amount calculation unit 1001 may calculate the subject blur amount by, for example, using a motion vector. The subject blur amount calculation unit 1001 may calculate the motion vector using temporally adjacent image signals. Temporarily adjacent image signals may be the image signals of subframe 1_1 and subframe 1_2 in Figure 7, or the image signals of mainframe MF1 and mainframe MF2. However, when using subframes, the exposure periods in each image signal are different. Therefore, in this case, the signal processing unit 609 may pre-process the image signals with sensitivity boosting or desensitizing to match the exposure of one of the image signals, and output this to the subject blur amount calculation unit 1001.

[0104] The subject blur amount calculation unit 1001 may calculate motion vectors using the block matching method or the gradient method. When using the block matching method, the subject blur amount calculation unit 1001 may divide the image signal into any number of blocks and perform correlation calculations for each block. The subject blur amount calculation unit 1001 may use the correlation calculation results as motion vectors to determine the amount of movement between temporally adjacent images, i.e., the subject blur amount.

[0105] The subject blur amount calculation unit 1001 may calculate the allowable exposure period for the next main frame based on the calculated subject blur amount. Here, the subject blur amount calculation unit 1001 should calculate the allowable exposure period such that there is no subject blur in the next main frame. The subject blur amount calculation unit 1001 may set the allowable exposure period to the time during which the product of the subject blur amount and the allowable number of blur pixels falls within the allowable number of pixels. Since the subject blur amount represents how many pixels moved per unit period, the subject blur amount calculation unit 1001 should calculate the subject blur amount based on the time difference of the image from which the motion vector was calculated and the motion vector. The subject blur amount calculation unit 1001 should set the allowable exposure period so as to satisfy the following conditions. Allowable exposure period ≤ Allowable number of pixels to blur / Subject blur amount

[0106] Since the exposure period that can be set in the imaging device is discrete, the subject blur amount calculation unit 1001 may readjust the main period so that it satisfies the above-mentioned allowable exposure period and is also an exposure period that can be set in the imaging device.

[0107] Figure 11 is a flowchart of the imaging process of the imaging device according to the third embodiment. Steps that overlap with those in Figures 8 and 9 are explained in a simplified or omitted manner.

[0108] In the third embodiment, if blur is detected in the frame of the previous main period, in S1101, the subject blur amount calculation unit 1001 calculates the allowable exposure period. Specifically, in S1103, which processes the frame of the previous main period, the subject blur amount calculation unit 1001 calculates the subject blur amount for each subframe and calculates the allowable exposure period based on the subject blur amount. Here, a flowchart is shown in which the subject blur amount calculation unit 1001 calculates the subject blur amount on a subframe basis. Note that if the subject blur amount is calculated on a main frame basis, the subject blur amount calculation unit 1001 may calculate the subject blur amount after S811 or S809 and before the start of exposure for the next main frame, and then calculate the allowable exposure period.

[0109] In S1102, the exposure period control unit 606 sets the exposure period for the main frame (i.e., the main period) based on the allowable exposure period calculated by the subject blur amount calculation unit 1001, and starts exposure. Here, the exposure period control unit 606 may set the allowable exposure period as the main period if it is possible to set it as the allowable exposure period. On the other hand, if the exposure period control unit 606 cannot set the allowable exposure period as the main period, it may set a period shorter than the allowable exposure period as the main period. This makes it possible to take pictures in which subject blur is within an acceptable range.

[0110] As described above, the third embodiment controls exposure based on the amount of subject blur, using a main period based on the allowable exposure period. This allows the third embodiment to suppress blur even if blur occurs in the previous frame, while extending the exposure period within an allowable range.

[0111] In the third embodiment, if no blur is detected in the previous frame, the detection exposure period is used as the main exposure period, so that blur can be suppressed while exposure can be performed for an appropriate exposure period or an exposure period desired by the user.

[0112] (Other examples) Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the embodiments described above may be combined as appropriate. Regarding the method for detecting subject blur, a detection method based on motion vectors has been described, but the invention is not limited to this, and subject blur may be detected in a region where the difference amount of image signals between temporally adjacent frames exceeds a certain threshold.

[0113] The embodiments described above may be combined. When embodiments are combined, the user may be allowed to select any of the configurations or processes of the embodiments, or the information processing device may be configured to set them automatically.

[0114] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0115] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) An information processing device for controlling an imaging device having a photoelectric conversion element that counts pulse signals corresponding to the incidence of light and generates an image signal, Exposure period control means for controlling the exposure period of the main frame based on the main period, which is the exposure period of the main frame, A motion blur detection means reads out the image signal of the exposed subframe for each unit period shorter than the main period, from the start of exposure to the end of the unit period, and detects motion blur of the subject. An output means that outputs an output signal based on an image signal selected from either the main frame or the subframe, based on the results of blur detection, An information processing device characterized by comprising: (Item 2) The output means is If no blur is detected, select the image signal from the main frame. If blur is detected, select the image signal from the subframe prior to the blur detection. The information processing device described in item 1, characterized by the features described herein. (Item 3) The main period and the sub-period are integer multiples of the unit period. An information processing device according to item 1 or item 2, characterized in that it is an information processing device according to item 1 or item 2. (Item 4) If the blur detection means does not detect blur in the read subframe, it saves the image signal of the subframe. An information processing device according to any one of items 1 to 3, characterized by the features described in item 1 to 3. (Item 5) If the aforementioned vibration is detected, The exposure period control means stops the exposure, The output means outputs an output signal based on the image signal of the stored subframe. The information processing device described in item 4, characterized by the features described herein. (Item 6) The exposure period for determination is either the exposure period calculated by photometry or the exposure period set by the user. The exposure period control means is If blur was detected in the previous main period, the longest sub-period from the start of exposure during which no blur was detected will be saved as the storage period. If blurring is detected during the previous main period, and the detection exposure period is longer than the storage period, the storage period is used as the main period for exposure control. An information processing device according to any one of items 1 to 5, characterized by the features described in item 1 to 5. (Item 7) The exposure period for determination is either the exposure period calculated by photometry or the exposure period set by the user. The exposure period control means is If blur was detected in the previous main period, the longest sub-period from the start of exposure during which no blur was detected will be saved as the storage period. If no blur was detected in the previous main period, or if the detection exposure period is not longer than the storage period, the detection exposure period is used as the main period for exposure control. An information processing device according to any one of items 1 to 6, characterized by the features described in item 1 to 6. (Item 8) The output means, when blur is detected, performs sensitivity enhancement processing on the image signal of the subframe. An information processing device according to any one of items 1 to 7, characterized by the features described in item 1 to 7. (Item 9) The output means performs the sensitization process by multiplying the image signal by a ratio (= main period / the longest sub-period among the sub-periods in which no blur was detected). The information processing device described in item 8, characterized by the features described herein. (Item 10) The system includes a blur amount calculation means that calculates an exposure period as an allowable exposure period, based on the blur amount, which is the amount of blur during the previous main period, such that the product of the blur amount and the blur amount is smaller than a predetermined threshold. The exposure period control means controls the exposure during the main period based on the allowable exposure period if blurring is detected during the previous main period. An information processing device according to any one of items 1 to 9, characterized in that it is an information processing device. (Item 11) The exposure period for determination is either the exposure period calculated by photometry or the exposure period set by the user. The exposure period control means controls the exposure during the determination exposure period if no blur is detected during the previous main period. The information processing device according to item 10, characterized in that it is a processing device. (Item 12) The information processing device described in item 1, Multiple pixels having photoelectric conversion means for converting light into electrical signals, A waveform shaping means for generating pulses based on signals from an image, a counter means for starting to count the pulses upon receiving the pulses and counting the number of pulses as a count value, and a processing means associated with each pixel having a memory for recording the count value as the image signal, An imaging device equipped with the following features. (Item 13) An information processing method for controlling an imaging device having a photoelectric conversion element that counts pulse signals corresponding to the incidence of light to generate an image signal, An exposure period control process that controls the exposure period of the mainframe based on the main period, which is the exposure period of the mainframe, A blur detection process is performed to detect subject blur by reading out the image signal of the exposed subframe during a sub-period from the start of exposure to the end of the unit period, for each unit period shorter than the main period, An output step that outputs an output signal based on an image signal selected from either the main frame or the subframe, based on the results of blur detection. An information processing method characterized by comprising: (Item 14) A program to cause a computer to function as one of the information processing devices described in any one of items 1 through 11.

[0116] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0117] 100...Photoelectric conversion element, 103...Signal processing unit, 600...Imaging device, 603...Shake detection unit, 606...Exposure period control unit, 608...Output signal selection unit, 609...Signal processing unit, 1200...Information processing unit, 1001...Subject blur amount calculation unit.

Claims

1. An information processing device for controlling an imaging device having a photoelectric conversion element that counts pulse signals corresponding to the incidence of light and generates an image signal, Exposure period control means for controlling the exposure period of the main frame based on the main period, which is the exposure period of the main frame, A motion blur detection means reads out the image signal of the exposed subframe for each unit period shorter than the main period, from the start of exposure to the end of the unit period, and detects motion blur of the subject. An output means that outputs an output signal based on an image signal selected from either the main frame or the subframe, based on the results of blur detection, An information processing device characterized by comprising:

2. The output means is If no blur is detected, select the image signal from the main frame. If blur is detected, select the image signal from the subframe prior to the blur detection. The information processing apparatus according to feature 1.

3. The main period and the sub-period are integer multiples of the unit period. The information processing apparatus according to feature 1.

4. If the blur detection means does not detect blur in the read subframe, it saves the image signal of the subframe. The information processing apparatus according to feature 1.

5. If the aforementioned vibration is detected, The exposure period control means stops the exposure, The output means outputs an output signal based on the image signal of the stored subframe. The information processing apparatus according to feature 4.

6. The exposure period for determination is either the exposure period calculated by photometry or the exposure period set by the user. The exposure period control means is If blur was detected in the previous main period, the longest sub-period from the start of exposure during which no blur was detected will be saved as the storage period. If blurring is detected during the previous main period, and the detection exposure period is longer than the storage period, the storage period is used as the main period for exposure control. The information processing apparatus according to feature 1.

7. The exposure period for determination is either the exposure period calculated by photometry or the exposure period set by the user. The exposure period control means is If blur was detected in the previous main period, the longest sub-period from the start of exposure during which no blur was detected will be saved as the storage period. If no blur was detected in the previous main period, or if the detection exposure period is not longer than the storage period, the detection exposure period is used as the main period for exposure control. The information processing apparatus according to feature 1.

8. The output means, when blur is detected, performs sensitivity enhancement processing on the image signal of the subframe. The information processing apparatus according to feature 1.

9. The output means performs the sensitization process by multiplying the image signal by a ratio (= main period / the longest sub-period among the sub-periods in which no blur was detected). The information processing apparatus according to feature 8.

10. The system includes a blur amount calculation means that calculates an exposure period as an allowable exposure period, based on the blur amount, which is the amount of blur during the previous main period, such that the product of the blur amount and the blur amount is smaller than a predetermined threshold. The exposure period control means controls the exposure during the main period based on the allowable exposure period if blurring is detected during the previous main period. The information processing apparatus according to feature 1.

11. The exposure period for determination is either the exposure period calculated by photometry or the exposure period set by the user. The exposure period control means controls the exposure during the determination exposure period if no blur is detected during the previous main period. The information processing apparatus according to feature 10.

12. The information processing apparatus according to claim 1, Multiple pixels having photoelectric conversion means for converting light into electrical signals, A waveform shaping means for generating pulses based on signals from an image, a counter means for starting to count the pulses upon receiving the pulses and counting the number of pulses as a count value, and a processing means associated with each pixel having a memory for recording the count value as the image signal, An imaging device equipped with the following features.

13. An information processing method for controlling an imaging device having a photoelectric conversion element that counts pulse signals corresponding to the incidence of light to generate an image signal, An exposure period control process that controls the exposure period of the mainframe based on the main period, which is the exposure period of the mainframe, A blur detection process is performed to detect subject blur by reading out the image signal of the exposed subframe during a sub-period from the start of exposure to the end of the unit period, for each unit period shorter than the main period, An output step that outputs an output signal based on an image signal selected from either the main frame or the subframe, based on the results of blur detection. An information processing method characterized by comprising:

14. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 11.

Images