Image processing device, image processing method, program, and storage medium

Abstract

To reduce processing load while maintaining the accuracy of subject recognition. [Solution] An image processing apparatus comprising: an acquisition unit that acquires multiple subframes from a photoelectric conversion element capable of non-destructive readout for each of multiple subframes with different exposure periods during the imaging period of one frame; a recognition unit that performs a predetermined recognition process on the multiple subframes acquired by the acquisition unit in the first frame; and a specification unit that identifies a subframe from among the multiple subframes for which the result of the predetermined recognition process by the recognition unit satisfies predetermined conditions, wherein when the recognition unit performs a predetermined recognition process on the multiple subframes acquired by the acquisition unit in the second frame, which is a frame later than the first frame, it performs a recognition process on a subframe from among the multiple subframes acquired by the acquisition unit in the second frame that corresponds to a subframe for which the specification unit identified in the first frame satisfies predetermined conditions.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, a program, and a storage medium.

Background Art

[0002] In recent years, a photoelectric conversion element has been developed that digitally counts the number of photons incident on an avalanche photodiode (APD) and outputs the count value from a pixel as a digitally converted photoelectric signal. It is known that a photoelectric conversion element equipped with an APD enables non-destructive reading, unlike conventional CMOS sensors. Patent Document 1 discloses an imaging device that has a photoelectric conversion element equipped with an APD and can acquire a plurality of images whose exposure periods overlap each other.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the field of image recognition technology, it is well known that when an image is too bright, too dark, or there is image blur in the subject, the recognition rate tends to decrease.

[0005] Therefore, in an imaging device that can acquire a plurality of images whose exposure periods overlap each other as in Patent Document 1, if recognition processing is executed on all of the captured plurality of images, the processing load increases.

[0006] Therefore, the problem to be solved by the present invention is to reduce the processing load caused by image recognition processing.

Means for Solving the Problems

[0007] To solve the above problems, an image processing apparatus according to one aspect of the present invention includes: an acquisition unit that acquires a plurality of subframes from a photoelectric conversion element capable of non-destructive readout for each of a plurality of subframes with different exposure periods during the imaging period of one frame; a recognition unit that performs a predetermined recognition process on the plurality of subframes acquired by the acquisition unit in the first frame; and a specification unit that identifies a subframe from the plurality of subframes in which the result of the predetermined recognition process by the recognition unit satisfies predetermined conditions, wherein when the recognition unit performs the predetermined recognition process on the plurality of subframes acquired by the acquisition unit in the second frame, which is a frame later than the first frame, the recognition unit performs the recognition process on a subframe from the plurality of subframes acquired by the acquisition unit in the second frame that corresponds to a subframe in which the specification unit identified in the first frame satisfies predetermined conditions. [Effects of the Invention]

[0008] According to the present invention, the processing load due to image recognition can be reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the configuration of a photoelectric conversion element. [Figure 2] This figure shows an example configuration of the sensor board 11. [Figure 3] This figure shows an example of the configuration of the circuit board 21. [Figure 4] Figures 2 and 3 show the equivalent circuits of pixel 101 and the signal processing circuit 103 corresponding to pixel 101. [Figure 5] This diagram schematically illustrates the relationship between the operation of the APD201 and its output signals. [Figure 6] This is a functional block diagram of the photoelectric converter 600 and the mobile unit 700. [Figure 7] This is a diagram illustrating the photoelectric conversion method performed by the camera control unit 605. [Figure 8]This figure shows an example of an image divided into multiple frames. [Figure 9] This diagram shows the relationship between the memory circuit and the buffer. [Figure 10] This flowchart shows details of an example of driving a photoelectric conversion element. [Figure 11] This is a flowchart that continues from Figure 10. [Figure 12] This flowchart shows details of an example of a drive that reduces processing load. [Figure 13] This flowchart shows details of an example of a drive that reduces processing load when there are multiple subjects. [Modes for carrying out the invention]

[0010] The embodiments for carrying out the present invention will be described in detail below with reference to the attached drawings. The embodiments described below are examples of means for realizing the present invention and should be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions, and the present invention is not limited to the embodiments described below. Furthermore, some of the embodiments described later may be combined as appropriate.

[0011] Figure 1 shows an example of the configuration of a photoelectric conversion element according to this embodiment. In the following description, a photoelectric conversion device having a so-called stacked structure, in which the photoelectric conversion element 100 is composed of two substrates, a sensor substrate 11 and a circuit substrate 21, stacked and electrically connected, will be used as an example. However, it may also be a so-called non-stacked structure in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes the signal detected in the pixel region 12.

[0012] Figure 2 shows an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes multiple pixels 101 arranged in a two-dimensional manner across multiple rows and columns. Each pixel 101 is equipped with a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as APD).

[0013] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the light reception frequency of photons. Note that the number of rows and columns of the pixel array forming the pixel region 12 is not particularly limited.

[0014] FIG. 3 is a diagram showing a configuration example of the circuit board 21. The circuit board 21 includes a signal processing circuit 103, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, vertical signal lines 113, a vertical scanning circuit 110, and an output circuit 114 that process the charges photoelectrically converted by each photoelectric conversion unit 102 in FIG. 2.

[0015] The vertical scanning circuit 110 receives the control pulses supplied from the control pulse generation unit 115 and sequentially supplies the control pulses to a plurality of pixels arranged in the row direction. Logic circuits such as a shift register and an address decoder are used for the vertical scanning circuit 110.

[0016] The signals output from the photoelectric conversion unit 102 of each pixel are processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and digital values are held in the memory. The horizontal scanning circuit 111 inputs a control pulse for sequentially selecting each column to the signal processing circuit 103 in order to read out signals from the memories of each pixel in which digital signals are held.

[0017] Signals are output from the signal processing circuit 103 of the pixels in the row selected by the vertical scanning circuit 110 to the vertical signal lines 113. The signals output to the vertical signal lines 113 are output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 incorporates a plurality of buffers connected to each vertical signal line 113.

[0018] As shown in Figures 2 and 3, multiple signal processing circuits 103 are arranged in the region that overlaps with the pixel region 12 in a plan view. Then, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged so as to overlap between the edge of the sensor substrate 11 and the edge of the pixel region 12 in a plan view.

[0019] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in a plan view.

[0020] Note that the arrangement of the vertical signal line 113, the readout circuit 112, and the output circuit 114 is not limited to the example shown in Figure 3. For example, the vertical signal line 113 may be arranged extending in the row direction, and the readout circuit 112 may be placed at the end of the vertical signal line 113. Also, the signal processing circuit 103 does not necessarily need to be provided for each photoelectric conversion unit; a single signal processing unit may be shared by multiple photoelectric conversion units, and sequential signal processing may be performed.

[0021] Figure 4 shows the equivalent circuit of the pixel 101 in Figures 2 and 3 and the signal processing circuit 103 corresponding to the pixel 101.

[0022] The APD201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD201 is connected to a power line to which a drive voltage VL (first voltage) is supplied. The other of the two nodes of the APD201 is connected to a power line to which a drive voltage VH (second voltage), which is higher than voltage VL, is supplied.

[0023] In Figure 4, one node of the APD201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD201 so that the APD201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated.

[0024] When a reverse bias voltage is supplied, there are two modes of operation: Geiger mode, where the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, where the voltage difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0025] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power line to which a drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD 201.

[0026] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD201 and thereby suppressing avalanche multiplication (quench operation). In addition, the quench element 202 also works to restore the voltage supplied to the APD201 to the drive voltage VH by flowing the current that compensates for the voltage drop caused by the quench operation (recharge operation).

[0027] Figure 4 shows an example in which the signal processing circuit 103 has a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202. However, the signal processing circuit 103 only needs to have at least one of the following components in addition to the quench element 202: the waveform shaping unit 210, the counter circuit 211, and the memory circuit 212.

[0028] The waveform shaping unit 210 shapes the cathode voltage change of the APD201 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 210. Figure 4 shows an example in which one inverter is used as the waveform shaping unit 210, but a circuit in which multiple inverters are connected in series may be used, or other circuits that have a waveform shaping effect may be used.

[0029] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When the control pulse RES is supplied via the drive line 213, the signal held by the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation time (exposure period).

[0030] The memory circuit 212 receives control pulses SEL from the vertical scanning circuit 110 in Figure 3 via the drive line 214 (not shown in Figure 3) in Figure 4, which switches the electrical connection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the counter's count value and outputs the output signal from the pixel's counter circuit 211 to the vertical signal line 113.

[0031] Furthermore, switches such as transistors may be placed between the quench element 202 and the APD201, or between the photoelectric conversion unit 102 and the signal processing circuit 103, to switch the electrical connections. Similarly, the supply of voltage VH or voltage VL to the photoelectric conversion unit 102 may be electrically switched using switches such as transistors.

[0032] Figure 5 schematically shows the relationship between the operation of APD201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to APD201. When a photon is incident on APD201 at time t1, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0033] As the voltage drop increases further and the potential difference applied to APD201 decreases, the avalanche multiplication of APD201 stops, as at time t2, and the voltage level at nodeA no longer drops below a certain value. Subsequently, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA settles to its original potential level. At this time, any portion of the output waveform at nodeA that exceeds a certain threshold is shaped by the waveform shaping unit 210 and output as a pulse signal at nodeB.

[0034] Next, the photoelectric converter 600 of this embodiment will be described. Figure 6 is a functional block diagram of the photoelectric converter 600 according to this embodiment. Note that some of the functional blocks shown in Figure 6 are realized by having a computer (not shown) included in the photoelectric converter 600 execute a computer program stored in a memory (not shown) which is a storage medium.

[0035] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, each functional block shown in Figure 6 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0036] (Regarding photoelectric converters) The photoelectric conversion device 600 includes the photoelectric conversion element 100, imaging optical system 601, image processing unit 603, recognition unit 604, camera control unit 605, memory unit 606, communication unit 607, etc., as described in Figures 1 to 5. The photoelectric conversion element 100 is composed of an avalanche photodiode, as described in Figures 1 to 5, for photoelectric conversion of an optical image. Since the photoelectric conversion element 100 uses an APD, non-destructive readout is possible. That is, the charge does not degrade when the signal is read out. Therefore, even if the signal is repeatedly read out in multiple subframes during the exposure period of one frame, the original signal can be read out without degradation. In this embodiment, the case in which a photoelectric conversion element using an APD is used is described, but it is not necessarily required to have an APD as long as the image sensor is capable of non-destructive readout.

[0037] The image processing unit (acquisition unit) 603 performs image processing on the image signal acquired by the photoelectric conversion element 100, such as black level correction, gamma curve adjustment, noise reduction, and data compression, to generate the final image signal. Furthermore, if the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable for the image processing unit 603 to perform processing such as white balance correction and color conversion.

[0038] Furthermore, the output of the image processing unit 603 is supplied to the recognition unit 604 and the camera control unit 605. The recognition unit 604 performs predetermined recognition processing based on the image signal. Predetermined recognition processing includes, for example, subject recognition of people or vehicles, character recognition, image recognition, or specific anomaly recognition such as a specific fire. These recognition processes are affected by the quality of the image signal and include recognition processes affected by changes in exposure time, such as image brightness and subject blur. In this embodiment, subject recognition will be described as the premise. The recognition unit 604 also calculates the recognition rate as the confidence (likelihood) of the subject. For example, if the subject to be recognized is a person, the confidence (likelihood) that the subject in the image is a person is called the recognition rate, and in this embodiment, the recognition rate is expressed as 0 to 100%. A higher recognition rate indicates a higher confidence (likelihood).

[0039] The camera control unit 605 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control each part of the photoelectric converter 600.

[0040] Furthermore, the camera control unit 605 functions as a control means, and for example, via the control pulse generation unit for the photoelectric conversion element 100, it controls the length of the exposure period for each frame of the photoelectric conversion element 100 and the timing of the control signal CLK.

[0041] The storage unit 606 includes, for example, a recording medium such as a memory card or hard disk, and can store and read image signals. The communication unit 607 is equipped with wireless and wired interfaces and outputs the generated image signals to the outside of the photoelectric converter 600 and receives various signals from the outside.

[0042] 608 is a network consisting of multiple routers, switches, cables, etc. that satisfy communication standards such as Ethernet (registered trademark), and clients transmit control signals and image signals of the photoelectric converter 600 via the network 608.

[0043] Note that the image processing unit 603, recognition unit 604, storage unit 606, etc., shown in Figure 6 do not necessarily have to be mounted on the photoelectric converter 600. For example, they may be provided on an external terminal for remotely controlling the photoelectric converter 600, which is provided separately from the photoelectric converter 600.

[0044] Figure 7 is a diagram illustrating the photoelectric conversion method by the camera control unit 605 according to this embodiment. In this embodiment, one frame with a length of 33.3 ms is divided into four parts. That is, as shown in Figure 7, frame 1 (the first frame) is divided into frames 1_1, 1_2, 1_3, and 1_4 with equal periods of time (8.33 ms). In other words, frames 1_1, 1_2, 1_3, and 1_4 are multiple subframes in the first frame, and non-destructive readout is possible from the photoelectric conversion element 100 for each of the multiple subframes.

[0045] Frame 1_1 has an accumulation time (exposure period) from the start time T0 to time T1 of Frame 1, and Frame 1_2, being a non-destructive readout, has an accumulation time (exposure period) from time T0 to time T2. Frame 1_3 has an accumulation time (exposure period) from time T0 to time T3, and Frame 1_4 has an accumulation time (exposure period) from time T0 to time T4. In other words, the multiple subframes read out during the accumulation time (exposure period) of Frame 1 each have different accumulation time (exposure period) lengths.

[0046] Then, at time T0, the counter circuit 211 is reset, and at times T1 to T4, the count values ​​C1_1, C1_2, C1_3, and C1_4 are obtained from the counter circuit 211, respectively.

[0047] Furthermore, the count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory circuit 212. The signal for one line temporarily stored in the memory circuit 212 is then output sequentially from the photoelectric conversion element 100 via the buffer of the read circuit 112.

[0048] Thus, according to this embodiment, the signal accumulated during the period of frame 1_1 is read out from time T1 to T2 and processed quickly by the recognition unit 604. Therefore, image recognition can be performed quickly. Similarly, the signals accumulated during the periods of frame 1_2, frame 1_3, and frame 1_4 are read out sequentially from time T2 to T3, T3 to T4, and T4 to T1, respectively, and image recognition can be performed repeatedly. In other words, during the imaging period of one frame, the image processing unit (acquisition unit) 603 acquires multiple subframes (image signals) with different exposure periods, and the recognition unit 604 performs predetermined recognition processing on the acquired subframes.

[0049] Figure 8 shows an example of images divided into multiple frames. As shown in Figure 8, the image in frame 1_1 is dark because the exposure time is short, but there is little motion blur in the moving person. On the other hand, motion blur occurs in the order of frame 1_2, frame 1_3, and frame 1_4 as the exposure time increases. Notably, there is no blur in stationary vehicles or white lines, and the contrast improves as the exposure time increases.

[0050] Thus, in this embodiment, there is a first storage time (exposure period) and a second storage time (exposure period) within one frame. The first storage time is shorter than the second storage time, and the signal generated during the first storage time is controlled to be output between the end of the first storage time and the end of the second storage time (exposure period).

[0051] Furthermore, in this embodiment, the first storage time (exposure period) and the second storage time (exposure period) overlap, and the first and second storage times begin simultaneously. Moreover, the end of the second storage time marks the end of a frame, and the second storage time is an integer multiple of the first storage time.

[0052] In other words, images with short and long storage times (exposure periods) are created, and the timing of the end of the short storage time (exposure period) is set to be earlier than the timing of the end of the long storage time (exposure period). Then, as soon as the short storage time (exposure period) is finished, the image is output and sent to the subsequent recognition unit 604. The recognition unit then recognizes the subject based on the image signal generated during at least the first storage time (exposure period). The recognition unit 604 recognizes the subject based on the signal generated during at least the first storage time (exposure period).

[0053] Figure 9 shows the relationship between the memory circuit and the buffer in this embodiment. In Figure 9, the memory circuits 212 in the signal processing circuit 103 of Figure 3 are arranged in an N row and M column configuration, and each memory circuit is represented as memory 1-1 to memory NM. Also, buffers 1 to buffer M in Figure 9 represent the buffers included in the read circuit 112 in Figure 3. The output circuit 114 in Figure 9 corresponds to the output circuit 114 in Figure 3.

[0054] Figure 10 is a flowchart showing details of an example of driving the photoelectric conversion element in this embodiment, and Figure 11 is a flowchart that continues from Figure 10. The CPU and other components of the computer within the camera control unit 605 execute the computer program stored in memory, thereby sequentially performing the operations of each step in the flowcharts of Figures 10 and 11.

[0055] In step S101 of Figure 10, i is set to 1. Next, in step S102, the count value Count of the counter circuit 211 at time Ti is output to the memory circuit 212. At this time, the output is made simultaneously to all memory circuits. This operation corresponds to the operation at time T1 in Figure 7.

[0056] Next, in step S103, j=1 is set, and in step S104, the count value Count(jki) in the memory circuit jk in Figure 9 is output to the buffer. At this time, columns 1 to M are output to the buffer simultaneously. This operation means that the count value of the first row in Figure 9 is taken into the buffer.

[0057] Next, in step S105, k is set to 1, and in step S106, the count value of buffer k, Count(jki), is output to the output circuit 114. This operation corresponds to reading the buffer signal in the leftmost column of Figure 9 from the output circuit.

[0058] Next, proceed to step S107 in FIG. 11 via A, and determine in step S107 whether k < M. If No, then in step S108, increment k by 1 such that k = k + 1, return to step S106 via B, and perform the operation of step S106. This operation corresponds to the operation of reading the signals of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0059] When it becomes No in step S107, that is, when k = M, it means that the signals of the buffer in the M-th column of FIG. 9 have been read out from the output circuit. Next, proceed to step S109 and determine whether j < N. If Yes in step S109, then in step S110, increment j by 1 such that j = j + 1, and return to step S104 via C. This corresponds to the operation for starting the reading of the next line.

[0060] When it is determined as No in step S109, it means that the reading of all lines has ended. So, proceed to step S111 and determine whether i < 4. If it is determined as Yes in step S111, then increment i by 1 such that i = i + 1 and return to step S102 via D. This operation corresponds to the operation of starting the reading at the next time T2.

[0061] When it is determined as No in step S111, it means that the reading at time T4 has been completed. So, proceed to step S113 and reset the counter circuit 211 with a reset signal. This operation corresponds to the reset operation of the counter circuit 211 at time T4 in FIG. 7. In the above manner, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0062] As mentioned above, the image becomes darker during the first storage time (exposure period), but the blur of moving subjects is reduced. The image becomes brighter during the second storage time (exposure period), but the blur of moving subjects is increased. Therefore, the optimal storage time (exposure period) in terms of subject recognition rate differs depending on the brightness of the shooting environment and the speed of the subject's movement. In this case, it is necessary to have the recognition unit 604 process the images acquired for multiple storage times (exposure periods) and calculate the recognition rate. In this embodiment, by performing image recognition on each of frames 1_1, 1_2, 1_3, and 1_4, calculating the recognition rate for each, and comparing them, it is possible to determine which storage time (exposure period) is optimal for recognizing the subject. However, if the recognition unit 604 performs recognition processing on multiple images acquired for multiple storage times (exposure periods), the number of images to be processed increases, and the processing load increases. Performing recognition processing on multiple images acquired for multiple storage times (exposure periods) in each frame further increases the processing load.

[0063] However, if the recognition rate of the subject does not change significantly, there is no need to perform recognition processing every frame and compare the recognition rates. Therefore, under conditions where the recognition rate of the subject does not change significantly, the recognition rate of images from multiple storage times (exposure periods) acquired once is calculated, and from the next frame onward, recognition processing is performed only on images from storage times (exposure periods) where the recognition rate was above a certain level. This makes it possible to reduce the processing load of the recognition unit 604 while maintaining the recognition rate.

[0064] Figure 12 is a flowchart detailing an example of a drive that reduces the processing load in this embodiment. Note that the CPU and other components within the camera control unit 605 execute a computer program stored in memory, which sequentially performs the operation of each step in the flowchart of Figure 12.

[0065] In step S114, the recognition unit 604 determines whether the target subject has been recognized. If No, it means that the target subject has not been recognized, and monitoring continues in step S114 until the target subject is recognized. If Yes, the target subject has been recognized, and the process proceeds to step S101.

[0066] Step S101 is the same process as in Figure 10, so the explanation is omitted. In step S115, the recognition unit 604 calculates the recognition rate of the target subject.

[0067] In step S116, it is determined whether the recognition rate calculated in step S115 is equal to or greater than a predetermined recognition rate. If the answer is Yes, it means that the recognition rate of the subject is high, so the process proceeds to step S117, and the accumulation time (exposure period) information when the target image was captured is stored in the memory of the camera control unit 605. If the answer is No, it means that the recognition rate of the subject is low, so the process proceeds to step S118, and the accumulation time (exposure period) information when the target image was captured is not stored in the memory of the camera control unit 605. In other words, the camera control unit 605 functions as a identifying unit that identifies a subframe in which the result of the recognition processing by the recognition unit 604 satisfies predetermined conditions. In this embodiment, the predetermined conditions are described as the recognition rate of the subject being equal to or greater than a predetermined recognition rate. Other predetermined conditions will be described later.

[0068] Steps S111 and S112 involve the same process as shown in Figure 11, so their explanation is omitted.

[0069] If the result in step S111 is determined to be "No", the calculation of the recognition rate for each of the frames 1_1, 1_2, 1_3, and 1_4 is completed. Consider the case where the recognition unit 604 performs recognition processing on frames 2 and later (second frames), which are frames after frame 1. In this case, the recognition processing is performed on the subframes read out at the storage time (exposure period) stored in the memory of the camera control unit 605 in step S117. That is, the recognition processing is performed on the subframes read out at the storage time (exposure period) corresponding to the subframes that satisfy predetermined conditions identified by the camera control unit 605.

[0070] In step S120, it is determined whether the recognition rate of the same subject as the subject recognized in step S114 in the acquired image is above a predetermined recognition rate. If yes, the process returns to step S119 and continues the recognition process only for the storage time (exposure period) stored in the memory of the camera control unit 605 in step S117. The recognition unit 604 performs recognition processing on subframes read out at the storage time (exposure period) corresponding to subframes that satisfy the predetermined conditions. On the other hand, it does not perform recognition processing on subframes read out at the storage time (exposure period) corresponding to subframes that do not satisfy the predetermined conditions. If no, it means that the recognition rate of the subject did not meet the predetermined recognition rate in any of the images, and it is necessary to compare the recognition rates of multiple images acquired at different storage times (exposure periods) again, so this flowchart is terminated. Possible events that cause a decrease in the recognition rate of the subject include sudden changes in the brightness of the shooting environment or the speed of the subject's movement. Also, if the subject goes out of frame, the recognition rate of the subject will not meet the predetermined recognition rate.

[0071] In this embodiment, we demonstrated an example in which the processing load can be reduced by performing recognition processing on subframes read out with an accumulation time (exposure period) at or above a predetermined recognition rate. However, the predetermined condition may be set to, for example, the recognition rate being the highest among multiple subframes. That is, by performing recognition processing on the subframe corresponding to the accumulation time (exposure period) at which the recognition rate is highest, it is possible to reduce the number of recognition processes and thus reduce the processing load.

[0072] In this embodiment, the explanation was based on the assumption that there is only one subject in the acquired image. If there are multiple subjects in the acquired image, the recognition rate of the images acquired for multiple storage times (exposure periods) is calculated, and thereafter, only the images with a recognition rate above a certain level are processed for recognition. This makes it possible to reduce the processing load of the recognition process.

[0073] Furthermore, in this embodiment, a case was described in which the subframe in which the recognition process is performed is determined based on whether the recognition rate of the subject is above a predetermined recognition rate. In other words, the decision of which subframe in the next frame to perform the recognition process on is made based on whether the result of the recognition process by the recognition unit 604 satisfies predetermined conditions.

[0074] These predetermined conditions may include not only whether the aforementioned recognition rate is above a predetermined recognition rate, but also, for example, whether the recognition rate is above a predetermined recognition rate in facial recognition. In other words, a subframe corresponding to an accumulation time (exposure period) with a high recognition rate is identified and the authentication process is executed. In addition, the predetermined conditions may include not only a comparison of recognition rates, but also recognition feasibility, such as whether a specific string or subject was recognized. In other words, the recognition process is executed on a subframe corresponding to an accumulation time (exposure period) in which only a specific subject is recognized. This is applied to consider additional conditions, such as identifying a subframe corresponding to an accumulation time (exposure period) in which only the bright / dark subject is recognized among subjects in bright and dark areas in an environment with a large difference in brightness.

[0075] Figure 13 is a flowchart detailing an example of a drive that reduces the processing load when there are multiple subjects in this embodiment. Note that the CPU and other components acting as a computer within the camera control unit 605 execute the computer program stored in memory, thereby sequentially performing the operations of each step in the flowchart in Figure 12.

[0076] In step S121, the recognition unit 604 determines whether multiple subjects have been recognized. If No, it means that multiple subjects have not been recognized, and monitoring continues in step S121 until the target subject is recognized. If Yes, multiple subjects have been recognized, and the process proceeds to step S101.

[0077] Step S101 is the same process as in Figure 10, so its explanation is omitted.

[0078] In step S122, the recognition unit 604 calculates the recognition rate for each of the multiple subjects.

[0079] In step S123, it is determined whether the recognition rates calculated in step S122 are all above a predetermined recognition rate. If the answer is Yes, it means that the recognition rate of multiple subjects is high; if the answer is No, it means that the recognition rate of one or more subjects is low. Steps S117 and S118 are the same processes as in Figure 12, so their explanation is omitted.

[0080] Steps S111 and S112 involve the same process as shown in Figure 11, so their explanation is omitted. Similarly, step S118 involves the same process as shown in Figure 12, so its explanation is omitted.

[0081] In step S124, it is determined whether the recognition rate of multiple subjects identical to the subject recognized in step S121 in the acquired image is above a predetermined recognition rate. If yes, the process returns to step S119 and continues the recognition process only for the storage time (exposure period) stored in memory in step S117. If no, it means that the recognition rate of multiple subjects in any image did not meet the predetermined recognition rate, and it is necessary to compare the recognition rates of images acquired again for multiple storage times (exposure periods), so this flowchart is terminated. Possible events that cause a decrease in the recognition rate include sudden changes in the brightness of the shooting environment or the movement speed of the subject. Also, if the subject goes out of frame, the recognition rate of the subject will not meet the predetermined recognition rate.

[0082] In this embodiment, we have shown an example where the processing load can be reduced by processing only the accumulation time (exposure period) where the recognition rate of each of the multiple subjects is above a predetermined recognition rate. However, this is not limited to this example. For example, the processing may be limited to the accumulation time (exposure period) where the sum of the recognition rates for each subject is the highest. For example, consider an example where the recognition rate of people in frame 1_1 is 70% and the recognition rate of vehicles is 80%, and the recognition rate of people in frame 1_2 is 80% and the recognition rate of vehicles is 60%. In this case, the sum of the recognition rates in frame 1_1 is 150%, and the recognition rate in frame 1_2 is 140%, so only frame 1_1, where the sum of the recognition rates is the highest, is processed. If the predetermined recognition rate in the embodiment is 70% or higher, the recognition rates of multiple subjects in both frame 1_1 and frame 1_2 are above the predetermined recognition rate. Therefore, it is necessary to process both frame 1_1 and frame 1_2, but by processing only the frame where the sum of the recognition rates is the highest, the number of images to be processed can be reduced. Thus, the processing load can be further reduced.

[0083] Alternatively, the recognition process may be performed on each of the accumulation times (exposure periods) in which the recognition rate of multiple subjects was highest. That is, when multiple subjects are recognized in the first frame, the identification unit (camera control unit) 605 identifies a subframe from the first frame that satisfies predetermined conditions for each of the multiple subjects. For example, consider a case where the recognition rate of people is 60% and the recognition rate of vehicles is 80% in frame 1_1, and the recognition rate of people is 80% and the recognition rate of vehicles is 60% in frame 1_2. In this case, the recognition rate of people is highest in frame 1_1, and the recognition rate of vehicles is highest in frame 1_2. Therefore, in the second frame (i.e., a frame after frame 1), recognition processing should be performed on the subframes corresponding to frame 1_1 and frame 1_2, respectively.

[0084] Furthermore, a priority is set for each of the multiple subjects to be recognized. For example, if a person and a vehicle are recognized simultaneously, the person may be given priority. In this case, the recognition process may be performed on the subframe with the highest priority, where the person has been recognized and the storage time (exposure period) has a high recognition rate. This priority ranking is stored as a table in the storage unit 606, and when multiple types of subjects are recognized, the recognition rate of the subject with the highest priority among the recognized subjects is obtained. Then, it is determined whether the recognition rate of the subject with the highest priority is equal to or higher than a predetermined recognition rate.

[0085] In this embodiment, an example is shown in which the recognition rate is compared within a single frame. However, the recognition rate can also be compared across multiple frames, and the storage time (exposure period) for performing recognition processing in subsequent frames can be determined accordingly.

[0086] In this embodiment, an example is shown in which the processing load can be reduced by performing recognition processing only for the accumulation time (exposure period) during which the recognition rate of the subject is equal to or greater than a predetermined recognition rate. However, the camera control unit 605 may also control the photoelectric conversion element 100 so that it outputs only for the accumulation time (exposure period) during which the recognition rate of the subject is equal to or greater than a predetermined recognition rate. In frames from frame 2 onward (second frames), the identification unit (camera control unit) 605 has identified subframes that satisfy a predetermined condition and subframes that do not satisfy the predetermined condition. Therefore, in frames from frame 2 onward, non-destructive readout may not be performed during the exposure period of subframes that do not satisfy the predetermined condition; that is, the acquisition unit (image processing unit) 603 may not acquire the subframes themselves. This provides the additional effect of reducing the power consumed by reading out subframes. Alternatively, the camera control unit (determination unit) 605 may determine the exposure period for one frame's imaging period based on the accumulation time (exposure period) of the subframe identified by the identification unit (camera control unit) 605. This provides the additional effect of enabling imaging in each frame with an optimal accumulation time for the recognition processing of the recognition unit 604. In other words, it becomes unnecessary to read the subframe itself from the photoelectric conversion element 100.

[0087] Furthermore, since this embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not degrade when read out, so the accumulation time (exposure period) can be overlapped. Also, since there is no readout noise, the original signal does not degrade no matter how many times it is read out after a single accumulation.

[0088] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible in accordance with the spirit of the present invention, and these modifications are not excluded from the scope of the present invention.

[0089] For example, in the above embodiment, accumulation is performed for a minimum of 1 / 4 frame period, but the minimum accumulation time (exposure period) may be changed to, for example, 1 / 5 frame period or 1 / 3 frame period, depending on the recognition accuracy of the recognition unit 604. Alternatively, the minimum accumulation time (exposure period) may be changed depending on the brightness of the subject.

[0090] Furthermore, even if the readout cycle is set to every 1 / 4 frame period, the actual storage time (exposure period) may be shorter than the 1 / 4 frame period, depending on the brightness of the subject and the image recognition accuracy. In other words, the counter circuit may be reset midway through the storage time (exposure period) of frame 1_1 in Figure 7.

[0091] Alternatively, the counter circuit may be reset at time T1 in Figure 7. This may adjust the count value read at time T4. This embodiment includes the following combinations.

[0092] Furthermore, although the above embodiment shows an example in which the image signal generated during the first storage time (exposure period) is output by the end of the second storage time (exposure period), the embodiment is not limited to this.

[0093] The function of the photoelectric converter 600 as an imaging device may be a separate component. In that case, it becomes an image processing device that acquires multiple subframes from the photoelectric converter element 100 by non-destructive readout during the exposure period of one frame. The image processing device only needs to have the functions of an authentication unit 604 and a identification unit (camera control unit) 605.

[0094] In addition, some or all of the control in this embodiment may be performed by supplying a computer program that realizes the functions of the above-described embodiment to the photoelectric converter via a network or various storage media. The computer (or CPU, MPU, etc.) in the photoelectric converter may then read and execute the program.

[0095] The disclosures herein include the following image processing apparatus, image processing method, program, and storage medium.

[0096] (Item 1) An acquisition unit that acquires multiple subframes from a photoelectric conversion element capable of non-destructive readout for each of multiple subframes with different exposure periods during the imaging period of one frame, A recognition unit that performs a predetermined recognition process on the plurality of subframes acquired by the acquisition unit in the first frame, The system includes a selection unit that identifies a subframe from among the plurality of subframes whose result of the predetermined recognition processing by the recognition unit satisfies predetermined conditions, When the recognition unit performs the predetermined recognition process on the plurality of subframes acquired by the acquisition unit in a second frame which is a frame later than the first frame, An image processing apparatus characterized in that, among the plurality of subframes acquired by the acquisition unit in the second frame, the recognition process is performed on the subframes corresponding to the subframes that satisfy the predetermined conditions identified by the identification unit in the first frame.

[0097] (Item 2) The image processing apparatus according to item 1, characterized in that, in the second frame, the recognition process is not performed on subframes corresponding to subframes that do not satisfy the predetermined conditions among the plurality of subframes acquired by the acquisition unit.

[0098] (Item 3) The results of the recognition process include the recognition rate of the object to be recognized. The image processing apparatus according to item 1, characterized in that the aforementioned predetermined condition is that the recognition rate is equal to or greater than a predetermined recognition rate.

[0099] (Item 4) The image processing apparatus according to item 3, characterized in that the predetermined condition is the maximum recognition rate among the recognition rates for each of the plurality of subframes in which the recognition unit performed the recognition process.

[0100] (Item 5) The image processing apparatus according to item 1, characterized in that the predetermined condition is whether or not the recognition object has been recognized by the recognition unit.

[0101] (Item 6) The image processing apparatus according to item 1, characterized in that the acquisition unit does not acquire subframes during the exposure period corresponding to subframes that do not satisfy the predetermined conditions among a plurality of subframes read out during the exposure period of the second frame.

[0102] (Item 7) A priority is set for each of the multiple subjects to be recognized by the recognition unit. The image processing apparatus according to item 3, characterized in that the predetermined conditions are that the recognition rate is equal to or greater than the predetermined recognition rate, and the subject with the highest priority is a subframe containing the recognized subject.

[0103] (Item 8) The photoelectric conversion device according to item 4, characterized in that when a plurality of subjects are recognized in the first frame by a predetermined recognition process of the recognition unit, the identifying unit identifies a subframe that satisfies the predetermined conditions for each of the plurality of subjects.

[0104] (Item 9) The system further includes a determination unit that determines the exposure period during the imaging period for one frame, The photoelectric conversion apparatus according to item 1, characterized in that the determination unit determines the exposure period for the imaging period of one frame so as to approach the exposure period of the subframe identified by the identification unit.

[0105] (Item 10) The image processing apparatus according to item 1, characterized in that the exposure period of the first frame and the exposure periods of the plurality of subframes acquired by the acquisition unit in the first frame start simultaneously.

[0106] (Item 11) The image processing apparatus according to item 1, characterized in that the photoelectric conversion element includes an avalanche photodiode.

[0107] (Item 12) An acquisition step in which multiple subframes are acquired from a photoelectric conversion element capable of non-destructive readout for each of multiple subframes with different exposure periods during the imaging period of one frame, A recognition step in which a predetermined recognition process is performed on the plurality of subframes acquired in the acquisition step in the first frame, The process includes a selection step of identifying a subframe from among the plurality of subframes whose result of the predetermined recognition process performed by the recognition step satisfies predetermined conditions, In the recognition step, if the predetermined recognition process is performed on the plurality of subframes acquired in the acquisition step in the second frame, which is a frame later than the first frame, An image processing method characterized in that, in the second frame, the recognition process is performed on a subframe that corresponds to a subframe that satisfies the predetermined conditions identified in the identification step in the first frame, among the plurality of subframes acquired in the acquisition step.

[0108] (Item 13) A program that causes a computer to execute the image processing method described in item 12.

[0109] (Item 14) A computer-readable storage medium containing the program described in item 13. [Explanation of symbols]

[0110] 11 Sensor board 12-pixel area 21 Circuit board 22 Circuit area 100 Photoelectric conversion elements 101 pixels 102 Photoelectric conversion unit 103 Signal Processing Circuit 110 Vertical scanning circuit 111 Horizontal scanning circuit 112 Readout Circuit 113 Vertical signal line 114 Output Circuit 115 Control pulse generation unit 201 Avalanche Photodiode 202 Quench 210 Waveform shaping section 211 Counter Circuit 212 Memory Circuit 213 Drive Line 600 Photoelectric converter 601 Imaging Optical System 603 Signal Processing Unit 604 Recognition part 605 Camera Control Unit 606 Storage section 607 Communications Department 608 Network

Claims

1. An acquisition unit that acquires multiple subframes from a photoelectric conversion element capable of non-destructive readout for each of multiple subframes with different exposure periods during the imaging period of one frame, A recognition unit that performs a predetermined recognition process on the plurality of subframes acquired by the acquisition unit in the first frame, The system includes a selection unit that identifies a subframe from among the plurality of subframes whose result of the predetermined recognition processing by the recognition unit satisfies predetermined conditions, When the recognition unit performs the predetermined recognition process on the plurality of subframes acquired by the acquisition unit in a second frame which is a frame later than the first frame, An image processing apparatus characterized in that, among the plurality of subframes acquired by the acquisition unit in the second frame, the predetermined recognition process is performed on the subframes corresponding to the subframes that satisfy the predetermined conditions identified by the identification unit in the first frame.

2. The image processing apparatus according to claim 1, characterized in that, in the second frame, the recognition process is not performed on subframes corresponding to subframes that do not satisfy the predetermined conditions among the plurality of subframes acquired by the acquisition unit.

3. The results of the recognition process include the recognition rate of the object to be recognized. The image processing apparatus according to claim 1, characterized in that the aforementioned predetermined condition is that the recognition rate is equal to or greater than a predetermined recognition rate.

4. The image processing apparatus according to claim 3, characterized in that the predetermined condition is the maximum recognition rate among the recognition rates for each of the plurality of subframes in which the recognition unit performed the recognition process.

5. The image processing apparatus according to claim 1, characterized in that the predetermined condition is whether or not the recognition object has been recognized by the recognition unit.

6. The image processing apparatus according to claim 1, characterized in that the acquisition unit does not acquire subframes during the exposure period corresponding to subframes that do not satisfy the predetermined conditions among a plurality of subframes read out during the exposure period of the second frame.

7. A priority is set for each of the multiple subjects to be recognized by the recognition unit. The image processing apparatus according to claim 3, characterized in that the predetermined conditions are that the recognition rate is equal to or greater than a predetermined recognition rate, and the subject with the highest priority is a subframe containing the recognized subject.

8. The photoelectric conversion apparatus according to claim 1, characterized in that the identifying unit identifies a subframe that satisfies the predetermined conditions for each of the multiple subjects when a plurality of subjects are recognized in the first frame by a predetermined recognition process of the recognition unit.

9. The system further includes a determination unit that determines the exposure period during the imaging period for one frame, The photoelectric conversion apparatus according to claim 1, characterized in that the determination unit determines the exposure period for the imaging period of one frame so as to approach the exposure period of the subframe identified by the identification unit.

10. The image processing apparatus according to claim 1, characterized in that the exposure period of the first frame and the exposure periods of the plurality of subframes acquired by the acquisition unit in the first frame are started simultaneously.

11. The image processing apparatus according to claim 1, characterized in that the photoelectric conversion element includes an avalanche photodiode.

12. An acquisition step in which multiple subframes are acquired from a photoelectric conversion element capable of non-destructive readout for each of multiple subframes with different exposure periods during the imaging period of one frame, A recognition step in which a predetermined recognition process is performed on the plurality of subframes acquired in the acquisition step in the first frame, The process includes a selection step of identifying a subframe from among the plurality of subframes whose result of the predetermined recognition process performed by the recognition step satisfies predetermined conditions, In the recognition step, if the predetermined recognition process is performed on the plurality of subframes acquired in the acquisition step in the second frame, which is a frame later than the first frame, An image processing method characterized in that, in the second frame, the predetermined recognition process is performed on a subframe that corresponds to a subframe that satisfies the predetermined conditions identified in the identification step in the first frame, among the plurality of subframes acquired in the acquisition step.

13. A program for causing a computer to execute the image processing method described in claim 12.

14. A computer-readable storage medium storing the program described in claim 13.

Images