Signal processing device, signal processing method, and detection sensor

Abstract

This technology relates to a signal processing device, a signal processing method and a detection sensor for enabling flicker information to be detected from the output indicating a brightness change. The signal processing device comprises: a counting unit that counts, in an image indicating brightness changes outputted from a light receiving unit at a predetermined frame rate, a first count number that is the count number of pixels in which a first brightness change in a plus direction is detected and a second count number that is the count number of pixels in which a second brightness change in a minus direction is detected; a coefficient generating unit that generates coefficients in accordance with times at which the brightness changes are detected; and an integrating unit that integrates multiplication results obtained by multiplying the count numbers of pixels by the coefficients. This technology can be applied, for example, to an event detection sensor or the like.

Description

Technical Field

[0001] The present technology relates to a signal processing device, a signal processing method, and a detection sensor, and more particularly to a signal processing device, a signal processing method, and a detection sensor capable of detecting flicker information from the output of an event detection sensor that detects the presence or absence of a luminance change at a fixed period.

Background Art

[0002] An image sensor has been proposed that outputs, at a fixed frame rate, whether or not an event has occurred by treating a change in the luminance of a pixel as an event (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such an event detection sensor that detects the presence or absence of a luminance change at a fixed period, for example, when the light source in a room is flickering and a luminance change occurs over the entire screen, resulting in many events, the events originally desired to be acquired are buried among a large number of events caused by flicker, and the information originally desired to be acquired cannot be captured.

[0005] The present technology has been made in view of such a situation, and enables detection of flicker information from the output of an event detection sensor that detects the presence or absence of a luminance change at a fixed period.

Means for Solving the Problems

[0006] The signal processing device according to the first aspect of the present technology includes a counting unit that counts a first count number, which is the number of pixels in an image indicating a luminance change output from a light receiving unit at a predetermined frame rate, in which a first luminance change in the positive direction is detected, and a second count number, which is the number of pixels in which a second luminance change in the negative direction is detected; a coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; and an integration unit that integrates the multiplication result of the count number of the pixels and the coefficient.

[0007] The signal processing method according to the first aspect of the present technology is such that a signal processing device counts a first count number, which is the number of pixels in an image indicating a luminance change output from a light receiving unit at a predetermined frame rate, in which a first luminance change in the positive direction is detected, and a second count number, which is the number of pixels in which a second luminance change in the negative direction is detected, generates a coefficient according to the time when the luminance change is detected, and integrates the multiplication result of the count number of the pixels and the coefficient.

[0008] In the first aspect of the present technology, in an image indicating a luminance change output from a light receiving unit at a predetermined frame rate, a first count number, which is the number of pixels in which a first luminance change in the positive direction is detected, and a second count number, which is the number of pixels in which a second luminance change in the negative direction is detected, are counted, a coefficient according to the time when the luminance change is detected is generated, and the multiplication result of the count number of the pixels and the coefficient is integrated.

[0009] The detection sensor according to the second aspect of the present technology includes a light receiving unit in which pixels that perform photoelectric conversion of incident light to generate an electrical signal are arranged in a grid pattern, a counting unit that counts a first count number, which is the number of pixels in an image indicating a luminance change output from the light receiving unit at a predetermined frame rate, in which a first luminance change in the positive direction is detected, and a second count number, which is the number of pixels in which a second luminance change in the negative direction is detected; a coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; and an integration unit that integrates the multiplication result of the count number of the pixels and the coefficient.

[0010] In a second aspect of the present technology, in a pixel of a light-receiving unit, photoelectric conversion of incident light is performed to generate an electrical signal. In an image showing a luminance change output from the light-receiving unit at a predetermined frame rate, a first count number which is the count number of pixels in which a first luminance change in the positive direction is detected, and a second count number which is the count number of pixels in which a second luminance change in the negative direction is detected are counted, a coefficient corresponding to the time when the luminance change is detected is generated, and a multiplication result of the count number of the pixels and the coefficient is integrated.

[0011] Note that the signal processing device according to the first aspect of the present technology can be realized by causing a computer to execute a program. The program to be executed by the computer can be transmitted via a transmission medium or recorded on a recording medium and provided.

[0012] The signal processing device and the detection sensor may be independent devices or internal blocks constituting one device.

Brief Description of Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted. The description will be made in the following order. 1. Configuration example of event detection sensor 2. Principle of event generation by light source flicker 3. Configuration example of flicker detection unit 4. Configuration example of sensitivity control unit 5. Processing flow of flicker control process 6. Example of processing result of flicker control process 7. Other configuration examples of event detection sensor 8. Configuration example of imaging device 9. Configuration example of electronic device 10. Application example to a moving body

[0015] <1. Configuration example of event detection sensor> FIG. 1 is a block diagram showing a configuration example of an embodiment of an event detection sensor which is a sensor to which the present technology is applied.

[0016] The event detection sensor 1 includes a pixel array unit 11 which is a light receiving unit, and a signal processing circuit 12 which processes a signal generated by the light receiving unit.

[0017] The pixel array unit 11 is configured by arranging pixels 21 that receive incident light and perform photoelectric conversion in a grid pattern. Further, in the pixel array unit 11, a detection circuit 22 that detects a luminance change (light amount change) generated in the pixel 21 as an event is configured in a different layer at the same planar position as the pixel 21, for example, corresponding to each pixel 21. Therefore, the detection circuits 22 are also arranged and configured in a grid pattern.

[0018] Each detection circuit 22 detects whether there is a luminance change in the plus direction exceeding a predetermined threshold (hereinafter referred to as a + change), a luminance change in the minus direction exceeding a predetermined threshold (hereinafter referred to as a - change), or no luminance change exceeding a predetermined threshold within a predetermined period corresponding to the frame rate in the corresponding pixel 21, and outputs it as a detection signal.

[0019] The pixel array unit 11 outputs the detection signals of the respective detection circuits 22 to the signal processing circuit 12 in a predetermined order in accordance with the control by a pixel driving unit (not shown).

[0020] Therefore, the pixel array unit 11 detects the presence or absence of a luminance change in units of pixels 21 (detection circuits 22) at a constant frame rate, and outputs, as event data, the image data of the change image in which the detection results are stored as the pixel values of each pixel 21 to the signal processing circuit 12. The pixel value of each pixel in the change image is a value indicating either a + change, a - change, or no change.

[0021] FIG. 2 shows an example of a change image when a certain shooting scene is detected by the event detection sensor 1.

[0022] As shown on the left side of Fig. 2, assume that the event detection sensor 1 has detected a scene in which a person is moving in the direction indicated by the arrow. The person, who is the subject, appears brighter than the surrounding background. In this case, for the pixels in the contour portion on the advancing direction side of the person in the output change image, since a luminance change from dark (low luminance) to bright (high luminance) is detected, a pixel value indicating a + change is stored. On the other hand, for the pixels in the contour portion on the side opposite to the advancing direction of the person, since a luminance change from bright (high luminance) to dark (low luminance) is detected, a pixel value indicating a - change is stored. The other pixels have pixel values indicating no change.

[0023] The change image takes three values for the pixel values, indicating either a + change, a - change, or no change, and does not require high gradations such as 8 bits or 10 bits like a normal image sensor. Therefore, the exposure time and the time for AD conversion can also be performed in an extremely short time. As a result, compared with the frame rate of a normal image sensor such as 30 fps or 60 fps, a change image can be output at an extremely high frame rate. For example, output at a high frame rate such as 1000 fps is possible.

[0024] Returning to Fig. 1, the signal processing circuit 12 includes an event data acquisition unit 31, an event count unit 32, a flicker detection unit 33, and a sensitivity control unit 34.

[0025] The event data acquisition unit 31 acquires the change image output from the pixel array unit 11 at a predetermined frame rate, outputs it to the outside of the sensor, and supplies it to the event count unit 32.

[0026] The event count unit 32 counts the number of pixels with a + change (hereinafter also referred to as the + count number) and the number of pixels with a - change (hereinafter also referred to as the - count number) for the change image sequentially supplied from the event data acquisition unit 31, and supplies the count result to the flicker detection unit 33 and the sensitivity control unit 34.

[0027] The flicker detection unit 33 detects (estimates) the amount of flicker in a predetermined period using the + count number and the - count number supplied from the event counting unit 32, and outputs the result to the outside of the event detection sensor 1 and the sensitivity control unit 34.

[0028] The sensitivity control unit 34 determines whether flicker occurs in a predetermined period based on the amount of flicker supplied from the flicker detection unit 33. When it is determined that flicker occurs in a predetermined period, the sensitivity control unit 34 adjusts (controls) the sensitivity parameters of each detection circuit 22 of the pixel array unit 11 using the + count number and the - count number supplied from the event counting unit 32. For example, when the amount of flicker is large, the sensitivity control unit 34 raises the threshold value of the luminance change to be regarded as an event, making it difficult for an event to occur. When the amount of flicker is small, the sensitivity control unit 34 lowers the threshold value of the luminance change to make it easier for an event to occur. A control value for controlling the threshold value of the luminance change is supplied from the sensitivity control unit 34 to each detection circuit 22 of the pixel array unit 11.

[0029] <2. Principle of event generation due to light source flicker> With reference to FIG. 3, the principle of event generation due to light source flicker will be described.

[0030] When event detection is performed in an environment using a light source with a power frequency of 50 Hz, the light source generates flicker at 100 Hz, which is twice the power frequency of 50 Hz.

[0031] As shown in FIG. 3, one cycle of the light source with a power frequency of 50 Hz is 20 msec, and flicker occurs at a cycle of 10 msec, which is half of that.

[0032] Furthermore, when the luminance change is divided into an event of + change (hereinafter also referred to as a positive event) and an event of - change (hereinafter referred to as a negative event), as shown in FIG. 3, the positive event and the negative event are alternately detected every 5 msec.

[0033] FIG. 4 shows the result of actually detecting an event under a light source with a power frequency of 50 Hz.

[0034] FIG. 4 is a graph showing the number of positive events and negative events obtained by generating a change image every 1 msec at a frame rate of 1000 fps for the event detection sensor 1 and integrating them for 10 msec, respectively, for each 1 msec.

[0035] Looking at the positive events and negative events respectively, under a light source with a power frequency of 50 Hz, events with a period of 10 msec are occurring.

[0036] Therefore, the flicker detection unit 33 of the event detection sensor 1 detects the presence or absence of flicker with a predetermined period by detecting whether periodicity of change occurs at a predetermined period based on the number of positive events and negative events.

[0037] In the following description, the case where the flicker detection unit 33 detects flicker with a period of 10 msec occurring under a light source with a power frequency of 50 Hz will be described as an example.

[0038] With reference to FIGS. 5 and 6, an outline of flicker detection by the flicker detection unit 33 and sensitivity parameter control by the sensitivity control unit 34 will be described.

[0039] As described with reference to FIG. 4, under a light source with a power frequency of 50 Hz, flicker-induced events with a period of 10 msec occur in both positive events and negative events.

[0040] Therefore, when trying to detect flicker occurring under a light source with a power frequency of 50 Hz, the flicker detection unit 33 detects the flicker using the count numbers of positive events and negative events with a detection period of 10 msec.

[0041] FIG. 5 is a schematic image diagram of flicker occurring under a light source with a power frequency of 50 Hz.

[0042] As shown in FIG. 5, at the timing when flicker occurs, a peak in the count number of positive or negative events occurs.

[0043] Describing the appearance image of the flicker in FIG. 5 more accurately in terms of frames, it becomes as shown in FIG. 6.

[0044] In this embodiment, when the frame rate of the event detection sensor 1 is 1000 fps, 10 changed images are generated during a detection period of 10 msec. Among the 10 frames in one cycle (10 msec), there are frames in which light source flicker at a power supply frequency of 50 Hz occurs and frames in which it does not occur.

[0045] The sensitivity control unit 34 controls the sensitivity parameter of the detection circuit 22 in terms of frames. Specifically, the sensitivity control unit 34 lowers the detection sensitivity only for the frames in which light source flicker occurs among the 10 frames, and keeps the detection sensitivity high (unchanged) for the frames in which light source flicker does not occur.

[0046] In other words, the sensitivity control unit 34 controls the sensitivity parameter for each phase occurring within the detection period. A frame corresponding to the phase within the detection period, which represents which frame among the 10 frames corresponding to the detection period, is called a frame phase.

[0047] Note that the detection period may be a period that is an integer multiple of the flicker period, rather than 10 msec corresponding to the flicker period.

[0048] As shown in FIG. 4, since positive events and negative events occur at different timings, the sensitivity control unit 34 controls the sensitivity separately for positive events and negative events. That is, the sensitivity control unit 34 changes the threshold for detecting a + change with respect to the frame phase in which a positive event occurs due to light source flicker, and controls to change the threshold for detecting a - change with respect to the frame phase in which a negative event occurs due to light source flicker.

[0049] <3. Configuration Example of Flicker Detection Unit> FIG. 7 is a block diagram showing a detailed configuration example of the flicker detection unit 33 in FIG. 1.

[0050] The flicker detection unit 33 includes a subtractor 51, a convolution coefficient generation unit 52, an integration unit 53, and a flicker amount estimation unit 54.

[0051] The integration unit 53 includes multipliers 71 and 72, integrators 73 and 74, and output units 75 and 76.

[0052] The flicker detection unit 33 is supplied with the + count number and the - count number from the event counting unit 32, and the supplied + count number and - count number are input to the subtractor 51.

[0053] The subtractor 51 subtracts the - count number from the + count number and outputs the subtraction result to the multipliers 71 and 72 of the integration unit 53.

[0054] The convolution coefficient generation unit 52 generates function values obtained by substituting values corresponding to the time of the flicker period into the sine function and cosine function of the flicker period to be detected, and supplies them as convolution coefficients to the multipliers 71 and 72 of the integration unit 53.

[0055] The convolution coefficient generation unit 52 is supplied with a vertical drive signal Vsync corresponding to the frame rate at which the pixel array unit 11 outputs a changing image from a timing control unit (not shown), and the convolution coefficient generation unit 52 generates a value corresponding to the time of the flicker period based on the vertical drive signal Vsync and substitutes it into the sine function and cosine function of the flicker period to be detected.

[0056] Note that instead of the sin function and cos function, an approximation function approximating the sin function and cos function may be used, and an approximate value obtained by substituting, as an argument, a value corresponding to the time of the flicker period into the approximation function may be supplied to multipliers 71 and 72 of the integration unit 53. In the present embodiment, as described with reference to FIGS. 8 to 10, the sin approximation function and cos approximation function obtained by approximating the sin function and cos function with a signal taking two values of 1 and -1 are used to calculate function values corresponding to the time of the flicker period. The calculated function value of the sin approximation function is supplied to multiplier 72, and the calculated function value of the cos approximation function is supplied to multiplier 71.

[0057] Further, the convolution coefficient generation unit 52 generates an enable signal indicating the timing at which the integration unit 53 outputs the integration result, and supplies it to output units 75 and 76 of the integration unit 53. The integration period of the integration unit 53 determined at the timing when the enable signal becomes High can be, for example, the same 10 msec as the detected flicker period (one period). Alternatively, the integration period may be a period that is an integer multiple of the flicker period.

[0058] The integration unit 53 integrates the multiplication result obtained by multiplying the subtraction result obtained by subtracting the - count number from the + count number supplied from the subtractor 51 and the convolution coefficient using the sin function and cos function or the sin approximation function and cos approximation function approximating the same supplied from the convolution coefficient generation unit 52.

[0059] The multiplier 71 supplies the multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 and the function value of the cos approximation function supplied from the convolution coefficient generation unit 52 to the integrator 73.

[0060] The multiplier 72 supplies the multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 and the function value of the sin approximation function supplied from the convolution coefficient generation unit 52 to the integrator 74.

[0061] The integrator 73 integrates the multiplication result supplied from the multiplier 71 and supplies it to the output unit 75. The integrator 74 integrates the multiplication result supplied from the multiplier 72 and supplies it to the output unit 76.

[0062] The output unit 75 is composed of, for example, flip - flops, and at the timing when the enable signal supplied from the convolution coefficient generation unit 52 is High, it acquires the integrated value cos_sum of the integrator 73 and supplies it to the flicker amount estimation unit 54.

[0063] The output unit 76 is composed of, for example, flip - flops, and at the timing when the enable signal supplied from the convolution coefficient generation unit 52 is High, it acquires the integrated value sin_sum of the integrator 74 and supplies it to the flicker amount estimation unit 54.

[0064] The flicker amount estimation unit 54 uses the integration results supplied from the output units 75 and 76 respectively to calculate the amplitude component of the flicker frequency and estimates the flicker amount (flicker - like degree) occurring at a specific frequency (flicker frequency).

[0065] Specifically, the flicker amount estimation unit 54 calculates the flicker amount EST_FL by the formula (1) or formula (2) using the frame integration count sum, the integrated values cos_sum and sin_sum.

Equation

[0066] Here, the frame integration count sum is equal to the number of frames integrated by the integration unit 53. In this embodiment, since the integration period is the same as the flicker period, which is 10 msec, sum = 10.

[0067] In formula (1) and formula (2), when there is no flicker at the target period, the flicker amount EST_FL becomes a small value, and when there is flicker at the target period, the flicker amount EST_FL becomes a large value.

[0068] FIG. 8 shows an example of a sin function and a cos function generated by the convolution coefficient generation unit 52, or a sin approximation function and a cos approximation function approximating the same.

[0069] As shown in the upper part of FIG. 8, the convolution coefficient generation unit 52 may generate function values obtained by substituting values corresponding to the times of the flicker period into a sin function and a cos function with one period of the flicker period to be detected. However, in the present embodiment, as shown in the lower part of FIG. 8, function values corresponding to the times of the flicker period are calculated using a sin approximation function and a cos approximation function obtained by approximating the sin function and the cos function with a signal taking two values of 1 and -1.

[0070] When the sin approximation function and the cos approximation function are represented by sin_approx(t) and cos_approx(t), respectively, the sin approximation function and the cos approximation function can be expressed by the following equations.

Equation

[0071] In Equations (3) and (4), f is the flicker period, and t is a value corresponding to the time of the flicker period. By this approximation, as shown in FIG. 8, the sin approximation function and the cos approximation function are approximated to a signal that outputs +1 when the sin function and the cos function are positive and -1 when the sin function and the cos function are negative. Since the flicker period is now set to 10 msec, the outputs of the sin approximation function and the cos approximation function switch between +1 and -1 in units of 5 msec.

[0072] Such a sin approximation function and a cos approximation function can adopt, for example, a configuration in which a table associating +1 or -1 with each time of one period is stored in the convolution coefficient generation unit 52, and function values of the sin approximation function and the cos approximation function are output based on the table.

[0073] In addition, the sin approximation function and the cos approximation function can also be realized by a logic circuit as shown in FIG. 9.

[0074] FIG. 9 shows a circuit configuration example of the convolution coefficient generation unit 52 when the sin approximation function and the cos approximation function shown in FIG. 8 are adopted.

[0075] The convolution coefficient generation unit 52 in FIG. 9 includes a counter 101 that performs counting corresponding to the flicker period.

[0076] Further, the convolution coefficient generation unit 52 has a comparator 102 and 103, a selector 104, and a flip-flop 105 configured to output a function value cos_approx of the cos approximation function according to the time of the flicker period.

[0077] Furthermore, the convolution coefficient generation unit 52 has a comparator 111 and 112, a selector 113, and a flip-flop 114 configured to output a function value sin_approx of the sin approximation function according to the time of the flicker period.

[0078] Furthermore, the convolution coefficient generation unit 52 has a comparator 121 configured to output an enable signal.

[0079] A vertical drive signal Vsync and a count number cycle corresponding to the flicker period are input to the counter 101. The counter 101 starts counting from 0 and counts up according to the vertical drive signal Vsync. Then, when the counter 101 counts up to the count number cycle, it resets the count value cnt and repeats the process of counting from 0 again. Now, since the flicker period is 10 msec and the vertical drive signal Vsync is a signal that goes High at 1 msec intervals corresponding to a frame rate of 1000 fps, "10" is input as the count number cycle.

[0080] The count value cnt of the counter 101 is supplied to the comparators 102, 103, 111, 112, and 121.

[0081] The comparator 102 is supplied with the set value cos_ptim. The comparator 102 compares the count value cnt supplied from the counter 101 with the set value cos_ptim, and outputs +1 to the selector 104 at the timing when they match. For count values cnt other than the set value cos_ptim, for example, 0 is output.

[0082] The comparator 103 is supplied with the set value cos_ntim. The comparator 103 compares the count value cnt supplied from the counter 101 with the set value cos_ntim, and outputs -1 to the selector 104 at the timing when they match. For count values cnt other than the set value cos_ntim, for example, 0 is output.

[0083] The selector 104 selects +1 and outputs it to the flip-flop 105 at the timing when +1 is supplied from the comparator 102, selects -1 and outputs it to the flip-flop 105 at the timing when -1 is supplied from the comparator 102, and outputs the value fed back from the flip-flop 105 to the flip-flop 105 at other timings.

[0084] The flip-flop 105 holds and outputs the value (+1 or -1) input from the selector 104 until it is updated next. The value output from the flip-flop 105 becomes the function value cos_approx of the cos approximation function.

[0085] The comparator 111 is supplied with the set value sin_ptim. The comparator 111 compares the count value cnt supplied from the counter 101 with the set value sin_ptim, and outputs +1 to the selector 113 at the timing when they match. For count values cnt other than the set value sin_ptim, for example, 0 is output.

[0086] The comparator 112 is supplied with a set value sin_ntim. The comparator 112 compares the count value cnt supplied from the counter 101 with the set value sin_ntim, and outputs -1 to the selector 113 at the timing when they match. For count values cnt other than the set value sin_ntim, for example, 0 is output.

[0087] The selector 113 selects +1 and outputs it to the flip-flop 114 at the timing when +1 is supplied from the comparator 111, selects -1 and outputs it to the flip-flop 114 at the timing when -1 is supplied from the comparator 112, and outputs the value fed back from the flip-flop 114 to the flip-flop 114 at other timings.

[0088] The flip-flop 114 holds and outputs the value (+1 or -1) input from the selector 113 until it is updated next. The value output from the flip-flop 114 becomes the function value sin_approx of the sin approximation function.

[0089] The comparator 121 is supplied with the count number cycle. The comparator 121 compares the count value cnt supplied from the counter 101 with the count number cycle, and sets the enable signal to High at the timing when they match. For count values cnt other than the count number cycle, a Low enable signal is output. Since the count value cnt supplied from the counter 101 repeats as 1, 2, 3, ··· 10, a High enable signal is output when the count value cnt is 10.

[0090] Figure 10 shows a timing chart when operating with the logic circuit shown in Figure 9.

[0091] The sin approximation function sin_approx(t) and the cos approximation function cos_approx(t) shown in Figure 8 are realized by the logic circuit of Figure 9.

[0092] The enable signal is High in units of 10 msec when detecting flicker occurring under a light source with a frame period, specifically a power frequency of 50 Hz.

[0093] <4. Configuration Example of Sensitivity Control Unit> Next, the control of the sensitivity parameter by the sensitivity control unit 34 in FIG. 1 will be described.

[0094] The sensitivity control unit 34 is supplied with the + count number and the - count number from the event count unit 32, and the detection result of the flicker amount is supplied from the flicker detection unit 33. When the detected flicker amount is large (greater than a predetermined flicker determination threshold FL_TH), the sensitivity control unit 34 changes the control value for controlling the threshold of luminance change as the sensitivity parameter to perform control to lower the detection sensitivity. In the present embodiment, the sensitivity control unit 34 directly changes the threshold of luminance change itself as the control value, and separately controls the + side threshold Vrefp for controlling the detection sensitivity of the + change and the - side threshold Vrefn for controlling the detection sensitivity of the - change.

[0095] Since the flicker amount for each detection period is supplied from the flicker detection unit 33 with a detection period of 10 msec, the sensitivity control unit 34 determines the + side threshold Vrefp and the - side threshold Vrefn, which are the control values for the next detection period, for each frame phase based on the event count number of each frame phase of the detection period.

[0096] With reference to FIGS. 11 and 12, the method for determining the + side threshold Vrefp for controlling the detection sensitivity of the + change will be described.

[0097] In FIGS. 11 and 12, the flicker amount EST_FL of the detection period DT2 is supplied from the flicker detection unit 33 at a predetermined timing, and the + count number P_count(i) of each frame phase i (i is an integer from 0 to 9) of the detection period DT2 is sequentially supplied from the event count unit 32. The sensitivity control unit 34 determines the control value sense(i), that is, the + side threshold value Vrefp(i), of each frame phase i of the next detection period DT3 based on the + count number P_count(i) of each frame phase i of the detection period DT2.

[0098] First, the sensitivity control unit 34 calculates the minimum value min(DT2) of the + count number P_count(i) of the frame phase i of the detection period DT2 by the following formula (5). MIN() in formula (5) represents a function for calculating the minimum value (i is an integer from 0 to 9). min(DT2) = MIN(P_count(i)) ·········(5)

[0099] Next, the sensitivity control unit 34 calculates the dynamic range DR(i) of each frame phase i of the detection period DT2 by subtracting the minimum value min(DT2) from the + count number P_count(i) of each frame phase i of the detection period DT2 as shown in formula (6). DR(i) = P_count(i) - min(DT2) ·········(6)

[0100] Then, when the calculated dynamic range DR(i) of each frame phase i is smaller than a preset first threshold value DR_TH1, the sensitivity control unit 34 changes the control value sense(i) of the frame phase i of the next detection period DT3 to increase the detection sensitivity. When increasing the detection sensitivity, the sensitivity control unit 34 changes the control value sense(i), which is the + side threshold value Vrefp(i), in a direction of decreasing it. Specifically, the sensitivity control unit 34 calculates the control value sense’(i) of the frame phase i of the next detection period DT3 by the following formula (7). sense’(i) = MAX(sense(i) - VALUE, LOWER_LIMIT) ···(7)

[0101] In Equation (7), MAX() is a function that selects the maximum value, VALUE represents the change range of the detection sensitivity, and LOWER_LIMIT represents the limit value when increasing the detection sensitivity. According to Equation (7), when the value {sense(i) - VALUE} obtained by subtracting the change range VALUE from the control value sense(i) of the detection period DT2 is greater than or equal to the limit value LOWER_LIMIT, the sensitivity control unit 34 determines that subtraction value as the control value sense’(i) of the frame phase i of the next detection period DT3. When it is less than the limit value LOWER_LIMIT, the sensitivity control unit 34 determines the limit value LOWER_LIMIT as the control value sense’(i) of the frame phase i of the next detection period DT3.

[0102] On the other hand, when the calculated dynamic range DR(i) is greater than a preset second threshold DR_TH2, the sensitivity control unit 34 changes the control value sense(i) of the frame phase i of the next detection period DT3 to lower the detection sensitivity. When lowering the detection sensitivity, the sensitivity control unit 34 changes the control value sense(i), which is the + side threshold Vrefp(i), in the direction of increasing it. Specifically, the sensitivity control unit 34 calculates the control value sense’(i) of the frame phase i of the next detection period DT3 according to the following Equation (8). sense’(i) = MIN(sense(i) + VALUE, UPPER_LIMIT) ···(8)

[0103] In Equation (8), MIN() is a function that selects the minimum value, VALUE represents the change range of the detection sensitivity, and UPPER_LIMIT represents the limit value when lowering the detection sensitivity. According to Equation (8), when the value {sense(i) + VALUE} obtained by adding the change range VALUE to the control value sense(i) of the detection period DT2 is less than or equal to the limit value UPPER_LIMIT, the sensitivity control unit 34 determines that addition value as the control value sense’(i) of the frame phase i of the next detection period DT3. When it is greater than the limit value UPPER_LIMIT, the sensitivity control unit 34 determines the limit value UPPER_LIMIT as the control value sense’(i) of the frame phase i of the next detection period DT3.

[0104] Note that in this example, the change width VALUE in the addition direction for increasing the detection sensitivity and the change width VALUE in the subtraction direction for decreasing the detection sensitivity are the same value, but they may also be different values.

[0105] When the calculated dynamic range DR(i) of each frame phase i is greater than or equal to the first threshold DR_TH1 and less than or equal to the second threshold DR_TH2, the control value sense(i) of that frame phase i is not changed, and the current control value sense(i) is maintained.

[0106] In the example of FIG. 11, since the dynamic ranges DR(5) to DR(9) of each frame phase from the 5th frame phase to the 9th frame phase of the detection period DT2 are greater than the second threshold DR_TH2, the control values sense’(5) to sense’(9) of each frame phase from the 5th frame phase to the 9th frame phase of the next detection period DT3 are changed in the direction of lowering the detection sensitivity. In other words, the control values sense’(5) to sense’(9) of the detection period DT3 are changed in the direction of increasing by the change width VALUE compared to the control values sense(5) to sense(9) of the detection period DT2.

[0107] On the other hand, the dynamic ranges DR(0) to DR(4) of each frame phase from the 0th frame phase to the 4th frame phase of the detection period DT2 are smaller than the first threshold DR_TH1, but since the limit value LOWER_LIMIT in the case of increasing the detection sensitivity is reached, the control values sense’(0) to sense’(4) of each frame phase from the 0th frame phase to the 4th frame phase of the next detection period DT3 are not changed.

[0108] FIG. 12 shows an example of setting the control value sense(i) according to the + count number P_count(i) of each frame phase i of four consecutive detection periods DT1 to DT4.

[0109] Since each of the dynamic ranges DR(5) to DR(9) from the 5th frame phase to the 9th frame phase of the detection period DT2 is larger than the second threshold value DR_TH2, the control values sense(5) to sense(9) from the 5th frame phase to the 9th frame phase of the next detection period DT3 are changed in the direction of lowering the detection sensitivity. In other words, the control values sense(5) to sense(9) of the detection period DT3 are changed in the direction higher by the change width VALUE than the control values sense(5) to sense(9) of the detection period DT2.

[0110] In the next detection period DT3, since only the dynamic range DR(7) of the 7th frame phase is larger than the second threshold value DR_TH2, in the next detection period DT4, only the control value sense(7) of the 7th frame phase is changed in the direction of lowering the detection sensitivity. In other words, the control value sense(7) of the detection period DT4 is changed in the higher direction than the control value sense(7) of the detection period DT3, and the control values sense(5) and sense(6) of the 5th and 6th frame phases and the control values sense(8) and sense(8) of the 8th and 9th frame phases remain the same as the control values of the detection period DT3.

[0111] Although the control value sense(i) of each frame phase i for controlling the detection sensitivity of the + change has been described, the sensitivity control unit 34 also performs the same control for the control value sense(i) of each frame phase i for controlling the detection sensitivity of the - change.

[0112] FIG. 13 is a diagram showing an image of the control value sense(i) which is the output of the sensitivity control unit 34 with respect to the count number from the event count unit 32 which is the input of the sensitivity control unit 34, for each of the control of the detection sensitivity of the + change and the control of the detection sensitivity of the - change.

[0113] Since the control of the detection sensitivity for + changes and the control of the detection sensitivity for - changes have opposite polarities, in the control of the detection sensitivity for - changes, when reducing the detection sensitivity, the sensitivity control unit 34 changes the control value sense(i), which is the - side threshold Vrefn, in a direction to lower it, and when increasing the detection sensitivity, the sensitivity control unit 34 changes the control value sense(i), which is the - side threshold Vrefn, in a direction to raise it. Note that since the magnitude relationship of the control value sense(i) also changes depending on the structure of the pixels of the light receiving unit and the polarity at the time of detection, it can be appropriately determined according to the conditions.

[0114] In the control of the detection sensitivity for - changes, the change width VALUE when increasing the detection sensitivity and when decreasing the detection sensitivity may be the same value or different values, similar to the control of the detection sensitivity for + changes. Also, the change width VALUE of the detection sensitivity may be the same value or different values between the control of the detection sensitivity for + changes and the control of the detection sensitivity for - changes.

[0115] Hereinafter, the control value sense(i) for the detection sensitivity of + changes is referred to as the control value p_sense(i), and the control value sense(i) for the detection sensitivity of - changes is referred to as the control value n_sense(i) for distinction.

[0116] FIG. 14 is a block diagram showing a detailed configuration example of the sensitivity control unit 34.

[0117] The sensitivity control unit 34 includes a positive control value generation unit 151 that generates a control value p_sense(i) for controlling the detection sensitivity of + changes, and a negative control value generation unit 152 that generates a control value n_sense(i) for controlling the detection sensitivity of - changes.

[0118] The positive control value generation unit 151 includes a minimum value detection unit 171, a DR calculation unit 172, a comparison unit 173, and an update determination unit 174.

[0119] The negative control value generation unit 152 includes a minimum value detection unit 181, a DR calculation unit 182, a comparison unit 183, and an update determination unit 184.

[0120] The positive control value generation unit 151 and the negative control value generation unit 152 basically have the same configuration as shown in FIG. 14.

[0121] The sensitivity control unit 34 is supplied with the + count number and the - count number from the event counting unit 32, and the detected flicker amount from the flicker detection unit 33. The + count number from the event counting unit 32 is input to the positive control value generation unit 151, and the - count number is input to the negative control value generation unit 152. Also, the flicker amount from the flicker detection unit 33 is input to the update determination units 174 and 184.

[0122] First, the positive control value generation unit 151 will be described.

[0123] The minimum value detection unit 171 accumulates the sequentially input + count numbers in units of the detection period DT, detects the minimum value min(DT(x)) of the + count numbers for each detection period DT, and supplies it to the DR calculation unit 172 (x = 1, 2, 3, ···). That is, the minimum value detection unit 171 performs the operation of the above-described formula (5) for each detection period DT.

[0124] The DR calculation unit 172 subtracts the minimum value min(DT(x)) of the detection period DT(x) supplied from the minimum value detection unit 171 from the + count number P_count(i) of each phase frame i of the current detection period DT(x) supplied from the event counting unit 32, thereby calculating the dynamic range DR(i) of each frame phase i of the current detection period DT(x). That is, the DR calculation unit 172 performs the operation of the above-described formula (6) for each detection period DT. The calculated dynamic range DR(i) of each frame phase i of the current detection period DT(x) is supplied to the comparison unit 173.

[0125] The comparison unit 173 compares the dynamic range DR(i) of each frame phase i of the current detection period DT(x) with the first threshold value DR_TH1, and determines whether it is smaller than the first threshold value DR_TH1. When the dynamic range DR(i) of each frame phase i is smaller than the first threshold value DR_TH1, the comparison unit 173 calculates the control value p_sense(i) at each frame phase i of the next detection period DT(x + 1).

[0126] Also, the comparison unit 173 compares the dynamic range DR(i) of each frame phase i of the current detection period DT(x) with the second threshold value DR_TH2, and determines whether it is larger than the second threshold value DR_TH2. When the dynamic range DR(i) of each frame phase i is larger than the second threshold value DR_TH2, the comparison unit 173 calculates the control value p_sense(i) at each frame phase i of the next detection period DT(x + 1).

[0127] That is, the comparison unit 173 calculates the control value p_sense’(i) at each frame phase i of the next detection period DT(x + 1) by performing the operations of the above-described equations (7) and (8), and supplies it to the update determination unit 174.

[0128] The update determination unit 174 determines whether flicker is occurring based on the flicker amount EST_FL supplied from the flicker detection unit 33. For example, when the flicker amount EST_FL supplied from the flicker amount estimation unit 54 is larger than a predetermined flicker determination threshold value FL_TH, the sensitivity control unit 34 determines that flicker is occurring, and when it is equal to or less than the flicker determination threshold value FL_TH, determines that flicker is not occurring. Alternatively, when a state where the flicker amount EST_FL calculated in units of the detection period DT is larger than the flicker determination threshold value FL_TH occurs a predetermined number of times or more within a predetermined period, it may be determined that flicker is occurring.

[0129] Then, when the update determination unit 174 determines that flicker is occurring, it performs control to update, for each frame phase, a control value that controls the threshold value of luminance change as a sensitivity parameter. That is, the update determination unit 174 supplies the control value p_sense’(i) at each frame phase i of the next detection period DT(x + 1), which is supplied from the comparison unit 173, to each detection circuit 22 of the pixel array unit 11 as the updated control value p_sense(i).

[0130] The operations of the minimum value detection unit 181, DR calculation unit 182, comparison unit 183, and update determination unit 184 of the negative control value generation unit 152 are the same as those of the minimum value detection unit 171, DR calculation unit 172, comparison unit 173, and update determination unit 174 of the positive control value generation unit 151, except that the - count number is used instead of the + count number. Therefore, the description is omitted. However, as shown in FIG. 13, since the polarities of the control of the detection sensitivity of the + change and the control of the detection sensitivity of the - change are opposite, the magnitude relationship of the control values is opposite.

[0131] In the negative control value generation unit 152, the control value n_sense(i) on the - change side, that is, the - side threshold value Vrefn(i) at each frame phase i of the next detection period DT(x + 1) is determined and supplied to each detection circuit 22 of the pixel array unit 11.

[0132] <5. Processing Flow of Flicker Control Processing> With reference to the flowchart of FIG. 15, the flicker control processing by the event detection sensor 1 will be described. This processing is started, for example, when an event detection (imaging) start is instructed to the event detection sensor 1.

[0133] First, in step S1, the pixel array unit 11 performs exposure with a predetermined exposure time corresponding to the frame rate, and generates a change image in which information indicating the presence or absence of luminance change is stored for each pixel. The generated change image is supplied to the signal processing circuit 12. The pixel value of each pixel of the change image represents either a + change, a - change, or no change.

[0134] In step S2, the event data acquisition unit 31 acquires the change image supplied from the pixel array unit 11, outputs it to the outside of the event detection sensor 1, and supplies it to the event count unit 32.

[0135] In step S3, the event count unit 32 counts the + count number and the - count number for the change image supplied from the event data acquisition unit 31, and supplies the count result to the flicker detection unit 33 and the sensitivity control unit 34.

[0136] In step S4, the flicker detection unit 33 subtracts the - count number from the + count number supplied from the event count unit 32, integrates the multiplication result obtained by multiplying the subtraction result by the function value of the cosine approximation function (cosine function), and integrates the multiplication result obtained by multiplying the subtraction result by the function value of the sine approximation function (sine function).

[0137] More specifically, the subtractor 51 of the flicker detection unit 33 subtracts the - count number from the + count number, and outputs the subtraction result to the multipliers 71 and 72 of the integration unit 53. The multiplier 71 supplies the multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 by the function value of the cosine approximation function generated by the convolution coefficient generation unit 52 to the integrator 73, and the integrator 73 integrates the multiplication result supplied from the multiplier 71. Also, the multiplier 72 supplies the multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 by the function value of the sine approximation function generated by the convolution coefficient generation unit 52 to the integrator 74, and the integrator 74 integrates the multiplication result supplied from the multiplier 72.

[0138] In step S5, the flicker detection unit 33 determines whether the enable signal supplied from the convolution coefficient generation unit 52 has become High. The enable signal becoming High indicates that the change image of the number of frames corresponding to the detection period DT has been input from the pixel array unit 11.

[0139] In step S5, if it is determined that the enable signal is not High, the process returns to step S3, and steps S3 to S5 described above are repeated.

[0140] On the other hand, in step S5, if it is determined that the enable signal is High, the process proceeds to step S6. The output unit 75 of the flicker detection unit 33 acquires the integrated value cos_sum of the integrator 73 and supplies it to the flicker amount estimation unit 54. The output unit 76 acquires the integrated value sin_sum of the integrator 74 and supplies it to the flicker amount estimation unit 54.

[0141] In step S7, the flicker amount estimation unit 54 of the flicker detection unit 33 estimates the flicker amount generated at the flicker frequency of the detection target using the integration results supplied from the output units 75 and 76 respectively. Specifically, the flicker amount estimation unit 54 calculates the flicker amount EST_FL according to the above-described formula (1) or formula (2). The calculated flicker amount EST_FL is output to the outside of the event detection sensor 1 and is also supplied to the sensitivity control unit 34.

[0142] In step S8, the sensitivity control unit 34 determines whether flicker is occurring based on the flicker amount EST_FL supplied from the flicker amount estimation unit 54. For example, when the flicker amount EST_FL supplied from the flicker amount estimation unit 54 is greater than a predetermined flicker determination threshold value FL_TH, the sensitivity control unit 34 determines that flicker is occurring. When it is below the flicker determination threshold value FL_TH, the sensitivity control unit 34 determines that no flicker is occurring. Alternatively, when the state where the flicker amount EST_FL calculated in units of the detection period DT is greater than the flicker determination threshold value FL_TH occurs a predetermined number of times or more within a predetermined period, it may be determined that flicker is occurring.

[0143] In step S8, if it is determined that no flicker is occurring, steps S9 to S11 described later are skipped, and the process returns to step S1.

[0144] On the other hand, if it is determined in step S8 that flicker is occurring, the processes of the next steps S9 to S11 are executed.

[0145] In step S9, the sensitivity control unit 34 detects the minimum value min(DT(x)) of the count numbers of a plurality of change images accumulated in units of the detection period DT for each of the + count number and the - count number. Specifically, the minimum value detection unit 171 detects the minimum value min(DT(x)) of the + count numbers of a plurality of change images accumulated in units of the detection period DT, and the minimum value detection unit 181 detects the minimum value min(DT(x)) of the - count numbers of a plurality of change images accumulated in units of the detection period DT.

[0146] Next, in step S10, the sensitivity control unit 34 calculates the control value sense(i) at each frame phase i of the next detection period DT(x + 1).

[0147] For example, in the positive control value generation unit 151 that processes the + count number, the DR calculation unit 172 subtracts the minimum value min(DT(x)) of the + count numbers from the + count number P_count(i) of each phase frame i of the current detection period DT(x) to calculate the dynamic range DR(i) of the + count numbers of each frame phase i of the detection period DT(x). The comparison unit 173 compares the dynamic range DR(i) of the + count numbers of each frame phase i of the current detection period DT(x) with the first threshold DR_TH1 and the second threshold DR_TH2 to calculate the control value p_sense(i) at each frame phase i of the next detection period DT(x + 1).

[0148] Similarly, for the negative control value generation unit 152 that processes the - count number, the control value n_sense(i) at each frame phase i of the next detection period DT(x + 1) is calculated by the DR calculation unit 182 and the comparison unit 183.

[0149] Next, in step S11, the update determination units 174 and 184 of the sensitivity control unit 34 respectively supply the control value sense(i) at each frame phase i of the next detection period DT(x + 1) supplied from the comparison units 173 and 183 to each detection circuit 22 of the pixel array unit 11 as the updated control value sense(i) at the timing corresponding to each frame phase i of the next detection period DT(x + 1). More specifically, the update determination unit 174 supplies the control value p_sense(i) on the + change side, that is, the + side threshold value Vrefp, to each detection circuit 22, and the update determination unit 184 supplies the control value n_sense(i) on the - change side, that is, the - side threshold value Vrefn, to each detection circuit 22.

[0150] After step S11, the process returns to step S1, and the above-described process is repeated.

[0151] <6. Example of Processing Result of Flicker Control Processing> FIG. 16 shows an example of the processing result of the flicker control processing by the event detection sensor 1.

[0152] In the shooting scene shown in the image 201 of FIG. 16, event detection by the event detection sensor 1 was executed in an environment using a light source with a power frequency of 50 Hz. The shooting scene shows a person walking and moving from right to left within the screen in front of the background. The image 201 is an image captured by a general CMOS image sensor.

[0153] The image 202 is frame data (frame image) of a change image by the event detection sensor 1. The image 202 is in a state where the sensitivity of the event detection sensor 1 is high, events are detected throughout the screen, and the events of the moving person are buried in noise.

[0154] The image 203 is frame data (frame image) of a change image after the sensor sensitivity is reduced compared to the image 202 by the flicker control processing.

[0155] In the image 203, the noise capturing the flicker is reduced, and only the movement larger than the flicker (mainly the moving person) is detected as an event.

[0156] Therefore, according to the flicker control process, the flicker amount (flicker information) can be detected from the changing images output at a fixed period. And when the flicker amount is large, the flicker can be suppressed and only the dominant events can be extracted. Also, the + change and the - change can be controlled independently, each specialized for the timing when they are likely to occur.

[0157] <7. Other Configuration Examples of the Event Detection Sensor> FIG. 17 is a block diagram showing a configuration example as another embodiment of the event detection sensor 1.

[0158] In FIG. 17, the parts corresponding to those in FIG. 1 are denoted by the same reference numerals, and the description of those parts will be omitted as appropriate.

[0159] In FIG. 17, flicker detection units 33-1 to 33-R (R>1) are provided, and the difference from the event detection sensor 1 in FIG. 1 is that a plurality of (R) flicker detection units 33 are provided, and other points are common to the event detection sensor 1 in FIG. 1.

[0160] The event detection sensor 1 in FIG. 17 can detect a plurality of flicker frequencies by including a plurality of flicker detection units 33-1 to 33-R. That is, each of the flicker detection units 33-1 to 33-R has a different flicker frequency set as the detection target.

[0161] For example, when R = 2 and the flicker frequencies detected by the flicker detection units 33-1 and 33-2 are set to 100 Hz and 120 Hz, respectively, it becomes possible to detect the flicker corresponding to the western region and the eastern region of Japan.

[0162] Alternatively, if the frequencies of flicker detected by the flicker detectors 33-1 to 33-R are set to 25 Hz, 50 Hz, 100 Hz, 200 Hz, 400 Hz, ···, it is possible to detect only events of arbitrary frequencies, and frequency analysis including flicker becomes possible.

[0163] <8. Configuration Example of Imaging Device> FIG. 18 is a block diagram showing a configuration example of an imaging device including the above-described event detection sensor 1 as an imaging element.

[0164] The imaging device 300 includes an optical unit 311, an imaging element 312, a recording unit 313, and a control unit 314. Examples of the imaging device 300 include a camera mounted on an industrial robot and an in-vehicle camera.

[0165] The optical unit 311 condenses light from a subject and makes it incident on the imaging element 312. The imaging element 312 photoelectrically converts the incident light incident via the optical unit 311 to generate image data, and supplies it to the recording unit 313. As this imaging element 312, the event detection sensor 1 shown in FIG. 1, FIG. 17, etc. is mounted.

[0166] The recording unit 313 records and stores the image data supplied from the imaging element 312 in a predetermined recording medium. The control unit 314 controls the imaging element 312. For example, the control unit 314 instructs the imaging element 312 to start and end imaging, and designates the frame rate during imaging.

[0167] FIG. 19 is a perspective view showing a schematic configuration example of the imaging element 312.

[0168] The imaging element 312 is configured with a stacked structure in which a light-receiving chip 321 and a detection chip 322 are bonded and stacked. The light-receiving chip 321 and the detection chip 322 are electrically connected via connection parts such as vias, Cu-Cu junctions, and bumps.

[0169] FIG. 20 is a plan view showing a configuration example of the light-receiving chip 321.

[0170] The light-receiving chip 321 includes a light-receiving portion 341 formed at the central portion of the chip and one or more via arrangement portions 342 formed at the outer peripheral portion outside the light-receiving portion 341. In the example of FIG. 20, three via arrangement portions 342 are provided at the corner portions of the chip outer periphery.

[0171] In the light-receiving portion 341, a plurality of shared blocks 343 are arranged in a two-dimensional lattice. In the via arrangement portion 342, vias electrically connected to the detection chip 322 are arranged.

[0172] In each of the shared blocks 343, a plurality of logarithmic response portions 351 are arranged. For example, in one shared block 343, four logarithmic response portions 351 are arranged in 2 rows × 2 columns. These four logarithmic response portions 351 share a circuit on the detection chip 322. Details of the shared circuit will be described later. Note that the number of logarithmic response portions 351 in the shared block 343 is not limited to four.

[0173] The logarithmic response portion 351 generates a voltage signal corresponding to the logarithmic value of the photocurrent. Each of the logarithmic response portions 351 is assigned a pixel address composed of a row address and a column address.

[0174] FIG. 21 is a plan view showing a configuration example of the detection chip 322.

[0175] The detection chip 322 includes one or more via arrangement portions 361, an address event detection portion 362, a row drive circuit 363, a column drive circuit 364, and a signal processing circuit 365.

[0176] The via arrangement portion 361 is provided at a position corresponding to the via arrangement portion 342 of the light-receiving chip 321 and is electrically connected to the light-receiving chip 321 via vias. In FIG. 21, via arrangement portions 361 are provided at positions corresponding to the three via arrangement portions 342 of FIG. 20, and a total of three via arrangement portions 361 are formed on the detection chip 322.

[0177] The address event detection unit 362 detects the presence or absence of an event for each logarithmic response unit 351 of the light receiving chip 321, and generates a detection signal indicating the detection result. The detection signal is generated as three-value (2-bit) information representing either a + change, a - change, or no change.

[0178] The row drive circuit 363 selects a predetermined row address of the address event detection unit 362, and causes the detection signal of the selected row address to be output to the signal processing circuit 365.

[0179] The column drive circuit 364 selects a predetermined column address of the address event detection unit 362, and causes the detection signal of the selected column address to be output to the signal processing circuit 365.

[0180] The signal processing circuit 365 performs predetermined signal processing on the detection signal output from the address event detection unit 362. For example, the signal processing circuit 365 acquires image data using the detection signal as a pixel signal. Then, based on the image data, the signal processing circuit 365 executes a process of detecting (estimating) the amount of flicker in a predetermined period, and controls the address event detection unit 362 so as to suppress flicker when flicker occurs in a predetermined period. Therefore, in the imaging device 312, the process executed by the signal processing circuit 12 in FIG. 1 is executed by the signal processing circuit 365.

[0181] FIG. 22 is a plan view showing the details of the address event detection unit 362.

[0182] In the address event detection unit 362, a plurality of detection blocks 371 are arranged in a two-dimensional grid pattern. The detection blocks 371 are arranged for each shared block 343 on the light receiving chip 321. That is, when the number of shared blocks 343 on the light receiving chip 321 is N (N is an integer), N detection blocks 371 are arranged in the detection chip 322. Each detection block 371 is electrically connected to the corresponding shared block 343 by vias, Cu-Cu junctions, or the like.

[0183] FIG. 23 is a block diagram showing a configuration example of one detection block 371.

[0184] The detection block 371 includes four detection units 381, a selector 382, a comparison unit 383, and a transfer circuit 384.

[0185] Each of the four detection units 381 is composed of a logarithmic response unit 351, a buffer 352, and a differentiator 353. The logarithmic response unit 351 generates a voltage signal corresponding to the logarithmic value of the photocurrent and outputs it to the buffer 352. The buffer 352 buffers the voltage signal from the logarithmic response unit 351 and outputs it to the differentiator 353. This buffer 352 can ensure the isolation of noise associated with the subsequent switching operation and improve the driving force for driving the subsequent stage. Note that this buffer 352 can also be omitted. The differentiator 353 outputs the change amount of the voltage signal (change amount of luminance change) as a differential signal Sin.

[0186] As shown in FIG. 20, the logarithmic response unit 351 is also provided in the shared block 343 of the light receiving chip 321, and is distributed between the shared block 343 of the light receiving chip 321 and the detection unit 381 of the detection block 371. Therefore, the four detection units 381 correspond to the 2-row × 2-column logarithmic response units 351 in the shared block 343. When distinguishing each of the four detection units 381, they are referred to as detection units 381-1 to 381-4, and the differential signals Sin output by each of the detection units 381-1 to 381-4 are distinguished as differential signal Sin1, differential signal Sin2, differential signal Sin3, and differential signal Sin4.

[0187] Selector 382 selects the output of any one of the four detection units 381 according to the selection signals SEL1 to SEL4 from the row driving circuit 363, and supplies the acquired differential signal Sin to the comparison unit 383 as the differential signal Sout. Specifically, when the selection signal SEL1 is supplied from the row driving circuit 363, the selector 382 selects the differential signal Sin1 from the detection unit 381-1; when the selection signal SEL2 is supplied, it selects the differential signal Sin2 from the detection unit 381-2; when the selection signal SEL3 is supplied, it selects the differential signal Sin3 from the detection unit 381-3; and when the selection signal SEL4 is supplied, it selects the differential signal Sin4 from the detection unit 381-4 and supplies it to the comparison unit 383 as the differential signal Sout.

[0188] The comparison unit 383 compares the differential signal Sout supplied from the selector 382 with a predetermined threshold value, and supplies the comparison result to the transfer circuit 384. As the predetermined threshold values to be compared with the differential signal Sout, the above-described + side threshold value Vrefp and - side threshold value Vrefn are supplied from the sensitivity control unit 34 of the signal processing circuit 365 (signal processing circuit 12).

[0189] The comparison unit 383 outputs a detection signal DET+ indicating whether the differential signal Sout indicating the amount of change in luminance change exceeds the + side threshold value Vrefp to the transfer circuit 384, and outputs a detection signal DET- indicating whether the differential signal Sout exceeds the - side threshold value Vrefn to the transfer circuit 384.

[0190] The transfer circuit 384 transfers (outputs) the detection signal to the signal processing circuit 365 according to the column drive signal from the column drive circuit 364. Here, the transfer circuit 384 generates the detection signal as three - valued (2 - bit) information representing either a + change, a - change, or no change, and outputs it to the signal processing circuit 365. Specifically, when the detection signal DET+ indicating that the change amount of the luminance change exceeds the + side threshold value Vrefp is supplied from the comparison unit 383, the transfer circuit 384 outputs a detection signal representing a + change; when the detection signal DET - indicating that it exceeds the - side threshold value Vrefn is supplied, it outputs a detection signal representing a - change; and when it does not exceed either the + side threshold value Vrefp or the - side threshold value Vrefn, it outputs a detection signal representing no change.

[0191] FIG. 24 is a circuit showing the detailed configuration of the detection unit 381, and in particular, shows a detailed configuration example of the logarithmic response unit 351 and the differentiator 353.

[0192] The logarithmic response unit 351 is composed of a PD (PhotoDiode) 411 as a photoelectric conversion element and FETs 412 to 414. As the FETs 412 and 414, for example, N - type MOS (NMOS) FETs can be adopted, and as the FET 413, for example, P - type MOS (PMOS) FETs can be adopted.

[0193] The PD 411 receives the incident light, performs photoelectric conversion, and generates and conducts a photocurrent as an electric signal. The logarithmic response unit 351 converts the photocurrent from the PD 411 into a voltage (hereinafter also referred to as a photovoltage) Vo corresponding to the logarithm of the photocurrent, and outputs it to the differentiator 353 via the buffer 352.

[0194] The source of the FET 412 is connected to the gate of the FET 414, and the photocurrent by the PD 411 flows through the connection point between the source of the FET 412 and the gate of the FET 414. The drain of the FET 412 is connected to the power supply VDD, and its gate is connected to the drain of the FET 414.

[0195] The source of FET413 is connected to the power supply VDD, and its drain is connected to the connection point between the gate of FET412 and the drain of FET414. A predetermined bias voltage Vbias is applied to the gate of FET413. The source of FET414 is grounded.

[0196] The drain of FET412 is connected to the power supply VDD side and forms a source follower. PD411 is connected to the source of FET412 which forms a source follower. Thus, a photocurrent due to the charge generated by the photoelectric conversion of PD411 flows through FET412 (from the drain to the source). FET412 operates in the subthreshold region, and a photovoltage Vo corresponding to the logarithm of the photocurrent flowing through FET412 appears at the gate of FET412. As described above, in the logarithmic response section 351, the photocurrent from PD411 is converted by FET412 into a photovoltage Vo corresponding to the logarithm of the photocurrent.

[0197] The photovoltage Vo is output from the connection point between the gate of FET412 and the drain of FET414 to the differentiator 353 via the buffer 352.

[0198] The differentiator 353 calculates the difference between the current photovoltage and the photovoltage at a timing slightly different from the current one for the photovoltage Vo from the logarithmic response section 351, and outputs a difference signal Vout corresponding to the difference.

[0199] The differentiator 353 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434.

[0200] One end of the capacitor 431 is connected to the output of the buffer 352, and the other end is connected to the input terminal of the operational amplifier 432. Therefore, the photovoltage Vo is input to the (inverting) input terminal of the operational amplifier 432 via the capacitor 431.

[0201] The output terminal of the operational amplifier 432 is connected to the selector 382 in FIG. 23.

[0202] One end of the capacitor 433 is connected to the input terminal of the operational amplifier 432, and the other end is connected to the output terminal of the operational amplifier 432.

[0203] The switch 434 is connected to the capacitor 433 so as to turn on / off the connection between both ends of the capacitor 433. The switch 434 turns on / off the connection between both ends of the capacitor 433 by turning on / off according to the row drive signal of the row drive circuit 363.

[0204] The capacitor 433 and the switch 434 constitute a switched capacitor. When the off switch 434 is temporarily turned on and then turned off again, the capacitor 433 is reset to a state where the charge is discharged and new charge can be accumulated.

[0205] When the photovoltage Vo on the logarithmic response unit 351 side of the capacitor 431 when the switch 434 is turned on is represented as Vinit, and the capacitance (capacitance) of the capacitor 431 is represented as C1. The input terminal of the operational amplifier 432 is at virtual ground, and the charge Qinit accumulated in the capacitor 431 when the switch 434 is on is represented by Equation (9). Qinit = C1 ×Vinit ···(9)

[0206] Also, when the switch 434 is on, both ends of the capacitor 433 are short-circuited, so the charge accumulated in the capacitor 433 becomes zero.

[0207] Then, when the photovoltage Vo on the logarithmic response unit 351 side of the capacitor 431 when the switch 434 is turned off is represented as Vafter, the charge Qafter accumulated in the capacitor 431 when the switch 434 is turned off is represented by Equation (10). Qafter = C1×Vafter ···(10)

[0208] Assuming that the capacitance of the capacitor 433 is represented as C2, the charge Q2 stored in the capacitor 433 is expressed by Equation (11) using the differential signal Vout which is the output voltage of the operational amplifier 432.

[0209] Q2 = -C2×Vout ···(11)

[0210] Before and after the switch 434 is turned off, since the total charge amount obtained by combining the charge of the capacitor 431 and the charge of the capacitor 433 does not change, Equation (12) holds. Qinit = Qafter + Q2 ···(12)

[0211] Substituting Equations (9) to (11) into Equation (12), Equation (13) is obtained. Vout = -(C1 / C2)×(Vafter - Vinit) ···(13)

[0212] According to Equation (13), in the differentiator 353, subtraction of the photovoltages Vafter and Vinit, that is, calculation of the differential signal Vout corresponding to the difference (Vafter - Vinit) between the photovoltages Vafter and Vinit is performed. According to Equation (13), the gain of the subtraction in the differentiator 353 is C1 / C2. Therefore, the differentiator 353 outputs, as the differential signal Vout, a voltage obtained by multiplying the change in the photovoltage Vo after reset of the capacitor 433 by C1 / C2. This differential signal Vout is output as the differential signal Sin.

[0213] The differentiator 353 outputs the differential signal Sin by turning the switch 434 on and off by means of the line drive signal output by the line drive circuit 363.

[0214] FIG. 25 shows a configuration example of the comparison unit 383 of the detection block 371 in FIG. 23.

[0215] The comparison unit 383 includes comparators 451 and 452. The + side threshold value Vrefp and the - side threshold value Vrefn are supplied from the sensitivity control unit 34 of the signal processing circuit 365 (signal processing circuit 12) to the comparison unit 383.

[0216] Comparator 451 compares the differential signal Sout from selector 382 with the + side threshold value Vrefp, and supplies the comparison result to transfer circuit 384 as detection signal DET+. This detection signal DET+ indicates whether or not the amount of change in luminance exceeds the + side threshold value Vrefp.

[0217] Comparator 452 compares the differential signal Sout from selector 382 with the - side threshold value Vrefn, and supplies the comparison result to transfer circuit 384 as detection signal DET-. This detection signal DET- indicates whether or not the amount of change in luminance exceeds the - side threshold value Vrefn.

[0218] FIG. 26 is a timing chart showing a control example of the row drive circuit 363.

[0219] At timing T0, the row drive circuit 363 selects the first row by the row drive signal L1 and drives the differentiator 353 of that row. The capacitor 433 in the differentiator 353 of the first row is initialized by this row drive signal L1. Further, the row drive circuit 363 supplies the selection signal SEL1 to the selector 382 to select the upper left detection section 381 of the 2 rows × 2 columns in the shared block 343 for a certain period. Thereby, the presence or absence of an event is detected in the detection section 381 of the odd-numbered columns of the first row.

[0220] Next, at timing T1, the row drive circuit 363 drives the differentiator 353 of the first row again by the row drive signal L1. Further, the row drive circuit 363 selects the upper right detection section 381 of the 2 rows × 2 columns in the shared block 343 for a certain period by the selection signal SEL2. Thereby, the presence or absence of an event is detected in the detection section 381 of the even-numbered columns of the first row.

[0221] At timing T2, the row driving circuit 363 drives the differentiator 353 in the second row with the row driving signal L2. The capacitor 433 in the differentiator 353 in the second row is initialized by this row driving signal L2. Also, the row driving circuit 363 selects the lower left detection unit 381 out of the 2 rows × 2 columns in the shared block 343 for a certain period with the selection signal SEL3. Thereby, the presence or absence of an event is detected in the detection unit 381 of the odd-numbered columns in the second row.

[0222] Subsequently, at timing T3, the row driving circuit 363 drives the differentiator 353 in the second row again with the row driving signal L2. Also, the row driving circuit 363 selects the lower right detection unit 381 out of the 2 rows × 2 columns in the shared block 343 for a certain period with the selection signal SEL4. Thereby, the presence or absence of an event is detected in the detection unit 381 of the even-numbered columns in the second row.

[0223] Similarly hereinafter, the row driving circuit 363 sequentially selects the rows in which the logarithmic response units 310 are arranged, and drives the selected rows with a row driving signal. Also, every time the row driving circuit 363 selects a row, it sequentially selects each of the detection units 381 in the shared block 343 of the selected row with the selection signal SEL. For example, when the detection units 381 of 2 rows × 2 columns are arranged in the shared block 343, every time a row is selected, the odd-numbered columns and the even-numbered columns in that row are sequentially selected.

[0224] By performing the above driving control in order for the entire address event detection unit 362 (light receiving unit 341) in which the detection units 381 are arranged, a change image indicating the presence or absence of a luminance change is generated at a predetermined frame rate and output to the signal processing circuit 365.

[0225] The signal processing circuit 365 acquires the change image output at a predetermined frame rate, determines whether flicker at a predetermined period is occurring, and if flicker is occurring, controls (adjusts) the thresholds for detecting luminance change, that is, the + side threshold Vrefp and the - side threshold Vrefn.

[0226] <9. Configuration Example of Electronic Device> The above-described event detection sensor 1 can be mounted on electronic devices such as smartphones, tablet terminals, mobile phones, personal computers, game machines, television receivers, wearable terminals, digital still cameras, and digital video cameras.

[0227] FIG. 27 is a block diagram showing a configuration example of a smartphone as an electronic device equipped with an event detection sensor.

[0228] As shown in FIG. 27, the smartphone 601 is configured by connecting an event detection sensor 602, an imaging device 603, a display 604, a speaker 605, a microphone 606, a communication module 607, a sensor unit 608, a touch panel 609, and a control unit 610 via a bus 611. Further, in the control unit 610, the CPU executes a program to function as an application processing unit 621 and an operating system processing unit 622.

[0229] The event detection sensor 1 shown in FIG. 1 is applied to the event detection sensor 602. For example, the event detection sensor 602 is disposed on the front surface of the smartphone 601 and can detect and output a luminance change of a subject such as the face, hand, or finger of the user of the smartphone 601 as an event. Note that the event detection sensor 602 may be disposed on the back surface of the smartphone 601.

[0230] The imaging device 603 is disposed on the front surface of the smartphone 601 and acquires an image of the user by performing imaging with the user of the smartphone 601 as a subject. Although not shown, the imaging device 603 may be disposed on the back surface of the smartphone 601.

[0231] The display 604 displays an operation screen for performing processing by the application processing unit 621 and the operating system processing unit 622, an image captured by the imaging device 603, and the like. The speaker 605 and the microphone 606 output the voice of the other party and collect the voice of the user when making a call using the smartphone 601, for example.

[0232] The communication module 607 performs network communication via communication networks such as the Internet, the public telephone line network, wide-area communication networks for wireless mobile bodies such as so-called 4G lines and 5G lines, WAN (Wide Area Network), and LAN (Local Area Network), and short-range wireless communication such as Bluetooth (registered trademark) and NFC (Near Field Communication). The sensor unit 608 senses speed, acceleration, proximity, etc., and the touch panel 609 acquires touch operations by the user on the operation screen displayed on the display 604.

[0233] The application processing unit 621 performs processing for providing various services by the smartphone 601. For example, the application processing unit 621 can perform processing to cause the imaging device 603 to perform imaging based on the luminance change supplied from the event detection sensor 602 and display the image obtained as a result of the imaging on the display 604. Also, for example, the application processing unit 621 can perform processing to specify the area of interest when the imaging device 603 performs imaging based on the luminance change supplied from the event detection sensor 602.

[0234] The operation system processing unit 622 performs processing for realizing the basic functions and operations of the smartphone 601. For example, the operation system processing unit 622 can perform processing for authenticating the user's face based on the imaging result of the imaging device 603 and unlocking the smartphone 601. Further, the operation system processing unit 622 can perform processing for recognizing, for example, the user's gesture based on the imaging result of the imaging device 603, and perform processing for inputting various operations according to the gesture.

[0235] In the smartphone 601 configured as described above, by applying the event detection sensor 1 in FIG. 1 as the event detection sensor 602, for example, it is possible to detect the movement or state change of a predetermined object, or perform processing such as creating and displaying data of a place where there is a change in luminance.

[0236] <10. Application Example to Mobile Objects> The technology according to the present disclosure (this technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile object such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, a robot, or the like.

[0237] FIG. 28 is a block diagram showing a schematic configuration example of a vehicle control system which is an example of a mobile control system to which the technology according to the present disclosure can be applied.

[0238] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in FIG. 28, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside vehicle information detection unit 12030, an inside vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0239] The drive system control unit 12010 controls the operations of devices related to the vehicle's drive system according to various programs. For example, the drive system control unit 12010 functions as a control device for a driving force generation device for generating the driving force of the vehicle, such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating the braking force of the vehicle.

[0240] The body system control unit 12020 controls the operations of various devices installed in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as a headlamp, a backlamp, a brake lamp, a turn signal, or a fog lamp. In this case, radio waves transmitted from a portable device that substitutes for a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these inputs of radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0241] The vehicle exterior information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing, such as for a person, a vehicle, an obstacle, a sign, or characters on the road surface, based on the received image.

[0242] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of received light. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. Also, the light received by the imaging unit 12031 may be visible light or non-visible light such as infrared light.

[0243] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 may calculate the degree of driver fatigue or concentration based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0244] Based on the information inside and outside the vehicle acquired by the out-vehicle information detection unit 12030 or the in-vehicle information detection unit 12040, the microcomputer 12051 can calculate control target values for the driving force generation device, the steering mechanism, or the braking device, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control for realizing functions of an ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation of the vehicle, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, collision warning of the vehicle, or lane departure warning of the vehicle.

[0245] In addition, based on the information around the vehicle acquired by the out-vehicle information detection unit 12030 or the in-vehicle information detection unit 12040, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, etc., which controls the driving force generation device, the steering mechanism, or the braking device, etc., to drive autonomously without relying on the driver's operation.

[0246] In addition, based on the out-vehicle information acquired by the out-vehicle information detection unit 12030, the microcomputer 12051 can output a control command to the body system control unit 12020. For example, the microcomputer 12051 can perform cooperative control for the purpose of anti-glare, such as controlling the headlamp according to the position of the preceding vehicle or oncoming vehicle detected by the out-vehicle information detection unit 12030 and switching the high beam to the low beam.

[0247] The audio-visual output unit 12052 transmits at least one output signal of audio and image to an output device capable of notifying information visually or aurally to the vehicle occupants or outside the vehicle. In the example of FIG. 28, as the output devices, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are illustrated. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0248] FIG. 29 is a diagram showing an example of the installation position of the imaging unit 12031.

[0249] In FIG. 29, the vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.

[0250] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose of the vehicle 12100, side mirrors, rear bumper, back door, and the upper part of the front glass inside the vehicle cabin. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the front glass inside the vehicle cabin mainly acquire images in front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images on the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or the back door mainly acquires images behind the vehicle 12100. The front images acquired by the imaging units 12101 and 12105 are mainly used for detecting a preceding vehicle or a pedestrian, an obstacle, a traffic signal, a traffic sign, or a lane.

[0251] Note that FIG. 29 shows an example of the shooting ranges of imaging units 12101 to 12104. The shooting range 12111 indicates the shooting range of the imaging unit 12101 provided on the front nose. The shooting ranges 12112 and 12113 indicate the shooting ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively. The shooting range 12114 indicates the shooting range of the imaging unit 12104 provided on the rear bumper or the back door. For example, by overlapping the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0252] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0253] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 obtains the distance to each solid object within the shooting ranges 12111 to 12114 and the temporal change of this distance (relative speed with respect to the vehicle 12100), and thus can extract, as the preceding vehicle, the closest solid object on the traveling path of the vehicle 12100 that travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, 0 km / h or more). Further, the microcomputer 12051 can set the inter-vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control), etc. In this way, cooperative control for the purpose of automatic driving, etc., which runs autonomously without relying on the driver's operation, can be performed.

[0254] For example, based on the distance information obtained from imaging units 12101 to 12104, the microcomputer 12051 classifies and extracts solid object data regarding solid objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other solid objects, and can use it for automatic avoidance of obstacles. For example, the microcomputer 12051 discriminates obstacles around the vehicle 12100 into obstacles visible to the driver of the vehicle 12100 and obstacles difficult to visually recognize. Then, the microcomputer 12051 determines a collision risk indicating the degree of risk of collision with each obstacle, and when the collision risk is equal to or higher than a set value and there is a possibility of collision, it outputs an alarm to the driver via the audio speaker 12061 or the display unit 12062, or performs forced deceleration or avoidance steering via the drive system control unit 12010, thereby providing driving assistance for collision avoidance.

[0255] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian exists in the captured image of the imaging units 12101 to 12104. Such recognition of a pedestrian is performed, for example, by a procedure of extracting feature points in the captured image of the imaging units 12101 to 12104 as infrared cameras and a procedure of performing pattern matching processing on a series of feature points indicating the outline of an object to determine whether it is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the captured image of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio and image output unit 12052 controls the display unit 12062 to superimpose and display a rectangular contour line for emphasis on the recognized pedestrian. Further, the audio and image output unit 12052 may control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0256] The above has described an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the vehicle external information detection unit 12030 and the vehicle internal information detection unit 12040 among the configurations described above. Specifically, by mounting the event detection sensor 1 or the imaging device 300 as the vehicle external information detection unit 12030 or the vehicle internal information detection unit 12040, it is possible to perform a process of detecting the driver's operation or to detect a change in the external situation of the vehicle and reflect it in the control of the vehicle.

[0257] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0258] For example, a form in which all or part of the above-described plurality of configuration examples are combined can be adopted.

[0259] In addition, each step described in the above flowchart can be executed by one device or can be executed in cooperation by a plurality of devices.

[0260] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or can be executed in cooperation by a plurality of devices.

[0261] Note that the effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0262] Note that the present technology can adopt the following configuration. (1) A counting unit that counts a first count number, which is the number of pixels in which a first luminance change in the positive direction is detected, and a second count number, which is the number of pixels in which a second luminance change in the negative direction is detected, in an image showing a luminance change output from a light receiving unit at a predetermined frame rate; A coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; An integrating unit that integrates the multiplication result of the count of the pixels and the coefficient A signal processing device comprising the same. (2) The signal processing device further includes a subtraction unit that subtracts the second count number from the first count number, and the integrating unit integrates the multiplication result of the subtraction result of the subtraction unit and the coefficient. The signal processing device according to (1) above. (3) The coefficient generation unit generates a coefficient corresponding to the time based on the predetermined frame rate. The signal processing device according to (1) or (2) above. (4) The integrating unit integrates the multiplication result by an integer multiple of the number of frames corresponding to the period to be detected. The signal processing device according to any one of (1) to (3) above. (5) The coefficient generation unit generates values of a sine function and a cosine function corresponding to the time as the coefficient. The signal processing device according to any one of (1) to (4) above. (6) The integrating unit multiplies the count of the pixels by the values of the sine function and the cosine function respectively, and calculates the multiplication result. The signal processing device according to (5) above. (7) The coefficient generation unit generates values of the sine function and the cosine function corresponding to the period to be detected as the coefficient. The signal processing device according to (5) or (6) above. (8) The coefficient generation unit generates values of a sine approximation function and a cosine approximation function obtained by approximating the sine function and the cosine function as the coefficient. The signal processing device according to any one of (5) to (7) above. (9) The sine approximation function and the cosine approximation function are functions obtained by approximating the sine function and the cosine function with a signal taking two values of 1 and -1. The signal processing device according to (8) above. (10) The coefficient generation unit outputs 1 or -1 based on a table associating 1 or -1 with the time. The signal processing device according to (9) above. (11) The signal processing device further includes a flicker amount estimation unit that estimates the amount of flicker in which the luminance change occurs at a specific frequency based on the integration result of the integration unit. The signal processing device according to any one of (1) to (10) above. (12) The signal processing device further includes a control unit that controls the sensitivity parameter of the light receiving unit based on the estimation result of the flicker amount estimation unit. The signal processing device according to (11) above. (13) The control unit controls the sensitivity parameter of the light receiving unit separately for the first luminance change and the second luminance change. The signal processing device according to (12) above. (14) The control unit controls the sensitivity parameter of the light receiving unit for each phase of the period to be detected. The signal processing device according to (12) or (13) above. (15) The signal processing device In an image showing a luminance change output from a light receiving unit at a predetermined frame rate, a first count number that is the count number of pixels in which a first luminance change in the positive direction is detected, and a second count number that is the count number of pixels in which a second luminance change in the negative direction is detected are counted. A coefficient corresponding to the time when the luminance change is detected is generated. The multiplication result of the pixel count number and the coefficient is integrated. Signal processing method. (16) A light receiving unit in which pixels that perform photoelectric conversion of incident light to generate an electrical signal are arranged in a grid pattern, In an image showing the luminance change output from the light receiving unit at a predetermined frame rate, a first count number which is the count number of pixels in which a first luminance change in the plus direction is detected, and a second count number which is the count number of pixels in which a second luminance change in the minus direction is detected, a counting unit that counts them; a coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; an integration unit that integrates the multiplication result of the count number of the pixels and the coefficient; A detection sensor comprising:

Explanation of Signs

[0263] 1 Event detection sensor, 11 Pixel array unit, 12 Signal processing circuit, 21 Pixel, 22 Detection circuit, 31 Event data acquisition unit, 32 Event counting unit, 33 Flicker detection unit, 34 Sensitivity control unit, 51 Subtractor, 52 Convolution coefficient generation unit, 53 Integration unit, 54 Flicker amount estimation unit, 71, 72 Multiplier, 73, 74 Integrator, 75, 76 Output unit, 151 Positive control value generation unit, 152 Negative control value generation unit, 171 Minimum value detection unit, 172 DR calculation unit, 173 Comparison unit, 174 Update determination unit, 181 Minimum value detection unit, 182 DR calculation unit, 183 Comparison unit, 184 Update determination unit, 300 Imaging device, 312 Image sensor, 601 Smartphone, 602 Event detection sensor, 603 Imaging device

Claims

1. In an image showing a luminance change output from a light receiving unit at a predetermined frame rate, a counting unit that counts a first count number that is the count number of pixels in which a first luminance change in the plus direction is detected, and a second count number that is the count number of pixels in which a second luminance change in the minus direction is detected; A subtraction unit that subtracts the second count number from the first count number; A coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; An integration unit that integrates the multiplication result of the subtraction result of the subtraction unit and the coefficient; A signal processing device comprising the above.

2. In an image showing a luminance change output from a light receiving unit at a predetermined frame rate, a counting unit that counts a first count number that is the count number of pixels in which a first luminance change in the plus direction is detected, and a second count number that is the count number of pixels in which a second luminance change in the minus direction is detected; A coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; An integration unit that integrates the multiplication result of the count number of the pixels and the coefficient; Comprising: The integration unit integrates the multiplication result by an integer multiple of the number of frames according to the period to be detected. A signal processing device.

3. In an image showing a luminance change output from a light receiving unit at a predetermined frame rate, a counting unit that counts a first count number that is the count number of pixels in which a first luminance change in the plus direction is detected, and a second count number that is the count number of pixels in which a second luminance change in the minus direction is detected; A coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; An integration unit that integrates the multiplication result of the count number of the pixels and the coefficient; Comprising: The coefficient generation unit generates values of a sine function and a cosine function according to the time as the coefficient. A signal processing device.

4. The integration unit multiplies the count number of the pixels by the values of the sine function and the cosine function respectively, and calculates the multiplication result. The signal processing device according to claim 3.

5. The coefficient generation unit generates the values of the sine function and the cosine function corresponding to the period to be detected as the coefficients. The signal processing device according to claim 3.

6. The coefficient generation unit generates the values of the sine approximation function and the cosine approximation function obtained by approximating the sine function and the cosine function as the coefficients. The signal processing device according to claim 3.

7. The sine approximation function and the cosine approximation function are functions obtained by approximating the sine function and the cosine function with a signal taking two values of 1 and -1. The signal processing device according to claim 6.

8. The coefficient generation unit outputs 1 or -1 based on a table associating 1 or -1 with the time. The signal processing device according to claim 7.

9. In an image indicating a luminance change output from a light receiving unit at a predetermined frame rate, a first count number which is the count number of pixels in which a first luminance change in the positive direction is detected, and a second count number which is the count number of pixels in which a second luminance change in the negative direction is detected, a counting unit for counting them, A coefficient generation unit for generating a coefficient corresponding to the time when the luminance change is detected; An integration unit for integrating the multiplication result of the count number of the pixels and the coefficient; A flicker amount estimation unit for estimating the flicker amount in which the luminance change occurs at a specific frequency based on the integration result of the integration unit A signal processing device comprising.

10. Further comprising a control unit that controls the sensitivity parameter of the light receiving unit based on the estimation result of the flicker amount estimation unit The signal processing apparatus according to claim 9.

11. The control unit controls the sensitivity parameter of the light receiving unit separately for the first luminance change and the second luminance change The signal processing apparatus according to claim 10.

12. The control unit controls the sensitivity parameter of the light receiving unit for each phase of the period to be detected The signal processing apparatus according to claim 10.

13. The coefficient generation unit generates a coefficient corresponding to the time based on the predetermined frame rate The signal processing apparatus according to any one of claims 1 to 12.

14. A signal processing apparatus In an image showing a luminance change output from a light receiving unit at a predetermined frame rate, a first count number that is the count number of pixels in which a first luminance change in the plus direction is detected, and a second luminance change in the minus direction is detected A second count number that is the count number of pixels is counted, Subtract the second count number from the first count number, Generate a coefficient corresponding to the time when the luminance change is detected, Integrate the multiplication result of the subtraction result and the coefficient A signal processing method.

15. A light receiving unit in which pixels that perform photoelectric conversion of incident light to generate an electrical signal are arranged in a lattice pattern, In an image showing a luminance change output from the light receiving unit at a predetermined frame rate, a first count number that is the count number of pixels in which a first luminance change in the plus direction is detected, and a second luminance change in the minus direction is detected A counting unit that counts a second count number that is the count number of pixels, A subtraction unit that subtracts the second count number from the first count number A coefficient generation unit that generates a coefficient according to the time when the luminance change is detected; An integration unit that integrates the multiplication result of the subtraction result of the subtraction unit and the coefficient; A detection sensor comprising the above.

Images