Signal processing device, signal processing method and detection sensor

Abstract

The present technology relates to a signal processing device, a signal processing method, and a detection sensor, which are used to enable flicker information to be detected from an output indicating a brightness change. The signal processing device includes: a counting unit that counts a first count number and a second count number in an image indicating a brightness change output from a light receiving unit at a predetermined frame rate, the first count number being the count number of pixels in which a first brightness change in a positive direction is detected, and the second count number being the count number of pixels in which a second brightness change in a negative direction is detected; a coefficient generating unit that generates a coefficient according to the time when the brightness change is detected; and an accumulation unit that accumulates a multiplication result obtained by multiplying the count number of pixels by the coefficient. The technology can be applied to, for example, an event detection sensor, etc.

Description

Technical Field

[0001] The present technology relates to a signal processing device, a signal processing method and a detection sensor, and more particularly, to a signal processing device, a signal processing method and a detection sensor capable of detecting flicker information from the output of an event detection sensor that detects the presence or absence of brightness changes at a constant period. Background Art

[0002] An image sensor has been proposed which, when taking a change in brightness of a pixel as an event, periodically outputs whether the event has occurred at a constant frame rate (for example, see Patent Document 1).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Publication No. 2017-535999 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] In this event detection sensor that detects the presence or absence of brightness changes at a constant cycle, when brightness changes occur across the entire screen and many events occur, such as when a light source in a room flickers, the event that was originally intended to be acquired is hidden in the many events caused by the flickering, and the information that was originally intended to be acquired cannot be captured.

[0008] The present technology has been made in view of such circumstances, and enables detection of flicker information from the output of an event detection sensor that detects the presence or absence of a brightness change at a constant cycle.

[0009] Solution to the problem

[0010] According to a first aspect of the present technology, a signal processing device includes: a counting unit that counts a first count number and a second count number in an image output from a light receiving unit at a predetermined frame rate and indicating a brightness change, the first count number being the count number of pixels in which a first brightness change in a positive direction is detected, and the second count number being the count number of pixels in which a second brightness change in a negative direction is detected; a coefficient generating unit that generates a coefficient corresponding to a time when the brightness change is detected; and an accumulation unit that accumulates a multiplication result of the count number of pixels and the coefficient.

[0011] In a signal processing method according to the first aspect of the present technology, the signal processing device performs the following operations: in an image output from a light receiving unit at a predetermined frame rate and indicating a brightness change, counts a first count number and a second count number, the first count number being the count number of pixels in which a first brightness change in a positive direction is detected, and the second count number being the count number of pixels in which a second brightness change in a negative direction is detected; generates a coefficient corresponding to the time when the brightness change is detected; and accumulates the multiplication result of the pixel count number and the coefficient.

[0012] In a first aspect of the present technology, in an image output from a light receiving unit at a predetermined frame rate and indicating a brightness change, a first count number and a second count number are counted, the first count number being the count number of pixels in which a first brightness change in a positive direction is detected, the second count number being the count number of pixels in which a second brightness change in a negative direction is detected, a coefficient corresponding to the time at which the brightness change is detected is generated, and the multiplication result of the pixel count number and the coefficient is accumulated.

[0013] A detection sensor according to a second aspect of the present technology includes: a light receiving unit in which pixels that perform photoelectric conversion of incident light and generate electrical signals are arranged in a grid pattern; a counting unit that counts a first count number and a second count number in an image that is output from the light receiving unit at a predetermined frame rate and indicates a brightness change, the first count number being the count number of pixels in which a first brightness change in a positive direction is detected, and the second count number being the count number of pixels in which a second brightness change in a negative direction is detected; a coefficient generating unit that generates a coefficient corresponding to a time when a brightness change is detected; and an accumulation unit that accumulates a multiplication result of the count number of pixels and the coefficient.

[0014] In a second aspect of the present technology, an electrical signal is generated by performing photoelectric conversion of incident light in a pixel of a light receiving unit, and in an image that is output from the light receiving unit at a predetermined frame rate and indicates a brightness change, a first count number and a second count number are counted, the first count number being the count number of pixels in which a first brightness change in a positive direction is detected, and the second count number being the count number of pixels in which a second brightness change in a negative direction is detected, a coefficient corresponding to the time at which the brightness change is detected is generated, and the multiplication result of the pixel count number and the coefficient is accumulated.

[0015] Note that the signal processing device according to the first aspect of the present technology can be implemented by causing a computer to execute a program. The program executed by the computer can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.

[0016] The signal processing device and the detection sensor may be independent devices, or internal blocks constituting one device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 : is a block diagram showing a configuration example of an embodiment of an event detection sensor as a sensor to which the present technology is applied.

[0018] Figure 2 It is shown by Figure 1 FIG. 10 is a diagram of an example of an image in which an event detection sensor detects changes.

[0019] Figure 3 2 is a diagram showing the principle of the occurrence of events caused by light source flickering.

[0020] Figure 4 is a graph showing the result of detecting an event under a light source having a power frequency of 50 Hz.

[0021] Figure 5 A diagram showing an overview of flicker detection by a flicker detection unit and sensitivity parameter control by a sensitivity control unit.

[0022] Figure 6 A diagram showing an overview of flicker detection by a flicker detection unit and sensitivity parameter control by a sensitivity control unit.

[0023] Figure 7 It is shown Figure 1 A block diagram of a detailed configuration example of a flicker detection unit in FIG.

[0024] Figure 8 is a block diagram showing an example of a sine function and a cosine function or a sine approximation function and a cosine approximation function generated by a convolution coefficient generation unit.

[0025] Fig. 9 It shows that the Figure 8 A diagram of a circuit configuration example of a convolution coefficient generating unit in the case of a sine approximate function and a cosine approximate function shown in FIG.

[0026] Fig.10 Is Fig. 9 A timing diagram showing the case where the logic circuit shown performs an operation.

[0027] Fig.11 2 is a diagram showing a method of determining a +-side threshold value for controlling the detection sensitivity of a + change.

[0028] Fig.122 is a diagram showing a method of determining a +-side threshold value for controlling the detection sensitivity of a + change.

[0029] Fig.13 is an image diagram of the input and output of the sensitivity control unit.

[0030] Fig.14 is a block diagram showing a detailed configuration example of a sensitivity control unit.

[0031] Fig.15 It is shown by Figure 1 Flowchart of the flicker control process performed by the event detection sensor in FIG.

[0032] Fig.16 It is shown by Figure 1 FIG. 1 is a diagram of an example of a processing result of a flicker control process performed by an event detection sensor in FIG.

[0033] Fig.17 : is a block diagram showing a configuration example of another embodiment as an event detection sensor to which the present technology is applied.

[0034] Fig.18 is a block diagram showing a configuration example of an imaging device including an event detection sensor as an imaging element.

[0035] Fig.19 It is shown Fig.18 A perspective view of a schematic configuration example of an imaging element in FIG.

[0036] Fig. 20 is a plan view showing a configuration example of a light receiving chip.

[0037] Fig.21 is a plan view showing a configuration example of a detection chip.

[0038] Fig. 22 is a plan view showing details of the address event detection unit.

[0039] Fig.23 is a block diagram showing a configuration example of a detection block.

[0040] Fig.24 is a circuit diagram showing a detailed configuration of a detection unit.

[0041] Fig.25 It is shown Fig.23 FIG. 1 is a diagram of a configuration example of a comparison unit of a detection block in FIG.

[0042] Fig.26 is a timing chart showing a control example of the row driving circuit.

[0043] Fig. 27is a block diagram showing a configuration example of a smartphone as an electronic device to which the present technology is applied.

[0044] Fig.28 is a block diagram showing an example of a schematic configuration of a vehicle control system.

[0045] Fig.29 1 is an explanatory diagram showing an example of installation positions of the vehicle exterior information detection section and the imaging section. DETAILED DESCRIPTION

[0046] Hereinafter, a mode for executing the present technology (hereinafter referred to as an embodiment) will be described with reference to the accompanying drawings. Note that in this specification and the accompanying drawings, components having substantially the same functional configuration are designated by the same reference numerals to omit repeated descriptions. The description will be given in the following order.

[0047] 1. Configuration example of event detection sensor

[0048] 2. Principle of events caused by light source flickering

[0049] 3. Configuration example of flicker detection unit

[0050] 4. Configuration example of sensitivity control unit

[0051] 5. Processing flow of flicker control process

[0052] 6. Example of processing results of the flicker control process

[0053] 7. Another configuration example of event detection sensor

[0054] 8. Configuration example of imaging device

[0055] 9. Example of electronic device configuration

[0056] 10. Application examples for mobile objects

[0057] <1. Configuration example of event detection sensor>

[0058] Figure 1 : is a block diagram showing a configuration example of an embodiment of an event detection sensor as a sensor to which the present technology is applied.

[0059] The event detection sensor 1 includes a pixel array unit 11 as a light receiving unit and a signal processing circuit 12 that processes a signal generated by the light receiving unit.

[0060] In the pixel array unit 11, pixels 21 that receive incident light and perform photoelectric conversion are arranged in a lattice pattern. In addition, in the pixel array unit 11, a detection circuit 22 that detects a brightness change (light amount change) generated in the pixel 21 as an event corresponds to each pixel 21, and is configured in a different layer at the same plane position as the pixel 21, for example. Therefore, the detection circuit 22 is also arranged in a lattice pattern.

[0061] In the corresponding pixel 21, each detection circuit 22 detects whether there is a brightness change in the positive direction exceeding a predetermined threshold (hereinafter referred to as "+ change"), a brightness change in the negative direction exceeding a predetermined threshold (hereinafter referred to as "- change"), or a brightness change exceeding a predetermined threshold within a predetermined time period corresponding to the frame rate, and outputs the result as a detection signal.

[0062] The pixel array unit 11 outputs the detection signals of the respective detection circuits 22 to the signal processing circuit 12 in sequence in a predetermined order under the control of the pixel driving unit (not shown).

[0063] Therefore, the pixel array unit 11 detects the presence or absence of brightness change in units of pixels 21 (in units of detection circuits 22) at a constant frame rate, and outputs image data of a change image storing the detection result as a pixel value of each pixel 21 as event data to the signal processing circuit 12. The pixel value of each pixel of the change image is a value indicating any of +change, -change, and no change.

[0064] Figure 2 An example of a changing image in a case where a certain imaging scene is detected by the event detection sensor 1 is shown.

[0065] like Figure 2 As shown on the left side of , it is assumed that the event detection sensor 1 detects a scene in which a person moves in the direction indicated by the arrow. The person as the subject appears brighter than the surrounding background. In this case, in the pixels of the contour portion on the side of the person's traveling direction in the output change image, a brightness change from dark (low brightness) to bright (high brightness) is detected, and thus a pixel value indicating a +change is stored. On the other hand, in the pixels of the contour portion on the side opposite to the person's traveling direction, a brightness change from bright (high brightness) to dark (low brightness) is detected, and thus a pixel value indicating a -change is stored. The other pixels have pixel values ​​indicating no change.

[0066] In the change image, the pixel value takes any ternary value indicating + change, - change or no change, and does not require a high grayscale such as 8 bits or 10 bits like a normal image sensor, so the operation can also be performed in an extremely short time of exposure time and AD conversion time. Therefore, compared with the frame rate of a normal image sensor such as 30fps or 60fps, the change image can be output at an extremely high frame rate. For example, the change image can be output at a high frame rate such as 1000fps.

[0067] Back to Figure 1 The signal processing circuit 12 includes an event data acquisition unit 31 , an event counting unit 32 , a flicker detection unit 33 , and a sensitivity control unit 34 .

[0068] The event data acquisition unit 31 acquires a change image output from the pixel array unit 11 at a predetermined frame rate, outputs the change image to the outside of the sensor, and supplies the change image to the event counting unit 32 .

[0069] The event counting unit 32 counts the number of +-changed pixels (hereinafter also referred to as the +-count number) and the number of --changed pixels (hereinafter also referred to as the “--count number”) relative to the change image sequentially provided from the event data acquisition unit 31, and provides the counting results to the flicker detection unit 33 and the sensitivity control unit 34.

[0070] The flicker detection unit 33 detects (estimates) a flicker amount of a predetermined cycle by using the +count number and the −count number supplied from the event counting unit 32 , and outputs the flicker amount to the outside of the event detection sensor 1 and the sensitivity control unit 34 .

[0071] The sensitivity control unit 34 determines whether flicker of a predetermined period occurs based on the flicker amount provided from the flicker detection unit 33. In the case where it is determined that flicker of a predetermined period occurs, the sensitivity control unit 34 adjusts (controls) the sensitivity parameter of each detection circuit 22 of the pixel array unit 11 by using the + count number and the - count number provided from the event counting unit 32. For example, the sensitivity control unit 34 performs control so that in the case where the flicker amount is large, the threshold of the brightness change captured as an event is increased, making the occurrence of the event difficult, and in the case where the flicker amount is small, the threshold of the brightness change is lowered, making the occurrence of the event easy. The control value for controlling the threshold of the brightness change is provided from the sensitivity control unit 34 to each detection circuit 22 of the pixel array unit 11.

[0072] <2. Principle of events caused by light source flickering>

[0073] Will refer to Figure 3 Describe the principle of what happens when a light source flashes.

[0074] When an event is detected in an environment using a light source having a power frequency of 50 Hz, the light source generates flicker at 100 Hz which is twice the power frequency of 50 Hz.

[0075] like Figure 3 As shown, one cycle of a light source having a power frequency of 50 Hz is 20 milliseconds, and flickering occurs with a cycle of 10 milliseconds which is 1 / 2 of the cycle.

[0076] Furthermore, when the brightness change is divided into a + change event (hereinafter also referred to as a positive event) and a - change event (hereinafter also referred to as a negative event), as Figure 3 As shown, positive events and negative events are detected alternately every 5 milliseconds.

[0077] Figure 4 The results of actual detection events under a light source having a power frequency of 50 Hz are shown.

[0078] Figure 4 This is a graph in which the frame rate of the event detection sensor 1 is set to 1000 fps, a change image is generated every 1 millisecond, and the number of each of positive events and negative events accumulated for 10 milliseconds is displayed every 1 millisecond.

[0079] When looking at each of the positive event and the negative event, the event occurs with a cycle of 10 milliseconds under a light source having a power frequency of 50 Hz.

[0080] In this regard, the flicker detection unit 33 of the event detection sensor 1 detects the presence or absence of flicker of a predetermined period by detecting whether the periodicity changes in a predetermined period based on the number of events of the positive event and the negative event.

[0081] Note that in the following description, a case where the flicker detection unit 33 detects flicker with a cycle of 10 milliseconds generated under a light source having a power frequency of 50 Hz will be described as an example.

[0082] Will refer to Figure 5 and Figure 6 An outline of flicker detection performed by the flicker detection unit 33 and sensitivity parameter control performed by the sensitivity control unit 34 is described.

[0083] As reference Figure 4 As described, under a light source having a power frequency of 50 Hz, flicker inducing events with a period of 10 milliseconds occur in each of the positive events and the negative events.

[0084] In this regard, in the case of detecting flicker occurring under a light source having a power frequency of 50 Hz, the flicker detection unit 33 detects flicker by using the counted numbers of positive events and negative events with 10 milliseconds as a detection cycle.

[0085] Figure 5 This is a schematic image diagram of flicker that occurs under a light source with a power frequency of 50 Hz.

[0086] like Figure 5 As shown in , the peak of the count number of positive events or negative events appears at the timing when flicker occurs.

[0087] like Figure 6 As shown, it is more accurately described in frames Figure 5 The flickering appearance image in.

[0088] In this embodiment, when the frame rate of the event detection sensor 1 is set to 1000 fps, 10 change images are generated in a cycle of 10 milliseconds as a detection cycle. Then, among the ten frames in one cycle (10 milliseconds), there are frames in which the light source having a power frequency of 50 Hz flickers and frames in which flickers do not occur.

[0089] The sensitivity control unit 34 controls the sensitivity parameter of the detection circuit 22 in units of frames. Specifically, the sensitivity control unit 34 lowers the detection sensitivity only for frames where light source flicker occurs among the 10 frames, and keeps the detection sensitivity high (unchanged) for frames where light source flicker does not occur.

[0090] In other words, the sensitivity control unit 34 controls the sensitivity parameter for each phase generated in the detection cycle. A frame corresponding to a phase in the detection cycle indicating the number of a frame among ten frames corresponding to the detection cycle is referred to as a frame phase.

[0091] Note that, instead of 10 milliseconds corresponding to the blinking period, the detection period may be a period that is an integral multiple of the blinking period.

[0092] like Figure 4 As shown, the positive event and the negative event occur at different timings, and therefore, the sensitivity control unit 34 performs sensitivity control on the positive event and the negative event, respectively. That is, the sensitivity control unit 34 performs control to change the frame phase for detecting a + change threshold for a positive event due to light source flickering, and to change the frame phase for detecting a - change threshold for a negative event due to light source flickering.

[0093] <3. Configuration Example of Flicker Detection Unit>

[0094] Figure 7 It is shown Figure 1A block diagram of a detailed configuration example of the flicker detection unit 33 in FIG.

[0095] The flicker detection unit 33 includes a subtractor 51 , a convolution coefficient generation unit 52 , an integration unit 53 , and a flicker amount estimation unit 54 .

[0096] The accumulation unit 53 includes multipliers 71 and 72 , accumulators 73 and 74 , and output units 75 and 76 .

[0097] The +count number and the −count number are supplied from the event counting unit 32 to the flicker detecting unit 33 , and the supplied +count number and the −count number are input to the subtractor 51 .

[0098] The subtractor 51 subtracts the −count number from the +count number, and outputs the subtraction result to the multipliers 71 and 72 of the accumulation unit 53 .

[0099] The convolution coefficient generating unit 52 generates a function value obtained by substituting a value corresponding to the time of the flicker cycle as an argument into the sine function and cosine function of the flicker cycle to be detected, and supplies the function value as a convolution coefficient to the multipliers 71 and 72 of the integrating unit 53.

[0100] A vertical drive signal Vsync corresponding to the frame rate of the changing image output by the pixel array unit 11 is provided from a timing control unit (not shown) to the convolution coefficient generation unit 52, and the convolution coefficient generation unit 52 generates a value corresponding to the time of the flicker cycle based on the vertical drive signal Vsync, and substitutes the value into the sine function and cosine function of the flicker cycle to be detected.

[0101] Note that, instead of the sine function and the cosine function, an approximate function obtained by approximating the sine function and the cosine function may be used, and an approximate value obtained by substituting a value corresponding to the time of the blinking cycle as an argument into the approximate function may be supplied to the multipliers 71 and 72 of the accumulating unit 53. Figures 8 to 10 As described, the function value corresponding to the time of the flickering period is calculated by using the sine approximation function and the cosine approximation function which approximate the sine function and the cosine function as signals taking two values ​​1 and -1. The function value of the calculated sine approximation function is provided to the multiplier 72, and the function value of the calculated cosine approximation function is provided to the multiplier 71.

[0102] In addition, the convolution coefficient generation unit 52 generates an enable signal indicating the timing at which the accumulation unit 53 outputs the accumulation result, and supplies the enable signal to the output units 75 and 76 of the accumulation unit 53. The accumulation period of the accumulation unit 53 determined at the timing at which the enable signal becomes High may be, for example, 10 milliseconds, which is the same as the flicker cycle (one cycle) to be detected. Alternatively, the accumulation period may be a cycle that is an integer multiple of the flicker cycle.

[0103] The accumulation unit 53 accumulates the multiplication result obtained by multiplying the subtraction result provided from the subtractor 51 and obtained by subtracting the -count number from the +count number by the convolution coefficient provided from the convolution coefficient generating unit 52, and uses the sine function and cosine function or the sine approximation function and cosine approximation function obtained by approximating the sine function and cosine function.

[0104] The multiplier 71 supplies a multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 by the function value of the cosine approximation function supplied from the convolution coefficient generation unit 52 to the accumulator 73 .

[0105] The multiplier 72 supplies a multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 by the function value of the sine approximation function supplied from the convolution coefficient generation unit 52 to the accumulator 74 .

[0106] The accumulator 73 accumulates the multiplication results supplied from the multiplier 71 and supplies the result to the output unit 75. The accumulator 74 accumulates the multiplication results supplied from the multiplier 72 and supplies the result to the output unit 76.

[0107] The output unit 75 includes, for example, a flip-flop, acquires the cumulative value cos_sum of the accumulator 73 at the timing when the enable signal supplied from the convolution coefficient generation unit 52 is high, and supplies the cumulative value cos_sum to the flicker amount estimation unit 54 .

[0108] The output unit 76 includes, for example, a flip-flop, acquires the integrated value sin_sum of the accumulator 74 at the timing when the enable signal supplied from the convolution coefficient generation unit 52 is high, and supplies the integrated value sin_sum to the flicker amount estimation unit 54 .

[0109] The flicker amount estimation unit 54 calculates the amplitude component of the flicker frequency by using the integration result supplied from each of the output units 75 and 76 , and estimates the flicker amount (flicker possibility) occurring at a specific frequency (flicker frequency).

[0110] Specifically, the flicker amount estimation unit 54 calculates the flicker amount EST_FL by formula (1) or formula (2) using the frame accumulation number sum, the accumulation values ​​cos_sum, and sin_sum.

[0111] [Mathematical formula 1]

[0112] EST_FL=(|sin_sum|+|cos_sum|) / sum…(1)

[0113]

[0114] Here, the frame accumulation number sum is equal to the number of frames accumulated by the accumulation unit 53, and in the present embodiment, the accumulation period is 10 milliseconds which is the same as the blinking cycle, so sum=10.

[0115] In formulas (1) and (2), when flicker of the target cycle does not occur, the flicker amount EST_FL is a small value, and when flicker of the target cycle occurs, the flicker amount EST_FL is a large value.

[0116] Figure 8 Examples of the sine function and the cosine function generated by the convolution coefficient generation unit 52 or the sine approximation function and the cosine approximation function obtained by approximating the sine function and the cosine function are shown.

[0117] like Figure 8 As shown in the upper part of , the convolution coefficient generation unit 52 can generate a function value obtained by substituting a value corresponding to the time of the flicker cycle into a sine function and a cosine function with the flicker cycle to be detected as one cycle. However, in the present embodiment, Figure 8 As shown in the lower part of , the function value corresponding to the time of the flickering period is calculated by using the sine approximation function and the cosine approximation function that approximate the sine function and the cosine function as signals taking two values ​​1 and -1.

[0118] When the sine approximation function and the cosine approximation function are expressed by sin_approx(t) and cos_approx(t), the sine approximation function and the cosine approximation function can be expressed by the following formulas.

[0119] [Mathematical formula 2]

[0120]

[0121]

[0122] In formulas (3) and (4), f represents the flickering period, and t represents the value corresponding to the time of the flickering period. By this approximation, Figure 8As shown in , the sine approximation function and the cosine approximation function are approximated to output +1 when the sine function and the cosine function are positive and output -1 when the sine function and the cosine function are negative. Since the current flicker cycle is 10 milliseconds, the outputs of the sine approximation function and the cosine approximation function switch between +1 and -1 in units of 5 milliseconds.

[0123] As such a sine approximation function and a cosine approximation function, for example, a configuration may be adopted in which a table in which +1 or -1 is associated with each time of one cycle is stored in the convolution coefficient generation unit 52, and function values ​​of the sine approximation function and the cosine approximation function are output based on the table.

[0124] In addition, the sine approximation function and the cosine approximation function can be obtained by Fig. 9 The logic circuit shown is used to implement it.

[0125] Fig. 9 Shown in the use of Figure 8 An example of the circuit configuration of the convolution coefficient generating unit 52 in the case of the sine approximation function and the cosine approximation function is shown.

[0126] Fig. 9 The convolution coefficient generating unit 52 in EMBODIMENT 1 includes a counter 101 which performs counting corresponding to the flickering period.

[0127] Furthermore, the convolution coefficient generation unit 52 includes comparators 102 and 103 , a selector 104 , and a flip-flop 105 as a configuration for outputting a function value cos_approx of a cosine approximation function corresponding to the time of the flicker period.

[0128] Furthermore, the convolution coefficient generation unit 52 includes comparators 111 and 112 , a selector 113 , and a flip-flop 114 as a configuration for outputting a function value sin_approx of a sinusoidal approximation function corresponding to the time of the blinking period.

[0129] Furthermore, the convolution coefficient generation unit 52 includes a comparator 121 as a configuration for outputting an enable signal.

[0130] The vertical drive signal Vsync and the count number cycle corresponding to the flicker cycle are input to the counter 101. The counter 101 starts the count value cnt from 0 and counts up according to the vertical drive signal Vsync. Then, when the count value cnt is counted up to the count number cycle, the counter 101 resets the count value cnt and repeats the process of counting from 0 again. Since the current flicker cycle is 10 milliseconds and the vertical drive signal Vsync is a signal corresponding to the frame rate of 1000fps and becomes high at intervals of 1 millisecond, "10" is input as the count number cycle.

[0131] The count value cnt of the counter 101 is supplied to the comparators 102 , 103 , 111 , 112 , and 121 .

[0132] The set value cos_ptim is supplied to the comparator 102, and the comparator 102 compares the count value cnt supplied from the counter 101 with the set value cos_ptim, and outputs a +1 value to the selector 104 at the timing when the count value cnt matches the set value cos_ptim. For the count value cnt other than the set value cos_ptim, for example, 0 is output.

[0133] The set value cos_ntim is supplied to the comparator 103, and the comparator 103 compares the count value cnt supplied from the counter 101 with the set value cos_ntim, and outputs -1 to the selector 104 at the timing when the count value cnt matches the set value cos_ntim. For the count value cnt other than the set value cos_ntim, for example, 0 is output.

[0134] The selector 104 selects +1 and outputs +1 to the flip-flop 105 at a timing when +1 is provided from the comparator 102, selects -1 and outputs -1 to the flip-flop 105 at a timing when -1 is provided from the comparator 102, and outputs the value fed back from the flip-flop 105 to the flip-flop 105 at other timings.

[0135] The flip-flop 105 holds and outputs the value (+1 or -1) input from the selector 104 until the value is updated next time. The value output from the flip-flop 105 is the function value cos_approx of the cosine approximation function.

[0136] The set value sin_ptim is supplied to the comparator 111, and the comparator 111 compares the count value cnt supplied from the counter 101 with the set value sin_ptim, and outputs +1 to the selector 113 at the timing when the count value cnt matches the set value sin_ptim. For the count value cnt other than the set value sin_ptim, for example, 0 is output.

[0137] The set value sin_ntim is supplied to the comparator 112, and the comparator 112 compares the count value cnt supplied from the counter 101 with the set value sin_ntim, and outputs -1 to the selector 113 at the timing when the count value cnt matches the set value sin_ntim. For the count value cnt other than the set value sin_ntim, for example, 0 is output.

[0138] The selector 113 selects +1 and outputs +1 to the trigger 114 at the timing when +1 is provided from the comparator 111, selects -1 and outputs -1 to the trigger 114 at the timing when -1 is provided from the comparator 112, and outputs the value fed back from the trigger 114 to the trigger 114 at other timings.

[0139] The flip-flop 114 holds and outputs the value (+1 or -1) input from the selector 113 until the value is updated next time. The value output from the flip-flop 114 is the function value sin_approx of the sine approximation function.

[0140] The count number cycle is provided to the comparator 121, and the comparator 121 compares the count value cnt provided from the counter 101 with the count number cycle, and sets the enable signal to high at the timing when the count value cnt matches the count number cycle. For the count value cnt other than the count number cycle, a low enable signal is output. Since the count value cnt provided from the counter 101 is a repetition of 1, 2, 3, ..., 10, when the count value cnt is 10, a high enable signal is output.

[0141] Fig.10 Shown in Fig. 9 The timing diagram of the case where the logic circuit shown is operated.

[0142] Figure 8 The sine approximation function sin_approx(t) and cosine approximation function cos_approx(t) shown are given by Fig. 9 Logic circuit implementation.

[0143] The enable signal is high in units of 10 milliseconds in a frame cycle, specifically, in a case where flickering is detected in a light source having a power frequency of 50 Hz.

[0144] <4. Configuration Example of Sensitivity Control Unit>

[0145] Next, we will describe Figure 1 The sensitivity control unit 34 in the control of the sensitivity parameters.

[0146] The + count number and the - count number are provided from the event counting unit 32 to the sensitivity control unit 34, and the detection result of the flicker amount is provided from the flicker detection unit 33. In the case where the detected flicker amount is large (greater than the predetermined flicker determination threshold FL_TH), the sensitivity control unit 34 performs control to reduce the detection sensitivity by changing the control value of the threshold for controlling the brightness change as a sensitivity parameter. In the present embodiment, the sensitivity control unit 34 directly changes the threshold of the brightness change itself as a control value, and controls the + side threshold Vrefp for controlling the detection sensitivity of the + change and the - side threshold Vrefn for controlling the detection sensitivity of the - change, respectively.

[0147] Since the flicker amount of each detection cycle is provided by the flicker detection unit 33 with a detection cycle of 10 milliseconds, the sensitivity control unit 34 determines the + side threshold value Vrefp and the - side threshold value Vrefn as the control value of the next detection cycle for each frame phase based on the number of event counts for each frame phase of the detection cycle.

[0148] Reference Fig.11 and Fig.12 , a method of determining the +-side threshold value Vrefp for controlling the detection sensitivity of +-change will be described.

[0149] exist Fig.11 and Fig.12 In the embodiment, the flicker amount EST_FL of the detection period DT2 is provided from the flicker detection unit 33 at a predetermined timing, and the + count number P_count(i) (i is an integer from 0 to 9) of each frame phase i of the detection period DT2 is sequentially provided from the event counting unit 32. The sensitivity control unit 34 determines the control value sense(i) of each frame phase i of the next detection period DT3, that is, the + side threshold value Vrefp(i), based on the + count number P_count(i) of each frame phase i of the detection period DT2.

[0150] First, the sensitivity control unit 34 calculates the minimum value min(DT2) of the frame phase i+count number P_count(i) of the detection period DT2 by formula (5). MIN() in formula (5) represents a function for calculating the minimum value (i is an integer from 0 to 9).

[0151] min(DT2)=NIN(P_count(i))………(5)

[0152] Next, as shown in Formula (6), the sensitivity control unit 34 calculates the dynamic range DR(i) of each frame phase i of the detection period DT2 by subtracting the minimum value min(DT2) from the +count number P_count(i) of each frame phase i of the detection period DT2.

[0153] DR(i)=P_count(i)-min(DT2)…………(6)

[0154] Then, in the case where the calculated dynamic range DR(i) of each frame phase i is less than the preset first threshold DR_TH1, the sensitivity control unit 34 changes the control value sense(i) of the frame phase i of the next detection cycle DT3 to increase the detection sensitivity. In the case of increasing the detection sensitivity, the sensitivity control unit 34 changes the control value sense(i) as the + side threshold Vrefp(i) in the direction of reducing the control value sense(i). Specifically, the sensitivity control unit 34 calculates the control value sense'(i) of the frame phase i of the next detection cycle DT3 by the following formula (7).

[0155] sense'(i)=MAX(sense(i)+VALUE,LOWER_LIMIT)...(7)

[0156] In formula (7), MAX() is a function for selecting a maximum value, VALUE represents the variation amplitude of the detection sensitivity, and LOWER_LIMIT represents the limit value in the case of increasing the detection sensitivity. According to formula (7), in the case where the value {sense(i)-VALUE} obtained by subtracting the variation amplitude VALUE from the control value sense(i) of the detection period DT2 is equal to or greater than the limit value LOWER_LIMIT, the sensitivity control unit 34 determines the subtraction value as the control value sense'(i) of the frame phase i of the next detection period DT3, and in the case where the value {sense(i)-VALUE} is less than the limit value LOWER_LIMIT, the sensitivity control unit determines the limit value LOWER_LIMIT as the control value sense'(i) of the frame phase i of the next detection period DT3.

[0157] On the other hand, when the calculated dynamic range DR(i) is greater than the preset second threshold DR_TH2, the sensitivity control unit 34 changes the control value sense(i) of the frame phase i of the next detection period DT3 to reduce the detection sensitivity. In the case of reducing the detection sensitivity, the sensitivity control unit 34 changes the control value sense(i) as the + side threshold Vrefp(i) in the direction of increasing the control value sense(i). Specifically, the sensitivity control unit 34 calculates the control value sense'(i) of the frame phase i of the next detection period DT3 by the following formula (8).

[0158] sense'(i)=MIN(sense(i)+VALUE, UPPER_LIMIT)...(8)

[0159] In formula (8), MIN() is a function for selecting a minimum value, VALUE represents the variation amplitude of the detection sensitivity, and UPPER_LIMIT represents the limit value in the case of reducing the detection sensitivity. According to formula (8), in the case where the value {sense(i)+VALUE} obtained by adding the variation amplitude VALUE to the control value sense(i) of the detection period DT2 is equal to or less than the limit value UPPER_LIMIT, the sensitivity control unit 34 determines the added value as the control value sense'(i) of the frame phase i of the next detection period DT3, and in the case where the value {sense(i)+VALUE} is greater than the limit value UPPER_LIMIT, the sensitivity control unit 34 determines the limit value UPPER_LIMIT as the control value sense'(i) of the frame phase i of the next detection period DT3.

[0160] Note that in this example, the change amplitude VALUE in the addition direction for increasing the detection sensitivity and the change amplitude VALUE in the subtraction direction for decreasing the detection sensitivity have the same value, but may have different values.

[0161] When the calculated dynamic range DR(i) of each frame phase i is equal to or greater than the first threshold DR_TH1 and equal to or less than the second threshold DR_TH2, the control value sense(i) of the frame phase i is not changed and the current control value sense(i) is maintained.

[0162] exist Fig.11 In the example, each dynamic range DR(5) to DR(9) of the fifth frame phase to the ninth frame phase of the detection period DT2 is greater than the second threshold DR_TH2, and therefore the control values ​​sense'(5) to sense'(9) of the fifth frame phase to the ninth frame phase of the next detection period DT3 are changed in the direction of reducing the detection sensitivity. In other words, the control values ​​sense'(5) to sense'(9) of the detection period DT3 are changed to be higher than the control values ​​sense(5) to sense(9) of the detection period DT2 by a variation amplitude VALUE.

[0163] On the other hand, the respective dynamic ranges DR(0) to DR(4) from the 0th frame phase to the fourth frame phase of the detection period DT2 are less than the first threshold DR_TH1, but reach the limit value LOWER_LIMIT with increasing detection sensitivity, and therefore, the control values ​​sense'(0) to sense'(4) from the 0th frame phase to the fourth frame phase of the next detection period DT3 are not changed.

[0164] Fig.12A setting example of the control value sense(i) corresponding to the +count number P_count(i) of each frame phase i of the continuous four detection periods DT1 to DT4 is shown.

[0165] The respective dynamic ranges DR(5) to DR(9) of the fifth frame phase to the ninth frame phase of the detection period DT2 are greater than the second threshold DR_TH2, and therefore, the control values ​​sense(5) to sense(9) of the fifth frame phase to the ninth frame phase of the next detection period DT3 are changed in a direction of reducing the detection sensitivity. In other words, the control values ​​sense(5) to sense(9) of the detection period DT3 are changed in a direction having a higher variation amplitude VALUE than the control values ​​sense(5) to sense(9) of the detection period DT2.

[0166] In the next detection cycle DT3, only the dynamic range DR(7) of the seventh frame phase is greater than the second threshold DR_TH2, and therefore, in the next detection cycle DT4, only the control value sense(7) of the seventh frame phase is changed in the direction of reducing the detection sensitivity. In other words, the control value sense(7) of the detection cycle DT4 is changed to a direction higher than the control value sense(7) of the detection cycle DT3, and the control values ​​sense(5) and sense(6) of the fifth and sixth frame phases and the control values ​​sense(8) and sense(9) of the eighth and ninth frame phases remain the same as the control values ​​of the detection cycle DT3.

[0167] Although the control value sense(i) for each frame phase i for controlling the detection sensitivity of +change has been described, the sensitivity control unit 34 also performs similar control on the control value sense(i) for each frame phase i for controlling the detection sensitivity of -change.

[0168] Fig.13 It is a diagram showing an image of a control value sense(i) as an output of the sensitivity control unit 34 relative to the count number of the event counting unit 32 as an input of the sensitivity control unit 34 for each of the control of the +change detection sensitivity and the control of the -change detection sensitivity.

[0169] Since the polarity is opposite between the control of the detection sensitivity of the + change and the control of the detection sensitivity of the - change, in the control of the detection sensitivity of the change, the sensitivity control unit 34 changes the control value sense(i) as the - side threshold value Vrefn to the direction of reducing the detection sensitivity in the case of reducing the detection sensitivity, and changes the control value sense(i) as the - side threshold value Vrefn in the direction of increasing the detection sensitivity in the case of increasing the detection sensitivity. Note that the amplitude relationship of the control value sense(i) also changes according to the structure of the pixel of the light receiving unit and the polarity at the time of detection, and therefore, can be appropriately determined according to the situation.

[0170] In the control of the -variable detection sensitivity, the variation amplitude VALUE in the case of increasing and decreasing the detection sensitivity may be the same value similarly to the control of the +variable detection sensitivity, or may be different values. In addition, the variation amplitude VALUE of the detection sensitivity may be the same value between the control of the +variable detection sensitivity and the control of the -variable detection sensitivity, or may be different values.

[0171] Hereinafter, the control value sense(i) of the +changed detection sensitivity is referred to as the control value p_sense(i), and the control value sense(i) of the −changed detection sensitivity is referred to as the control value n_sense(i) for distinction.

[0172] Fig.14 : is a block diagram showing a detailed configuration example of the sensitivity control unit 34 .

[0173] The sensitivity control unit 34 includes a positive control value generating unit 151 that generates a control value p_sense(i) for controlling the detection sensitivity of +change and a negative control value generating unit 152 that generates a control value n_sense(i) for controlling the detection sensitivity of -change.

[0174] The positive control value generating unit 151 includes a minimum value detecting unit 171 , a DR calculating unit 172 , a comparing unit 173 , and an updating determining unit 174 .

[0175] The negative control value generating unit 152 includes a minimum value detecting unit 181 , a DR calculating unit 182 , a comparing unit 183 , and an updating determining unit 184 .

[0176] like Fig.14 As shown, the positive control value generating unit 151 and the negative control value generating unit 152 have substantially the same configuration.

[0177] The + count number and the - count number are provided from the event counting unit 32 to the sensitivity control unit 34, and the detected flicker amount is provided from the flicker detecting unit 33 to the sensitivity control unit 34. The + count number from the event counting unit 32 is input to the positive control value generating unit 151, and the - count number is input to the negative control value generating unit 152. In addition, the flicker amount from the flicker detecting unit 33 is input to the update determining units 174 and 184.

[0178] First, the positive control value generating unit 151 will be described.

[0179] The minimum value detection unit 171 accumulates the number of + counts input sequentially in units of detection period DT, detects the minimum value min(DT(x)) of the number of + counts in each detection period DT, and provides the minimum value min(DT(x)) to the DR calculation unit 172 (x=1, 2, 3, etc.). That is, the minimum value detection unit 171 performs the calculation of the above formula (5) for each detection period DT.

[0180] The DR calculation unit 172 calculates the dynamic range DR(i) of each frame phase i of the current detection period DT(x) by subtracting the minimum value min(DT(x)) of the detection period DT(x) provided by the minimum value detection unit 171 from the + count number P_count(i) of each phase frame i of the current detection period DT(x) provided by the event counting unit 32. That is, the DR calculation unit 172 performs the calculation of the above formula (6) for each detection period DT. The calculated dynamic range DR(i) of each frame phase i of the current detection period DT(x) is provided to the comparison unit 173.

[0181] The comparison unit 173 compares the dynamic range DR(i) of each frame phase i of the current detection period DT(x) with the first threshold DR_TH1, and determines whether the dynamic range DR(i) is less than the first threshold DR_TH1. Then, in the case where the dynamic range DR(i) of each frame phase i is less than the first threshold DR_TH1, the comparison unit 173 calculates the control value p_sense(i) in each frame phase i of the next detection period DT(x+1).

[0182] In addition, the comparison unit 173 compares the dynamic range DR(i) of each frame phase i of the current detection period DT(x) with the second threshold DR_TH2, and determines whether the dynamic range DR(i) is greater than the second threshold DR_TH2. Then, when the dynamic range DR(i) of each frame phase i is greater than the second threshold DR_TH2, the control value p_sense(i) in each frame phase i of the next detection period DT(x+1) is calculated.

[0183] That is, the comparison unit 173 calculates the control value p_sense′(i) in each frame phase i of the next detection period DT(x+1) by performing the calculation of the above formulas (7) and (8), and supplies the control value p_sense′(i) to the update determination unit 174 .

[0184] The update determination unit 174 determines whether flicker occurs based on the flicker amount EST_FL supplied from the flicker detection unit 33. For example, in a case where the flicker amount EST_FL supplied from the flicker amount estimation unit 54 is greater than a predetermined flicker determination threshold FL_TH, the sensitivity control unit 34 determines that flicker occurs, and in a case where the flicker amount EST_FL is equal to or less than the flicker determination threshold FL_TH, the sensitivity control unit 34 determines that flicker does not occur. Alternatively, in a case where a state in which the flicker amount EST_FL calculated in units of the detection cycle DT is greater than the flicker determination threshold FL_TH occurs a predetermined number of times or more within a predetermined period, it may be determined that flicker occurs.

[0185] Then, in the case where it is determined that flicker has occurred, the update determination unit 174 performs control to update the control value for controlling the threshold value of the brightness change as the sensitivity parameter for each frame phase. That is, the update determination unit 174 provides the control value p_sense'(i) in each frame phase i of the next detection cycle DT(x+1) provided from the comparison unit 173 as the updated control value p_sense(i) to each detection circuit 22 of the pixel array unit 11.

[0186] The operations of the minimum value detection unit 181, the DR calculation unit 182, the comparison unit 183, and the update determination unit 184 of the negative control value generation unit 152 are similar to the operations of the minimum value detection unit 171, the DR calculation unit 172, the comparison unit 173, and the update determination unit 174 of the positive control value generation unit 151, except that the - count number is used instead of the + count number, and therefore the description thereof is omitted. Fig.13 As shown, the polarity between the control of the detection sensitivity of the +change and the control of the detection sensitivity of the -change is opposite, and therefore, the magnitude relationship of the control values ​​is opposite.

[0187] In the negative control value generating unit 152 , the control value n_sense(i) on the negative change side, that is, the negative side threshold Vrefn(i) in each frame phase i of the next detection cycle DT(x+1) is determined and provided to each detection circuit 22 of the pixel array unit 11 .

[0188] <5. Processing flow of flicker control process>

[0189] Will refer to Fig.15The flowchart in describes the flicker control process performed by the event detection sensor 1. The process starts, for example, when the event detection sensor 1 is instructed to start event detection (imaging).

[0190] First, in step S1, the pixel array unit 11 performs exposure in a predetermined exposure time corresponding to a frame rate, and generates a change image in which information indicating the presence or absence of a brightness change is stored for each pixel. The generated change image is provided to the signal processing circuit 12. The pixel value of each pixel of the change image indicates any of +change, -change, or no change.

[0191] In step S2 , the event data acquisition unit 31 acquires the change image supplied from the pixel array unit 11 , outputs the change image to the outside of the event detection sensor 1 , and supplies the change image to the event counting unit 32 .

[0192] In step S3 , the event counting unit 32 counts the + count number and the − count number with respect to the change image supplied from the event data acquiring unit 31 , and supplies the count result to the flicker detecting unit 33 and the sensitivity control unit 34 .

[0193] In step S4, the flicker detection unit 33 subtracts the -count number from the +count number provided by the self-event counting unit 32, accumulates the multiplication results obtained by multiplying the subtraction result by the function value of the cosine approximation function (cosine function), and accumulates the multiplication results obtained by multiplying the subtraction result by the function value of the sine approximation function (sine function).

[0194] More specifically, the subtractor 51 of the flicker detection unit 33 subtracts the -count number from the +count number, and outputs the subtraction result to the multipliers 71 and 72 of the accumulation unit 53. The multiplier 71 supplies the multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 by the function value of the cosine approximation function generated by the convolution coefficient generation unit 52 to the accumulator 73, and the accumulator 73 accumulates the multiplication result supplied from the multiplier 71. Furthermore, the multiplier 72 supplies the multiplication result obtained by multiplying the subtraction result of the count number supplied from the subtractor 51 by the function value of the sine approximation function generated by the convolution coefficient generation unit 52 to the accumulator 74, and the accumulator 74 accumulates the multiplication result supplied from the multiplier 72.

[0195] In step S5, the flicker detection unit 33 determines whether the enable signal supplied from the convolution coefficient generation unit 52 becomes high. The fact that the enable signal becomes high indicates that a change image of the number of frames corresponding to the detection period DT is input from the pixel array unit 11.

[0196] In the event that it is determined in step S5 that the enable signal is not high, the process returns to step S3 , and the above-described steps S3 to S5 are repeated.

[0197] On the other hand, when it is determined in step S5 that the enable signal is high, the processing proceeds to step S6, the output unit 75 of the flicker detection unit 33 obtains the cumulative value cos_sum of the accumulator 73 and provides the cumulative value cos_sum to the flicker amount estimation unit 54, and the output unit 76 obtains the cumulative value sin_sum of the accumulator 74 and provides the cumulative value sin_sum to the flicker amount estimation unit 54.

[0198] In step S7, the flicker amount estimation unit 54 of the flicker detection unit 33 estimates the flicker amount occurring at the flicker frequency of the detection target by using the respective accumulation results provided from the output units 75 and 76. Specifically, the flicker amount estimation unit 54 calculates the flicker amount EST_FL by the above formula (1) or formula (2). The calculated flicker amount EST_FL is output to the outside of the event detection sensor 1 and is also provided to the sensitivity control unit 34.

[0199] In step S8, the sensitivity control unit 34 determines whether flicker occurs based on the flicker amount EST_FL supplied from the flicker amount estimation unit 54. For example, in a case where the flicker amount EST_FL supplied from the flicker amount estimation unit 54 is greater than a predetermined flicker determination threshold FL_TH, the sensitivity control unit 34 determines that flicker occurs, and in a case where the flicker amount EST_FL is equal to or less than the flicker determination threshold FL_TH, the sensitivity control unit 34 determines that flicker does not occur. Alternatively, in a case where a state in which the flicker amount EST_FL calculated in units of the detection cycle DT is greater than the flicker determination threshold FL_TH occurs a predetermined number of times or more within a predetermined period, it may be determined that flicker occurs.

[0200] In the event that it is determined in step S8 that flicker has not occurred, steps S9 to S11 described later are skipped, and the process returns to step S1 .

[0201] On the other hand, in the case where it is determined in step S8 that flicker has occurred, the processing of the following steps S9 to S11 is performed.

[0202] In step S9, the sensitivity control unit 34 detects the minimum value min(DT(x)) of the count number of multiple change images accumulated in the detection period DT for each of the + count number and the - count number. Specifically, the minimum value detection unit 171 detects the minimum value min(DT(x)) of the + count number of multiple change images accumulated in the detection period DT, and the minimum value detection unit 181 detects the minimum value min(DT(x)) of the - count number of multiple change images accumulated in the detection period DT.

[0203] Next, in step S10 , the sensitivity control unit 34 calculates the control value sense(i) in each frame phase i of the next detection period DT(x+1).

[0204] For example, in the positive control value generation unit 151 for processing the + count number, the DR calculation unit 172 calculates the dynamic range DR(i) of the + count number for each frame phase i of the detection cycle DT(x) by subtracting the minimum value min(DT(x)) of the + count number from the + count number P_count(i) of each phase frame i of the current detection cycle DT(x). The comparison unit 173 compares the dynamic range DR(i) of the + count number for each frame phase i of the current detection cycle DT(x) with the first threshold DR_TH1 and the second threshold DR_TH2, and calculates the control value p_sense(i) in each frame phase i of the next detection cycle DT(x+1).

[0205] Similarly, for the negative control value generating unit 152 that processes the count number, the DR calculating unit 182 and the comparing unit 183 calculate the control value n_sense(i) in each frame phase i of the next detection period DT(x+1).

[0206] Next, in step S11, the update determination units 174 and 184 of the sensitivity control unit 34 provide the control value sense(i) in each frame phase i of the next detection period DT(x+1) provided from the corresponding comparison units 173 and 183 as the updated control value sense(i) to each detection circuit 22 of the pixel array unit 11 at a timing corresponding to each frame phase i of the next detection period DT(x+1). More specifically, the update determination unit 174 provides the control value p_sense(i) on the +change side, that is, the +side threshold value Vrefp, to each detection circuit 22, and the update determination unit 184 provides the control value n_sense(i) on the -change side, that is, the -side threshold value Vrefn, to each detection circuit 22.

[0207] After step S11, the process returns to step S1, and the above-mentioned process is repeated.

[0208] <6. Example of Processing Result of Flicker Control Process>

[0209] Fig.16 An example of a processing result of the flicker control process performed by the event detection sensor 1 is shown.

[0210] exist Fig.16 In the imaging scene indicated by the image 201 of FIG. 1 , event detection by the event detection sensor 1 is performed in an environment using a light source having a power frequency of 50 Hz. The imaging scene shows a person walking from the right side to the left side in the picture in front of the background. The image 201 is an image captured by a general CMOS image sensor.

[0211] Image 202 is frame data (frame image) of a changing image passing through the event detection sensor 1. Image 202 is in a state where the sensitivity of the event detection sensor 1 is high, an event is detected in the entire screen, and an event of a moving person is detected buried in noise.

[0212] Image 203 is frame data (frame image) of a changed image after the sensor sensitivity is reduced to below the sensitivity of image 202 by the flicker control process.

[0213] In image 203, the noise of captured flickers is reduced and only movements larger than flickers (mainly moving people) are detected as events.

[0214] Therefore, according to the flicker control process, the flicker amount (flicker information) can be detected from the changing image output at a constant cycle. Then, in the case where the flicker amount is large, the flicker can be suppressed, and only the priority event (superiorevent) can be extracted. In addition, a dedicated control can be performed independently on each of the + change and the - change at the timing at which each of the + change and the - change may occur.

[0215] <7. Another Configuration Example of Event Detection Sensor>

[0216] Fig.17 : is a block diagram showing a configuration example as another embodiment of the event detection sensor 1 .

[0217] exist Fig.17 In, with Figure 1 The corresponding parts in the figure are represented by the same figure marks, and their descriptions will be appropriately omitted.

[0218] exist Fig.17 In the embodiment, flicker detection units 33-1 to 33-R (R>1) are provided, and a plurality of (R) flicker detection units 33 are provided, which is consistent with Figure 1 The event detection sensor 1 in is different, but the other points are the same Figure 1 The point in the event detection sensor 1 is the same.

[0219] Fig.17 The event detection sensor 1 in FIG. 1 can detect a plurality of flicker frequencies by including a plurality of flicker detection units 33 - 1 to 33 -R. That is, the flicker detection units 33 - 1 to 33 -R have different flicker frequencies set as detection targets.

[0220] For example, when the frequencies of flicker detected by the flicker detection units 33 - 1 and 33 - 2 are set to 100 Hz and 120 Hz in the case of R=2, flicker corresponding to the western region of Japan and the eastern region of Japan can be detected.

[0221] Alternatively, when the frequency of flicker detected by the flicker detection units 33 - 1 to 33 -R is set to 25 Hz, 50 Hz, 100 Hz, 200 Hz, 400 Hz, . . . , etc., only events of arbitrary frequencies can be detected, and frequency analysis including flicker becomes possible.

[0222] <8. Configuration Example of Imaging Device>

[0223] Fig.18 : is a block diagram showing a configuration example of an imaging device including the above-described event detection sensor 1 as an imaging element.

[0224] The imaging device 300 includes an optical unit 311, an imaging element 312, a recording unit 313, and a control unit 314. For example, as the imaging device 300, a camera mounted on an industrial robot, a vehicle-mounted camera, or the like is assumed.

[0225] The optical unit 311 condenses light from the subject and causes the light to enter the imaging element 312. The imaging element 312 photoelectrically converts the incident light incident via the optical unit 311 to generate image data, and supplies the image data to the recording unit 313. As the imaging element 312, a Figure 1 , Fig.17 Event detection sensor 1 in the like.

[0226] The recording unit 313 records and accumulates, in a predetermined recording medium, the image data supplied from the imaging element 312. The control unit 314 controls the imaging element 312. For example, the control unit 314 instructs the imaging element 312 to start and end imaging, and specifies a frame rate at the time of imaging.

[0227] Fig.19 is a transmission diagram showing a schematic configuration example of the imaging element 312 .

[0228] The imaging element 312 has a stacked structure in which a light receiving chip 321 and a detection chip 322 are bonded and stacked. The light receiving chip 321 and the detection chip 322 are electrically connected via a connection portion such as a through hole, a Cu-Cu bond, a bump.

[0229] Fig. 20 3 is a plan view showing a configuration example of the light receiving chip 321 .

[0230] The light receiving chip 321 includes a light receiving unit 341 formed in a chip center portion and one or more through hole arrangement units 342 formed in a peripheral portion outside the light receiving unit 341. Fig. 20 In the example of FIG. 3 , three through-hole arrangement units 342 are arranged at the corners of the chip periphery.

[0231] In the light receiving unit 341, a plurality of shared blocks 343 are arranged in a two-dimensional lattice pattern. In the through-hole arrangement unit 342, through-holes electrically connected to the detection chip 322 are arranged.

[0232] A plurality of logarithmic response units 351 are arranged in each shared block 343. For example, in one shared block 343, four logarithmic response units 351 are arranged in 2 rows × 2 columns. These four logarithmic response units 351 share the circuit on the detection chip 322. The details of the shared circuit will be described later. Note that the number of logarithmic response units 351 in a shared block 343 is not limited to four.

[0233] The logarithmic response unit 351 generates a voltage signal corresponding to the logarithmic value of the photocurrent. A pixel address including a row address and a column address is assigned to each logarithmic response unit 351.

[0234] Fig.21 3 is a plan view showing a configuration example of the detection chip 322 .

[0235] The detection chip 322 includes one or more via arrangement units 361 , an address event detection unit 362 , a row driving circuit 363 , a column driving circuit 364 , and a signal processing circuit 365 .

[0236] The through-hole arrangement unit 361 is provided at a position corresponding to the through-hole arrangement unit 342 of the light receiving chip 321, and is electrically connected to the light receiving chip 321 via a through-hole. Fig.21 In the embodiment, the through hole arrangement unit 361 is arranged at Fig. 20 At positions corresponding to the three through-hole arrangement units 342 in the detection chip 322 , a total of three through-hole arrangement units 361 are formed on the detection chip 322 .

[0237] The address event detection unit 362 detects the presence or absence of an event for each logarithmic response unit 351 of the light receiving chip 321 and generates a detection signal indicating the detection result. The detection signal is generated as ternary (2-bit) information indicating any of +change, -change or no change.

[0238] The row driving circuit 363 selects a predetermined row address of the address event detecting unit 362 , and outputs a detection signal of the selected row address to the signal processing circuit 365 .

[0239] The column driving circuit 364 selects a predetermined column address of the address event detecting unit 362 , and outputs a detection signal of the selected column address to the signal processing circuit 365 .

[0240] The signal processing circuit 365 performs predetermined signal processing on the detection signal output from the address event detection unit 362. For example, the signal processing circuit 365 acquires image data in which the detection signal is a pixel signal. Then, the signal processing circuit 365 performs a process of detecting (estimating) the amount of flicker of a predetermined cycle based on the image data, and controls the address event detection unit 362 to suppress flicker in the event that flicker of a predetermined cycle occurs. Therefore, in the imaging element 312, the processing performed by the signal processing circuit 365 Figure 1 The process is performed by the signal processing circuit 12 in.

[0241] Fig. 22 is a plan view showing details of the address event detection unit 362.

[0242] In the address event detection unit 362, a plurality of detection blocks 371 are arranged in a two-dimensional lattice pattern. The detection block 371 is arranged for each shared block 343 on the light receiving chip 321. That is, when the number of shared blocks 343 on the light receiving chip 321 is N (N is an integer), N detection blocks 371 are arranged in the detection chip 322. Each detection block 371 is electrically connected to the corresponding shared block 343 through a through hole, Cu-Cu bonding, or the like.

[0243] Fig.23 is a block diagram showing a configuration example of one detection block 371 .

[0244] The detection block 371 includes four detection units 381 , a selector 382 , a comparison unit 383 , and a transfer circuit 384 .

[0245] Each of the four detection units 381 includes a logarithmic response unit 351, a buffer 352, and a differentiator 353. The logarithmic response unit 351 generates a voltage signal corresponding to the logarithmic value of the photocurrent and outputs the voltage signal to the buffer 352. The buffer 352 buffers the voltage signal from the logarithmic response unit 351 and outputs the voltage signal to the differentiator 353. By using the buffer 352, the isolation of the noise accompanying the switching operation of the subsequent stage can be ensured, and the driving force for driving the subsequent stage can be improved. Note that the buffer 352 can be omitted. The differentiator 353 outputs the change amount of the voltage signal (the change amount of the brightness change) as the differential signal Sin.

[0246] like Fig. 20 As shown, the logarithmic response unit 351 is also provided in the shared block 343 of the optical receiving chip 321, and is dispersedly arranged in the shared block 343 of the optical receiving chip 321 and the detection unit 381 of the detection block 371. Therefore, the four detection units 381 correspond to the logarithmic response units 351 of 2 rows×2 columns in the shared block 343. In the case of distinguishing each of the four detection units 381, the detection unit is referred to as a detection unit 381-1 to 381-4, and the differential signal Sin output from each of the detection units 381-1 to 381-4 is distinguished as a differential signal Sin1, a differential signal Sin2, a differential signal Sin3, and a differential signal Sin4.

[0247] The selector 382 selects the output of any one of the four detection units 381 according to the selection signals SEL1 to SEL4 from the row driving circuit 363, and supplies the acquired differential signal Sin as the differential signal Sout to the comparison unit 383. Specifically, the selector 382 selects the differential signal Sin1 from the detection unit 381-1 in the case where the selection signal SEL1 is supplied from the row driving circuit 363, selects the differential signal Sin2 from the detection unit 381-2 in the case where the selection signal SEL2 is supplied, selects the differential signal Sin3 from the detection unit 381-3 in the case where the selection signal SEL3 is supplied, and selects the differential signal Sin4 from the detection unit 381-4 in the case where the selection signal SEL4 is supplied, and supplies the differential signals to the comparison unit 383 as the differential signal Sout.

[0248] The comparison unit 383 compares the differential signal Sout supplied from the selector 382 with a predetermined threshold value, and supplies the comparison result to the transmission circuit 384. As the predetermined threshold value to be compared with the differential signal Sout, the above-mentioned + side threshold value Vrefp and - side threshold value Vrefn are supplied from the sensitivity control unit 34 of the signal processing circuit 365 (signal processing circuit 12).

[0249] The comparison unit 383 outputs a detection signal DET+ indicating whether the differential signal Sout representing the amount of change in brightness exceeds the + side threshold Vrefp to the transmission circuit 384, and outputs a detection signal DET- indicating whether the differential signal Sout exceeds the - side threshold Vrefn to the transmission circuit 384.

[0250] The transmission circuit 384 transmits (outputs) the detection signal to the signal processing circuit 365 according to the column driving signal from the column driving circuit 364. Here, the transmission circuit 384 generates the detection signal as ternary (2-bit) information indicating any of +change, -change, and no change, and outputs the detection signal to the signal processing circuit 365. Specifically, the transmission circuit 384 outputs the detection signal indicating +change in the case where the detection signal DET+ indicating that the amount of change in brightness exceeds the +-side threshold value Vrefp is provided from the comparison unit 383, outputs the detection signal indicating -change in the case where the detection signal DET- indicating that the amount of change exceeds the --side threshold value Vrefn is provided, and outputs the detection signal indicating no change in the case where neither the +-side threshold value Vrefp nor the --side threshold value Vrefn is exceeded.

[0251] Fig.24 is a circuit showing a detailed configuration of the detection unit 381 , and particularly shows a detailed configuration example of the logarithmic response unit 351 and the differentiator 353 .

[0252] The logarithmic response unit 351 includes a photodiode (PD) 411 as a photoelectric conversion element and FETs 412 to 414. As FETs 412 and 414, for example, N-type metal oxide semiconductor (NMOS) FETs can be used, and as FET 413, for example, P-type metal oxide semiconductor (PMOS) FETs can be used.

[0253] PD 411 receives incident light, performs photoelectric conversion, and generates photocurrent as an electrical signal and causes it to flow. Logarithmic response unit 351 converts the photocurrent from PD 411 into a voltage (hereinafter also referred to as photovoltage) Vo corresponding to the logarithm of the photocurrent, and outputs the voltage Vo to differentiator 353 via buffer 352.

[0254] The source of FET 412 is connected to the gate of FET 414, and the photocurrent generated by PD 411 flows through the connection point between the source of FET 412 and the gate of FET 414. The drain of FET 412 is connected to the power supply VDD, and its gate is connected to the drain of FET 414.

[0255] The source of the FET 413 is connected to the power supply VDD, and the drain thereof is connected to a connection point between the gate of the FET 412 and the drain of the FET 414. A predetermined bias voltage Vbias is applied to the gate of the FET 413. The source of the FET 414 is grounded.

[0256] The drain of FET 412 is connected to the power supply VDD side and is a source follower. PD 411 is connected to the source of FET 412 as a source follower, and therefore, a photocurrent caused by the charge generated by the photoelectric conversion of PD 411 flows through FET 412 (drain to source). FET 412 operates in a subthreshold region, and a photovoltage Vo corresponding to the logarithm of the photocurrent flowing through FET 412 appears at the gate of FET 412. As described above, in the logarithmic response unit 351, the photocurrent from PD 411 is converted by FET 412 into a photovoltage Vo corresponding to the logarithm of the photocurrent.

[0257] The photovoltage Vo is output from a connection point between the gate of the FET 412 and the drain of the FET 414 to the differentiator 353 via the buffer 352 .

[0258] Regarding the photovoltage Vo from the logarithmic response unit 351 , the differentiator 353 calculates the difference between the current photovoltage and the photovoltage at a timing slightly different from the current timing, and outputs a difference signal Vout corresponding to the difference.

[0259] The differentiator 353 includes a capacitor 431 , an operational amplifier 432 , a capacitor 433 , and a switch 434 .

[0260] One end of the capacitor 431 is connected to the output terminal of the buffer 352, and the other end is connected to the input terminal of the operational amplifier 432. Therefore, the photovoltage Vo is input to the (inverting) input terminal of the operational amplifier 432 via the capacitor 431.

[0261] The output terminal of the operational amplifier 432 is connected to Fig.23 Selector 382.

[0262] One end of the capacitor 433 is connected to the input terminal of the operational amplifier 432 , and the other end is connected to the output terminal of the operational amplifier 432 .

[0263] The switch 434 is connected to the capacitor 433 to turn on / off the connection between both ends of the capacitor 433. The switch 434 turns on / off the connection between both ends of the capacitor 433 by turning on / off according to the row driving signal of the row driving circuit 363.

[0264] The capacitor 433 and the switch 434 constitute a switched capacitor. When the switch 434 that has been turned off is temporarily turned on and turned off again, the capacitor 433 is reset to a state in which the charge is discharged and the charge can be accumulated again.

[0265] The photovoltage Vo of the capacitor 431 on the logarithmic response unit 351 side when the switch 434 is turned on is represented by Vinit, and the capacitance (electrostatic capacitance) of the capacitor 431 is represented by C1. The input terminal of the operational amplifier 432 is virtually grounded, and the charge Qinit accumulated in the capacitor 431 when the switch 434 is turned on is expressed by formula (9).

[0266] Qinit=C1×Vinit…(9)

[0267] Furthermore, when the switch 434 is turned on, both ends of the capacitor 433 are short-circuited, so that the charge accumulated in the capacitor 433 becomes zero.

[0268] Thereafter, when the photovoltage Vo of the capacitor 431 on the logarithmic response unit 351 side when the switch 434 is turned off is represented by Vafter, the charge Qafter accumulated in the capacitor 431 when the switch 434 is turned off is expressed by formula (10).

[0269] Qafter=C1×Vafter…(10)

[0270] When the capacitance of the capacitor 433 is represented by C2, the charge Q2 accumulated in the capacitor 433 is expressed by the formula (11) using the difference signal Vout as the output voltage of the operational amplifier 432.

[0271] Q2=-C2×Vout…(11)

[0272] Before and after the switch 434 is turned off, the total charge amount of the charge of the capacitor 431 and the charge of the capacitor 433 does not change, so that the formula (12) is established.

[0273] Qinit=Qafter+Q2…(12)

[0274] When formulas (9) to (11) are substituted into formula (12), formula (13) is obtained.

[0275] Vout=-(C1 / C2)×(Vafter-Vinit)…(13)

[0276] According to formula (13), the differentiator 353 subtracts the photovoltage Vafter from Vinit, that is, calculates the difference signal Vout corresponding to the difference (Vafter-Vinit) between the photovoltage Vafter and Vinit. According to formula (13), the subtraction gain performed by the differentiator 353 is C1 / C2. Therefore, the differentiator 353 outputs a voltage obtained by multiplying the change in the photovoltage Vo after the capacitor 433 is reset by C1 / C2 as the difference signal Vout. The difference signal Vout is output as the differential signal Sin.

[0277] When the switch 434 is turned on and off by the row driving signal output from the row driving circuit 363 , the differentiator 353 outputs the differential signal Sin.

[0278] Fig.25 Show Fig.23 2 is a configuration example of the comparison unit 383 of the detection block 371 in FIG.

[0279] The comparison unit 383 includes comparators 451 and 452. The +side threshold value Vrefp and the -side threshold value Vrefn are supplied to the comparison unit 383 from the sensitivity control unit 34 of the signal processing circuit 365 (signal processing circuit 12).

[0280] The comparator 451 compares the differential signal Sout from the selector 382 with the + side threshold value Vrefp, and supplies the comparison result as a detection signal DET+ to the transmission circuit 384. The detection signal DET+ indicates whether the amount of change in brightness exceeds the + side threshold value Vrefp.

[0281] The comparator 452 compares the differential signal Sout from the selector 382 with the - side threshold value Vrefn, and supplies the comparison result as a detection signal DET- to the transmission circuit 384. The detection signal DET- indicates whether the amount of change in brightness exceeds the - side threshold value Vrefn.

[0282] Fig.26 363 is a timing chart showing a control example of the row driving circuit 363.

[0283] At timing T0, the row driving circuit 363 selects the first row by the row driving signal L1 and drives the differentiator 353 of the selected row. The capacitor 433 in the differentiator 353 of the first row is initialized by the row driving signal L1. In addition, the row driving circuit 363 provides the selection signal SEL1 to the selector 382 for a certain period of time and selects the upper left detection unit 381 of 2 rows×2 columns in the shared block 343. Therefore, the detection unit 381 in the odd-numbered column of the first row detects the presence or absence of an event.

[0284] Next, at timing T1, the row drive circuit 363 drives the differentiator 353 of the first row again by the row drive signal L1. In addition, the row drive circuit 363 selects the upper right detection unit 381 in the 2 rows×2 columns in the shared block 343 for a certain period of time by the selection signal SEL2. Therefore, the detection unit 381 in the even-numbered column of the first row detects the presence or absence of an event.

[0285] At timing T2, the row drive circuit 363 drives the differentiator 353 in the second row by the row drive signal L2. The capacitor 433 in the differentiator 353 of the second row is initialized by the row drive signal L2. In addition, the row drive circuit 363 selects the lower left detection unit 381 in the 2 rows×2 columns in the shared block 343 for a certain period of time by the selection signal SEL3. Therefore, the detection unit 381 in the odd-numbered column of the second row detects the presence or absence of an event.

[0286] Subsequently, at timing T3, the row drive circuit 363 drives the differentiator 353 in the second row again by the row drive signal L2. In addition, the row drive circuit 363 selects the lower right detection unit 381 in the 2 rows×2 columns in the shared block 343 for a certain period of time by the selection signal SEL4. Therefore, the detection unit 381 in the even-numbered column of the 2nd row detects the presence or absence of an event.

[0287] Similarly, the row driving circuit 363 sequentially selects the row in which the logarithmic response unit 310 is arranged, and drives the selected row by the row driving signal. In addition, each time a row is selected, the row driving circuit 363 sequentially selects each detection unit 381 in the shared block 343 of the selected row by the selection signal SEL. For example, in the case where 2 rows×2 columns of detection units 381 are arranged in the shared block 343, each time a row is selected, the odd columns and even columns in the row are sequentially selected.

[0288] The above-mentioned drive control is sequentially performed on the entire address event detection unit 362 (light receiving unit 341) arranged with the detection unit 381, so that a change image indicating the presence or absence of brightness change is generated at a predetermined frame rate and output to the signal processing circuit 365.

[0289] The signal processing circuit 365 acquires a changing image output at a predetermined frame rate, determines whether flicker of a predetermined period occurs, and if flicker occurs, controls (adjusts) the thresholds for detecting brightness changes, i.e., the + side threshold Vrefp and the - side threshold Vrefn.

[0290] <9. Configuration Example of Electronic Device>

[0291] The above-mentioned event detection sensor 1 can be mounted on an electronic device such as a smartphone, a tablet terminal, a mobile phone, a personal computer, a game console, a television receiver, a wearable terminal, a digital still camera, or a digital video camera.

[0292] Fig. 27 is a block diagram showing a configuration example of a smartphone as an electronic device mounted with an event detection sensor.

[0293] like Fig. 27 As shown, the smartphone 601 is configured by connecting an event detection sensor 602, an imaging device 603, a display 604, a speaker 605, a microphone 606, a communication module 607, a sensor unit 608, a touch panel 609, and a control unit 610 via a bus 611. In addition, the control unit 610 has functions as an application processing unit 621 and an operating system processing unit 622 by executing a program by the CPU.

[0294] Figure 1 The event detection sensor 1 in FIG. 6 is applied as the event detection sensor 602. For example, the event detection sensor 602 is arranged on the front of the smartphone 601, and can detect and output a brightness change of a subject such as the face, hand, or finger of the user of the smartphone 601 as an event. Note that the event detection sensor 602 can be arranged on the rear surface of the smartphone 601.

[0295] The imaging device 603 is arranged in front of the smartphone 601, and performs imaging of the user of the smartphone 601 as a subject to acquire an image in which the user is captured. Note that although not shown, the imaging device 603 may also be arranged on the rear surface of the smartphone 601.

[0296] The display 604 displays an operation screen for the application processing unit 621 and the operating system processing unit 622 to perform processing, an image captured by the imaging device 603, etc. For example, when making a call using the smartphone 601, the speaker 605 and the microphone 606 output the voice of the other party and collect the user's voice.

[0297] The communication module 607 performs network communication via the Internet, a public telephone line network, a wide area communication network for wireless mobile bodies such as a so-called 4G line or 5G line, and a communication network such as a wide area network (WAN) or a local area network (LAN), a short-range wireless communication such as Bluetooth (registered trademark) or near field communication (NFC), etc. The sensor unit 608 senses speed, acceleration, proximity, etc., and the touch panel 609 acquires a touch operation of the operation screen displayed on the display 604 by the user.

[0298] The application processing unit 621 performs a process for providing various services through the smartphone 601. For example, the application processing unit 621 may perform a process of causing the imaging device 603 to perform imaging based on the brightness change provided from the event detection sensor 602 and displaying an image obtained as a result of the imaging on the display 604. In addition, for example, the application processing unit 621 may perform a process of specifying a region of interest when the imaging device 603 performs imaging based on the brightness change provided from the event detection sensor 602.

[0299] The operating system processing unit 622 performs processes for implementing basic functions and operations of the smartphone 601. For example, the operating system processing unit 622 may perform a process of authenticating the user's face and unlocking the smartphone 601 based on the imaging result of the imaging device 603. In addition, the operating system processing unit 622 may perform processes such as recognizing the user's gesture based on the imaging result of the imaging device 603 and inputting various operations according to the gesture.

[0300] In the smartphone 601 configured as described above, when Figure 1 When the event detection sensor 1 in is applied as the event detection sensor 602, for example, a process of detecting movement or state change of a predetermined object or creating and displaying data of a place where a brightness change has occurred can be performed.

[0301] <10. Application examples for mobile objects>

[0302] The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device installed on any type of mobile body such as a car, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, and a robot.

[0303] Fig.28 : is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a moving body control system to which the technology according to an embodiment of the present disclosure can be applied.

[0304] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig.28 In the example depicted in FIG. 1 , the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an inside information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as a functional configuration of the integrated control unit 12050.

[0305] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various types of programs. For example, the drive system control unit 12010 is used as a control device to control: a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine, a drive motor, etc., a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating a braking force of the vehicle, etc.

[0306] The body system control unit 12020 controls the operation of various types of devices provided to the body according to various types of programs. For example, the body system control unit 12020 is used as a control device to control: a keyless entry system, a smart key system, a power window device, or various types of lights such as headlights, reverse lights, brake lights, turn lights, fog lights, etc. In this case, a radio wave transmitted from a mobile device may be input to the body system control unit 12020 as a substitute for a key or a signal of various types of switches. The body system control unit 12020 receives these input radio waves or signals to control the door lock device, power window device, lights, etc. of the vehicle.

[0307] The vehicle exterior information detection unit 12030 detects information related to the exterior of the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to an imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to image the exterior of the vehicle and receives the imaged image. Based on the received image, the vehicle exterior information detection unit 12030 may perform a process of detecting an object (e.g., a person, vehicle, obstacle, sign, symbol, etc. on the road), or perform a process of detecting the distance to the object.

[0308] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0309] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. The in-vehicle information detection unit 12040 is connected to, for example, a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that images the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or the driver's concentration level, or can determine whether the driver is dozing off.

[0310] The microcomputer 12051 can calculate a control target value for a driving force generating device, a steering mechanism, or a braking device based on information about the interior or exterior of the vehicle obtained by the exterior information detecting unit 12030 or the interior information detecting unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS), including: collision avoidance or impact buffering for the vehicle, following driving based on vehicle-to-vehicle distance, vehicle speed maintenance driving, vehicle collision alarm, vehicle lane deviation alarm, etc.

[0311] In addition, the microcomputer 12051 can control the driving force generating device, steering mechanism, braking device, etc. based on the information about the outside or inside of the vehicle obtained by the outside information detection unit 12030 or the inside information detection unit 12040, thereby performing collaborative control intended for autonomous driving, etc., where the autonomous driving enables the vehicle to travel automatically without relying on the driver's operation.

[0312] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information about the exterior of the vehicle acquired by the exterior information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare by controlling the headlights to change them from high beam to low beam based on the position of a preceding vehicle or an oncoming vehicle detected by the exterior information detection unit 12030, for example.

[0313] The sound / image output unit 12052 sends an output signal of at least one of sound and image to an output device, which can visually or auditorily notify information to the passengers of the vehicle or the outside of the vehicle. Fig.28 In the example of FIG. 1 , an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are shown as output devices. The display portion 12062 may include, for example, at least one of an in-vehicle display and a head-up display.

[0314] Fig.29 The figure depicts an example of the installation position of the imaging unit 12031.

[0315] exist Fig.29 In the figure, vehicle 12100 includes imaging units 12101, 12102, 12103, 12104 and 12105 as imaging unit 12031.

[0316] For example, the imaging units 12101, 12102, 12103, 12104, and 12105 may be arranged at the front nose, side mirrors, rear bumper, and rear door of the vehicle 12100, and at the upper portion of the windshield inside the vehicle. The imaging unit 12101 arranged at the front nose and the imaging unit 12105 arranged at the upper portion of the windshield inside the vehicle mainly obtain images in front of the vehicle 12100. The imaging units 12102 and 12103 arranged at the side mirrors mainly obtain images of the sides of the vehicle 12100. The imaging unit 12104 arranged at the rear bumper or rear door mainly obtains images of the rear of the vehicle 12100. The front images obtained by the imaging units 12101 and 12105 are mainly used to detect front vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0317] Notice, Fig.29 An example of the imaging range of the imaging units 12101 to 12104 is shown. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided at the front nose. Imaging ranges 12112 and 12113 respectively indicate the imaging ranges of the imaging units 12102 and 12103 provided at the side mirrors. Imaging range 12114 indicates the imaging range of the imaging unit 12104 provided at the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 12100 observed from above is obtained by superimposing image data imaged by the imaging units 12101 to 12104.

[0318] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0319] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and thereby extract the nearest three-dimensional object as the leading vehicle, which is particularly present on the driving path of the vehicle 12100 and is traveling at a predetermined speed (e.g., equal to or greater than 0 km / h) in substantially the same direction as the vehicle 12100. In addition, the microcomputer 12051 can pre-set the following distance to be maintained from the leading vehicle, and perform automatic braking control (including follow-up parking control), automatic acceleration control (including follow-up start control), etc. Therefore, it is possible to perform cooperative control intended for autonomous driving, which allows the vehicle to travel automatically without relying on the driver's operation, etc.

[0320] For example, the microcomputer 12051 can classify the three-dimensional object data about the three-dimensional object into three-dimensional object data of two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes whether the obstacles around the vehicle 12100 are obstacles that can be visually identified by the driver of the vehicle 12100, or obstacles that are difficult for the driver of the vehicle 12100 to visually identify. Then, the microcomputer 12051 determines the collision risk, which indicates the risk of collision with each obstacle. In the case where the collision risk is equal to or higher than the set value and therefore there is a possibility of collision, the microcomputer 12051 outputs an alarm to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or evasive steering via the drive system control unit 12010. Thus, the microcomputer 12051 can assist driving to avoid collisions.

[0321] At least one of the imaging units 12101 to 12104 may be an infrared camera device that detects infrared rays. For example, the microcomputer 12051 may identify a pedestrian by determining whether there is a pedestrian in the imaging images of the imaging units 12101 to 12104. This pedestrian recognition is performed, for example, by extracting characteristic points in the imaging images of the imaging units 12101 to 12104 as infrared cameras, and determining whether it is a pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the imaging images of the imaging units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the identified pedestrian. The sound / image output unit 12052 may also control the display unit 12062 so that it displays an icon representing a pedestrian at a desired position, etc.

[0322] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the vehicle exterior information detection unit 12030 and the vehicle interior information detection unit 12040 in the above configuration. Specifically, when the event detection sensor 1 and the imaging device 300 are installed as the vehicle exterior information detection unit 12030 and the vehicle interior information detection unit 12040, a process of detecting a driver's operation or detecting a change in a vehicle exterior condition and reflecting the change in vehicle control can be performed.

[0323] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present technology.

[0324] For example, a mode combining all or some of the above-described multiple configuration examples may be adopted.

[0325] Furthermore, each step described in the above flowchart may be performed by one device or may be shared and performed by a plurality of devices.

[0326] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in one step may be executed by one device or may be shared and executed by a plurality of devices.

[0327] Note that the effects described in this specification are merely examples and are not restrictive, and effects other than those described in this specification may be provided.

[0328] Note that the present technology can have the following configurations. (1)

[0330] A signal processing device, comprising:

[0331] a counting unit that counts, in an image output from the light receiving unit at a predetermined frame rate and indicating a brightness change, a first count number that is a count number of pixels in which a first brightness change in a positive direction is detected, and a second count number that is a count number of pixels in which a second brightness change in a negative direction is detected;

[0332] a coefficient generating unit that generates a coefficient corresponding to a time when the brightness change is detected; and

[0333] An accumulation unit accumulates a multiplication result of the count number of the pixels and the coefficient. (2)

[0335] The signal processing device according to (1), further comprising:

[0336] a subtraction unit that subtracts the second count number from the first count number, wherein

[0337] The accumulation unit accumulates the subtraction result of the subtraction unit and the multiplication result of the coefficient. (3)

[0339] The signal processing device according to (1) or (2), wherein:

[0340] The coefficient generation unit generates a coefficient corresponding to the time based on the predetermined frame rate. (4)

[0342] The signal processing device according to any one of (1) to (3), wherein:

[0343] The accumulation unit accumulates the multiplication result by an integral multiple of the number of frames corresponding to the cycle to be detected. (5)

[0345] The signal processing device according to any one of (1) to (4), wherein:

[0346] The coefficient generation unit generates values ​​of a sine function and a cosine function corresponding to the time as the coefficients. (6)

[0348] The signal processing device according to (5), wherein:

[0349] The accumulation unit multiplies each of the values ​​of the sine function and the cosine function by the count number of the pixels to calculate the multiplication result. (7)

[0351] The signal processing device according to (5) or (6), wherein:

[0352] The coefficient generation unit generates values ​​of the sine function and the cosine function corresponding to a period to be detected as the coefficients. (8)

[0354] The signal processing device according to any one of (5) to (7), wherein:

[0355] The coefficient generation unit generates, as the coefficients, values ​​of a sine approximation function and a cosine approximation function obtained by approximating the sine function and the cosine function. (9)

[0357] The signal processing device according to (8), wherein:

[0358] The sine approximation function and the cosine approximation function are functions obtained by approximating the sine function and the cosine function to signals taking two values ​​of 1 and -1. (10)

[0360] The signal processing device according to (9), wherein:

[0361] The coefficient generation unit outputs 1 or -1 based on a table that associates 1 or -1 with the time. (11)

[0363] The signal processing device according to any one of (1) to (10), further comprising:

[0364] A flicker amount estimation unit estimates a flicker amount in which the brightness change occurs at a specific frequency based on the integration result of the integration unit. (12)

[0366] The signal processing device according to (11), further comprising:

[0367] A control unit controls a sensitivity parameter of the light receiving unit based on an estimation result of the flicker amount estimation unit. (13)

[0369] The signal processing device according to (12), wherein:

[0370] The control unit controls the first brightness variation and the second brightness variation in sensitivity parameters of the light receiving unit, respectively. (14)

[0372] The signal processing device according to (12) or (13), wherein:

[0373] The control unit controls a sensitivity parameter of the light receiving unit for each phase of a cycle to be detected. (15)

[0375] A signal processing method performed by a signal processing device, the method comprising:

[0376] In an image output from a light receiving unit at a predetermined frame rate and indicating a brightness change, counting a first count number which is a count number of pixels in which a first brightness change in a positive direction is detected and a second count number which is a count number of pixels in which a second brightness change in a negative direction is detected;

[0377] generating a coefficient corresponding to a time at which the brightness change is detected; and

[0378] The multiplication result of the count number of the pixels and the coefficient is accumulated. (16)

[0380] A detection sensor, comprising:

[0381] a light receiving unit in which pixels that perform photoelectric conversion of incident light and generate electric signals are arranged in a lattice pattern;

[0382] a counting unit that counts, in an image output from the light receiving unit at a predetermined frame rate and indicating a brightness change, a first count number that is a count number of pixels in which a first brightness change in a positive direction is detected, and a second count number that is a count number of pixels in which a second brightness change in a negative direction is detected;

[0383] a coefficient generating unit that generates a coefficient corresponding to a time when the brightness change is detected; and

[0384] An accumulation unit accumulates a multiplication result of the count number of the pixels and the coefficient.

[0385] Reference numerals list

[0386] 1 event detection sensor, 11 pixel array unit, 12 signal processing circuit, 21 pixel 22 detection circuit, 31 event data acquisition unit, 32 event counting unit, 33 flicker detection unit, 34 sensitivity control unit, 51 subtractor, 52 convolution coefficient generation unit, 53 accumulation unit, 54 flicker amount estimation unit, 71, 72 multipliers, 73, 74 accumulators, 75, 76 output unit, 151 positive control value generation unit, 152 negative control value generation unit, 171 minimum value detection unit, 172 DR calculation unit, 173 comparison unit, 174 update determination unit, 181 minimum value detection unit, 182 DR calculation unit, 183 comparison unit, 184 update determination unit, 300 imaging device, 312 imaging element, 601 smart phone, 602 event detection sensor, 603 imaging device.

Claims

1. A signal processing device, include: a counting unit that counts, in an image output from the light receiving unit at a predetermined frame rate and indicating a brightness change, a first count number that is a count number of pixels in which a first brightness change in a positive direction is detected, and a second count number that is a count number of pixels in which a second brightness change in a negative direction is detected; a coefficient generating unit which generates a coefficient corresponding to a time when the brightness change is detected; as well as An accumulation unit accumulates a multiplication result of the count number of the pixels and the coefficient.

2. The signal processing device according to claim 1, further comprising: include: a subtraction unit that subtracts the second count number from the first count number, wherein The accumulation unit accumulates the subtraction result of the subtraction unit and the multiplication result of the coefficient.

3. The signal processing device according to claim 1, in, The coefficient generation unit generates a coefficient corresponding to the time based on the predetermined frame rate.

4. The signal processing device according to claim 1, in, The accumulation unit accumulates the multiplication result by an integral multiple of the number of frames corresponding to the cycle to be detected.

5. The signal processing device according to claim 1, in, The coefficient generation unit generates values ​​of a sine function and a cosine function corresponding to the time as the coefficients.

6. The signal processing device according to claim 5, in, The accumulation unit multiplies each of the values ​​of the sine function and the cosine function by the count number of the pixels to calculate the multiplication result.

7. The signal processing device according to claim 5, in, The coefficient generation unit generates values ​​of the sine function and the cosine function corresponding to a period to be detected as the coefficients.

8. The signal processing device according to claim 5, in, The coefficient generation unit generates, as the coefficients, values ​​of a sine approximation function and a cosine approximation function obtained by approximating the sine function and the cosine function.

9. The signal processing device according to claim 8, in, The sine approximation function and the cosine approximation function are functions obtained by approximating the sine function and the cosine function to signals taking two values ​​of 1 and -1.

10. The signal processing device according to claim 9, in, The coefficient generation unit outputs 1 or -1 based on a table that associates 1 or -1 with the time.

11. The signal processing device according to claim 1, further comprising: include: A flicker amount estimation unit estimates the flicker amount of the brightness change occurring at a specific frequency based on the integration result of the integration unit.

12. The signal processing device according to claim 11, further comprising: include: A control unit controls a sensitivity parameter of the light receiving unit based on an estimation result of the flicker amount estimation unit.

13. The signal processing device according to claim 12, in, The control unit controls the first brightness variation and the second brightness variation in sensitivity parameters of the light receiving unit, respectively.

14. The signal processing device according to claim 12, in, The control unit controls a sensitivity parameter of the light receiving unit for each phase of a cycle to be detected.

15. A signal processing method performed by a signal processing device, the method include: In an image output from a light receiving unit at a predetermined frame rate and indicating a brightness change, counting a first count number which is a count number of pixels in which a first brightness change in a positive direction is detected and a second count number which is a count number of pixels in which a second brightness change in a negative direction is detected; generating a coefficient corresponding to a time at which the brightness change is detected; as well as The multiplication result of the count number of the pixels and the coefficient is accumulated.

16. A detection sensor, include: a light receiving unit in which pixels that perform photoelectric conversion of incident light and generate electric signals are arranged in a lattice pattern; a counting unit that counts, in an image output from the light receiving unit at a predetermined frame rate and indicating a brightness change, a first count number that is a count number of pixels in which a first brightness change in a positive direction is detected, and a second count number that is a count number of pixels in which a second brightness change in a negative direction is detected; a coefficient generating unit which generates a coefficient corresponding to a time when the brightness change is detected; as well as An accumulation unit accumulates a multiplication result of the count number of the pixels and the coefficient.

Images