Imaging system and imaging method

Abstract

Suppresses events caused by light source flicker. [Solution] An imaging system is provided, comprising: a signal generation unit that generates a time-series event signal in which the first event signal, which shows in chronological order the number of times the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and the second event signal, which shows in chronological order the number of times the absolute value of the negative change amount of the pixel signal exceeds a predetermined threshold, are reversed in sign at the baseline; and a determination unit that determines whether or not the event signal is caused by flicker based on a value relating to the phase in the multiplication value of the event signal and a flicker detection signal having the same frequency as the frequency to be detected.

Description

[Technical Field]

[0001] This disclosure relates to an imaging system and an imaging method. [Background technology]

[0002] In recent years, development has been progressing on image sensors that detect changes in brightness for each pixel as events in real time (hereinafter referred to as EVS (Event-Based Vision Sensor)). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-093778 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, using it in an environment with a light source that has a periodic emission may cause events to occur due to the flicker of the light source.

[0005] Therefore, this disclosure provides an imaging system and an imaging method capable of suppressing events caused by light source flicker. [Means for solving the problem]

[0006] To solve the above problems, the present disclosure provides a signal generation unit that generates a time-series event signal in which the positive and negative values ​​of a first event signal, which indicates in chronological order the number of times the absolute value of the positive change in the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and the negative change in the pixel signal, which indicates in chronological order the number of times the absolute value of the negative change in the pixel signal exceeds a predetermined threshold, are reversed in sign at the baseline. A determination unit determines whether the event signal is caused by flicker based on the phase value in the convolution calculation of the event signal and a flicker detection signal having the same frequency as the flicker frequency to be detected. An imaging system is provided that includes the following features.

[0007] The system further includes a flicker intensity calculation unit that performs a convolution operation on the aforementioned time-series event signal, which consists of a cosine wave and a sine wave having the same frequency as the flicker frequency to be detected, uses the convolution result of the cosine wave as the real part of the amplitude component, generates the convolution value of the sine wave as the imaginary part of the amplitude component, and generates an absolute value by adding the real and imaginary parts of the amplitude component, respectively. The determination unit may determine that the event signal is caused by flicker if the absolute value obtained by adding the real and imaginary parts of the amplitude component increases monotonically with increasing frame count.

[0008] The flicker frequency to be detected may also be the frequency of the light source that generates the flicker.

[0009] The determination unit may determine whether the event signal is caused by flicker if the absolute value over a predetermined number of frames exceeds a predetermined value.

[0010] The system further includes an evaluation processing unit that generates a second absolute value that suppresses random fluctuations in the aforementioned absolute value, The determination unit may determine that the event signal is caused by flicker if the second absolute value increases monotonically in accordance with the increase in the second absolute value of the number of frames.

[0011] The evaluation processing unit may generate the moving average of the absolute values ​​as the second absolute value with the random fluctuations suppressed.

[0012] The evaluation processing unit increases the evaluation value indicating stability when the moving average of the absolute value is larger on the side with the increased number of frames, and decreases the evaluation value when the moving average of the adjacent frames is smaller. The determination unit may determine that the event signal is caused by flicker if the evaluation value exceeds a predetermined value.

[0013] The absolute value may also be the square root of the sum of the squares of the real-part sums obtained by adding the real parts of each of the amplitude components and the squares of the imaginary-part sums obtained by adding the imaginary parts of each of the amplitude components.

[0014] Multiple photoelectric conversion elements, each converting incident light into photoelectric signals to generate electrical signals, A plurality of detection units generate event information, each corresponding to one of the plurality of photoelectric conversion elements, which determines the polarity as positive when the absolute value of the positive change in the voltage of the pixel signal corresponding to the electrical signal generated by each of the plurality of photoelectric conversion elements exceeds the predetermined threshold, and determines the polarity as negative when the absolute value of the negative change in the voltage of the pixel signal exceeds the predetermined threshold, coordinate information based on the photoelectric conversion element that exceeded the threshold, and time information at the time when the threshold was exceeded. An event signal generation unit generates the first event signal and the second event signal using at least one of the event information generated by the plurality of detection units, It may also be provided with the following:

[0015] The flicker intensity calculation unit may perform convolution integration on the event signal over a predetermined period using sine waves and cosine waves with sequentially increasing rotational phases in predetermined time units, thereby generating the real and imaginary parts of the amplitude components at the different rotational phases.

[0016] The event signal generation unit may generate the first event signal and the second event signal using event information generated by one or more detection units.

[0017] The system may further include an analog-to-digital converter that converts pixel signals corresponding to the electrical signals generated by each of the aforementioned photoelectric conversion elements into digital image signals.

[0018] The system may further include a region selection unit that, upon receiving the event information, causes the analog-to-digital converter to selectively output an electrical signal generated by the photoelectric conversion element, which is identified by the coordinate information contained in the event information.

[0019] The determination unit may further include a control unit that controls the supply of event information from the detection unit used to generate the event signal to the region selection unit when the determination unit determines that the event signal is caused by flicker.

[0020] The imaging data based on the digital image signal further includes a recognition processing unit that performs recognition processing on a region identified by coordinate information included in the event information, If the determination unit determines that the event signal is caused by flicker, it may perform the recognition process without using the event information supplied from the detection unit used to generate the event signal.

[0021] The determination unit may, with respect to the time-series event signal, determine whether the event signal is caused by flicker based on the phase of the amplitude component, where the convolution result of the flicker detection signal, which is composed of a cosine wave and a sine wave, is the real part of the amplitude component, and the convolution result of the event signal and the sine wave is the imaginary part of the amplitude component.

[0022] The determination unit may determine that the event signal is caused by flicker if the phase of each of the amplitude components is within a predetermined range.

[0023] The determination unit may determine that the phase of each amplitude component is within a predetermined range if the magnitude of the complex vector composed of the real and imaginary parts of the amplitude component increases monotonically in accordance with the increase in the number of frames.

[0024] To solve the above problems, the present disclosure provides a signal generation process that generates a time-series event signal in which a first event signal shows, in a time-series manner, the number of times, in a time-series manner, the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and a second event signal shows, in a time-series manner, the number of times, in a time-series manner, the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold, and the two event signals are inverted in positive and negative signs at the baseline. A determination step in which a determination unit determines whether or not the event signal is caused by flicker, based on the phase value in the multiplication value of the event signal and a flicker detection signal having the same frequency as the frequency to be detected, An imaging method comprising the above is provided. [Brief explanation of the drawing]

[0025] [Figure 1] A block diagram showing an example of the system configuration of an imaging system to which the technology described herein is applied. [Figure 2A] A schematic diagram showing a streetlamp, the subject of the captured image, and a person. [Figure 2B] This figure illustrates multiple imaging modes in the imaging device according to this embodiment. [Figure 3A] A block diagram showing an example of the configuration of an imaging device. [Figure 3B] A block diagram showing an example of the configuration of the pixel array section. [Figure 4] A circuit diagram showing an example of a pixel circuit configuration. [Figure 5] A block diagram showing a first configuration example of the address event detection unit. [Figure 6] A diagram illustrating an example configuration for flicker detection. [Figure 7A] A diagram showing frame images in chronological order. [Figure 7B]A diagram illustrating an example of event signal generation. [Figure 8] A diagram showing the characteristics of the median flicker intensity. [Figure 9] This figure shows the characteristics of the median flicker intensity in relation to the number of events. [Figure 10] A diagram showing the characteristics of the median flicker intensity. [Figure 11] This figure shows examples of the characteristics of low-intensity flicker and noisy pseudo-flicker in comparative examples. [Figure 12] This figure shows an example of smoothing processing for increasing frame numbers in the midpoint of flicker intensity. [Figure 13] A diagram showing the change in evaluation values. [Figure 14] A diagram illustrating another configuration for flicker detection. [Figure 15] A diagram illustrating yet another configuration example related to flicker detection. [Figure 16] A flowchart illustrating an example of flicker detection processing. [Figure 17] A flowchart illustrating another example of flicker detection. [Figure 18] A flowchart showing an example of imaging control by the control unit 13. [Modes for carrying out the invention]

[0026] The following description will focus on the main components of the imaging system and imaging method, with reference to the drawings. While the following description will primarily focus on the main components of the imaging system and imaging method, the imaging device and imaging method may contain components and functions not shown or described. The following description does not exclude any components or functions not shown or described.

[0027] Figure 1 is a block diagram showing an example of the system configuration of an imaging system to which the technology described herein is applied. As shown in Figure 1, the imaging system 1 to which the technology described herein is applied comprises an imaging lens 11, an imaging device 10, and a processing device 20. This imaging system 1 is an example of the electronic equipment described herein, and examples of such electronic equipment include camera systems mounted on industrial robots and in-vehicle camera systems.

[0028] Furthermore, the imaging device 10 has a first signal processing unit 25. The processing unit 20 is, for example, an application processor (AP), and performs processing on the electronic image received from the imaging device 10 according to its intended use (application). The processing unit 20 includes a recording unit 12, a control unit 13, a second signal processing unit 26, and a CPU 55.

[0029] In the imaging system 1 with the above configuration, the imaging lens 11 captures incident light from the subject and forms an image on the imaging surface of the imaging device 10. The imaging device 10 converts the incident light captured by the imaging lens 11 into photoelectric data using a photoelectric conversion element at the pixel level to acquire imaging data.

[0030] The first signal processing unit 25 performs predetermined signal processing, such as image recognition processing, on the captured image data and outputs the processing results to the recording unit 12. The first signal processing unit 25 also performs flicker detection processing together with the second signal processing unit 26. Details of the detection processing will be described later.

[0031] The recording unit 12 stores data supplied from the imaging device 10 via the signal line 14. The control unit 13, for example, is configured by a microcomputer and controls the imaging operation of the imaging device 10. The CPU 55 performs various information processing based on event signals output from the imaging device 10. The CPU 55 may be installed inside the imaging device 10.

[0032] [Imaging device] Here, we will explain examples of imaging modes of the imaging device 10 using Figures 2A and 2B. Figure 2A schematically shows the streetlamp O2 and the person O4, which are the targets of imaging in the captured image g0. In the following explanation, for the sake of simplicity, we may explain using an example in which the figure is divided into eight regions sa1 to sa8, but we are not limited to this. Region sa4, which has a circular region a4, is imaged with respect to the lamp part of the streetlamp O2 that generates an event caused by flicker. On the other hand, region sa6, which has a circular region a6, is imaged with respect to the person O4. The person O generates an event due to movement.

[0033] Figure 2B is a diagram illustrating multiple imaging modes in the imaging device 10 according to this embodiment. The imaging device 10 according to this embodiment has multiple imaging modes. Images E0 and E2 are diagrams in which event information is represented as a two-dimensional image.

[0034] In the first mode, after acquiring the initial image g0, it is possible to read out images of row ranges L30 and L32, which include the event regions A20 and A40 in image E0. The first mode is a mode in which the flicker occurrence region A20 can be determined based on the read-out event information.

[0035] The second mode, as an example of an embodiment shown in image E2, can suppress event information in the flicker-causing region A20. In other words, the second mode is a mode that can read out image information from the range excluding the flicker-causing region A20. For example, in the second mode, the image of the row range L32, which includes the event region A40 that occurred in the range excluding the flicker-causing region A20, is read out. As a result, for example, the image of the row range L32 is updated to the captured image. Therefore, the update time of the captured image is shortened and power consumption can be suppressed.

[0036] [Imaging device] Figure 3A is a block diagram showing an example of the configuration of an imaging device. As shown in Figure 3A, the imaging device 10 according to the first configuration example of the imaging device of this disclosure is an asynchronous imaging device called an EVS, and comprises a pixel array unit 21, a drive unit 22, a readout area selection unit 23, a column processing unit 24, and a first signal processing unit 25.

[0037] In the imaging device 10 with the above configuration, the pixel array section 21 has multiple pixels 30 arranged in a matrix in a two-dimensional arrangement. A vertical signal line VSL, described later, is wired to each pixel row in this matrix-like pixel arrangement.

[0038] Each of the multiple pixels 30 generates an analog signal of voltage corresponding to an electrical signal as a pixel signal. Furthermore, each of the multiple pixels 30 detects the presence or absence of an address event based on whether the absolute value of the positive change in the pixel signal exceeds a first threshold, or whether the absolute value of the negative change in the pixel signal exceeds a second threshold. Note that address events are sometimes simply referred to as events.

[0039] In the first mode, when an event occurs in pixel 30, event information (x, y, t, p) is generated, which includes the coordinates (x, y) of pixel 30, a timestamp t indicating the time the event occurred, and the polarity p of the event. The polarity p is assigned, for example, 1 if it exceeds a first threshold, and -1 if it exceeds a second threshold. In the first mode, pixel 30 outputs the event information (x, y, t, p) to the first signal processing unit 25 and the second signal processing unit 26.

[0040] In the first mode, when an event occurs, each pixel 30 outputs event information (x, y, t, p) to the first signal processing unit 25, the second signal processing unit 26, and the readout area selection unit 23. The readout area selection unit 23 drives the pixel 30 corresponding to the coordinates (x, y) of the event information to the drive unit 22. As a result, the drive unit 22 drives the pixel 30 at coordinates (x, y) and outputs the pixel signal to the column processing unit 24. The drive unit 22 and the column processing unit 24 are circuit units for acquiring grayscale information. Therefore, in the first mode, which acquires only event information, it is possible to have a configuration without the drive unit 22 and the column processing unit 24. In the first mode, the pixel area caused by flicker is determined and stored as the coordinates (x, y) of the pixel 30.

[0041] In the second mode, flicker suppression processing is performed on the pixel 30 or the pixel signal of pixel 30 that was detected as flicker in the first mode. This makes it possible to suppress flicker caused by the light source. As mentioned above, for example, it is possible to stop the transmission or reading of event information (x, y, t, p) from the coordinates (x, y) of the pixel region caused by flicker. Details of the flicker suppression processing will be described later.

[0042] The column processing unit 24 consists of, for example, an analog-to-digital converter. The column processing unit 24 performs a process to convert the analog pixel signal output from the pixels 30 of each pixel row in the pixel array unit 21 into a digital signal. The column processing unit 24 then supplies the digital signal after analog-to-digital conversion to the first signal processing unit 25.

[0043] In the second mode, the first signal processing unit 25 performs predetermined signal processing, such as CDS (Correlated Double Sampling) processing and image recognition processing, on the digital signal supplied from the column processing unit 24. The first signal processing unit 25 can also supply data indicating the processing result and event information (x, y, t, p) to the recording unit 12 (see Figure 1) via the signal line 14.

[0044] [Example of pixel array configuration] Figure 3B is a block diagram showing an example of the configuration of the pixel array unit 21. As shown in Figure 3B, in the pixel array unit 21, which is made up of a plurality of pixels 30 arranged in a matrix in a two-dimensional arrangement, each of the plurality of pixels 30 has a configuration in which a light receiving unit 31, an image signal generation unit 32, and an address event detection unit 33 are included.

[0045] In the pixel 30 with the above configuration, the light receiving unit 31 converts incident light into electrical signals. The light receiving unit 31 then supplies the generated electrical signals to either the image signal generation unit 32 or the address event detection unit 33, in accordance with the control of the drive unit 22 (see Figure 3A).

[0046] The image signal generation unit 32 generates an image signal SIG as a voltage signal corresponding to the electrical signal supplied from the light receiving unit 31, and supplies this generated image signal SIG to the column processing unit 24 (see Figure 3A) via the vertical signal line VSL.

[0047] The address event detection unit 33 detects the presence or absence of an address event based on whether the amount of change in the pixel signal from each of the light receiving units 31 exceeds a predetermined threshold. As described above, when an event occurs, event information (x, y, t, p) is generated and stored.

[0048] When an address event occurs, the address event detection unit 33 sends a request to the read area selection unit 23 (see Figure 3A) requesting the transmission of an address event detection signal. Upon receiving a response to the request from the read area selection unit 23, the address event detection unit 33 sends it to the drive unit 22, the first signal processing unit 25, and the second signal processing unit 26.

[0049] [Example of pixel circuit configuration] Figure 4 is a circuit diagram showing an example of the circuit configuration of a pixel 30. As described above, each of the multiple pixels 30 has a configuration that includes a light receiving unit 31, an image signal generation unit 32, and an address event detection unit 33.

[0050] In the pixel 30 with the above configuration, the light-receiving section 31 has a configuration comprising a light-receiving element (photoelectric conversion element) 311, a transfer transistor 312, and an OFG (Over Flow Gate) transistor 313. For example, N-type MOS (Metal Oxide Semiconductor) transistors are used as the transfer transistor 312 and the OFG transistor 313. The transfer transistor 312 and the OFG transistor 313 are connected in series with each other.

[0051] The light-receiving element 311 is connected between the common connection node N1 between the transfer transistor 312 and the OFG transistor 313 and ground, and converts incident light into electricity to generate an amount of charge corresponding to the amount of incident light.

[0052] A transfer signal TRG is supplied to the gate electrode of the transfer transistor 312 from the drive unit 22 shown in Figure 3A. In response to the transfer signal TRG, the transfer transistor 312 supplies the charge converted photoelectrically by the photodetector 311 to the image signal generation unit 32.

[0053] The gate electrode of the OFG transistor 313 is supplied with a control signal OFG from the drive unit 22. In response to the control signal OFG, the OFG transistor 313 supplies an electrical signal generated by the photodetector 311 to the address event detection unit 33. The electrical signal supplied to the address event detection unit 33 is an electrical signal consisting of electric charge.

[0054] The image signal generation unit 32 has a configuration that includes a reset transistor 321, an amplification transistor 322, a selection transistor 323, and a floating diffusion layer 324. For example, N-type MOS transistors are used as the reset transistor 321, the amplification transistor 322, and the selection transistor 323.

[0055] The image signal generation unit 32 is supplied with charge converted photoelectrically by the photodetector 311 via the transfer transistor 312 from the photodetector 31. The charge supplied from the photodetector 31 is stored in the floating diffusion layer 324. The floating diffusion layer 324 generates a voltage signal with a voltage value corresponding to the amount of charge stored. In other words, the floating diffusion layer 324 converts charge into voltage.

[0056] The reset transistor 321 is connected between the power supply line VDD (power supply voltage) and the floating diffusion layer 324. The gate electrode of the reset transistor 321 is supplied with a reset signal RST from the drive unit 22. In response to the reset signal RST, the reset transistor 321 initializes (resets) the charge amount of the floating diffusion layer 324.

[0057] The amplifying transistor 322 is connected in series with the selection transistor 323 between the power supply line VDD (power supply voltage) and the vertical signal line VSL (vertical signal line). The amplifying transistor 322 amplifies the voltage signal that has been converted to a charge-voltage in the floating diffusion layer 324.

[0058] A selection signal SEL is supplied to the gate electrode of the selection transistor 323 from the drive unit 22. In response to the selection signal SEL, the selection transistor 323 outputs the voltage signal amplified by the amplification transistor 322 as an image signal SIG to the column processing unit 24 (see Figure 3A) via the vertical signal line VSL.

[0059] (Mode 1) In the first mode, when the control unit 13 shown in Figure 1 instructs the drive unit 22 to start detecting an address event, it drives the OFG transistor 313 of the light receiving unit 31 by supplying a control signal OFG to the OFG transistor 313, thereby supplying an electrical signal to the address event detection unit 33.

[0060] When the drive unit 22 detects an address event in a pixel 30, it turns off the OFG transistor 313 of that pixel 30, stopping the supply of electrical signals to the address event detection unit 33. Next, the drive unit 22 drives the transfer transistor 312 by supplying a transfer signal TRG to it, in accordance with the control of the read area selection unit 23, thereby transferring the charge photoelectrically converted by the photodetector 311 to the floating diffusion layer 324.

[0061] (Second mode) In the second mode, for example, when the control unit 13 shown in Figure 1 instructs the drive unit 22 according to this embodiment to start detecting an address event, it turns off the OFG transistor 313 of the pixel 30 in the flicker region determined in the first mode, thereby stopping the supply of electrical signals to the address event detection unit 33.

[0062] On the other hand, when the drive unit 22 detects an address event in a pixel 30 outside the flicker region, the drive unit 22 turns off the OFG transistor 313 of that pixel 30, stopping the supply of electrical signals to the address event detection unit 33. Next, the drive unit 22 drives the transfer transistor 312 by supplying a transfer signal TRG to the transfer transistor 312 in accordance with the control of the readout region selection unit 23, thereby transferring the charge photoelectrically converted by the photodetector 311 to the floating diffusion layer 324.

[0063] In this way, the imaging device 10, which has a pixel array section 21 in which the pixels 30 with the above configuration are arranged in two dimensions, outputs only the pixel signals in the pixel row where the pixel 30 for which an address event has been detected is located to the column processing section 24. This reduces the power consumption of the imaging device 10 and the amount of image processing required compared to the case where the pixel signals of all pixels are output regardless of whether an address event has occurred. In particular, in the second mode, the address events of pixels 30 in the flicker region are not output, so the power consumption of the imaging device 10 and the amount of image processing required can be further reduced.

[0064] The pixel configuration 30 illustrated here is merely an example and is not limited to this configuration. For example, a pixel configuration without an image signal generation unit 32 is also possible. In this pixel configuration, the OFG transistor 313 can be omitted in the light receiving unit 31, and the function of the OFG transistor 313 can be assigned to the transfer transistor 312.

[0065] [Example of address event detection unit configuration] Figure 5 is a block diagram showing a first configuration example of the address event detection unit 33. As shown in Figure 5, the address event detection unit 33 according to this configuration example has a configuration comprising a current-voltage conversion unit 331, a buffer 332, a subtractor 333, a quantizer 334, and a transfer unit 335.

[0066] The current-voltage conversion unit 331 converts the electrical signal from the light-receiving unit 31 of the pixel 30 into a logarithmic pixel signal. The current-voltage conversion unit 331 supplies the converted pixel signal to the buffer 332. The buffer 332 buffers the voltage signal supplied from the current-voltage conversion unit 331 and supplies it to the subtractor 333.

[0067] The subtractor 333 is supplied with a row drive signal from the drive unit 22. The subtractor 333 reduces the level of the pixel signal supplied from the buffer 332 according to the row drive signal. The subtractor 333 then supplies the reduced-level pixel signal to the quantizer 334. The quantizer 334 quantizes the pixel signal supplied from the subtractor 333 into a digital signal and outputs it to the transfer unit 335 as an address event detection signal.

[0068] The transfer unit 335 reads the detection signal of the address event supplied from the quantizer 334, generates event information (x, y, t, p), and transfers it to the region selection unit 23, etc. As described above, the transfer unit 335 sets (x, y) to the coordinates of the pixel 30 where the address event occurred, and t to the time when the address event occurred. The transfer unit 335 also sets p to 1 when the absolute value of the positive change in the pixel signal generated by the pixel 30 exceeds a predetermined first threshold. Similarly, p is set to -1 when the absolute value of the negative change in the pixel signal generated by the pixel 30 exceeds a predetermined second threshold.

[0069] Furthermore, when the transfer unit 335 receives a response to a request from the read area selection unit 23, it supplies an address event detection signal to the first signal processing unit 25 in the first mode. On the other hand, in the second mode, it supplies it to the drive unit 22 and the first signal processing unit 25.

[0070] [Example of Flicker Detection Unit Configuration] Figure 6 illustrates the configuration of the first signal processing unit 25 and the second signal processing unit 26 for flicker detection. The flicker detection processing units of the first signal processing unit 25 and the second signal processing unit 26 constitute the flicker detection processing unit 27. In Figure 6, for the sake of simplicity, only one of the multiple pixels 30 is shown.

[0071] The first signal processing unit 25 includes a storage unit 40, an event count unit 42, an event difference calculation unit 44, a flicker intensity calculation unit 46, and a flicker stability calculation unit 48. The second signal processing unit 26 includes a flicker determination unit 50 and a subsequent processing unit 60.

[0072] Referring to Figure 2A, we will now explain an example of event generation due to light source flicker using Figures 7A and 7B. Figure 7A is a diagram showing frame images 0 to i in chronological order. Frame images 0 to i are images in which event information (x, y, t, p) is extracted at the same sampling interval Δt and arranged in coordinate order. Frame numbers n=0 to i indicate that time has passed as the frame number increases. Note that the sampling interval Δt in this implementation corresponds to a predetermined time unit.

[0073] In other words, the elapsed time t from the time origin of the image corresponds to the sampling interval Δt × n. To put it another way, the frame number n = Δt × k, where k is an integer k ≥ 0. That is, when k is 1, n is 1 and corresponds to Δt; when k is 2, n is 2 and corresponds to Δt × 2; when k is 3, n is 3 and corresponds to Δt × 3. Thus, the frame number n corresponds to discrete time with a sampling interval Δt × k and k ≥ 0.

[0074] Figure 7A shows a 4x8 pixel region ga4 corresponding to circular region a4 and a 4x8 pixel region ga6 corresponding to circular region a6. In this embodiment, a positive event is defined as a case where the absolute value of the change in the positive polarity p-side, where the pixel signal increases, exceeds a predetermined threshold. A negative event is defined as a case where the absolute value of the change in the negative polarity n-side, where the pixel signal decreases, exceeds a predetermined threshold. Furthermore, positive events may be referred to as on-events and negative events as off-events. Additionally, positive events may be referred to as positive polarity events and negative events as negative polarity events.

[0075] As shown in pixel region ga4, flicker events tend to repeat periodically in the same pixel 30, with positive and negative events alternating. On the other hand, as shown in pixel region ga6, object movement events, with positive and negative events, do not exhibit periodicity and occur in accordance with the movement of the object.

[0076] Figure 7B illustrates an example of event signal x(n) generation. Figure 7B(a) shows the number of events in each region in time series. The vertical axis represents the number of events, and the horizontal axis represents the file number n. The first event signal xp(n) shows the value obtained by adding the number of events with positive polarity p in region sa6 for each file in time series. Similarly, the first event signal xn(n) shows the value obtained by adding the number of events with negative polarity n in region sa6 for each file in time series. For example, in an environment using a flicker light source that blinks at a frequency of 100 Hz, the light emission intensity detected by the address event detection unit 33 will similarly increase and decrease at a frequency of 100 Hz.

[0077] Figure 7B(b) shows the p / n event count difference, which is obtained by subtracting the number of negative events on the negative polarity n side from the number of positive events on the positive polarity p side. The vertical axis represents the number of events, and the horizontal axis represents the file number n. Since the period of the first event signal xp(n) and the period of the second event signal xn(n) are the same, the period of the event signal x(n), which is the p / n event count difference, is the same as that of the first event signal xp(n) and the second event signal xn(n). In this embodiment, the time origin of the first event signal xp(n) and the second event signal xn(n) is assumed to be the same point in time.

[0078] The memory unit 40 stores event information (x, y, t, p) having a timestamp t corresponding to the period of the first mode. The period of the first mode can be set to a desired period.

[0079] The event count unit 42 adds the number of positive events and the number of negative events for each set region sa1 to sa8, based on the timestamp t. In this case, the event count unit 42 adds the count for each sampling interval Δt. As described above, the number of repetitions of Δt corresponds to the number of frames n. The set regions sa1 to sa8 for each pixel can be set to any range.

[0080] The event number difference calculation unit 44 calculates an event signal x(n) by adding one of the positive events and negative events as a negative value. As shown in FIG. 7B(b), the event signal x(n) has an alternating positive and negative AC waveform in which the positive and negative events alternately take positive and negative values.

[0081] In this way, the event number difference calculation unit 44 includes a first event signal xp(n) that shows, for each time in time series, the number of times the absolute value of the positive change amount by which the pixel signal generated in the pixel 30 having the photoelectric conversion element 311 increases exceeds a predetermined threshold value, and a second event signal xn(n) that shows, for each time in time series, the number of times the absolute value of the negative change amount by which the pixel signal decreases exceeds a predetermined threshold value, and generates an event signal time series event signal x(n) in which the positive and negative are inverted at the baseline. In FIG. 7(b), for simplicity of explanation, the baseline is set to the event number = 0, but it is not limited to this. For example, the baseline may be the event number = constant value Ofset.

[0082] The flicker intensity calculation unit 46 generates the real part and the imaginary part of the amplitude component at the angular velocity ω with respect to the time series event signal x(n), and adds the real part and the imaginary part of the amplitude component, respectively, to obtain the real part Re n , and the imaginary part Im n to generate an absolute value. More specifically, the flicker intensity calculation unit 46 calculates the flicker intensity intermediate value Amp n of the time series event signal x(n) according to equations (1) and (3). In the present embodiment, F(n) is sometimes referred to as a value related to the phase when it is set as the vector V1 n = Re n + iIm n . Also, the vector V2 n is sometimes referred to as an amplitude component when it is set as V2 n = V1 n - V1 n-1 = Rn + iIn. The vector V1 n and the vector V2 n have a phase (see FIGS. 9 and 10 described later). <0所定の0427>

[0083] For example, the flicker intensity calculation unit 46 performs a convolution operation of a sine wave and a cosine wave of angular velocity ω on the event signal x(n) which changes with respect to the frame number n. In this embodiment, the sine wave and cosine wave of angular velocity ω may be referred to as the flicker detection signal.

[0084] As described above, the elapsed time t from the time origin of the image corresponds to the sampling interval Δt × n. In other words, the frame number n = Δt × k, where k is an integer such that k ≥ 0. That is, when k is 1, n is 1, and n is equivalent to the number of repetitions of the sampling interval Δt.

[0085] Here, the frame number n is an integer between 0 and N-1. In this case, n is monotonically increased. As described above, the frame number n = Δt × k, and k is an integer k ≥ 0, so equation (1) becomes a discrete convolution integral for each sampling interval Δt. N is a constant and is the number of frames at which the convolution operation is terminated (the flicker stability is extracted). Note that the sampling interval Δt in this embodiment corresponds to a predetermined time unit.

[0086]

number

[0087]

number

[0088] When the flicker detection target frequency f is 100 [Hz] and the EVS sampling frequency fevs is 1000 [Hz] (frame interval Δt: 0.001 [sec]), the flicker detection angular velocity ω is ω = (2π × 100 [Hz]) / 1000 [Hz] = 2π / 10 [rad / Δt], and 10 frames constitute one period (= 2π).

[0089]

number

[0090] In the calculation of the intermediate flicker intensity Amp0, when n=0, it is the square root of the sum of the squares of R0, which is the value obtained by dividing the event signal x(0) multiplied by cos(ω×0) by N, and I0, which is the value obtained by dividing the event signal x(0) multiplied by sin(ω×0) by N. In the calculation of the intermediate flicker intensity Amp1, when n=1, the flicker intensity calculation unit 46 calculates R1(R0+x(1)*cos(ω×1)÷N), which is the sum of R0 and the value obtained by dividing the product of x(1) and cos(ω×1) by N, and I1(I0+x(1)*sin(ω×1)÷N), which is the sum of I0 and sin(ω×1) by N. Then, the square root of the sum of the squares of R1 and I1 is calculated as the intermediate flicker intensity Amp1.

[0091] Similarly, in the calculation of the intermediate flicker intensity Amp2, when n=2, the flicker intensity calculation unit 46 calculates R2 (R1 + x(2) * cos(ω × 2) ÷ N) by adding the value obtained by dividing the product of x(2) and cos(ω × 2) by N to R1, and I2 (I1 + x(2) * sin(ω × 2) ÷ N) by adding the value obtained by dividing the product of x(2) and sin(ω × 2) by N to I1. Then, the square root of the sum of the squares of R2 and I2 is calculated as the intermediate flicker intensity Amp2.

[0092] Thus, the flicker intensity calculation unit 46 is Rn-1 and I n-1 Vector V1 consisting of n-1 The vector V2 is obtained by multiplying the time-series event signal x(n) and the flicker detection signal. n By taking the sum of, the real part Re n , and the imaginary part Im n Vector V1 consisting of n This generates (see equation (1)). Then, the real part Re n , and the imaginary part Im n By taking the square root of the sum of the squares of each, we obtain vector V1 n Generates the absolute value of the magnitude (see equation (3)).

[0093] Figure 8 shows the median flicker intensity (Amp). n This diagram shows the characteristics of the frame. The horizontal axis represents the frame number n, and the vertical axis represents the median flicker intensity (Amp). n This shows an example in an environment using a flicker light source that blinks at 100 Hz, where the flicker detection target frequency f is 100 [Hz] and the EVS sampling frequency fevs is 1000 [Hz] (frame interval Δt: 0.001 [sec]).

[0094] Line L10 is the median flicker intensity Amp n This shows that line L10 increases monotonically with increasing frame number n. That is, as the number of convolution operations n shown in equations (1) and (3) increases, the intermediate flicker intensity Amp n It increases monotonically.

[0095] On the other hand, line L12 is the midpoint of the flicker intensity Amp for the noisy pseudo-flicker in Figure 8(b). n This shows the median flicker intensity Amp in noisy pseudo-flicker. n This value increases or decreases randomly as the frame number n increases.

[0096] Line L14 shows the midpoint value of the flicker intensity when the flicker detection target frequency f is not set to the same 100 [Hz] as the flicker light source. nThis is shown. As shown in line L14, if the flicker detection target frequency f is not the same as the flicker light source (100 Hz), the midpoint value of the flicker intensity Amp increases with increasing frame number n. n For example, it repeats a periodic increase or decrease.

[0097] Thus, for low-intensity flicker under a light source with a certain flicker frequency, the flicker detection target frequency f is set to the same value as the flicker frequency of the light source, and the intermediate flicker intensity Amp n When calculated, it increases monotonically with increasing frame number n. In contrast, the median flicker intensity Amp in noisy pseudo-flicker j This increases or decreases randomly with increasing frame number n. As can be seen from this, for a low-intensity flicker at a certain flicker frequency, the flicker detection target frequency f is set to the same value as the flicker frequency and the flicker intensity Amp of equation (3) is calculated. n By performing this calculation, it becomes possible to distinguish it from noisy pseudo-flicker.

[0098] Figure 9 shows the midpoint of the flicker intensity (Amp) for the event signal x(n). n This diagram shows the characteristics of [the compound]. The horizontal axis represents the real part Re of equation (1). n The vertical axis represents the imaginary part Im of equation (1). n This shows that, as mentioned above, vector V1 n Re n +iIm n This shows that vector V1 n This is represented by a black filled vector. On the other hand, as mentioned above, vector V2 n V1 n -V1 n-1 This demonstrates that, in a time-series event signal x(n) under a light source with flicker frequency f, Re n +iIm n Phase and V2 n The phase of is the same regardless of n. Therefore, vector V1 n The size increases monotonically.

[0099] Figure 10 shows the midpoint flicker intensity Amp when the flicker detection target frequency f is not the same frequency as the flicker light source. n Alternatively, the median flicker intensity Amp for noisy pseudo-flicker. n This diagram shows the characteristics of [the compound]. The horizontal axis represents the real part Re of equation (1). n The vertical axis represents the imaginary part Im of equation (1). n This shows that, as mentioned above, vector V1 n Re n +iIm n This shows that vector V1 n This is represented by a black filled vector. On the other hand, as mentioned above, vector V2 n V1 n -V1 n-1 This indicates.

[0100] Because the flicker detection target frequency f is not the same frequency as the flicker light source, vector V2 n The phase is vector V1 n-1 It is thought that it will shift with a predetermined angular velocity difference. As a result, vector V1 n The size does not increase monotonically.

[0101] Figure 11 shows examples of the characteristics of low-intensity flicker and noisy pseudo-flicker in a comparative example. The horizontal axis represents frequency, and the vertical axis represents the power spectrum. Figure 11(a) is an example of low-intensity flicker that blinks at 100 Hz. Here, low intensity refers to cases where the light intensity of the light source is small or where the illumination area of ​​the light source is small relative to the measurement area.

[0102] In the example shown in Figure 11(a), a peak appears in the power spectrum at a frequency of 100 Hz, exceeding the threshold th. Therefore, flicker is detected. To detect low-intensity flicker, the threshold th needs to be set to a smaller value.

[0103] On the other hand, Figure 11(b) shows an example of noisy pseudo-flicker. When the threshold th is small, noisy pseudo-flicker may exceed the threshold th at a frequency of 100 Hz even though it has no peak. As a result, it is incorrectly identified as flicker. Thus, as in the comparative example, when setting the threshold th to a smaller value to detect low-intensity flicker, the possibility of identifying noisy pseudo-flicker as flicker increases.

[0104] In contrast, the intermediate flicker intensity value Amp according to this embodiment n For low-intensity flicker under a light source with power frequency f, the midpoint increases monotonically with increasing n, regardless of the light source intensity. Therefore, the midpoint amplifier... n By checking whether or not the flicker is increasing monotonically, it becomes possible to distinguish low-intensity flicker from noisy pseudo-flicker with higher accuracy.

[0105] The flicker stability calculation unit 48 calculates the intermediate flicker intensity value Amp n An evaluation value is calculated that indicates the degree of monotonically increasing with respect to the increase in frame number n. The flicker determination unit 50 calculates an evaluation value that indicates the degree of monotonically increasing flicker intensity Amp with respect to the increase in frame number n. n If the value increases monotonically, it is determined that the flicker is caused by a flicker light source. For example, the flicker determination unit 50 determines whether or not the flicker is caused by flicker based on the evaluation value calculated by the flicker stability calculation unit 48.

[0106] Here, we will explain an example of the processing of the flicker stability calculation unit 48 using Figures 12 and 13. Figure 12 shows the flicker intensity Amp n This figure shows an example of smoothing processing for increasing frame number n. The horizontal axis represents the frame number n, and the vertical axis represents the median flicker intensity (Amp). n This shows the midpoint of the flicker intensity (Amp). n The solid line represents the median flicker intensity (amplitude). n This shows the result after smoothing.

[0107] As shown in Figure 12, the flicker stability calculation unit 48 calculates, for example, the intermediate value of the flicker intensity Amp nCalculate the moving average of the window size m. Here, m=3. This gives the median flicker intensity Amp after smoothing. n Variation is suppressed.

[0108] The flicker stability calculation unit 48 calculates the intermediate flicker intensity value Amp n After calculating the moving average, the evaluation value count is calculated using the conditional expression according to equation (4). In other words, this flicker stability calculation unit 48 calculates the midpoint value of the flicker intensity Amp at frame number n-1. n-1 Moving average value and the median flicker intensity Amp at frame number n n Determine the relative magnitude of the moving average value and the median flicker intensity Amp n-1 The median flicker intensity Amp is greater than the moving average value. n If the moving average is large, add 1 to the evaluation value count. On the other hand, the median flicker intensity Amp n-1 The median flicker intensity Amp is greater than the moving average value. n If the moving average is small, -1 is added to the evaluation value count.

[0109] The flicker stability calculation unit 48 calculates the intermediate flicker intensity value Amp without smoothing. j It is also possible to generate an evaluation value for this. In this embodiment, the evaluation value "count" may be referred to as the flicker stability.

[0110] The closer the evaluation value `count` is to the final value of frame number `n`, the higher the stability. In other words, the midpoint of the flicker intensity `Amp`. n As the moving average increases monotonically over time, the stability of the variable increases.

[0111]

number

[0112] Figure 13 shows the change in the evaluation value count. The horizontal axis represents the frame number n, and the vertical axis represents the evaluation value count. Since the window size m used when calculating the moving average is 3, the evaluation value count is set to 0 for period P10. The solid line represents the case where flicker occurs, and the dashed line represents noisy pseudo-flicker. As shown in Figure 13, when flicker occurs, the evaluation value count increases in accordance with the increase in frame number n. On the other hand, in the case of noisy pseudo-flicker, the evaluation value count increases or decreases randomly in accordance with the increase in frame number n.

[0113] The flicker determination unit 50 of the second signal processing unit 26 determines whether or not there is flicker based on the evaluation value count calculated by the flicker stability calculation unit 48. That is, the flicker determination unit 50 determines that there is flicker if the evaluation value count exceeds a predetermined value. In this way, the flicker determination unit 50 determines the intermediate flicker intensity value Amp n A flicker is identified when the moving average of the parameters increases monotonically by a predetermined percentage or more.

[0114] The flicker detection unit 50 records information about the pixel coordinates (x, y) that it has determined to be flicker in the recording unit 12. In the second mode, the control unit 13 stops the transfer unit 335 (see Figure 5) from transmitting event information of the pixel coordinates (x, y) that it has determined to be flicker to the readout area selection unit 23. As a result, even if an address event is detected, the reading of the image signal from the pixel coordinates (x, y) that it has determined to be flicker is suppressed, thus suppressing flicker. In this case as well, since noisy pseudo-flicker is not distinguished as flicker, noisy information can be transmitted in the same way as natural images.

[0115] The subsequent processing unit 60 performs recognition processing on a region based on the coordinates (x, y) included in the event information, for example. This allows it to perform processing to recognize the category of moving objects in the imaging region as people, animals, etc. In the second mode, the control unit 13 controls the subsequent processing unit 60 so as not to use the event information of the pixel coordinates (x, y) that have been determined to be flicker. In such cases, the subsequent processing unit 60 can perform recognition processing without using the region information of the pixel coordinates (x, y) that have been determined to be flicker, thereby reducing misrecognition.

[0116] Figure 14 is a diagram illustrating another configuration example for flicker detection. It differs from the imaging system 1 described in Figure 6 in that the event number difference calculation unit 44, flicker intensity calculation unit 46, and flicker stability calculation unit 48 of the first signal processing unit 25 are located on the processing unit 20 side. Thus, it is also possible to mainly place the processing unit related to event detection on the imaging device 10 side and the processing unit related to signal processing on the processing unit 20 side.

[0117] Figure 15 illustrates yet another configuration example related to flicker detection. It differs from the imaging system 1 described in Figure 6 in that all of the processing units of the first signal processing unit 25 are located on the imaging device 10 side. Thus, it is also possible to place the processing unit related to flicker detection on the imaging device 10 side.

[0118] Figure 16 is a flowchart showing an example of the flicker detection process. The control unit 13 sets the period N for performing the flicker detection and the regions sa1 to sa8 for determining the flicker, and sets the evaluation value count and frame number n to their initial values ​​of 0 (step S100).

[0119] Next, the event count unit 42 uses the event information (x, y, t, p) that occurred up to period N to determine the total number of positive events and the total number of negative events that occurred in t = Δ × n for each region sa1 to sa8. Then, the event difference calculation unit 44 takes one of the total number of positive events and the total number of negative events as negative and adds them to calculate the event signal x(n) (step S102).

[0120] Next, the flicker intensity calculation unit 46 calculates the intermediate flicker intensity Amp for each region sa1 to sa8 according to equations (1) and (3). n Calculate (Step S104).

[0121] Next, the flicker stability calculation unit 48 calculates the intermediate flicker intensity value Amp for n. n The moving average value is calculated and an evaluation value count is generated (step S106). The control unit 13 determines whether n is less than N-1 (S108). If the control unit 13 determines that n is less than N-1 (Yes in S108), it adds 1 to n (S110) and repeats the process from step S102. On the other hand, if the control unit 13 determines that n is N-1 or greater (No in S108), the flicker stability calculation unit 48 supplies the evaluation value count for N-1 for each region sa1 to sa8 to the flicker determination unit 50 (S110).

[0122] Next, the flicker detection unit 50 determines whether the evaluation value count is above a predetermined threshold for each region sa1 to sa8 (S114). If it is above the predetermined threshold (Yes in S114), it determines that flicker is occurring (step S114) and terminates the process. On the other hand, if the evaluation value count is below a predetermined threshold (No in S114), the flicker detection unit 50 determines that flicker is not occurring (step S118) and terminates the process.

[0123] Figure 17 is a flowchart showing another example of flicker detection processing. Here, we explain an example in which the flicker detection unit 50 adds yet another detection criterion to the flicker detection.

[0124] After the control unit 13 has performed the processing from steps S100 to S112, the flicker determination unit 50 determines the intermediate flicker intensity value Amp when n=N-1. N-1 The system determines whether the value is above a predetermined second threshold (step S120), and if it is above the predetermined second threshold (Yes in S120), it determines that flicker is occurring (step S114), and the process ends.

[0125] On the other hand, the flicker determination unit 50 determines that no flicker has occurred (step S116) if the evaluation value count is less than a predetermined threshold (No. in S120), and terminates the process.

[0126] Thus, the median flicker intensity Amp when n=N-1 N-1 By adding this to the judgment, it becomes possible to suppress the detection of flicker caused by weak light that has little effect on the naked eye. In this way, the flicker detection unit 50 can also perform flicker detection using multiple judgment criteria.

[0127] Figure 18 is a flowchart showing an example of imaging control by the control unit 13. As shown in Figure 18, first the control unit 13 sets the period N for the first mode and the period for the second mode (step S200).

[0128] Next, the control unit 13 causes the imaging device 10 and the second signal processing unit 26 to perform the first mode, which is flicker detection processing (step S202). Then, the control unit 13 causes the recording unit 12 to record information about the area where flicker is occurring as pixel coordinates (x, y).

[0129] Next, the control unit 13 causes the imaging device 10 and the second signal processing unit 26 to perform the imaging process, which is the second mode (step S204). In the second mode, the control unit 13 stops the transfer unit 335 from sending event information of pixel coordinates (x, y) that have been determined to be flicker to the readout area selection unit 23. In addition, in the second mode, the control unit 13 controls the subsequent processing unit 60 so as not to use event information of pixel coordinates (x, y) that have been determined to be flicker.

[0130] Next, the control unit 13 determines whether the number of imaging cycles in the second mode has reached a predetermined number (step S206). If the control unit 13 does not determine that the number of imaging cycles has reached a predetermined number (No. in step S206), it repeats the process from step S204.

[0131] On the other hand, when the control unit 13 determines that the number of imaging operations has reached a predetermined number (Yes in step S206), it determines whether to end the overall control process (step S208). When the control unit 13 determines not to end the control process (No in step S206), it repeats the process from step S202. On the other hand, when the control unit 13 determines to end the control process (Yes in step S206), it ends the overall control process.

[0132] As described above, after executing the first mode, it is possible to repeatedly execute imaging a predetermined number of times in the second mode. As a result, it is possible to periodically determine the flicker region, and even when the flicker region fluctuates, the influence of flicker can be reduced in the captured image in the second mode and the processing after imaging.

[0133] As described above, according to the present embodiment, the event number difference calculation unit 44 shows the number of times the absolute value of the positive change amount by which the electrical signal generated by the photoelectric conversion element 311 increases exceeds a predetermined threshold value at each time in a time series as the first event signal xp(n), and the number of times the absolute value of the negative change amount by which the electrical signal decreases exceeds a predetermined threshold value at each time in a time series as the second event signal xn(n). A time-series event signal x(n) obtained by inverting the sign of one of them is generated. Then, the flicker determination unit 27 calculates a value (vector V1 n , equation (1)) regarding the convolution of the flicker detection target frequency f with respect to the event signal x(n) and the flicker detection signal determined by the sampling frequency of the EVS, and determines whether the event signal x(n) is caused by flicker based on the monotonic increase property.

[0134] Amplitude component V2 n (V1 n -V1 n-1 ]]() becomes in the same phase as V1 when the time-series event signal x(n) has periodicity (see FIG. 9). Therefore, a value regarding the phase of the amplitude component (for example, vector V1 n-1 ), and the phase of the amplitude component (for example, vector V1 n, by using the formula (1), it is possible to determine whether it is caused by flicker. That is, the flicker intensity calculation unit 46 generates, for the time-series event signal x(n), the real part Rn of the convolution operation result of the cosine wave of the flicker detection signal composed of a cosine wave and a sine wave with an angular velocity ω, and the imaginary part In of the convolution operation result of the sine wave, and generates the square root of the sum of the squares of the real part R n and the imaginary part I n respectively (see formula (3)). The flicker determination unit 27 determines that when the flicker intensity intermediate value Amp n (formula (2)) monotonically increases as the number of frames increases, the vector V2 n (V1 n -V1 n-1 ) is objectively determined to be in the same phase as V1 n-1 , and the time-series event signal x(n) can be determined to be caused by flicker.

[0135] Note that the present technology can adopt the following configuration.

[0136] (1) A first event signal showing the number of times the absolute value of the positive change amount in which the pixel signal generated by the pixel having a photoelectric conversion element increases exceeds a predetermined threshold value in time series, and a second event signal showing the number of times the absolute value of the negative change amount in which the pixel signal decreases exceeds a predetermined threshold value in time series, a signal generation unit that generates a time-series event signal that is positive and negative inverted at the baseline, A determination unit that determines whether the event signal is caused by flicker based on a value related to the phase in the convolution operation value of the event signal and a flicker detection signal having the same frequency as the flicker frequency to be detected, An imaging system comprising.

[0137] (2) The system further includes a flicker intensity calculation unit that performs a convolution operation on the aforementioned time-series event signal, which consists of a cosine wave and a sine wave having the same frequency as the flicker frequency to be detected, uses the convolution result of the cosine wave as the real part of the amplitude component, generates the convolution value of the sine wave as the imaginary part of the amplitude component, and generates an absolute value by adding the real and imaginary parts of the amplitude component, respectively. The imaging system according to (1), wherein the determination unit determines that the event signal is caused by flicker when the absolute value obtained by adding the real and imaginary parts of the amplitude component increases monotonically with increasing frame count.

[0138] (3) The imaging system according to (1), wherein the flicker frequency to be detected is the frequency of the light source that generates the flicker.

[0139] (4) The imaging system according to (3), wherein the determination unit determines whether the event signal is caused by flicker when the absolute value in a predetermined number of frames exceeds a predetermined value.

[0140] (5) The system further includes an evaluation processing unit that generates a second absolute value that suppresses random fluctuations in the aforementioned absolute value, The imaging system according to (3), wherein the determination unit determines that the event signal is caused by flicker when the second absolute value increases monotonically in accordance with the increase in the second absolute value of the number of frames.

[0141] (6) The evaluation processing unit calculates the moving average of the absolute value from the second absolute value with the random fluctuations suppressed. n The imaging system described in (5) generates as follows.

[0142] (7) The evaluation processing unit increases the evaluation value indicating stability when the moving average of the absolute value is larger on the side with the increased number of frames, and decreases the evaluation value when the moving average of the adjacent frames is smaller. The imaging system according to (6), wherein the determination unit determines that the event signal is caused by flicker when the evaluation value exceeds a predetermined value.

[0143] (8) The imaging system according to (7), wherein the absolute value is the square root of the sum of the squares of the real-part sums obtained by adding the real parts of each of the amplitude components and the squares of the imaginary-part sums obtained by adding the imaginary parts of each of the amplitude components.

[0144] (9) Multiple photoelectric conversion elements, each converting incident light into photoelectric signals to generate electrical signals, A plurality of detection units generate event information, each corresponding to one of the plurality of photoelectric conversion elements, which determines the polarity as positive when the absolute value of the positive change in the voltage of the pixel signal corresponding to the electrical signal generated by each of the plurality of photoelectric conversion elements exceeds the predetermined threshold, and determines the polarity as negative when the absolute value of the negative change in the voltage of the pixel signal exceeds the predetermined threshold, coordinate information based on the photoelectric conversion element that exceeded the threshold, and time information at the time when the threshold was exceeded. An event signal generation unit generates the first event signal and the second event signal using at least one of the event information generated by the plurality of detection units, The imaging system described in (1), further comprising:

[0145] (10) The imaging system according to (2), wherein the flicker intensity calculation unit performs convolution integration on the event signal over a predetermined period of time using sine waves and cosine waves with sequentially increasing rotational phases, and generates the real and imaginary parts of the amplitude components at the different rotational phases.

[0146] (11) The imaging system according to (9), wherein the event signal generation unit generates the first event signal and the second event signal using event information generated by one or more detection units.

[0147] (12) The imaging system according to (9), further comprising an analog-to-digital converter that converts pixel signals corresponding to electrical signals generated by each of the plurality of photoelectric conversion elements into digital image signals.

[0148] (13) The imaging system according to (10), further comprising a region selection unit that, when the event information is supplied, causes the analog-to-digital converter to selectively output an electrical signal generated by the photoelectric conversion element, which is identified by the coordinate information contained in the event information.

[0149] (14) The imaging system according to (13), further comprising a control unit that controls the supply of event information from the detection unit used to generate the event signal to the region selection unit when the determination unit determines that the event signal is caused by flicker.

[0150] (15) The imaging data based on the digital image signal further includes a recognition processing unit that performs recognition processing on a region identified by coordinate information included in the event information, The imaging system according to (13), wherein, when the determination unit determines that the event signal is caused by flicker, it performs the recognition process without using the event information supplied from the detection unit used to generate the event signal.

[0151] (16) The imaging system according to (2), wherein the determination unit determines whether the event signal is caused by flicker based on the phase of the amplitude component, when the convolution result of the flicker detection signal, which is composed of a cosine wave and a sine wave, is used as the real part of the amplitude component, and the convolution result of the event signal and the sine wave is used as the imaginary part of the amplitude component.

[0152] (17) The imaging system according to (3), wherein the determination unit determines that the event signal is caused by flicker when the phase of each of the amplitude components is within a predetermined range.

[0153] (18) The imaging system according to (17), wherein the determination unit determines that the phase of each amplitude component is within a predetermined range when the magnitude of the complex vector composed of the real and imaginary parts of the amplitude component increases monotonically in accordance with the increase in the number of frames.

[0154] (19) A signal generation process that generates a time-series event signal in which a first event signal shows, in a time-series manner, the number of times, in a time-series manner, the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and a second event signal shows, in a time-series manner, the number of times, in a time-series manner, the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold, and the two event signals are inverted in positive and negative signs at the baseline. A determination step in which a determination unit determines whether or not the event signal is caused by flicker, based on the phase value in the multiplication value of the event signal and a flicker detection signal having the same frequency as the frequency to be detected, An imaging method comprising the following features.

[0155] The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents. [Explanation of Symbols]

[0156] 1: Imaging system, 10: Imaging device, 20: Processing unit, 13: Control unit, 33: Address event detection unit (detection unit), 42: Event count unit (event signal generation unit), 44: Event difference calculation unit (signal generation unit), 46: Flicker intensity calculation unit (amplitude component generation unit), 48: Flicker stability calculation unit (evaluation processing unit), 50: Flicker determination unit (determination unit), 60: Subsequent processing unit (recognition processing unit), 311: Photodetector (photoelectric conversion element)

Claims

1. A signal generation unit generates a time-series event signal in which the first event signal, which shows in a time-series manner the number of times the absolute value of the positive change in the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and the second event signal, which shows in a time-series manner the number of times the absolute value of the negative change in the pixel signal exceeds a predetermined threshold, are reversed in sign at the baseline. A determination unit determines whether the event signal is caused by flicker based on the phase value in the convolution calculation of the event signal and a flicker detection signal having the same frequency as the flicker frequency to be detected. An imaging system equipped with the following features.

2. The system further includes a flicker intensity calculation unit that performs a convolution operation on the aforementioned time-series event signal, which consists of a cosine wave and a sine wave having the same frequency as the flicker frequency to be detected, uses the convolution result of the cosine wave as the real part of the amplitude component, generates the convolution value of the sine wave as the imaginary part of the amplitude component, and generates an absolute value by adding the real and imaginary parts of the amplitude component, respectively. The imaging system according to claim 1, wherein the determination unit determines that the event signal is caused by flicker when the absolute value obtained by adding the real and imaginary parts of the amplitude component increases monotonically with increasing frame count.

3. The imaging system according to claim 1, wherein the flicker frequency to be detected is the frequency of the light source that generates the flicker.

4. The imaging system according to claim 3, wherein the determination unit determines whether the event signal is caused by flicker when the absolute value in a predetermined number of frames exceeds a predetermined value.

5. The system further includes an evaluation processing unit that generates a second absolute value that suppresses random fluctuations in the aforementioned absolute value, The imaging system according to claim 3, wherein the determination unit determines that the event signal is caused by flicker when the second absolute value increases monotonically in accordance with the increase in the second absolute value of the number of frames.

6. The evaluation processing unit calculates the moving average value of the absolute value from the second absolute value with the random fluctuations suppressed. n The imaging system according to claim 5, which generates as follows.

7. The evaluation processing unit increases the evaluation value indicating stability when the moving average of the absolute value is larger on the side with the increased number of frames, and decreases the evaluation value when the moving average of the adjacent frames is smaller. The imaging system according to claim 6, wherein the determination unit determines that the event signal is caused by flicker when the evaluation value exceeds a predetermined value.

8. The imaging system according to claim 7, wherein the absolute value is the square root of the sum of the squares of the real-part sums obtained by adding the real parts of each of the amplitude components and the squares of the imaginary-part sums obtained by adding the imaginary parts of each of the amplitude components.

9. Multiple photoelectric conversion elements, each converting incident light into photoelectric signals to generate electrical signals, A plurality of detection units generate event information, each corresponding to one of the plurality of photoelectric conversion elements, which determines the polarity as positive when the absolute value of the positive change in the voltage of the pixel signal corresponding to the electrical signal generated by each of the plurality of photoelectric conversion elements exceeds the predetermined threshold, and determines the polarity as negative when the absolute value of the negative change in the voltage of the pixel signal exceeds the predetermined threshold, coordinate information based on the photoelectric conversion element that exceeded the threshold, and time information at the time when the threshold was exceeded. An event signal generation unit generates the first event signal and the second event signal using at least one of the event information generated by the plurality of detection units, The imaging system according to claim 1, further comprising the following:

10. The imaging system according to claim 2, wherein the flicker intensity calculation unit performs convolution integration on sine waves and cosine waves with rotational phases sequentially increased in predetermined time units for the event signal over a predetermined period, and generates the real and imaginary parts of the amplitude components at the different rotational phases.

11. The imaging system according to claim 9, wherein the event signal generation unit generates the first event signal and the second event signal using event information generated by one or more detection units.

12. The imaging system according to claim 9, further comprising an analog-to-digital converter that converts pixel signals corresponding to electrical signals generated by each of the plurality of photoelectric conversion elements into digital image signals.

13. The imaging system according to claim 10, further comprising a region selection unit that, upon receiving the event information, causes the analog-to-digital converter to selectively output an electrical signal generated by the photoelectric conversion element, which is identified by the coordinate information contained in the event information.

14. The imaging system according to claim 13, further comprising a control unit that controls the supply of event information from the detection unit used to generate the event signal to the region selection unit when the determination unit determines that the event signal is caused by flicker.

15. The imaging data based on the aforementioned digital image signal further includes a recognition processing unit that performs recognition processing on a region identified by coordinate information included in the event information, The imaging system according to claim 13, wherein, when the determination unit determines that the event signal is caused by flicker, it performs the recognition process without using the event information supplied from the detection unit used to generate the event signal.

16. The imaging system according to claim 2, wherein the determination unit determines whether the event signal is caused by flicker based on the phase of the amplitude component, when the convolution result of the flicker detection signal, which is composed of a cosine wave and a sine wave, is used as the real part of the amplitude component, and the convolution result of the event signal and the sine wave is used as the imaginary part of the amplitude component.

17. The imaging system according to claim 3, wherein the determination unit determines that the event signal is caused by flicker when the phase of each of the amplitude components is within a predetermined range.

18. The imaging system according to claim 17, wherein the determination unit determines that the phase of each amplitude component is within a predetermined range when the magnitude of the complex vector composed of the real and imaginary parts of the amplitude component increases monotonically in accordance with the increase in the number of frames.

19. A signal generation process that generates a time-series event signal in which a first event signal shows, in a time-series manner, the number of times, in a time-series manner, the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and a second event signal shows, in a time-series manner, the number of times, in a time-series manner, the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold, and the positive and negative inversions of these event signals occur at the baseline. A determination step in which a determination unit determines whether or not the event signal is caused by flicker, based on the phase value in the multiplication value of the event signal and a flicker detection signal having the same frequency as the frequency to be detected, An imaging method comprising the following features.

Images