Signal processing apparatus, imaging apparatus, and signal processing method

By using the acquisition unit, counter and threshold change unit in the event vision sensor, the threshold value is dynamically adjusted to adapt to the changes in the imaging scene, the problem of low robustness in the prior art is solved, and more efficient event detection performance is achieved.

CN118947128APending Publication Date: 2024-11-12SONY SEMICON SOLUTIONS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380029231.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-29
Filing Date
2023-03-07
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing event vision sensors (EVS) are low in robustness in imaging scenarios, making it difficult to automatically set appropriate event detection thresholds, especially the expected number of events depends on the scene changes.

Method used

A signal processing device and method are designed, including a acquisition unit, a counter and a threshold value change unit, count noise events by obtaining the amount of electrical signal changes in response to incident light, and dynamically adjusting the threshold according to the number of noise events to adapt to different imaging scenarios. demand.

Benefits of technology

Improves the robustness and adaptability of event detection, ensures that the number of noise events is within an acceptable range, and appropriately adjusts the threshold to optimize event detection performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118947128A_ABST
    Figure CN118947128A_ABST
Patent Text Reader

Abstract

A signal processing apparatus according to an aspect of the present disclosure includes an acquisition unit, a counting unit, and a threshold value changing unit. The acquisition unit acquires, as an event signal, a signal in which the amount of change over time exceeds a predetermined value from among a plurality of electric signals generated by a plurality of photoelectric conversion elements from incident light. The counting unit counts a signal corresponding to noise from among the plurality of event signals acquired by the acquisition unit. The threshold value changing unit changes the threshold value based on the noise event count obtained by the counting unit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a signal processing device, an imaging device, and a signal processing method. Background Art

[0002] An imaging device is known that acquires data of a portion where a brightness level changes only when a certain event occurs in an imaging scene. This type of imaging device may be referred to as an event-based vision sensor (EVS).

[0003] The existing EVS basically has a fixed threshold for event detection, and therefore has a disadvantage in that robustness with respect to the imaging scene is low. To solve this problem, for example, the inventions according to Patent Documents 1 and 2 propose to automatically set a threshold for event detection.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: International Publication No. WO 2021 / 235323

[0007] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2020-161993 Summary of the invention

[0008] However, the expected number of events varies depending on the imaging scene; therefore, there is a problem that it is difficult to set the expected number of events for automatically setting the threshold for event detection. Therefore, it is desirable to provide a signal processing device, an imaging device, and a signal processing method so that automatic setting of the threshold for event detection can be performed more appropriately.

[0009] A signal processing device according to one aspect of the present disclosure includes an acquisition unit, a counter, and a threshold change unit. The acquisition unit acquires a signal whose signal change amount exceeds a predetermined threshold value among a plurality of electrical signals generated by a plurality of photoelectric conversion elements in response to incident light as an event signal. The counter counts the signals corresponding to noise among the plurality of event signals acquired by the acquisition unit. The threshold change unit changes the threshold value based on the number of noise events obtained by the counter.

[0010] An imaging device according to one aspect of the present disclosure includes a plurality of photoelectric conversion elements that perform photoelectric conversion on incident light and generate electrical signals. The imaging device also includes a plurality of signal output units configured for corresponding photoelectric conversion elements, as well as a counter and a threshold change unit. When the amount of change in the electrical signal generated by the corresponding photoelectric conversion element exceeds a predetermined threshold, each signal output unit outputs an event signal. The counter counts the signals corresponding to noise among the plurality of event signals output from the plurality of signal output units. The threshold change unit changes the threshold based on the number of noise events obtained by the counter.

[0011] The signal processing method according to aspects of the present invention comprises the following three steps:

[0012] (A) acquiring, as an event signal, a signal whose signal change amount exceeds a predetermined threshold value among a plurality of electrical signals generated by a plurality of photoelectric conversion elements in response to incident light;

[0013] (B) counting signals corresponding to noise among the acquired plurality of event signals; and

[0014] (C) Changing the threshold based on the number of noise events obtained.

[0015] In a signal processing device and a signal processing method according to one aspect of the present disclosure, among a plurality of electrical signals generated by a plurality of photoelectric conversion elements in response to incident light, a signal whose signal change exceeds a predetermined threshold is acquired as an event signal. Among the acquired plurality of event signals, the signal corresponding to the noise is counted, and the threshold is changed according to the number of noise events thus obtained. Therefore, for example, in the case where the sensitivity to the imaging scene is too high, the number of noise events increases; therefore, the threshold can be changed to make the number of noise events smaller. In addition, for example, the threshold can also be changed to make the number of noise events close to an acceptable range.

[0016] In an imaging device according to one aspect of the present disclosure, when the amount of change in the electrical signal generated by the photoelectric conversion element in response to the incident light exceeds a predetermined threshold value, an event signal is output from each signal output unit. Among the multiple event signals output, the signal corresponding to the noise is counted, and the threshold value is changed based on the number of noise events obtained in this way. Therefore, for example, in the case where the sensitivity to the imaging scene is too high, the number of noise events increases; therefore, the threshold value can be changed to make the number of noise events smaller. In addition, for example, the threshold value can also be changed to make the number of noise events close to an acceptable range. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a diagram showing a schematic configuration example of an information processing system.

[0018] Figure 2 It is shown Figure 1 Schematic diagram of an exemplary configuration embodiment of an imaging device in .

[0019] Figure 3 It is shown Figure 2 A diagram of an exemplary configuration embodiment of each pixel in FIG.

[0020] Figure 4 It is shown Figure 2 Schematic diagram of an embodiment of a circuit configuration of each pixel in FIG.

[0021] Figure 5 It is shown Figure 3 Schematic diagram of an embodiment of a circuit configuration of a pixel signal generator in FIG.

[0022] Figure 6 It is shown Figure 2 Schematic diagram of an embodiment of functional blocks of a signal processor in.

[0023] Figure 7 is a diagram showing an embodiment of how to identify a noise event.

[0024] Figure 8 is a diagram showing an embodiment of how to identify a noise event.

[0025] Fig. 9 It is shown Figure 2 Schematic diagram of an embodiment of the functional blocks of a converter in.

[0026] Fig.10 It is shown Fig. 9 Graph of an embodiment of output data of a converter in FIG.

[0027] Fig.11 It is shown Fig. 9 Schematic diagram of an embodiment of output data of a transmitter in .

[0028] Fig.12 It is shown from Fig. 9 Graph of an embodiment of output data of a converter in FIG.

[0029] Fig.13 It is shown from Fig. 9 Schematic diagram of an embodiment of output data of a transmitter in FIG.

[0030] Fig.14 It is shown in Figure 1 Schematic diagram of an embodiment of a threshold value changing process in an imaging device in.

[0031] Fig.15 is a diagram illustrating an embodiment in which the number of noise events changes as a threshold value changes.

[0032] Fig.16 is a diagram showing an embodiment of event image data according to an expected number of events.

[0033] Fig.17 is a diagram illustrating an embodiment of the functional blocks of a processor.

[0034] Fig.18 is a diagram illustrating an embodiment of the functional blocks of a processor.

[0035] Fig.19 is a diagram illustrating an embodiment of the functional blocks of a processor.

[0036] Fig. 20 It is shown Figure 1 Schematic diagram of a modified example of a threshold value changing procedure in an imaging device in FIG.

[0037] Fig.21 is a diagram illustrating an embodiment in which the number of noise events changes as a threshold value changes.

[0038] Fig. 22 It is shown Figure 1 2 is a diagram showing a modified example of a schematic configuration of an imaging device in an embodiment.

[0039] Fig.23 It is shown Figure 2 Schematic diagram of a modified example of the functional blocks of the signal processor in FIG.

[0040] Fig.24 is a diagram showing an embodiment of how to identify a noise event.

[0041] Fig.25 It is shown from Fig. 9 Graph of an embodiment of output data of a converter in FIG.

[0042] Fig.26 It is shown Figure 2 Schematic diagram of a modified example of the pixel array portion in FIG.

[0043] Fig. 27 It is shown Figure 2 Schematic diagram of a modified example of the pixel array portion in FIG.

[0044] Fig.28 is a diagram showing an embodiment of how to identify a noise event.

[0045] Fig.29 It is shown Figure 2 Schematic diagram of a modified example of the functional blocks of the signal processor in FIG.

[0046] Fig.30 is a block diagram showing an embodiment of a schematic configuration of a vehicle control system.

[0047] Fig.31 The diagram is a diagram showing an example of the installation positions of the vehicle exterior information detection unit and the imaging unit for auxiliary explanation. DETAILED DESCRIPTION

[0048] With reference to the accompanying drawings, embodiments of the present disclosure will be described in detail below. The following description is a specific example of the present disclosure, and the present disclosure is not limited to the following modes.

[0049] <1. Implementation Method>

[0050] Figure 1A schematic configuration example of an information processing system 1000 according to an embodiment of the present disclosure is shown. The information processing system 1000 is a system applicable to a system equipped with a camera device. For example, the information processing system 1000 may be applied to a vehicle-mounted camera, a surveillance camera, an industrial device (e.g., a device having a function of fault detection through a camera), a robot (e.g., a drone equipped with a camera), a mobile device (e.g., a smart phone equipped with a camera), and the like. The information processing system 1000 includes an imaging device 100, a processor 200, and a bus 300 coupling them. The bus 300 may be installed between devices in the information processing system 1000 or inside the device. The type of transmission data in the bus 300 is, for example, mipiCSI-2 data or mipiCSI-3 data.

[0051] Figure 2 FIG. 1 shows a schematic configuration example of an imaging device 100. The imaging device 100 is an asynchronous imaging device called EVS. Figure 2 As shown, the imaging device 100 includes a pixel array portion 10 including a plurality of pixels 11 , a driving unit 12 , an arbitrator 13 , a column processor 14 , a signal processor 15 , a converter 16 , a transmitter 17 , and a controller 18 .

[0052] In the pixel array section 10, a plurality of pixels 11 are arranged two-dimensionally in a matrix. With respect to the pixel array in the matrix, vertical signal lines (described later) are provided, with two for each pixel column. Each pixel 11 generates a photocurrent in response to incident light, and generates an analog signal (electric signal) of a voltage according to the generated photocurrent as a pixel signal. Each pixel 11 detects the presence or absence of an address event by whether the amount of change in the pixel signal exceeds a predetermined threshold value. The pixel 11 in which an address event occurs outputs a request to the arbitrator 13.

[0053] The driving unit 12 drives the plurality of pixels 11, and causes it to output a pixel signal generated by each pixel 11 to the column processor 14 through one of the vertical signal lines. The arbiter 13 discriminates requests from the plurality of pixels 11, and sends a response based on the discriminated result to each pixel 11. The pixel 11 that has received the response from the arbiter 13 outputs a detection signal (event signal) indicating the detection result to the driving unit 12 and the signal processor 15 through another vertical signal line. For the readout of the event signal from the pixel 11, it is also possible to cause it to be read out from the pixels 11 in a plurality of rows.

[0054] The column processor 14 includes, for example, an analog-to-digital converter. For example, for each pixel column of the pixel array section 10, the column processor 14 performs processing to convert the analog pixel signal output from the pixel 11 in the column into a digital signal. And, the column processor 14 outputs the digital signal (digital pixel signal) after the analog-to-digital conversion to the signal processor 15.

[0055] The signal processor 15 counts the signals corresponding to the noise among the multiple event signals acquired from the pixel 11, and based on the number of noise events thus obtained, changes the threshold for detecting the presence or absence of the address event. The signal processor 15 changes the threshold so that the number of noise events is close to an acceptable range. The signal processor 15 outputs the multiple event signals acquired from the pixel 11 to the converter 16 as event image data. The signal processor 15 also outputs reliability determination data to the converter 16. The reliability judgment data includes, for example, reliability judgment data such as the number of noise events, the judgment result of whether the number of noise events is within an acceptable range, and the threshold. The signal processor 15 also outputs the multiple pixel signals acquired from the column processor 14 to the converter 16 as image data.

[0056] The converter 16 converts various data obtained from the signal processor 15 into the format of transmission data in the bus 300. The converter 16 outputs the converted data to the transmitter 17. The transmitter 17 transmits the data obtained from the converter 16 to the processor 200 via the bus 300. The controller 18 controls operations of, for example, the pixel array section 10, the driving unit 12, the arbiter 13, the column processor 14, the signal processor 15, the converter 16, and the transmitter 17.

[0057] Subsequently, the configuration of each pixel 11 is described. Figure 3 Schematic configuration examples of each pixel 11 are shown. Figure 3 As shown, each pixel 11 includes a light receiving unit 20 , a pixel signal generator 30 , and an event detector 40 .

[0058] The light receiving unit 20 performs photoelectric conversion on the incident light to generate a photocurrent. And, according to the control of the driving unit 12, the light receiving unit 20 outputs the photocurrent generated by the photoelectric conversion to either the pixel signal generator 30 or the event detector 40. The pixel signal generator 30 generates a voltage signal corresponding to the photocurrent input from the light receiving unit 20 as a pixel signal Sd, and outputs the generated pixel signal Sd to the column processor 14 through the vertical signal line.

[0059] The event detector 40 generates an analog signal (electric signal) of a voltage as a pixel signal according to the photocurrent from the light receiving unit 20. The event detector 40 detects the presence or absence of an address event by whether the change amount of the generated pixel signal exceeds a predetermined threshold value. Address events include, for example, an ON event indicating that the change amount of the pixel signal exceeds an upper threshold value and an OFF event indicating that the change amount is lower than a lower threshold value. In addition, the address event detection signal includes, for example, 1 bit indicating the detection result of the ON event and 1 bit indicating the detection result of the OFF event. It should be noted that the event detector 40 can be configured to detect only ON events.

[0060] When an address event has occurred, the event detector 40 outputs a request to request transmission of an address-event detection signal to the arbitrator 13. And, when a response to the request has been received from the arbitrator 13, the event detector 40 outputs an address event detection signal (event signal Ed) to the drive unit 12 and the signal processor 15.

[0061] Figure 4 FIG. 1 shows an example of a circuit configuration of the pixel 11. As described above, each pixel 11 includes, for example, Figure 4 The light receiving unit 20, the pixel signal generator 30 and the event detector 40 are shown.

[0062] The light receiving unit 20 includes a light receiving element (photoelectric conversion element) PD, a transfer transistor Tr1, and an overflow gate (OFG) transistor Tr2. The transfer transistor Tr1 and the OFG transistor Tr2 are, for example, N-type metal oxide semiconductor (MOS) transistors. The transfer transistor Tr1 and the OFG transistor Tr2 are coupled to each other in series.

[0063] The light receiving element PD is coupled between the common connection node N of the transfer transistor Tr1 and the OFG transistor Tr2 and the ground, and performs photoelectric conversion on the incident light and generates an amount of charge according to the amount of the incident light. The gate electrode of the transfer transistor Tr1 is supplied with a transfer signal Trg from the drive unit 12. In response to the transfer signal Trg, the transfer transistor Tr1 outputs the charge into which the incident light has been photoelectrically converted by the light receiving element PD to the pixel signal generator 30. The gate electrode of the OFG transistor Tr2 is supplied with a control signal Ofg from the drive unit 12. In response to the control signal Ofg, the OFG transistor Tr2 supplies the electrical signal generated by the light receiving element PD to the event detector 40. The electrical signal supplied to the event detector 40 is a photocurrent including the charge.

[0064] The pixel signal generator 30 includes, for example, a reset transistor Tr3, an amplifier transistor Tr4, a selection transistor Tr5, and a floating diffusion layer FD. For example, the reset transistor Tr3, the amplifier transistor Tr4, and the selection transistor Tr5 are N-type MOS transistors.

[0065] The charge in which the incident light has been photoelectrically converted by the light receiving element PD is supplied from the light receiving unit 20 to the pixel signal generator 30 by the transfer transistor Tr1. The charge supplied from the light receiving unit 20 is accumulated in the floating diffusion layer FD. The floating diffusion layer FD generates a voltage signal of a voltage value according to the amount of accumulated charge. That is, the floating diffusion layer FD converts the charge into a voltage.

[0066] The reset transistor Tr3 is coupled between a power supply line of a power supply voltage and the floating diffusion layer FD. A gate electrode of the reset transistor Tr3 is supplied with a reset signal Rst from the drive unit 12. In response to the reset signal Rst, the reset transistor Tr3 initializes (resets) the charge amount of the floating diffusion layer FD.

[0067] The amplifier transistor Tr4 and the selection transistor Tr5 are coupled between the power supply line of the power supply voltage and the vertical signal line. The amplifier transistor Tr4 and the selection transistor Tr5 are coupled to each other in series. The floating diffusion layer FD is coupled to the gate electrode of the amplifier transistor Tr4. The amplifier transistor Tr4 amplifies the voltage signal of the voltage into which the charge has been converted by the floating diffusion layer FD. The gate electrode of the selection transistor Tr5 is supplied with a selection signal Sel from the drive unit 12. In response to the selection signal Sel, the selection transistor Tr5 outputs the voltage signal amplified by the amplifier transistor Tr4 as a pixel signal Sd to the column processor 14 through the vertical signal line.

[0068] In an imaging device 100 provided with a plurality of pixels 11 having the above-described configuration, if the driving unit 12 is instructed by the controller 18 to start detecting an address event, the driving unit 12 supplies a control signal Ofg to the OFG transistor Tr2 of the light receiving unit 20, thereby driving the OFG transistor Tr2 and causing the OFG transistor Tr2 to supply a photocurrent to the event detector 40.

[0069] And, if an address event has been detected in the pixel 11, the driving unit 12 brings the OFG transistor Tr2 of the pixel 11 into an off state and stops it from supplying photocurrent to the event detector 40. Then, the driving unit 12 supplies a transfer signal Trg to the transfer transistor Tr1, thereby driving the transfer transistor Trg and causing the transfer transistor Trg to transfer the charge that has been photoelectrically converted into incident light by the light receiving element PD to the floating diffusion layer FD.

[0070] In this manner, the imaging device 100 provided with the plurality of pixels 11 having the above-described configuration outputs only the pixel signal Sd of the pixel 11 in which the address event has been detected to the column processor 14. Therefore, compared with the case where the pixel signals Sd of all pixels are output regardless of the presence or absence of the address event, the power consumption and the image processing amount of the imaging device 100 can be reduced.

[0071] It should be noted that the configuration of the pixel 11 given as an embodiment here is only an embodiment and is not limited to this configuration embodiment. For example, each pixel 11 may have a pixel configuration in which the pixel signal generator 30 is not included. In this case, in the light receiving unit 20, the OFG transistor Tr2 can be eliminated, and the function of the OFG transistor Tr2 can be provided for the transfer transistor Tr1.

[0072] Figure 5 4 shows an embodiment of a circuit configuration of the event detector 40. For example, Figure 5 As shown, the event detector 40 includes an IV converter 41 , a subtractor 42 , a quantizer 43 , and a transmission unit 44 .

[0073] The IV converter 41 converts the photocurrent from the light receiving unit 20 into a voltage signal of the logarithm of the photocurrent. The IV converter 41 supplies the converted voltage signal to the subtractor 42. For example, Figure 5 As shown, the IV converter 41 includes an N-type transistor Tr6, a P-type transistor Tr7, and an N-type transistor Tr8. For example, the N-type transistor Tr6, the P-type transistor Tr7, and the N-type transistor Tr8 are MOS transistors.

[0074] The N-type transistor Tr6 is coupled between the power supply line of the power supply voltage and the signal input line. The P-type transistor Tr7 and the N-type transistor Tr8 are coupled between the power supply line of the power supply voltage and the ground. The P-type transistor Tr7 and the N-type transistor Tr8 are coupled in series with each other. The gate electrode of the N-type transistor Tr6 and the input terminal of the subtractor 42 are coupled to the common connection node of the P-type transistor Tr7 and the N-type transistor Tr8.

[0075] A predetermined bias voltage Vb is applied to the gate electrode of the P-type transistor Tr7. Therefore, the P-type transistor Tr7 provides a constant current to the N-type transistor Tr8. The photocurrent from the light receiving unit 20 is input to the gate electrode of the N-type transistor Tr8 through the signal input line. The corresponding drains of the N-type transistors Tr6 and Tr8 are coupled to the power supply side; this circuit is called a source follower. The photocurrent from the light receiving unit 20 is converted into a voltage signal of the logarithm of the photocurrent by these two source followers coupled in a loop.

[0076] The subtractor 42 reduces the level of the voltage signal supplied from the IV converter 41 according to the row driving signal Ct1 supplied from the driving unit 12. And, the subtractor 42 supplies the voltage signal after the level reduction to the quantizer 43. For example, Figure 5 As shown, the subtractor 42 includes a capacitance element C1, an inverter circuit IN, a capacitance element C2, and a switch element SW.

[0077] One end of the capacitor element C1 is coupled to the output terminal of the IV converter 41, and the other end of the capacitor element C1 is coupled to the input terminal of the inverter circuit IN. The capacitor element C2 is coupled to the inverter circuit IN in parallel. The switch element SW is coupled between the two ends of the capacitor element C2. The switch element SW is provided with a row drive signal Ct1 from the drive unit 12 as its opening and closing control signal. According to the row drive signal Ct1, the switch element SW opens or closes the path connecting the two ends of the capacitor element C2. The inverter circuit IN reverses the polarity of the voltage signal input through the capacitor element C1.

[0078] In the subtractor 42 having the above configuration, when the switch element SW has entered the on (closed) state, the voltage signal Vinit is input to the terminal of the capacitance element C1 on the IV converter 41 side, and its opposite side terminal is used as a virtual ground terminal. For convenience, the potential of the virtual ground terminal should be zero. At this time, the charge Qinit accumulated in the capacitance element C1 is expressed by the following equation (1), where C1 represents the capacitance value of the capacitance element C1.

[0079] Qinit = C1 × Vinit (1)

[0080] Next, assuming that the switch element SW has entered an OFF (disconnected) state and the voltage of the terminal of the capacitance element C1 on the IV converter 41 side has changed to Vafter, the charge Qafter accumulated in the capacitance element C1 is expressed by the following equation (2).

[0081] Qafter = C1 × Vafter (2)

[0082] Meanwhile, the charge Q2 accumulated in the capacitance element C2 is expressed by the following equation (3), where C2 represents the capacitance value of the capacitance element C2 and Vout represents the output voltage.

[0083] Q2 = -C2 × Vout (3)

[0084] At this time, the total amount of charge of the capacitance elements C1 and C2 does not change; therefore, the following equation (4) holds true.

[0085] Qinit = Qafter + Q2 (4)

[0086] If equation (4) is modified by substituting equations (1) to (3) into equation (4), the following equation (5) is obtained.

[0087] Vout = -(C1 / C2) × (Vafter - Vinit) (5)

[0088] Equation (5) represents the subtraction operation of the voltage signal, and the gain of the subtraction result is C1 / C2. Generally, it is desired to maximize the gain; therefore, it is preferred to design C1 to be large and C2 to be small. At the same time, if C2 is too small, the kTC noise increases, and there is a possibility of noise characteristic degradation; therefore, the reduction of C2 capacity is limited to the range of acceptable noise. In addition, an event detector 40 including a subtractor 42 is installed on each pixel 11; therefore, the capacitive elements C1 and C2 have constraints on the area. Taking this into account, the respective capacitance values ​​of the capacitive elements C1 and C2 are determined.

[0089] The quantizer 43 quantizes the voltage signal supplied from the subtractor 42 into a digital signal, and outputs the digital signal as an address event detection signal (event signal) to the transmission unit 44. Figure 5 As shown, the quantizer 43 includes a comparator CMP. The comparator CMP sets the output signal from the inverting circuit IN (i.e., the voltage signal from the subtractor 42) as a non-inverting (+) input, and sets a predetermined threshold voltage Vth as an inverting (-) input. Furthermore, the comparator CMP compares the voltage signal from the subtractor 42 with the predetermined threshold voltage Vth, and outputs a signal indicating the comparison result to the transmission unit 44 as an address event detection signal (event signal). The quantizer 43 sets (changes) an upper threshold voltage Vthp based on a threshold setting signal thp from the controller 18. The quantizer 43 sets (changes) a lower threshold voltage Vthn based on a threshold setting signal thn from the controller 18.

[0090] The transmission unit 44 outputs a request for sending an address event detection signal to the arbitrator 13. When receiving a response to the request from the arbitrator 13, the transmission unit 44 outputs an address event detection signal (event signal Ed) to the drive unit 12 and the signal processor 15.

[0091] Figure 6 1 shows an embodiment of the functional blocks of the signal processor 15. For example, Figure 6 As shown in , the signal processor 15 includes an output unit 151 , a noise event counter 152 , a noise determination unit 153 , and a threshold value changing unit 154 .

[0092] The noise determination module including the noise event counter 152 and the noise determination unit 153 determines whether each event signal Ed is a signal corresponding to noise based on the event signal Ed output from each pixel 11. For example, in the pixel array section 10, it is assumed that a plurality of pixels 11 are two-dimensionally arranged in an effective pixel area into which light from the outside enters. At this time, the noise determination module scans an inspection area including at least three or more pixels 11 in mapping data that associates the plurality of event signals Ed with the arrangement of the plurality of pixels 11 in the effective pixel area. Based on the distribution of the event signals Ed included in the inspection area, the noise determination module determines whether the event signal Ed of the pixel of interest in the inspection area is a signal corresponding to noise.

[0093] Figure 7 and 8 An embodiment of map data is shown. Figure 7 and Figure 8 In , "POS" indicates that the event signal Ed is a positive event signal Ed, and "NEG" indicates that the event signal Ed is a negative event signal Ed. Figure 7 In FIG. 1 , the inspection region CR is a region including 9 pixels 11 (3×3 pixels 11). Figure 8 In the inspection region CR, the pixel 11 corresponding to the center is the focus pixel, and the pixels adjacent to the focus pixel are the surrounding pixels. The inspection region CR is a region including the focus pixel and a plurality of surrounding pixels. Figure 7 and 8 , the event signal Ed of the circled pixel 11 indicates a signal determined to be noise.

[0094] For example, the noise event counter 152 scans the inspection area CR in the above-mentioned map data. At this time, for example, each time the inspection area CR shifts by one pixel, the noise event counter 152 counts the number of event signals Ed having the same polarity as the event signal Ed of the pixel of interest in the inspection area CR (the number of counts Nx). When the number of counts Nx in each inspection area CR is equal to or less than α (α≥1), the noise determination unit 153 determines that the event signal Ed of the pixel of interest is noise. When the number of counts Nx in each inspection area CR is greater than α (α≥1, the noise determination unit 153 determines that the event signal E of the pixel of interest is not noise.

[0095] It should be noted that the noise determination module can perform noise determination without setting the inspection area CR. For example, when the polarities of the event signal Ed of a certain pixel (pixel of interest) and each pixel adjacent to the pixel of interest are different from each other, the noise determination module can determine that the event signal Ed of the pixel of interest is noise.

[0096] The noise determination module (noise determination unit 153) outputs information (noise address information) about the address of the pixel 11 determined to be noise to the output unit 151. The noise determination module (noise determination unit 153) further outputs reliability determination data Rdd in the above-mentioned mapping data to the threshold value changing unit 154 and the converter 16. For example, the reliability determination data Rdd includes the number of noise events in the above-mentioned mapping data, the determination result of whether the number of noise events in the above-mentioned mapping data is within an acceptable range, a threshold value (threshold value setting signal th) described later, and the like.

[0097] The output unit 151 generates event raw data Erd based on the event signal Ed output from the pixel 11 and the noise address information input from the noise determination module (noise determination unit 153). For example, the output unit 151 excludes the event signal Ed of the pixel 11 determined to be noise from the event raw data Erd' including the event signal Ed output from the pixel 11. Exclusion is, for example, setting all bits of the event signal Ed of the pixel 11 determined to be noise to "0". By doing so, the output unit 151 generates event raw data Erd from the event raw data Erd', and outputs the generated event raw data Erd to the converter 16.

[0098] The threshold value changing unit 154 determines whether to change the current threshold value setting signal th input from the controller 18 based on the reliability determination data Rdd input from the noise determination module (noise determination unit 153) and the predetermined standard for determination. In the case where the threshold value changing unit 154 has determined that it is necessary to change the threshold value setting signal th using a predetermined method, the threshold value changing unit 154 changes the threshold value setting signal th and outputs the threshold value setting signal th after the change to the controller 18. The "predetermined standard for determination" and the "predetermined method" will be described in detail later.

[0099] The controller 18 outputs, for example, a current threshold setting signal thp(t) for setting an upper threshold voltage Vthp in the quantizer 43 and a current threshold setting signal thn(t) for setting a lower threshold voltage Vthn in the quantizer 43 to the threshold changing unit 154. For example, in the case where it is determined that the threshold setting signal thp(t) needs to be changed, the controller 18 changes the threshold setting signal thp(t) using a predetermined method, and outputs the threshold setting signal thp(t+1) after the change to the controller 18. For example, in the case where it is determined that the threshold setting signal thn(t) needs to be changed, the controller 18 changes the threshold setting signal thn(t) using a predetermined method, and outputs the threshold setting signal thn(t+1) after the change to the controller 18. For example, in the case where the threshold changing unit 154 has performed a process of changing the threshold, the controller 18 sets the threshold setting signals thp(t+1) and thn(t+1) of the quantizer 43 after the change as the threshold setting signals thp and thn. For example, in a case where the threshold changing unit 154 does not perform processing for changing the threshold, the controller 18 sets the (original) threshold setting signals thp(t) and thn(t) as the threshold setting signals thp and thn before changing in the quantizer 43 .

[0100] For example, Fig. 9 As shown, the converter 16 includes an output format converter 16A and an output timing controller 16B. The output format converter 16A converts various data obtained from the signal processor 15 into the format of the transmission data in the bus 300. For example, the output format converter 16A converts the event raw data Erd and the reliability determination data Rdd into the format of the transmission data in the bus 300. The output format converter 16A, for example, converts the image data Id including a plurality of pixel signals obtained from the column processor 14 into the format of the transmission data in the bus 300. The output format converter 16A outputs the converted data Da to the transmitter 17 according to the timing control signal from the output timing controller 16B. The transmitter 17 transmits the data Db obtained from the converter 16 to the processor 200 via the bus 300.

[0101] Fig.10 An example of data Da output from the converter 16 is shown. Fig.11 An embodiment of the data Db output from the transmitter 17 is shown. The data Da and Db are mipi CSI-2 data or mipi CSI-3 data.

[0102] For example, in data Da and Db for sending an image of one frame, between a frame start (FS) indicating the start of a frame and a frame end (FE) indicating the end of a frame, for each row of event raw data Erd, a packet having data of the row stored is generated, or, for each part into which the row is divided, a packet having data of the part stored is generated. In data Da and Db, a packet header (PH) is set at the head, and a packet footer (PF) is set at the foot. And, a payload having stored event raw data Erd is arranged between a packet header and a packet footer. In data Da and Db, a footer area includes footer information containing embedded data. The embedded data indicates additional information that can be embedded in the header or footer of data Da and Db. At this time, the embedded data includes the number of noise events in the above-mentioned mapping data, a determination result of whether the number of noise events in the above-mentioned mapping data is within an acceptable range, etc. as reliability determination data Rdd.

[0103] Fig.12 An example of data Da output from the converter 16 is shown. Fig.13 An embodiment of the data Db output from the transmitter 17 is shown. In the data Da and Db, the packet header includes the number of noise events with respect to each row of the event raw data Erd.

[0104] It should be noted that in the case where image data including a plurality of pixel signals acquired from the column processor 14 is transmitted to the processor 200 through the converter 16 and the transmitter 17 , the image data is stored in the payload.

[0105] [operate]

[0106] Subsequently, a threshold value changing process in the imaging device 100 is described. Fig.14 An embodiment of a threshold value changing procedure in the imaging device 100 is shown.

[0107] First, the imaging device 100 starts imaging (step S101). Next, each pixel 11 of the pixel array unit 10 generates a photocurrent in response to incident light, and generates an analog signal (electric signal) of a voltage as a pixel signal according to the generated photocurrent. In addition, when the amount of change in the pixel signal exceeds a predetermined threshold, each pixel 11 of the pixel array unit 10 generates an event signal. The pixel array unit 10 outputs the generated event signal to the signal processor 15. Thus, the signal processor 15 acquires event image data including a plurality of event signals (step S102).

[0108] Next, the signal processor 15 (noise determination module) obtains the event number Na and the noise event number Nb included in the event image data (a plurality of event signals obtained from the pixel 11) (steps S103 and S104). Next, the signal processor 15 (noise determination module) determines whether the current mode is the noise reduction mode (step S105). As a result, in the case where the current mode is the noise reduction mode (yes in step S105), the signal processor 15 (noise determination module) changes the threshold value based on the noise event number Nb. Specifically, the signal processor 15 (noise determination module) determines whether the noise event number Nb is within an acceptable range (step S106). As a result, in the case where the noise event number Nb is not within an acceptable range (N in step S106), the signal processor 15 (noise determination module) changes the threshold value Vth (threshold setting signal th) to make the noise event number Nb close to the acceptable range (step S107), and returns to step S102.

[0109] In the case where the number of noise events Nb is within the acceptable range (Y in step S106), or in the case where the current mode is not the noise reduction mode (N in step S105), the signal processor 15 (threshold value changing unit 154) determines whether the number of events Na is within the acceptable range (step S108). As a result, in the case where the number of events Na is not within the acceptable range (No in step S108), the signal processor 15 (threshold value changing unit 154) changes the threshold value Vth (threshold value setting signal th) so that the number of events Na is close to the acceptable range (step S109), and returns to step S102. In the case where the number of events Na is within the acceptable range (Yes in step S108), the pixel array unit 10 determines whether to terminate imaging. Therefore, in the case where a control signal for terminating imaging is not input (No in step S110), the pixel array unit 10 continues imaging, and the process returns to step S102. In the case where a control signal for terminating imaging has been input (Yes in step S110), the pixel array unit 10 terminates imaging (step S111).

[0110] Fig.15 FIG. 4 shows an embodiment in which the number of noise events Nb varies with a threshold. Fig.15 As shown, the signal processor 15 (threshold value changing unit 154) gradually changes (increases) the threshold value Vth (threshold value setting signal th) so that the number of noise events Nb approaches the acceptable range. Then, the number of noise events Nb gradually approaches the acceptable range. Therefore, the event image data, for example, changes from Fig.16 The noisy data shown in (A) becomes Fig.16 (B) shows data with less noise.

[0111] Even in the case where the number of noise events Nb is within the acceptable range, the signal processor 15 (threshold value changing unit 154) continues to change (increase) the threshold value Vth (threshold value setting signal th) so that the number of events Na approaches the acceptable range (the expected number of events Nc). As a result, when the number of events Na has fallen into the acceptable range (has become equal to or less than the expected number of events Nc), the signal processor 15 (threshold value changing unit 154) stops changing the threshold value Vth (threshold value setting signal th). That is, using the fixed threshold value Vth (fixed threshold value setting signal th), each pixel 11 generates an event signal Ed. At this time, if the expected number of events Nc is too small, the event image data, for example, Fig.16 As shown in (A), not only the noise is reduced but also the object edge is reduced. Therefore, it is preferable to set the expected number of events Nc to a value according to the imaging scene.

[0112] Subsequently, the processor 200 is described. The processor 200 performs processing using various data obtained from the imaging device 100. The processor 200 acquires, for example, event raw data Erd and reliability determination data Rdd as various data, and acquires image data Id. At this time, the processor 200 may include, for example, a reliability determination unit 210 and an interpolation processing unit 220, such as Fig.17 shown.

[0113] The reliability determination unit 210 determines the reliability of the event raw data Erd based on the reliability determination data Rdd obtained from the imaging device 100. For example, in a case where the determination result included in the reliability determination data Rdd indicates that the number of noise events Nb is outside the acceptable range, the reliability determination unit 210 determines that the reliability of the event raw data Erd is low. The reliability determination unit 210 does not perform the transmission of the event raw data Erd determined to be of low reliability to the interpolation processing unit 220. In addition, for example, in a case where the determination result included in the reliability determination data Rdd indicates that the number of noise events Nb is within the acceptable range, the reliability determination unit 210 determines that the reliability of the event raw data Erd is high. The reliability determination unit 210 sends the event raw data Erd determined to be of high reliability to the interpolation processing unit 220. In a case where the image data Id has been input, the reliability determination unit 210 performs the transmission of the input image data Id to the interpolation processing unit 220.

[0114] When the event raw data Erd has been input from the reliability determination unit 210, for example, the interpolation processing unit 220 outputs the input event raw data Erd to the outside at intervals of a period Ta. When the image data Id has been input from the reliability determination unit 210, for example, the interpolation processing unit 220 outputs the input image data Id to the outside at intervals of a period Tb longer than the period Ta. By doing so, it becomes possible to interpolate data with a plurality of event raw data Erd until the image data Id is output next time.

[0115] For example, Fig.17 As shown, a vehicle-mounted sensor processing unit 230 may be provided instead of the interpolation processing unit 220. The vehicle-mounted sensor processing unit 230 uses the event raw data Erd input from the reliability determination unit 210, for example, as sensor information of a vehicle equipped with the information processing system 1000, for various controls.

[0116] For example, Fig.18 As shown, a facial expression detector 240 may be provided instead of the interpolation processing unit 220. For example, the facial expression detector 240 uses the event raw data Erd input from the reliability determination unit 210 as image data for detecting a person's facial expression.

[0117] [Effect]

[0118] Next, the effects of the imaging device 100 are described.

[0119] The existing EVS basically has a fixed threshold for event detection, and therefore has a disadvantage in that robustness with respect to the imaging scene is low. To solve this problem, for example, the inventions according to Patent Documents 1 and 2 propose to automatically set a threshold for event detection. However, the expected number of events varies depending on the imaging scene; therefore, there is a problem in that it is difficult to set the expected number of events for automatically setting the threshold for event detection.

[0120] At the same time, in the present embodiment, among the multiple electrical signals generated by the multiple photoelectric conversion elements PD in response to the incident light, a signal whose signal change exceeds a predetermined threshold is obtained by each pixel 11 as an event signal Ed. Among the multiple event signals Ed obtained by the pixel 11, the signal corresponding to the noise is counted, and the threshold is changed based on the noise event number Nb obtained thereby. Therefore, for example, in the case where the sensitivity to the imaging scene is too high, the noise event number Nb increases; therefore, the threshold can be changed to reduce the noise event number Nb. In addition, for example, the threshold can also be changed to make the noise event number Nb close to an acceptable range. Thus, the automatic setting of the threshold for event detection can be performed more appropriately.

[0121] In an imaging device according to one aspect of the present disclosure, when the amount of change in the electrical signal generated by the photoelectric conversion element in response to incident light exceeds a predetermined threshold value, an event signal Ed is output from each pixel 11. Among the multiple event signals Ed output from the pixel 11, the signal corresponding to the noise is counted, and the threshold value is changed based on the noise event number Nb obtained thereby. Therefore, for example, in the case where the sensitivity to the imaging scene is too high, the noise event number Nb increases; therefore, the threshold value can be changed to make the noise event number Nb smaller. In addition, for example, the threshold value can also be changed to make the noise event number Nb close to an acceptable range. Thus, the automatic setting of the threshold value for event detection can be performed more appropriately.

[0122] In addition, in the present embodiment, the number of events Na included in the event image data (a plurality of event signals obtained from the pixel 11) is brought close to the allowable range (the expected number of events Nc) by changing the threshold. Therefore, the number of noise events Nb can be set according to the expected number of events Nc; therefore, the automatic setting of the threshold for event detection can be performed more appropriately.

[0123] Furthermore, in the present embodiment, in the above-mentioned mapping data, the inspection region CR including at least three pixels 11 is scanned, and it is determined whether the event signal Ed of the pixel of interest in the inspection region CR is a signal corresponding to noise based on the distribution of the event signal Ed included in the inspection region CR. Thus, the number of noise events Nb can be appropriately detected regardless of the imaging scene. As a result, the automatic setting of the threshold for event detection can be performed more appropriately.

[0124] <2. Modifications>

[0125] [Variation A]

[0126] In the above-mentioned embodiment, the signal processor 15 can automatically set the expected number of events Nc. At this time, the threshold value changing process in the imaging device 100 can be, for example, as follows.

[0127] Fig. 20An embodiment of the threshold value changing process in the imaging device 100 according to the present modification is shown. First, the imaging device 100 performs the above-mentioned steps S101 to S107. Next, when the number of noise events Nb is within an acceptable range (Y in step S106), or when the current mode is not a noise reduction mode (N in step S105), the signal processor 15 (threshold value changing unit 154) determines whether the current mode is a mode for automatically setting the expected number of events Nc (automatic setting mode) (step S112). As a result, when the current mode is the automatic setting mode (yes in step S112), the signal processor 15 (threshold value changing unit 154) automatically sets the expected number of events Nc (step S113). For example, when the number of noise events Nb has fallen within an acceptable range, the signal processor 15 (threshold value changing unit 154) sets the number of events Na at this time to the expected number of events Nc. After the signal processor 15 (threshold value changing unit 154) has completed step S113, or in the case where the current mode is not the automatic setting mode (No in step S112), the signal processor 15 (threshold value changing unit 154) executes steps S108 and thereafter.

[0128] Fig.21 FIG. 4 shows an embodiment in which the number of noise events Nb varies with a threshold. Fig.21 As shown, the signal processor 15 (threshold value changing unit 154) gradually changes (increases) the threshold value Vth (threshold value setting signal th) so that the number of noise events Nb approaches the acceptable range. Then, the number of noise events Nb gradually approaches the acceptable range. In addition, when the number of noise events Nb has fallen into the acceptable range, the signal processor 15 (threshold value changing unit 154) sets the number of events Na at this time to the expected number of events Nc. As a result, the number of noise events Nb remains approximately constant.

[0129] In this modification, when the number of events Na included in the event image data (a plurality of event signals obtained from the pixel 11) reaches the allowable range (the expected number of events Nc), the number of events Na at this time is set to the expected number of events Nc. Therefore, the number of noise events Nb can be set according to the expected number of events Nc; therefore, the automatic setting of the threshold for event detection can be performed more appropriately.

[0130] [Variation Example B]

[0131] In the above-described embodiment and modification A, the imaging device 100 may include, for example, Fig. 22The illuminance meter 19 shown in FIG. The illuminance meter 19 is arranged adjacent to the pixel array section 10, and measures the illuminance of light entering the adjacent area of ​​the pixel array section 10. The illuminance meter 19 outputs the measured illuminance data Ld to the noise determination module. The illuminance obtained by the illuminance meter 19 is substantially equal to the illuminance of light entering the pixel array section 10.

[0132] In this modification, for example, Fig.23 and Fig.24 As shown, the noise determination module determines whether the event signal Ed of the pixel of interest in the inspection area CR is a signal corresponding to noise based on a criterion derived based on the illuminance data Ld obtained by the illuminance meter 19 and the distribution of the event signal Ed output from the pixel 11. For example, as the above-mentioned criterion, the noise determination module can set the expected number of events Nc based on the illuminance data Ld. In this way, by setting the expected number of events Nc based on the illuminance data Ld, it is possible to more appropriately perform automatic setting of the threshold for event detection.

[0133] In this modification, for example, Fig.25 As shown, the header area may include header information including embedded data. At this time, the embedded data includes illuminance data Ld as reliability determination data Rdd. In this way, the illuminance data Ld is transmitted to the processor 200, which allows the processor 200 to perform processing according to the illuminance data Ld.

[0134] [Variant C]

[0135] In the above-mentioned embodiment and modification examples A and B, for example, Fig.26 and Fig. 27 As shown, the pixel array section 10 may include a plurality of light-shielding pixels 11a located in the periphery (light-shielding region) of the effective pixel region. Each light-shielding pixel 11a includes, for example, a pixel 11 and a light-shielding layer configured to protect the pixel 11 from incidence of light from the outside. Fig.26 A case is shown in which a plurality of light-shielding pixels 11 a are arranged in a horizontal light-shielding region adjacent to an effective pixel region. Fig. 27 A case is shown in which a plurality of light-shielding pixels 11 a are arranged in a vertical light-shielding region adjacent to an effective pixel region.

[0136] In this modified example, the plurality of pixels provided in the pixel array section 10 include a plurality of pixels 11 provided in an effective pixel region and a plurality of light-shielding pixels 11a provided in a light-shielding region. The noise determination module sets a plurality of event signals obtained from the light-shielding pixels 11a provided in the light-shielding region among the plurality of event signals obtained from the pixels 11 provided in the pixel array section 10 as signals corresponding to noise. That is, the noise determination module sets the number of event signals obtained from the light-shielding pixels 11a provided in the light-shielding region as the number of noise events Nb. Therefore, for example, in the case where the sensitivity to the imaging scene is too high, the number of noise events Nb increases; therefore, the threshold value can be changed to reduce the number of noise events Nb. In addition, for example, the threshold value can also be changed to bring the number of noise events Nb close to an acceptable range. Thus, the automatic setting of the threshold value for event detection can be performed more appropriately.

[0137] In this modification, for example, Fig.28 As shown in , the noise determination unit 153 can derive a horizontal histogram of multiple event signals Ed obtained from multiple shading pixels 11a arranged in the horizontal shading area. At this time, the noise determination unit 153 can set the noise amount in the same row coordinates as the focus pixel within the derived histogram to a value (α) for determining whether the event signal Ed of the focus pixel is a signal corresponding to noise. When the count Nx in each inspection area CR is equal to or less than α, the noise determination unit 153 determines that the event signal Ed of the focus pixel is noise. When the count Nx in each inspection area CR is greater than α, the noise determination unit 153 determines that the event signal E of the focus pixel is not noise. Even in this case, automatic setting of the threshold for event detection can be performed more appropriately.

[0138] In this modification, for example, the noise determination unit 153 may derive a vertical histogram of a plurality of event signals Ed acquired from a plurality of light-shielding pixels 11a arranged in a vertical light-shielding region. At this time, the noise determination unit 153 may set the amount of noise in the same column coordinates as the pixel of interest within the derived histogram as a value (α) for determining whether the event signal Ed of the pixel of interest is a signal corresponding to noise. Even in this case, automatic setting of the threshold for event detection may be performed more appropriately.

[0139] [Variant D]

[0140] In the above-mentioned embodiment and modifications A to C, for example, Fig.29 As shown in , the signal processor 15 may include a noise reduction unit 155 that removes noise included in a plurality of event signals Ed (event image data) acquired from a plurality of pixels 11 .

[0141] The noise reduction unit 155 may output the event image data obtained by noise removal to the output unit 151. At this time, the output unit 151 generates event raw data Erd based on the event image data output from the noise reduction unit 155 and the noise address information input from the noise determination module (noise determination unit 153).

[0142] In this modified embodiment, for example, Fig.29 As shown in , the signal processor 15 may further include an event counter 156 that detects the event number Na included in the event image data output from the noise reduction unit 155. At this time, the event counter 156 outputs the obtained event number Na to the noise determination unit 153. The noise reduction unit 155 further outputs information about the address of the pixel 11 determined to be noise (noise address information) to the noise determination unit 153.

[0143] The noise determination unit 153 calculates the number of noise events Nb based on the noise address information output from the noise reduction unit 155. The noise determination unit 153 calculates α based on the calculated number of noise events Nb. In addition, when the number of events included in the inspection region CR among the number of events Na obtained from the event counter 156 is equal to or less than α, the noise determination unit 153 determines that the event signal Ed of the pixel of interest is noise. When the number of events included in the inspection region CR among the number of events Na obtained from the event counter 156 is greater than α, the noise determination unit 153 determines that the event signal E of the pixel of interest is not noise.

[0144] In the present modification, noise determination is performed using the noise reduction unit 155. Even in this case, automatic setting of a threshold value for event detection can be performed more appropriately.

[0145] [Variant E]

[0146] In the above-mentioned embodiment and modified examples A to D, the arbiter 13 may be eliminated. In addition, in the above-mentioned embodiment and modified examples A to D, the noise event counter 152 may calculate the illuminance data Ld based on the pixel signal obtained from each pixel 11. In addition, in the above-mentioned embodiment and modified examples A to D, a pixel array section including a plurality of pixels may be provided separately from the pixel array section 10, and each pixel includes a light receiving unit 20 and a pixel signal generator 30. In this case, the noise event counter 152 may calculate the illuminance data Ld based on the pixel signal obtained from the pixel array section provided separately from the pixel array section 10.

[0147] <3. Application examples of mobile objects>

[0148] The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device installed on any kind of mobile body (such as a motor vehicle, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal transporter, an airplane, a drone, a ship, and a robot).

[0149] Fig.30 : is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a moving body control system to which the technology according to the embodiment of the present disclosure can be applied.

[0150] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig.30 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, as a functional structure of the integrated control unit 12050, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are exemplified.

[0151] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 serves as a control device for a drive force generating device (such as an internal combustion engine, a drive motor, etc.) for generating a drive force of the vehicle, a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating a braking force of the vehicle, and the like.

[0152] The body system control unit 12020 controls the operation of various devices provided on the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as a headlight, a backup light, a brake light, a turn signal, a fog light, etc. In this case, a radio wave transmitted from a mobile device as a substitute for a key or a signal of various switches may be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls the door lock device, the power window device, the lights, etc. of the vehicle.

[0153] The vehicle exterior information detection unit 12030 detects information outside the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to an imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receive the captured image. In addition, the vehicle exterior information detection unit 12030 may also detect objects such as people, vehicles, obstacles, signs, and text on the road surface, or detect their distances, based on the received image.

[0154] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 may output the electrical signal as an image, or may output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0155] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. The in-vehicle information detection unit 12040 is connected to, for example, a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that photographs the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue or the driver's concentration, or can determine whether the driver is dozing off.

[0156] The microcomputer 12051 can calculate the control target value of the driving force generation device, the steering mechanism, or the braking device based on the information about the inside or outside of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of the advanced driver assistance system (ADAS), which includes collision prevention or shock absorption for the vehicle, follow-up driving based on the following distance, maintaining the vehicle speed for driving, warning of vehicle collision, warning of vehicle deviation from the lane, etc.

[0157] In addition, the microcomputer 12051 can perform collaborative control for automatic driving by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about outside or inside the vehicle obtained by the outside information detection unit 12030 or the inside information detection unit 12040, thereby enabling the vehicle to travel automatically without relying on the driver's operation, etc.

[0158] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information about the outside of the vehicle obtained by the outside vehicle information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare by controlling the headlights to change from high beam to low beam according to the position of the front vehicle or the oncoming vehicle detected by the outside vehicle information detection unit 12030.

[0159] The sound / image output unit 12052 sends an output signal of at least one of sound and image to an output device, which can visually or auditorily notify the occupants of the vehicle or the outside of the vehicle of information. Fig.30 In the embodiment of FIG. 1 , an audio speaker 12061, a display portion 12062, and a dashboard 12063 are shown as output devices. For example, the display portion 12062 may include at least one of an on-board display and a head-up display.

[0160] Fig.31 This is a diagram showing an example of the setting position of the imaging unit 12031.

[0161] exist Fig.31 In the figure, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104 and 12105.

[0162] The imaging units 12101, 12102, 12103, 12104 and 12105 are, for example, disposed at positions on the front nose, side mirrors, rear bumper and rear door of the vehicle 12100 and at positions on the upper portion of the windshield inside the vehicle. The imaging unit 12101 disposed at the front nose inside the vehicle and the imaging unit 12105 disposed at the upper portion of the windshield mainly obtain images in front of the vehicle 12100. The imaging units 12102 and 12103 disposed to the side mirrors mainly obtain images on the sides of the vehicle 12100. The imaging unit 12104 disposed to the rear bumper or rear door mainly obtains images of the rear of the vehicle 12100. The imaging unit 12105 disposed at the upper portion of the windshield inside the vehicle is mainly used to detect vehicles ahead, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0163] By the way, Fig.31An embodiment of the shooting range of the imaging units 12101 to 12104 is described. The imaging range 12111 represents the imaging range of the imaging unit 12101 set to the front nose. The imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging units 12102 and 12103 set to the side mirrors. The imaging range 12114 represents the imaging range of the imaging unit 12104 set to the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 12100 viewed from above is obtained by superimposing image data imaged by the imaging units 12101 to 12104.

[0164] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0165] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging range 12111 to 12114 and the time change of the distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, thereby extracting the nearest three-dimensional object that is particularly present on the driving path of the vehicle 12100 and is traveling at a prescribed speed (e.g., equal to or greater than 0 km / hour) in the same direction as the vehicle 12100 as the leading vehicle. In addition, the microcomputer 12051 can pre-set the following distance to keep in front of the leading vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. As a result, coordinated control for automatic driving that allows the vehicle to automatically travel without relying on the driver's operation, etc. can be performed.

[0166] For example, the microcomputer 12051 can classify the three-dimensional object data related to the three-dimensional object into three-dimensional object data of a two-wheeled vehicle, a standard vehicle, a large vehicle, a pedestrian, a utility pole, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic obstacle avoidance. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that can be visually recognized by the driver of the vehicle 12100 and obstacles that are difficult for the driver of the vehicle 12100 to visually recognize. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle. In the case where the collision risk is equal to or higher than the set value and therefore there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or evasive steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist driving to avoid collisions.

[0167] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 may, for example, identify a pedestrian by determining whether a pedestrian exists in the captured images of the imaging units 12101 to 12104. This identification of a pedestrian is performed, for example, by a process of extracting feature points in the imaged images of the imaging units 12101 to 12104 as infrared cameras and by performing pattern matching processing on a series of feature points representing the outline of an object to determine whether it is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the imaged images of the imaging units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 so that a square outline for emphasis is displayed as superimposed on the identified pedestrian. The sound / image output unit 12052 may also control the display unit 12062 so that an icon representing a pedestrian or the like is displayed at a desired position.

[0168] As described above, an embodiment of a vehicle control system to which the technology according to the present disclosure can be applied has been described. The technology according to the present disclosure can be applied to the imaging unit 12031 among the above-mentioned components. By applying the technology according to the present disclosure to the imaging unit 12031, highly robust image data can be obtained. As a result, various controls can be performed more appropriately using the image data.

[0169] The present disclosure has been described above by the embodiments and their variations; however, the present disclosure is not limited to the above-mentioned embodiments, etc., and various modifications may be made. It should be noted that the effects described in this specification are only examples. The effects of the present disclosure are not limited to those described in this specification. The present disclosure may have other effects in addition to those described in this specification.

[0170] Furthermore, for example, the present disclosure may have the following configurations.

[0171] (1) A signal processing device comprising:

[0172] an acquisition unit that acquires, as an event signal, a signal whose signal change amount exceeds a predetermined threshold value among a plurality of electrical signals generated by a plurality of photoelectric conversion elements in response to incident light;

[0173] a counter for counting signals corresponding to noise among the plurality of event signals acquired by the acquisition unit; and

[0174] The threshold value changing unit changes the threshold value based on the number of noise events obtained by the counter.

[0175] (2) The signal processing device according to (1), wherein:

[0176] The threshold changing unit changes the threshold so that the number of noise events is close to an acceptable range.

[0177] (3) The signal processing device according to (2), wherein:

[0178] The threshold value changing unit changes the threshold value so that the event number approaches the expected event number, the event number being the number of event signals acquired by the acquisition unit.

[0179] (4) The signal processing device according to (3), wherein:

[0180] When the number of noise events has fallen into an acceptable range, the threshold value changing unit sets the number of events at this time as the expected number of events.

[0181] (5) The signal processing device according to any one of (1) to (4), wherein:

[0182] A plurality of photoelectric conversion elements are two-dimensionally arranged in an effective pixel area, and

[0183] The counter also includes: scanning an inspection area including at least three pixels in mapping data that associates multiple event signals with the configuration of multiple photoelectric conversion elements in an effective pixel area; and determining whether the event signal of a focus pixel in the inspection area is a signal corresponding to noise based on the distribution of event signals contained in the inspection area.

[0184] (6) The signal processing device according to (5), wherein:

[0185] The counter determines whether the event signal of the pixel of interest in the inspection area is a signal corresponding to noise based on the distribution and a criterion derived based on the illuminance of incident light, an electrical signal, or an output signal of a photoelectric conversion element provided separately from the photoelectric conversion element.

[0186] (7) The signal processing device according to any one of (1) to (4), wherein:

[0187] The plurality of photoelectric conversion elements include a plurality of first photoelectric conversion elements provided in the effective pixel region and a plurality of second photoelectric conversion elements provided in the light shielding region, and

[0188] The counter sets the number of signals whose signal change amounts exceed a threshold value among the plurality of electrical signals generated by the plurality of second photoelectric conversion elements as the number of noise events.

[0189] (8) An imaging device comprising:

[0190] A plurality of photoelectric conversion elements for performing photoelectric conversion on incident light and generating electrical signals;

[0191] a plurality of signal output units provided for corresponding photoelectric conversion elements, each of the plurality of signal output units outputting an event signal when a change amount of an electrical signal generated by the corresponding photoelectric conversion element exceeds a predetermined threshold value;

[0192] a counter for counting signals corresponding to noise among a plurality of event signals output from a plurality of signal output units; and

[0193] The threshold value changing unit changes the threshold value based on the number of noise events obtained by the counter.

[0194] (9) The imaging device according to (8), wherein

[0195] The threshold changing unit changes the threshold so that the number of noise events is close to an acceptable range.

[0196] (10) The imaging device according to (9), wherein

[0197] The threshold value changing unit changes the threshold value so that the event number approaches the expected event number, the event number being the number of event signals acquired by the acquisition unit.

[0198] (11) The imaging device according to (10), wherein

[0199] When the number of noise events has fallen into an acceptable range, the threshold value changing unit sets the number of events at this time as the expected number of events.

[0200] (12) The imaging device according to any one of (8) to (11), wherein

[0201] A plurality of photoelectric conversion elements are two-dimensionally arranged in an effective pixel area, and

[0202] Counter: In mapping data that associates multiple event signals with the configuration of multiple photoelectric conversion elements in an effective pixel area, scans an inspection area including at least three pixels; and the counter determines whether the event signal of a focus pixel in the inspection area is a signal corresponding to noise based on the distribution of the event signals contained in the inspection area.

[0203] (13) The imaging device according to (12), wherein

[0204] The counter determines whether the event signal of the pixel of interest in the inspection area is a signal corresponding to noise based on the distribution and a criterion derived based on the illuminance of incident light, an electrical signal, or an output signal of a photoelectric conversion element provided separately from the photoelectric conversion element.

[0205] (14) The imaging device according to any one of (8) to (11), wherein

[0206] The plurality of photoelectric conversion elements include a plurality of first photoelectric conversion elements provided in the effective pixel region and a plurality of second photoelectric conversion elements provided in the light shielding region.

[0207] The plurality of signal output units include a plurality of first signal output units corresponding to the plurality of first photoelectric conversion elements and a plurality of second signal output units corresponding to the plurality of second photoelectric conversion elements, and

[0208] The counter sets the number of event signals output from the plurality of second signal output units as the number of noise events.

[0209] (15) A signal processing method, comprising:

[0210] acquiring, as an event signal, a signal in which a signal change amount exceeds a predetermined threshold value among a plurality of electrical signals generated by a plurality of photoelectric conversion elements in response to incident light;

[0211] counting signals corresponding to noise among the acquired multiple event signals; and

[0212] The threshold is varied based on the number of noise events obtained.

[0213] (16) The signal processing method according to (15), further comprising:

[0214] Change the threshold to bring the number of noise events closer to an acceptable range.

[0215] (17) The signal processing method according to (16), further comprising:

[0216] The threshold is changed so that the event number approaches the expected event number, the event number being the number of event signals acquired by the acquisition unit.

[0217] (18) The signal processing method according to (17), further comprising:

[0218] When the number of noise events falls within an acceptable range, the number of events at this time is set as the expected number of events.

[0219] (19) The signal processing method according to any one of (15) to (18), wherein:

[0220] A plurality of photoelectric conversion elements are two-dimensionally arranged in an effective pixel area, and

[0221] The signal processing method also includes:

[0222] scanning an inspection area including at least three or more pixels in mapping data associating a plurality of event signals with arrangements of a plurality of photoelectric conversion elements in an effective pixel area; and

[0223] Based on the distribution of event signals included in the inspection region, it is determined whether the event signal of the pixel of interest in the inspection region is a signal corresponding to noise.

[0224] (20) The signal processing method according to (19), wherein:

[0225] Whether the event signal of the pixel of interest in the inspection area is a signal corresponding to noise is determined based on the distribution and a criterion derived based on the illuminance of incident light, an electric signal, or an output signal of a photoelectric conversion element provided separately from the photoelectric conversion element.

[0226] (21) The signal processing method according to any one of (15) to (18), wherein:

[0227] The plurality of photoelectric conversion elements include a plurality of first photoelectric conversion elements provided in the effective pixel region and a plurality of second photoelectric conversion elements provided in the light shielding region, and

[0228] The signal processing method also includes:

[0229] The number of signals whose signal change amounts exceed a threshold value among the plurality of electrical signals generated by the plurality of second photoelectric conversion elements is set as the number of noise events.

[0230] This application claims the benefit of Japanese Priority Patent Application JP2022-054433 filed in the Japan Patent Office on Mar. 29, 2022, the entire contents of which are incorporated herein by reference.

[0231] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

Claims

1. A signal processing device, comprising: an acquisition unit that acquires, as an event signal, a signal whose signal change amount exceeds a predetermined threshold value among a plurality of electrical signals generated by a plurality of photoelectric conversion elements in response to incident light; a counter, for counting signals corresponding to noise among the plurality of event signals acquired by the acquisition unit; as well as A threshold value changing unit changes the threshold value based on the number of noise events obtained by the counter.

2. The signal processing device according to claim 1, wherein: The threshold changing unit changes the threshold so that the number of noise events is close to an acceptable range.

3. The signal processing device according to claim 2, wherein: The threshold value changing unit changes the threshold value so that the number of events approaches the expected number of events, the number of events being the number of the event signals acquired by the acquiring unit.

4. The signal processing device according to claim 3, wherein: When the number of noise events has fallen into the acceptable range, the threshold changing unit sets the number of events at this time as the expected number of events.

5. The signal processing device according to claim 1, wherein: The multiple photoelectric conversion elements are two-dimensionally arranged in the effective pixel area, and the counter also includes: scanning an inspection area including at least three pixels in mapping data that associates the multiple event signals with the configuration of the multiple photoelectric conversion elements in the effective pixel area; and determining whether the event signal of the focus pixel in the inspection area is a signal corresponding to noise based on the distribution of the event signals contained in the inspection area.

6. The signal processing device according to claim 5, wherein: The counter determines whether the event signal of the pixel of interest in the inspection area is a signal corresponding to noise based on the distribution and a standard derived based on the illumination of the incident light, the electrical signal, or an output signal of a photoelectric conversion element arranged separately from the photoelectric conversion element.

7. The signal processing device according to claim 1, wherein: The multiple photoelectric conversion elements include multiple first photoelectric conversion elements arranged in the effective pixel area and multiple second photoelectric conversion elements arranged in the light-shielding area, and the counter sets the number of signals whose signal change amounts exceed the threshold among the multiple electrical signals generated by the multiple second photoelectric conversion elements as the number of noise events.

8. An imaging device comprising: A plurality of photoelectric conversion elements for performing photoelectric conversion on incident light and generating electrical signals; a plurality of signal output units provided for corresponding photoelectric conversion elements, each of the plurality of signal output units outputting an event signal when a change amount of an electrical signal generated by the corresponding photoelectric conversion element exceeds a predetermined threshold value; a counter for counting signals corresponding to noise among the plurality of event signals output from the plurality of signal output units; as well as A threshold value changing unit changes the threshold value based on the number of noise events obtained by the counter.

9. The imaging device according to claim 8, wherein: The threshold changing unit changes the threshold so that the number of noise events is close to an acceptable range.

10. The imaging device according to claim 9, wherein: The threshold value changing unit changes the threshold value so that the number of events approaches the expected number of events, the number of events being the number of the event signals acquired by the acquiring unit.

11. The imaging device according to claim 10, wherein: When the number of noise events has fallen into the acceptable range, the threshold changing unit sets the number of events at this time as the expected number of events.

12. The imaging device according to claim 8, wherein: The plurality of photoelectric conversion elements are two-dimensionally arranged in an effective pixel region, and the counter: scans an inspection region including at least three or more pixels in mapping data associating a plurality of the event signals with the arrangement of the plurality of photoelectric conversion elements in the effective pixel region; And the counter determines whether the event signal of the pixel of interest in the inspection area is a signal corresponding to noise based on the distribution of the event signals included in the inspection area.

13. The imaging device according to claim 12, wherein: The counter determines whether the event signal of the pixel of interest in the inspection area is a signal corresponding to noise based on the distribution and a standard derived based on the illumination of the incident light, the electrical signal, or an output signal of a photoelectric conversion element arranged separately from the photoelectric conversion element.

14. The imaging device according to claim 8, wherein: The plurality of photoelectric conversion elements include a plurality of first photoelectric conversion elements disposed in the effective pixel region and a plurality of second photoelectric conversion elements disposed in the light shielding region. The plurality of signal output units include a plurality of first signal output units corresponding to the plurality of first photoelectric conversion elements and a plurality of second signal output units corresponding to the plurality of second photoelectric conversion elements, and The counter sets the number of the event signals output from the plurality of second signal output units as the noise event number.

15. A signal processing method, comprising: acquiring, as an event signal, a signal in which a signal change amount exceeds a predetermined threshold value among a plurality of electrical signals generated by a plurality of photoelectric conversion elements in response to incident light; Counting signals corresponding to noise among the acquired multiple event signals; as well as The threshold is varied based on the number of noise events obtained.

16. The signal processing method according to claim 15, further comprising: The threshold is changed to make the number of noise events close to an acceptable range.

17. The signal processing method according to claim 16, further comprising: The threshold is changed so that the event number approaches the expected event number, the event number being the number of the event signals acquired by the acquisition unit.

18. The signal processing method according to claim 17, further comprising: When the number of noise events has fallen into the acceptable range, the number of events at this time is set as the expected number of events.

19. The signal processing method according to claim 15, wherein: The plurality of photoelectric conversion elements are two-dimensionally arranged in an effective pixel area, and the signal processing method further includes: scanning an inspection area including at least three pixels in mapping data associating a plurality of the event signals with the arrangement of the plurality of photoelectric conversion elements in the effective pixel area; and Based on the distribution of the event signals included in the inspection region, it is determined whether the event signal of the pixel of interest in the inspection region is a signal corresponding to noise.

20. The signal processing method according to claim 19, wherein: Based on the distribution and a standard derived based on the illumination of the incident light, the electrical signal, or an output signal of a photoelectric conversion element provided separately from the photoelectric conversion element, it is determined whether the event signal of the pixel of interest in the inspection area is a signal corresponding to noise.

21. The signal processing method according to claim 15, wherein: The plurality of photoelectric conversion elements include a plurality of first photoelectric conversion elements disposed in an effective pixel region and a plurality of second photoelectric conversion elements disposed in a light shielding region, and the signal processing method further includes: The number of signals whose signal change amounts exceed the threshold value among the plurality of electrical signals generated by the plurality of second photoelectric conversion elements is set as the number of noise events.