Imaging element and imaging apparatus

Abstract

An imaging element (10) according to an embodiment of the present disclosure is provided with: an imaging unit (11) in which a plurality of pixels (110) each including a photoelectric conversion element are arranged in a matrix; a convolution circuit (20, 20A) that performs convolution processing on a plurality of pixel signals that are analog signals output from each of the plurality of pixels (110) on the basis of a convolution coefficient; and a pooling circuit (150) that performs pooling processing on the plurality of pixel signals on which the convolution processing has been performed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an imaging element and an imaging apparatus. BACKGROUND

[0002] A convolutional neural network (CNN) is actively used in the field of image recognition. In CNN processing, a received input image reaches a fully connected layer (FC layer) through a convolution layer and a pooling layer. For example, when an image is output from a contact image sensor (CIS) and image recognition (e.g., character recognition and object recognition) is performed using a CNN, a system outside the CIS is indispensable. In addition, a CNN can require a high-specification graphics processing unit (GPU), depending on the processing content. When real-time processing is intended to be performed, an image recognition system can become large.

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: JP 2012-227695 A SUMMARY

[0006] TECHNICAL PROBLEM

[0007] To simplify an image recognition system, it can be conceivable to implement a CNN in CIS logical processing. However, convolution and pooling operations are bottlenecks. It is difficult to perform processing within a real time. A lead time for recognizing an image is increased. Therefore, it is required to shorten the lead time for recognizing an image while simplifying the image recognition system.

[0008] Therefore, the present disclosure provides an imaging element and an imaging apparatus capable of simplifying an image recognition system and shortening a lead time.

[0009] SOLUTION TO PROBLEM

[0010] An imaging element according to an embodiment of the present disclosure includes: an imaging section in which a plurality of pixels each including a photoelectric conversion element are arranged in a matrix; a convolution circuit that performs convolution processing on a plurality of pixel signals that are analog signals respectively output from the plurality of pixels, on the basis of a convolution coefficient; and a pooling circuit that performs pooling processing on the plurality of pixel signals that have undergone the convolution processing. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a block diagram depicting an example of a schematic configuration of an imaging element according to a first embodiment.

[0012] Figure 2 is a circuit diagram depicting an example of a schematic configuration of a pixel according to the first embodiment.

[0013] Figure 3is a block diagram depicting a CNN operation flow of the imaging element according to the first embodiment.

[0014] Figure 4 is a block diagram depicting a CNN learning flow related to the CNN operation in Figure 3

[0015] Figure 5 is a block diagram depicting a normal image output operation flow of the imaging element according to the first embodiment.

[0016] Figure 6 is a block diagram depicting a flow of a variation of the CNN operation of the imaging element according to the first embodiment.

[0017] Figure 7 is a block diagram depicting a CNN learning flow related to the CNN operation in Figure 6

[0018] Figure 8 An example of a schematic configuration of a convolution circuit based on a convolution filter according to the first embodiment is depicted.

[0019] Figure 9 An example of an operation of the convolution circuit when performing normal image output processing according to the first embodiment is depicted.

[0020] Figure 10 An example of an operation of the convolution circuit when performing convolution processing according to the first embodiment is depicted.

[0021] Figure 11 An example of a schematic configuration of an AD conversion circuit including a pooling circuit according to the first embodiment is depicted.

[0022] Figure 12 An example of a schematic configuration of a convolution circuit based on a convolution filter according to the second embodiment is depicted.

[0023] Figure 13 A slide of a convolution filter according to the third embodiment is depicted.

[0024] Figure 14 is a first diagram depicting an example of an operation of the convolution circuit when performing convolution processing according to the third embodiment.

[0025] Figure 15 is a second diagram depicting an example of an operation of the convolution circuit when performing convolution processing according to the third embodiment.

[0026] Figure 16 A use example using the imaging element according to each embodiment is depicted.

[0027] Figure 17 ​​An example of a schematic configuration of an imaging device is depicted.

[0028] Figure 18 is a block diagram depicting an example of a schematic configuration of a vehicle control system.

[0029] Figure 19 is a view that is an example of assisting in explaining the mounting positions of the outside-vehicle information detecting portion and the imaging portion. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In addition, in the following embodiments, the same reference numerals are assigned to the same parts, and repetitive explanations are omitted.

[0031] Further, the present disclosure will be described in the following item order.

[0032] 1. First Embodiment

[0033] 1-1. Example of Schematic Configuration of Imaging Element

[0034] 1-2. Example of Schematic Configuration of Pixel

[0035] 1-3. Example of CNN Operation and Example of Normal Image Output Operation of Imaging Element

[0036] 1-4. Example of Schematic Configuration of Convolution Circuit

[0037] 1-5. Example of Schematic Configuration of AD Conversion Circuit Including Pooling Circuit

[0038] 1-6. Effects / Advantages

[0039] 2. Second Embodiment

[0040] 3. Third Embodiment

[0041] 4. Other Embodiments

[0042] 5. Application Examples

[0043] 6. Applications

[0044] 7. Appendix

[0045] <1. First Embodiment>

[0046] <1-1. Example of Schematic Configuration of Imaging Device>

[0047] Figure 1 is a block diagram depicting an example of a schematic configuration of an imaging element 10 according to the first embodiment. As shown in the figure, the imaging element 10 includes a pixel array 11, a vertical scanning circuit 12, a horizontal scanning circuit 13, a signal processing circuit 14, and a control circuit 15. Figure 1As illustrated, the imaging element 10 includes an imaging section (pixel array section) 11, a vertical scanning circuit 12, a horizontal scanning circuit 13, a capacitive addition circuit (addition circuit) 14, an analog-digital (AD) conversion circuit 15, a full connection processing section 16, a storage section 17, and a control section 18. The imaging element 10 is, for example, a complementary metal-oxide semiconductor (CMOS) image sensor. The imaging element 10 is incorporated in various imaging devices such as a CIS, for example.

[0048] The imaging section 11 includes a plurality of pixels (pix) 110 capable of performing photoelectric conversion. The pixels 110 are arranged in a two-dimensional lattice pattern in a vertical direction (column direction) and a horizontal direction (row direction). In the present embodiment, the arrangement in the vertical direction is defined as a column, and the arrangement in the horizontal direction is defined as a row. Each row is also referred to as a line. Each column is also referred to as a column. A pixel signal line 121 is connected to each pixel 110 of each row. A vertical signal line 122 is connected to each pixel 110 of each column. Each pixel signal line 121 is connected to the vertical scanning circuit 12. Each vertical signal line 122 is connected to the horizontal scanning circuit 13. The pixel signal line 121 or the vertical signal line 122 includes a plurality of signal lines, for example. Figure 1

[0049] Under the control of the control section 18, the vertical scanning circuit 12 transmits various signals such as a drive pulse for reading a pixel signal from the pixels 110 to the imaging section 11 via each pixel signal line 121. The vertical scanning circuit 12 supplies various signals such as a drive pulse to each pixel 110 via the pixel signal line 121 for each row, and causes each pixel 110 to output a pixel signal to the vertical signal line 122, for example. That is, each pixel 110 of each row is driven by a drive signal supplied from the vertical scanning circuit 12 via the pixel signal line 121. The vertical scanning circuit 12 includes a shift register and an address decoder, for example.

[0050] Under the control of the control section 18, the horizontal scanning circuit 13 scans the pixel signals output (read) by each pixel 110 in the horizontal direction (row direction), and outputs each pixel signal to the capacitive addition circuit 14. Further, under the control of the control section 18, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitive addition circuit 14 in accordance with the processing to be performed. When normal image output processing of outputting a normal image is performed, for example, the horizontal scanning circuit 13 connects all the vertical signal lines 122 to the capacitive addition circuit 14. Further, when CNN processing is performed, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitive addition circuit 14 based on the convolution coefficients stored in the storage section 17 (details will be described later).

[0051] ​Under the control of the control section 18, the capacitance adder circuit 14 outputs the pixel signal output from each pixel 110 to the AD conversion circuit (ADC) 15 in accordance with the processing to be performed. For example, when performing normal image output processing, the capacitance adder circuit 14 directly outputs each pixel signal output from the horizontal scanning circuit 13 to the AD conversion circuit 15 (bypass). Further, when performing CNN processing, the capacitance adder circuit 14 adds (for example, weighted sum addition) each pixel signal output from the horizontal scanning circuit 13 based on the convolution coefficient stored in the storage section 17, and outputs the result to the AD conversion circuit 15 (details will be described later).

[0052] Here, the horizontal scanning circuit 13 and the capacitance adder circuit 14 function as a convolution circuit 20 that performs convolution processing on each pixel signal output from each pixel 110. In normal CNN processing, an input image reaches a fully connected layer (FC layer) through a convolution layer and a pooling layer. In the convolution layer, a feature amount is extracted from the input image based on a convolution coefficient (convolution filter). For example, the convolution filter is sequentially applied to the input image to extract a feature amount. The convolution circuit 20 performs processing (convolution processing) related to the convolution layer in an analog manner. The convolution circuit 20 is an analog circuit.

[0053] The AD conversion circuit 15 performs AD conversion processing on each pixel signal output from the capacitance adder circuit 14 (that is, a pixel signal output from each pixel 110), and outputs each pixel signal as a digital signal to the fully connected processing section 16. The AD conversion circuit 15 can use various AD conversion circuits.

[0054] The AD conversion circuit 15 has a pooling circuit 150. Under the control of the control section 18, the pooling circuit 150 performs pooling processing on the pixel signal output from each pixel 110 in accordance with the processing to be performed. For example, when performing normal image output processing, the pooling circuit 150 directly outputs each pixel signal to the fully connected processing section 16 (bypass). Further, when performing CNN processing, the AD conversion circuit 15 performs pooling processing on each pixel signal, and outputs each pixel signal that has undergone the pooling processing to the fully connected processing section 16. The pooling circuit 150 performs pooling processing on each pixel signal that has undergone the convolution processing (details will be described later).

[0055] Here, in normal CNN processing, in the pooling layer, a maximum value or an average value is extracted in a predetermined number (for example, 3 x 3) of windows. A feature amount is extracted from the input image. In this case, the amount of data is reduced. The pooling circuit 150 performs processing (convolution processing) related to the pooling layer in an analog manner. The pooling circuit 150 and the AD conversion circuit 15 are analog circuits.

[0056] Under the control of the control unit 18, the fully connected processing unit 16 outputs each pixel signal, which is a digital signal output from the AD conversion circuit 15, to the outside of the element (sensor) according to the processing to be performed. For example, when performing normal image output processing, the fully connected processing unit 16 directly outputs each pixel signal to the outside of the element (bypass). Furthermore, when performing CNN processing, the fully connected processing unit 16 performs fully connected processing on each pixel signal and outputs each pixel signal that has undergone fully connected processing to the outside of the element.

[0057] Here, in normal CNN processing, the output from the pooling layer is collected in the fully connected layer. The fully connected processing unit 16 logically performs processing related to the fully connected layer (fully connected processing). The fully connected processing unit 16 is, for example, a digital circuit (logic circuit).

[0058] Incidentally, the pixel signal output from the fully connected processing unit 16 is input to an external device, such as a signal processing unit. For example, the external device generates a digital pixel signal (i.e., pixel data), performs various processes on the pixel data, and finally generates image data.

[0059] Storage unit 17 stores various data such as convolution coefficients (convolution filters). For example, random access memory (RAM) and flash memory are used as storage unit 17. Convolution coefficients read from storage unit 17 are temporarily set in register 17a.

[0060] The control unit 18 controls each part, such as the vertical scanning circuit 12, the horizontal scanning circuit 13, the capacitor adding circuit 14, the pooling circuit 150, and the fully connected processing unit 16. For example, the control unit 18 controls the horizontal scanning circuit 13, the capacitor adding circuit 14, etc., based on the convolution coefficients set in register 18a. For example, a processor such as a central processing unit (CPU) is used as the control unit 18. The CPU includes read-only memory (ROM), RAM, etc. The CPU controls the operation of each circuit and each part by using RAM as working memory according to a program pre-stored in the ROM.

[0061] <1-2. Example of a schematic configuration of pixels>

[0062] Figure 2 An example of a schematic configuration of pixel 110 according to the first embodiment is shown. Figure 2 As shown, pixel 110 includes a photoelectric conversion element 111, a trigger transistor 112, a reset transistor 114, an amplifying transistor 115, and a selection transistor 116. For example, a PN junction photodiode is used as the photoelectric conversion element 111. Furthermore, for example, an N-type metal-oxide-semiconductor (MOS) transistor is used as the trigger transistor 112, the reset transistor 114, the amplifying transistor 115, and the selection transistor 116.

[0063] A pixel signal line 121 is connected to such a pixel 110. The pixel signal line 121 supplies a reset pulse RST, a transfer pulse TRG, and a selection signal SEL to the pixel 110. Therefore, the pixel signal line 121 includes a plurality of signal lines.

[0064] The photoelectric conversion element 111 photoelectrically converts incident light into an amount of electric charge (here, electrons) corresponding to the amount of light. The cathode of the photoelectric conversion element 111 is grounded. The anode of the photoelectric conversion element 111 is connected to the drain of the trigger transistor 112.

[0065] The source of the trigger transistor 112 is connected to the floating diffusion layer 113. The transfer pulse TRG is supplied to the gate of the trigger transistor 112. The trigger transistor 112 is turned on (closed) when the transfer pulse TRG is in a high state, and is turned off (open) when the transfer pulse TRG is in a low state. In a case where the trigger transistor 112 is in an on state, the electric charge output from the photoelectric conversion element 111 is supplied to the floating diffusion layer 113. The floating diffusion layer 113 accumulates the electric charge supplied from the photoelectric conversion element 111. The floating diffusion layer 113 generates a voltage corresponding to the amount of accumulated electric charge.

[0066] The source of the reset transistor 114 is connected to the floating diffusion layer 113. The power supply VDD of the pixel 110 is connected to the drain of the reset transistor 114. The reset pulse RST is supplied to the gate of the reset transistor 114. The reset transistor 114 is turned on when the reset pulse RST is in a high state, and is turned off when the reset pulse RST is in a low state.

[0067] The gate of the amplification transistor 115 is connected to the floating diffusion layer 113. The power supply VDD is connected to the drain of the amplification transistor 115. The drain of the selection transistor 116 is connected to the source of the amplification transistor 115. The source of the selection transistor 116 is connected to a vertical signal line (VSL) 122. The selection signal SEL is supplied to the gate of the selection transistor 116. The selection transistor 116 is turned on when the selection signal SEL is in a high state, and is turned off when the selection signal SEL is in a low state.

[0068] In this configuration, in an initial state, each of the selection signal SEL, the reset pulse RST, and the transfer pulse TRG is in a low state. Furthermore, the photoelectric conversion element 111 is exposed, and the trigger transistor 112 is turned off in a low state by the transfer pulse TRG. Therefore, the electric charge generated by exposure accumulates in the photoelectric conversion element 111.

[0069] When the selection signal SEL is set to the high state, the selection transistor 116 is turned on. The reset pulse RST is set to the high state, and the charge of the floating diffusion layer 113 is discharged to the power supply VDD. Thus, the potential of the floating diffusion layer 113 is reset to a prescribed potential. After a predetermined time has elapsed from the return of the reset pulse RST to the low state, the transfer pulse TRG is set to the high state. The charge accumulated in the photoelectric conversion element 111 by exposure is supplied to the floating diffusion layer 113 and accumulated in the floating diffusion layer 113. A voltage corresponding to the charge accumulated in the floating diffusion layer 113 is generated. The voltage is amplified by the amplification transistor 115 and output to the vertical signal line 122 as a pixel signal via the selection transistor 116.

[0070] <1-3. CNN operation example and normal image output operation example of imaging element>

[0071] Figure 3 is a block diagram depicting the flow of CNN operation of the imaging element 10 according to the first embodiment. In the CNN operation, as shown in Figure 3 , the imaging section 11 acquires each pixel signal (analog signal). The horizontal scan circuit 13 and the capacitive addition circuit 14 (convolution circuit 20) perform convolution processing on each pixel signal based on a convolution coefficient (convolution filter). The pooling circuit 150 in the AD conversion circuit 15 performs pooling processing on each pixel signal subjected to the convolution processing. The fully connected processing section 16 performs fully connected processing on each pixel signal subjected to the pooling processing. Each pixel signal that has undergone the CNN processing (pixel signal and classification label) is output. The imaging element 10 can complete the CNN processing in the sensor. Incidentally, there are various types of convolution filters, and the feature amount to be extracted changes depending on the convolution filter to be applied.

[0072] Here, Figure 4 is a block diagram depicting the flow of CNN learning related to the CNN operation in Figure 3 . In order to realize the CNN operation in Figure 3 , as shown in Figure 4 , the external sensor CNN 51 as a system outside the sensor preliminarily learns the CNN, and the storage section 17 preliminarily stores the convolution coefficient and the processing coefficient of the full connection. The external sensor CNN 51 can learn the CNN (convolution layer to pooling layer to fully connected layer) based on the learning data and the label 52, and acquire the convolution coefficient and the fully connected processing coefficient. The convolution circuit 20 performs convolution processing based on the convolution coefficient stored in the storage section 17. In addition, the fully connected processing section 16 performs fully connected processing based on the fully connected processing coefficient stored in the storage section 17. In Figure 3 and Figure 4In the example of the imaging element 10, the convolution processing, the pooling processing, and the full connection processing are executed in the sensor. Incidentally, the external sensor CNN 51 can include both of hardware and software or either of hardware and software.

[0073] Figure 5 is a block diagram showing a flow of a normal image output operation of the imaging element 10 according to the first embodiment. In the normal image output operation, as shown in Figure 5 , the imaging section 11, the horizontal scanning circuit 13, and the AD conversion circuit 15 (excluding the pooling circuit 150) operate to output a normal image (output all pixels). All of the CNN-based processing is bypassed. The capacitance addition circuit 14, the pooling circuit 150, and the full connection processing section 16 do not perform actions. This causes a normal image in which the CNN processing is not executed to be output.

[0074] Incidentally, although in the example of the imaging element 10, Figure 3 and Figure 4 , the full connection processing section 16 is provided in the sensor, this is not restrictive. Figure 6 is a block diagram depicting a variation of the CNN operation of the imaging element 10 according to the first embodiment. As shown in Figure 6 , instead of the full connection processing section 16, an external sensor CNN 50 that executes the full connection processing can be provided outside the sensor. That is, the full connection processing can be executed outside the sensor.

[0075] Here, Figure 7 is a block diagram depicting a flow of CNN learning related to the CNN operation in Figure 6 . To implement the CNN operation in Figure 6 , as shown in Figure 7 , the external sensor CNN 51 as a system outside the sensor preliminarily learns the CNN, and the storage section 17 preliminarily stores the convolution coefficients (first layer). Then, the external sensor CNN 50 executes the CNN processing in the first layer and subsequent layers. In this case, the convolution processing in the first layer is executed in the sensor, and the subsequent processing is executed outside the sensor.

[0076] Incidentally, the CNN operation and the normal image output operation are switchable, for example, by a user. The user operates an input section (for example, a switch and a button) to cause the control section 18 to switch the CNN operation and the normal image output operation. The input section is electrically connected to the control section 18. The input section receives an input operation from the user, and transmits an input signal (switching instruction signal) to the control section 18 in response to the input operation. The control section 18 switches the CNN operation and the normal image output operation in response to the input signal.

[0077] For example, the control section 18 switches the operation mode from the CNN operation to the normal image output operation in response to an input signal for restricting the CNN operation, and restricts (disables) the convolution processing and the pooling processing. In contrast, the control section 18 switches the operation mode from the normal image output operation to the CNN operation in response to an input signal for permitting the CNN operation, and permits the convolution processing and the pooling processing. This allows a user to be given a selection right of whether to perform the CNN operation, and can improve the convenience of the user.

[0078] <1-4. Example of schematic configuration of convolution circuit>

[0079] Figure 8 An example of a schematic configuration of the convolution circuit 20 based on a convolution filter according to the first embodiment is shown. As shown in FIG. 1, the horizontal scanning circuit 13 includes a multiplexer 13a. Each vertical signal line 122 of each column is connected to the multiplexer 13a. The multiplexer 13a selects a column to which each vertical signal line 122 is to be connected by switching (VSL connection / disconnection switch). The control section 18 controls the multiplexer 13a. Figure 8

[0080] The capacitive addition circuit 14 includes a plurality of multiplexers 14a, a plurality of capacitors 14b, and a plurality of switches 14c. Each multiplexer 14a is connected to the multiplexer 13a of the horizontal scanning circuit 13 via a predetermined number (three in this example) of vertical signal lines 122. Each capacitor 14b is connected at one end to the multiplexer 14a. Each capacitor 14b is connected at the other end of each multiplexer 14a to one signal line 122a. Each multiplexer 14a changes the connection of the capacitors 14b, and sets an addition ratio (addition capacitance setting). The control section 18 controls these multiplexers 14a. Figure 8

[0081] In each multiplexer 14a, a signal line 122a is provided. These signal lines 122a are connected with the adjacent signal line 122a through a connection line 122b. Each switch 14c is individually provided in each signal line 122a, and is individually provided in each connection line 122b. The control section 18 controls the on / off of these switches 14c.

[0082] Figure 9 An example of the operation of the convolution circuit 20 when performing the normal image output processing according to the first embodiment is depicted. In the normal image output processing, as shown in FIG. 2, the control section 18 sets the operation mode to the normal image output operation, and controls the horizontal scanning circuit 13 and the capacitive addition circuit 14. Figure 9 ​​As shown, the horizontal scanning circuit 13 connects all vertical signal lines 122 to the capacitor adder circuit 14 without disconnecting them. Furthermore, the capacitor adder circuit 14 connects all vertical signal lines 122 to signal line 122a via multiplexer 14a and capacitor 14b. At this time, switch 14c connects all signal lines 122a without connecting wiring 122b.

[0083] Figure 10 An example of the operation of the convolution circuit 20 during convolution processing according to the first embodiment is depicted. In convolution processing, such as... Figure 10 As shown, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitor adder circuit 14 and the multiplexer 13a based on a convolutional filter. Furthermore, the capacitor adder circuit 14 controls the connection between each vertical signal line 122 and each signal line 122a, and each multiplexer 14a and each capacitor 14b based on a convolutional filter. In this case, three adjacent signal lines 122a are defined as a group. Each switch 14c connects (horizontally connects) the connecting line 122b for each group and connects to the central signal line 122a. Incidentally, in addition to... Figure 10 Similar connection processing is performed in every pixel 110 outside of each pixel 110 in the image.

[0084] exist Figure 10 In the example, in the 3×3 convolution filter, "first row: 2, 0, and 1; second row: 0, 1, and 0; and third row: 1, 0, and 1" are established. The horizontal scanning circuit 13 and the capacitor adder circuit 14 implement the convolution filter (the number of matrices and each value). The capacitor adder circuit 14 includes each multiplexer 14a and each capacitor 14b for setting the convolution coefficients. For example, in... Figure 10 In the 3×3 convolution filter, two filters are set in the first row and the first column. Figure 10 The pixels 110 in the first row and first column (top left pixel) are connected to two capacitors 14b via vertical signal lines 122. Similarly, zeros are provided in the first row and second column. Figure 10 Pixel 110 in the first row and second column is not connected to capacitor 14b. One is set in the first row and third column. Figure 10 Pixel 110 in the first row and third column is connected to a capacitor 14b. Similarly, the connection / disconnection between pixel 110 and capacitor 14b is performed in the second and third rows.

[0085] Here, and in Figure 11Similar to the convolutional filter depicted, a convolutional filter including zeros (0 filter) can be implemented using the horizontal scanning circuit 13 and the capacitor adder circuit 14. This allows for the implementation of the filter with a simple configuration. Incidentally, the increased number of capacitors 14b in the capacitor adder circuit 14 increases the degrees of freedom for the convolution coefficients (convolutional filter).

[0086] <1-5. Examples of illustrative configurations of AD conversion circuits including pooling circuits>

[0087] Figure 11 An example of a schematic configuration of an AD conversion circuit 15 including a pooling circuit 150 according to a first embodiment is shown. Figure 12 As shown, the AD conversion circuit 15 includes multiple comparators 15a, multiplexers 15b, and multiple counters 15c (CN). For example, an AD conversion circuit using a voltage-controlled oscillator (VCO) is used as the AD conversion circuit 15. Comparators 15a are located on signal line 122a. Multiplexers 15b are connected to each signal line 122a. Each counter 15c is connected to the stage following the multiplexer 15b (downstream of the signal flow).

[0088] Pooling circuit 150 includes OR circuit 150a. OR circuit 150a is disposed between multiplexer 15b and counter 15c. Pooling circuit 150 performs max pooling (maximum pooling) on ​​the pixel signal output from comparator 15a using OR circuit 150a. However, note that pooling is not limited to max pooling. For example, average pooling can be used.

[0089] During normal image output processing, multiplexer 15b directly sends the pixel signal output from comparator 15a to counter 15c (CN). Furthermore, during CNN processing, multiplexer 15b sends the pixel signal output from comparator 15a to OR circuit 150a. OR circuit 150a performs max pooling on each pixel signal and sends each pooled image signal to counter 15c (CN). Each pixel signal output from counter 15c is a digital signal. Control unit 18 controls multiplexer 15b.

[0090] <1-6. Functions / Effects>

[0091] As described above, according to the first embodiment, it is possible to process operations relating to the convolution processing and the pooling processing in an analog manner by incorporating the convolution processing and the pooling processing of the CNN processing in the imaging element 10, and it is possible to speed up the processing time. For example, it is possible to reduce the proportion of memories, logic circuits, and the like by implementing the convolution processing and the pooling processing in an analog circuit. Furthermore, it is possible to reduce the load of the logic processing, and it is possible to shorten the lead time. Thereby, it is possible to simplify the image recognition system, and it is possible to shorten the lead time.

[0092] <2. Second Embodiment>

[0093] Next, with reference to Figure 12 , a convolution circuit 20A according to a second embodiment is described. Figure 12 An example of a schematic configuration of the convolution circuit 20A based on a convolution filter according to the second embodiment is depicted. Differences from the first embodiment will be mainly described below, and other descriptions will be omitted.

[0094] As shown in Figures 13 to 15 , the convolution circuit 20A according to the second embodiment includes a capacitance addition circuit (addition circuit) 14A that is partially different from the capacitance addition circuit (addition circuit) 14A of the first embodiment. In addition to the plurality of capacitors 14b and the plurality of switches 14c according to the first embodiment, the capacitance addition circuit 14A includes a plurality of amplifiers (column amplifiers) 14d for setting a convolution coefficient. These amplifiers 14d are provided for each vertical signal line 122. The gain of the amplifiers 14d is changed to implement a convolution filter. For example, when the convolution filter has a value of 2, the gain of the amplifiers 14d is doubled. Such amplifiers 14d are used to implement a convolution filter. Incidentally, although the capacitance ratio of the capacitors 14b is fixed, the element configuration can be simplified compared to the element configuration in the case of adding by the capacitance ratio.

[0095] As described above, according to the second embodiment, by using a plurality of amplifiers 14d as part of the capacitance addition circuit 14A of the convolution circuit 20A, it is possible to simplify the configuration of the imaging element 10, and thereby it is possible to simplify the image recognition system. In addition, according to the second embodiment, the same effects as the first embodiment can be obtained.

[0096] <3. Third Embodiment>

[0097] Next, with reference to Figure 13 , convolution processing (including sliding processing) according to a third embodiment is described. Differences from the first embodiment will be mainly described below, and other descriptions will be omitted.

[0098] Figure 13 Sliding of a convolution filter according to the third embodiment is depicted. As shown in , the sliding of the convolution filter is performed by moving the convolution filter in the horizontal direction. The sliding of the convolution filter is performed by moving the convolution filter in the horizontal direction.Figure 13 In the third embodiment, the convolution filter is horizontally shifted by a predetermined shift amount, as shown in FIG. 12. Figure 14 In the example of FIG. 12, the predetermined shift amount (stride) is 1, and the convolution filter is slid from A to B with a shift amount = 1. Incidentally, the sliding indicates that the convolution filter is moved, and the stride is a unit indicating the shift amount of the moved convolution filter.

[0099] Figure 15 FIG. 13 is a first diagram showing an embodiment of the operation of the convolution circuit 20 at the time of performing the convolution processing according to the third embodiment. Figure 14 FIG. 14 is a second diagram depicting an example of the operation of the convolution circuit 20 at the time of the convolution processing according to the third embodiment. As shown in FIG. 14, the horizontal scanning circuit 13 and the capacitive adder circuit 14 according to the third embodiment have a similar configuration to that of the first embodiment. Further, the horizontal scanning circuit 13 and the capacitive adder circuit 14 can perform the convolution processing including the sliding processing. Figure 15 Figure 14 As shown in FIG. 12, the horizontal scanning circuit 13 and the capacitive adder circuit 14 according to the third embodiment have a similar configuration to that of the first embodiment. Further, the horizontal scanning circuit 13 and the capacitive adder circuit 14 can perform the convolution processing including the sliding processing.

[0100] As shown in FIG. 12, in the convolution processing, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitive adder circuit 14 on the basis of the convolution filter. Further, the capacitive adder circuit 14 controls the connection between each vertical signal line 122 and each signal line 122a and each multiplexer 14a and each capacitor 14b on the basis of the convolution filter. In this case, 3 adjacent signal lines 122a are defined as a group. Each switch 14c connects the connection line 122b for each group (lateral connection) and connects the central one signal line 122a. As shown in FIG. 12, the convolution filter is implemented by the horizontal scanning circuit 13 and the capacitive adder circuit 14. Figure 14 Figure 15 In the example of FIG. 12, in the 3 x 3 convolution filter, "first row: 2, 0, and 1, second row: 0, 1, and 0, and third row: 1, 0, and 1" are established. The horizontal scanning circuit 13 and the capacitive adder circuit 14 implement the convolution filter (the number of the matrix and each value).

[0101] As shown in FIG. 14, when the shift amount = 1 (see FIG. 12), the horizontal scanning circuit 13 and the capacitive adder circuit 14 perform the convolution processing. Figure 13 Figure 16 ​​​) The horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitive addition circuit 14 based on the convolution filter and the moving amount when the convolution filter is moved. In addition, the capacitive addition circuit 14 controls the connection between each vertical signal line 122 and each signal line 122a to each multiplexer 14a and each capacitor 14b based on the convolution filter and the moving amount. At this time, the groups of 3 signal lines 122a are offset by 1 signal line 122a. Each switch 14c connects the line 122b for each group connection (lateral connection) and connects the center signal line 122a. This causes the convolution filter to slide in the horizontal direction (row direction). This sliding is repeated in the imaging section 11.

[0102] As described above, according to the third embodiment, the convolution processing including the sliding processing can be performed so that the deformation of the convolution processing can be increased and the convenience of the user can be improved. In addition, according to the third embodiment, the same effects as the first embodiment can be obtained.

[0103] Although in the above-described third embodiment, the form in which the AD conversion is performed individually at different times when the AD conversion is performed simultaneously in the convolution filter A and the convolution filter B is adopted, this is not a limitation because the column circuit needs to be added. For example, the column circuit such as a vertical column can be added to perform the AD conversion simultaneously in the convolution filter A and the convolution filter B.

[0104] In addition, although in the above-described third embodiment, the convolution filter is moved in the horizontal direction (row direction), this is not a limitation. For example, the convolution filter can be shifted in the vertical direction (column direction). Also in this case, the convolution filter can be shifted vertically by adding the column circuit such as a vertical column.

[0105] <4. Other Embodiments>

[0106] Incidentally, although in each of the above-described embodiments, one convolution layer and one pooling layer are provided and one convolution processing and one pooling processing are performed, this is not a limitation. For example, a plurality of convolution layers and a plurality of pooling layers can be provided and a plurality of convolution processing times and a plurality of pooling processing times can be repeatedly performed.

[0107] In addition, the imaging element 10 of each of the above-described embodiments can also be formed on one chip. Alternatively, the imaging element 10 can also be formed in a divided manner on a plurality of chips. In addition, the imaging element 10 can be formed as a laminated structure obtained by joining these chips. For example, the imaging element 10 can have a structure in which a light-receiving chip and a circuit chip are laminated vertically (a two-layer laminated structure). The light-receiving chip includes the imaging section 11. The circuit chip includes the storage section 17 and the control section 18.

[0108] <5. Application Examples>

[0109] Next, application examples of the imaging element 10 of each embodiment will be described. Figure 17 Examples of use using the imaging element 10 according to each embodiment are shown.

[0110] For example, as described below, the imaging element 10 described above can be used in various cases of sensing light such as visible light, infrared light, ultraviolet light, and X-rays. For example, as shown in Figure 18 The imaging element 10 is used for "an apparatus for capturing an image provided for appreciation, such as a digital still camera and a portable apparatus having a camera function", "an apparatus provided for transportation, such as a vehicle-mounted sensor, a monitoring camera, and a distance measuring sensor, the vehicle-mounted sensor photographing, for example, a front, a rear, a surrounding environment; and an interior of a vehicle for safe driving, such as automatic stop, a state of a driver, and the like, the monitoring camera monitors a traveling vehicle and a road, and the distance measuring sensor measures, for example, a distance between vehicles", "a household appliance for photographing a user's gesture and operating an apparatus according to the gesture, such as a television, a refrigerator, an air conditioner, and the like", "an apparatus provided for medical care or health care, such as an endoscope and an apparatus performing angiography by receiving infrared light", "an apparatus provided for security, such as a monitoring camera for security and a camera for person authentication", "an apparatus provided for beauty care, such as a skin measuring instrument photographing skin and a microscope photographing a scalp", "an apparatus provided for sports, such as an action camera for sports and a wearable camera", "an apparatus provided for agriculture, such as a camera for monitoring a state of a field and a crop", and the like.

[0111] For example, the imaging element 10 described above can be applied to various electronic apparatuses such as an imaging apparatus including a digital still camera and a digital camera, a mobile phone having an imaging function, and other apparatuses having an imaging function.

[0112] Figure 18 is a block diagram depicting a configuration example of an imaging apparatus 300 that is an electronic apparatus to which the technology according to the present disclosure is applicable. The imaging apparatus 300 includes an optical system 301, a shutter apparatus 302, a solid-state imaging apparatus 303, a control circuit (control section) 304, a signal processing circuit 305, a monitor 306, and a memory 307. The imaging apparatus 300 can capture still images and moving images.

[0113] The optical system 301 includes one or a plurality of lenses. The optical system 301 guides light (incident light) from an object to the solid-state imaging apparatus 303, and forms an image of the light on a light-receiving surface of the solid-state imaging apparatus 303.

[0114] A shutter device 302 is provided between the optical system 301 and the solid-state imaging device 303. The shutter device 302 controls a light application period and a light shielding period of the solid-state imaging device 303 under the control of a control circuit 304.

[0115] The solid-state imaging device 303 includes, for example, a package including the imaging element 10 described above. The solid-state imaging device 303 accumulates signal charges according to light that forms an image on a light-receiving surface via the optical system 301 and the shutter device 302 for a certain period of time. The signal charges accumulated in the solid-state imaging device 303 are transferred according to a drive signal (timing signal) supplied from the control circuit 304.

[0116] The control circuit 304 outputs a drive signal for controlling a transfer operation of the solid-state imaging device 303 and a shutter operation of the shutter device 302 to drive the solid-state imaging device 303 and the shutter device 302.

[0117] The signal processing circuit 305 performs various types of signal processing on the signal charges output from the solid-state imaging device 303. An image (image data) obtained by the signal processing performed by the signal processing circuit 305 is supplied to be displayed on a monitor 306 or supplied to be stored (recorded) in a memory 307.

[0118] Also in the imaging device 300 configured as described above, by applying the imaging element 10 described above as the solid-state imaging device 303, it is possible to simplify an image recognition system and to shorten a lead time.

[0119] <6. Applications>

[0120] Further, the technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device such as an electronic device mounted in any type of mobile body, such as a car, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, a construction machine, and an agricultural machine (tractor). Further, for example, the technology according to the present disclosure can be applied to an endoscopic surgery system, a microscopic surgery system, and the like.

[0121] Figure 18 is a block diagram of an example of a schematic configuration of a vehicle control system 7000 that is an example of a mobile body control system to which the technology according to the embodiments of the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected to each other via a communication network 7010. In the vehicle control system 7000, the plurality of electronic control units are connected to each other via the communication network 7010, and the plurality of electronic control units share information and cooperate with each other. The vehicle control system 7000 includes, for example, an engine control unit (ECU) 7020, a transmission control unit (TCU) 7030, a hybrid control unit (HCU) 7040, a body control unit (BCU) 7050, a head unit (HU) 7060, and a drive assist ECU 7070. Figure 19In the example shown in FIG. 7, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, a vehicle exterior information detection unit 7400, a vehicle interior information detection unit 7500, and an integrated control unit 7600. For example, a communication network 7010 that connects a plurality of control units to each other can be an in-vehicle communication network that conforms to any standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), or the like.

[0122] Each control unit includes a microcomputer that performs operation processing in accordance with various programs, a storage section that stores programs executed by the microcomputer, parameters for various operations, and the like, and a drive circuit that drives various control target devices. Each control unit also includes a network interface (I / F) for performing communication with other control units via the communication network 7010, and a communication I / F for performing communication with devices, sensors, and the like in and outside the vehicle by wired communication or radio communication. Figure 19 The functional configuration of the integrated control unit 7600 shown includes a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning section 7640, a beacon reception section 7650, a vehicle-mounted device I / F 7660, a sound / image output section 7670, a vehicle-mounted network I / F 7680, and a storage section 7690. The other control units similarly include a microcomputer, a communication I / F, a storage section, and the like.

[0123] The drive system control unit 7100 controls the operation of devices related to the drive system of the vehicle in accordance with various programs. For example, the drive system control unit 7100 functions as a control device for a drive force generation device such as an internal combustion engine, a drive motor, or the like that generates a drive force of the vehicle, a drive force transmission mechanism that transmits the drive force to the wheels, a steering mechanism that adjusts the steering angle of the vehicle, a brake device that generates a braking force of the vehicle, and the like. The drive system control unit 7100 can have a function as a control device of an anti-lock brake system (ABS), an electronic stability control (ESC), or the like.

[0124] The drive system control unit 7100 is connected with a vehicle state detection section 7110. The vehicle state detection section 7110 includes, for example, at least one of a gyro sensor that detects the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, and a sensor that detects the operation amount of an accelerator pedal, the operation amount of a brake pedal, the steering angle of a steering wheel, the engine speed, or the rotational speed of the wheels, or the like. The drive system control unit 7100 performs operation processing using signals input from the vehicle state detection section 7110, and performs control of the internal combustion engine, the drive motor, the electric power steering device, the brake device, and the like.

[0125] The body system control unit 7200 controls the operation of various devices provided to the vehicle body in accordance with various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, and the like. In this case, radio waves emitted from a mobile device can be input to the body system control unit 7200 as a substitute for signals of a key or various switches. The body system control unit 7200 receives these input radio waves or signals and controls a door lock device, a power window device, a lamp, or the like of the vehicle.

[0126] The battery control unit 7300 controls a secondary battery 7310 that is a power supply for driving a motor in accordance with various programs. For example, information about a battery temperature, a battery output voltage, a remaining charge amount in the battery, and the like is supplied from a battery device including the secondary battery 7310 to the battery control unit 7300. The battery control unit 7300 performs an arithmetic operation process using these signals and performs control for adjusting the temperature of the secondary battery 7310 or control of a cooling device provided to the battery device or the like.

[0127] The outside information detecting unit 7400 detects information outside the vehicle including the vehicle control system 7000. For example, the outside information detecting unit 7400 is connected with at least one of an imaging unit 7410 and an outside information detecting unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. For example, the outside information detecting unit 7420 includes at least one of an environmental sensor for detecting a current atmospheric condition or a weather condition and a peripheral information detecting sensor for detecting other vehicles, obstacles, pedestrians, and the like in the periphery of a vehicle including the vehicle control system 7000.

[0128] For example, the environmental sensor can be at least one of a raindrop sensor that detects rain, a fog sensor that detects fog, a sunlight sensor that detects a degree of sunlight, and a snow sensor that detects snowfall. The peripheral information detecting sensor can be at least one of an ultrasonic sensor, a radar device, and a LIDAR device (light detection and ranging device, or laser imaging detection and ranging device). Each of the imaging unit 7410 and the outside information detecting unit 7420 can be provided as an independent sensor or device, or can be provided as a device in which a plurality of sensors or devices are integrated.

[0129] Here, Figure 18Examples of mounting positions of the imaging sections 7410 and the external vehicle information detection sections 7420 are depicted. The imaging sections 7910, 7912, 7914, 7916, and 7918 are provided, for example, at at least one of positions on the front nose, side mirrors, rear bumper, and rear door of the vehicle 7900 and a position on the upper portion of the windshield in the vehicle interior. The imaging section 7910 provided on the front nose portion in the vehicle interior and the imaging section 7918 provided on the upper portion of the windshield mainly obtain images of the front of the vehicle 7900. The imaging sections 7912 and 7914 provided on the side mirrors mainly obtain images of the sides of the vehicle 7900. The imaging section 7916 provided on the rear bumper or the rear door mainly obtains an image of the rear of the vehicle 7900. The imaging section 7918 provided on the upper portion of the windshield in the vehicle interior is mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, and the like.

[0130] Incidentally, Figure 18 Examples of imaging ranges of each of the imaging sections 7910, 7912, 7914, and 7916 are depicted. The imaging range a indicates an imaging range of the imaging section 7910 provided to the front nose. The imaging ranges b and c indicate imaging ranges of the imaging sections 7912 and 7914 provided to the side mirrors, respectively. The imaging range d indicates an imaging range of the imaging section 7916 provided to the rear bumper or the rear door. By superimposing image data imaged by the imaging sections 7910, 7912, 7914, and 7916, for example, a bird's-eye image of the vehicle 7900 viewed from above can be obtained.

[0131] The vehicle exterior information detection sections 7920, 7922, 7924, 7926, 7928, 7930 provided to the front, rear, side, corners, and the upper portion of the windshield in the vehicle interior of the vehicle 7900 can also be ultrasonic sensors or radar devices, for example. The vehicle exterior information detection sections 7920, 7926, 7930 provided to the front nose of the vehicle 7900, the rear bumper, the rear door of the vehicle 7900, and the upper portion of the windshield in the vehicle interior can also be LIDAR devices, for example. These vehicle exterior information detection sections 7920 to 7930 are mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, and the like.

[0132] Returning Figure 18The outside information detecting section 7400 causes the imaging section 7410 to take an image of the outside of the vehicle, and receives the taken image data. In addition, the outside information detecting section 7400 receives detection information from the outside information detecting section 7420 connected to the outside information detecting section 7400. In a case where the outside information detecting section 7420 is an ultrasonic sensor, a radar device, a LIDAR device, or the like, the outside information detecting section 7400 transmits an ultrasonic wave, an electromagnetic wave, or the like, and receives information of a reflected wave. The outside information detecting section 7400 can perform a process of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or a process of detecting a distance to the object, on the basis of the received information. In addition, the outside information detecting section 7400 can perform an environmental recognition process of recognizing rain, fog, a road surface condition, or the like, on the basis of the received information. The outside information detecting section 7400 can calculate a distance to an object outside the vehicle on the basis of the received information.

[0133] In addition, the outside information detecting section 7400 can perform an image recognition process of recognizing a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or a process of detecting a distance thereto, on the basis of the received image data. The outside information detecting section 7400 can perform a process such as distortion correction, alignment, or the like, on the received image data, and combine image data imaged by a plurality of different imaging sections 7410 to generate an aerial view image or a panoramic image. The outside information detecting section 7400 can perform a viewpoint conversion process using image data taken by the imaging section 7410 including mutually different imaging sections.

[0134] The in-vehicle information detecting section 7500 detects information about the inside of the vehicle. For example, the in-vehicle information detecting section 7500 is connected to a driver state detecting section 7510 that detects a state of a driver. The driver state detecting section 7510 can include a camera that images the driver, a biological sensor that detects biological information of the driver, a microphone that collects sound within the inside of the vehicle, or the like. The biological sensor is provided in a seat surface, a steering wheel, or the like, for example, and detects biological information of an occupant seated on the seat or the driver holding the steering wheel. On the basis of detection information input from the driver state detecting section 7510, the in-vehicle information detecting section 7500 can calculate a degree of fatigue of the driver or a degree of concentration of the driver, or can determine whether the driver is dozing off. The in-vehicle information detecting section 7500 can subject an audio signal obtained by collecting sound to a process such as a noise canceling process or the like.

[0135] The integrated control unit 7600 controls general operations within the vehicle control system 7000 in accordance with various programs. The integrated control unit 7600 is connected with an input section 7800. The input section 7800 is implemented by a device capable of performing an input operation by an occupant, such as a touch panel, a button, a microphone, a switch, a lever, or the like. The integrated control unit 7600 can be supplied with data obtained by voice recognition of a voice input through the microphone. The input section 7800 can be, for example, a remote control device using infrared rays or another radio wave, or an external connection device, such as a mobile phone, a personal digital assistant (PDA), or the like, that supports operations of the vehicle control system 7000. The input section 7800 can be, for example, a camera. In this case, the occupant can input information by a gesture. Alternatively, data obtained by detecting a motion of a wearable device worn by the occupant can be input. Further, the input section 7800 can include, for example, an input control circuit or the like that generates an input signal based on information input by the occupant or the like using the above-described input section 7800 and outputs the generated input signal to the integrated control unit 7600. The occupant or the like inputs various data or instructs processing operations to the vehicle control system 7000 by operating the input section 7800.

[0136] The storage section 7690 can include a read only memory (ROM) that stores various programs executed by the microcomputer and a random access memory (RAM) that stores various parameters, operation results, sensor values, and the like. Further, the storage section 7690 can be implemented by a magnetic storage device such as a hard disk drive (HDD) or the like, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0137] The general-purpose communication I / F 7620 is a communication I / F that mediates communication with various devices existing in the external environment 7750, which is widely used. The general-purpose communication I / F 7620 can implement a cellular communication protocol such as Global System for Mobile Communications (GSM (registered trademark)), Worldwide Interoperability for Microwave Access (WiMAX (registered trademark)), Long Term Evolution (LTE (registered trademark)), LTE-Advanced (LTE-A), or the like, or another wireless communication protocol such as wireless LAN, also known as Wireless Fidelity (Wi-Fi (registered trademark)), Bluetooth (registered trademark), or the like. For example, the general-purpose communication I / F 7620 can be connected to a device (for example, an application server or a control server) existing on an external network (for example, the Internet, a cloud network, or a company-specific network) via a base station or an access point. Further, for example, the general-purpose communication I / F 7620 can be connected to a terminal (which is, for example, a terminal of a driver, a pedestrian, or a shop, or a machine type communication (MTC) terminal) existing in the vicinity of the vehicle using peer-to-peer (P2P) technology.

[0138] The dedicated communication I / F 7630 is a communication I / F that supports development of a communication protocol for use in a vehicle. The dedicated communication I / F 7630 can implement a standard protocol, for example, like Wireless Access in Vehicular Environments (WAVE) which is a combination of Institute of Electrical and Electronics Engineers (IEEE) 802.11p as a lower layer and IEEE 1609 as a higher layer, Dedicated Short-Range Communications (DSRC), or a cellular communication protocol. The dedicated communication I / F 7630 generally performs V2X communication as a concept including one or more of communication between vehicles (vehicle-to-vehicle), communication between a road and a vehicle (vehicle-to-infrastructure), communication between a vehicle and a home (vehicle-to-home), and communication between a pedestrian and a vehicle (vehicle-to-pedestrian).

[0139] The positioning section 7640 performs positioning, for example, by receiving Global Navigation Satellite System (GNSS) signals (e.g., GPS signals from Global Positioning System (GPS) satellites) from GNSS satellites, and generates position information including latitude, longitude, and altitude of the vehicle. Incidentally, the positioning section 7640 can identify a current position by exchanging signals with a wireless access point, or can obtain position information from a terminal such as a mobile phone, Personal Handyphone System (PHS), or a smart phone having a positioning function.

[0140] The beacon receiving section 7650, for example, receives radio waves or electromagnetic waves emitted from a radio station installed on a road or the like, and thereby obtains information on a current position, congestion, a closed road, a necessary time, and the like. Incidentally, the function of the beacon receiving section 7650 can be included in the above-described dedicated communication I / F 7630.

[0141] The in-vehicle device I / F 7660 is a communication interface that mediates connection between the microcomputer 7610 and various in-vehicle devices 7760 present within the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), Near Field Communication (NFC), or Wireless Universal Serial Bus (WUSB). Furthermore, the in-vehicle device I / F 7660 can establish a wired connection via a connection terminal (and a cable, if necessary) that is not shown in the drawing, by Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI (registered trademark)), Mobile High-Definition Link (MHL), or the like. The in-vehicle devices 7760 can include, for example, at least one of a mobile device and a wearable device owned by an occupant, and an information device that is carried or attached to the vehicle. The in-vehicle devices 7760 can also include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0142] The in-vehicle network I / F 7680 is an interface that mediates communication of the microcomputer 7610 with the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals and the like in accordance with a predetermined protocol supported by the communication network 7010.

[0143] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 in accordance with various programs on the basis of information obtained via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning section 7640, the beacon receiving section 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 can calculate a control target value of a driving force generation device, a steering mechanism, or a braking device on the basis of obtained information about the inside and outside of the vehicle, and output a control command to the drive system control unit 7100. For example, the microcomputer 7610 can perform cooperative control intended to realize a function of an advanced driver assistance system (ADAS) including collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, maintenance of a vehicle speed of driving, warning of a vehicle collision, warning of deviation of the vehicle from a lane, and the like. In addition, the microcomputer 7610 can perform cooperative control intended for automatic driving, which automatically drives the vehicle without depending on an operation of a driver or the like, by controlling the driving force generation device, the steering mechanism, the braking device, and the like on the basis of obtained information about the environment around the vehicle.

[0144] The microcomputer 7610 can generate three-dimensional distance information between the vehicle and an object such as a surrounding structure, a person, or the like on the basis of information obtained via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning section 7640, the beacon receiving section 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680, and generate local map information including information about the surroundings of the current position of the vehicle. Furthermore, the microcomputer 7610 can predict a danger such as a collision of the vehicle, approach of a pedestrian, or the like, entry into a closed road, and the like on the basis of the obtained information, and generate a warning signal. The warning signal may, for example, be a signal for generating a warning sound or lighting a warning lamp.

[0145] The sound / image output section 7670 transmits an output signal of at least one of a sound and an image to an output device capable of visually or aurally notifying information to an occupant of the vehicle or outside of the vehicle. In Figure 1In the example shown in FIG. 7, the audio speaker 7710, the display section 7720, and the instrument panel 7730 are shown as output devices. The display section 7720 can include at least one of an on-board display and a head-up display, for example. The display section 7720 can have an augmented reality (AR) display function. The output devices can be different from these devices, and can be another device such as headphones, a wearable device such as a glasses-type display worn by an occupant, a projector, a lamp, or the like. In the case where the output device is a display device, the display device visually displays a result obtained through various processing performed by the microcomputer 7610 or information received from another control unit, in various forms such as text, images, tables, graphs, and the like. Further, in the case where the output device is an audio output device, the audio output device converts an audio signal composed of reproduced audio data or sound data and the like into an analog signal, and outputs the analog signal auditorily.

[0146] Incidentally, in the above description, the control units are described as being separate from each other. However, the control units can be integrated into one control unit. Alternatively, each individual control unit can include a plurality of control units. Further, the vehicle control system 7000 can include another control unit not shown in the drawing. In addition, part or all of the functions performed by one of the control units in the above description can be assigned to another control unit. That is, as long as information is transmitted and received via the communication network 7010, predetermined arithmetic processing can be performed by any control unit. Similarly, a sensor or a device connected to one of the control units can be connected to another control unit, and a plurality of control units can transmit and receive detection information to and from each other via the communication network 7010. Figure 1 Incidentally, the control units described with reference to

[0147] the above description can be implemented in any control unit or the like. Further, a computer-readable recording medium storing such a computer program can be provided. The recording medium is a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory, for example. Further, the above-described computer program can be distributed via a network, for example, without using a recording medium. Figure 18 In the above-described vehicle control system 7000, the imaging element 10 according to the present embodiment described with reference to

[0148] the above description can be applied to the integrated control unit 7600 in the application Figure 1 In the above-described vehicle control system 7000, the imaging element 10 according to the present embodiment described with reference to Figure 18 the above description can be applied to the integrated control unit 7600 in the application In the imaging device, by applying the technology according to the present disclosure to the imaging section 7410, for example, the image recognition system can be simplified and the lead time can be shortened.

[0149] Further, referring to Figure 1 At least some components of the imaging element 10 according to the present embodiment described above can be implemented in a module (for example, an integrated circuit module including one chip) of the integrated control unit 7600 in Figure 18 Alternatively, the imaging element 10 according to the present embodiment described above can be implemented by ​ ​ the plurality of control units of the vehicle control system 7000 in

[0150] While the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments themselves and various modifications can be made without departing from the spirit of the present disclosure. Further, components of different embodiments and variations can be combined as appropriate.

[0151] Further, the effects described in each of the embodiments described in this specification are merely examples and the disclosure is not limited thereto. Other effects can be exhibited.

[0152] <7. Appendix>

[0153] Incidentally, the present technology can also have the following configuration. (1)

[0155] An imaging element comprising:

[0156] an imaging section in which a plurality of pixels each including a photoelectric conversion element are arranged in a matrix;

[0157] a convolution circuit that performs convolution processing on a plurality of pixel signals that are analog signals respectively output from the plurality of pixels, based on a convolution coefficient; and

[0158] a pooling circuit that performs pooling processing on the plurality of pixel signals that have undergone the convolution processing. (2)

[0160] The imaging element according to the above (1), further comprising:

[0161] a full integration processing section that performs full integration processing on the plurality of pixel signals that have undergone the pooling processing. (3)

[0163] The imaging element according to the above (1) or (2),

[0164] wherein the convolution circuit includes:

[0165] a horizontal scanning circuit that selects a plurality of pixels from which pixel signals are output, based on the convolution coefficient; and

[0166] ​An addition circuit adds a plurality of pixel signals respectively output from a plurality of pixels which have been selected, based on a convolution coefficient. (4)

[0168] The imaging device according to (3) above,

[0169] The addition circuit includes:

[0170] A multiplexer and a plurality of capacitors for setting the convolution coefficient. (5)

[0172] The imaging device according to (3) above,

[0173] The addition circuit includes:

[0174] A plurality of amplifiers for setting the convolution coefficient. (6)

[0176] The imaging device according to any one of (1) to (5) above, further comprising:

[0177] A storage section that stores the convolution coefficient. (7)

[0179] The imaging device according to any one of (1) to (6) above, further comprising:

[0180] A control section that controls the convolution circuit and the pooling circuit based on the convolution coefficient. (8)

[0182] The imaging device according to (7) above,

[0183] The control section restricts or allows the convolution processing and the pooling processing in response to an input signal input to the control section. (9)

[0185] The imaging device according to any one of (1) to (8) above,

[0186] The convolution coefficient is a convolution filter, and

[0187] The convolution circuit shifts the convolution filter by a predetermined shift amount and performs the convolution processing. (10)

[0189] The imaging device according to any one of (1) to (9) above,

[0190] The convolution coefficient is a convolution filter including zero. (11)

[0192] An imaging device comprising:

[0193] imaging element; and

[0194] an optical system that forms an image of light on a light-receiving surface of the imaging element,

[0195] wherein the imaging element includes:

[0196] an imaging section in which a plurality of pixels each including a photoelectric conversion element are arranged in a matrix;

[0197] a convolution circuit that performs convolution processing on a plurality of pixel signals that are analog signals respectively output from the plurality of pixels, based on a convolution coefficient; and

[0198] a pooling circuit that performs pooling processing on the plurality of pixel signals that have undergone the convolution processing. (12)

[0200] An imaging apparatus including:

[0201] the imaging element according to any one of (1) to (10) above; and

[0202] an optical system that forms an image of light on a light-receiving surface of the imaging element.

[0203] Reference numeral list

[0204] 10 imaging element

[0205] 11 imaging section

[0206] 12 vertical scanning circuit

[0207] 13 horizontal scanning circuit

[0208] 13a multiplexer

[0209] 14 capacitive addition circuit

[0210] 14a multiplexer

[0211] 14b capacitor

[0212] 14c switch

[0213] 15 AD conversion circuit

[0214] 16 fully connected processing section

[0215] 17 storage section

[0216] 17a register

[0217] 18 control section

[0218] 20 convolution circuit

[0219] 20A convolution circuit

[0220] 110 pixel

[0221] 121 pixel signal line

[0222] 122 vertical signal line

[0223] 150 pooling circuit

[0224] 300 imaging device

[0225] 301 optical system

Claims

1. An imaging element, comprising: An imaging unit, wherein multiple pixels, each including a photoelectric conversion element, are arranged in a matrix; A convolution circuit performs convolution processing on multiple pixel signals based on convolution coefficients, wherein the multiple pixel signals are analog signals output from the multiple pixels respectively; The convolution circuit includes: A horizontal scanning circuit selects a plurality of pixels from a plurality of pixels to output the pixel signal based on the convolution coefficients; and An adder circuit adds multiple pixel signals output from multiple selected pixels based on the convolution coefficients. The addition circuit includes: A multiplexer and multiple capacitors for setting the convolution coefficients; and A pooling circuit performs pooling processing on the plurality of pixel signals that have already undergone the convolution processing.

2. The imaging element according to claim 1, further comprising: The fully integrated processing unit performs fully integrated processing on the multiple pixel signals that have already undergone the pooling process.

3. The imaging element according to claim 1, in, The adder circuit includes: Multiple amplifiers are used to set the convolution coefficients.

4. The imaging element according to claim 1, further comprising: The storage unit stores the convolution coefficients.

5. The imaging element according to claim 1, further comprising: The control unit controls the convolution circuit and the pooling circuit based on the convolution coefficients.

6. The imaging element according to claim 5, in, The control unit restricts or allows the convolution and pooling processes in response to input signals.

7. The imaging element according to claim 1, in, The convolution coefficients are convolution filters, and The convolution circuit offsets the convolution filter by a predetermined shift amount and performs the convolution process.

8. The imaging element according to claim 1, in, The convolution coefficients are convolution filters that include zeros.

9. An imaging device, comprising: Imaging elements; as well as An optical system that forms an image of light on the light-receiving surface of the imaging element. The imaging element includes: An imaging unit, wherein each of the multiple pixels, including a photoelectric conversion element, is arranged in a matrix; A convolution circuit performs convolution processing on multiple pixel signals based on convolution coefficients, wherein the multiple pixel signals are analog signals output from the multiple pixels respectively; The convolution circuit includes: A horizontal scanning circuit selects a plurality of pixels from a plurality of pixels to output the pixel signal based on the convolution coefficients; and An adder circuit adds multiple pixel signals output from multiple selected pixels based on the convolution coefficients. The addition circuit includes: A multiplexer and multiple capacitors for setting the convolution coefficients; and A pooling circuit performs pooling processing on the plurality of pixel signals that have already undergone the convolution processing.

Images