Image pickup element and image pickup device

Abstract

An image capturing element (10) according to an embodiment of the present disclosure is provided with: an image capturing unit (11) in which a plurality of pixels (110) individually including a photoelectric conversion element are arranged in a matrix; a convolution circuit (20, 20A) which performs convolution processing on the basis of a convolution coefficient, with respect to a plurality of pixel signals, which are analog signals output from each of the plurality of pixels (110); and a pooling circuit (150) which performs pooling processing with respect to the plurality of pixel signals for which the convolution processing has been performed.

Description

[Technical Field]

[0001] The present disclosure relates to an imaging element and an imaging device. [Background technology]

[0002] Convolutional neural networks (CNNs) are widely used in the field of image recognition. CNNs process an input image, passing through convolutional and pooling layers before reaching a fully connected layer (FC layer). For example, when outputting an image from a contact image sensor (CIS) and performing image recognition using a CNN (e.g., character recognition, object recognition, etc.), a system other than the CIS is required. Furthermore, depending on the processing content, CNNs may require a high-spec GPU (Graphics Processing Unit), which raises concerns that the image recognition system may become large when attempting to achieve real-time processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-227695 Summary of the Invention [Problem to be solved by the invention]

[0004] To simplify the image recognition system, it is conceivable to implement CNN within the CIS logic processing. However, convolution and pooling operations become a bottleneck, making it difficult to complete the processing in a realistic timeframe, and lengthening the lead time for image recognition. For this reason, there is a need to shorten the lead time for image recognition while simplifying the image recognition system.

[0005] Therefore, the present disclosure provides an imaging element and an imaging device that can simplify the image recognition system and shorten the lead time. [Means for solving the problem]

[0006] An imaging element according to one embodiment of the present disclosure includes an imaging section in which a plurality of pixels, each including a photoelectric conversion element, are arranged in a matrix, a convolution circuit that performs convolution processing on pixel signals, which are analog signals output from the plurality of pixels, based on convolution coefficients, and a pooling circuit that performs pooling processing on the plurality of pixel signals that have been subjected to the convolution processing. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing a schematic configuration example of an imaging element according to a first embodiment. [Figure 2] 1 is a circuit diagram illustrating a schematic configuration example of a pixel according to a first embodiment. [Figure 3] FIG. 2 is a block diagram showing the flow of CNN operation of the image sensor according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing a flow of CNN learning related to the CNN operation shown in FIG. 3. [Figure 5] 4 is a block diagram showing the flow of a normal image output operation of the imaging element according to the first embodiment. FIG. [Figure 6] FIG. 10 is a block diagram showing the flow of a modified example of the CNN operation of the image sensor according to the first embodiment. [Figure 7] FIG. 7 is a block diagram showing a flow of CNN learning related to the CNN operation shown in FIG. 6. [Figure 8] FIG. 1 is a diagram illustrating an example of a schematic configuration of a convolution circuit based on a convolution filter according to a first embodiment. [Figure 9] 5A and 5B are diagrams illustrating an example of the operation of a convolution circuit during normal image output processing according to the first embodiment. [Figure 10] 4A and 4B are diagrams illustrating an example of the operation of the convolution circuit during convolution processing according to the first embodiment. [Figure 11] 1 is a diagram illustrating an example of a schematic configuration of an AD conversion circuit including a pooling circuit according to a first embodiment. [Figure 12]FIG. 10 is a diagram illustrating an example of a schematic configuration of a convolution circuit based on a convolution filter according to a second embodiment. [Figure 13] FIG. 10 is a diagram for explaining sliding of a convolution filter according to the third embodiment. [Figure 14] FIG. 11 is a first diagram illustrating an example of the operation of the convolution circuit during convolution processing according to the third embodiment. [Figure 15] FIG. 11 is a second diagram illustrating an example of the operation of the convolution circuit during convolution processing according to the third embodiment. [Figure 16] 1A to 1C are diagrams illustrating examples of use of the imaging element according to each embodiment. [Figure 17] FIG. 1 is a diagram illustrating an example of a schematic configuration of an imaging device. [Figure 18] 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; [Figure 19] FIG. 2 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The present disclosure will be described in the following order. 1. First embodiment 1-1. Example of the general configuration of an image sensor 1-2. Example of pixel configuration 1-3. Example of CNN operation and normal image output operation of image sensor 1-4. Example of a schematic configuration of a convolution circuit 1-5. Example of a schematic configuration of an AD conversion circuit including a pooling circuit 1-6. Actions and Effects 2. Second embodiment 3. Third embodiment 4. Other Embodiments 5. Application Examples 6. Application Examples 7. Supplementary Notes

[0010] <1. First embodiment> <1-1. Example of the general configuration of an imaging device> FIG. 1 is a block diagram showing a schematic configuration example of an image sensor 10 according to a first embodiment. As shown in FIG. 1, the image sensor 10 includes an imaging section (pixel array section) 11, a vertical scanning circuit 12, a horizontal scanning circuit 13, a capacitance addition circuit (addition circuit) 14, an AD (Analog to Digital) conversion circuit 15, a full-coupling processing section 16, a storage section 17, and a control section 18. The image sensor 10 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 10 is incorporated into various image sensing devices such as a CIS.

[0011] The imaging unit 11 has a plurality of pixels 110 capable of photoelectric conversion. These pixels 110 are arranged in a two-dimensional grid in the vertical direction (column direction) and the horizontal direction (row direction). In FIG. 1 , the vertical arrangement is referred to as a column, and the horizontal arrangement is referred to as a row. Each row is also called a line, and each column is also called a column. To each pixel 110, a pixel signal line 121 is connected for each row, and a vertical signal line 122 is connected for each column. Each pixel signal line 121 is connected to a vertical scanning circuit 12, and each vertical signal line 122 is connected to a horizontal scanning circuit 13. For example, the pixel signal line 121 or the vertical signal line 122 includes a plurality of signal lines.

[0012] The vertical scanning circuit 12, under the control of the control unit 18, transmits various signals such as drive pulses for reading pixel signals from the pixels 110 to the imaging unit 11 via each pixel signal line 121. For example, the vertical scanning circuit 12 supplies various signals such as drive pulses to each pixel 110 for each line via the pixel signal line 121, and causes each pixel 110 to output a pixel signal to a vertical signal line 122. In other words, each pixel 110 is driven for each line by the drive signal supplied from the vertical scanning circuit 12 via the pixel signal line 121. The vertical scanning circuit 12 includes, for example, a shift register, an address decoder, etc.

[0013] The horizontal scanning circuit 13, under the control of the control unit 18, scans pixel signals output (read) from each pixel 110 in the horizontal direction (line direction) and outputs each pixel signal to the capacitance adding circuit 14. Also, under the control of the control unit 18, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitance adding circuit 14 depending on the processing to be performed. For example, when performing normal image output processing to output a normal image, the horizontal scanning circuit 13 connects all vertical signal lines 122 and the capacitance adding circuit 14. Also, when performing CNN processing, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitance adding circuit 14 based on convolution coefficients stored in the storage unit 17 (this will be described in detail later).

[0014] The capacitive addition circuit 14 outputs pixel signals output from each pixel 110 to an AD conversion circuit (ADC) 15 according to the processing to be performed under the control of the control unit 18. For example, when performing normal image output processing, the capacitive addition circuit 14 outputs each pixel signal output from the horizontal scanning circuit 13 directly to the AD conversion circuit 15 (bypass). When performing CNN processing, the capacitive addition circuit 14 adds (e.g., weighted addition) each pixel signal output from the horizontal scanning circuit 13 based on the convolution coefficients stored in the storage unit 17 and outputs the result to the AD conversion circuit 15 (details will be described later).

[0015] Here, the horizontal scanning circuit 13 and the capacitance adding circuit 14 function as a convolution circuit 20 that performs convolution processing on individual pixel signals output from each pixel 110. Normal CNN processing is a process in which an input image passes through a convolution layer and a pooling layer and reaches a fully connected layer (FC layer). In the convolution layer, features are extracted from the input image based on convolution coefficients (convolution filters). For example, a convolution filter is sequentially applied to the input image to extract features. The processing related to this convolution layer (convolution processing) is performed analogically by the convolution circuit 20. The convolution circuit 20 is an analog circuit.

[0016] The AD conversion circuit 15 performs AD conversion processing on each pixel signal output from the capacitance addition circuit 14, i.e., the pixel signal output from each pixel 110, and outputs each pixel signal, which is a digital signal, to the full connection processing unit 16. As the AD conversion circuit 15, various AD conversion circuits can be used.

[0017] The AD conversion circuit 15 includes a pooling circuit 150. The pooling circuit 150 performs pooling processing on pixel signals output from each pixel 110 according to the processing to be performed under the control of the control unit 18. For example, when performing normal image output processing, the pooling circuit 150 outputs each pixel signal as is to the fully connected processing unit 16 (bypass). When performing CNN processing, the AD conversion circuit 15 performs pooling processing on each pixel signal and outputs each pixel signal that has undergone pooling processing to the fully connected processing unit 16. The pooling circuit 150 performs pooling processing on each pixel signal that has undergone convolution processing (this will be described in detail later).

[0018] In normal CNN processing, the pooling layer extracts the maximum or average value from a predetermined number of windows (e.g., 3 × 3) and extracts features from the input image. At this time, the amount of data is reduced. The processing related to this pooling layer (convolution processing) is performed analogically by the pooling circuit 150. The pooling circuit 150 and the AD conversion circuit 15 are analog circuits.

[0019] The all-connected processor 16 outputs each pixel signal, which is a digital signal output from the AD conversion circuit 15, to the outside of the element (outside the sensor) according to the processing to be performed under the control of the control unit 18. For example, when performing normal image output processing, the all-connected processor 16 outputs each pixel signal as is to the outside of the element (bypass). Furthermore, when performing CNN processing, the all-connected processor 16 performs all-connected processing on each pixel signal and outputs each pixel signal that has been subjected to all-connected processing to the outside of the element.

[0020] In normal CNN processing, the fully connected layer aggregates outputs from the pooling layer. Processing related to this fully connected layer (fully connected processing) is logically executed by the fully connected processing unit 16. The fully connected processing unit 16 is, for example, a digital circuit (logic circuit).

[0021] The pixel signals output from the full connection processor 16 are input to an external device such as a signal processor, etc. In the external device, for example, a digital pixel signal, i.e., pixel data, is generated, and various processes are performed on the pixel data to finally generate image data.

[0022] The storage unit 17 stores various data such as convolution coefficients (convolution filters). For example, a RAM (Random Access Memory), a flash memory, etc. are used as the storage unit 17. The convolution coefficients read from the storage unit 17 are temporarily set in a register 17a.

[0023] The control unit 18 controls each unit, such as the vertical scanning circuit 12, the horizontal scanning circuit 13, the capacitance adding circuit 14, the pooling circuit 150, and the fully connected processing unit 16. For example, the control unit 18 controls the horizontal scanning circuit 13, the capacitance adding circuit 14, etc. based on the convolution coefficients set in the register 18a. For example, a processor such as a CPU (Central Processing Unit) is used as the control unit 18. The CPU includes a ROM (Read Only Memory), a RAM, etc., and controls the operation of each circuit and unit according to a program pre-stored in the ROM, using the RAM as a work memory.

[0024] <1-2. Example of pixel configuration> 2 is a diagram showing a schematic configuration example of a pixel 110 according to the first embodiment. As shown in FIG. 2, the pixel 110 has a photoelectric conversion element 111, a trigger transistor 112, a reset transistor 114, an amplification transistor 115, and a selection transistor 116. For example, a PN junction photodiode is used as the photoelectric conversion element 111. Furthermore, for example, an N-type MOS (Metal Oxide Semiconductor) transistor is used as the trigger transistor 112, the reset transistor 114, the amplification transistor 115, and the selection transistor 116.

[0025] A pixel signal line 121 is connected to the pixel 110. The pixel signal line 121 supplies a reset pulse RST, a transfer pulse TRG, and a selection signal SEL to the pixel 110. For this reason, the pixel signal line 121 includes a plurality of signal lines.

[0026] The photoelectric conversion element 111 photoelectrically converts incident light into an electric charge (electrons in this case) in an amount corresponding to the amount of light. The cathode of the photoelectric conversion element 111 is connected to ground, and the anode of the photoelectric conversion element 111 is connected to the drain of the trigger transistor 112.

[0027] The source of the trigger transistor 112 is connected to a floating diffusion layer 113. A transfer pulse TRG is supplied to the gate of the trigger transistor 112. The trigger transistor 112 is turned on (closed) when the transfer pulse TRG is high, and is turned off (open) when the transfer pulse TRG is low. When the trigger transistor 112 is on, the charge output from the photoelectric conversion element 111 is supplied to the floating diffusion layer 113. The floating diffusion layer 113 accumulates the charge supplied from the photoelectric conversion element 111. The floating diffusion layer 113 generates a voltage according to the amount of accumulated charge.

[0028] The source of the reset transistor 114 is connected to the floating diffusion layer 113. The drain of the reset transistor 114 is connected to a power supply VDD for the pixel 110. A reset pulse RST is supplied to the gate of the reset transistor 114. The reset transistor 114 is turned on when the reset pulse RST is high and turned off when the reset pulse RST is low.

[0029] The gate of the amplification transistor 115 is connected to the floating diffusion layer 113. A power supply VDD is connected to the drain of the amplification transistor 115, and the drain of the selection transistor 116 is connected to the source of the amplification transistor 115. The source of the selection transistor 116 is connected to a vertical signal line (VSL) 122. A selection signal SEL is supplied to the gate of the selection transistor 116. The selection transistor 116 is turned on when the selection signal SEL is high and turned off when it is low.

[0030] In this configuration, in the initial state, the selection signal SEL, reset pulse RST, and transfer pulse TRG are all low. Furthermore, the photoelectric conversion element 111 is exposed to light, and the trigger transistor 112 is turned off by the low-state transfer pulse TRG. Therefore, charges generated by exposure are accumulated in the photoelectric conversion element 111.

[0031] When the selection signal SEL is set to a high state, the selection transistor 116 is turned on. The reset pulse RST is set to a high state, and the charge in the floating diffusion layer 113 is discharged to the power supply VDD. This resets the potential of the floating diffusion layer 113 to a predetermined potential. A predetermined time after the reset pulse RST is returned to a low state, the transfer pulse TRG is set to a high state. The charge accumulated in the photoelectric conversion element 111 by exposure is supplied to and accumulated in the floating diffusion layer 113. A voltage corresponding to the charge accumulated in the floating diffusion layer 113 is generated, amplified by the amplification transistor 115, and output to the vertical signal line 122 as a pixel signal via the selection transistor 116.

[0032] <1-3. Example of CNN operation of image sensor and example of normal image output operation> FIG. 3 is a block diagram showing the flow of CNN operation of the image sensor 10 according to the first embodiment. In CNN operation, as shown in FIG. 3, each pixel signal (analog signal) is obtained by the image sensor 11. The horizontal scanning circuit 13 and the capacitance adding circuit 14 (convolution circuit 20) perform convolution processing on each pixel signal based on convolution coefficients (convolution filters). The pooling circuit 150 in the AD conversion circuit 15 performs pooling processing on each pixel signal that has undergone convolution processing. The fully connected processing unit 16 performs fully connected processing on each pixel signal that has undergone pooling processing, and each pixel signal (pixel signal and classification label) that has undergone CNN processing is output. The image sensor 10 is capable of completing CNN processing within the sensor. Note that there are multiple types of convolution filters, and the feature amount extracted varies depending on the convolution filter used.

[0033] FIG. 4 is a block diagram showing the flow of CNN learning related to the CNN operation shown in FIG. 3. To realize the CNN operation shown in FIG. 3, as shown in FIG. 4, a CNN is trained in advance in a sensor-external CNN 51, which is a system outside the sensor, and convolution coefficients and fully connected processing coefficients are stored in a storage unit 17. The sensor-external CNN 51 can train the CNN (convolution layer → pooling layer → fully connected layer) based on training data and labels 52 and acquire convolution coefficients and fully connected processing coefficients. The convolution circuit 20 performs convolution processing based on the convolution coefficients stored in the storage unit 17. The fully connected processing unit 16 also performs fully connected processing based on the fully connected processing coefficients stored in the storage unit 17. In the examples of FIGS. 3 and 4, the convolution processing, pooling processing, and fully connected processing are performed within the sensor. The sensor-external CNN 51 may be configured using hardware and / or software.

[0034] 5 is a block diagram showing the flow of the normal image output operation of the image sensor 10 according to the first embodiment. In the normal image output operation, as shown in FIG. 5, the image pickup unit 11, horizontal scanning circuit 13, and AD conversion circuit 15 (except for the pooling circuit 150) operate to output a normal image (to output all pixels). All CNN-related processing is bypassed, and the capacitance addition circuit 14, pooling circuit 150, and full connection processing unit 16 do not operate. As a result, a normal image that has not undergone CNN processing is output.

[0035] 3 and 4, the full connection processor 16 is provided inside the sensor (image sensor 10), but this is not limiting. FIG. 6 is a block diagram showing a flow of a modified example of the CNN operation of the image sensor 10 according to the first embodiment. As shown in FIG. 6, instead of the full connection processor 16, an external CNN 50 that performs full connection processing may be provided outside the sensor. In other words, the full connection processing may be performed outside the sensor.

[0036] Here, Fig. 7 is a block diagram showing the flow of CNN learning related to the CNN operation shown in Fig. 6. To realize the CNN operation shown in Fig. 6, as shown in Fig. 7, CNN is trained in advance by a sensor-external CNN 51, which is a system outside the sensor, and the convolution coefficients (first layer) are stored in a storage unit 17. Then, CNN processing from the first layer onwards is executed by the sensor-external CNN 50. In this case, the convolution processing of the first layer is executed within the sensor, and subsequent processing is executed outside the sensor.

[0037] Note that the CNN operation and the normal image output operation may be switchable by the user, for example. The user operates an input unit (for example, a switch, a button, or the like), which causes the control unit 18 to switch between the CNN operation and the normal image output operation. The input unit is electrically connected to the control unit 18. This input unit accepts an input operation by the user, and sends an input signal (a switching instruction signal) to the control unit 18 in response to the input operation. The control unit 18 switches between the CNN operation and the normal image output operation in response to the input signal.

[0038] For example, in response to an input signal for restricting CNN operation, the control unit 18 switches the operation mode from CNN operation to normal image output operation and restricts (prohibits) convolution processing and pooling processing. On the other hand, in response to an input signal for permitting CNN operation, the control unit 18 switches the operation mode from normal image output operation to CNN operation and permits convolution processing and pooling processing. This gives the user the option to select whether or not to perform CNN operation, thereby improving user convenience.

[0039] <1-4. Example of a schematic configuration of a convolution circuit> 8 is a diagram showing a schematic configuration example of a convolution circuit 20 based on a convolution filter according to the first embodiment. As shown in FIG. 8, the horizontal scanning circuit 13 has a multiplexer 13a. The vertical signal lines 122 for each column are connected to the multiplexer 13a. The multiplexer 13a selects which column each vertical signal line 122 is connected to by switching (VSL connection / disconnection switch). The multiplexer 13a is controlled by the control unit 18.

[0040] The capacitance addition circuit 14 includes a plurality of multiplexers 14a, a plurality of capacitors 14b, and a plurality of switches 14c. Each multiplexer 14a is connected to a multiplexer 13a of the horizontal scanning circuit 13 via a predetermined number (three in FIG. 8) of vertical signal lines 122. One end of each capacitor 14b is connected to the corresponding multiplexer 14a, and the other end is connected to one signal line 122a for each multiplexer 14a. Each multiplexer 14a changes the connection of the capacitors 14b to set the addition ratio (addition capacitance setting). These multiplexers 14a are controlled by the control unit 18.

[0041] A signal line 122a is provided for each multiplexer 14a. These signal lines 122a are connected to adjacent signal lines 122a via connection lines 122b. Each switch 14c is provided individually for each signal line 122a and also for each connection line 122b. The on / off of these switches 14c is controlled by the control unit 18.

[0042] 9 is a diagram showing an example of the operation of the convolution circuit 20 during normal image output processing according to the first embodiment. In normal image output processing, as shown in FIG. 9, the horizontal scanning circuit 13 does not disconnect any of the vertical signal lines 122 but connects them to the capacitance adding circuit 14. The capacitance adding circuit 14 also connects all of the vertical signal lines 122 to the signal lines 122a via the multiplexers 14a and the capacitors 14b. At this time, the switches 14c do not connect the connection line 122b but connect all of the signal lines 122a.

[0043] FIG. 10 is a diagram illustrating an example of the operation of the convolution circuit 20 during convolution processing according to the first embodiment. In the convolution processing, as shown in FIG. 10, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and each capacitance adding circuit 14 using a multiplexer 13a based on a convolution filter. The capacitance adding circuit 14 also controls the connection between each vertical signal line 122 and each signal line 122a using each multiplexer 14a and each capacitor 14b based on a convolution filter. At this time, each switch 14c groups three adjacent signal lines 122a together, connects each group to a connecting line 122b (horizontal connection), and connects the central signal line 122a. Similar connection processing is performed for each pixel 110 other than the pixels 110 illustrated in FIG. 10.

[0044] In the example of FIG. 10, the 3×3 convolution filter has the following arrangement: row 1: 2, 0, 1; row 2: 0, 1, 0; row 3: 1, 0, 1. This convolution filter (the number of rows and columns and each value) is realized by a horizontal scanning circuit 13 and a capacitance adding circuit 14. This capacitance adding circuit 14 has multiplexers 14a and capacitors 14b for setting convolution coefficients. For example, in the 3×3 convolution filter of FIG. 10, the value in the first row and first column is 2. The pixel 110 (the upper left pixel) in FIG. 10 is connected to two capacitors 14b via a vertical signal line 122. Similarly, the value in the second row and first column is 0, and the pixel 110 in the second row and first column in FIG. 10 is not connected to any capacitor 14b. The value in the third row and first column is 1, and the pixel 110 in the third row and first column in FIG. 10 is connected to one capacitor 14b. In the second and third rows, the pixels 110 are similarly connected / disconnected to the capacitors 14b.

[0045] 10, a convolution filter including zeros (a zero filter) can be realized by the horizontal scanning circuit 13 and the capacitance adding circuit 14. This makes it possible to realize a zero filter with a simple configuration. Note that the greater the number of capacitors 14b in the capacitance adding circuit 14, the greater the degree of freedom in the convolution coefficient (convolution filter).

[0046] <1-5. Example of a schematic configuration of an AD conversion circuit including a pooling circuit> FIG. 11 is a diagram illustrating a schematic configuration example of an AD conversion circuit 15 including a pooling circuit 150 according to the first embodiment. As illustrated in FIG. 11, the AD conversion circuit 15 includes a plurality of comparators 15a, a multiplexer 15b, and a plurality of counters 15c (CN). For example, an AD conversion circuit using a voltage-controlled oscillator (VCO) is used as this AD conversion circuit 15. Each comparator 15a is provided for each signal line 122a. The multiplexer 15b is connected to each signal line 122a, and each counter 15c is connected to the subsequent stage of this multiplexer 15b (downstream in the signal flow).

[0047] The pooling circuit 150 includes an OR circuit 150a. This OR circuit 150a is provided between the multiplexer 15b and the counter 15c. By using the OR circuit 150a, the pooling circuit 150 performs maximum value pooling processing (Max pooling processing) on ​​the pixel signals output from each comparator 15a. However, the pooling processing is not limited to maximum value pooling processing, and for example, average value pooling processing may also be used.

[0048] When performing normal image output processing, the multiplexer 15b sends the pixel signals output from each comparator 15a directly to the counter 15c (CN). When performing CNN processing, the multiplexer 15b sends the pixel signals output from each comparator 15a to the OR circuit 150a. The OR circuit 150a performs maximum value pooling processing on each pixel signal, and sends each image signal after the pooling processing to the counter 15c (CN). Each pixel signal output from the counter 15c becomes a digital signal. The multiplexer 15b is controlled by the control unit 18.

[0049] <1-6. Actions and Effects> As described above, according to the first embodiment, by incorporating the convolution processing and pooling processing of the CNN processing into the image sensor 10, it becomes possible to perform the calculations related to the convolution processing and pooling processing in an analog manner, thereby speeding up the processing time. For example, by implementing the convolution processing and pooling processing using analog circuits, it is possible to reduce the scale of the memory, logic circuit, etc. Furthermore, it is possible to reduce the burden of logic processing and shorten the lead time. Therefore, it is possible to simplify the image recognition system and shorten the lead time.

[0050] 2. Second embodiment Next, a convolution circuit 20A according to a second embodiment will be described with reference to Fig. 12. Fig. 12 is a diagram showing a schematic configuration example of a convolution circuit 20A based on a convolution filter according to the second embodiment. The following description will focus on differences from the first embodiment, and other descriptions will be omitted.

[0051] As shown in FIG. 12, a convolution circuit 20A according to the second embodiment includes a capacitance addition circuit (addition circuit) 14A that is partially different from that of the first embodiment. The capacitance addition circuit 14A includes a plurality of amplifiers (column amplifiers) 14d for setting convolution coefficients, in addition to the plurality of capacitors 14b and the plurality of switches 14c according to the first embodiment. These amplifiers 14d are provided for each vertical signal line 122. A convolution filter is realized by changing the gain of the amplifier 14d. For example, if the numerical value of the convolution filter is 2, the gain of the amplifier 14d is doubled. Such an amplifier 14d is used to realize a convolution filter. Note that although the capacitance ratio of the capacitors 14b is fixed, the element configuration can be simplified compared to addition based on capacitance ratio.

[0052] As described above, according to the second embodiment, by using a plurality of amplifiers 14d in part of the capacitance addition circuit 14A of the convolution circuit 20A, it is possible to simplify the configuration of the image sensor 10, thereby realizing a simplification of the image recognition system. Note that according to the second embodiment, the same effects as those of the first embodiment can be obtained.

[0053] 3. Third Embodiment Next, convolution processing (including slide processing) according to the third embodiment will be described with reference to Fig. 13 to Fig. 15. The following description will focus on differences from the first embodiment, and other descriptions will be omitted.

[0054] Fig. 13 is a diagram for explaining sliding of a convolution filter according to the third embodiment. As shown in Fig. 13, in the third embodiment, the convolution filter is shifted in the horizontal direction by a predetermined amount of movement. In the example of Fig. 13, the predetermined amount of movement (stride) is 1, and the convolution filter is slid from A to B by the amount of movement = 1. Note that "slide" means shifting the convolution filter, and "stride" is a unit indicating the amount of movement by which the convolution filter is shifted.

[0055] Fig. 14 is a first diagram showing an example of the operation of the convolution circuit 20 during convolution processing according to the third embodiment. Fig. 15 is a second diagram showing an example of the operation of the convolution circuit 20 during convolution processing according to the third embodiment. As shown in Figs. 14 and 15, the horizontal scanning circuit 13 and the capacitance adding circuit 14 according to the third embodiment have the same configuration as those in the first embodiment, but are capable of performing convolution processing including slide processing.

[0056] As shown in FIG. 14, in the convolution process, the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitance adding circuit 14 based on a convolution filter. The capacitance adding circuit 14 also controls the connection between each vertical signal line 122 and each signal line 122a using each multiplexer 14a and each capacitor 14b based on the convolution filter. At this time, each switch 14c groups three adjacent signal lines 122a together, connects each group with a connection line 122b (horizontal connection), and connects the central signal line 122a. In the example of FIG. 14, the 3×3 convolution filter is "1st row: 2, 0, 1; 2nd row: 0, 1, 0; 3rd row: 1, 0, 1." This convolution filter (the number of rows and columns and each value) is realized by the horizontal scanning circuit 13 and the capacitance adding circuit 14.

[0057] As shown in FIG. 15, when the convolution filter is shifted by a shift amount of 1 (see FIG. 13), the horizontal scanning circuit 13 controls the connection between each vertical signal line 122 and the capacitance adding circuit 14 based on the convolution filter and the shift amount. The capacitance adding circuit 14 also controls the connection between each vertical signal line 122 and each signal line 122a using each multiplexer 14a and each capacitor 14b based on the convolution filter and the shift amount. At this time, each switch 14c shifts a group of three signal lines 122a by one signal line 122a, connects the connection line 122b for each group (horizontal connection), and connects the central signal line 122a. This causes the convolution filter to slide horizontally (row direction). This sliding is repeated within the imaging unit 11.

[0058] As described above, according to the third embodiment, it is possible to perform convolution processing including slide processing, thereby increasing the variety of convolution processing and improving user convenience. Note that according to the third embodiment, the same effects as those of the first embodiment can be obtained.

[0059] In the third embodiment described above, when AD conversion is performed simultaneously on convolution filter A and convolution filter B, a column circuit must be added, so a format is adopted in which AD conversion is not performed simultaneously but is performed separately. However, this is not limiting. For example, a column circuit such as a vertical column may be added so that AD conversion is performed simultaneously on convolution filter A and convolution filter B.

[0060] In the third embodiment, the convolution filters are shifted in the horizontal direction (row direction), but this is not limiting. For example, the convolution filters may be shifted in the vertical direction (column direction). In this case, too, the convolution filters can be shifted in the vertical direction by adding a column circuit such as a vertical column.

[0061] <4. Other embodiments> In the above-described embodiments, both the convolution layer and the pooling layer are each one layer, and the convolution process and the pooling process are performed only once, but this is not limiting. For example, both the convolution layer and the pooling layer may each be multiple layers, and both the convolution process and the pooling process may be repeated multiple times.

[0062] The imaging element 10 according to each of the above-described embodiments may be formed on a single chip, or the imaging element 10 may be divided and formed on multiple chips. Furthermore, the imaging element 10 may be formed as a laminated structure in which these chips are bonded together. For example, the imaging element 10 may be configured as a two-layer laminated structure in which a light-receiving chip having the imaging unit 11 and the like and a circuit chip having the memory unit 17, control unit 18, and the like are laminated one above the other.

[0063] <5. Application Examples> Next, an application example of the image sensor 10 according to each embodiment will be described. Fig. 16 is a diagram showing an example of using the image sensor 10 according to each embodiment.

[0064] The above-described image pickup device 10 can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows. For example, as shown in FIG. 16 , the imaging element 10 is used in “devices for capturing images for viewing, such as digital cameras and portable devices with camera functions,” “devices for traffic purposes, such as in-vehicle sensors for capturing images of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping and for recognizing the driver's state, surveillance cameras for monitoring moving vehicles and roads, and distance measuring sensors for measuring distances between vehicles,” “devices for home appliances such as TVs, refrigerators, and air conditioners for capturing images of user gestures and operating the device in accordance with the gestures,” “devices for medical and healthcare purposes, such as endoscopes and devices for capturing blood vessel images by receiving infrared light,” “devices for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication,” “devices for beauty purposes, such as skin measuring devices for capturing images of the skin and microscopes for capturing images of the scalp,” “devices for sports, such as action cameras and wearable cameras for sports, and “devices for agricultural purposes, such as cameras for monitoring the condition of fields and crops.”

[0065] For example, the above-described imaging element 10 can be applied to various electronic devices, such as imaging devices such as digital still cameras and digital video cameras, mobile phones with imaging functions, and other devices with imaging functions.

[0066] 17 is a block diagram showing an example configuration of an imaging device 300 as an electronic device to which the technology according to the present disclosure can be applied. The imaging device 300 includes an optical system 301, a shutter device 302, a solid-state imaging device 303, a control circuit (control unit) 304, a signal processing circuit 305, a monitor 306, and a memory 307. The imaging device 300 is capable of capturing still images and moving images.

[0067] The optical system 301 includes one or more lenses. The optical system 301 guides light from a subject (incident light) to the solid-state imaging device 303, and forms an image on the light-receiving surface of the solid-state imaging device 303.

[0068] The shutter device 302 is disposed between the optical system 301 and the solid-state imaging device 303. The shutter device 302 controls the light irradiation period and the light blocking period for the solid-state imaging device 303 under the control of the control circuit 304.

[0069] The solid-state imaging device 303 is configured, for example, by a package including the above-described imaging element 10. The solid-state imaging device 303 accumulates signal charges for a certain period of time in response to light that is imaged on the light-receiving surface via the optical system 301 and the shutter device 302. The signal charges accumulated in the solid-state imaging device 303 are transferred in accordance with a drive signal (timing signal) supplied from the control circuit 304.

[0070] The control circuit 304 outputs drive signals that control the transfer operation of the solid-state imaging device 303 and the shutter operation of the shutter device 302 , thereby driving the solid-state imaging device 303 and the shutter device 302 .

[0071] The signal processing circuit 305 performs various signal processing on the signal charges output from the solid-state imaging device 303. The image (image data) obtained by the signal processing performed by the signal processing circuit 305 is supplied to a monitor 306 to be displayed, or supplied to a memory 307 to be stored (recorded).

[0072] Even in the imaging device 300 configured in this manner, by applying the above-mentioned imaging element 10 as the solid-state imaging device 303, it is possible to simplify the image recognition system and shorten the lead time.

[0073] <6. Application Examples> Furthermore, the technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be realized as an electronic device or other device mounted on any type of moving body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, construction machinery, or agricultural machinery (tractor). For example, the technology according to the present disclosure may be applied to an endoscopic surgery system, a microsurgery system, or the like.

[0074] 18 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected via a communication network 7010. In the example shown in FIG. 18, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside-vehicle information detection unit 7400, an inside-vehicle information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units may be an in-vehicle communication network conforming to any standard, such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), or FlexRay (registered trademark).

[0075] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a storage unit that stores the programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various devices to be controlled. Each control unit includes a network I / F for communicating with other control units via a communication network 7010, and a communication I / F for communicating with devices or sensors inside and outside the vehicle via wired or wireless communication. Figure 18 illustrates the functional configuration of the integrated control unit 7600, including a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio / video output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Similarly, the other control units also include a microcomputer, a communication I / F, a storage unit, and the like.

[0076] The drivetrain control unit 7100 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 7100 functions as a control device for a driving force generating device for generating driving force for the vehicle, such as an internal combustion engine or a drive motor, a driving force transmission mechanism for transmitting driving force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device for an ABS (Antilock Brake System) or an ESC (Electronic Stability Control), etc.

[0077] A vehicle state detection unit 7110 is connected to the drivetrain control unit 7100. The vehicle state detection unit 7110 includes at least one of a gyro sensor that detects the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, or a sensor that detects the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine rotation speed, the rotation speed of the wheels, etc. The drivetrain control unit 7100 performs arithmetic processing using signals input from the vehicle state detection unit 7110, and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.

[0078] Body system control unit 7200 controls the operation of various devices mounted on the vehicle body in accordance with various programs. For example, body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches may be input to body system control unit 7200. Body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0079] The battery control unit 7300 controls the secondary battery 7310, which is the power supply source for the drive motor, in accordance with various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input to the battery control unit 7300 from a battery device equipped with the secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, and controls the temperature regulation of the secondary battery 7310 or a cooling device or the like provided in the battery device.

[0080] The outside vehicle information detection unit 7400 detects information outside the vehicle equipped with the vehicle control system 7000. For example, at least one of an imaging unit 7410 and an outside vehicle information detection unit 7420 is connected to the outside vehicle information detection unit 7400. The imaging unit 7410 includes at least one of a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The outside vehicle information detection unit 7420 includes at least one of an environmental sensor for detecting the current weather or climate, or a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle equipped with the vehicle control system 7000.

[0081] The environmental sensor may be, for example, at least one of a raindrop sensor that detects rain, a fog sensor that detects fog, a sunshine sensor that detects the degree of sunshine, and a snow sensor that detects snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. The imaging unit 7410 and the outside vehicle information detection unit 7420 may each be provided as an independent sensor or device, or may be provided as a device in which multiple sensors or devices are integrated.

[0082] 19 shows an example of the installation positions of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least one of the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 7900. The imaging unit 7910 provided on the front nose and the imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 provided on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin is mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0083] 19 shows an example of the imaging ranges of the imaging units 7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of the imaging unit 7910 provided on the front nose, imaging ranges b and c indicate the imaging ranges of the imaging units 7912 and 7914 provided on the side mirrors, respectively, and imaging range d indicates the imaging range of the imaging unit 7916 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained.

[0084] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided on the front, rear, sides, and corners of the vehicle 7900 and above the windshield inside the vehicle cabin may be, for example, ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926, and 7930 provided on the front nose, rear bumper, back door, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, LIDAR devices. These vehicle exterior information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, etc.

[0085] Returning to FIG. 18 , the explanation will be continued. The outside vehicle information detection unit 7400 causes the imaging unit 7410 to capture an image outside the vehicle and receives the captured image data. The outside vehicle information detection unit 7400 also receives detection information from the connected outside vehicle information detection unit 7420. If the outside vehicle information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the outside vehicle information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. The outside vehicle information detection unit 7400 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, text on the road, etc. based on the received information. The outside vehicle information detection unit 7400 may also perform environment recognition processing for recognizing rainfall, fog, road conditions, etc. based on the received information. The outside vehicle information detection unit 7400 may also calculate the distance to an object outside the vehicle based on the received information.

[0086] The outside vehicle information detection unit 7400 may also perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, characters on the road, etc., based on the received image data. The outside vehicle information detection unit 7400 may perform processing such as distortion correction or alignment on the received image data, and may also generate an overhead image or a panoramic image by combining image data captured by different image capturing units 7410. The outside vehicle information detection unit 7400 may also perform viewpoint conversion processing using image data captured by different image capturing units 7410.

[0087] The interior information detection unit 7500 detects information inside the vehicle. A driver state detection unit 7510 that detects the state of the driver is connected to the interior information detection unit 7500, for example. The driver state detection unit 7510 may include a camera that captures an image of the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sound from within the vehicle cabin. The biosensor is provided, for example, on the seat or steering wheel, and detects the biometric information of a passenger sitting in the seat or the driver gripping the steering wheel. The interior information detection unit 7500 may calculate the driver's level of fatigue or concentration, or determine whether the driver is dozing, based on the detection information input from the driver state detection unit 7510. The interior information detection unit 7500 may perform processing such as noise canceling on the collected audio signal.

[0088] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 in accordance with various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 may be implemented by a device that can be operated by a passenger, such as a touch panel, a button, a microphone, a switch, or a lever. Data obtained by voice recognition of a voice input through a microphone may be input to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or a personal digital assistant (PDA) that can operate the vehicle control system 7000. The input unit 7800 may be, for example, a camera, in which case the passenger can input information by gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by the passenger or the like using the input unit 7800 and outputs the input signal to the integrated control unit 7600. By operating this input unit 7800, passengers and the like input various data to the vehicle control system 7000 and instruct processing operations.

[0089] The storage unit 7690 may include a ROM (Read Only Memory) that stores various programs executed by the microcomputer, and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, etc. The storage unit 7690 may also be realized by a magnetic storage device such as an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0090] The general-purpose communication I / F 7620 is a general-purpose communication I / F that mediates communication between various devices present in the external environment 7750. The general-purpose communication I / F 7620 may implement a cellular communication protocol such as GSM (Global System of Mobile communications), WiMAX (registered trademark), LTE (Long Term Evolution), or LTE-Advanced (LTE-A), or other wireless communication protocols such as wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The general-purpose communication I / F 7620 may connect to devices (e.g., application servers or control servers) present on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via, for example, a base station or an access point. The general-purpose communication I / F 7620 may also connect to terminals present near the vehicle (e.g., terminals of drivers, pedestrians, or stores, or machine-type communication (MTC) terminals) using, for example, P2P (Peer to Peer) technology.

[0091] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol designed for use in vehicles. The dedicated communication I / F 7630 may implement a standard protocol such as WAVE (Wireless Access in Vehicle Environment), which is a combination of a lower layer IEEE802.11p and an upper layer IEEE1609, a dedicated short range communications (DSRC), or a cellular communication protocol. The dedicated communication I / F 7630 typically performs V2X communication, which is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0092] The positioning unit 7640 performs positioning by receiving, for example, GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites), and generates position information including the latitude, longitude, and altitude of the vehicle. Note that the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, PHS, or smartphone that has a positioning function.

[0093] The beacon receiver 7650 receives, for example, radio waves or electromagnetic waves transmitted from radio stations or the like installed on the road, and acquires information such as the current location, congestion, road closures, required travel time, etc. The function of the beacon receiver 7650 may be included in the dedicated communication I / F 7630 described above.

[0094] The in-vehicle device I / F 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 may establish wireless connections using wireless communication protocols such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). The in-vehicle device I / F 7660 may also establish a wired connection such as a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI (registered trademark), or a Mobile High-Definition Link (MHL)) via a connection terminal (and a cable, if necessary) not shown. The in-vehicle device 7760 may include, for example, at least one of a mobile device or a wearable device owned by a passenger, or an information device carried into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0095] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals in accordance with a predetermined protocol supported by the communication network 7010.

[0096] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 in accordance with various programs based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on acquired information inside and outside the vehicle, and output control commands to the drivetrain control unit 7100. For example, the microcomputer 7610 may perform cooperative control aimed at realizing functions of an Advanced Driver Assistance System (ADAS), including vehicle collision avoidance or impact mitigation, following driving based on the following distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc. In addition, the microcomputer 7610 may perform cooperative control for the purpose of autonomous driving, in which the vehicle travels autonomously without relying on driver operation, by controlling a driving force generating device, a steering mechanism, a braking device, etc. based on information acquired about the vehicle's surroundings.

[0097] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people, and create local map information including information about the vicinity of the vehicle's current location, based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. Furthermore, the microcomputer 7610 may predict dangers, such as a vehicle collision, the approach of a pedestrian, or entry into a closed road, based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or turning on a warning lamp.

[0098] The audio / video output unit 7670 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying vehicle occupants or the outside of the vehicle of information. In the example of FIG. 18 , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are illustrated as output devices. The display unit 7720 may include, for example, at least one of an on-board display and a head-up display. The display unit 7720 may have an AR (Augmented Reality) display function. The output device may be other devices besides these devices, such as headphones, a wearable device such as an eyeglass-type display worn by the occupant, a projector, or a lamp. When the output device is a display device, the display device visually displays results obtained by various processes performed by the microcomputer 7610 or information received from other control units in various formats, such as text, images, tables, and graphs. When the output device is an audio output device, the audio output device converts audio signals consisting of reproduced audio data or acoustic data into analog signals and audibly outputs the analog signals.

[0099] In the example shown in FIG. 18 , at least two control units connected via the communication network 7010 may be integrated into one control unit. Alternatively, each control unit may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not shown. In the above description, some or all of the functions performed by one control unit may be performed by another control unit. In other words, as long as information is transmitted and received via the communication network 7010, predetermined arithmetic processing may be performed by one of the control units. Similarly, a sensor or device connected to one control unit may be connected to another control unit, and multiple control units may transmit and receive detection information to and from each other via the communication network 7010.

[0100] A computer program for realizing each function of the image sensor 10 according to this embodiment described with reference to FIG. 1 can be implemented in any control unit or the like. A computer-readable recording medium storing such a computer program can also be provided. The recording medium can be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0101] In the vehicle control system 7000 described above, the imaging element 10 according to the present embodiment described with reference to Fig. 1 can be applied to the integrated control unit 7600 of the application example shown in Fig. 18. For example, the storage unit 17 and control unit 18 of the imaging element 10 correspond to the microcomputer 7610, storage unit 7690, and in-vehicle network I / F 7680 of the integrated control unit 7600. By applying the technology according to the present disclosure to an imaging device such as the imaging unit 7410, it is possible to simplify the image recognition system in the imaging device and shorten the lead time.

[0102] Furthermore, at least some of the components of the image sensor 10 according to this embodiment described with reference to Fig. 1 may be realized in a module (for example, an integrated circuit module configured on a single die) for the integrated control unit 7600 shown in Fig. 18. Alternatively, the image sensor 10 according to this embodiment described with reference to Fig. 1 may be realized by a plurality of control units of the vehicle control system 7000 shown in Fig. 18.

[0103] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0104] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0105] <7. Notes> The present technology can also be configured as follows. (1) an imaging unit in which a plurality of pixels each including a photoelectric conversion element are arranged in a matrix; a convolution circuit that performs convolution processing on a plurality of pixel signals, which are analog signals output from the plurality of pixels, based on convolution coefficients; a pooling circuit that performs a pooling process on the plurality of pixel signals that have been subjected to the convolution process; An imaging element comprising: (2) a total integration processing unit that performs total integration processing on the plurality of pixel signals that have been subjected to the pooling processing, The imaging device according to (1) above. (3) The convolution circuit comprises: a horizontal scanning circuit that selects, from among the plurality of pixels, a plurality of pixels from which the pixel signals are to be output based on the convolution coefficients; an adder circuit that adds the plurality of pixel signals output from the plurality of selected pixels based on the convolution coefficient; having The imaging element according to (1) or (2) above. (4) The adding circuit a multiplexer and a plurality of capacitors for setting the convolution coefficients; The imaging element according to (3) above. (5) The adding circuit a plurality of amplifiers for setting the convolution coefficients; The imaging element according to (3) above. (6) Further comprising a storage unit that stores the convolution coefficients. The imaging device according to any one of (1) to (5) above. (7) a control unit that controls the convolution circuit and the pooling circuit based on the convolution coefficients; The imaging device according to any one of (1) to (6) above. (8) The control unit restricts or permits the convolution processing and the pooling processing in response to an input signal input to the control unit. The imaging element according to (7) above. (9) the convolution coefficients are convolution filters; the convolution circuit shifts the convolution filter by a predetermined amount and performs the convolution process; The imaging device according to any one of (1) to (8) above. (10) The convolution coefficients are convolution filters including 0. The imaging device according to any one of (1) to (9) above. (11) An imaging element; an optical system that forms an image of light on the light receiving surface of the image sensor; Equipped with The imaging element is an imaging unit in which a plurality of pixels each including a photoelectric conversion element are arranged in a matrix; a convolution circuit that performs convolution processing on a plurality of pixel signals, which are analog signals output from the plurality of pixels, based on convolution coefficients; a pooling circuit that performs a pooling process on the plurality of pixel signals that have been subjected to the convolution process; An imaging device comprising: (12) The imaging element according to any one of (1) to (10) above; an optical system that forms an image of light on the light receiving surface of the image sensor; An imaging device comprising: [Explanation of symbols]

[0106] 10. Image sensor 11 Imaging unit 12 Vertical scanning circuit 13 Horizontal scanning circuit 13a Multiplexer 14 Capacitive Adder Circuit 14a Multiplexer 14b capacitor 14c switch 15 AD conversion circuit 16 Fully connected processing section 17 Memory section 17a Register 18 Control Unit 20 Convolution Circuit 20A convolution circuit 110 pixels 121 pixel signal line 122 vertical signal line 150 Pooling Circuit 300 Imaging device 301 Optical system

Claims

1. an imaging unit in which a plurality of pixels each including a photoelectric conversion element are arranged in a matrix; a convolution circuit that performs convolution processing on a plurality of pixel signals, which are analog signals output from the plurality of pixels, based on convolution coefficients; a pooling circuit that performs a pooling process on the plurality of pixel signals that have been subjected to the convolution process; Equipped with The convolution circuit comprises: a horizontal scanning circuit that selects, from among the plurality of pixels, a plurality of pixels from which the pixel signals are to be output based on the convolution coefficients; an adder circuit that adds the plurality of pixel signals output from the plurality of selected pixels based on the convolution coefficient; and The adding circuit a multiplexer and a plurality of capacitors for setting the convolution coefficients; Image sensor.

2. further comprising a full connection processing unit that performs full connection processing on the plurality of pixel signals that have been subjected to the pooling processing, The imaging device according to claim 1 .

3. Further comprising a storage unit that stores the convolution coefficients. The imaging device according to claim 1 .

4. a control unit that controls the convolution circuit and the pooling circuit based on the convolution coefficients; The imaging device according to claim 1 .

5. The control unit restricts or permits the convolution processing and the pooling processing in response to an input signal input to the control unit. The imaging device according to claim 4 .

6. the convolution coefficients are convolution filters; the convolution circuit shifts the convolution filter by a predetermined amount and performs the convolution process; The imaging device according to claim 1 .

7. The convolution coefficients are convolution filters including zeros. The imaging device according to claim 1 .

8. An imaging element; an optical system that forms an image of light on the light receiving surface of the image sensor; Equipped with The imaging element is an imaging unit in which a plurality of pixels each including a photoelectric conversion element are arranged in a matrix; a convolution circuit that performs convolution processing on a plurality of pixel signals, which are analog signals output from the plurality of pixels, based on convolution coefficients; a pooling circuit that performs a pooling process on the plurality of pixel signals that have been subjected to the convolution process; Equipped with The convolution circuit comprises: a horizontal scanning circuit that selects, from among the plurality of pixels, a plurality of pixels from which the pixel signals are to be output based on the convolution coefficients; an adder circuit that adds the plurality of pixel signals output from the plurality of selected pixels based on the convolution coefficient; and The adding circuit a multiplexer and a plurality of capacitors for setting the convolution coefficients; Imaging device.

Images