Solid-state imaging devices and electronic devices

Abstract

[Problem] A solid-state imaging device and an electronic device are provided, by which the size of a circuit for performing a neural network operation can be suppressed. [Solution] A solid-state imaging device according to one aspect of the present disclosure includes a pixel array unit and a processing unit. The pixel array unit has a plurality of first pixels that generate an electrical signal as a first pixel signal, the electrical signal having a logarithmic characteristic relative to the amount of light. The processing unit performs an arithmetic operation of a first neural network based on a plurality of first input data and a plurality of logarithmic weighting factors, the plurality of first input data being based on the plurality of first pixel signals read from the pixel array unit, the plurality of logarithmic weighting factors representing the strength of a connection between a plurality of first nodes as a logarithm.

Description

Technical Field

[0001] The present disclosure relates to a solid-state imaging device and an electronic device. Background Art

[0002] Recently, a data learning method using a neural network is widely used as a type of machine learning. Neural networks are used in every field, including applications such as image recognition of image data captured by an imaging device such as a camera.

[0003] In the case of image recognition using a neural network, data as the target of image recognition is provided for the nodes in the input layer (input nodes). In the nodes (intermediate nodes) of the intermediate layer of the latter stage as the input layer, arithmetic operations are performed based on the following items: data input from each input node; and a weighting factor indicating the strength of connection with each input node. In other words, the weighted sum of the input data and the weighting factor is calculated. In this calculation, a product-sum operation is performed, that is, the amount of operation is large. In addition, if a multiplication circuit is used for product-sum operation, the circuit scale becomes larger (that is, another problem). This applies to the nodes of each layer in the subsequent stages of the intermediate layer.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent literature 1: Daisuke Miyashita, Edward H. Lee, and Boris Murmann, "Convolutional neural networks using logarithmic data representation", CoRR, vol. abs / 1603.01025, 2016.

[0007] Non-patent document 2: D. Kim, "An Enhanced Dynamic-Range CMOS Image Sensor Using a Digital Logarithmic Single-Slope ADC", IEEE Transactions on circuit and systems 2012.

[0008] Non-patent document 3: Alex Omid-Zohoor, "Toward Always-On Mobile ObjectDetection: Energy Versus Performance Tradeoffs for Embedded HOG FeatureExtraction", IEEE Transactions on circuit and systems for video technology 2018. Summary of the invention

[0009] Technical Problems to be Solved by the Invention

[0010] The method for solving the above problem is to convert the weighting factor between the data provided to the input node and each node into a logarithm with a base of 2 (log2 conversion) (see non-patent document 1). According to this method, the operation at each node can be performed by adding in the logarithmic space, so it is not necessary to perform multiplication at each node. In other words, a multiplication circuit is not required. However, it is additionally necessary to perform a circuit for log2 conversion of the data provided to the input node. This leads to an increase in circuit scale and an increase in power consumption.

[0011] The present disclosure provides a solid-state imaging apparatus and an electronic device in which the scale of a circuit for performing arithmetic operations of a neural network is suppressed.

[0012] Solution to the problem

[0013] A solid-state imaging device according to one aspect of the present disclosure includes a pixel array unit and a processing unit. The pixel array unit has a plurality of first pixels that generate an electrical signal as a first pixel signal, the electrical signal having a logarithmic characteristic with respect to the amount of light. The processing unit performs arithmetic processing for a first neural network based on a plurality of first input data based on the plurality of first pixel signals read from the pixel array unit and a plurality of logarithmic weighting factors that logarithmically represent the strength of a connection between a plurality of first nodes.

[0014] The first pixel signal indicates a voltage value, and the voltage value is proportional to a logarithm of the current value based on a predetermined value; and the processing unit may generate the first input data by multiplying a value of the first pixel signal by the logarithm, an antilogarithm of the logarithm being a predetermined value.

[0015] The predetermined value may be a Napier constant.

[0016] The base of the logarithm whose antilog is the predetermined value may be 2.

[0017] The solid-state imaging device includes a control unit. The pixel array unit includes a plurality of second pixels, and the type of the plurality of second pixels is different from that of the plurality of first pixels. The control unit selectively performs a first reading process to read first pixel signals from the plurality of first pixels, or performs a second reading process to read a plurality of second pixel signals from the plurality of second pixels. In the case of performing the first reading process, the processing unit performs arithmetic processing of a first neural network. In the case of performing the second reading process, the processing unit performs arithmetic processing of a second neural network based on a plurality of second input data and a weighting factor, the plurality of second input data is based on the plurality of second pixel signals, and the weighting factor indicates the strength of a connection between the plurality of second nodes.

[0018] The control unit may select one of the first reading process and the second reading process according to the instruction data indicating one of the first reading process and the second reading process.

[0019] The second pixel may refer to a pixel that generates an electric signal having a linear characteristic with respect to the amount of light as a second pixel signal.

[0020] The second pixel may refer to a pixel that generates a count value of the number of incoming photons as a second pixel signal.

[0021] The second pixel may refer to a pixel that generates a charge amount indicating a phase difference between reflected light and emitted light of emitted light as a second pixel signal.

[0022] The solid-state imaging device may include a control unit. The first pixel may include: a photoelectric conversion unit; a first reading unit that reads an electrical signal having a logarithmic characteristic from the photoelectric conversion unit as a first pixel signal; and a second reading unit that reads an electrical signal having a linear characteristic from the photoelectric conversion unit as a second pixel signal; the control unit may selectively perform a first reading process to read the first pixel signal using the first reading unit, or perform a second reading process to read the second pixel signal using the second reading unit. In the case of performing the first reading process, the processing unit may perform arithmetic processing of a first neural network; and in the case of performing the second reading process, the processing unit may perform arithmetic processing of a second neural network based on a plurality of second input data and a weighting factor, the plurality of second input data being based on a plurality of second pixel signals, and the reinforcement factor representing the strength of a connection between a plurality of second nodes.

[0023] The processing unit may perform image recognition on the image indicated by the plurality of first pixel signals using arithmetic processing of the first neural network.

[0024] An electronic device according to one aspect of the present disclosure includes: an image pickup element, including a pixel array unit having a plurality of first pixels that generate an electrical signal as a first pixel signal, the electrical signal having a logarithmic characteristic relative to the amount of light; and a processing circuit configured to perform arithmetic processing of a first neural network based on a plurality of first input data and a plurality of logarithmic weighting factors, the plurality of first input data being based on the plurality of first pixel signals read from the pixel array unit, the plurality of logarithmic weighting factors representing the strength of a connection between the plurality of first nodes by logarithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 : is a block diagram depicting a configuration example of a solid-state imaging device according to Embodiment 1 of the present disclosure.

[0026] Figure 2 It is a graph schematically showing the characteristics of logarithmic pixels.

[0027] Figure 3 is a diagram depicting an example of a circuit configuration of a logarithmic pixel.

[0028] Figure 4 is a diagram schematically indicating an example of a typical structure of a neural network.

[0029] Figure 5 is a diagram for describing an example of arithmetic processing at a node.

[0030] Figure 6 is a block diagram depicting an example of a log2 addition circuit.

[0031] Figure 7 is a flowchart of an example of the operation of the solid-state imaging device according to Embodiment 1 of the present disclosure.

[0032] Figure 8 is a diagram depicting a variation of the circuit configuration of a logarithmic pixel.

[0033] Fig. 9 FIG. 4 is a diagram depicting another variation of the circuit configuration of a logarithmic pixel.

[0034] Fig.10 is a diagram depicting an example of a circuit configuration of a subtractor and a quantizer.

[0035] Fig.11 : is a block diagram depicting a configuration example of a solid-state imaging device according to Embodiment 2 of the present disclosure.

[0036] Fig.12 is a graph schematically showing the characteristics of a linear pixel.

[0037] Fig.13 is a diagram depicting an example of a circuit configuration of a linear pixel.

[0038] Fig.14 is a flowchart of an example of the operation of the solid-state imaging device according to Embodiment 2 of the present disclosure.

[0039] Fig.15 yes Fig.14 A flowchart of an example of the operation of FIG.

[0040] Fig.16 : is a block diagram depicting a configuration example of a solid-state imaging device according to Embodiment 3.

[0041] Fig.17 is a diagram depicting an example of a circuit configuration of a hybrid pixel.

[0042] Fig.18 is a diagram depicting a variation of the circuit configuration of a hybrid pixel.

[0043] Fig.19 is a diagram depicting a variation of the circuit configuration of a hybrid pixel.

[0044] Fig. 20 This is a flowchart of a variation of the operation of the solid-state imaging device according to Embodiment 3.

[0045] Fig.21 1 is a flowchart of another variation of the operation of the solid-state imaging device according to Embodiment 3.

[0046] Fig. 22 is a block diagram depicting an example of a configuration of an electronic device according to the present disclosure. DETAILED DESCRIPTION

[0047] Embodiments of the present disclosure will be described with reference to the accompanying drawings. In at least one embodiment described in the present disclosure, constituent elements included in each embodiment may be combined, and the combination is within the scope of the embodiment according to the present disclosure.

[0048] Figure 1 : is a block diagram depicting a configuration example of a solid-state imaging device according to Embodiment 1 of the present disclosure. Figure 1 The solid-state imaging device in the embodiment includes an image pickup element 10, a signal processing unit 16, a central processing unit (CPU) 17, a data storage unit 18, an arithmetic processing unit 19, and a bus 20. The image pickup element 10 includes a pixel array unit 11 having a plurality of pixels 21, a vertical drive unit 12, a column processing unit 13, a horizontal drive unit 14, and a control unit 15. The CPU 17 and the arithmetic processing unit 19 constitute a processing unit 26.

[0049] For example, Figure 1The solid-state imaging device in is a complementary metal oxide semiconductor (CMOS) type solid-state imaging device and can be used in various electronic devices. Examples of electronic devices include mobile devices with imaging functions (e.g., smartphones, tablets, portable phones), imaging devices (e.g., digital still cameras, video cameras), surveillance cameras, and vehicle-mounted cameras.

[0050] The structure of the image pickup element 10 can be a back-illuminated type, wherein the pixel array unit 11 (vertical drive line unit 12, column processing unit 13, horizontal drive unit 14, and control unit 15) and the logic circuit are arranged on the same supporting substrate and illuminated from the back of the substrate. In addition, the structure of the image pickup element 10 can be a stacked type, wherein the pixel array unit 11 and the logic circuit are manufactured separately and stacked. In this case, each stacked layer is electrically connected using a connecting portion (e.g., a via). In addition to the via, the connection can be performed by copper-copper bonding or by a bump. The structure of the image pickup element can be a stacked type, which includes a pixel array 11, a vertical drive unit 12, a column processing unit 13, a horizontal drive unit 14, a control unit 15, a signal processing unit 16, a CPU 17, a data storage unit 18, an arithmetic processing unit 19, and a bus 20.

[0051] The pixel array unit 11 includes a plurality of pixels 21 arranged in two dimensions (in a matrix form). The region in which the plurality of pixels 21 are arranged corresponds to the pixel region. The pixel 21 includes a photoelectric conversion unit, and the photoelectric conversion unit includes a photodiode (PD) and a plurality of pixel transistors. The photodiode refers to an element that converts incident light into an electrical signal (signal charge) by photoelectric conversion. For example, the plurality of pixel transistors are metal oxide semiconductor (MOS) transistors.

[0052] The photoelectric conversion unit of the pixel 21 receives light via an optical lens (not shown) and generates an electrical signal as a pixel signal, which changes logarithmically with respect to the amount of incident light. The generated electrical signal (pixel signal) is a voltage signal indicating the output voltage of the pixel 21. In the pixel 21, the current flows exponentially with respect to the voltage generated in the photodiode, so the voltage and the current are logarithmically related, and a logarithmic voltage signal is output. The pixel 21 that generates a voltage signal as a pixel signal is called a "logarithmic pixel", which has a logarithmic characteristic with respect to the amount of incident light. In the following description, the pixel 21 may be referred to as a "logarithmic pixel 21". The logarithmic pixel 21 corresponds to the example of the first pixel, and the pixel signal read from the logarithmic pixel 21 corresponds to the first pixel signal read from the first pixel. The logarithmic pixel has greater noise than a pixel having a linear characteristic (referred to as a "linear signal"), and its output signal increases linearly with respect to the amount of light, but has a wider dynamic range characteristic.

[0053] Figure 2 Graph schematically showing the relationship between the amount of incident light and the output voltage of a logarithmic pixel. As the amount of incident light increases, the output voltage increases logarithmically.

[0054] For example, when the current of a logarithmic pixel is i and its voltage is v, the relationship between i and v can be approximated by the following expression of a natural logarithm. A is a predetermined coefficient. When the logarithm is expressed as logab, a is called the "base" and b is called the "antilog". A constant term may be included on the right side of expression (1). Here, the relationship between i and v approximates the natural logarithm, but a logarithm whose base is a predetermined value other than the Napier constant (e) may be approximated.

[0055] [Mathematical formula 1]

[0056] v=A*log e i…(1)

[0057] Figure 3 An example of a circuit configuration of the logarithmic pixel 21 is indicated. Here, the configuration of one logarithmic pixel 21 is indicated as an example, but each of the other logarithmic pixels 21 has the same configuration.

[0058] The logarithmic pixel 21 includes a photodiode 31 corresponding to the photoelectric conversion unit and a plurality of pixel transistors. The plurality of transistors include an output transistor 32, an amplifier transistor 33, a bias transistor 34, an amplifier transistor 35, and a selection transistor 36. For example, the output transistor 32, the amplifier transistor 33, the amplifier transistor 35, and the selection transistor 36 are N-type transistors. For example, the bias transistor 34 is a P-type transistor. However, the conductivity type of these pixel transistors can be opposite to these examples. In this case, it is necessary to change the logic of the control signal and the voltage of the power supply line of these pixel transistors.

[0059] The drain terminal of the output transistor 32 is connected to the power supply voltage, and the source terminal thereof is connected to the cathode terminal of the photodiode 31. The anode terminal of the photodiode 31 is connected to a reference voltage. For example, the reference voltage is a ground voltage. The cathode terminal of the photodiode 31 is also connected to the gate terminal of the amplifying transistor 33. The connection node N1 of the photodiode 31 and the amplifying transistor 33 is connected to the source terminal of the output transistor 32.

[0060] The bias transistor 34 is connected in series with the amplifier transistor 33 between the power supply voltage and the reference voltage. A connection node N2 between the bias transistor 34 and the amplifier transistor 33 is connected to the gate terminal of the output transistor 32 and the gate terminal of the amplifier transistor 35. The output transistor 32 and the amplifier transistor 33 are connected to form a loop via the connection node N2 and the connection node N1.

[0061] A bias voltage Vbias is applied to a gate terminal of the bias transistor 34 .

[0062] The drain terminal of the amplification transistor 35 is connected to the power supply voltage, and the source terminal thereof is connected to the drain terminal of the selection transistor 36. The source terminal of the selection transistor 36 is connected to the vertical signal line 23. The gate terminal of the selection transistor 36 is connected to the pixel driving line 22, and the selection transistor 36 is controlled by the selection signal SEL supplied from the vertical driving unit 12.

[0063] The photodiode 31 outputs a current i that changes linearly with respect to the amount of incident light. According to the current i, a voltage is applied to the gate terminal of the amplifier transistor 33. The output transistor 32 and the amplifier transistor 33 operate as source followers in the sub-threshold region, respectively. By connecting the output transistor 32 and the amplifier transistor 33 in a loop, the voltage in the connection node N1 is fed back to the gate terminal of the output transistor 32 via the amplifier transistor 33. Thus, a voltage signal of the logarithm of the current i is generated in the connection node N2, and the generated voltage signal is provided to the amplifier transistor 35 via the buffer circuit 40. The voltage signal is amplified by the amplifier transistor 35. The amplified voltage signal is output to the vertical signal line 23 via the selection transistor 36. In this way, a voltage signal that changes logarithmically with respect to the amount of incident light is output as a pixel signal.

[0064] Figure 3 The configuration of the logarithmic pixels indicated in is an example, and the configuration of the logarithmic pixels is not limited to Figure 3 Any configuration can be used for logarithmic pixels as long as a voltage signal that changes according to the logarithm of the photoelectric current can be detected. In addition, a circuit with a specified function can be added to Figure 3 For example, a sensor circuit that detects brightness changes may be connected to the connection node N2 so as to be connected in parallel with the amplification transistor 35.

[0065] Figure 1 The vertical driving unit 12 in the vertical direction sequentially selects the pixel driving line 22 in the pixel array unit 11, and supplies a pulse (driving signal) to drive a plurality of pixels 21, which are commonly connected to the selected pixel driving line 22. For example, the vertical driving unit 12 sequentially selects the pixels 21 in the pixel array unit 11 for each row in the vertical direction, and supplies a driving signal such as a selection signal SEL and a reset signal RST mentioned later to the pixel driving line 22 of the selected row. Thus, the pixel signal generated in each pixel 21 is supplied to the column processing unit 13 via the vertical signal line 23. For example, the vertical driving unit 12 is composed of a circuit such as a shift register.

[0066] The column processing unit 13 processes the signal output from a row of pixels 21 for each pixel column. The column processing unit 13 includes a unit circuit corresponding to each pixel column. The unit circuit corresponding to each pixel column receives the signal output from the corresponding pixel in a row of pixels 21 and processes the signal. For example, the signal processing may include analog-to-digital (A / D) conversion, and may also include noise elimination and signal amplification performed before the A / D conversion.

[0067] The horizontal driving unit 14 sequentially selects the unit circuits in the column processing unit 13 by sequentially outputting horizontal scanning pulses. Thus, pixel signals are sequentially output from each unit circuit in the column processing unit 13 to the signal processing unit 16. For example, the horizontal driving unit 14 is composed of a circuit such as a shift register.

[0068] The control unit 15 is a circuit that controls the operation of each unit in the solid-state imaging device. For example, based on the vertical synchronization signal, the horizontal synchronization signal, and the master clock, the control unit 15 generates various signals such as a clock signal and a control signal, which become references for the operations of the vertical drive unit 12, the column processing unit 13, the horizontal drive unit 14, etc. The control unit 15 outputs the generated signals to the vertical drive unit 12, the column processing unit 13, the horizontal drive unit 14, etc.

[0069] The signal processing unit 16 performs various signal processing on the pixel signal supplied from the column processing unit 13. In some cases, the signal processing can only buffer the pixel signal. Other examples of signal processing include black level adjustment, column variation correction, and various digital signal processing. For example, the signal processing unit 16 is composed of a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a CPU circuit. The signal processing unit 16 is connected to the bus 20.

[0070] The CPU 17 is connected to the bus 20, and controls the signal processing unit 16, the data storage unit 18, and the arithmetic processing unit 19. The CPU 17 performs control to write the data of the pixel signal processed by the signal processing unit 16 into the data storage unit 18 as pixel data.

[0071] The data storage unit 18 internally stores the pixel data processed by the signal processing unit 16. By storing the pixel data corresponding to each pixel 21 in the data storage unit 18, the image data of one frame is stored in the data storage unit 18. The pixel data is a digital value having a predetermined bit length, and the bit length is determined by the resolution setting of the A / D conversion. In the case where the solid-state imaging device performs imaging in a predetermined cycle, the image data is acquired in a predetermined cycle and stored in the data storage unit 18.

[0072] For example, the data storage unit 18 is composed of a volatile or non-volatile memory device such as a static random access memory (SRAM), a dynamic random access memory (DRAM), a NAND flash memory, and a magnetoresistive memory. The data storage unit 18 may also be composed of a storage medium such as a register and a digital circuit other than a memory device. The data storage unit 18 is connected to the bus 20.

[0073] Under the control of the CPU 17, the arithmetic processing unit 19 performs arithmetic processing of a neural network by inputting a frame of data stored in the data storage unit 18 to recognize the image of the image data. In other words, the arithmetic processing unit 19 performs image recognition of the image based on the image signal read from each pixel 21. An example of image recognition is to recognize whether a predetermined object is captured in the image data. For example, recognize whether a person's face is captured. In some cases, the type of object captured in the image is predicted from a plurality of alternatives. For example, predict whether the object is a human, an animal, or an object. In this example, the number of alternatives is 3, but it may be 2 or 4 or more. As long as image recognition is performed based on image data, image recognition may be different from these examples. The arithmetic processing unit 19 stores data indicating the result of image recognition in the data storage unit 18. The data storage unit 18 stores data calculated by the arithmetic processing unit 19 indicating the result of image recognition.

[0074] For example, the arithmetic processing unit 19 is composed of a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a CPU circuit. The arithmetic processing unit 19 may include an internal buffer for storing data. The internal buffer is composed of any circuit such as a register, a memory, and a digital circuit. The memory may be a volatile memory or a non-volatile memory.

[0075] A part of the processing performed by the arithmetic processing unit 19 may be executed by the CPU 17 .

[0076] The data storage unit 18 stores neural network parameters used by the arithmetic processing unit 19 to perform arithmetic processing of the neural network. The parameter refers to a value generated by converting a weighting factor into a logarithm with a base of 2, and the weighting factor indicates the strength of the connection (link) between each node of the neural network after learning. Thereafter, the converted weighting factor may be referred to as a "logarithmic weighting factor" or a "log2 weighting factor". The logarithmic weighting factor refers to the strength of the connection between multiple nodes represented by a logarithm. In embodiment 1, in the arithmetic processing of the neural network, the logarithmic weighting factor is used instead of the learned weighting factor. In order to store the logarithmic weighting factor, the data storage unit 18 stores information that associates an identifier (representing the connection between each node of the neural network) with the logarithmic weighting factor. For example, the information can be represented as a table or function that returns a logarithmic weighting factor based on a given identifier.

[0077] All or part of the data stored in the above-mentioned data storage unit 18 may be stored in the internal buffer of the arithmetic processing unit 19. For example, the above-mentioned parameters may be stored in the internal buffer of the arithmetic processing unit 19. The data storage unit 18 may store the learned weighting factors.

[0078] Figure 4 : is a diagram schematically indicating an example of a typical structure of a neural network. In Embodiment 1, an example of a fully connected layered neural network in which all nodes between layers are interconnected is indicated as a neural network, but other types of neural networks such as a conventional neural network and an interconnected neural network may be used.

[0079] First, refer to Figure 4 A conventional neural network is described. The neural network includes an input layer 41, an intermediate layer (hidden layer) 42, and an output layer 43. In this example, there is one intermediate layer 42, but there may be multiple intermediate layers. In the case where there are multiple intermediate layers, the intermediate layers are sequentially referred to as "first intermediate layer, second intermediate layer, ..., last intermediate layer" from the side close to the input layer 41. As the number of intermediate layers increases, the accuracy of identification is improved overall.

[0080] The input layer 41 includes a plurality of input units A1 to A q The middle layer 42 includes a plurality of arithmetic units B1 to B j The output layer 43 includes a plurality of arithmetic units C1 to C n . Then, input cells A1 to A q is called the "input node A1 to A q ", the middle layer arithmetic units B1 to B j is called "intermediate node B1 to B j", and the arithmetic units C1 to C n is called the "output node C1 to C n ”.

[0081] Each intermediate node of the intermediate layer 42 is connected to all input nodes A1 to A1 of the input layer 41. q In a conventional neural network, a learned weighting factor is set for each link. Each output node of the output layer 43 is associated with all the intermediate nodes B1 to B2 of the intermediate layer 42. j Link. Sets the learned weighting factor for each link.

[0082] The arithmetic processing of a conventional neural network will be described. Input data is supplied to input nodes A1 to A q Here, it is assumed that the input data is each pixel data of the image data. For example, the number of input nodes is the same as the number of pixels of the input image data, and the pixel data of each pixel is provided to the input node. Each input node outputs the provided input data to the intermediate nodes B1 to B j .

[0083] Each input node enables the connection from input node A1 to A q The input data is multiplied by the input nodes A1 to A q The corresponding weighting factor is calculated and the sum of the multiplication results is determined. In other words, the product sum (weighted sum) of the input data and the weighting factor is determined. Then, the result of the product sum operation is used as the input variable of the activation function to calculate the activation function. Each intermediate node outputs the output value of the activation function to the output nodes C1 to C n .

[0084] For example, suppose that the intermediate node B1 is connected to the input nodes A1 to A q The weighting factors between w1 and w q , and from input node A1 to A q The data input to the intermediate node B1 is d1 to d q In this case, the product-sum operation value (X) at the intermediate node B1 is given by the following expression: B1 ) and the output value of the activation function (Y B1 ).

[0085] [Mathematical formula 2]

[0086]

[0087]

[0088] Function f indicates an activation function. Σ indicates weighting factors w1 to w q With input data d1 to dq Here, an example of arithmetic operation in the intermediate node B1 is indicated, but in the intermediate nodes B2 to B j The same arithmetic operation is also performed in the intermediate nodes B1 to B j The output value becomes the input to the output node C1 to C m data.

[0089] For example, as the activation function, an arbitrary nonlinear function such as a ReLU function and a step function is used. However, a linear function may be used.

[0090] The ReLU function refers to a function that outputs 0 if the input variable is less than 0, and directly outputs the value of the input variable if the input variable exceeds 0. When the ReLU function is ReLu(x) and the input variable is x, ReLu(x) is defined as follows.

[0091] [Mathematical formula 3]

[0092]

[0093] A step function is a function that outputs 0 if the input variable is less than 0 and outputs 1 if the input variable exceeds 0. When the step function is Step(x) and the input variable is x, Step(x) is defined as follows.

[0094] [Formula 4]

[0095]

[0096] Here, the activation function of each intermediate node of the intermediate layer is the same, however, the activation function may be different depending on the intermediate node.

[0097] The arithmetic operation of each output node is the same as that of the intermediate node. Specifically, each output node makes the number of nodes from the intermediate node B1 to B j The input data is multiplied by the intermediate nodes B1 to B j The corresponding weighting factors are added and the sum of the multiplication results is determined. In other words, the sum of the products of the input data and the weighting factors (weighted sum) is determined. Then, the activation function is calculated using the result of the product-sum operation as the input variable of the activation function. The output value of the activation function at each output node becomes the output of each output node.

[0098] Here, the activation function of each output node of the output layer is the same, however, the activation function may be different depending on the output node. It does not matter whether the activation function of the output layer is the same or different from the activation function of the intermediate layer.

[0099] The arithmetic processing unit 19 performs determination processing based on the output value of each output node, and outputs a result of the determination processing as a result of image recognition.

[0100] For example, a category is assigned to each output node and the category of the output node that outputs the maximum output value is regarded as the recognition result. For example, it is assumed that there are two output nodes, and the first category assigned to one output node (the first output node) is a category indicating that a person's face is captured in the image. It is also assumed that the second category assigned to the other output node (the second output node) is a category indicating that a person's face is not captured in the image. In this case, if the output value of the first output node is greater than the output value of the second output node, a signal indicating the first category is output. In other words, it is determined that a person's face is captured in the image data provided as input to the neural network. On the other hand, if the output value of the second output node is greater than the output value of the first output node, a signal indicating the second category is output. In other words, it is determined that a person's face is not captured in the image data provided as input to the neural network.

[0101] The above-mentioned determination process based on the output value of each output node is only an example, and a different determination process can be used. For example, the output value of each output node is compared with a threshold value, and when the number ratio of the output nodes (whose output value is above the threshold value) is above a predetermined value, it is determined to obtain a first recognition result (for example, it is determined that the face of a person is captured in the image). In the case where the number ratio is less than a predetermined value, it is determined to obtain a second recognition result (for example, it is determined that the face of a person is not captured in the image). Another example is to specify the maximum output value among the output values ​​of the output nodes, and when the specified output value is above a predetermined value, it is determined to obtain a first recognition result, and when the specified output value is less than a predetermined value, it is determined to obtain a second recognition result. The threshold is stored in advance in the internal buffer of the data storage unit 18 or the arithmetic processing unit 19. The determination method based on the output value of each output node can be arbitrarily defined according to the configuration of the neural network of the learning method, etc.

[0102] In the case where there are multiple intermediate layers, each intermediate node of the intermediate layer in the first level is connected to all input nodes A1 to A qLink, and set the weighting factor of each link. In addition, between adjacent intermediate layers, each intermediate node of the intermediate layer in the next level is linked with all the intermediate nodes of the intermediate layer in the previous level, and set the weighting factor of each link. Then, each output node of the output layer 43 is linked with all the intermediate nodes of the intermediate layer in the last level, and set the weighting factor of each link. In each intermediate node of the intermediate layer in the second or next stage, the output value of all the intermediate nodes of the intermediate layer in the previous level becomes the input data. The arithmetic operation of each intermediate node of the intermediate layer in the second or next level is the same as that of the intermediate layer 42 described above.

[0103] For this learning of the neural network, a plurality of image data whose recognition results (correct solutions) have been known are provided as learning data, and the weighting factors between the nodes are adjusted so that when the image data is input to the input layer, the recognition result (correct recognition result) is obtained. The recognition result known in advance is called "teacher signal". This adjustment of the weighting factors can be performed by methods such as the back propagation method. The back propagation method refers to a method of adjusting the weighting factors in sequence from the output layer side so that the displacement between the output of the neural network and the teacher signal is reduced.

[0104] The above is a description of a commonly used neural network, and reference will be made to Figures 4 to 6 A neural network according to Embodiment 1 is described. Figure 4 indicates the overall structure of the neural network and will also refer to Figure 4 A description is given of a neural network according to Embodiment 1. For example, the neural network according to Embodiment 1 corresponds to a first neural network, and for example, each node in the first neural network corresponds to a first node.

[0105] In embodiment 1, for the data provided to each input node of the neural network, the arithmetic processing unit 19 converts the pixel data and uses the converted data as input data (first input data). Specifically, in expression (1), the current value i indicated by the pixel data of each pixel 21 and related to the value (voltage value) is converted into a logarithm with a base of 2, and the data (log2i) is used. The converted data is called "logarithmic pixel data". Using the change of the basic formula, the above-mentioned expression (1) is converted into expression (5-1), and further converted into expression (5-2), thereby deriving the logarithmic pixel data (log2i).

[0106] [Mathematical formula 5]

[0107]

[0108]

[0109] Because A and log2e are fixed values, A' is also a fixed value. Therefore, log2i can be calculated by multiplying the value (v) of the pixel data by A'. Because A' is a fixed value, the arithmetic operation is simple and can be implemented with a small amount of arithmetic operations or by a small circuit. In other words, there is no need for a circuit (log2 conversion circuit) for converting the pixel data into a logarithm with a base of 2 (log2 conversion). log2e is an example of a logarithm whose antilogarithm is a predetermined value (in this case, the Napier constant).

[0110] In the arithmetic processing of the neural network according to Embodiment 1, a logarithmic weighting factor is used as a weighting factor between each node, and the logarithmic weighting factor is generated by converting the learned weighting factor into a logarithm with a base of 2. "Between each node" refers to "between an input node and an intermediate node" and "between an intermediate node and an output node". In the case where there are multiple intermediate layers, "between each node" also refers to "between intermediate nodes of different intermediate layers". As described above, the arithmetic processing of the neural network according to Embodiment 1 is performed based on the logarithmic pixel data provided to each input node and the logarithmic weighting factor set between each node.

[0111] Next, the content of the arithmetic processing at the intermediate node will be described. First, logarithmic pixel data is input from each input node connected to the intermediate node and the logarithmic weighting factors relative to each input node are added respectively. Because the target of addition is a logarithm, the result of addition is also a logarithm. Here, log2i+log2w=log2(i*w) holds true, so the value determined by adding the logarithmic pixel data (log2i) and the logarithmic weighting factor (log2w) is equal to the value generated by converting the product value (i*w) of the current data i and the weighting factor w into a logarithm with a base of 2. Log2(i*w) can be calculated only by adding the input logarithms without directly calculating i*w. Therefore, the multiplication operation of the arithmetic processing in conventional neural networks can be replaced by the addition operation.

[0112] Then, the log2 of the sum of the antilogarithms of the addition results (logarithmic values) of each input node is calculated. Calculating the log2 of the sum of the antilogarithms of multiple logarithmic values ​​is called "log2 addition". The value calculated by log2 addition is called "log2 added value".

[0113] For example, when the addition result of each input node is log2(i1*w i ),...,log2(i q-1 *w q-1 )、log2(i q *w q), then its log2 addition value is log2(i1*w1+,...,+i q-1 *w q-1 +i q *w q ). Log2 addition can be realized with a small amount of arithmetic operations or through a small circuit.

[0114] The output value of the activation function is calculated using the calculated log2 addition value as the input variable of the activation function. Figure 4 A specific example is described by taking the intermediate node B1 in FIG. 1 as an example.

[0115] Figure 5 is a diagram for describing an example of arithmetic processing in the intermediate node B1. Assume that the intermediate node B1 is connected to the input nodes A1 to A q The logarithmic weighting factors are set between log2w1 and log2w q Assume that from input node A1 to A q The data input to the intermediate node B1 is log2i1 to log2i q .

[0116] In this case, the calculated value of the activation function at the intermediate node B1 (X') is given by the following expression: B1 ) and output value (Y' B1 ).exist Figure 5 In the example, the ReLU function is used as the activation function.

[0117] [Mathematical formula 6]

[0118]

[0119]

[0120] At the intermediate node B2 to B j The same operation as that of the intermediate node B1 is performed at the node.

[0121] In order to understand expression (6-1) by comparing it with expression (2-1), the calculated value (log2 addition value) at the node in the neural network according to implementation mode 1 is made to correspond to the value generated by converting the calculated value (product sum value) at the node in the conventional neural network into log2.

[0122] Output nodes C1 to C n The arithmetic processing at the output node is the same as that at the intermediate node, but the data input from the layer of the previous level is the output value from each intermediate layer. As in the case of the intermediate node, because the output value of the intermediate node is the logarithm with a base of 2, log2 addition can be performed at the output node.

[0123] For example, when the activation function at the intermediate node is ReLU and the calculated value is greater than 0, the calculated value is directly output, so the output value of ReLU is a logarithm with a base of 2. If the calculated value is below 0, 0 is output, but 0=log21, so in this case, the output value is also a logarithm with a base of 2. Therefore, the same arithmetic operation as in the case of the intermediate node can also be performed at the output node.

[0124] The arithmetic processing unit 19 outputs the nodes C1 to C n The output value (logarithm with base 2) of the image recognition unit 10 is used to perform a determination process, and the result of the determination process is output as the result of image recognition. The details of the determination process are the same as the determination process of the conventional neural network described above. For example, a category is assigned to each output node, and the category of the output node that outputs the maximum output value is regarded as the result of image recognition. Another possible method is the above-mentioned method of comparing the output value of each output node with a threshold value. In the case of comparison with a threshold value, a value generated by converting the threshold value into a logarithm with base 2 (logarithmic threshold value) can be used. Therefore, comparison between logarithms becomes feasible. The logarithmic threshold value is stored in advance in the data storage unit 18 or in the internal buffer of the arithmetic processing unit 19.

[0125] Therefore, in the neural network according to Embodiment 1, multiplication operations are not required at intermediate nodes and output nodes, so a multiplication circuit is not required and the circuit scale is reduced. In addition, the amount of arithmetic operations is reduced, that is, power consumption is reduced.

[0126] A configuration example of a circuit (log2 addition circuit) that efficiently performs log2 addition will be described. Here, log2 addition of log2(i1*w1) and log2(i2*w2) can be approximated by the following expression (see Patent Document 1).

[0127] [Mathematical formula 7]

[0128]

[0129] The first term in expression (7) indicates the larger value of log2(i1*w1) and log2(i2*w2). The second term is 2 indexed by the negative absolute value of the difference between log2(i1*w1) and log2(i2*w2). The second term is equivalent to shifting 1 to the right (in the direction of smaller numbers) by the absolute value of the difference. In other words, the bit shift of the bit string representing decimal 1 in bits is |log2(i1*w2)-log2(i1*w1)|. Because the power exponent of 2 is a negative value, a bit shift is performed on the right. For example, in the case of representing 1 as 00001000, if the absolute value of the difference mentioned above is 3, 3 bit shifts are performed to become 00000001. The role of the bit symbol can be determined arbitrarily, and the surface is only an example.

[0130] As described above, log2 addition can be performed by selecting the maximum value, calculating the absolute value of the difference, and shifting the bits, so the arithmetic operation is simple. In addition, if the value of |log2(i1*w2)-log2(i1*w1)| is not an integer, the value can be approximated to an integer using rounding processing (round-up processing or round-down processing), a ceiling function, a floor function, etc. Each item in log2(i1*w2) and log2(i1*w1) can be approximated to an integer, respectively, and then, |log2(i1*w2)-log2(i1*w1)| can be calculated.

[0131] Figure 6 1 is a block diagram depicting an example of a log2 addition circuit. The arithmetic processing unit 19 includes a log2 addition circuit. In this example of the circuit configuration, the accumulation is performed by making a plurality of input nodes A1 to A q The logarithmic pixel data of each input node in is added to the logarithmic weight to calculate log2(i n *w n )(n is an integer in the range of 1 to q). Figure 6 The circuit in includes a selector 51, an adder 52, a subtractor 53, an absolute value arithmetic unit 54, an exponential arithmetic unit 55, and a trigger circuit 56.

[0132] The selector 51 includes a first input terminal and a second input terminal. n *w n ) is input to the first input terminal and is regarded as data P n The value log2(i1*w1+...i n-1 *w n-1 ) is input to the second input terminal and regarded as data S n-1Initially, log2(i2*w2)(=P2) is input to the first input terminal, and then, log2(i1*w1)(=S1) is input to the second input terminal.

[0133] Also the data P n and data S n-1 Input to subtractor 53. Subtractor 53 obtains P n Subtract S from n-1 The output of the subtractor 53 is connected to the control terminal of the selector 51. The subtractor 53 outputs a signal indicating the sign of the subtraction result to the control terminal of the selector 51 and outputs a signal indicating the subtraction result to the absolute value arithmetic unit 54. If the subtraction result indicates positive, the selector 51 outputs the data P n , and if the subtraction result indicates negative, the output data S n-1 In other words, the output data P n and data S n-1 The subtractor 53 can be configured to obtain the larger value from S n-1 Subtract P n , and in this case, if the subtraction result indicates positive, the selector 51 outputs data S n-1 , and if the subtraction result indicates a negative, the output data P n .

[0134] The absolute value arithmetic unit 54 is connected to the output of the subtractor 53 and calculates the absolute value of the subtraction result generated by the subtractor 53. n -S n-1 |.

[0135] The exponential arithmetic unit 55 is connected to the output of the absolute value arithmetic unit 54 and calculates the exponential value of 2. n -S n-1 As mentioned above, by shifting 1 to the right by |P n -S n-1 |This calculation can be achieved.

[0136] The adder 52 is connected to the output of the selector 51 and the output of the exponential arithmetic unit 55 , and adds the value selected by the selector 51 and the value after bit shifting is performed in the exponential arithmetic unit 55 .

[0137] The trigger circuit 56 is connected to the output of the adder 52 and stores the addition result of the adder 52 (the approximate value of log2(i1*w1+i2*w2)) internally. The trigger circuit 56 is connected to the second input terminal of the selector 51 and feeds the above value back to the second input terminal and the subtractor 53 as data S2. The log2(i3*w3) to be the next calculation target is input to the first input terminal of the selector 51 as data P3. After that, by repeating the process in the same manner, the log2(i n *w n ) cumulatively performs log2 addition, and finally, log2(i1*w1+...i q *w q The output value of trigger circuit 56 becomes the input variable of the activation function.

[0138] Figure 7 1 is a flowchart of an example of the operation of the solid-state imaging device according to Embodiment 1. When a predetermined event is established in a device equipped with the present device, a trigger for executing the process is generated. The predetermined event may be any event generated or detected by the device equipped with the present device. For example, the event may be: a user operates a device equipped with the present device to perform a predetermined operation; a predetermined time has elapsed; a vibration of a predetermined value or higher is detected, etc., but the event is not limited to these examples. By the trigger, an image of one frame is captured by the pixel array unit 11 under the control of the control unit 15. In other words, in each pixel 21, a pixel signal (voltage signal) that changes logarithmically with respect to the amount of light of the incident light is generated by photoelectric conversion, and the generated pixel signal is read and sent to the signal processing unit 16 (S101). A / D conversion is performed on these pixel signals by the column processing unit 13 (S102), and a pixel data set of each pixel acquired by the A / D conversion is stored as image data in the data storage unit 18 (S103).

[0139] The arithmetic processing unit 19 reads the image data from the data storage unit 18 and performs data conversion on each pixel data included in the image data (S104). For example, the data conversion operation multiplies the data by log2e / A (see Expression (5-1) and Expression (5-2)). In other words, the value of the pixel data is multiplied by the logarithm of the Napier constant (predetermined value) whose antilogarithm is. Thus, the value of the pixel data is converted into a logarithm with a base of 2, and the converted data (log2i) becomes logarithmic pixel data (S104).

[0140] The arithmetic processing unit 19 reads a logarithmic weighting factor (log2 weighting factor) from the data storage unit 18 (S105), which is the logarithm of the weighting factor set between each node in the neural network. The short sentence "between each node" means "between the input node and the intermediate node" and "between the intermediate node and the output node". If there are multiple intermediate layers, [the meaning of "between each node"] includes "between intermediate nodes of different intermediate layers". The logarithmic weighting factor may be stored in advance in the internal buffer of the arithmetic processing unit 19 instead of the data storage unit 18. In the following description, it is assumed that the neural network has one intermediate layer (see Figure 4 ), however, a case of a neural network having a plurality of intermediate layers is also possible. In this case, in the intermediate layer of the second or later stage, data is input from the intermediate nodes in the intermediate layer of the previous stage, not from the input nodes, and the processing content is the same as in the case of the intermediate layer of the first stage.

[0141] The arithmetic processing unit 19 assigns the logarithmic pixel data of each pixel 21 as input data to each input node of the neural network (S106). In the learning process, the correspondence between the specified pixel 21 and the specified input node is determined in advance. The one-to-one correspondence between the pixel 21 and the input node is only an example, and the input data assigned to each input node can be generated by another method.

[0142] For example, a pixel group is formed by a predetermined number of pixels 21, and each pixel group corresponds to each input node. Determine the representative value of the logarithmic pixel data of the pixel group, and assign the representative value as input data to the input node. The representative value can be any statistical value such as mean, maximum, and minimum. The pixel data of the difference between the logarithmic pixel data read this time and the logarithmic pixel data read last time can be assigned to the input node as input data.

[0143] The arithmetic processing unit 19 performs arithmetic operations on each intermediate node based on the input data assigned to each input node and the logarithmic weighting factor set between each intermediate node and each input node (S107). In other words, for each input node, the logarithmic weighting factor is added to the input data, and log2 addition is performed on all the addition results. The arithmetic processing unit 19 uses the log2 addition value as an input variable to calculate the activation function (S107). The output value of the activation function is regarded as the output value of each intermediate node. The output value of each intermediate node becomes the data input to the output layer of the next stage.

[0144] Then, [arithmetic processing unit 19] performs arithmetic operation on each output node based on the output value of each intermediate node and the logarithmic weighting factor set between each intermediate node and each output node (S108). In other words, for each intermediate node, the logarithmic weighting factor is added to the data to be input, and log2 addition is performed on all the addition results. Arithmetic processing unit 19 calculates an activation function using the log2 addition value as an input variable (S108). The output value of the activation function becomes the output value of the output node.

[0145] The arithmetic processing unit 19 performs determination processing based on the output value of each output node (S109). For example, the output node whose maximum output value is calculated is specified, and the category assigned to the specified output node is obtained as the result of image recognition. As mentioned above, various other algorithms can be used for determination recognition, such as processing using a threshold value. Any algorithm can be used. The arithmetic processing unit 19 stores the data of the result of the image recognition in the data storage unit 18.

[0146] The CPU 17 reads data indicating the result of image recognition from the data storage unit 18 and outputs the data (S110). For example, the data is output to an application executed by a device equipped with the present apparatus. In this case, the application performs processing according to the result indicated by the data. The data indicating the result of image recognition can be displayed on a screen of a display included in the device equipped with the present apparatus.

[0147] The above-mentioned order of steps is an example, and the order of some steps may be reversed. For example, the order of step S104 and step S105 may be reversed.

[0148] As described above, according to Embodiment 1, the data of the pixel signal (voltage signal) read from the logarithmic pixel is converted, thereby generating input data of a logarithm with a base of 2. Since a circuit (log2 conversion circuit) that performs conversion on a logarithm with a base of 2 is not required, the circuit scale can be reduced and power consumption can also be reduced.

[0149] In Embodiment 1, the data of the pixel signal read from the logarithmic pixel is used, so the high dynamic range characteristics of the logarithmic pixel can be used in image recognition. In the case of the prior art, the data of the pixel signal read from the linear pixel is converted into a logarithm with a base of 2 by a log2 conversion circuit, but in this case, the dynamic range of the converted data is limited by the dynamic range of the linear pixel. Therefore, the effect of the high dynamic range cannot be obtained. On the other hand, in Embodiment 1, data is read from the logarithmic pixel, so the effect of the high dynamic range can be obtained.

[0150] In addition, according to Embodiment 1, logarithmic pixel data (logarithm with base 2) and logarithmic weighting factor (logarithm with base 2) converted from pixel data are used, and therefore, addition can be used for arithmetic operation of each node. This means that multiplication (multiplication of weighting factor and input data) does not need to be performed at each node. Therefore, the amount of arithmetic operation is reduced and power consumption can be reduced. In addition, since a multiplication circuit is not required, the circuit scale can be reduced.

[0151] (Transformation 1)

[0152] In Embodiment 1, a logarithm with a base of 2 is used, but the base of the logarithm does not need to be 2. Even when the base is not 2, addition operations can be performed in the logarithmic region of each node, thereby reducing the amount of arithmetic operations.

[0153] (Transformation 2)

[0154] The arrangement of logarithmic pixels indicated in Embodiment 1 ( Figure 3 ) are examples, and various modifications are possible. Figure 8 Describes the deformation of logarithmic pixels.

[0155] Figure 8 3 is a circuit diagram depicting a variation of a logarithmic pixel. Here, a reset transistor 37, a buffer transistor 38, and a bias transistor 39 are added as pixel transistors to the Figure 3 The buffer transistor 38 and the bias transistor 39 constitute a buffer circuit 40. For example, the reset transistor 37 is an N-type transistor. For example, the buffer transistor 38 and the bias transistor 39 are P-type transistors.

[0156] The connection node N2 is connected to the gate terminal of the output transistor 32 and the gate terminal of the buffer transistor 38. The voltage signal generated in the connection node N2 is supplied to the amplification transistor 35 via the buffer circuit 40. The voltage signal is amplified by the amplification transistor 35.

[0157] The bias transistor 39 and the buffer transistor 38 are connected in series between the power supply voltage and the reference voltage. A bias voltage Vbias2 is applied to the gate terminal of the bias transistor 39. The connection node of the bias transistor 39 and the buffer transistor 38 (i.e., the output terminal of the buffer circuit 40) is connected to the source terminal of the reset transistor 37 and the gate terminal of the amplifying transistor 35. The drain terminal of the reset transistor 37 is connected to the power supply voltage. The gate terminal of the reset transistor 37 is connected to the pixel drive line 22.

[0158] The reset transistor 37 is controlled by the reset signal RST supplied by the vertical drive unit 12. In the state where the reset transistor 37 is turned on, the signal processing unit 16 or the column processing unit 13 performs a correlated double sampling (CDS) process to detect a signal that does not include a pixel signal, and removes (subtracts) the detected signal from the pixel signal. Thereby, fixed pattern muting caused by irregularities of the pixel transistor, etc. is removed. The pixel signal can be read before or after reading the signal that does not include a pixel signal. The signal obtained by the CDS process can be used as the pixel signal read from the logarithmic pixel. The buffer circuit 40 has a suppressive effect so that the potential of the logarithmic pixel (the potential of the node N2) is less affected during the reset process. Thereby, the potential of the logarithmic pixel can be output constantly according to the amount of light. Even if used Figure 8 The circuit configuration in , CDS processing cannot be performed either.

[0159] According to Modification 2, fixed pattern noise included in a pixel signal can be removed by performing CDS processing.

[0160] (Transformation 3)

[0161] Fig. 9 is a circuit diagram depicting another variation of a logarithmic pixel. Here, an event detection unit 80 is added to Figure 8 Circuit configuration in. In some cases, a pixel circuit to which an event detection unit 80 is added is referred to as a "dynamic vision sensor (DVS) circuit". Here, only the circuit symbol is used to indicate the configuration of the buffer circuit 40. The event detection unit 80 detects whether an event is generated based on whether the change in photocurrent from the photodiode 31 exceeds a predetermined threshold. For example, the event is a turn-on event detected when the change exceeds an upper threshold, a turn-off event detected when the change is lower than a lower threshold, or a turn-on event and a turn-off event. For example, detecting a turn-on event or a turn-off event corresponds to detecting a change in contrast (e.g., detecting an object). The event detection unit 80 outputs a detection signal or a non-detection signal to the control unit 15 based on whether an event is detected.

[0162] The event detection unit 80 includes a subtractor 81 and a quantizer 82. The subtractor 81 is connected to the output terminal of the buffer circuit 40. The quantizer 82 is connected to the output terminal of the subtractor 81. The event detection unit 80 may be added to Figure 3 The circuit configuration in .

[0163] The subtractor 81 drops the level of the voltage signal from the buffer circuit 40 according to the drive signal from the vertical drive unit 12. The subtractor 81 supplies the dropped voltage signal to the quantizer 82.

[0164] The quantizer 82 quantizes the voltage signal from the subtractor 81 into a digital signal. The quantizer 82 compares the quantized signal with the threshold voltage Vth and generates a signal (a detection signal or a non-detection signal) indicating the comparison result. For example, in the case where the threshold voltage Vth is an upper threshold, if the quantized signal is greater than the upper threshold, a detection signal is generated, and if the quantized signal is below the upper threshold, a non-detection signal is generated. The quantizer 82 outputs the generated detection signal or non-detection signal to the control unit 15. In the case where the threshold voltage Vth is also a lower threshold, a detection signal or a non-detection signal can be generated in the same manner.

[0165] In the case where a detection signal is input, the control unit 15 determines to read a pixel signal from the logarithmic pixel 21, and controls the vertical drive unit 12 to read the pixel signal from the logarithmic pixel 21. On the other hand, in the case where a non-detection signal is input, the control unit 15 determines not to read a pixel signal from the logarithmic pixel 21. In the case where no event is detected, the quantizer 82 may omit outputting the non-detection signal to the control unit 15.

[0166] Fig.10 8 is a circuit diagram depicting a configuration example of a subtractor 81 and a quantizer 82. The subtractor 81 includes a capacitor 83, a capacitor 84, an inverter 85, and a switch 86. The quantizer 82 includes a comparator 87.

[0167] One end of the capacitor 83 is connected to the output terminal of the buffer circuit 40, and the other end is connected to the input terminal of the inverter 85. The capacitor 84 is connected in parallel with the inverter 85. The switch 86 connects both ends of the capacitor 84. The switch 86 performs switching according to the drive signal from the vertical drive unit 12.

[0168] The inverter 85 inverts the voltage signal input via the capacitor 83. The inverter 85 outputs the inverted signal to the non-inverting input terminal (+) of the comparator 87.

[0169] When the switch 86 is turned on, the voltage signal Vinit is input to the buffer circuit 40 side of the capacitor 83, and the opposite side of the capacitor 83 becomes a virtual ground terminal. For convenience, it is assumed that the potential of the virtual ground terminal is zero. If the capacitance of the capacitor 83 at this time is C1, the change Qinit stored in the capacitor 83 is given by the following expression. On the other hand, because both ends of the capacitor 84 are short-circuited, the charge stored in the capacitor 84 is zero.

[0170] Qinit=C1×Vinit···(8-1)

[0171] Then, the switch 86 is turned off, and the voltage on the buffer circuit 40 side of the capacitor 83 becomes Vafter. At this time, the charge Qafter stored in the capacitor 83 is given by the following expression.

[0172] Qafter=C1×Vafter···(8-2)

[0173] On the other hand, the charge Q2 stored in the capacitor 84 is given by the following expression in which the output voltage is Vout.

[0174] Q2=-C2×Vout···(8-3)

[0175] The total amount of charge of capacitor 83 and capacitor 84 does not change, so the following expression holds.

[0176] Qinit=Qafter+Q2···(8-4)

[0177] If Expression (8-1) to Expression (8-3) are substituted into Expression (8-4) and transformed, the following expression is obtained.

[0178] Vout=-(C1 / C2)×(Vafter-Vinit)···(8-5)

[0179] Expression (8-5) indicates the subtraction operation of the voltage signal, and the gain in the subtraction result is C1 / C2. Generally, the gain needs to be maximized, so it is preferred to design C1 to be larger and C2 to be smaller. However, if C2 is too small, the kTC noise increases and the noise characteristics may deteriorate, so the reduction of the C2 capacitance is limited by the allowable range of noise.

[0180] The comparator 87 compares the voltage signal from the subtractor 81 with the threshold voltage Vth applied to the inverter input terminal (-). The comparator 87 outputs a detection signal or a non-detection signal according to the comparison result.

[0181] According to Modification 3, when the event detection unit 80 detects an event, a pixel signal is read, and thus, pixel data can be generated at a high speed.

[0182] (Implementation Method 2)

[0183] In Embodiment 2, two types of pixels (logarithmic pixels and linear pixels) are used as pixels set in the pixel array unit 11. Then, according to the indication data provided externally, the pixels from which the pixel signals are read are selected between the logarithmic pixels and the linear pixels. In the case of reading the pixel signals from the logarithmic pixels, the subsequent processing is the same as in Embodiment 1. In the case of reading the pixel signals from the linear pixels, the subsequent processing is the same as the arithmetic processing of the conventional neural network. In other words, at each node (intermediate node and output node), a product-sum operation is performed based on the pixel data and the weighting factor of each pixel, and the activation function is calculated using the result of the product-sum operation as an input variable. Embodiment 2 will be described in detail.

[0184] Fig.11 2 is a block diagram illustrating a configuration example of a solid-state imaging device according to Embodiment 2 of the present disclosure. Here, an input / output terminal 24 and a mode instruction unit 25 are added to Figure 1 The same reference numerals are used to denote the solid-state imaging device having the same name as that of the Figure 1 The same elements in the present invention are described herein and their detailed description is omitted except for the expansion or change processing.

[0185] The pixel array unit 11 includes two types of pixels (logarithmic pixels 21A and linear pixels 21B). The linear pixels 21B correspond to an example of a second pixel, which is a different type from the logarithmic pixels 21A. The pixel signal read from the linear pixels 21B corresponds to the second pixel signal read from the second pixel.

[0186] The logarithmic pixels 21A and the linear pixels 21B are alternately arranged in the horizontal direction and the vertical direction (arranged in a checkerboard pattern). However, the arrangement is not limited to this and can be arbitrarily arranged. For example, in each square pixel block consisting of two vertical pixels and two horizontal pixels, three pixels in the pixel block can be logarithmic pixels, and another pixel can be a linear pixel. In addition, the same type of pixels can be arranged in each column, and different types of pixels can be alternately arranged in the row direction (horizontal direction).

[0187] As described in Embodiment 1, a linear pixel generates an electrical signal as a pixel signal that changes linearly with respect to incident light. In other words, a linear pixel generates an output signal that increases linearly with respect to the amount of light. Compared with a logarithmic pixel, a linear pixel has a narrower dynamic range but has lower noise characteristics.

[0188] Fig.12 is a graph schematically showing the relationship between the amount of incident light entering a linear pixel and the output voltage. As the amount of incident light increases, the output voltage increases linearly (eg, proportionally).

[0189] Fig.13 Indicates an example of a circuit configuration of a linear pixel 21B. Here, the configuration of one linear pixel 21B is indicated as an example, but each of the other linear pixels 21B has the same configuration.

[0190] The linear pixel 21B includes a photodiode 61 corresponding to a photoelectric conversion unit and a plurality of pixel transistors. The plurality of pixel transistors include a transfer transistor 62, an amplifying transistor 63, a selection transistor 64, and a reset transistor 65. For example, these transistors are metal oxide semiconductor (MOS) transistors and are N-type transistors. However, these transistors may be P-type transistors. In this case, it is necessary to change the logic of the control signal and the voltage of the power supply line of these pixel transistors.

[0191] The photodiode 61 is an element that performs photoelectric conversion to convert incident light into an electrical signal (signal change) proportional to the light amount of the incident light and stores the electrical signal. The anode terminal of the photodiode 61 is connected to a reference voltage, and the cathode terminal thereof is connected to the source terminal of the transfer transistor 62.

[0192] The transfer transistor 62 is connected between the cathode terminal of the photodiode 61 and the FD node 66. The gate terminal of the transfer transistor 62 is connected to the vertical drive unit 12. The transfer transistor 62 is controlled by a transfer signal TRG from the vertical drive unit 12. When the transfer transistor 62 is turned on, the signal charge stored in the photodiode 61 is transferred to the FD node 66.

[0193] The FD node 66 is a floating diffusion region connected between the drain terminal of the transfer transistor 62 and the gate terminal of the amplification transistor 63. Thus, the FD node 66 can store signal charges transferred from the photodiode 61. The FD node 66 is also connected to the source terminal of the reset transistor 65.

[0194] The amplifier transistor 63 is connected between a power supply voltage and the selection transistor 64. A gate terminal of the amplifier transistor 63 is connected to the FD node 66. The amplifier transistor 63 enters a conductive state according to the signal charge stored in the FD node 66.

[0195] The selection transistor 64 is connected between the amplifying transistor 63 and the vertical signal line 23. The gate terminal of the selection transistor 64 is connected to the vertical driving unit 12. The selection transistor 64 is controlled by the selection signal SEL from the vertical driving unit 12. The selection transistor 64 connects the linear pixel 21B to the vertical signal line 23 or disconnects the linear pixel 21B from the vertical signal line 23. When the selection transistor 64 is turned on, current can be supplied to the amplifying transistor 63, and current is supplied from the power supply voltage according to the conductive state of the amplifying transistor 63. The vertical signal line 23 is set to the voltage of the amplifying transistor 63.

[0196] The reset transistor 65 is connected between the power supply voltage and the FD node 66. The gate terminal of the reset transistor 65 is connected to the vertical drive unit 12. The reset transistor 65 is controlled by a reset signal RST from the vertical drive unit 12. When the reset transistor 65 is turned on, the signal charge stored in the FD node 66 is released to the power supply voltage. Thus, the FD node 66 of the linear pixel 21B is reset.

[0197] In this way, a voltage signal proportional to the amount of incident light is output as a pixel signal. Fig.13 The configuration of the linear pixels in is an example, and the configuration of the linear pixels is not limited to Fig.13 Any configuration can be used for linear signals as long as a voltage signal proportional to the amount of light (charge) can be detected.

[0198] The input / output terminal 24 is a terminal for sending or receiving a signal to or from an external device. The input / output terminal 24 sends or receives a signal to or from an external device using any communication system such as serial communication. The external device may be a device for performing internal settings of a device equipped with the solid-state imaging device from the outside of the device, or may be an operation unit (e.g., a button) with which a user performs various settings.

[0199] The mode indication unit 25 receives mode data indicating the mode of the pixel reading process via the input / output terminal 24. The pixel reading mode includes a logarithmic reading mode and a linear reading mode. In the logarithmic reading mode, a first reading process of reading from a logarithmic pixel is performed. In the linear reading mode, a second reading process of reading from a linear pixel is performed. The mode data indicates one of these modes. For example, the mode indication unit 25 is composed of a storage element such as a register. The CPU 17 can provide the mode data to the mode indication unit 25.

[0200] The control unit 15 reads the mode data from the mode indication unit 25, and in the case where the mode data indicates the logarithmic reading mode, the control unit 15 controls each unit so that a pixel signal is read from each logarithmic pixel 21A in the pixel array unit 11. In the case where the mode data indicates the linear reading mode, the control unit 15 controls each unit so that a pixel signal is read from each linear pixel 21B in the pixel array unit 11. In other words, the control unit 15 selectively performs a process of reading a pixel signal from each logarithmic pixel 21A or a process of reading a pixel signal from each linear pixel 21B.

[0201] As in Embodiment 1, the column processing unit 13 processes signals output from a row of pixels 21 for each pixel column. For example, the signal processing may include A / D conversion, and may also include at least one of noise removal and amplification of the signal before A / D conversion.

[0202] The data storage unit 18 stores the parameters of the neural network, and the arithmetic processing unit 19 performs arithmetic processing of the neural network for the logarithmic reading mode and the linear reading mode respectively using the parameters of the neural network. The neural network of the logarithmic reading mode and the neural network of the linear reading mode are different from each other and also perform learning independently of each other.

[0203] As in Embodiment 1, as a parameter of the neural network in the logarithmic reading mode, the data storage unit 18 stores a value generated by converting a weighting factor indicating the strength of the connection between each node into a logarithm (logarithmic weighting factor) with a base of 2. Instead of the data storage unit 18, the logarithmic weighting factor may be stored in an internal buffer of the logarithmic processing unit 19. For example, the neural network in the logarithmic reading mode corresponds to the first neural network, and for example, each node in the first neural network corresponds to the first node.

[0204] As a parameter of the neural network in the linear reading mode, the data storage unit 18 stores a weighting factor indicating the strength of the connection between each node. Instead of the data storage unit 18, the weighting factor may be stored in an internal buffer of the arithmetic processing unit 19. For example, the neural network in the linear reading mode corresponds to the second neural network, and for example, each node in the second neural network corresponds to the second node.

[0205] The CPU 17 reads the mode data from the mode indicating unit 25, and in the case where the mode data indicates the logarithmic reading mode, the CPU 17 controls the arithmetic processing unit 19 in the same manner as in Embodiment 1. In other words, the arithmetic processing unit 19 converts the pixel data of each pixel into logarithmic pixel data (see expression (5-2)) and performs arithmetic processing of the neural network using the logarithmic pixel data and the logarithmic weighting factor between each node. As described in Embodiment 1, in each intermediate node and each output node, the data input from the layer of the previous level is added to the logarithmic weighting factor, and log2 addition is performed on these addition results (see expression (6-1)). Using the result of the log2 addition as the input variable of the activation function, the output value of the activation function is calculated (see expression (6-2)).

[0206] In the case where the mode data indicates a linear reading mode, the CPU 17 controls the arithmetic processing unit 19 so that an arithmetic operation similar to that of the conventional neural network described in Implementation 1 is performed. In other words, the arithmetic processing unit 19 regards the pixel data of each pixel as the input data (second input data) of the neural network, and performs arithmetic processing of the neural network using the input data and the weighting factor between each node. In the intermediate nodes and the output nodes, the data input from the layer of the previous level is multiplied by the weighting factor (see expression (2-1)), and the multiplication results are added. In other words, a product-sum operation is performed on the input data and the weighting factor. Then, the result of the product-sum operation is used as the input variable of the activation function, and the output value of the activation function is calculated (see expression (2-2)).

[0207] As described above, the arithmetic processing unit 19 includes a circuit that performs arithmetic processing of a neural network for the logarithmic reading mode and a circuit that performs arithmetic processing of a neural network for the linear reading mode.

[0208] Here, the neural networks of these modes are neural networks that perform learning independently of each other, but the present invention is not limited to this. For example, in the case where the contents of image recognition of these modes are the same, a neural network that performs learning together can be used. In this case as well, logarithmic pixel data and logarithmic weighting factors are used in the logarithmic reading mode, and pixel data and weighting factors are used in the linear reading mode. As described above, the arithmetic operations in each intermediate node and each output node are also different according to the mode.

[0209] Fig.14 1 is a flowchart of an example of the operation of the solid-state imaging device according to Embodiment 2. The control unit 15 reads the mode data from the mode indicating unit 25 (S201). Then, the [control unit 15] checks which mode of the logarithmic reading mode and the linear reading mode the mode data indicates (S202). In the case where the logarithmic reading mode is indicated, the control unit 15 controls each unit so that a pixel signal is read from each logarithmic pixel 21A of the pixel array unit 11 (S203). Subsequent processing is performed in accordance with Figure 7 The steps S102 to S110 in are the same as (S204).

[0210] In the case where the linear reading mode is instructed, the control unit 15 controls each unit so that a pixel signal is read from each linear pixel 21B in the pixel array unit 11 (S205). Fig.15 Subsequent processing is indicated in .

[0211] Fig.15 It is the successor Fig.14 Flow chart of an example of the operation of step S205 in FIG. Figure 7The same steps as in the flowchart are described in the processing unless otherwise required.

[0212] A / D conversion is performed on the pixel signal read from each linear pixel 21B by the column processing unit 13 ( S206 ), and a pixel data set of each pixel acquired by the A / D conversion is stored in the data storage unit 18 as image data ( S207 ).

[0213] The arithmetic processing unit 19 reads the image data from the data storage unit 18 ( S208 ), and reads the weighting factor between each node in the neural network in the linear reading mode from the data storage unit 18 ( S209 ).

[0214] The arithmetic processing unit 19 assigns the pixel data of each pixel 21 as input data to each input node of the neural network (S210). In the learning process, the correspondence between the specified pixel 21 and the specified input node is determined in advance. The one-to-one correspondence between the pixel 21 and the input node is only an example, and the input data assigned to each node can be generated by another method.

[0215] The arithmetic processing unit 19 performs an arithmetic operation on each intermediate node based on the input data assigned to each input node and the weighting factor between each intermediate node and each input node (S211). In other words, for each input node, the weighting factor is multiplied by the input data, and the product of the multiplication result is calculated. This means performing a product-sum operation of the weighting factor and the input data. The arithmetic processing unit 19 uses the result of the product-sum operation as an input variable to calculate the activation function (S211). The output value of the activation function is regarded as the output value of each intermediate node. The output value of each intermediate node becomes the data input to the output layer of the next level.

[0216] Then, [arithmetic processing unit 19] performs an arithmetic operation on each output node based on the output value of each intermediate node and the weighting factor between each intermediate node and each output node (S212). In other words, the weighting factor of each intermediate node is multiplied by the input data, and the product of the multiplication result is calculated. This means performing a product-sum operation of the weighting factor and the input data. The arithmetic processing unit 19 uses the result of the product-sum operation as an input variable to calculate the activation function (S212). The output value of the activation function is regarded as the output value of each output node.

[0217] The arithmetic processing unit 19 performs determination processing based on the output value of each output node (S213). For example, the output node of the maximum output value of the calculator is specified, and the category assigned to the specified output node is obtained as the result of image recognition. The arithmetic processing unit 19 stores the data indicating the result of image recognition in the data storage unit 18.

[0218] The CPU 17 reads data indicating the result of image recognition from the data storage unit 18 and outputs the data to an external device (S214). An example of an external device is a processor that executes an application program, and the application program is executed in a device equipped with the solid-state imaging device. The external device can determine whether to switch the reading mode according to the result of image recognition. For example, the external device determines to switch the logarithmic reading mode to the linear reading mode. The external device writes the mode data in which the mode to be switched is specified to the mode indication unit 25 via the input / output terminal 24 or the CPU 17. The mode switching will be described in detail below.

[0219] As mentioned above, compared with linear pixels, logarithmic pixels have higher noise characteristics but a wider dynamic range, and compared with logarithmic pixels, linear pixels have lower noise characteristics but a narrower dynamic range. By switching between the logarithmic reading mode and the linear reading mode using this characteristic difference, image recognition can be performed according to the purpose. For example, in order to perform sensing with lower power consumption, the logarithmic reading mode is used, and in order to accurately identify the object detected during the sensing process, the linear reading mode is used.

[0220] For example, the logarithmic reading mode is used to detect whether a person's face is captured, and if the face of the person is detected and captured, the mode is switched to the linear reading mode to identify the person (for example, to identify one of multiple people registered in advance). For another example, the logarithmic reading mode is used to detect whether a license plate of a vehicle is captured, and if the license plate is detected and captured, the mode is switched to the linear reading mode to identify the number on the license plate.

[0221] (Transformation 1)

[0222] The resolution setting (number of bits) of the A / D conversion can be changed depending on whether the mode is a logarithmic reading mode or a linear reading mode. For example, the number of bits in the linear reading mode can be set higher than the number of bits in the logarithmic reading mode. Specifically, for example, the number of bits of the linear pixel is 10 to 12 bits, and the number of bits of the logarithmic reading mode is 8 bits. By reducing the number of bits, the amount of arithmetic operations can be reduced, and therefore, the power consumption in the logarithmic reading mode can be further reduced. These values ​​are examples only, and the number of bits in each mode is arbitrary. Between these modes, the number of bits of the A / D conversion can be the same.

[0223] (Transformation 2)

[0224] In Embodiment 2, an example in which logarithmic pixels and linear pixels are used as two types of pixels is described, but another type of pixel may be used instead of linear pixels.

[0225] For example, the number of incoming photons can be counted and output as a pixel signal, that is, a photon counting pixel (digital pixel) can be used. For a photon counting pixel, a single photon avalanche diode (SPAD) capable of detecting incoming single photons is used. The pixel signal is generated by counting the pulses output by the SPAD from the photons entering.

[0226] For another pixel example, a distance measuring pixel can be used to measure the distance to an object. An example of a distance measuring pixel is a pixel using indirect time of flight (ITOF) technology. In this pixel, when pulsed light modulated using a photodiode is reflected, a phase difference as an amount of charge is detected and regarded as a pixel signal.

[0227] The pixel illustrated here corresponds to the example of the second pixel, that is, a pixel of a different type from the logarithmic pixel (first pixel). In the case of using this type of pixel, the processing performed on the read pixel signal is the same as that of the linear pixel. In other words, the case of using the pixel illustrated here can be described in the same manner as described in Implementation 2, but the part about the linear reading mode described in Implementation 2 is regarded as the part of the reading mode for reading the pixel. For example, the neural network of the linear reading mode is regarded as the neural network of the reading mode for reading the pixel.

[0228] (Implementation 3)

[0229] In the second embodiment, the logarithmic pixel and the linear pixel exist independently as different pixels, but in the third embodiment, a pixel having both the functions of a logarithmic pixel and a linear pixel (hereinafter referred to as a "hybrid pixel") is used.

[0230] Fig.16 2 is a block diagram illustrating a configuration example of a solid-state imaging device according to Embodiment 3. Here, a hybrid pixel 21C is used instead of Fig.11 The logarithmic pixel 21A and the linear pixel 21B in the pixel array unit 11. The mixed pixel 21C corresponds to an example of a second pixel whose type is different from that of the logarithmic pixel 21A.

[0231] Fig.17 2 indicates an example of a circuit configuration of a hybrid pixel 21C. The configuration of the hybrid pixel 21C is basically Figure 3 The configuration of logarithmic pixels in Fig.13 The configuration of linear pixels in (which are integrated into one pixel). Fig.17 In the drawings, the same reference numerals are used to denote Figure 3 or Fig.13 However, Figure 3 Change SEL in to SEL1, and Fig.13The SEL and TRG in the code are changed to SEL2 and TRG2. Figure 3 and Fig.13 The differences in the configurations are described below.

[0232] The hybrid pixel 21C includes a photodiode 71, a logarithmic pixel reading unit (first reading unit) 75, and a linear pixel reading unit (second reading unit) 76. The logarithmic pixel reading unit 75 and the linear pixel reading unit 76 are connected. Figure 3 The configuration of the logarithmic pixel in corresponds to (excluding the photodiode 71), and the linear pixel reading unit 76 is Fig.13 The configuration of the linear pixel in FIG. 7 corresponds to that of the linear pixel in FIG. 7 (excluding the photodiode 71). The logarithmic pixel reading unit 75 further includes a transfer transistor 72. The transfer transistor 72 is provided between the phototransistor 71 and the connection node N1.

[0233] In the hybrid pixel 21C, the photodiode 71 is commonly connected to the logarithmic pixel reading unit 75 and the linear pixel reading unit 76. The photodiode 71 is commonly used in the logarithmic reading mode and the linear reading mode. In the logarithmic reading mode of embodiment 3, the logarithmic pixel reading unit 75 is used to consult the reading (first reading process), and in the linear reading mode, the linear pixel reading unit 76 is used to perform the reading (second reading process). The cathode terminal of the photodiode 71 is connected to the source terminal of the transfer transistor 62 and the source terminal of the transfer transistor 72. The drain terminal of the transfer transistor 72 is connected to the connection node N1. The gate terminal of the transfer transistor 72 is connected to the vertical drive unit 12. The transfer transistor 72 is controlled by the transfer signal TRG1 from the vertical drive unit 12. In the case of executing the logarithmic reading mode, the transfer transistor 72 is turned on.

[0234] In the case of executing the logarithmic reading mode, the control unit 15 keeps the transfer transistor 72 in the on state and keeps the transfer transistor 62 in the off state. In these states, [the control unit 15] reads the pixel signal using the logarithmic pixel reading unit 75. This operation is the same as the reading of the logarithmic pixel described in Embodiment 1.

[0235] On the other hand, in the case of executing the linear reading mode, the control unit 15 keeps the transfer transistor 72 in the off state. In this state, the [control unit 15] reads the pixel signal using the linear pixel reading unit 76. This operation is the same as the reading of the linear pixel described in Embodiment 2.

[0236] As described above, according to Embodiment 3, since mixed pixels are used, the pixel area can be used effectively, and therefore, more pixels can be provided even if the spatial area is the same.

[0237] (Transformation 1)

[0238] Fig.17 The configuration of the hybrid pixel 21C in FIG. 1 is an example, and various modifications are possible.

[0239] For example, you can Figure 8 The buffer circuit 40 and the reset transistor 37 are added to Fig.17 The logarithmic pixel reading unit 75 in.

[0240] Fig.18 is to add the buffer circuit 40 and the reset transistor 37 to Fig.17 An example of the circuit configuration of the logarithmic pixel reading unit 75 in FIG. The configuration and operation of the buffer circuit 40 and the reset transistor 37 are similar to those of the reference Figure 8 The CDS process performed using the buffer circuit 40 and the reset transistor 37 is also the same as that described in reference Figure 8 Same as described.

[0241] (Transformation 2)

[0242] You can Fig. 9 The event detection unit 80 in is added to Fig.17 The logarithmic pixel reading unit 75 in.

[0243] Fig.19 Instructs to add event detection unit 80 to Fig.18 An example of the circuit configuration of the logarithmic pixel reading unit 75 in FIG. The configuration and operation of the event detection unit 80 are similar to those of the reference Fig. 9 and Fig.10 The event detection unit 80 can be added to Fig.17 The logarithmic pixel reading unit 75 in.

[0244] (Transformation 3)

[0245] Fig. 20 yes Fig.16 Here, it is assumed that Fig.19 The circuit configuration of the event detection unit 80 in FIG. 8 is used for the configuration of the hybrid pixel 21C.

[0246] The control unit 15 controls the event detection unit 80 of each mixed pixel 21C and determines whether an event is detected (S301). For example, the detection of an event corresponds to the detection of a change in contrast (e.g., object detection). In the case of receiving a detection signal from each mixed pixel 21C, it is determined that an event is detected in each mixed pixel 21C, and in the case of receiving a non-detection signal or not receiving a detection signal, it is determined that an event is not detected in each mixed pixel 21C.

[0247] In the case where it is determined that an event is not detected in all mixed pixels 21C (No), when a predetermined time has elapsed, this step is repeated again. In the case where there is a mixed pixel 21C for which an event is determined to be detected (Yes), the mode data is read from the mode indication unit 25 (S302), and the value of the mode data is checked (S303).

[0248] In the case where the value of the mode data indicates the logarithmic reading mode, the processing in the logarithmic reading mode is performed on the mixed pixel 21C, and it is determined that an event is detected for the mixed pixel 21C (S304). Fig.14 In this case, CDS processing can be performed using the reset transistor 37, or the pixel signal can be read without CDS processing. When pixel data is assigned to each input node of the neural network, the same pixel data as before (data of the pixel signal read from the logarithmic pixel reading unit 75) can be used for mixed pixels in which no event is detected.

[0249] In the case where the value of the mode data indicates the linear reading mode, the processing in the linear reading mode is performed on the mixed pixel 21C, and it is determined that an event is detected for the mixed pixel 21C (S305). Fig.14 Step S205 and Fig.15 The same as steps S206 to S214 in . Still for reading from the linear pixel reading unit 76, CDS processing can be performed in the same manner as the logarithmic pixel reading unit 75 (this is also the same as implementation mode 2). When pixel data is assigned to each input node of the neural network, the same pixel data as before (data of the pixel signal read from the linear pixel reading unit 76) can be used for mixed pixels, and it is determined that no event is detected for the mixed pixels. It is assumed that in the internal buffer of the data storage unit 18 or the arithmetic processing unit 19, pixel data for the linear reading mode and the logarithmic reading mode are stored separately. The processing in the flowchart can be repeated at predetermined intervals.

[0250] Fig.21 yes Fig.16 Flow chart of another variation of the operation of the solid-state imaging device in FIG. 10 . Step S306 and step S307 after step S304 are added to Fig. 20In the flowchart. In step S306, it is determined whether the result of the image recognition in the logarithmic reading mode in step S304 satisfies the mode switching condition. For example, the mode switching condition refers to detecting a predetermined object, but the condition is not limited to this and can be any other condition. In the case where the mode switching condition is not met (no), the processing in the flowchart ends. In the case where the mode switching condition is met, the value of the mode data of the mode indication unit 25 is changed to indicate the linear reading mode. The value of the mode data can be changed by the CPU 17, or by an external device, or by any other device. Thus, the processing in the linear reading mode is performed from the next operation (S305). In other words, image recognition can be performed in the logarithmic reading mode with lower power consumption until the mode switching condition is met, and after the mode switching condition is met, the mode is switched to the linear reading mode and image recognition with higher accuracy can be performed.

[0251] According to the solid-state imaging device (see Fig.11 ) can also be executed with Fig. 20 and Fig.21 . For example, the logarithmic pixel 21A and the linear pixel 21B are made to correspond to each other in advance, and the corresponding information is stored in the internal buffer of the control unit 15 or in a storage unit accessible to the control unit 15. The control unit 15 determines whether an event is detected using the event detection unit 80 in each logarithmic pixel 21A. In the case where an event is detected, the value of the mode data is checked. If the value of the mode data indicates a logarithmic reading mode, reading is performed from the logarithmic pixel 21A. If the value of the mode data indicates a linear reading mode, reading is performed from the linear pixel 21B corresponding to the logarithmic pixel 21A. Other operations are the same as described in Variation 3.

[0252] (Configuration of electrical equipment)

[0253] The first to third solid-state imaging devices can be used in various electronic devices, such as mobile devices with imaging functions (e.g., smartphones, tablet computers, portable phones), imaging devices such as digital still cameras and video cameras, surveillance cameras, and vehicle-mounted cameras.

[0254] Fig. 22 is a block diagram depicting an example of a configuration of an electronic device according to the present technology. Fig. 22The electronic device 100 in the embodiment has an optical system including a lens group 101 and the like, an image pickup element 102, a DSP 103, a CPU 104, a display unit 105, a DSP 106, an operation unit 107, a power supply unit 108, and a memory 110. The CPU 104 and the DSP 106 constitute a processing circuit 111. The DSP 103, the CPU 104, the display unit 105, the DSP 106, the operation unit 107, the power supply unit 108, and the memory 110 are interconnected via a bus 109. For example, the image pickup element 102 corresponds to the image pickup element 10 according to Embodiments 1 to 3, the DSP 103 corresponds to the signal processing unit 16, the DSP 106 corresponds to the arithmetic processing unit 19, and the memory 110 corresponds to the data storage unit 18. For example, the processing circuit 111 corresponds to the processing unit 26 according to Embodiments 1 to 3.

[0255] The lens group 101 receives incident light from an object and forms an image on an imaging surface of the image pickup element 102. The image pickup element 102 converts the light amount of the incident light forming an image on the imaging surface by the lens group 101 into an electrical signal in units of pixels and outputs the electrical signal as a pixel signal.

[0256] The display unit 105 is a flat-panel display device such as a liquid crystal display device and an organic electroluminescent (EL) display device. [The display unit 105] displays a moving image or a still image captured by the image pickup element 102. The display unit 105 also displays the result of image recognition by the solid-state imaging device according to Embodiments 1 to 3.

[0257] The operation unit 107 sends operation instructions regarding various functions of the electronic device 100 based on the operation of the user. The power supply unit 108 supplies various types of operation power to the DSP 103, the memory 110, the display unit 105, the DSP 106, and the operation unit 107. Mode data can be supplied from the operation unit 107, the CPU 104, etc. to the mode instruction unit 25 via the input / output terminal 24 of the image pickup element 102.

[0258] The above embodiments are examples of implementing the present disclosure, and the present disclosure can be completed by various other modes. For example, the above embodiments can be modified, replaced, omitted, or combined without departing from the essence and scope of the present disclosure. The modes in which such modification, replacement, omission, etc. are performed are included within the scope of the present disclosure and are included within the scope of the present invention disclosed in the claims and their equivalents.

[0259] The effects of the present disclosure stated in the present description are examples, and other effects may also be exhibited.

[0260] The present disclosure may have the following configurations.

[0261] [1] A solid-state imaging device comprising: a pixel array unit having a plurality of first pixels, the plurality of first pixels generating electric signals having logarithmic characteristics relative to the amount of light as first pixel signals; and a processing unit configured to perform arithmetic processing of a first neural network based on a plurality of first input data and a plurality of logarithmic weighting factors, the plurality of first input data being based on the plurality of first pixel signals read from the pixel array unit, the plurality of logarithmic weighting factors logarithmically representing the strength of connections between the plurality of first nodes.

[0262] [2] A solid-state imaging device according to [1], wherein the first pixel signal indicates a voltage value, and the voltage value is proportional to the logarithm of the current value based on a predetermined value; and the processing unit generates the first input data by multiplying the value of the first pixel signal by a logarithm, the antilogarithm of the logarithm being the predetermined value.

[0263] [3] The solid-state imaging device according to [2], wherein the predetermined value is a Napier constant.

[0264] [4] The solid-state imaging device according to [2] or [3], wherein the base of the pair of numbers whose antilogarithm is a predetermined value is 2.

[0265] [5] The solid-state device according to any one of [1] to [4], further comprising a control unit, wherein:

[0266] The pixel array unit includes a plurality of second pixels, the type of the plurality of second pixels being different from that of the plurality of first pixels;

[0267] The control unit selectively performs a first reading process to read first pixel signals from a plurality of first pixels, or performs a second reading process to read a plurality of second pixel signals from a plurality of second pixels. In the case of performing the first reading process, the processing unit performs arithmetic processing of the first neural network; and

[0268] In the case of performing the second reading process, the processing unit performs arithmetic processing of the second neural network based on multiple second input data and weighting factors, the multiple second input data are based on multiple second pixel signals, and the weighting factors indicate the strength of the connection between the multiple second nodes.

[0269] [6] The solid-state imaging device according to [5], wherein the control unit selects one of the first reading process and the second reading process according to instruction data indicating one of the first reading process and the second reading process.

[0270] [7] The solid-state imaging device according to [5], wherein the second pixel is a pixel that generates an electric signal having a linear characteristic with respect to the amount of light as the second pixel signal.

[0271] [8] The solid-state imaging device according to [5], wherein the second pixel is a pixel that generates a count value of the number of incoming photons as a second pixel signal.

[0272] [9] The solid-state imaging device according to [5], wherein the second pixel is a pixel that generates a charge amount indicating a phase difference between reflected light of emitted light and emitted light as a second pixel signal.

[0273]

[10] A solid-state imaging device according to any one of [1] to [9], wherein the first pixel includes: a photoelectric conversion unit; a first reading unit that reads an electrical signal having a logarithmic characteristic from the photoelectric conversion unit as a pixel signal; and a second reading unit that reads an electrical signal having a linear characteristic from the photoelectric conversion unit as a second pixel signal. The control unit selectively performs a first reading process to read the first pixel signal using the first reading unit, or performs a second reading process to read the second pixel signal using the second reading unit. In the case of performing the first reading process, the processing unit performs arithmetic processing of the first neural network; and in the case of performing the second reading process, the processing unit performs arithmetic processing of the second neural network based on multiple second input data and weighting factors, the multiple second input data are based on the multiple second pixel signals, and the reinforcement factor represents the strength of the connection between the multiple second nodes.

[0274]

[11] A solid-state imaging device according to any one of [1] to

[10] , wherein image recognition is performed on an image indicated by a plurality of first pixel signals using arithmetic processing of a first neural network.

[0275]

[12] An electronic device comprising: an image pickup element including a pixel array unit having a plurality of first pixels, the first pixels generating an electric signal having a logarithmic characteristic relative to an amount of light as a first pixel signal; and a processing circuit configured to perform arithmetic processing on a first neural network based on a plurality of first input data and a plurality of logarithmic weighting factors, the plurality of first input data being based on the plurality of first pixel signals read from the pixel array unit, the plurality of logarithmic weighting factors logarithmically representing the strength of a connection between a plurality of first nodes.

[0276] [Reference Numbers List]

[0277] 10 Image pickup element

[0278] 11 Pixel Array Unit

[0279] 12 Vertical drive unit

[0280] 13 columns of processing units

[0281] 14 Horizontal drive unit

[0282] 15 Control Unit

[0283] 16 Signal Processing Unit

[0284] 17 CPU

[0285] 18 Data storage unit

[0286] 19 Arithmetic Processing Unit

[0287] 21 Logarithmic pixels

[0288] 21A Logarithmic Pixel

[0289] 21B Linear Pixel

[0290] 21C Hybrid Pixel

[0291] 22 pixel drive lines

[0292] 23 vertical signal line

[0293] 24 Input / Output Terminals

[0294] 25 Mode indication unit

[0295] 26 Processing Units

[0296] 31,61,71 Photodiode

[0297] 32,33,34,35,36,37,38,39,62,63,64,65,72 MOS transistors

[0298] N1, N2 connection nodes

[0299] N3 Reset Node

[0300] 40 Buffer circuit

[0301] 66 FD nodes

[0302] 41 Input layer

[0303] 42 Middle Layer

[0304] 43 Output layer

[0305] 51 Selector

[0306] 52 Adder

[0307] 53 Subtractor

[0308] 54 Absolute value arithmetic unit

[0309] 55 Exponential arithmetic unit

[0310] 56 Trigger Circuit

[0311] 80 Event Detection Unit

[0312] 81 Subtractor

[0313] 82 Quantizer

[0314] 83,84 Capacitors

[0315] 85 Inverter

[0316] 86 Switch

[0317] 87 Comparator

[0318] 100 Electronics

[0319] 101 lens group

[0320] 102 Image pickup element

[0321] 103,106 DSP

[0322] 104 CPU

[0323] 105 Display unit

[0324] 107 Operating unit

[0325] 108 Power supply unit

[0326] 109 Bus

[0327] 110 Memory

[0328] A1 to Aq Input nodes (input unit)

[0329] B1 to Bj Intermediate nodes (arithmetic unit)

[0330] C1 to Cn Output nodes (arithmetic unit).

Claims

1. A solid-state imaging device, comprising: A pixel array unit having a plurality of first pixels, wherein the plurality of first pixels generate a plurality of electrical signals having a logarithmic characteristic with respect to the amount of light as a plurality of first pixel signals; as well as a processing unit configured to perform arithmetic processing of a first neural network based on a plurality of first input data based on a plurality of first pixel signals read from the pixel array unit and a plurality of logarithmic weighting factors representing the strength of connections between a plurality of first nodes in the first neural network by logarithms; as well as A control unit, wherein The pixel array unit includes a plurality of second pixels, and the types of the plurality of second pixels are different from those of the plurality of first pixels; The control unit selectively performs a first reading process to read a plurality of first pixel signals from a plurality of first pixels, or performs a second reading process to read a plurality of second pixel signals from a plurality of second pixels; In case of performing the first reading process, the processing unit performs arithmetic processing of the first neural network; and When performing the second reading process, the processing unit performs arithmetic processing of the second neural network based on multiple second input data and weighting factors, wherein the multiple second input data are based on multiple second pixel signals, and the weighting factors indicate the strength of the connection between multiple second nodes in the second neural network.

2. The solid-state imaging device according to claim 1, wherein The first pixel signal indicates a voltage value, and the voltage value is proportional to a logarithm of a current value based on a predetermined value; and The processing unit generates the first input data by multiplying the value of the first pixel signal by a logarithm, an antilogarithm of the logarithm being the predetermined value.

3. The solid-state imaging device according to claim 2, wherein: The predetermined value is the Napier constant.

4. The solid-state imaging device according to claim 2, wherein: The base of the logarithm is 2, and the antilogarithm of the logarithm is the predetermined value.

5. The solid-state imaging device according to claim 1, wherein The control unit selects one of the first reading process and the second reading process according to instruction data indicating one of the first reading process and the second reading process.

6. The solid-state imaging device according to claim 1, wherein The second pixel is a pixel that generates an electric signal having a linear characteristic with respect to the light amount as the second pixel signal.

7. The solid-state imaging device according to claim 1, wherein The second pixel is a pixel that generates a count value of the number of incoming photons as the second pixel signal.

8. The solid-state imaging device according to claim 1, wherein The second pixel is a pixel that generates an amount of charge indicating a phase difference between reflected light of emitted light and the emitted light as the second pixel signal.

9. The solid-state imaging device according to claim 1, wherein The processing unit performs image recognition on an image indicated by a plurality of the first pixel signals using the arithmetic processing of the first neural network.

10. An electronic device comprising: An image pickup element including a pixel array unit having a plurality of first pixels, the plurality of first pixels generating a plurality of electric signals having a logarithmic characteristic with respect to the amount of light as a plurality of first pixel signals; as well as a processing circuit configured to perform arithmetic processing of a first neural network based on a plurality of first input data based on a plurality of first pixel signals read from the pixel array unit and a plurality of logarithmic weighting factors representing the strength of connections between a plurality of first nodes in the first neural network by logarithms; as well as A control unit, wherein The pixel array unit includes a plurality of second pixels, and the types of the plurality of second pixels are different from those of the plurality of first pixels; The control unit selectively performs a first reading process to read a plurality of first pixel signals from a plurality of first pixels, or performs a second reading process to read a plurality of second pixel signals from a plurality of second pixels; In case of performing the first reading process, the processing circuit performs arithmetic processing of the first neural network; and When performing the second reading process, the processing circuit performs arithmetic processing of the second neural network based on multiple second input data and weighting factors, wherein the multiple second input data are based on multiple second pixel signals, and the weighting factors indicate the strength of the connection between multiple second nodes in the second neural network.

11. A solid-state imaging device comprising: A pixel array unit having a plurality of mixed pixels, wherein the plurality of mixed pixels generate a plurality of electrical signals having logarithmic characteristics with respect to the amount of light as a plurality of first pixel signals; as well as a processing unit configured to perform arithmetic processing of a first neural network based on a plurality of first input data based on a plurality of first pixel signals read from the pixel array unit and a plurality of logarithmic weighting factors representing the strength of connections between a plurality of first nodes in the first neural network by logarithms, and A control unit, wherein The hybrid pixel includes: a photoelectric conversion unit; a first reading unit that reads an electrical signal having a logarithmic characteristic from the photoelectric conversion unit as the first pixel signal; and a second reading unit that reads an electrical signal having a linear characteristic from the photoelectric conversion unit as a second pixel signal; The control unit selectively performs a first reading process to read the first pixel signal using the first reading unit, or performs a second reading process to read the second pixel signal using the second reading unit; In case of performing the first reading process, the processing unit performs arithmetic processing of the first neural network; and When performing the second reading process, the processing unit performs arithmetic processing of the second neural network based on multiple second input data and weighting factors, wherein the multiple second input data are based on multiple second pixel signals, and the weighting factors indicate the strength of the connection between multiple second nodes in the second neural network.

Images