Solid-state imaging device and image processing system

Abstract

Provided is a solid-state imaging device capable of reducing power consumption by reducing operation cost compared with the prior art. This solid-state imaging device is provided with: a plurality of photoelectric conversion elements that photoelectrically convert optical signals each having a predetermined pixel color and output pixel signals; and a control circuit that performs control so as to simultaneously transfer the pixel signals from the plurality of photoelectric conversion elements to an output terminal, thereby outputting a pixel luminance signal having a predetermined characteristic quantity. Here, the feature quantity is a horizontal edge quantity. The plurality of photoelectric conversion elements include: a photoelectric conversion element that photoelectrically converts an optical signal having a red signal level R and outputs a pixel signal; and two photoelectric conversion elements that photoelectrically converts an optical signal having a green signal level G and outputs a pixel signal. And a photoelectric conversion element that photoelectrically converts an optical signal having a blue signal level B and outputs a pixel signal. The solid-state imaging device outputs, for example, a pixel luminance signal having a luminance conversion formula Y = 2G + R + B.

Description

Technical Field

[0001] The present invention relates to a solid-state imaging device such as an image sensor and an image processing system including the solid-state imaging device. Background Art

[0002] Image recognition systems extract image features. Some image recognition systems use machine learning models, for example. In image recognition using machine learning models, for example, features are extracted from a high-resolution input image, and the image is classified (clustered) based on the features.

[0003] For example, Non-Patent Document 1 discloses an image sensor that performs feature quantity calculations (convolution operations) within the chip. The image sensor comprises multiple pixels. Each pixel includes a photoelectric conversion element (photodiode) and a readout circuit. The readout circuit for each pixel outputs an electrical signal corresponding to the intensity of received light, known as a pixel signal. The image sensor disclosed in Non-Patent Document 1 requires a special manufacturing process (IGZO) to perform the accumulation and multiplication operations of the photoelectric conversion element used for the convolution operation.

[0004] For example, Patent Document 1 discloses an image sensor implemented using a suitable CMOS process for large-scale integrated circuits. This image sensor outputs a feature quantity used for image recognition and includes a plurality of pixel circuits and a controller configured to execute a first mode control of the plurality of pixel circuits. The plurality of pixel circuits each include a photoelectric conversion element and a charge storage unit that receives and transfers charge from the photoelectric conversion element, and the controller is configured to output a pixel signal corresponding to the amount of charge stored in the charge storage unit. The first mode control controls the transfer of charge between the photoelectric conversion element and the charge storage unit for the purpose of calculating the feature quantity.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2022-102604

[0008] Non-patent literature

[0009] Non-patent document 1: Seiichi Yoneda et al., "Image Sensor Capable of AnalogConvolution for Real-time Image Recognition System Using Crystalline OxideSemiconductor FET", International Image Sensor Workshop (IISW), 2019, pp.322-325.

[0010] Non-Patent Document 2: Yamamoto Kohei et al., "Image Classification Using Neural Networks for CMOS Image Sensors for Feature Extraction," Research Technology Report of the Society of Image Information and Media, Vol. 46, No. 29, pp. 21-24, published on September 15, 2022

[0011] Non-Patent Document 3: Okumura Shuhei et al., "Research on Low-Cost CMOS Color Image Sensors with Extractable Feature Quantities," Integrated Circuit Research Group (ICD), Institute of Electronics and Information Communications, Student and Youth Research Group, published on March 22, 2022

[0012] Non-Patent Document 4: Yuu Ohsuga et al., "Analysis of the Impact of Extractable Feature Quantities in CMOS Image Sensors on Image Recognition," Integrated Circuit Research Group (ICD), Institute of Electronics and Information Communications, Student and Youth Research Group, published on March 22, 2022

[0013] Non-Patent Literature 5: Norifumi Egami, "The Fundamentals of Image Input Devices (1st) - The Fundamentals of Photoelectric Conversion," Journal of the Society of Image Information and Media, Vol. 68, No. 1, 2014 Summary of the Invention

[0014] Problems to be solved by the invention

[0015] As in the conventional example described above, the image sensor capable of extracting feature quantities only performs feature quantity calculations within the chip, and other processing is performed in the image recognizer. Therefore, the output data volume of the solid-state imaging device is large, resulting in relatively high calculation costs and difficulty in reducing power consumption.

[0016] An object of the present invention is to solve the above-mentioned problems and to provide a solid-state imaging device capable of reducing power consumption by reducing computational costs compared to conventional technologies, and an image processing system including the solid-state imaging device.

[0017] Means for solving problems

[0018] A solid-state imaging device according to one embodiment of the present disclosure includes:

[0019] a plurality of photoelectric conversion elements for photoelectrically converting light signals having respective predetermined pixel colors and outputting pixel signals; and

[0020] The control circuit controls so as to simultaneously forward the pixel signals from the plurality of photoelectric conversion elements to the output terminal, thereby outputting a pixel brightness signal having a predetermined characteristic amount.

[0021] Effects of the Invention

[0022] Therefore, according to the solid-state imaging device according to one embodiment of the present disclosure, it is possible to reduce power consumption by reducing computational costs compared to conventional technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a block diagram showing a configuration example of an image sensor which is a solid-state imaging device according to the first embodiment.

[0024] Figure 2 Yes Figure 1 A block diagram of a structural example of a pixel portion.

[0025] Figure 3 Yes Figure 1 A circuit diagram showing a structural example of a pixel circuit and its peripheral circuits.

[0026] Figure 4 This is a timing chart showing an operation example of a conventional pixel circuit and its peripheral circuits.

[0027] Figure 5 Yes Figure 3 A timing diagram showing an operation example of a pixel circuit and its peripheral circuits.

[0028] Figure 6A It means for Figure 3 This diagram shows an example of the arrangement of color filters in a Bayer array of four photodiodes.

[0029] Figure 6B It means for Figure 3 Diagram showing an example of the arrangement of color filters for an RGBR array of four photodiodes.

[0030] Figure 6C It means for Figure 3 Diagram showing an example of the arrangement of color filters for an RBBG array of four photodiodes.

[0031] Figure 7 This is a graph showing the characteristics of light intensity versus light penetration depth in a general image sensor.

[0032] Figure 8 This is a circuit diagram showing a configuration example of a pixel circuit and its peripheral circuits according to Embodiment 2.

[0033] Figure 9 Yes Figure 8 A timing diagram showing an operation example of a pixel circuit and its peripheral circuits.

[0034] Figure 10A This is a graph showing the probability density distribution of a general horizontal edge signal.

[0035] Figure 10B Yes Figure 8 A diagram of the input-output characteristics of a comparator.

[0036] Figure 11 This is a circuit diagram showing a configuration example of a pixel circuit and its peripheral circuits according to Embodiment 3.

[0037] Figure 12 This is the method used in the fourth embodiment. Figure 1 A block diagram showing a configuration example of an image classification system using an image sensor.

[0038] Figure 13 This is the method used in the fifth embodiment. Figure 1 A block diagram of a configuration example of an image recognition system using an image sensor.

[0039] Figure 14 This is the method used in the sixth embodiment. Figure 1 A block diagram of a configuration example of an image recognition system using an image sensor. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments and modifications of the present invention will be described with reference to the accompanying drawings. The same or similar components are denoted by the same reference numerals.

[0041] (Implementation Method 1)

[0042] Figure 1 1 is a block diagram showing a configuration example of an image sensor 100 which is a solid-state imaging device according to the first embodiment. Figure 1 The image sensor 100 is an image sensor used in, for example, an image recognition system.

[0043] exist Figure 1In the image sensor 100, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used. The image sensor 100 includes a pixel unit 110 (an imaging unit) and a controller 120. The image sensor 100 also includes a readout circuit 130, a vertical scanning circuit 140, and a horizontal scanning circuit 150. The readout circuit 130 controls the reading of pixel signals from the pixel unit 110. The vertical scanning circuit 140 provides the pixel unit 110 with a signal that selects the pixel circuit (target pixel) from which pixel signals are to be read out, among the pixel circuits 111 (pixels) included in the pixel unit 110. The horizontal scanning circuit 150 provides the readout circuit 130 with a column select signal that selects the target pixel column from which pixel signals are to be output. Consequently, the readout circuit 130 outputs feature image data containing feature pixel signals from the pixel unit 110. Here, the feature image data includes, for example, a horizontal edge luminance signal, as described later.

[0044] The controller 120 is a control circuit that controls the operation of the pixel unit 110 and the readout circuit 130 via the vertical scanning circuit 140 and the horizontal scanning circuit 150. The controller is not limited to the aforementioned controller 120, but may also include the vertical scanning circuit 140 and the horizontal scanning circuit 150.

[0045] Figure 2 Yes Figure 1 A block diagram showing an example of the structure of the pixel unit 110. Figure 2 In FIG, the pixel portion 110 includes a plurality of pixel circuits 111, 111A, 111B, and 111C arranged in a two-dimensional array. Pixel circuits may also be simply referred to as pixels.

[0046] Figure 3 Yes Figure 1 A circuit diagram showing a configuration example of pixel circuits 111A and 111C and their peripheral circuits. Figure 3 , a pixel circuit 111A in the Jth row and a pixel circuit 111C in the J+1th row are shown. The plurality of pixel circuits 111, 111A, and 111C included in the pixel portion 110 have a common structure. Figure 3 , the pixel circuit 111C is depicted in a simplified manner compared to the pixel circuit 111A for the sake of convenience, but actually has the same structure as the pixel circuit 111A.

[0047] Here, the structure of the pixel circuit will be described using pixel circuit 111A in row J as an example. Pixel circuit 111A includes four photoelectric conversion elements PD1 through PD4. Photoelectric conversion elements PD1 through PD4 are, for example, photodiodes. Photoelectric conversion elements PD1 through PD4 accumulate an amount of electron charge corresponding to the amount of light. Here, the charge generated by photoelectric conversion is referred to as photoelectrically converted charge, and the electrons generated by photoelectric conversion are referred to as photoelectrically converted electrons.

[0048] The pixel circuit 111A includes forwarding gates TG1-TG4 connected to the photoelectric conversion elements PD1-PD4, respectively. The forwarding gates TG1-TG4 forward the charge accumulated in the photoelectric conversion elements PD1-PD4, respectively. Each of these gates is comprised of, for example, a MOS transistor. When an H-level signal is applied between the gate and source, the drain and source become conductive, turning the MOS transistor on. When an L-level signal is applied between the gate and source, the drain and source become non-conductive, turning the MOS transistor off. When each forwarding gate TG1-TG4 is conductive, the charge accumulated in the photoelectric conversion elements PD1-PD4 is forwarded to a charge storage unit described below. When the forwarding gate TG is off, the forwarding of the charge accumulated in the photoelectric conversion elements PD1-PD4 is prevented.

[0049] Pixel circuit 111A includes a charge storage unit that receives charge from photoelectric conversion elements PD1-PD4 via a forwarding gate TG. In an embodiment, the charge storage unit includes at least a floating diffusion (FD), a capacitor connected to the forwarding gates TG1-TG4. The floating diffusions FD are connected to the forwarding gates TG1-TG4, respectively, to receive charge from the photoelectric conversion elements PD1-PD4.

[0050] The pixel circuit 111A includes a reset gate RST connected to a charge storage unit. The charge storage unit is connected to a floating diffusion layer FD and a switching gate SG. The reset gate RST is connected to the floating diffusion layer FD and a power supply voltage V DD When the reset gate RST is turned on, the charge stored in the charge storage unit, i.e., the floating diffusion layer FD or the switch gate SG, is discharged to the power supply. This discharge of charge is called a reset. By resetting, the potential of the charge storage unit becomes the power supply voltage V DD If the reset gate RST is turned off, the discharge of charges is prevented.

[0051] The pixel circuit 111A includes a source follower transistor SF that forms a source follower circuit 20 together with a constant current source CS described later. The source follower transistor SF is also called an amplifier transistor. The gate terminal of the source follower transistor SF is connected to the floating diffusion layer FD (charge storage unit), and the drain terminal is connected to the power supply voltage V DD The source follower transistor SF generates a voltage vx (pixel signal) at its source terminal that corresponds to the amount of charge stored in the charge storage unit (and the capacitor of the charge storage unit). The voltage vx (pixel signal) generated at the source terminal of the source follower transistor SF is derived from the voltage V in the floating diffusion layer FD. FD Reduces the drain-source voltage V of the source follower transistor SF GS The voltage after the amount (V FD -V GS ).

[0052] Pixel circuit 111A further includes a select gate SEL. When select gate SEL is on, the source terminal of source follower transistor SF is connected to signal line 21 for reading pixel signals. Select gate SEL is used to select a target pixel circuit (target pixel) from among the plurality of pixel circuits 111A, 111C, and 111, from which the voltage in the charge storage unit is to be read. The voltage in the charge storage unit of the target pixel circuit is read by readout circuit 130. When select gate SEL is off, source follower transistor SF is disconnected from signal line 135.

[0053] Pixel circuit 111A further includes a switch gate SG. When switch gate SG is on, the charge storage unit of pixel circuit 111A is connected to the charge storage unit of another pixel circuit 111C. When switch gate SG is off, the charge storage unit of pixel circuit 111A is disconnected from the charge storage units of other pixel circuits 111C. Specifically, switch gate SG is an element that switches between connecting and disconnecting the charge storage unit provided in pixel circuit 111A, where switch gate SG is provided, and the charge storage units provided in other pixel circuits 111C adjacent to pixel circuit 111A. Furthermore, when switch gate SG of pixel circuit 111A is on, the charge storage unit (floating diffusion layer FD) of pixel circuit 111A is connected to the charge storage unit (floating diffusion layer FD) of pixel circuit 111C.

[0054] The readout circuit 130 includes a constant current source CS that forms the source follower circuit 20 together with the source follower transistor SF. The constant current source CS is connected to the signal line 135. The readout circuit 130 includes a CDS capacitor C provided in parallel with the constant current source CS and midway between the signal line 135. CDS . CDS capacitor C CDSThis is used for correlated double sampling, described later. The readout circuit 130 also includes a clamp gate CLP. When clamp gate CLP is on, signal line 135 is connected to ground GND. When clamp gate CLP is off, signal line 135 is disconnected from ground GND. The aforementioned gates are driven and controlled by the controller 120.

[0055] In the image sensor 100 constructed as described above, a pixel arrangement such as the Bayer array is typically used, with four pixels (photoelectric conversion elements PD1 through PD4) forming a single cell. These four pixels are assigned color filters, respectively, red (R), green (G), and blue (B), hereinafter referred to as the R filter, G filter, and B filter, respectively. For example, the R filter is assigned to photoelectric conversion element PD1, the G filter is assigned to photoelectric conversion elements PD2 and PD3, and the B filter is assigned to photoelectric conversion element PD4 (see Figure 6 below). Each pixel is incident with light of a specific wavelength that has passed through the color filters, and photoelectrically converted electrons are accumulated in each of the photoelectric conversion elements PD1 through PD4. To read out the electrons accumulated in each of the photoelectric conversion elements PD1 through PD4 to a node on the floating diffusion layer FD, forwarding gates TG1 through TG4 are connected. PDs 1 through 4 share a common source follower transistor SF for reading electrons forwarded to a node on the floating diffusion layer FD, a select gate SEL for selecting a specific pixel, and a reset gate RST for resetting the floating diffusion layer FD. Furthermore, PDs 1 through 4 share a common switch gate SG for connecting pixel cells.

[0056] Figure 4 : is a timing diagram showing an operation example of a pixel circuit and its peripheral circuits in a conventional example. Figure 5 Yes Figure 3 A timing diagram showing an operation example of the pixel circuits 111A and 111C and their peripheral circuits.

[0057] like Figure 4 As shown in the timing diagram of the conventional example, to output the feature image, the switch gate SG of the Jth row is first turned on, and then the reset gate RST of the Jth row is turned on to reset the floating diffusion layer FD. Next, the forwarding gate TG1 of the Jth row is turned on to forward the accumulated charge of the R pixel to the floating diffusion layer FD. Here, when the clamp gate CLP is turned off, the signal output from the pixel array (photoelectric conversion element PD1) via the source follower transistor SF is stored in the CDS capacitor C. CDS Maintain in.

[0058] The reset gate RST of the Jth row is turned on again to reset the floating diffusion layer FD. Next, the forwarding gate TG1 of the J+1th row is turned on to forward the accumulated charge of the R pixel to the floating diffusion layer FD. Based on the signal output from the pixel array (photoelectric conversion element PD1) via the source follower transistor SF and the CDS capacitor C CDS The output signal vout becomes the differential signal (=horizontal edge signal) between the R pixels (photoelectric conversion elements PD1 ) in the J+1th row and the Jth row.

[0059] Similarly, the differential signals of the pixels of the photoelectric conversion elements PD2, PD3, and PD4 are obtained. The differential signals of each color are used in, for example, an image recognition device ( Figure 13 40, Figure 14 After converting the signal into a brightness signal using the following formula, image recognition based on artificial intelligence (AI), for example, deep machine learning, is performed.

[0060] Y=0.299R+0.587G+0.114B (1)

[0061] On the other hand, as the present embodiment relates to Figure 5 As shown in the timing diagram of , by turning on the forward gates TG1, TG2, TG3, and TG4 simultaneously, the accumulated charges of the four photoelectric conversion elements PD1 to PD4 can be simply converted into the pixel brightness signal Y through the floating diffusion layer FD using the following formula.

[0062] Y=R+2G+B (2)

[0063] Here, R, G, and B represent the signal levels of the respective colors.

[0064] As described above, in the conventional example, to sequentially output horizontal edge signals for the four pixels of photoelectric conversion elements PD1 to PD4, a total of four horizontal edge signals must be output. In contrast, in this embodiment, only one horizontal edge signal is output, reducing the signal intensity by one-quarter. Furthermore, since the accumulated signals from the four photoelectric conversion elements PD1 to PD4 are combined in the floating diffusion layer FD, the exposure time required to achieve the same signal intensity as the conventional example can be shortened to one-quarter, thereby improving sensitivity. Since the signal intensity is quadrupled while the exposure time remains constant, the signal-to-noise ratio of the floating diffusion layer FD to reset noise can be quadrupled.

[0065] Figure 6A Indicates that Figure 3 Diagram showing an example of the arrangement of color filters for a Bayer array of four photodiodes. Figure 6A This is the arrangement of the color filters described in the above-mentioned embodiment 1. In addition to the Bayer array, a method in which the number of R filters is increased can also be used. Figure 6BRGBR array, or a larger number of B filters Figure 6C RBBG array.

[0066] Figure 7 This is a graph showing the characteristics of light intensity with respect to the depth into which light penetrates in a general image sensor (for example, refer to Non-Patent Document 5).

[0067] Humans are highly sensitive to changes in green brightness, so increasing the number of G filters can improve visual resolution. However, when considering image classification based on machine learning, RGBR arrays and RBBG arrays can be used. This is based on the relationship between the depth of light penetration into the image sensor and light intensity. In a typical image sensor, the depth of the photoelectric conversion element PD is approximately 2μm. While short-wavelength blue light is fully absorbed, long-wavelength red light is also transmitted by over 60%. This means that while nearly 100% of blue light is photoelectrically converted, only about 40% of red light is photoelectrically converted.

[0068] Therefore, when using an RGBR array, the luminance conversion formula is Y=2R+G+B, which can compensate for the lack of sensitivity to red light. Alternatively, when using an RGGB array, the luminance conversion formula is Y=R+G+2B, which can enhance the high sensitivity of blue light and produce a high-brightness signal.

[0069] As described above, this embodiment can generate and output feature image data containing, for example, horizontal edge signals (including horizontal edge feature quantities). The RGBR array can compensate for insufficient red light sensitivity, while the RGGB array can generate high-luminance signals, thereby improving classification (clustering) accuracy depending on the application and image classification target. In other words, it is possible to construct an image sensor capable of extracting feature quantities whose luminance conversion formulas are Y=2R+G+B or Y=R+G+2B.

[0070] (Implementation Method 2)

[0071] Figure 8 1 is a circuit diagram showing a configuration example of a pixel circuit 111 and its peripheral circuits according to Embodiment 2. Figure 9 Yes Figure 8 A timing diagram showing an operation example of the pixel circuit 111 and its peripheral circuits. Figure 8 The circuit and Figure 3 The circuit has the following differences compared to the

[0072] (1) An AD converter (ADC) 11 and a column switch COL are provided between the output terminal of the pixel circuit 111 of each column and the signal line 22. Here, the column switch COL is composed of, for example, a MOS transistor controlled by the controller.

[0073] (2) A feedback (return) circuit (threshold setting circuit) having a flip-flop 13 is provided between the signal line 22 and the output terminal 25 of the readout circuit 130 to provide a hysteresis characteristic ( Figure 10B ) of the comparator 12. That is, the comparator 12 is provided in the readout circuit 130.

[0074] The following describes the differences.

[0075] exist Figure 8 In the embodiment, the horizontal edge signal from the pixel circuit 111 of each column is converted into analog / digital by the AD converter 11 provided in each column. For example, the AD conversion resolution is set to 10 bits. The digital output signal from the pixel circuit 111 of each column is as follows: Figure 9 As shown in the timing diagram, the column switches COL of each column are sequentially turned on, and the signal is selectively read out to the signal line 22. The read 10-bit digital signal is binarized and converted into a 1-bit digital signal by the comparator 12, and then output as an output signal (pixel signal) via the output terminal 25. In other words, the comparator 12 functions as a binarizer. Here, when binarizing the signal of the (I+1)th row, the flip-flop 13 sets the already binarized and stored signal of the first column as the binarization threshold.

[0076] Figure 10A is a graph showing the probability density distribution of a general horizontal edge signal. Figure 10B Yes Figure 8 A graph of the input-output characteristics of the comparator 12 is shown.

[0077] The horizontal edge signal is as follows Figure 10A As shown, the values ​​are distributed around 0, which has no edge. However, in addition to tiny edges, the area around 0 also contains noise. Therefore, simply binarizing at values ​​above or below 0 amplifies the noise and degrades the horizontal edge signal. Therefore, when binarizing the (I+1)th row, if the signal value in row 1 is 0, the threshold is set high to increase the probability of judging it as 0 (L level). Conversely, if the signal value in row 1 is 1, the threshold is set high to increase the probability of judging it as 1 (H level).

[0078] As described above, according to Embodiment 2, by binarizing the horizontal edge signals of adjacent pixels, identical codes are more likely to be continuous, reducing high-frequency noise. This noise reduction improves image classification accuracy. Furthermore, binarization can reduce the amount of data to 1 / 10.

[0079] (Implementation Method 3)

[0080] Figure 11 This is a circuit diagram showing a configuration example of a pixel circuit 111 and its peripheral circuits according to Embodiment 3. Figure 11 The circuit and Figure 8 The circuit has the following differences compared to .

[0081] (1) Instead of the comparator 12 and the flip-flop 13 provided immediately before the output terminal 25, a comparator 12 and a column switch COL are provided between the AD converter 11 of each column and the output terminal 25. Here, for example, the output signal of the comparator 12 of the I column is used as the threshold value of the comparator 12 of the adjacent (I+1) column, thereby achieving the same Figure 8 The adjacent comparator 12 is similarly configured to receive a threshold value from the comparator 12 in the preceding adjacent column.

[0082] As described above, according to Embodiment 3, by binarizing the horizontal edge signals of adjacent pixels, identical codes are more likely to appear continuously, reducing high-frequency noise. This noise reduction improves image classification accuracy. Furthermore, binarization reduces the amount of data required to a tenth. Furthermore, by reducing the number of bits in the horizontal forwarding signal from 10 to 1, the power consumption required for horizontal forwarding is reduced.

[0083] (Implementation Method 4)

[0084] Figure 12 This indicates that the method according to the fourth embodiment is used. Figure 1 FIG. 1 is a block diagram of a configuration example of an image classification system using the image sensor 100 . Figure 12 The image classification system has the ability to extract feature quantities Figure 1 The image sensor 100 , the convolutional neural network (CNN) 30 , the feature extraction circuit 31 , the public dataset memory 32 with labels, and the feature dataset memory 33 with labels.

[0085] exist Figure 12 In the example, the public dataset memory 32 with labels stores a public dataset with labels consisting of general color image data (referring to the true labels used for clustering ( ) and color image data paired datasets). Feature extraction circuit 31 calculates and extracts predetermined features based on the labeled public dataset and stores them in labeled feature dataset memory 33. Specifically, labeled feature dataset memory 33 stores datasets that are paired with true labels, color image data, and features used for clustering.

[0086] The convolutional neural network (CNN) 30 is a well-known CNN. During learning, the convolutional neural network 30 uses a plurality of data sets stored in the feature data set memory 33 with labels for learning. During classification, the convolutional neural network 30 is based on the feature data sets that can extract features such as horizontal edges. Figure 1 The clustering process is performed on the color image data of the image sensor 100, thereby obtaining and outputting classification result data.

[0087] In this embodiment, features are extracted from a labeled feature dataset to generate a feature learning dataset for low-cost training of the image classification device, CNN 30. Using this feature learning dataset to train CNN 30 improves the accuracy of image classification using a CMOS image sensor capable of extracting features.

[0088] As described above, according to this embodiment, the training of the deep learning-based image classifier, namely CNN 30, requires the collection of a large number of images and the labeling of these images by skilled workers, which is very costly. However, by generating a feature value learning dataset based on a publicly available dataset with labels including color image data, this labor cost can be significantly reduced, and a high-precision image classification system involving feature values ​​can be implemented at a low cost.

[0089] In the above embodiment, the CNN 30 is provided, but the present invention is not limited thereto, and a deep neural network (DNN) configured by a well-known deep learning model may be provided.

[0090] (Implementation method 5)

[0091] Figure 13 This indicates that the method according to the fifth embodiment is used. Figure 1 FIG. 1 is a block diagram showing a configuration example of an image recognition system using an image sensor 100 . Figure 13 The image recognition system has the ability to extract features such as horizontal edges Figure 1 The image sensor 100 and the image recognition device 40 having the recognition detection circuit 41 are configured.

[0092] exist Figure 13In the image sensor 100, feature image data processed by an image filter including edges (differentiations) such as those used in edge extraction, is output to the recognition detection circuit 41 of the image recognition device 40. The recognition detection circuit 41 is configured using, for example, a machine learning model and outputs a recognition result of the image data based on the input feature image data.

[0093] As described above, according to this embodiment, a pixel filter can be constructed within the image sensor 100, reducing the amount of filtering required in image processing circuits such as the image recognition device 40. By simultaneously compressing the data volume with the filtering function, the interface speed between the solid-state imaging element of the image sensor 100 and the image recognition device 40 can be reduced, thereby reducing current consumption.

[0094] (Implementation Method 6)

[0095] Figure 14 This indicates that the method according to Implementation 6 is used. Figure 1 FIG. 1 is a block diagram showing a configuration example of an image recognition system using an image sensor 100 . Figure 14 Image recognition system and Figure 13 Compared with the image recognition system, the following points are different.

[0096] (1) Instead of the image recognition device 40 , an image recognition device 40A including a learning result memory 42 and a recognition detection circuit 41A is provided.

[0097] The following describes the differences.

[0098] exist Figure 14 In the example, learning result memory 42 stores a labeled dataset from which features such as horizontal edges are extracted. Recognition detection circuit 41A is configured using, for example, a machine learning model. During learning, it learns based on the dataset in learning result memory 42. During recognition, it outputs recognition results based on feature image data input from image sensor 100.

[0099] As described above, according to this embodiment, by using the recognition and detection circuit 41A that has been learned using the labeled dataset from which the feature quantities are extracted, the recognition and detection circuit 41A has an optimal learning model in the image sensor 100 and can perform recognition and detection with high accuracy.

[0100] (Variation)

[0101] The image classification system according to the fourth embodiment and the image recognition system according to the fifth or sixth embodiment are examples of image processing systems.

[0102] Industrial applicability

[0103] As described in detail above, the solid-state imaging device according to the present invention enables simplified luminance signal conversion simply by changing the pixel control timing within the solid-state imaging device, enabling low-noise binarization using simple circuitry. Consequently, by changing the pixel control method and adding a small-scale circuit, the amount of data required can be significantly reduced compared to conventional technologies, reducing computational costs and power consumption. In particular, by performing luminance signal conversion and binarization in addition to feature quantity calculations within the solid-state imaging device's image sensor, the amount of data communicated between the image sensor and the image recognizer can be reduced, significantly reducing the power consumption of the image recognition system.

[0104] Description of Reference Numerals

[0105] 11 AD converter (ADC)

[0106] 12 Comparators

[0107] 13 Triggers

[0108] 20 Source Follower Circuit

[0109] 21, 22, 23 signal lines

[0110] 25 output terminals

[0111] 30 Convolutional Neural Networks (CNNs)

[0112] 31 Feature extraction circuit

[0113] 32 public dataset storage with labels

[0114] 33 Labeled feature dataset storage

[0115] 40, 40A Image recognition device

[0116] 41, 41A identification detection circuit

[0117] 42 Learning result storage

[0118] 100 Image sensor (solid-state imaging device)

[0119] 110 pixel unit

[0120] 111, 111A, 111B, 111C pixel circuits

[0121] 120 controller

[0122] 130 Readout circuit

[0123] 140 vertical scanning circuit

[0124] 150 horizontal scanning circuit

[0125] C CDS CDS capacitors

[0126] COL column switch

[0127] CLP Clamp Gate

[0128] CS constant current source

[0129] FD floating diffusion layer

[0130] PD1~PD4 photoelectric conversion elements

[0131] Q1~Q15 MOS transistors

[0132] RST reset gate

[0133] SF source follower transistor

[0134] SEL select gate

[0135] SG Switch Gate

[0136] TG, TG1~TG4 forwarding grid

[0137] vout output signal (pixel signal)

[0138] vx pixel signal.

Claims

1. A solid-state imaging device comprising: a plurality of photoelectric conversion elements for photoelectrically converting light signals having respective predetermined pixel colors and outputting pixel signals; and The control circuit controls so as to simultaneously forward the pixel signals from the plurality of photoelectric conversion elements to the output terminal, thereby outputting a pixel brightness signal having a predetermined characteristic amount.

2. The solid-state imaging device according to claim 1, wherein The feature quantity is a horizontal edge quantity.

3. The solid-state imaging device according to claim 1, wherein The plurality of photoelectric conversion elements include: The photoelectric conversion element photoelectrically converts a light signal having a red signal level R into a pixel signal. Two photoelectric conversion elements, which photoelectrically convert a light signal having a green signal level G into a pixel signal; as well as The photoelectric conversion element photoelectrically converts a light signal having a blue signal level B into a pixel signal. The solid-state imaging device outputs a pixel brightness signal whose brightness conversion formula is Y=2G+R+B.

4. The solid-state imaging device according to claim 1, wherein The plurality of photoelectric conversion elements include: Two photoelectric conversion elements, which photoelectrically convert a light signal having a red signal level R into a pixel signal; The photoelectric conversion element performs photoelectric conversion on a light signal having a green signal level G and outputs a pixel signal; as well as The photoelectric conversion element photoelectrically converts a light signal having a blue signal level B into a pixel signal. The solid-state imaging device outputs a pixel brightness signal whose brightness conversion formula is Y=G+2R+B.

5. The solid-state imaging device according to claim 1, wherein The plurality of photoelectric conversion elements include: The photoelectric conversion element photoelectrically converts a light signal having a red signal level R into a pixel signal. The photoelectric conversion element performs photoelectric conversion on a light signal having a green signal level G and outputs a pixel signal; as well as The two photoelectric conversion elements photoelectrically convert the light signal with a blue signal level B into a pixel signal. The solid-state imaging device outputs a pixel brightness signal whose brightness conversion formula is Y=G+R+2B.

6. The solid-state imaging device according to any one of claims 1 to 5, further comprising: The binarizer uses the pixel luminance signals of adjacent columns as a threshold value, binarizes the pixel luminance signals from the solid-state imaging device, and outputs the binarized signals.

7. The solid-state imaging device according to claim 6, wherein The binarizer is provided at a subsequent stage of each column of pixel circuits including the plurality of photoelectric conversion elements.

8. The solid-state imaging device according to claim 6, wherein The binarizer is provided in a readout circuit that reads out pixel luminance signals from pixel circuits corresponding to respective columns of the plurality of photoelectric conversion elements.

9. An image processing system comprising: The solid-state imaging device according to any one of claims 1 to 5; and a neural network that performs clustering processing based on pixel luminance signals from the solid-state imaging device, in, The neural network is learned based on a feature quantity dataset with real labels, which includes feature quantities extracted from a public dataset with real labels.

10. An image processing system comprising: The solid-state imaging device according to any one of claims 1 to 5; and An image recognition device that performs image recognition processing based on a pixel luminance signal from the solid-state imaging device.

11. The image processing system according to claim 10, wherein: The image recognition device is learned based on a public dataset with real labels.

Images