Solid-state imaging device and image processing system

Abstract

To provide a solid-state imaging device that can reduce power consumption by reducing the computation cost compared to the prior art.SOLUTION: A solid-state imaging device comprises: a plurality of photoelectric conversion elements that output pixel signals by photoelectrically converting optical signals, each of which has a predetermined pixel color; and a control circuit that outputs a pixel luminance signal having a predetermined feature quantity, by performing control such that the pixel signals from the plurality of photoelectric conversion elements are transferred to an output terminal simultaneously. The feature quantity is a horizontal edge quantity. The plurality of photoelectric conversion elements include: a photoelectric conversion element that outputs a pixel signal by photoelectrically converting an optical signal having a red color signal level R; two photoelectric conversion elements that output a pixel signal by photoelectrically converting an optical signal having a green color signal level G; and a photoelectric conversion element that outputs a pixel signal by photoelectrically converting an optical signal having a blue color signal level B. The solid-state imaging device outputs, for example, a pixel luminance signal for which the luminance conversion equation is Y=2G+R+B.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a solid-state imaging device, such as an image sensor, and an image processing system including the solid-state imaging device. [Background technology]

[0002] In an image recognition system, feature amounts of an image are extracted. For example, a machine learning model is used for image recognition. In the case of image recognition using a machine learning model, for example, feature amounts of a high-resolution image as an input image are extracted, and the images are classified (clustered) based on the feature amounts.

[0003] For example, Non-Patent Document 1 discloses an image sensor that performs feature calculations (convolution calculations) inside a chip. The image sensor includes multiple pixels. Each pixel includes a photodiode, which is a photoelectric conversion element, and a readout circuit. The readout circuit of each pixel outputs a pixel signal, which is an electrical signal corresponding to the intensity of received light. The image sensor disclosed in Non-Patent Document 1 requires a special manufacturing process (IGZO) for accumulation and multiplication of the photoelectric conversion elements for performing the convolution calculations.

[0004] An image sensor for implementing feature calculations inside a chip using a CMOS process suitable for large-scale integrated circuits is disclosed in, for example, Patent Document 1. This image sensor is an image sensor that outputs feature amounts for image recognition, and includes a plurality of pixel circuits and a controller configured to execute a first mode control for controlling the plurality of pixel circuits, each of the plurality of pixel circuits including a photoelectric conversion element and a charge accumulation unit that receives charge transfer from the photoelectric conversion element, and is configured to output a pixel signal according to the amount of charge accumulated in the charge accumulation unit, and the first mode control includes controlling the transfer of the charge between the photoelectric conversion element and the charge accumulation unit for calculations to obtain the feature amounts. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-102604 A [Non-patent literature]

[0006] [Non-Patent Document 1] Seiichi Yoneda et al., "Image Sensor Capable of Analog Convolution for Real-time Image Recognition System Using Crystalline Oxide Semiconductor FET", International Image Sensor Workshop(IISW), 2019, pp.322-325. [Non-Patent Document 2] Kohei Yamamoto et al., "Image Classification Using Neural Networks for CMOS Image Sensors Capable of Feature Extraction," Institute of Image Information and Television Engineers Research and Technical Report, Vol. 46, No. 29, pp. 21-24, published September 15, 2022 [Non-Patent Document 3] Shuhei Okumura et al., "Study on low-cost CMOS color image sensor capable of feature extraction," Institute of Electronics, Information and Communication Engineers, Integrated Circuits Research Committee (ICD), Student and Young Researchers Research Committee, March 22, 2022 [Non-Patent Document 4] Hirotaka Osuga et al., "Analysis of the impact of features extractable within CMOS image sensors on image recognition," presented at the Institute of Electronics, Information and Communication Engineers' Integrated Circuits Study Group (ICD), Student and Young Researchers' Study Group, March 22, 2022 [Non-Patent Document 5] Norifumi Egami, "Fundamentals of Image Input Devices (Part 1) Fundamentals of Photoelectric Conversion", Journal of the Institute of Image Information and Television Engineers, Vol. 68, No. 1, 2014 Summary of the Invention [Problem to be solved by the invention]

[0007] In the image sensors capable of extracting features according to the above-described conventional examples, only feature calculation is performed within the chip, and other processing is performed by the image recognizer. As a result, the amount of output data from the solid-state imaging device is large, which poses the problem that the calculation costs are relatively high and it is difficult to reduce power consumption.

[0008] An object of the present invention is to solve the above problems and provide a solid-state imaging device capable of reducing power consumption by reducing the calculation cost compared to the conventional techniques, and an image processing system including the solid-state imaging device. [Means for solving the problem]

[0009] A solid-state imaging device according to an embodiment of the present disclosure includes: a plurality of photoelectric conversion elements, each of which photoelectrically converts an optical signal having a predetermined pixel color and outputs a pixel signal; a control circuit for controlling the pixel signals from the plurality of photoelectric conversion elements to be simultaneously transferred to an output terminal, thereby outputting a pixel luminance signal having a predetermined characteristic amount; Equipped with. Effect of the Invention

[0010] Therefore, according to the solid-state imaging device according to one aspect of the present disclosure, the calculation cost is smaller than that of the conventional technology, thereby reducing power consumption. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of an image sensor which is a solid-state imaging device according to a first embodiment. [Diagram 2] 2 is a block diagram showing an example of the configuration of a pixel unit in FIG. 1. [Diagram 3] 2 is a circuit diagram showing a configuration example of the pixel circuit and its peripheral circuits in FIG. 1. [Figure 4] 1 is a timing chart showing an example of the operation of a conventional pixel circuit and its peripheral circuits. [Diagram 5] 4 is a timing chart showing an example of the operation of the pixel circuit and its peripheral circuits in FIG. [Figure 6A] 4 is a diagram showing an example of the arrangement of color filters in a Bayer array for the four photodiodes in FIG. 3. [Figure 6B] 4 is a diagram showing an example of the arrangement of color filters in an RGBR arrangement for the four photodiodes in FIG. 3. [Figure 6C] FIG. 4 is a diagram showing an example of the arrangement of color filters in an RBBG arrangement for the four photodiodes in FIG. [Figure 7] 1 is a graph showing characteristics of light intensity versus the depth of penetration of light in a typical image sensor. [Figure 8] FIG. 11 is a circuit diagram showing a configuration example of a pixel circuit and its peripheral circuits according to a second embodiment. [Figure 9] 9 is a timing chart showing an example of the operation of the pixel circuit and its peripheral circuits in FIG. 8. [Figure 10A] 1 is a graph showing a probability density distribution of a typical horizontal edge signal. [Figure 10B] 9 is a graph showing input / output characteristics of the comparator of FIG. 8. [Figure 11] FIG. 11 is a circuit diagram showing a configuration example of a pixel circuit and its peripheral circuits according to a third embodiment. [Figure 12] FIG. 13 is a block diagram showing an example of the configuration of an image classification system using the image sensor of FIG. 1 according to a fourth embodiment. [Figure 13] FIG. 13 is a block diagram showing an example of the configuration of an image recognition system using the image sensor of FIG. 1 according to a fifth embodiment. [Figure 14] FIG. 13 is a block diagram showing an example of the configuration of an image recognition system using the image sensor of FIG. 1 according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments and modifications of the present invention will be described with reference to the drawings. Note that the same or similar components are denoted by the same reference numerals.

[0013] (Embodiment 1) Fig. 1 is a block diagram showing an example of the configuration of an image sensor 100 which is a solid-state imaging device according to embodiment 1. The image sensor 100 in Fig. 1 is an image sensor for, for example, an image recognition system.

[0014] In FIG. 1, the image sensor 100 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 100 includes a pixel unit 110 as an imaging unit and a controller 120. The image sensor 100 further includes a readout circuit 130, a vertical scanning circuit 140, and a horizontal scanning circuit 150. The readout circuit 130 controls the reading out of pixel signals from the pixel unit 110. The vertical scanning circuit 140 provides the pixel unit 110 with a signal for selecting a target pixel circuit (target pixel) from which a pixel signal is to be read out of the pixel circuits 111 (pixels) included in the pixel unit 110. The horizontal scanning circuit 150 provides a column selection signal for selecting a target pixel column from which a pixel signal is to be outputted out of the readout circuit 130. As a result, the readout circuit 130 outputs feature image data including a feature pixel signal from the pixel unit 110. Here, the feature image data is image data including, for example, a horizontal edge luminance signal as described later.

[0015] The controller 120 is a control circuit that controls the operations of the pixel unit 110 and the readout circuit 130 via the vertical scanning circuit 140 and the horizontal scanning circuit 150. Note that the controller may be a concept that includes not only the above-mentioned controller 120 but also the vertical scanning circuit 140 and the horizontal scanning circuit 150.

[0016] Fig. 2 is a block diagram showing an example of the configuration of the pixel section 110 of Fig. 1. In Fig. 2, the pixel section 110 includes a plurality of pixel circuits 111, 111A, 111B, and 111C arranged in a two-dimensional array. Note that the pixel circuits are also simply called pixels.

[0017] Fig. 3 is a circuit diagram showing a configuration example of the pixel circuits 111A and 111C in Fig. 1 and their peripheral circuits. In Fig. 3, one pixel circuit 111A in the Jth row and one pixel circuit 111C in the J+1th row are shown. The pixel circuits 111, 111A, and 111C included in the pixel unit 110 have a common structure. Note that in Fig. 3, the pixel circuit 111C is depicted in a simplified manner compared to the pixel circuit 111A for convenience, but in reality, it has a similar structure to the pixel circuit 111A.

[0018] Here, the structure of the pixel circuit will be described using the pixel circuit 111A in the Jth row as an example. The pixel circuit 111A includes four photoelectric conversion elements PD1 to PD4. The photoelectric conversion elements PD1 to PD4 are, for example, photodiodes. The photoelectric conversion elements PD1 to PD4 accumulate an amount of electronic charge according to the amount of light. Here, the charge generated by photoelectric conversion is called a photoelectric conversion charge, and the electrons generated by photoelectric conversion are called photoelectric conversion electrons.

[0019] The pixel circuit 111A includes transfer gates TG1 to TG4 connected to the photoelectric conversion elements PD1 to PD4, respectively. The transfer gates TG1 to TG4 transfer the charges accumulated in the photoelectric conversion elements PD1 to PD4, respectively. Each of the following gates is, for example, composed of a MOS transistor. When an H level signal is applied between the gate and source, the drain and source are brought into a conductive state and the MOS transistor is turned on, and when an L level signal is applied between the gate and source, the drain and source are brought into a non-conductive state and the MOS transistor is turned off. When each of the transfer gates TG1 to TG4 is turned on, it transfers the charges accumulated in the photoelectric conversion elements PD1 to PD4 to a charge accumulation section, which will be described later. When the transfer gate TG is turned off, the transfer of the charges accumulated in the photoelectric conversion elements PD1 to PD4 is prevented.

[0020] The pixel circuit 111A includes a charge accumulation unit that receives charge transfer from the photoelectric conversion elements PD1 to PD4 via the transfer gates TG. In the embodiment, the charge accumulation unit has at least a floating diffusion FD (floating diffusion layer) that is a capacitor connected to the transfer gates TG1 to TG4. Here, the floating diffusion FD is connected to the transfer gates TG1 to TG4, respectively, so as to receive the charge transfer from the photoelectric conversion elements PD1 to PD4.

[0021] The pixel circuit 111A includes a reset gate RST connected to a charge storage unit. The charge storage unit is connected to a floating diffusion FD and a switching gate SG. The reset gate RST is connected to the floating diffusion FD and a power supply voltage V DD The reset gate RST is connected between the charge storage section and the power supply voltage V. When the reset gate RST is turned on, the charge stored in the charge storage section, the floating diffusion FD or the switching gate SG, is discharged to the power supply. Discharging such charge is called resetting. By resetting, the potential of the charge storage section is reduced to the power supply voltage V. DD When the reset gate RST is turned off, discharge of the charge is prevented.

[0022] The pixel circuit 111A includes a source follower transistor SF that configures a source follower circuit 20 together with a constant current source CS described later. The source follower transistor SF is also called an amplification transistor. The source follower transistor SF has a gate terminal connected to a floating diffusion FD (charge storage unit) and a drain terminal connected to a power supply voltage V DD The source follower transistor SF generates a voltage vx (pixel signal) at its source terminal according to the amount of charge stored in the charge storage section (and the capacitor of the charge storage section). The voltage vx (pixel signal) generated at the source terminal of the source follower transistor SF is a voltage V FD to the drain-source voltage V of the source follower transistor SF GS The voltage drop (V FD-V GS ).

[0023] The pixel circuit 111A further includes a selection gate SEL. When the selection gate SEL is turned on, it connects the source terminal of the source follower transistor SF to a signal line 21 for reading out a pixel signal. The selection gate SEL is an element for selecting a target pixel circuit (target pixel) from among the multiple pixel circuits 111A, 111C, and 111, which is to be the target for reading out a voltage in a charge accumulation section. The voltage in the charge accumulation section of the target pixel circuit is read out by a readout circuit 130. When the selection gate SEL is turned off, it disconnects the source follower transistor SF from the signal line 135.

[0024] The pixel circuit 111A further includes a switching gate SG. When the switching gate SG is turned on, it connects the charge accumulation unit of the pixel circuit 111A to the charge accumulation unit of another pixel circuit 111C. When the switching gate SG is turned off, it disconnects the charge accumulation unit of the pixel circuit 111A from the charge accumulation unit of another pixel circuit 111C. That is, the switching gate SG is an element for switching between connection / disconnection between the charge accumulation unit provided in the pixel circuit 111A in which the switching gate SG is provided and the charge accumulation unit provided in another pixel circuit 111C adjacent to the pixel circuit 111A. When the switching gate SG of the pixel circuit 111A is turned on, the charge accumulation unit (floating diffusion FD) of the pixel circuit 111A is connected to the charge accumulation unit (floating diffusion FD) of the pixel circuit 111C.

[0025] The read circuit 130 includes a constant current source CS that constitutes the source follower circuit 20 together with the above-mentioned source follower transistor SF. The constant current source CS is connected to a signal line 135. The read circuit 130 includes a CDS capacitor C CDS The CDS capacitor C CDSis used for correlated double sampling, which will be described later. The readout circuit 130 further includes a clip gate CLP. When the clip gate CLP is turned on, it connects the signal line 135 to the ground GND. When the clip gate CLP is turned off, it disconnects the signal line 135 from the ground GND. Note that the above-mentioned gates are driven and controlled by the controller 120.

[0026] In the image sensor 100 configured as above, a pixel arrangement such as a Bayer arrangement is generally used, and four pixels (photoelectric conversion elements PD1 to PD4) form one cell. These four pixels are assigned color filters such as red (R), green (G), and blue (B), and are hereinafter referred to as an R filter, a G filter, and a B filter, respectively. For example, an R filter is arranged on the photoelectric conversion element PD1, G filters are arranged on the photoelectric conversion elements PD2 and PD3, and a B filter is arranged on the photoelectric conversion element PD4 (see FIG. 6 described later). Light of a specific wavelength that has passed through the color filters is incident on each pixel, and photoelectric conversion electrons are accumulated in each of the photoelectric conversion elements PD1 to PD4. Transfer gates TG1 to TG4 are connected to read out the electrons accumulated in each of the photoelectric conversion elements PD1 to PD4 to a node of the floating diffusion FD. A source follower transistor SF for reading out electrons transferred to the node of the floating diffusion FD, a selection gate SEL for selecting a specific pixel, and a reset gate RST for resetting the floating diffusion FD are shared by each of PD1 to PD4. In addition, a switching gate SG for connecting pixel cells is also shared by each of PD1 to PD.

[0027] Fig. 4 is a timing chart showing an example of the operation of a conventional pixel circuit and its peripheral circuits, and Fig. 5 is a timing chart showing an example of the operation of pixel circuits 111A and 111C in Fig. 3 and their peripheral circuits.

[0028] As shown in the conventional timing chart of FIG. 4, in order to output a feature image, the switching gate SG of the Jth row is first turned on, and then the reset gate RST of the Jth row is turned on to reset the floating diffusion FD. Next, the transfer gate TG1 of the Jth row is turned on to transfer the accumulated charge of the R pixel to the floating diffusion FD. Here, the signal output from the pixel array (photoelectric conversion element PD1) via the source follower transistor SF is transferred to the CDS capacitor C when the clip gate CLP is turned off. CDS is held in

[0029] The reset gate RST in the Jth row is turned on again to reset the floating diffusion FD. Next, the transfer gate TG1 in the J+1th row is turned on to transfer the accumulated charge of the R pixel to the floating diffusion FD. The signal output from the pixel array (photoelectric conversion element PD1) via the source follower transistor SF and the charge stored in the CDS capacitor C CDS From the signals held in , the output signal vout becomes a differential signal (=horizontal edge signal) of the R pixels (photoelectric conversion elements PD1) in the (J+1)th row and the Jth row.

[0030] Similarly, differential signals of the photoelectric conversion elements PD2, PD3, and PD4 are obtained. The differential signals of each color are converted into luminance signals using the following formula, for example, by an image recognition device (40 in FIG. 13, 40A in FIG. 14), and then image recognition is performed using, for example, deep machine learning artificial intelligence (AI).

[0031] Y=0.299R+0.587G+0.114B (1)

[0032] On the other hand, as shown in the timing chart of FIG. 5 according to this embodiment, by simultaneously turning on the transfer gates TG1, TG2, TG3, and TG4, the accumulated charges of the four photoelectric conversion elements PD1 to PD4 in the floating diffusion FD can be simply converted into a pixel luminance signal Y using the following equation.

[0033] Y=R+2G+B (2)

[0034] Here, R, G, and B represent the signal levels of each color.

[0035] As described above, in the conventional example, in order to sequentially output horizontal edge signals of the four pixels of the photoelectric conversion elements PD1 to PD4, it is necessary to output the horizontal edge signal a total of four times. In contrast, in this embodiment, the horizontal edge signal is output only once, and the signal amount can be reduced to 1 / 4. In addition, since the accumulations of the four photoelectric conversion elements PD1 to PD4 are added together by the floating diffusion FD, the exposure time required to obtain the same signal amount as in the conventional example can be shortened to 1 / 4, and the sensitivity can be improved. When the exposure time is constant, the signal amount is four times larger, and therefore the signal-to-noise ratio to the reset noise of the floating diffusion FD can be improved by four times.

[0036] Fig. 6A is a diagram showing an example of the arrangement of color filters in a Bayer array for the four photodiodes in Fig. 3. Fig. 6A shows the color filter arrangement described in the above-mentioned embodiment 1. In addition to this Bayer array, it is possible to use an RGBR array in Fig. 6B which increases the number of R filters, or an RBBG array in Fig. 6C which increases the number of B filters.

[0037] FIG. 7 is a graph showing the characteristics of light intensity versus the penetration depth of light in a typical image sensor (see, for example, Non-Patent Document 5).

[0038] Since humans are highly sensitive to changes in the brightness of green, increasing the number of G filters can increase the visual resolution. However, when considering image classification using machine learning, an RGBR array or an RBBG array can be adopted. This is based on the relationship between the depth at which light penetrates the image sensor and the light intensity. In a typical image sensor, the depth of the photoelectric conversion element PD is about 2 μm, and while short-wavelength blue light is sufficiently absorbed, more than 60% of long-wavelength red light is transmitted. This means that nearly 100% of blue light is photoelectrically converted, but only about 40% of red light is photoelectrically converted.

[0039] Therefore, when an RGBR array is used, the luminance conversion formula is Y=2R+G+B, which makes it possible to compensate for the lack of sensitivity to red light. Also, when an RGGB array is used, the luminance conversion formula is Y=R+G+2B, which makes it possible to obtain a high luminance signal by emphasizing the blue light, which has high sensitivity.

[0040] As described above, according to this embodiment, for example, feature image data including a horizontal edge signal (including a feature of a horizontal edge) can be generated and output, the RGBR array can compensate for the lack of sensitivity to red light, and the RGGB array can obtain a high luminance signal, thereby improving classification (clustering) accuracy according to the application and the image classification target. That is, an image sensor can be configured that can extract features such as Y=2R+G+B or Y=R+G+2B in the luminance conversion formula.

[0041] (Embodiment 2) Fig. 8 is a circuit diagram showing a configuration example of a pixel circuit 111 and its peripheral circuits according to embodiment 2, and Fig. 9 is a timing chart showing an operation example of the pixel circuit 111 and its peripheral circuits in Fig. 8. The circuit in Fig. 8 has the following differences from the circuit in Fig. 3. (1) An AD converter (ADC) 11 and a column switch COL are provided between the output terminal of the pixel circuit 111 of each column and the signal line 22. Here, the column switch COL is composed of, for example, a MOS transistor controlled by the controller. (2) A comparator 12 having a hysteresis characteristic (FIG. 10B) and a feedback circuit (threshold setting circuit) of a flip-flop 13 is provided between the signal line 22 and the output terminal 25 of the read circuit 130. That is, the comparator 12 is provided in the read circuit 130. The differences will be explained below.

[0042] In FIG. 8, horizontal edge signals from pixel circuits 111 of each column are analog-to-digital converted by AD converters 11 provided in each column. For example, the AD conversion resolution is assumed to be 10 bits. As shown in the timing chart of FIG. 9, the digital output signals from pixel circuits 111 of each column are selectively read out to signal lines 22 by sequentially turning on column switches COL of each column. The read out 10-bit digital signals are binarized into 1-bit digital signals by comparator 12, and then output as output signals (pixel signals) via output terminals 25. That is, comparator 12 functions as a binarizer (binarizer). Here, when binarizing the signal of the I+1th row, flip-flop 13 sets the signal of the Ith column, which has already been binarized and stored, as the binarization threshold value.

[0043] FIG. 10A is a graph showing a probability density distribution of a typical horizontal edge signal, and FIG. 10B is a graph showing the input / output characteristics of the comparator 12 of FIG.

[0044] As shown in FIG. 10A, the horizontal edge signal is distributed with its values ​​centered around 0, which is an edgeless value. However, in addition to minute edges, noise is included near 0. Therefore, simply binarizing above / below 0 amplifies the noise and deteriorates the horizontal edge signal. Therefore, when binarizing the I+1th row, if the signal value of the Ith row is 0, the threshold is set high to increase the probability of determining it as 0 (L level). Conversely, if the signal value of the Ith row is 1, the threshold is set high to increase the probability of determining it as 1 (H level).

[0045] As described above, according to the second embodiment, binarization is performed based on the horizontal edge signals of adjacent pixels, which makes it easier for the same code to appear consecutively, thereby reducing high-frequency noise. By reducing noise, the accuracy of image classification is improved. Furthermore, by binarizing, the amount of data can be reduced to 1 / 10.

[0046] (Embodiment 3) Fig. 11 is a circuit diagram showing a configuration example of a pixel circuit 111 and its peripheral circuits according to embodiment 3. The circuit in Fig. 11 has the following differences from the circuit in Fig. 8. (1) Instead of the comparator 12 and flip-flop 13 provided immediately before the output terminal 25, a comparator 12 and a column switch COL are inserted between the AD converter 11 of each column and the output terminal 25. Here, for example, by using the output signal of the comparator 12 in the Ith column as the threshold value of the comparator 12 in the adjacent (I+1)th column, a function similar to that of the flip-flop 13 in Fig. 8 is realized. The adjacent comparator 12 is similarly configured to input a threshold value from the comparator 12 in the column adjacent in the forward direction.

[0047] As described above, according to the third embodiment, binarization is performed based on the horizontal edge signals of adjacent pixels, which makes it easier for the same code to appear consecutively, thereby reducing high-frequency noise. By reducing noise, the accuracy of image classification is improved. Furthermore, by binarizing, the amount of data can be reduced to 1 / 10. Furthermore, by reducing the number of bits of the horizontal transfer signal from 10 bits to 1 bit, the power consumption required for horizontal transfer is reduced.

[0048] (Embodiment 4) Fig. 12 is a block diagram showing an example of the configuration of an image classification system using the image sensor 100 of Fig. 1 according to embodiment 4. The image classification system of Fig. 12 includes the image sensor 100 of Fig. 1 capable of extracting features, a convolutional neural network (CNN) 30, a feature extraction circuit 31, a labeled public dataset memory 32, and a labeled feature dataset memory 33.

[0049] 12, the labeled public dataset memory 32 stores a labeled public dataset (meaning a dataset in which a correct answer label for clustering is paired with color image data) consisting of general color image data. The feature extraction circuit 31 calculates and extracts a predetermined feature based on the labeled public dataset, and stores it in the labeled feature dataset memory 33. That is, the labeled feature dataset memory 33 stores a dataset in which a correct answer label for clustering is paired with color image data and a feature.

[0050] The convolutional neural network (CNN) 30 is a known CNN that, during learning, is trained using a plurality of data sets stored in a labeled feature dataset memory 33, and then, during classification, performs a clustering process based on color image data from the image sensor 100 in FIG. 1 that can extract features such as horizontal edges, and obtains and outputs classification result data.

[0051] In this embodiment, features are extracted from a labeled feature dataset to generate a feature training dataset for training an image classification device, CNN 30, at low cost. The accuracy of image classification using a CMOS image sensor capable of extracting features is improved by training CNN 30 using this feature training dataset.

[0052] As described above, according to this embodiment, in order to train the CNN30, which is an image classifier using deep learning, a large number of images must be collected and an expert must label the images, which requires a large cost. However, by generating a feature learning dataset based on a labeled public dataset that includes color image data, it is possible to significantly reduce personnel costs and realize a low-cost, highly accurate image classification system related to features.

[0053] In the above embodiment, the CNN 30 is provided, but the present invention is not limited to this, and the CNN 30 may be provided with a deep neural network (DNN) configured with a known deep learning model.

[0054] (Embodiment 5) Fig. 13 is a block diagram showing an example of the configuration of an image recognition system using the image sensor 100 of Fig. 1 according to embodiment 5. The image recognition system of Fig. 13 includes the image sensor 100 of Fig. 1 capable of extracting feature amounts such as horizontal edges, and an image recognition device 40 having a recognition detection circuit 41.

[0055] 13, an image sensor 100 outputs feature image data that has been subjected to image filtering processing, including edges (differentials) used in edge extraction, to a recognition detection circuit 41 of an image recognition device 40. The recognition detection circuit 41 is configured using, for example, a machine learning model, and outputs a recognition result of image data from the input feature image data.

[0056] As described above, according to this embodiment, a pixel filter can be configured completely within the image sensor 100, and the amount of filtering required in an image processing circuit such as the image recognition device 40 can be reduced. By compressing the amount of data simultaneously with the filtering function, the interface speed between the solid-state imaging element of the image sensor 100 and the image recognition device 40 can be reduced, thereby reducing current consumption.

[0057] (Embodiment 6) Fig. 14 is a block diagram showing an example of the configuration of an image recognition system using the image sensor 100 of Fig. 1 according to embodiment 6. The image recognition system of Fig. 14 differs from the image recognition system of Fig. 13 in the following points. (1) Instead of the image recognition device 40, an image recognition device 40A including a learning result memory 42 and a recognition detection circuit 41A is provided. The differences will be explained below.

[0058] 14, the learning result memory 42 stores a labeled data set after feature extraction such as a horizontal edge. The recognition detection circuit 41A is configured using, for example, a machine learning model, and during learning, the circuit learns based on the data set in the learning result memory 42, and during recognition, the circuit outputs a recognition result of image data based on the feature image data input from the image sensor 100.

[0059] As described above, according to this embodiment, by using the recognition and detection circuit 41A that has been trained using a labeled data set after feature extraction, the recognition and detection circuit 41A is provided with an optimal learning model for the image sensor 100, and is able to perform recognition / detection with high accuracy.

[0060] (Modification) The image classification system according to the fourth embodiment and the image recognition system according to the fifth or sixth embodiment are, for example, examples of an image processing system. [Industrial Applicability]

[0061] As described above in detail, the solid-state imaging device according to the present invention can perform simple luminance signal conversion by simply changing the pixel control timing in the solid-state imaging device, and can perform binarization with low noise using a simple circuit. As a result, by changing the pixel control method and adding a small-scale circuit, the amount of data can be significantly reduced compared to the conventional technology, and the calculation cost can be reduced, thereby reducing power consumption. In particular, by performing luminance signal conversion and binarization processing in addition to feature calculation within the image sensor of the solid-state imaging device, the amount of data communication between the image sensor and the image recognition device can be reduced, and the power consumption of the image recognition system can be significantly reduced. [Explanation of symbols]

[0062] 11 Analog-to-Digital Converter (ADC) 12 Comparator 13. Flip-flop 20 Source follower circuit 21, 22, 23 Signal lines 25 Output terminal 30 Convolutional Neural Network (CNN) 31 Feature Extraction Circuit 32 Labeled Public Dataset Memory 33 Labeled feature dataset memory 40,40A Image recognition device 41, 41A Recognition detection circuit 42 Learning result memory 100 Image sensor (solid-state imaging device) 110 Pixel section 111, 111A, 111B, 111C pixel circuit 120 Controller 130 Readout circuit 140 Vertical scanning circuit 150 Horizontal scanning circuit C CDS CDS Capacitor COL Column switch CLP Clip Gate CS constant current source FD Floating Diffusion PD1~PD4 Photoelectric conversion elements Q1~Q15 MOS transistors RST Reset gate SF Source follower transistor SEL Select Gate SG Switching Gate TG, TG1~TG4 Transfer gate vout Output signal (pixel signal) vx pixel signal

Claims

1. A plurality of photoelectric conversion elements that photoelectrically convert optical signals each having a predetermined pixel color and output pixel signals, and a control circuit that outputs a pixel luminance signal having a predetermined feature amount by controlling to transfer each pixel signal from the plurality of photoelectric conversion elements to an output terminal simultaneously. A solid-state imaging device comprising the above.

2. The feature amount is a horizontal edge amount. The solid-state imaging device according to Claim 1.

3. The plurality of photoelectric conversion elements include: a first photoelectric conversion element that photoelectrically converts an optical signal having a red signal level R and outputs a first pixel signal; two second photoelectric conversion elements that photoelectrically convert an optical signal having a green signal level G and output a second pixel signal; and a third photoelectric conversion element that photoelectrically converts an optical signal having a blue signal level B and outputs a third pixel signal. The solid-state imaging device outputs a pixel luminance signal whose luminance conversion formula is Y = 2G + R + B. The solid-state imaging device according to Claim 1.

4. The plurality of photoelectric conversion elements include: two first photoelectric conversion elements that photoelectrically convert an optical signal having a red signal level R and output a first pixel signal; a second photoelectric conversion element that photoelectrically converts an optical signal having a green signal level G and output a second pixel signal; and a third photoelectric conversion element that photoelectrically converts an optical signal having a blue signal level B and outputs a third pixel signal. The solid-state imaging device outputs a pixel luminance signal whose luminance conversion formula is Y = G + 2R + B. The solid-state imaging device according to Claim 1.

5. The plurality of photoelectric conversion elements include: a first photoelectric conversion element that photoelectrically converts an optical signal having a red signal level R and outputs a first pixel signal; a second photoelectric conversion element that photoelectrically converts an optical signal having a green signal level G and output a second pixel signal; and two third photoelectric conversion elements that photoelectrically convert an optical signal having a blue signal level B and output a third pixel signal. The solid-state imaging device outputs a pixel luminance signal whose luminance conversion formula is Y = G + R + 2B. The solid-state imaging device according to Claim 1.

6. The solid-state imaging device further includes a binarizer that binarizes and outputs the pixel luminance signal from the solid-state imaging device using the pixel luminance signals of adjacent columns as thresholds. The solid-state imaging device according to any one of Claims 1 to 5.

7. The binarizer is provided at a subsequent stage of a pixel circuit of each column including each of the plurality of photoelectric conversion elements. The solid-state imaging device according to Claim 6.

8. The solid-state imaging device further includes a binarizer that binarizes and outputs the pixel luminance signal from the solid-state imaging device using the pixel luminance signals of adjacent columns as thresholds. The solid-state imaging device according to Claim 6.

8. The binary converter is provided in a readout circuit that reads out pixel luminance signals from the pixel circuits of each column for the plurality of photoelectric conversion elements, The solid-state imaging device according to claim 6.

9. A solid-state imaging device according to any one of claims 1 to 5, An image processing system comprising: a neural network that executes clustering processing based on a pixel luminance signal from the solid-state imaging device, wherein the neural network is trained based on a feature amount dataset with correct labels including feature amounts extracted from a publicly available dataset with correct labels, Image processing system.

10. A solid-state imaging device according to any one of claims 1 to 5, and an image recognition device that executes image recognition processing based on a pixel luminance signal from the solid-state imaging device, An image processing system comprising the same.

11. The image recognition device is trained based on a publicly available dataset with correct labels, The image processing system according to claim 10.

Images