SOLID-STATE IMAGING DEVICE AND IMAGE PROCESSING SYSTEM

Abstract

A solid-state imaging device is provided that can reduce power consumption by lowering computation costs compared to the prior art. It includes: photoelectric conversion elements that output pixel signals by photoelectrically converting optical signals each having a predetermined pixel color; and a controller that outputs a pixel luminance signal with a predetermined feature amount by controlling the pixel signals from the photoelectric conversion elements to be simultaneously transmitted to an output terminal. The feature amount is a horizontal edge amount.The photoelectric conversion elements include: one photoelectric conversion element that outputs a pixel signal by photoelectrically converting an optical signal with a red color signal level R; two photoelectric conversion elements that output a pixel signal by photoelectrically converting an optical signal with a green color signal level G; and one photoelectric conversion element that outputs a pixel signal by photoelectrically converting an optical signal with a blue color signal level B. For example, the solid-state imaging device outputs a pixel luminance signal for which the luminance conversion equation is Y = 2 G + R + B.

Description

TECHNICAL AREA

[0001] The present invention relates to a solid-state imaging device, such as an image sensor, and an image processing system comprising the solid-state imaging device. TECHNICAL BACKGROUND

[0002] In an image recognition system, a set of features is extracted from an image. Image recognition includes, for example, a process that uses a machine learning model. In image recognition using a machine learning model, a set of features is extracted from a high-resolution image that serves as input, and image classification (clustering) is performed based on this set of features.

[0003] For example, non-patent literature 1 discloses an image sensor that performs feature set computation (convolution computation) within a chip. The image sensor contains a plurality of pixels. Each pixel contains a photodiode, which is a photoelectric conversion element, and a readout circuit. The readout circuit of each pixel outputs a pixel signal, which is an electrical signal corresponding to the intensity of the received light. The image sensor disclosed in non-patent literature 1 requires a special manufacturing process (IGZO) for storing and multiplying the photoelectric conversion elements for performing the convolution computation.

[0004] Patent document 1, for example, discloses an image sensor for realizing feature quantity calculation within a chip by means of a suitable complementary metal-oxide semiconductor (CMOS) process of a large-scale integrated circuit.This image sensor is an image sensor that outputs a feature set for image recognition, wherein the image sensor contains: a plurality of pixel circuits; and a controller configured to perform a first mode control of the plurality of pixel circuits, wherein each of the plurality of pixel circuits contains a photoelectric conversion element and a charge storage unit, which receives a transfer of charges from the photoelectric conversion element and is configured to output a pixel signal corresponding to a quantity of charges stored in the charge storage unit, and the first mode control includes the control of the transfer of charges between the photoelectric conversion element and the charge storage unit for the computation of obtaining the feature set. DOCUMENTS FROM THE STATE OF TECHNOLOGY PATENT LITERATURE

[0005] Patentdokument 1: Japanische Patentoffenlegungsschrift Nr. JP2022-102604A. NICHTPATENTIERTLITERATUR Nichtpatentliteratur 1: Seiichi Yoneda et al., „Image Sensor Capable of Analog Convolution for Real-time Image Recognition System Using Crystalline Oxide Semiconductor FET“, International Image Sensor Workshop (IISW), 2019, S. 322-325. Nichtpatentliteratur 2: Kohei Yamamoto et al., „Image Classification using Neural Network for Feature Extractable CMOS Image Sensor“, The Institute of Image Information and Television Engineers (ITE) Technical Report, Vol. 46, No. 29, pp. 21-24, veröffentlicht am 15. September 2022 Nichtpatentliteratur 3: Shuhei Okumura et al., „Study on Low Cost Feature Extractable CMOS Color Image Sensor“, The Institute of Electronics, Information and Communication Engineers, Technical Committee on Integrated Circuits and Devices (ICD), Workshop for Students and Young Researchers, veröffentlicht am 22. März 2022 Non-patent literature 4: Hiroyuki Osuga et al., “Analysis of Influence of Feature Extractable in CMOS Image Sensor on Image Recognition,” The Institute of Electronics, Information and Communication Engineers, Technical Committee on Integrated Circuits and Devices (ICD), Workshop for Students and Young Researchers, published March 22, 2022 Non-patent literature 5: Norifumi Egami, "Fundamentals of Image Input Device Technologies (1); Photoelectric Conversion", The journal of the Institute of Image Information and Television Engineers, Vol. 68, No. 1, 2014 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0006] In the image sensor that can extract the feature set according to the conventional example described above, since only the feature set calculation is performed in the chip and the other processing is carried out by the image recognizer, and the amount of output data from the solid-state imaging device is large, there are problems in that the computation costs are relatively high and the power consumption is difficult to reduce.

[0007] The objective of the present invention is to solve the above-mentioned problems and to provide a solid-state imaging device that can reduce power consumption by reducing computation costs compared to the prior art, as well as an image processing system that includes the solid-state imaging device. SOLUTIONS FOR THE PROBLEMS

[0008] According to one aspect of the present disclosure, a solid-state imaging device is provided which includes a plurality of photoelectric conversion elements and a control circuit. The photoelectric conversion elements photoelectrically convert optical signals, each having a predetermined pixel color, and output pixel signals. The control circuit is configured to output a pixel luminance signal with a predetermined set of features by controlling the pixel signals to transmit the pixel signals simultaneously from the multiple photoelectric conversion elements to an output terminal. EFFECTS OF THE INVENTION

[0009] Therefore, according to the solid-state imaging device as described in one aspect of the present disclosure, power consumption can be reduced by reducing computation costs compared to the prior art. BRIEF DESCRIPTION OF THE DRAWINGS [Fig. 1] Fig. Figure 1 is a block diagram showing a configuration example of an image sensor that is a solid-state imaging device according to a first embodiment. [ Fig. 2] Fig. 2 is a block diagram that shows an example of how to configure a pixel unit in Fig. 1 represents. [ Fig. 3] Fig. 3 is a circuit diagram that shows a configuration example of a pixel circuit in Fig. 2 and a peripheral circuit thereof. [ Fig. 4] Fig. Figure 4 is a timing diagram showing an operating example of a pixel circuit of a conventional example and a peripheral circuit thereof. [ Fig. 5] Fig. 5 is a timing diagram that shows an operational example of pixel switching in Fig. 3 and shows a peripheral circuit of it. [ Fig. 6A] Fig. Figure 6A is a diagram showing an example arrangement of color filters in a Bayer array for four photodiodes in Fig. 3 shows. [ Fig. 6B] Fig. Figure 6B is a diagram showing an example of the arrangement of color filters in an RGBR array for four photodiodes in Fig. 3 shows. [ Fig. 6C] Fig. Figure 6C is a diagram showing an example of the arrangement of color filters in an RBBG array for four photodiodes in Fig. 3 shows. [ Fig. 7] Fig. Figure 7 is a diagram showing the properties of light intensity in relation to the depth to which the light enters a general image sensor. [ Fig. 8] Fig. Figure 8 is a circuit diagram showing a configuration example of a pixel circuit according to a second embodiment and a peripheral circuit thereof. [ Fig. 9] Fig. 8 is a timing diagram that shows an operational example of the pixel circuit in Fig. 8 and a peripheral circuit of it. [ Fig. 10A] Fig. Figure 10A is a diagram showing a probability density distribution of a general horizontal edge signal. [ Fig. 10B] Fig. 10B is a diagram showing the input and output characteristics of a comparator in Fig. 8 shows. [ Fig. 11] Fig. Figure 11 is a circuit diagram showing a configuration example of a pixel circuit according to a third embodiment and a peripheral circuit thereof. [ Fig. 12] Fig. Figure 12 is a block diagram showing a configuration example of an image classification system according to a fourth embodiment, which includes the image sensor in Fig. 1 used. [ Fig. 13] Fig. 13 is a block diagram showing a configuration example of an image classification system according to a fifth embodiment using the image sensor in Fig. 1 shows. [ Fig. 14] Fig. Figure 14 is a block diagram showing a configuration example for an image classification system according to a sixth embodiment using the image sensor in Fig. 1 shows. DETAILED DESCRIPTION

[0010] Embodiments and modified embodiments of the present invention are described below with reference to the drawings. It should be noted that identical or similar components are designated by the same reference numerals. FIRST VERSION

[0011] Fig. Figure 1 is a block diagram showing a configuration example of an image sensor 100, which is a solid-state imaging device according to a first embodiment. The image sensor 100 in Fig. 1 is, for example, an image sensor for an image recognition system or similar.

[0012] According to Fig. In Figure 1, the image sensor 100 is, for example, a CMOS image sensor (Complementary Metal Oxide Semiconductor). The image sensor 100 comprises a pixel unit 110, which serves as the imaging unit, and a controller 120. The image sensor 100 further comprises a readout circuit 130, a vertical scanning circuit 140, and a horizontal scanning circuit 150. The readout circuit 130 controls the reading of a pixel signal from the pixel unit 110. The vertical scanning circuit 140 provides the pixel unit 110 with a signal to select a target pixel from which a pixel signal is read, from among the pixels 111 contained in the pixel unit 110. The horizontal scanning circuit 150 provides a column selection signal to the readout circuit 130 for selecting a target pixel column from which a pixel signal is output. In this configuration, the readout circuit 130 outputs feature set image data including a feature set pixel signal from the pixel unit 110.In this case, the feature set image data are, for example, image data that contain a horizontal edge luminance signal, as described later.

[0013] The controller 120 is a control circuit that controls the operation of the pixel unit 110 and the readout circuit 130 via the vertical scanning circuit 140 and the horizontal scanning circuit 150. It should be noted that the controller can be a concept that includes not only the controller 120 described above, but also the vertical scanning circuit 140 and the horizontal scanning circuit 150.

[0014] Fig. Figure 2 is a block diagram showing a configuration example for the pixel unit 110 in Fig. 1 shows. According to Fig. 2. The pixel unit 110 comprises a plurality of pixel circuits 111, 111A, 111B, and 111C arranged in a two-dimensional array. It should be noted that the pixel circuit is also simply referred to as a pixel.

[0015] Fig. Figure 3 is a circuit diagram showing a configuration example of the pixel circuits 111A and 111C in Fig. 2 and a peripheral circuit thereof. In Fig. Figure 3 shows a pixel circuit 111A in the J-th row and a pixel circuit 111C in the J+1-th row. The multiple pixel circuits 111, 111A, and 111C contained in the pixel unit 110 have a common structure. It is noted that in Fig. 3. For the sake of simplicity, the pixel circuit 111C is simplified compared to the pixel circuit 111A, but in reality it has the same structure as the pixel circuit 111A.

[0016] In this case, the structure of the pixel circuit is described using pixel circuit 111A in the J-th row as an example. Pixel circuit 111A comprises four photoelectric conversion elements PD1 to PD4. Photoelectric conversion elements PD1 to PD4 are, for example, photodiodes. These elements store electron charges in an amount corresponding to the amount of light. In this case, a charge generated by photoelectric conversion is called a photoelectric conversion charge, and an electron generated by photoelectric conversion is called a photoelectric conversion electron.

[0017] Pixel circuit 111A contains transfer gates TG1 to TG4, each connected to photoelectric conversion elements PD1 to PD4. Transfer gates TG1 to TG4 transfer the charges stored in photoelectric conversion elements PD1 to PD4. Each of these gates contains, for example, a MOS transistor. When a high-level signal (H) is applied between the gate and the source, the drain and source are brought into a conducting state to turn the MOS transistor on. Conversely, when a low-level signal (L) is applied between the gate and the source, the drain and source are brought into a non-conducting state to turn the MOS transistor off.When each of the transfer gates TG1 to TG4 is switched on, each transfers the charges stored in one of the corresponding photoelectric conversion elements PD1 to PD4 to a charge storage unit, which will be described later. When the transfer gates TG are switched off, the transfer of the charges stored in the photoelectric conversion elements PD1 to PD4 is prevented.

[0018] The pixel circuit 111A comprises the charge storage unit, which receives the transfer of charges from the photoelectric conversion elements PD1 to PD4 via the transfer gates TG. In this embodiment, the charge storage unit comprises at least one floating diffusion layer FD, which is a capacitor connected to the transfer gates TG1 to TG4. In this case, the floating diffusion layer FD is connected to each of the transfer gates TG1 to TG4 to enable the reception of charges transferred from each of the photoelectric conversion elements PD1 to PD4.

[0019] The pixel circuit 111A contains a reset gate RST, which is connected to the charge storage unit. The charge storage unit is connected to the floating diffusion layer FD and a switching gate SG. The reset gate RST is connected between the floating diffusion layer FD and a supply voltage V. DDThe device is connected and discharges the charges accumulated in the charge storage unit. In this case, when the reset gate RST is switched on, it discharges the charges stored in the floating diffusion layer FD or the switching gate SG, which is the charge storage unit, into the power supply. This discharge of charges is called a reset. The reset reduces the voltage potential of the charge storage unit to the supply voltage V. DD . When the reset gate RST is switched off, the reset gate RST prevents the discharge of charges.

[0020] The pixel circuit 111A contains a source-follower transistor SF, which, together with a constant current source CS (to be described later), forms a source-follower circuit 20. The source-follower transistor SF is also referred to as a gain transistor. The source-follower transistor SF has a gate terminal connected to the floating diffusion layer FD (charge storage unit) and a drain terminal connected to the supply voltage V. DD is connected. The source-follower transistor SF generates a voltage vx (pixel signal) at a source terminal, which corresponds to the amount of charge stored in the charge storage unit (and the capacitor of the charge storage unit). The voltage vx (pixel signal) generated at the source terminal of the source-follower transistor SF is a voltage (V FD -V GS ), which are achieved by reducing a voltage V FDat the suspended diffusion layer FD by an amount of a drain-source stress VG S of the source-follower transistor SF is obtained.

[0021] Pixel circuit 111A also includes a selection gate SEL. When the selection gate SEL is switched on, it connects the source terminal of the source follower transistor SF and a signal line 21 for reading the pixel signal. The selection gate SEL is used to select a target pixel circuit (target pixel) from which a voltage in the charge storage unit is to be read, from among the multiple pixel circuits 111A, 111C, and 111. The voltage in the charge storage unit contained in the target pixel circuit is read by the readout circuit 130. When the selection gate SEL is switched off, it disconnects the source follower transistor SF and the signal line 21.

[0022] Pixel circuit 111A also contains the switching gate SG. When the switching gate SG is switched on, it connects the charge storage unit of pixel circuit 111A and the charge storage unit of the other pixel circuit 111C. When the switching gate SG is switched off, it disconnects the charge storage unit of pixel circuit 111A and the charge storage unit of the other pixel circuit 111C. That is, the switching gate SG is a switching element for connecting and disconnecting the charge storage unit provided in pixel circuit 111A, in which the switching gate SG is located, and the charge storage unit provided in the other pixel circuit 111C, adjacent to pixel circuit 111A.When the switching gate SG of pixel circuit 111A is switched on, the charge storage unit (floating diffusion layer FD) of pixel circuit 111A and the charge storage unit (floating diffusion layer FD) of pixel circuit 111C are also connected together.

[0023] The readout circuit 130 comprises the constant current source CS, which configures the source follower circuit 20 together with the source follower transistor SF described above. The constant current source CS is connected to the signal line 21. The readout circuit 130 includes a CDS capacitor C. CDS , which is provided in parallel to the constant current source CS and is located in the middle of the signal line 21. The CDS capacitor C CDSThis is used for the correlated double sampling described later. The readout circuit 130 also includes a clip gate CLP. When the clip gate CLP is switched on, it connects signal line 21 and ground (GND). When the clip gate CLP is switched off, it disconnects signal line 21 and ground (GND). It should be noted that each of the gates described above is driven and controlled by the controller 120.

[0024] In the image sensor 100, configured as described above, a pixel arrangement similar to a Bayer array is generally used, with the four pixels (the photoelectric conversion elements PD1 to PD4) forming a cell. Color filters such as red (R), green (G), and blue (B) are assigned to these four pixels and are referred to below as the R filter, G filter, and B filter, respectively. For example, the R filter is located in photoelectric conversion element PD1, the G filter in photoelectric conversion elements PD2 and PD3, and the B filter in photoelectric conversion element PD4 (see Fig. 6, which will be described later). Light of a specific wavelength, having passed through the color filter, falls on each pixel, and the photoelectric conversion electrons are stored in each of the photoelectric conversion elements PD1 to PD4. The transfer gates TG1 to TG4 are configured to read the electrons stored in the photoelectric conversion elements PD1 to PD4 to the nodes of the floating diffusion layer FD. The source-follower transistor SF, for reading the electrons transferred to the node of the floating diffusion layer FD, the select gate SEL, for selecting a specific pixel, and the reset gate RST, for resetting the floating diffusion layer FD, are shared by PD1 to PD4. The switching gate SG, which connects the pixel cells, is also shared by PD1 to PD4.

[0025] Fig. Figure 4 is a timing diagram showing an operating example of a pixel circuit, a conventional example, and a peripheral circuit thereof. It also shows... Fig. 5 a timing diagram showing an operating example of the pixel circuit 111A and 111C in Fig. 3 and a peripheral circuit of it.

[0026] As in the time diagram of the conventional example in Fig. As shown in Figure 4, to output a feature image, the switching gate SG of the J-th row is first turned on, and then the reset gate RST of the J-th row is turned on to reset the floating diffusion layer FD. Next, the transfer gate TG1 of the J-th row is turned on, and the stored charge of the R pixel is transferred to the floating diffusion layer. In this case, when the clip gate CLP is off, the signal output by the pixel array (photoelectric conversion element PD1) via the source-follower transistor SF is stored in the CDS capacitor C. CDS held.

[0027] The reset gate RST of the J-th row is switched on again to reset the floating diffusion layer FD. Next, the transfer gate TG1 of the J+1 row is switched on, and the stored charge of the R pixel is transferred to the floating diffusion layer FD. This is derived from the signal output by the pixel array (photoelectric conversion element PD1) via the source-follower transistor SF and the value in the CDS capacitor C. CDS With the signal held, the output signal vout becomes a difference signal (= horizontal edge signal) between the R pixels (photoelectric conversion elements PD1) of the J+1 row.

[0028] Similarly, the difference signals of the photoelectric conversion elements PD2, PD3, and PD4 pixels are obtained. The difference signal of each color is converted into a luminance signal using the following equation, e.g., by an image recognition device (40 in Fig. 13 and Fig. 40A in Fig. 14), and then, for example, image recognition is performed using artificial intelligence (AI) of deep machine learning: Y=0.299 R+0.587 G+0.114 B

[0029] On the other hand, as shown in the time diagram in Fig. 5 according to the present embodiment, by simultaneously switching on the transfer gates TG1, TG2, TG3 and TG4, the stored charges in the four photoelectric conversion elements PD1 to PD4 are simply converted into a pixel luminance signal Y by using the following equation in flowing diffusion FD: ] Y=R+2G+B ] where R, G and B each represent a signal level of the individual colors.

[0030] As described above, in the conventional example, the horizontal edge signals of the four pixels of the photoelectric conversion elements PD1 to PD4 must be output a total of four times to output the horizontal edge signals sequentially. In the present embodiment, however, the horizontal edge signal is output only once, and the signal quantity can be reduced to 1 / 4. Since the memory of the four photoelectric conversion elements PD1 to PD4 is summed by the floating diffusion layer FD, the exposure time to achieve a signal quantity equivalent to that in the conventional example can be reduced to 1 / 4, and the sensitivity is improved. In a case where the exposure time is constant because the signal quantity is quadrupled, the SN-to-reset noise ratio of the floating diffusion layer FD can be improved fourfold.

[0031] Fig. Figure 6A is a diagram showing an example arrangement of the color filters in the Bayer array for four photodiodes in Fig. 3 shows. Fig. 6A is an arrangement of the color filters described in the first embodiment described above. In addition to this Bayer array, an RGBR array can be added. Fig. 6B, where the number of R filters is increased, or an RBBG array in Fig. 6C, in which the number of B filters is increased, is used.

[0032] Fig. Figure 7 is a diagram showing the properties of light intensity in relation to the depth to which light enters a general image sensor (see, for example, non-patent document 5).

[0033] Since humans are sensitive to changes in green luminance, increasing the number of G-filters can improve visual resolution. However, for image classification using machine learning, the RGBR or RBBG array can be used. This is based on the relationship between the depth at which light enters the image sensor and the light intensity. In a typical image sensor, the depth of the photoelectric conversion element (PD) is approximately 2 µm, and short-wavelength blue light is sufficiently absorbed, while long-wavelength red light is transmitted at 60% or more. This means that blue light is almost 100% photoelectrically converted, while red light is only about 40% photoelectrically converted.

[0034] In a case where the RGBR array is used, the luminance conversion equation is therefore Y = 2R + G + B, and the insufficient sensitivity of the red light can be compensated for. In a case where the RBBG array is used, the luminance conversion equation is Y = R + G + 2B, and a highly sensitive blue light is emphasized to obtain a high luminance signal.

[0035] As described above, the present embodiment can, for example, generate and output the feature set image data, including the horizontal edge signal (including the feature set of the horizontal edge), compensate for the insufficient sensitivity of the red light by the RGBR array, and improve the classification accuracy (clustering) in accordance with the application and the image classification goal by obtaining the high-luminance signal through the RBBG array. That is, an image sensor capable of extracting the feature set obtained by the luminance conversion equation such as Y = 2R + G + B or Y = R + G + 2B can be configured. SECOND VERSION

[0036] Fig. Figure 8 is a circuit diagram showing a configuration example of a pixel circuit 111 and a peripheral circuit thereof according to a second embodiment, and Fig. Figure 9 is a timing diagram showing an operating example of the pixel circuit 111 and its peripheral circuitry. Fig. 8 shows. The circuit in Fig. 8 shows the following differences to the circuit in Fig. 3 on: (1) An analog-to-digital converter (ADC) 11 and a column switch COL are provided between the output terminal of the pixel circuit 111 of each column and a signal line 22, wherein the column switch COL contains, for example, a MOS transistor controlled by the controller; and (2) a comparator 12 with hysteresis characteristic ( Fig. 10B), which contains a feedback circuit (threshold setting circuit) of a flip-flop 13, is provided between the signal line 22 and an output terminal 25 of a readout circuit 130. That is, the comparator 12 is included in the readout circuit 130.

[0037] The differences are described below.

[0038] As in Fig. As shown in Figure 8, the horizontal edge signal from the pixel circuit 111 of each column is converted analog-to-digital by the analog-to-digital converter 11 provided in each column. The resolution of the analog-to-digital conversion is set to, for example, 10 bits. The digital output signal from the pixel circuit 111 of each column is selectively read out onto the signal line 22 by sequentially switching on the column switch COL of each column, as shown in the timing diagram in Figure 8. Fig. The read 10-bit digital signal is binarized by comparator 12 and converted into a 1-bit digital signal, then output as a pixel signal via output terminal 25. That is, comparator 12 acts as a binarizer. In this case, at the time of binarization of the signal in row I+1, flip-flop 13 sets the already binarized signal from column I and stores it as the threshold for binarization.

[0039] Fig. Figure 10A is a diagram showing a probability density distribution of a general horizontal edge signal, and Fig. Figure 10B is a diagram showing the input and output characteristics of comparator 12. Fig. 8 shows.

[0040] As in Fig. As shown in Figure 10A, the values ​​of the horizontal edge signals are distributed around 0, without any actual edge being present. Near 0, however, noise is present in addition to a tiny edge. Therefore, when the signal is binarized at 0 or higher and / or lower, the noise is amplified and the horizontal edge signal is degraded. For this reason, when the I+1 row is binarized, if the value of the I row signal is 0, the threshold is set high to increase the probability of it being determined as 0 (low level). Conversely, if the value of the I row signal is 1, the threshold is set high to increase the probability of it being determined as 1 (high level).

[0041] As described above, in the second embodiment, by performing binarization based on the horizontal edge signal of the adjacent pixel, the same code is likely to be continued, resulting in a reduction of high-frequency noise. This noise reduction improves the accuracy of image classification. Furthermore, binarization can reduce the data volume to one-tenth. THIRD VERSION

[0042] Fig. Figure 11 is a circuit diagram showing a configuration example of a pixel circuit 111 according to a third embodiment and a peripheral circuit thereof. The circuit in Fig. 11 differs from the circuit in the following points: Fig. 8: (1) Instead of the comparator 12 and the flip-flop 13 provided immediately before the output terminal 25, a comparator 12 and a column switch COL are provided, which are inserted between an analog-to-digital converter 11 and an output terminal 25 of each column. In this case, for example, by using the output signal of the comparator 12 of the I-th column as the threshold of the comparator 12 of the adjacent (I+1)-th column, a function similar to that of the flip-flop 13 is achieved. Fig. 8 is realized. Similarly, the adjacent comparator 12 is configured to input the threshold value from comparator 12 of the column adjacent in the preceding direction.

[0043] As described above, according to the third embodiment, by performing binarization based on the horizontal edge signal of the adjacent pixel, the same code is likely to be continued, resulting in a reduction of high-frequency noise. This noise reduction improves the accuracy of image classification. Furthermore, binarization can reduce the amount of data to one-tenth. Reducing the number of bits in the horizontal transmission signal from 10 bits to 1 bit also reduces the power consumption required for horizontal transmission. FOURTH VERSION

[0044] Fig. Figure 12 is a block diagram showing a configuration example of an image classification system according to a fourth embodiment using the image sensor 100 in Fig. Figure 1 shows the image classification system in Fig. 12 includes the image sensor 100 in Fig. 1, which can extract a feature set, a convolutional neural network (CNN) 30, a feature set extraction circuit 31, a memory 32 for a tagged public data set and a memory for a tagged feature set data set 33.

[0045] Referring to Fig. Memory 32 stores the labeled public record (which is a record in which a correct response label for clustering and color image data are paired), containing general color image data. Feature set extraction circuit 31 computes and extracts a predetermined feature set based on the labeled public record and stores it in memory 33 for the labeled feature set record. That is, memory 33 for the labeled feature set stores a record in which the correct response label for clustering, the color image data, and the feature set are paired.

[0046] The convolutional neural network (CNN) 30 is a well-known CNN and, after training using a variety of data sets stored in memory 33 for the labeled feature set data set at the time of learning, receives classification result data at the time of classification and outputs classification result data by performing clustering processing based on color image data from the image sensor 100 in Fig. 1 executes, from which the feature set such as the horizontal edge can be extracted.

[0047] The present embodiment generates the feature set training dataset to subject the CNN 30, which is the image classification device, to cost-effective training by extracting the feature set from the labeled feature set dataset. By subjecting the CNN 30 to training using this feature set training dataset, the accuracy of image classification using a CMOS image sensor capable of extracting a feature set is improved.

[0048] As described above, according to the present embodiment, in order for the CNN 30, which is an image classifier, to perform deep learning, a large number of images must be collected and the correct response labeling must be carried out on the images by a skilled person, which incurs high costs. However, by generating the feature set training dataset based on the labeled public dataset, including the color image data, a feature set-accurate image classification system can be implemented at low cost and with a significant reduction in personnel costs.

[0049] In the above embodiment, the CNN 30 is provided, but the present invention is not limited thereto, and a deep neural network (DNN) configured to include a known deep learning model can be provided. FIFTH VERSION

[0050] Fig. Figure 13 is a block diagram showing a configuration example of an image recognition system according to a fifth embodiment using the image sensor 100 in Fig. 1 shows. The image recognition system in Fig. 13 includes the image sensor 100 in Fig. 1, which can extract a feature set such as a horizontal edge, and an image recognition device 40 with a recognition detector circuit 41.

[0051] With reference to Fig. 13. The image sensor 100 outputs the feature set image data to the recognition detector circuit 41 of the image recognition device 40. This data is subjected to image filter processing, including, for example, an edge (difference) used in edge extraction. The recognition detector circuit 41 is configured, for example, by using a machine learning model, and outputs a recognition result from the input feature set image data.

[0052] As described above, the pixel filter according to the present embodiment can be fully configured in the image sensor 100, thereby reducing the filter processing required in the image processing circuitry, such as the image recognition device 40. Power consumption can be reduced by decreasing the interface speed between the solid-state imaging element of the image sensor 100 and the image recognition device 40 by compressing the data volume simultaneously with the filter function. SIXTH VERSION

[0053] Fig. Figure 14 is a block diagram showing a configuration example of an image recognition system according to a sixth embodiment using the image sensor 100 in Fig. 1 shows. The image recognition system in Fig. 14 differs from the image recognition system in Fig. 13 in the following points. (1) Instead of the image recognition device 40, an image recognition device 40A with a learning result memory 42 and a recognition detector circuit 41A is provided.

[0054] The differences are described below.

[0055] Referring to Fig. For example, the learning result memory 42 stores a labeled data set after extraction of a feature set, such as a horizontal edge. The detection detector circuit 41A is configured using a machine learning model and, at the time of learning, performs training based on a data set in the learning result memory 42 and, at the time of detection, outputs a detection result of image data based on feature set image data input by the image sensor 100.

[0056] As described above, according to the present embodiment, the recognition detector circuit 41A can contain an optimal learning model for the image sensor 100 by using the learned recognition detector circuit 41A using the labeled data set after feature set extraction and perform recognition and detection with high accuracy. MODIFIED VERSION

[0057] The image classification system according to the fourth embodiment and the image recognition system according to the fifth or sixth embodiment are, for example, each an example of an image processing system. COMMERCIAL APPLICABILITY

[0058] As described in detail above, according to the solid-state imaging device of the present invention, simple luminance signal conversion can be performed simply by changing the control timing of the pixel in the solid-state imaging device, and binarization can be carried out with low noise using a simple circuit. This allows the amount of data to be significantly reduced compared to the prior art by changing the pixel control method and adding a small circuit, and power consumption can be reduced by lowering the computational costs. In particular, by performing the luminance signal conversion and binarization processing in addition to the feature set calculation in the image sensor of the solid-state imaging device, the amount of data transmitted between the image sensor and the image recognizer can be reduced, and the power consumption of the image recognition system can be significantly lowered. REFERENCE MARK LIST 11 AD converters (ADCs) 12 Comparator 13 Flip-flops 20 Source-follower circuit Signal lines 21, 22 and 23 25 Output terminal 30 Convolutional Neural Network (CNN) 31 Feature set extraction circuit 32 Memory slots for a labeled public record 33 Storage for a labeled set of attributes 40, and 40A image recognition device 41 and 41A detection detector circuit 42 learning outcome storage 100 Image sensor (solid-state image capture device) 110 pixel units 111, 111A, 111B, and 111C Pixels32chaltung 120 control 130 Readout circuit 140 Vertical scanning circuit 150 Horizontal scanning circuit C CDS CDS capacitor COL column switch CLP Clip-Gate CS constant current source FD Floating Diffusion Layer PD1 to PD4 Photoelectric conversion element Q1 to Q15 MOS transistor RST Reset Gate SF Source-Follower Transistor SEL Select gate SG Switchable Gate TG, and TG1 to TG4 Transfer Gate output signal (pixel signal) vx pixel signal QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2022-102604A

[0005] Cited non-patent literature

[0000] Seiichi Yoneda et al., „Image Sensor Capable of Analog Convolution for Real-time Image Recognition System Using Crystalline Oxide Semiconductor FET“, International Image Sensor Workshop (IISW), 2019, S. 322-325

[0005] Kohei Yamamoto et al., „Image Classification using Neural Network for Feature Extractable CMOS Image Sensor“, The Institute of Image Information and Television Engineers (ITE) Technical Report, Vol. 46, No. 29, pp. 21-24, veröffentlicht am 15. September 2022

[0005] Shuhei Okumura et al., „Study on Low Cost Feature Extractable CMOS Color Image Sensor“, The Institute of Electronics, Information and Communication Engineers, Technical Committee on Integrated Circuits and Devices (ICD), Workshop for Students and Young Researchers, veröffentlicht am 22. März 2022

[0005] Hiroyuki Osuga et al., „Analysis of Influence of Feature Extractable in CMOS Image Sensor on Image Recognition“, The Institute of Electronics, Information and Communication Engineers, Technical Committee on Integrated Circuits and Devices (ICD), Workshop for Students and Young Researchers, veröffentlicht am 22. März 2022

[0005] Norifumi Egami, „Fundamentals of Image Input Device Technologies (1); Photoelectric Conversion“, The journal of the Institute of Image Information and Television Engineers, Vol. 68, No. 1, 2014

[0005]

Claims

[1] Solid-state imaging device comprising: a plurality of photoelectric conversion elements, wherein the photoelectric conversion elements each photoelectrically convert optical signals with a predetermined pixel color and output pixel signals; and a controller configured to output a pixel luminance signal with a predetermined set of features by controlling the pixel signals to transmit the pixel signals simultaneously from the multiple photoelectric conversion elements to an output terminal. [2] Solid-state imaging device according to claim 1, wherein the feature set is a horizontal edge set. [3] Solid-state imaging device according to claim 1, wherein the multiple photoelectric conversion elements comprise: a photoelectric conversion element that photoelectrically converts an optical signal with a red signal level R and outputs a pixel signal; two photoelectric conversion elements, each of which photoelectrically converts optical signals with a green signal level G and outputs pixel signals; and a photoelectric conversion element that photoelectrically converts an optical signal with a blue signal level B and outputs a pixel signal, and wherein the solid-state imaging device outputs a pixel luminance signal with a luminance conversion equation of Y = 2 G+ R+ B. [4] Solid-state imaging device according to claim 1, wherein the multiple photoelectric conversion elements comprise: two photoelectric conversion elements that photoelectrically convert optical signals, each with a red signal level R, and output pixel signals; a photoelectric conversion element that photoelectrically converts an optical signal with a green signal level G and outputs a pixel signal; and a photoelectric conversion element that photoelectrically converts an optical signal with a blue signal level B and outputs a pixel signal, and wherein the solid-state imaging device outputs a pixel luminance signal with a luminance conversion equation of Y = G + 2 R + B. [5] Solid-state imaging device according to claim 1, wherein the multiple photoelectric conversion elements comprise: a photoelectric conversion element that photoelectrically converts an optical signal with a red signal level R and outputs a pixel signal; a photoelectric conversion element that photoelectrically converts an optical signal with a green signal level G and outputs a pixel signal; and two photoelectric conversion elements, each of which photoelectrically converts optical signals with a blue signal level B and outputs pixel signals, and wherein the solid-state imaging device outputs a pixel luminance signal with a luminance conversion equation of Y = G+ R + 2 B. [6] Solid-state imaging device according to any one of claims 1 to 5, further comprising a binarizer configured to binarize and output a pixel luminance signal from the solid-state imaging device by using a pixel luminance signal of an adjacent column as a threshold. [7] Solid-state imaging device according to claim 6, wherein the binarizer is provided in a subsequent stage of a pixel circuit of each column which contains the plurality of photoelectric conversion elements. [8] Solid-state imaging device according to claim 6, wherein the binarizer is provided in a readout circuit which reads a pixel luminance signal from a pixel circuit of each column which contains the plurality of photoelectric conversion elements. [9] Image processing system with: the solid-state imaging device according to any one of claims 1 to 5; and a neural network configured to perform clustering processing based on a pixel luminance signal from the solid-state imaging device, where the neural network is subjected to learning based on a dataset containing a feature set labeled as the correct answer, which contains a feature set extracted from a public dataset labeled as the correct answer. [10] Image processing system, comprising: the solid-state imaging device according to any one of claims 1 to 5; and an image recognition device configured to perform image recognition processing based on a pixel luminance signal from the solid-state imaging device. [11] Image processing system according to claim 10, wherein the image recognition device is subjected to a learning process based on a public data set labeled with correct answers.

Images