Solid-state imaging device

Abstract

[Problem] To obtain a high-precision image including multi-wavelength information using a compact and inexpensive device. [Solution] This solid-state imaging device is provided with a sensor and a signal processing circuit. A plurality of regions including light receiving elements arranged in a two-dimensional array in a first direction and a second direction intersecting the first direction are arranged in the first direction; and each region receives light of a different wavelength or input light of a different wavelength. The signal processing circuit obtains an image in which information of an object moving in the second direction is corrected based on signals obtained by the light receiving elements belonging to each of the plurality of regions.

Description

Technical Field

[0001] The present disclosure relates to a solid-state imaging device or a solid-state imaging system. Background Art

[0002] Image sensors that capture images with multi-wavelength information, such as multispectral cameras and hyperspectral cameras, are widely used in various fields. For example, in the agricultural field, applications are widely considered in situations where it is difficult for the human eye to observe, such as delivering water, fertilizers, or pesticides to necessary places at the right time, defect inspection of factories or plants, etc. For example, indicators such as the Normalized Difference Water Index (NDWI) and the Normalized Difference Vegetation Index (NDVI) in the agricultural field can be obtained from multi-wavelength information based on images with high reflectivity and images with low reflectivity in water, vegetation, etc.

[0003] However, currently, equipment such as conventional hyperspectral cameras are large and expensive, and research has been conducted, but actual introduction has not made much progress. In order to promote more efficient introduction, a method of taking these images compactly and inexpensively is needed. In addition, if one camera can simultaneously acquire visible light region information and invisible light region information, these images can be acquired from these pieces of information at the same viewing angle, and the camera or computer does not need pixel unit alignment, etc., and calculation becomes easy; however, in the current situation where multiple cameras are required as described above, it is also difficult to achieve this.

[0004] Citation list

[0005] Patent Documents

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2013-238579 Summary of the invention

[0007] Technical problem to be solved by the invention

[0008] Therefore, the present disclosure provides a solid-state imaging device or a solid-state imaging system that compactly and inexpensively acquires an image including high-precision multi-wavelength information.

[0009] Solutions to technical problems

[0010] According to an embodiment, a solid-state imaging device includes a sensor and a signal processing circuit. The sensor includes a light receiving element in a two-dimensional array along a first direction and a second direction intersecting the first direction, and is arranged with a plurality of regions, wherein the regions are arranged along the first direction and receive light beams with different wavelengths respectively or light beams with different wavelengths are incident on the regions respectively. The signal processing circuit acquires an image in which information of an object moving along the second direction is corrected based on a signal acquired in the light receiving element to which each region in the plurality of regions belongs.

[0011] The signal processing circuit may acquire an image in which a positional shift of the object along the second direction is corrected based on the captured images with respect to wavelengths acquired from the plurality of regions.

[0012] A plurality of color filters having transmission characteristics in different wavelength regions may be arranged along the first direction on the light receiving surface side of the sensor to form the plurality of regions in the sensor.

[0013] The plurality of filters may be connectable to and detachable from the sensor.

[0014] In the light receiving surface of the sensor, regions where light passing through one of the plurality of color filters is incident may be set as the plurality of regions.

[0015] The signal processing circuit may acquire an image in which information of the object is corrected based on a time difference in acquiring the same area of ​​the object in the plurality of areas.

[0016] The signal processing circuit may output an image for the corrected wavelength of each of the plurality of regions.

[0017] The signal processing circuit may calculate and output a predetermined index based on the image for the corrected wavelength of each of the plurality of regions.

[0018] The predetermined index may be the Normalized Difference Water Index (NDWI) or the Normalized Difference Vegetation Index (NDVI).

[0019] The plurality of regions may include at least a region receiving wavelengths in a visible light region, a region receiving wavelengths in a near infrared region, and a region receiving wavelengths in a short-wave infrared region.

[0020] The sensor and the signal processing circuit may be included in the same chip.

[0021] According to an embodiment, a solid-state imaging system includes: the solid-state imaging device according to any one of the above; and

[0022] A stage capable of moving an object in the second direction, wherein an image of the object moving with the stage is captured by using the solid-state imaging device, and the captured image of the object for each wavelength is acquired.

[0023] A light source may be further included that emits light of a plurality of wavelengths to a predetermined area along the first direction, wherein reflected light or transmitted light of the light emitted from the light source may be received in each of the plurality of areas arranged in the sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a diagram schematically showing an example of a solid-state imaging system according to the embodiment.

[0025] Figure 2 is a diagram schematically showing an example of a solid-state image pickup element according to the embodiment.

[0026] Figure 3 is a diagram showing an implementation example of an image pickup element according to an embodiment.

[0027] Figure 4 is a diagram showing an implementation example of an image pickup element according to an embodiment.

[0028] Figure 5 is a diagram showing an implementation example of an image pickup element according to an embodiment.

[0029] Figure 6 is a diagram showing an example of a mask according to the embodiment.

[0030] Figure 7 is a diagram showing an example of a color filter configuration according to an embodiment.

[0031] Figure 8 is a diagram showing an example of a color filter configuration according to an embodiment.

[0032] Fig. 9 is a diagram showing a configuration example of a mask and a pixel array according to the embodiment.

[0033] Fig.10 : is a diagram showing an example of ROI setting of a light receiving area according to the embodiment.

[0034] Fig.11 is a diagram showing an example of imaging an object in each light receiving area according to the embodiment.

[0035] Fig.12 is a diagram showing an example of imaging an object in each light receiving area according to the embodiment.

[0036] Fig.13 is a diagram showing an example of imaging an object in each light receiving area according to the embodiment.

[0037] Fig.14 is a diagram showing a configuration example of a light receiving region according to an embodiment.

[0038] Fig.15 is a flowchart showing the processing of the solid-state imaging device according to the embodiment.

[0039] Fig.16is a diagram schematically showing a solid-state imaging system according to the embodiment.

[0040] Fig.17 is a diagram schematically showing a solid-state imaging system according to the embodiment. DETAILED DESCRIPTION

[0041] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The accompanying drawings are for illustration, and the shapes and sizes of the components in the actual device, the size ratios with other components, etc. are not necessarily as shown in the accompanying drawings. In addition, since the accompanying drawings are shown in a simplified manner, it should be understood that components required for implementation other than those shown in the drawings are appropriately provided.

[0042] Figure 1 1 is a block diagram schematically showing a solid-state imaging system according to an embodiment. The solid-state imaging system 1 includes a solid-state imaging device 10, a storage device 20, and a processing device 22. The solid-state imaging system 1 is a system that causes the processing device 22 to perform some processing based on information acquired from the solid-state imaging device 10. As another example, the solid-state imaging system 1 may be a system that stores information acquired from the solid-state imaging device 10 in the storage device 20.

[0043] In addition, the solid-state imaging device 10 can acquire and store information required for processing, or the solid-state imaging device 10 can perform appropriate processing and output the processing result. That is, one or both of the storage device 20 and the processing device 22 are not essential components, and the solid-state imaging system 1 can be formed by the solid-state imaging device 10, and the processing of the solid-state imaging system 1 can be completed in the solid-state imaging device 10.

[0044] Furthermore, as the solid-state imaging system 1, the storage unit 104 in the solid-state imaging device 10 can operate as a first storage unit, and the storage device 20 can be included as a second storage unit. For example, in the case of performing processing such as signal processing, image processing, or some machine learning, the solid-state imaging system 1 can use the processing circuit included in the solid-state imaging device 10 or the processing device 22 outside thereof as a signal processing unit, an image processing unit, a learning unit, or an inference unit. For example, the first storage unit can be included inside or outside the same chip including the imaging element 102 of the solid-state imaging device 10.

[0045] That is, the first storage unit may be included inside the same chip including the imaging element 102, outside the same chip including the imaging element 102, or outside the imaging element 102. Furthermore, in the case where signal processing, image processing, learning, or inference is performed in the solid-state imaging device 10, a processing unit that performs signal processing and image processing, a learning unit that performs learning, or an inference unit that performs inference (including a processing circuit) may be included inside the same chip including the imaging element 102, outside the same chip including the imaging element 102, or outside the imaging element 102.

[0046] In addition, the solid-state imaging device 10 itself may be formed as one chip. That is, the imaging element 102 and the processing unit 106 including the signal processing circuit may be formed as the same chip. In this case, the solid-state imaging system 1 may include an electronic device including an imaging device such as a digital camera or a smartphone terminal including the solid-state imaging device 10, and a storage area and a processing area in the electronic device that are different from the solid-state imaging device 10, the storage device 20, and / or the processing device 22. These areas may be composed of electronic circuits.

[0047] As described above, in the present disclosure, the solid-state imaging device 10 is a concept that can be understood in a broad sense, such as a semiconductor chip (camera module) or an electronic device (a device or gadget having an imaging function) on which the semiconductor chip is mounted.

[0048] In addition, the storage device or storage circuit may be included inside or outside the semiconductor chip of the solid-state imaging device 10. In summary, the storage device or storage circuit of the present disclosure may be included on a chip forming a camera module, in the solid-state imaging device 10 outside the chip, or outside the solid-state imaging device 10 (including a file server connected via a network, etc.).

[0049] The solid-state imaging device 10 includes an optical system 100, an imaging element 102, a storage unit 104, a processing unit 106, and an input / output interface (hereinafter, referred to as an I / F 108). As a non-limiting example, the solid-state imaging device 10 may include a light receiving element capable of receiving visible light, near infrared (NIR) light, and short-wave infrared (SWIR) light, receive light in these wavelength bands to generate image data, and perform processing on the image data.

[0050] exist Figure 1 In the embodiment, the components of the solid-state imaging device 10 are connected to each other via a bus, but the present embodiment is not limited thereto, and the components may be electrically and directly connected to each other as appropriate.

[0051] The optical system 100 is arranged to receive light incident from the outside, such as light reflected from a target, light passing through a target, or light emitted by a target, in a light receiving element of an imaging element 102. The optical system 100 may include, for example, one or more lenses, modified, diffraction gratings, color filters, etc. As some non-limiting examples, the optical system 100 may be able to acquire at least multiple types of bands in visible light, NIR, SWIR, ultraviolet (UV), or obtain light of other bands as needed. Each of the multiple types of bands may be received by the imaging element 102 that is capable of appropriately receiving light of each band.

[0052] For example, the imaging element 102 receives light in the visible, NIR, and SWIR bands and generates image data. That is, the imaging element 102 includes, for example, pixels including photoelectric conversion elements (light receiving elements) that receive light in these bands and output analog signals; and pixel circuits that convert analog signals corresponding to the intensity of light output from the photoelectric conversion elements into image data.

[0053] Note that, for example, the imaging element 102 may receive light of other wavelength bands instead of the above three wavelength bands (visible, NIR, and SWIR), or may include pixels capable of acquiring multispectral or hyperspectral information received for more wavelength regions.

[0054] Figure 2 2 is a block diagram schematically showing an example of the outline of the image pickup element 102. The image pickup element 102 includes a vertical drive circuit 120, a horizontal drive circuit 122, and a pixel array 124 (sensor).

[0055] The vertical driving circuit 120 is, for example, a circuit that controls driving of pixels along the row direction.

[0056] The horizontal driving circuit 122 is a circuit that controls driving of pixels along the column direction, for example.

[0057] The pixel array 124 includes a plurality of pixels 126 in a two-dimensional (e.g., a row direction as a first direction and a column direction as a second direction intersecting the row direction) array. The pixels 126 belonging to the same row are selected by the vertical driving circuit 120, and for the selected row, under the control of the horizontal driving circuit 122, for each column, signals are transmitted from the pixels 126 to the processing unit 106. This transmission can be controlled using a row buffer.

[0058] The output from each pixel 126 is an analog signal, and the imaging element 102 may separately include an ADC (not shown) that converts the analog signal into a digital signal and an amplifier (not shown) that amplifies the signal before transmitting the signal to the processing unit 106, or these types of processing may be performed in the processing unit 106.

[0059] Furthermore, the imaging element 102 may separately include a control circuit for controlling the vertical drive circuit 120 and the horizontal drive circuit 122. The operation of the imaging element 102 and the configuration and operation of other circuits may be similar to those of a conventional solid-state imaging device, and thus detailed description thereof will be omitted.

[0060] The image pickup element 102 transmits a signal based on the intensity of light received by the pixel 126 to the processing unit 106 under the control of the vertical drive circuit 120 and the horizontal drive circuit 122. The processing unit 106 performs processing based on the received digital image signal or the digital image signal obtained by converting the received analog signal. As described above, the pixel 126 can receive light in the visible, NIR, and SWIR bands, for example, via the optical system 100, and the image pickup element 102 can acquire information about these bands as image information.

[0061] Return to Figure 1 , the storage unit 104 stores data required by the solid-state imaging device 10. In the solid-state imaging device 10, in the case of specifically implementing software-based information processing by using hardware resources, the storage unit 104 may store programs, execution files, etc. required for software-based information processing. The storage unit 104 may have a circuit configuration including at least a portion of a semiconductor, and may be a volatile or nonvolatile memory or storage.

[0062] The processing unit 106 performs appropriate signal processing and image processing on the signal acquired from the imaging element 102, and performs subsequent processing on the image data stored in the storage unit 104. Specifically, as described above, the processing unit 106 can perform processing to appropriately convert the signal output from the imaging element 102 into image data. In the present disclosure, the processing unit 106 has a configuration capable of implementing at least some processing using a model.

[0063] For example, a portion of the processing unit 106 may be an analog circuit. The processing unit 106 may include a digital circuit capable of processing digital image data. As a non-limiting example, the processing unit 106 may include a signal processing circuit. The signal processing circuit may specifically include an image processing circuit. These types of processing may be implemented by the processing unit 106 acquiring a program stored in the storage unit 104, etc.

[0064] In the case where the inference processing is performed in the processing unit 106, the model used for the inference may be a learning model optimized by machine learning including deep learning. In this case, the processing unit 106 may obtain a result by forming a model based on the parameters of the model stored in the storage unit 104 and inputting the image data to the formed model. The model may be, for example, a learning model that performs some determination on the target included in the image data. More specifically, the model may be, for example, a model that operates as a determiner that determines normality / abnormality, a model that operates as a classifier that classifies an input image or a target such as an object in the image, or a model that performs some processing on the input image.

[0065] The processing unit 106 can output the processed information to the outside via the I / F 108. The I / F 108 is an interface for inputting and outputting information between the inside and the outside of the solid-state imaging device 10.

[0066] As a non-limiting example, in the case where the solid-state imaging device 10 is a semiconductor chip, a selector for selecting data to be output may be separately included. The data processed by the processing unit 106 may be output to, for example, an external storage device 20 or a processing device 22 via the I / F 108. In addition, the I / F 108 may perform output such as displaying the image data processed by the processing unit 106 on a display, or may receive information from an input device such as a button, a touch panel, or a mouse and output a control signal to the solid-state imaging device 10 to the control unit or the processing unit 106.

[0067] As another non-limiting example, when the solid-state imaging device 10 is a device such as a digital camera, the I / F 108 may be a concept including a user interface. That is, the I / F 108 itself may include an output interface such as a display or an input interface such as a touch panel.

[0068] In any case, the I / F 108 may include a network interface connected to an external network. In this case, the solid-state imaging device 10 may be connected to the external network via the I / F 108 in a wired or wireless manner. The solid-state imaging device 10 may be connected to an external storage device 20 or a processing device 22 via a network in a wired or wireless manner.

[0069] The storage device 20 is a device for storage such as a memory or storage existing outside the solid-state imaging device 10. Information processed in the solid-state imaging device 10 can be stored in the storage device 20 via the I / F 108 temporarily or non-temporarily.

[0070] The processing device 22 is a device that exists outside the solid-state imaging device 10 and is used to perform certain processing. The processing device 22 may be, for example, a system included in determining moisture or vegetation in a farm or the like or controlling processing based on moisture or vegetation, an industrial automation system in a factory or the like, a system that performs inspections anywhere or performs processing based on inspections, or may be a device that performs at least some processing in various situations.

[0071] As another non-limiting example, the processing device 22 may be a device that performs processing outside the solid-state imaging device 10 based on image data acquired by the solid-state imaging device 10 or data obtained by performing any processing on the image data.

[0072] As a non-limiting example, in the case where the solid-state imaging device 10 is a semiconductor chip, the storage device 20 and the processing device 22 may be a storage circuit and a processing circuit included together with the solid-state imaging device 10 inside a device such as a digital camera including the solid-state imaging device 10. That is, a device such as a digital camera may include the solid-state imaging device 10, the storage device 20, and the processing device 22, and the solid-state imaging system 1 may include a device such as a digital camera. In this case, it is not excluded that the solid-state imaging device 10, the storage device 20, and the processing device 22 are also connected to an external device or a network.

[0073] As a non-limiting example, the solid-state imaging device 10 may be a device such as a digital camera, etc. In this case, the solid-state imaging device 10 operates as a digital camera, etc., and transmits and receives data to and from the storage device 20 and the processing device 22 outside the solid-state imaging device 10 in the solid-state imaging system 1 via the I / F 108.

[0074] In the above description, a digital camera or the like is a broad concept including, for example, a digital camera that acquires still images, a digital video camera that acquires moving images, and any of devices having both functions.

[0075] (Semiconductor Chip Configuration Example)

[0076] Figure 3 3 is a diagram showing a non-limiting example of a semiconductor chip on which at least a portion of the solid-state imaging device 10 is mounted. The substrate 30 includes a pixel region 300, a control circuit 302, and a logic circuit 304. Figure 3 As shown, the pixel region 300 , the control circuit 302 , and the logic circuit 304 may be included on the same substrate 30 .

[0077] The pixel region 300 is, for example, a region including the above-described pixel array 124 , etc. A pixel circuit, etc. may be appropriately included in the pixel region 300 , or may be included in other regions (not shown) of the substrate 30 .

[0078] The control circuit 302 includes a control unit. The control circuit 302 may include, for example, the vertical driving circuit 120, the horizontal driving circuit 122, or the above-mentioned control circuit as a part thereof.

[0079] The logic circuit 304 performs processing on, for example, a digital signal output from the pixel circuit and converted by the ADC. As described above, the substrate 30 may include a memory circuit (not shown), and the logic circuit 304 may process data stored in the memory circuit.

[0080] At least part of the operation of the processing unit 106 may be performed by the logic circuit 304. In addition, at least part of the processing unit 106 may not be mounted on this chip but on another signal processing chip included at a different location from the substrate 30, or may be mounted in other processors.

[0081] Figure 4 1 is a diagram showing a non-limiting example of a semiconductor chip on which at least a part of the solid-state imaging device 10 is mounted. As substrates, a first substrate 32 and a second substrate 34 are included. The first substrate 32 and the second substrate 34 have a stacked structure, and can transmit and receive signals to each other by appropriately connecting vias, micro bumps, or copper wires to each other in the respective substrates.

[0082] For example, the first substrate 32 may include the pixel region 300 and its peripheral circuits, and the second substrate 34 may include circuits that perform other signal processing.

[0083] Figure 5 1 is a diagram showing a non-limiting example of a semiconductor chip on which at least a part of the solid-state imaging device 10 is mounted. As substrates, a first substrate 32 and a second substrate 34 are included. The first substrate 32 and the second substrate 34 similarly have a structure of stacking and appropriately electrically connecting as described above.

[0084] The imaging element 102, the storage unit 104, the processing unit 106 and the I / F 108 may be included in Figures 3 to 5 By means of this semiconductor chip, processing from imaging to temporary or non-temporary storage of data, processing using a model, and output of processing results can also be completed in one chip.

[0085] Note that the present embodiment is not limited to these examples, and, for example, the solid-state imaging device 10 can be formed into a structure in which a third substrate for a storage area is included between the first substrate and the second substrate, or on the side of the second substrate opposite to the first substrate, and the three substrates are appropriately connected together and stacked.

[0086] A plurality of stacked substrates may be stacked and formed by any method such as chip on chip (CoC), chip on wafer (CoW), or wafer on wafer (WoW). In the present disclosure, as described above, in the pixel region 300, for example, pixels receiving light in at least visible, NIR, and SWIR bands may be included, and these semiconductor chips may be formed.

[0087] Hereinafter, an example of finally acquiring NDWI and NDVI will be described, but the application scope of the present disclosure is not limited thereto and may be applied as at least a part of an apparatus for acquiring any information from information in multi-wavelength band information.

[0088] (First embodiment)

[0089] Figure 6 is a diagram showing an example of a mask applied to the pixel array 124 according to an embodiment.

[0090] The mask 130 includes a color filter portion 132 and a light shielding portion 140. For example, the mask 130 is arranged to cover the light receiving surface side of the pixel array 124. As an example, the mask 130 may be bonded to the light receiving surface side of the pixel array 124 in any manner that appropriately transmits light controlled for each wavelength of the color filter portion 132.

[0091] In addition, as another example, the mask 130 may be attached to and detached from the image sensor in the form of an additional device. By adopting a form in which the mask 130 can be attached, the process in the present disclosure can be simply performed without processing at the processing stage of the image sensor.

[0092] As another example, for the mask 130, in the pre-processing of the image sensor, the color filter section 132 and the light shielding section 140 may be formed on the light receiving surface of the photoelectric conversion element. In this case, although the transfer of the image sensor to other applications is limited to a certain extent, a dedicated and more accurate process for performing photoelectric conversion can be realized to obtain images from multiple bands.

[0093] That is, for example, the mask 130 including a color filter that transmits light in a desired wavelength band may be arranged instead of the sealing glass by removing the sealing glass included on the light receiving surface side of the pixel array 124, the mask 130 may be arranged by being connected to the upper surface of the sealing glass or in other fixed forms, or the mask 130 may be arranged as a color filter and a light shielding filter on the light receiving surface side of the photoelectric conversion element in a form incorporated in the pixel array 124. As described above, the arrangement of the mask 130 may be achieved in any manner.

[0094] Figure 7 is a plan view showing an example of a filter portion according to the embodiment.

[0095] The color filter section 132 includes, for example, a first color filter 134 that transmits light in the visible band, a second color filter 136 that transmits light in the NIR band, and a third color filter 138 that transmits light in the SWIR band. Note that the combination of color filters is not limited thereto. In addition, color filters having four or more different transmission characteristics may be similarly included.

[0096] For example, in order to obtain information in multiple bands in the visible band, the color filter section 132 may include a color filter that transmits light in multiple different bands in the visible band. The same applies to the NIR band and the SWIR band. That is, the color filter section 132 may include a color filter that transmits light in multiple different NIR bands and / or a color filter that transmits light in multiple different SWIR bands.

[0097] Figure 8 1 is a diagram showing a configuration of another example of a color filter portion according to an embodiment. Figure 8 As shown, the color filter section 132 may further include a light shielding section 140 located between the first color filter 134 and the second color filter 136 and between the second color filter 136 and the third color filter 138 .

[0098] For example, in the case where the mask 130 is arranged to be connectable to the light receiving surface side of the pixel array 124, there may be a gap between the color filter section 132 and the light receiving surface of the pixel array 124. In this case, for example, in the pixel array 124, there may be a photoelectric conversion element that receives both the light passing through the first color filter 134 and the light passing through the second color filter 136.

[0099] like Figure 8 As shown, by arranging the light shielding portion 140 between the color filters, the color filter portion 132 can reduce the number of such pixels 126 into which light is incident via a plurality of color filters.

[0100] Fig. 9 1 is a diagram showing a state where a mask is arranged relative to a pixel array. In this figure, for ease of explanation, there is a large gap between the pixel array 124 and the mask 130, but in reality, the pixel array 123 and the mask 130 may be in close contact with each other, in a state close to close contact, or in a state where they are separated from each other by a sufficiently small distance relative to the area of ​​the pixel array 124 and the area of ​​the mask 130.

[0101] The color filter section 132 of the mask 130 is divided into, for example, three regions. In these regions, the color filter section 132 includes color filters having different wavelength transmission characteristics, respectively.

[0102] For the light beams reaching the pixel array 124 via the color filter, the light beams passing through the area other than the light shielding portion 140 are received in the light receiving area 128, mixed at the boundary portion of the area as shown by the oblique lines rising to the left, and then incident on the pixel 126. Therefore, in these areas, it is difficult to receive light of the required wavelength band. Therefore, in the present disclosure, the region of interest (ROI) is defined as not being affected by these boundary areas.

[0103] Fig.10 1 is a diagram showing an example of ROI setting in the light receiving area of ​​the pixel array. In the pixel array 124, the area other than the light receiving area 128 may be a light shielding area. The boundary of the light receiving area 128 may be set based on the position and size of the color filter section 132, for example, or may be set in the processing unit 106 or the processing device 22 based on the light receiving range after actually performing the test light reception.

[0104] In the light receiving area 128, there is a boundary area 1286 to which the pixel 126 belongs, on which the light beams passing through the plurality of color filters are incident as described above. The area to which the pixel 126 belongs that receives the light beams incident through the respective color filters is set not to include the boundary area 1286.

[0105] The light receiving area 128 includes: for example, a first light receiving area 1280 to which the pixel 126 belongs, which is Figure 7 The first color filter 134 shown receives light; the second light receiving area 1282 to which the pixel 126 belongs, which receives light through the second color filter 136; and the third light receiving area 1284 of the pixel 126, which receives light through the third color filter 138.

[0106] As an example, in the light receiving area 128, each light receiving area is arranged so that the pixels 126 along one or more rows belong to the light receiving area. In these areas, the pixels 126 belonging to each area receive light beams of different wavelengths of each area, or light rays of different wavelengths are incident. The pixels 126 belonging to each area generate and output analog signals based on the received intensity through the photoelectric conversion element.

[0107] The light receiving area corresponding to one color filter may be, for example, an area for eight rows. This is a non-limiting example, and the light receiving area corresponding to each color filter may be set to an area including fewer rows or an area including more rows.

[0108] For example, the solid-state imaging device 10 acquires a plurality of pieces of image information about targets (including biological objects, etc.) traveling in the column direction for each wavelength band in these areas. The acquired information is appropriately subjected to signal processing or image processing in the processing unit 106 and then output.

[0109] Fig.11 , Fig.12 and Fig.13 1 and 2 are diagrams showing an example of imaging an object in each light receiving area according to the embodiment. In these diagrams, the object Obj is shown as an image on the light receiving surface of the sensor as an example.

[0110] Omitted Fig.10 The solid-state imaging device 10 acquires information about the object Obj, for example, for each light receiving area. The object Obj moves along the column direction, such as Fig.11 In each light receiving area, information is acquired at a corresponding acquisition time of information in the pixel 126 (eg, a frame rate of light reception).

[0111] like Fig.11 As shown in FIG. 1 , for an object Obj traveling along the column direction, first, intensity information about the reflected light is obtained in the first light receiving area 1280. After a predetermined time has passed, as shown in FIG. Fig.12 As shown, for the object Obj traveling along the column direction, the intensity information about the reflected light and the like are acquired in the second light receiving area 1282. Fig.12 When the predetermined time further passes, if Fig.13 As shown, for the object Obj traveling along the column direction, the intensity information about the reflected light and the like starts to be acquired in the third light receiving area 1284 .

[0112] The information acquired in each light receiving area can be stored in a storage area such as a buffer. For example, each light receiving area works as a light receiving unit of a line scanner, and the line scanner stores the acquired information as time series data and outputs an image by appropriately combining a plurality of time series data. The combination of a plurality of time series data can be performed by a combination method similar to that of a conventional line scanner.

[0113] In the present embodiment, the solid-state imaging device 10 can acquire three images representing the intensity of light of different wavelength bands acquired from the three light receiving areas by combining a plurality of pieces of time series data acquired in each of the first light receiving area 1280, the second light receiving area 1282, and the third light receiving area 1284. These combinations can be performed by the imaging element 102 storing the time series data in the storage unit 104 and the processing unit 106 combining the plurality of pieces of time series data stored in the storage unit 104.

[0114] The solid-state image pickup device 10 can output each of these combined image data of the respective bands. In addition, the solid-state image pickup device 10 can output the results of signal processing or image processing of these combined image data individually or by using a plurality of pieces of image data.

[0115] That is, the solid-state imaging device 10 can acquire and output an image in which information on an object moving in the column direction is corrected based on signals acquired in the pixels 126 belonging to each of the plurality of light receiving regions.

[0116] The processing unit 106 may also correct the position of the image data acquired in each light receiving area. For example, the processing unit 106 may acquire the moving speed of the object based on the data acquired in the first light receiving area 1280.

[0117] The processing unit 106 can correct the timestamps of multiple time series data acquired in the second light receiving area 1282 and the third light receiving area 1284 based on the speed and span (e.g., frame rate) of information acquisition in the pixel 126 to match the timestamps of the time series data acquired in the first light receiving area 1280.

[0118] As another example, the processing unit 106 may also correct the position of the image data of the object reconstructed from the plurality of time series data acquired in the second light receiving area 1282 and the third light receiving area 1284 along the column direction based on the acquired speed information.

[0119] As another example, the speed along the column direction is known. In this case, the processing unit 106 can correct the positional shift of the image acquired in each area by using the known speed information.

[0120] Fig.14 is a diagram showing an example of the construction of a light receiving area according to an embodiment. It is assumed that the spacing d between the light receiving areas and the speed v in the column direction are known. In this case, for example, the time offset of the same area of ​​the object acquired in each light receiving area is a predetermined time of t=d / v. Therefore, the processing unit 106 can acquire a plurality of images corrected for the positional offset in the column direction by offsetting the time series data acquired from each light receiving area by a predetermined time t or by offsetting the image data acquired from each light receiving area by a number of frames corresponding to the predetermined time t. The number of frames corresponding to the predetermined time t can be set to F, and the target's motion speed v, spacing d, and frame rate can be set so that F is an integer value.

[0121] As described above, for an object captured in a plurality of light receiving areas, the processing unit 106 can acquire an image in which information (e.g., position) of the object is corrected based on the time difference in capturing the same area of ​​the object. Therefore, the solid-state imaging device 10 can acquire information of the object, e.g., position-corrected information, for each wavelength band.

[0122] In the light receiving area 128, the distances (number of pixels) between the first light receiving area 1280, the second light receiving area 1282, and the third light receiving area 1284 are known. This known distance can be calculated based on the configuration of the mask 130. In this case, more appropriate correction of positional deviation in the column direction can be achieved based on the speed information and the distance information.

[0123] In the above description, the solid-state imaging device 10 obtains image data from multiple pieces of information of different wavelength bands obtained for each light receiving area and outputs the image data, but the operation of the present disclosure is not limited to this. The processing unit 106 can also calculate and output predetermined indicators based on multiple pieces of image data obtained from multiple wavelength bands. These indicators can be statistical indicators based on one or more pieces of image data.

[0124] As another example, the processing unit 106 may acquire multiple pieces of image data such as NDWI, NDVI, etc., and output the multiple pieces of image data as predetermined indicators. For example, the first color filter may be a color filter that transmits visible light, the second color filter may be a color filter that transmits light in the NIR band, and the third color filter may be a color filter that transmits light in the SWIR band. In this case, the solid-state imaging device 10 may appropriately acquire image data corresponding to red (RED) from the light received by the first light receiving area 1280, acquire image data corresponding to the NIR band from the light received by the second light receiving area 1282, and acquire image data corresponding to the SWIR band from the light received by the third light receiving area 1284.

[0125] The processing unit 106 may combine images based on the following expression.

[0126] [Math.1]

[0127]

[0128] [Math.2]

[0129]

[0130] [Math.3]

[0131]

[0132] NDWI1 is a normalized water index for the ground surface, NDWI2 is a normalized water index for vegetation, and NDVI is a normalized vegetation index. As described above, the solid-state imaging device 10 can also calculate and output a predetermined index based on a plurality of pieces of image data in a plurality of wavelength bands.

[0133] Fig.15is a flowchart showing a processing example according to the embodiment. In this flowchart, as a non-limiting example, the distance between light receiving areas and the speed of the target in the column direction are known, but the present embodiment can also be applied to the case where the distance and speed are unknown as described above.

[0134] First, the processing unit 106 clears the FIFO used as a buffer (S100). In the case of including the first light receiving area 1280, the second light receiving area 1282, and the third light receiving area 1284 as light receiving areas of three bands, the storage unit 104 includes, for example, FIFO1 storing data of the first light receiving area 1280 and FIFO2 receiving data of the second light receiving area 1282. In the case where the number of bands to be acquired increases, it can be handled by increasing the number of FIFOs. The processing unit 106 initializes the contents of the FIFOs for each area.

[0135] Next, the processing unit 106 clears the frame buffer (S102). The frame buffer is a buffer for storing data output from the pixel 126 belonging to the pixel array 124. The size of the frame buffer can be set according to the size of the image to be acquired. In addition, the frame buffer is included corresponding to each light receiving area. In the above case, the storage unit 104 is included in the frame buffers corresponding to the first light receiving area 1280, the second light receiving area 1282 and the third light receiving area 1284, respectively.

[0136] Next, the solid-state image pickup device 10 captures a frame image (S104). This image pickup is performed using a normal image pickup method in the solid-state image pickup device 10.

[0137] Next, the processing unit 106 performs synchronization processing ( S106 to S110 ).

[0138] The processing unit 106 inputs the image data acquired in S104 to the FIFO (S106). For example, the processing unit 106 enqueues the data captured in the first light receiving area 1280 into FIFO1, and enqueues the data captured in the second light receiving area 1282 into FIFO2.

[0139] Next, the processing unit 106 dequeues the data of the first light receiving area 1280 and the second light receiving area 1282 corresponding to the data currently captured in the third light receiving area 1284 from the FIFO (S108).

[0140] Note that in the initial stage of image capture, this processing is omitted until appropriate past frame data is stored. For example, the processing unit 106 does not dequeue data from FIFO2 until captured data F frames ago is stored in FIFO2, nor does it dequeue data from FIFO1 until captured data 2×F frames ago is stored in FIFO1. After an appropriate time (frame) has passed from the initial state, the processing unit 106 dequeues data from FIFO1 and FIFO2 at appropriate times.

[0141] Next, the processing unit 106 appends the data to the image (S110). For example, the processing unit 106 appends the data corresponding to the width of the light receiving area 2×F frames ago acquired from FIFO1 to the frame buffer storing the image data of the first light receiving area 1280. Similarly, for example, the processing unit 106 appends the data corresponding to the width of the light receiving area F frames ago acquired from FIFO2 to the frame buffer storing the image data of the second light receiving area 1282, and appends the image data currently acquired in the third light receiving area 1284 to the frame buffer storing the image data of the third light receiving area 1284.

[0142] Note that, at this moment, in the event that there is an overlapping region between the data to be appended and the data for which appending is being performed, the processing unit 106 may perform calculations for appropriately overlapping these overlapping regions.

[0143] After adding the image data, the processing unit 106 determines whether the frame buffer for each light receiving area is full (S112). In the case where the frame buffer is full (S112: Yes), the processing unit 106 outputs the image data of the frame to the storage unit 104 or the storage device 20 (S114).

[0144] Note that since there is a 2F frame difference between the third light receiving area 1284 and the first light receiving area 1280, and a F frame difference between the third light receiving area 1284 and the second light receiving area 1282, the output from each frame buffer can be performed at an offset time.

[0145] In addition, after this processing, for each light receiving area, the processing starting from the subsequent S102 can be appropriately staggered. For example, the image data from the third light receiving area 1284 can be output, the frame buffer for the third light receiving area 1284 can be cleared, and after F frames, the output of the image data from the second light receiving area 1282 and the clearing of the frame buffer can be performed, and after F frames, the output of the image data from the first light receiving area 1280 and the clearing of the frame buffer can be performed. That is, the branch of S112 can be determined for each light receiving area.

[0146] The solid-state imaging device 10 may repeat the processing from S102 to S114 as many times as necessary. In addition, for example, in the case of acquiring image data for each band, the processing of S116 and S118 is not necessary and can be performed arbitrarily. In the case where the solid-state imaging device 10 performs the processing until the index output, the processing of S116 and S118 is performed as follows.

[0147] When the index needs to be output, the processing unit 106 performs calculations related to the index. For example, the processing unit 106 performs calculations corresponding to the above expressions (1), (2), and (3) in units of image frames (S116).

[0148] Then, the solid-state imaging device 10 outputs the index obtained in S116 (S118). As described above, multiple pieces of image data obtained for each light receiving area can be output, or some indexes that can be obtained from these images can be calculated inside and outside the chip and output. Of course, the image data and index data for each light receiving area can be output together.

[0149] As described above, according to the solid-state imaging device 10 or the solid-state imaging system 1 based on the present embodiment, by arranging appropriate color filters for an already manufactured image sensor and performing appropriate calculations, a line scan camera capable of covering a multi-spectral or high-spectral band can be realized without the need for control through a large housing and expensive module.

[0150] When the storage area is set as described above, a capacity for storing a plurality of frames of data is required, but, for example, it is not necessary to store image data for other areas in the pixel array 124 than the first light receiving area 1280, the second light receiving area 1282, and the third light receiving area 1284. Therefore, since the solid-state imaging device 10 can use the frame buffer or the like originally used for these other areas as necessary storage areas, even in a conventional configuration, sufficient storage capacity can be ensured by appropriately setting the size of the light receiving area.

[0151] Note that in order to maintain the speed in the column direction at a predetermined speed, a movable stage may be included.

[0152] Fig.16 2 is a diagram schematically showing an example of a solid-state imaging system 1 according to an embodiment. The solid-state imaging system 1 may further include a stage 24. Fig.16 The storage device 20 and the processing device 22 are not shown in the figure, but these components may be included as needed. For example, the processing device 22 may control the speed of the stage, etc.

[0153] The stage 24 is a movable stage that moves an object placed on the upper surface at a predetermined speed in a predetermined direction. The stage 24 may be a stage that moves in the direction of one axis. The solid-state imaging device 10 may be installed so that the direction of one axis of the stage 24 is the column direction.

[0154] The processing unit 106 or the processing device 22 can appropriately define the above-mentioned number of frames F based on the distance between the imaging surface (the light receiving surface of the pixel 126) of the solid-state imaging device 10 and the upper surface of the stage 24, the pitch of the pixels 126, and the moving speed of the upper surface of the stage 24. In addition, as described above, any of these parameters can be changed as needed so that F is an integer.

[0155] (Second embodiment)

[0156] In the above-described first embodiment, the aspect in which a color filter is included in the solid-state imaging device 10 to change the wavelength band of transmitted light has been described; however, it is also possible to implement without including a color filter in the solid-state imaging device 10 .

[0157] Fig.17 1 is a diagram schematically showing an example of a solid-state imaging system according to an embodiment. The solid-state imaging system 1 may include a solid-state imaging device 10, a stage 24, and a light source 26. Fig.16 Similar to the case of , the solid-state imaging system 1 may include at least one of the storage device 20 or the processing device 22 as necessary.

[0158] The light source 26 generates an area where information is desired to be acquired for each wavelength band on the upper surface of the stage 24 (the surface to be imaged by the solid-state imaging device 10). The light source 26 forms, for example, a first irradiation area 260, a second irradiation area 262, and a third irradiation area 264 on the upper surface of the stage 24. In the case of using an example similar to the above example, the light source 26 forms the first irradiation area 260 irradiated with visible light, the second irradiation area 262 irradiated with light in the NIR wavelength band, and the third irradiation area 264 irradiated with light in the SWIR wavelength band by irradiating the stage with light beams corresponding to the respective wavelength bands.

[0159] Note that, although only one light source 26 is shown, the present embodiment is not limited thereto. The irradiation area can be appropriately defined by a plurality of light sources 26 .

[0160] The solid-state imaging device 10 is arranged so that information about these areas can be appropriately acquired in predetermined areas of the pixel array 124. For example, the solid-state imaging device 10 is arranged so that the imaging range in the predetermined area of ​​the pixel array 124 is an imaging range 240 indicated by a dotted line. In addition, such an imaging range 240 can be set by adjusting the optical system 100 and the like of the solid-state imaging device 10 and the arrangement of the solid-state imaging device 10.

[0161] The solid-state imaging device 10 sets the light receiving range of each irradiation area of ​​the pixel array 124 as the light receiving area in the first embodiment. With this setting, the solid-state imaging device 10 can perform the operation similar to Fig.15 The image data for each band is appropriately acquired through the processing in .

[0162] As described above, according to the present embodiment, even in a manner in which no change such as mounting of a color filter is made to the configuration of the solid-state image pickup device 10 , similar operation can be realized.

[0163] The above-described embodiment may have the following modes. (1)

[0165] A solid-state imaging device, comprising:

[0166] A sensor comprising light receiving elements arranged in a two-dimensional array along a first direction and a second direction intersecting the first direction, wherein a plurality of regions are arranged in the sensor, wherein the regions are arranged along the first direction and respectively receive light beams having different wavelengths or light beams having different wavelengths are respectively incident on the regions; and

[0167] A signal processing circuit acquires an image in which information of the object moving in the second direction is corrected based on the signal acquired in the light receiving element to which each of the plurality of areas belongs. (2)

[0169] The solid-state imaging device according to (1), wherein

[0170] The signal processing circuit acquires an image in which a positional shift of the object along the second direction is corrected based on the captured images with respect to the wavelengths acquired from the plurality of regions. (3)

[0172] The solid-state imaging device according to (1) or (2), wherein

[0173] A plurality of color filters having transmission characteristics in different wavelength regions are arranged along the first direction on the light receiving surface side of the sensor to form the plurality of regions in the sensor. (4)

[0175] The solid-state imaging device according to (3), wherein

[0176] The plurality of filters may be connectable to and detachable from the sensor. (5)

[0178] The solid-state imaging device according to (3) or (4), wherein

[0179] In the light receiving surface of the sensor, regions where light passing through one of the plurality of color filters is incident are set as the plurality of regions. (6)

[0181] The solid-state imaging device according to any one of (1) to (5), wherein

[0182] The signal processing circuit acquires an image in which information of the object is corrected based on a time difference in acquiring the same area of ​​the object in the plurality of areas. (7)

[0184] The solid-state imaging device according to any one of (1) to (6), wherein

[0185] The signal processing circuit outputs an image for the corrected wavelength of each of the plurality of regions. (8)

[0187] The solid-state imaging device according to any one of (1) to (7), wherein

[0188] The signal processing circuit calculates and outputs a predetermined index based on the image for the corrected wavelength of each of the plurality of regions. (9)

[0190] The solid-state imaging device according to (8), wherein

[0191] The predetermined index is the Normalized Difference Water Index (NDWI) or the Normalized Difference Vegetation Index (NDVI). (10)

[0193] The solid-state imaging device according to any one of (1) to (9), wherein

[0194] The plurality of regions include at least a region receiving wavelengths in a visible light region, a region receiving wavelengths in a near infrared region, and a region receiving wavelengths in a short-wave infrared region. (11)

[0196] The solid-state imaging device according to any one of (1) to (10), wherein

[0197] The sensor and the signal processing circuit are included in the same chip. (12)

[0199] A solid-state camera system, comprising:

[0200] The solid-state imaging device according to any one of (1) to (11); and

[0201] a stage capable of moving the object along the second direction, wherein

[0202] An image of the object moving with the stage is captured by using the solid-state imaging device, and a captured image of the object for each wavelength is acquired. (13)

[0204] The solid-state imaging system according to (12) not depending on any one of (3) to (5), further comprising:

[0205] a light source that emits light of multiple wavelengths toward a predetermined area along the first direction, wherein

[0206] Reflected light or transmitted light of the light emitted from the light source is received in each of the plurality of regions arranged in the sensor.

[0207] The aspects of the present disclosure are not limited to the above-described embodiments, and include various conceivable variations, and the effects of the present disclosure are not limited to the above-described contents. The components in the various embodiments may be appropriately combined and applied. That is, various additions, variations, and partial deletions may be made without departing from the conceptual ideas and the gist of the present disclosure derived from the contents defined in the claims and their equivalents.

[0208] Reference numerals

[0209] 1Solid-state camera system

[0210] 10 Solid-state imaging device

[0211] 100 Optical System

[0212] 102 Camera Components

[0213] 120 vertical drive circuit

[0214] 122 Horizontal drive circuit

[0215] 124 pixel array

[0216] 126 pixels

[0217] 128 light receiving areas

[0218] 1280 first light receiving area

[0219] 1282 second light receiving area

[0220] 1284 third light receiving area

[0221] 1286 Boundary Area

[0222] 130 Mask

[0223] 132 Color filter unit

[0224] 134 First Color Filter

[0225] 136 Second color filter

[0226] 138 Third Color Filter

[0227] 140 shading part

[0228] 104 storage units

[0229] 106 processing units

[0230] 108I / F

[0231] 20 Storage Device

[0232] 22 Processing device

[0233] 24 Stage

[0234] 240 Camera Range

[0235] 26 Light Source

[0236] 260 First irradiation area

[0237] 262 Second irradiation area

[0238] 264 The third irradiation area

[0239] 30 substrate

[0240] 32. First substrate

[0241] 34 Second substrate

[0242] 300 pixel area

[0243] 302 Control Circuit

[0244] 304 logic circuit.

Claims

1. A solid-state imaging device, comprising: A sensor comprising light receiving elements in a two-dimensional array along a first direction and a second direction intersecting the first direction, wherein a plurality of regions are arranged in the sensor, wherein the regions are arranged along the first direction and respectively receive light having different wavelengths or light having different wavelengths is respectively incident on the regions; and A signal processing circuit acquires an image in which information of the object moving in the second direction is corrected based on the signal acquired in the light receiving element to which each of the plurality of areas belongs.

2. The solid-state imaging device according to claim 1, wherein The signal processing circuit acquires an image in which a positional shift of the object along the second direction is corrected based on the captured images with respect to the respective wavelengths acquired from the plurality of regions.

3. The solid-state imaging device according to claim 1, wherein A plurality of color filters having transmission characteristics in different wavelength regions are arranged along the first direction on the light receiving surface side of the sensor, and the plurality of regions are formed in the sensor.

4. The solid-state imaging device according to claim 3, wherein The plurality of filters may be connectable to and detachable from the sensor.

5. The solid-state imaging device according to claim 3, wherein In the light receiving surface of the sensor, regions where light passing through one of the plurality of color filters is incident are set as the plurality of regions.

6. The solid-state imaging device according to claim 1, wherein The signal processing circuit acquires information of the object based on a time difference of acquiring the same area of ​​the object in the plurality of areas to obtain a corrected image.

7. The solid-state imaging device according to claim 1, wherein The signal processing circuit outputs an image for the corrected wavelength of each of the plurality of regions.

8. The solid-state imaging device according to claim 1, wherein The signal processing circuit calculates and outputs a predetermined index based on the corrected image of the wavelength of each of the plurality of regions.

9. The solid-state imaging device according to claim 8, wherein The predetermined index is the Normalized Difference Water Index (NDWI) or the Normalized Difference Vegetation Index (NDVI).

10. The solid-state imaging device according to claim 1, wherein The plurality of regions include at least a region receiving wavelengths in a visible light region, a region receiving wavelengths in a near infrared region, and a region receiving wavelengths in a short-wave infrared region.

11. The solid-state imaging device according to claim 1, wherein The sensor and the signal processing circuit are included in the same chip.

12. A solid-state camera system, comprising: The solid-state imaging device according to claim 1; and a stage capable of moving the object along the second direction, wherein An image of the object moving with the stage is captured by using the solid-state imaging device, and a captured image of the object for each wavelength is acquired.

13. The solid-state imaging system according to claim 12, further comprising: a light source that emits light of multiple wavelengths toward a predetermined area along the first direction, wherein Reflected light or transmitted light of the light emitted from the light source is received in each of the plurality of regions arranged in the sensor.

Images