Imaging apparatus, imaging method, and electronic apparatus

Abstract

The present technology relates to an imaging device, an imaging method, and an electronic device that can improve image quality. The present invention includes two or more imaging units that can image or sense the same object. At least one first imaging unit of the two or more imaging units includes a first filter that transmits a plurality of wavebands, and at least one second imaging unit that is not the first imaging unit of the two or more imaging units includes a second filter that can change a waveband. For example, the present technology can be applied to a compound eye camera module, an imaging device including the compound eye camera module, and a device including the imaging device and providing virtual reality, and the like.

Description

Technical Field

[0001] This technology relates to imaging apparatus, imaging methods, and electronic devices, and for example, to imaging apparatus, imaging methods, and electronic devices that perform imaging using multiple spectra. Background Technology

[0002] Compound eye camera modules that combine multiple monocular camera modules are known (see, for example, Patent Document 1).

[0003] Reference List

[0004] Patent documents

[0005] Patent document 1: Japanese Patent Application Publication No. 2013-106229. Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] The aim is to improve image quality or increase the detection accuracy of predetermined objects by using a compound eye camera module.

[0008] This technique has been developed in view of this situation, and its purpose is to improve image quality and increase the detection accuracy of predetermined objects.

[0009] Solution to the problem

[0010] An imaging apparatus according to one aspect of the present technology includes two or more imaging units capable of imaging or sensing the same object, wherein at least one of the two or more imaging units, a first imaging unit, includes a first filter configured to transmit multiple bands, and at least one other second imaging unit, in addition to the first imaging unit, includes a second filter capable of changing bands.

[0011] An electronic device according to one aspect of the present technology includes: an imaging device comprising two or more imaging units capable of imaging or sensing the same object, wherein at least one first imaging unit of the two or more imaging units includes a first filter configured to transmit multiple bands, and at least one other second imaging unit of the two or more imaging units, in addition to the first imaging unit, includes a second filter capable of changing the bands; and a processing unit for processing signals from the imaging device.

[0012] In an imaging method according to one aspect of the present technology, the imaging apparatus includes two or more imaging units capable of imaging or sensing the same object, wherein at least one first imaging unit among the two or more imaging units includes a first filter configured to transmit multiple bands, and at least one second imaging unit among the two or more imaging units, other than the first imaging unit, includes a second filter capable of changing the bands, and the imaging method includes: estimating the object using a composite image obtained by synthesizing an image captured by the first imaging unit and an image captured by the second imaging unit.

[0013] An imaging element according to one aspect of the present technology includes two or more imaging units capable of imaging or sensing the same object, wherein at least one of the two or more imaging units, a first imaging unit, includes a first filter configured to transmit multiple bands, and at least one other second imaging unit, in addition to the first imaging unit, includes a second filter capable of changing the bands.

[0014] An electronic device according to one aspect of the present technology includes an imaging device.

[0015] Note that the imaging device and the electronic device can be separate devices or they can be internal blocks that make up a device. Attached Figure Description

[0016] Figure 1 It is a diagram used to illustrate the configuration related to compound eyes.

[0017] Figure 2 This is a diagram used to illustrate the configuration of the imaging device.

[0018] Figure 3 This is a diagram used to illustrate the configuration of the imaging elements.

[0019] Figure 4 This is a diagram used to illustrate the arrangement of filters.

[0020] Figure 5 This is a diagram used to illustrate the Fabry-Perot spectrometer.

[0021] Figure 6 This is a diagram illustrating the setup of the Fabry-Perot spectrometer.

[0022] Figure 7 It is a diagram used to illustrate the wavelength of the color to be interpolated.

[0023] Figure 8 This is a flowchart used to illustrate imaging processing 1.

[0024] Figure 9 This is a flowchart used to illustrate imaging processing 2.

[0025] Figure 10 This is a flowchart used to illustrate imaging processing 3.

[0026] Figure 11 It is a diagram used to illustrate the effect on a moving object.

[0027] Figure 12 It is a diagram used to illustrate the correction.

[0028] Figure 13 This is a flowchart used to illustrate the correction process.

[0029] Figure 14 This is a diagram showing an example of pixel configuration.

[0030] Figure 15 It is a diagram used to illustrate the color arrangement of multispectral pixels.

[0031] Figure 16 This is a diagram used to illustrate the output from multispectral pixels.

[0032] Figure 17 This is a diagram illustrating the configuration of an embodiment of the information processing system.

[0033] Figure 18 This is a diagram showing an example of a display device.

[0034] Figure 19 This is a diagram showing the external configuration of the AR-HMD.

[0035] Figure 20 This is a block diagram illustrating a configuration example of an AR-HMD as an information processing device.

[0036] Figure 21 This is a diagram illustrating an example of AR-HMD usage.

[0037] Figure 22 It is a diagram used to illustrate the recording medium.

[0038] Figure 23 This is a diagram illustrating an example of a schematic configuration of an endoscopic surgical system.

[0039] Figure 24 This is a block diagram illustrating an example of the functional configuration of the camera and CCU.

[0040] Figure 25 This is a block diagram illustrating an example of a schematic configuration of a vehicle control system.

[0041] Figure 26 This is an explanatory diagram showing an example of the installation location of the vehicle external information detection unit and the imaging unit. Detailed Implementation

[0042] In the following text, embodiments for implementing this technology will be described (hereinafter referred to as embodiments).

[0043] <Configuration of the compound eye camera module>

[0044] This technology can be applied to imaging devices that include compound eye camera modules. Figure 1 This is a diagram illustrating an example of the external configuration of a compound eye camera module to which this technology is applied.

[0045] exist Figure 1 middle, Figure 1 A is a stereoscopic view of the compound eye camera module 10, and Figure 1 B is the front view of the compound eye camera module 10.

[0046] The compound eye camera module 10 is a compound eye type camera module, and is configured by fixing the monocular camera module 21-1 and monocular camera module 21-2 by a connecting member 22 having a rectangular plate shape.

[0047] In the single-lens camera module 21-1, solid-state imaging elements such as complementary metal-oxide-semiconductor (CMOS) image sensors and lens units are installed.

[0048] In the single-lens camera module 21-1, the solid-state imaging element includes a pixel unit with multiple pixels arranged in two dimensions, and a peripheral circuit unit that drives the pixels and performs analog-to-digital (A / D) conversion, etc. In this solid-state imaging element, light (image light) incident from the lens in the lens unit forms an image on the light-receiving surface of the pixel unit, and the light of the formed image is photoelectrically converted to generate pixel signals.

[0049] Similar to the single-lens camera module 21-1, the single-lens camera module 21-2 is configured by mounting a CMOS image sensor, lens unit, etc. For example, in the compound-lens camera module 10, the single-lens camera module 21-1 can be used as the main camera, while the single-lens camera module 21-2 can be used as the sub-camera.

[0050] The connecting member 22 has a rectangular plate shape with a profile larger than the dimensions in the planar direction when the lens units of the monocular camera module 21-1 and the monocular camera module 21-2 are arranged. Furthermore, the rectangular insertion hole portions for inserting the lens units of the monocular camera module 21-1 and the rectangular insertion hole portions for inserting the lens units of the monocular camera module 21-2 are symmetrically arranged to penetrate the connecting member 22.

[0051] In the compound eye camera module 10, the lens units of the monocular camera module 21-1 and the monocular camera module 21-2 are respectively inserted and fixed into two rectangular insertion hole portions formed as through-connecting members 22. Therefore, the compound eye camera module 10 is configured as a compound eye type camera module including monocular camera module 21-1 and monocular camera module 21-2.

[0052] The compound eye camera module 10 is configured as described above.

[0053] Note that SLR camera module 21-1 and SLR camera module 21-2 are examples of multiple SLR camera modules connected by connecting member 22, and are referred to as SLR camera module 21 in the following text unless it is particularly necessary to distinguish these SLR camera modules.

[0054] Furthermore, a single-lens camera module is a camera module on which a solid-state imaging element (image sensor) is mounted. However, a compound-lens camera module is a camera module that mounts multiple solid-state imaging elements (image sensors) by connecting multiple single-lens camera modules. However, a module can be referred to by other names, such as a package.

[0055] Furthermore, here, as shown Figure 1 The structure shown is that the single-lens camera module 21-1 and the single-lens camera module 21-2 are connected by the connecting member 22. However, the simple arrangement of the single-lens camera module 21-1 and the single-lens camera module 21-2 without using the connecting member 22 is also within the scope of this technology.

[0056] <Imaging Device Configuration>

[0057] Figure 2 It shows including Figure 1 A diagram illustrating the configuration of an embodiment of the imaging device 100 of the compound eye camera module 10 shown.

[0058] Imaging apparatus 100 includes imaging unit 110 and imaging unit 120. While details will be described later, imaging unit 110 and imaging unit 120 receive and process light of different wavelengths. For example, the description will continue under the assumption that, in addition to the band difference such as red and blue wavelengths, the different wavelengths also include the difference between narrowband and broadband. Here, the description will continue under the assumption that imaging unit 110 is an imaging unit that receives and processes light with broadband wavelengths, and imaging unit 120 is an imaging unit that receives and processes light with narrowband wavelengths.

[0059] Note that the description of the imaging device 100 here includes imaging unit 110 and imaging unit 120 as an example, but the imaging device may include two or more imaging units.

[0060] Imaging unit 110, for example, corresponds to Figure 1 The compound eye camera module 10 in the compound eye camera module 10 is a single eye camera module 21-1, and the imaging unit 120 corresponds, for example, to the single eye camera module 21-1. Figure 1 The compound eye camera module 10 and the monocular camera module 21-2.

[0061] The imaging apparatus 100 includes: a processing unit 131 for acquiring and processing signals from imaging unit 110 and signals from imaging unit 120; and an image output unit 132 for outputting the image processed by the processing unit 131. Furthermore, the imaging apparatus 100 includes a memory 134 and a communication unit 135. The communication unit 135 is configured to exchange data with a predetermined server 151 via a network (not shown).

[0062] Server 151 is, for example, a server in the cloud. Alternatively, instead of a configuration for reading data from server 151, a configuration for reading data recorded on a predetermined recording medium (including a drive configuration) can be used.

[0063] The imaging unit 110 includes a lens 111, a broadband photoelectric conversion unit 112, an A / D conversion unit 113, a clamping unit 114, a color-specific output unit 115, a defect correction unit 116, and a linear matrix unit 117.

[0064] Lens 111 focuses the incident light onto broadband photoelectric conversion unit 112. Broadband photoelectric conversion unit 112 includes, for example, a color filter that receives and processes red (R), green (G), and blue (B) light, and outputs each signal to A / D conversion unit 113. A / D conversion unit 113 converts the analog signal of the image of the object undergoing photoelectric conversion by broadband photoelectric conversion unit 112 into a digital value.

[0065] The clamping unit 114 subtracts the black level of the digital data (image data) of the object's image provided by the A / D conversion unit 113. The color-specific output unit 115 supplements the color signal of the image data provided by the clamping unit 114 as needed. The defect correction unit 116 evaluates the defects of the pixels and corrects the pixels evaluated as defective as needed based on the evaluation results.

[0066] The linear matrix unit 117 applies a linear matrix to the image data provided from the defect correction unit 116 as needed to improve color reproduction, etc.

[0067] After processing by the linear matrix unit 117, a block can be set to perform gamma correction processing to make the brightness expression natural or to generate brightness and chromaticity signals on the image data before the image data is supplied to the processing unit 131 or after the processing unit 131 processes it.

[0068] The imaging unit 110 has functions similar to those of a regular camera called a digital camera, and has the function of imaging an object and generating a color image.

[0069] Imaging unit 120 includes a lens 121 and a narrowband photoelectric conversion unit 122. Lens 121 focuses incident light onto narrowband photoelectric conversion unit 122. Narrowband photoelectric conversion unit 122 includes, for example, a Fabry-Perot spectrometer, which receives and processes light in a predetermined wavelength band and outputs the light to A / D conversion unit 123. A / D conversion unit 123 converts the analog signal of the image of the object photoelectrically converted by narrowband photoelectric conversion unit 122 into a digital value and outputs the digital value to processing unit 131.

[0070] <Configuration of photoelectric conversion unit>

[0071] Figure 3 This is a diagram showing the configuration of the broadband photoelectric conversion unit 112 and the A / D conversion unit 113. (See diagram for example.) Figure 3 As shown, the broadband photoelectric conversion unit 112 includes: a pixel region 203, in which pixels 202 including a plurality of photoelectric conversion elements are regularly and two-dimensionally arranged on a semiconductor substrate (e.g., a silicon substrate); and a peripheral circuit region.

[0072] Pixel 202 includes a photoelectric conversion element (e.g., a photodiode (PD)) and a plurality of pixel transistors (so-called MOS transistors). The plurality of pixel transistors can be configured, for example, by three transistors of a transfer transistor, a reset transistor, and an amplification transistor, and can be configured by four transistors by further adding a selection transistor.

[0073] Furthermore, pixel 202 may also have a pixel-sharing structure. This pixel-sharing structure includes multiple photodiodes, multiple transmission transistors, a shared floating diffuser, and another shared pixel transistor for each. A photodiode is a photoelectric conversion element.

[0074] The peripheral circuit area includes a vertical drive circuit 204, a column signal processing circuit 205, a horizontal drive circuit 206, an output circuit 207, and a control circuit 208.

[0075] The control circuit 208 receives data such as an input clock and an indication of the operating mode, and outputs data such as internal information of the broadband photoelectric conversion unit 112. Specifically, the control circuit 208 generates clock signals or control signals based on the vertical synchronization signal, the horizontal synchronization signal, and the master clock, which serve as references for the operation of the vertical drive circuit 204, the column signal processing circuit 205, and the horizontal drive circuit 206. The control circuit 208 then inputs these signals to the vertical drive circuit 204, the column signal processing circuit 205, and the horizontal drive circuit 206.

[0076] The vertical drive circuit 204, configured, for example, by a shift register, selects pixel drive wiring, provides pulses for driving pixels 202 to the selected pixel drive wiring, and drives pixels 202 row by row. Specifically, the vertical drive circuit 204 sequentially and selectively scans each pixel 202 in the pixel region 203 in the vertical direction row by row, and provides a pixel signal based on the signal charge generated in the photoelectric conversion element of each pixel 202 according to the amount of light received to the column signal processing circuit 205 via the vertical signal line 209.

[0077] The signal processing circuit 205 corresponds to Figure 2 The A / D conversion unit 113 is shown. For example, the column signal processing circuit 205 is arranged for each column of pixels 202, and performs signal processing such as noise removal on the signal output from a row of pixels 202 for each column of pixels. Specifically, the column signal processing circuit 205 performs signal processing such as correlated double sampling (CDS) for removing fixed pattern noise specific to pixels 202, signal amplification, and analog-to-digital (A / D) conversion. In the output stage of the column signal processing circuit 205, a horizontal selection switch (not shown) is connected and disposed between the horizontal signal lines 210.

[0078] The horizontal drive circuit 206 is configured, for example, by a shift register and sequentially selects each of the column signal processing circuits 205 by sequentially outputting horizontal scan pulses, and causes each of the column signal processing circuits 205 to output a pixel signal to the horizontal signal line 210.

[0079] Output circuit 207 performs signal processing on each signal sequentially provided from column signal processing circuit 205 via horizontal signal line 210, and outputs the signal. For example, output circuit 207 may perform only buffering, or it may perform black level adjustment, column change correction, various types of digital signal processing (by...). Figure 2 (The processing performed by the clamping unit 114 in the middle) etc.

[0080] Input / output terminal 212 is configured to exchange signals externally.

[0081] The basic configuration of the narrowband photoelectric conversion unit 122 and the A / D conversion unit 123 is similar to that of the broadband photoelectric conversion unit 112 and the A / D conversion unit 113, and can be... Figure 3 The configuration shown is omitted here. Therefore, its description is omitted. In the following description, reference numerals for portions related to the narrowband photoelectric conversion unit 122 and the A / D conversion unit 123 are prefixed with dashes for descriptive purposes. For example, the pixel region 203 of the narrowband photoelectric conversion unit 122 is described as pixel region 203' by a dash.

[0082] <About Filters>

[0083] The broadband photoelectric conversion unit 112 and the narrowband photoelectric conversion unit 122 include different filters. Predetermined filters are disposed on pixel regions 203 (203'), and each pixel 202 is configured to receive light of a predetermined wavelength transmitted through each filter.

[0084] The filter of the broadband photoelectric conversion unit 112 can be, for example, as shown in... Figure 4 The RGB color filter shown in A. Figure 4 Figure A shows an array of RGB color filters known as the Bayer array. Four 2x2 pixels are assigned to R, G, G, and B pixels. Furthermore, this technique can also be applied to the following situations: four 2x2 pixels are assigned to R pixels, four pixels adjacent to the right of the R pixel are assigned to G pixels, four pixels below the R pixel are assigned to G pixels, and four pixels adjacent to the right of the G pixel are assigned to B pixels.

[0085] Furthermore, one of the two G pixels can be arranged as a transparent (white) pixel. Additionally, while RGB is used as an example here, other filters can also be used, such as filters for each color including cyan (Cy), magenta (Mg), yellow (Ye), and white (W).

[0086] In addition, filters other than color filters can be used, such as filters known as plasma filters. While any filter can be used, the filter of the broadband photoelectric conversion unit 112 is a filter that receives (transmits) light of a broadband wavelength. When the broadband photoelectric conversion unit 112 is configured with an RGB color filter, the broadband photoelectric conversion unit 112 serves as a photoelectric conversion unit that transmits and processes light from the blue (B) to red (R) band (e.g., the 400 nm to 700 nm band).

[0087] Narrowband photoelectric conversion unit 122 processes wavelengths within a narrower range than the wavelengths processed by broadband photoelectric conversion unit 112. Narrowband photoelectric conversion unit 122 includes a filter that transmits light within a predetermined range of wavelengths. As a filter, for example, a filter such as a Fabry-Perot spectrometer (interferometer) can be used. In the case of using a Fabry-Perot spectrometer, such as... Figure 4 As shown in B, the Fabry-Perot spectrometer 251 is arranged on the pixel region 203' of the narrowband photoelectric conversion unit 122 to cover all pixels.

[0088] Reference Figure 5 Describe the Fabry-Perot spectrometer 251. The Fabry-Perot spectrometer 251 can be used as a filter for transmitting light of a specific wavelength. For example... Figure 4As shown, the Fabry-Perot spectrometer 251 is an optical device comprising two semi-transparent mirrors 252 and 253, which are arranged opposite to and parallel to each other. The semi-transparent mirrors 252 and 253 are fabricated to have reflective surfaces with high reflectivity and low transmittance.

[0089] Light incident from one side (the upper side in the figure) of the Fabry-Perot spectrometer 251 is reflected between the two reflecting surfaces and oscillates back and forth multiple times to interfere with each other. Because the light oscillates back and forth multiple times with a constant optical path difference, the light transmitted through the semi-transparent mirror 253 becomes interference light with a considerable length. Therefore, when used as a spectrometer, very high resolution can be obtained.

[0090] That is, the wavelength that is desired to be analyzed by the Fabry-Perot spectrometer 251 can be selected from the incident light, and the selected light can be received by the pixel.

[0091] As described above, the Fabry-Perot spectrometer 251 is configured such that light of a predetermined wavelength is transmitted by causing light to reflect back and forth between and interfere with each other using semi-transparent mirrors 252 and 253. The wavelength of the light to be transmitted can be set by the distance between the semi-transparent mirrors 252 and 253. In other words, by changing the distance between the semi-transparent mirrors 252 and 253, light of a desired wavelength can be transmitted.

[0092] For example, a voltage can be applied to the semi-transparent mirrors 252 and 253, and the distance between them can be adjusted by electrostatic attraction. Such a Fabry-Perot spectrometer, capable of changing the wavelength of the light to be transmitted, is called a microelectromechanical system (MEMS) Fabry-Perot spectrometer, etc. Here, as an example, the case of a MEMS Fabry-Perot spectrometer capable of variably setting the wavelength of the light to be transmitted to a desired wavelength will be described as Fabry-Perot spectrometer 251. Note that, again in the following description, the MEMS Fabry-Perot spectrometer will continue to be simply referred to as Fabry-Perot spectrometer 251.

[0093] When the Fabry-Perot spectrometer 251 is used as a filter for the narrowband photoelectric conversion unit 122, such as Figure 4 As shown in B, a Fabry-Perot spectrometer 251 can be arranged to cover the entire surface of pixel region 203'.

[0094] In addition, such as Figure 6 As shown in Figure A, a configuration can be adopted in which pixel region 203' is divided into four 2×2 regions, and Fabry-Perot spectrometers 251-1 to 251-4 are arranged in the corresponding regions. Furthermore, as... Figure 6As shown in B, a configuration can be adopted in which pixel region 203' is divided into four regions in the horizontal direction and Fabry-Perot spectrometers 251-1 to 251-4 are arranged in the corresponding regions.

[0095] Although not shown, the pixel region 203' can be configured such that it is divided into four regions in the vertical direction and Fabry-Perot spectrometers 251-1 to 251-4 are arranged in the corresponding regions. The number of divisions can be arbitrary, and a configuration can be adopted in which multiple Fabry-Perot spectrometers 251 are arranged in the pixel region 203' according to the number of divisions. Furthermore, a configuration can be adopted where a Fabry-Perot spectrometer 251 is set for each pixel.

[0096] For reference Figure 4 As described, in the Fabry-Perot spectrometer 251, the wavelength of the light to be transmitted is set according to the distance between two transmissive mirrors 252 and 253; however, the distance between these two transmissive mirrors is not always uniform. For example, in the central portion of the Fabry-Perot spectrometer 251, transmissive mirror 252 may be relaxed, and the distance between transmissive mirrors 252 and 253 may be shorter than the distance between transmissive mirrors 252 and 253 at their ends.

[0097] To prevent this, arrangements can be made as shown in the reference. Figure 6 Multiple Fabry-Perot spectrometers 251 are described to reduce the area of ​​each semi-transparent mirror and prevent relaxation of the central portion. Furthermore, although details will be described later, by applying this technique, even as... Figure 4 As shown in B, a Fabry-Perot spectrometer 251 is set in pixel region 203', so that correction and processing can be performed appropriately even if the distance between the two semi-transparent mirrors 252 and 253 of the Fabry-Perot spectrometer 251 is not uniform.

[0098] In the following description, it will be described as follows: Figure 4 The case of setting a Fabry-Perot spectrometer 251 in pixel region 203' in B is taken as an example.

[0099] In this manner, the imaging unit 110 includes a broadband photoelectric conversion unit 112, which receives and processes light of a broadband wavelength from a color filter, etc., and generates a color image. In the following description, the imaging unit 110 will be appropriately described as a broadband imaging unit 110.

[0100] Imaging unit 120 includes a narrowband photoelectric conversion unit 122, which receives and processes narrowband wavelength light from the Fabry-Perot spectrometer 251 and the like, and generates an image. In the following description, imaging unit 120 will be appropriately described as narrowband imaging unit 120.

[0101] Since the imaging device 100 includes a broadband imaging unit 110 and a narrowband imaging unit 120, it can receive and process data such as... Figure 7 The light in the indicated wavelength band. When referenced... Figure 7 During the A to C phases, the broadband imaging unit 110 receives and processes light in the blue (B) band B, the green (G) band G, and the red (R) band R.

[0102] Since the narrowband imaging unit 120 can be set to the desired receiving and processing band, the band can be set according to the object or purpose to be imaged. Figure 7 In A, the wavelength of light to be transmitted through the Fabry-Perot spectrometer 251 is set to be a wavelength shorter than blue, such as the violet (ultraviolet) wavelength F.

[0103] Furthermore, the half-width at half-maximum (WHM) of band F of the Fabry-Perot spectrometer 251 is smaller than the WHM of the band of the color filter, for example, smaller than the WHM of band B. That is, also in this respect, the Fabry-Perot spectrometer 251 can be considered a filter that transmits narrow-band wavelengths and is a filter that can selectively transmit light of desired wavelengths.

[0104] like Figure 7 As shown in A, when imaging is performed in the ultraviolet band F by the narrowband imaging unit 120, for example, imaging can be performed in a band suitable for human skin analysis. For example, pigmentation on a person's face can be imaged by the narrowband imaging unit 120, and the image can be provided to the user, wherein the pigmentation is superimposed on a color image of the face imaged by the broadband imaging unit 110.

[0105] When imaging skin conditions such as pigmentation, imaging in the ultraviolet band can be performed with higher sensitivity compared to imaging in the visible light region (bands R, G, and B). Therefore, by imaging skin conditions using the narrowband imaging unit 120, the sensitivity of such imaging of skin conditions can be improved.

[0106] Furthermore, this technology can be applied not only to the analysis of skin conditions such as pigmentation, but also to imaging, for example, to detect unapplied areas of cosmetics such as sunscreen. For instance, the imaging device 100 using this technology can be installed in a smartphone, etc., and can provide applications for imaging a user's face, detecting unapplied areas of cosmetics, and notifying the user.

[0107] Furthermore, when ultraviolet light is captured by the narrowband imaging unit 120, the amount of ultraviolet light from the external light can be measured by analyzing the captured image, and an alarm can be issued based on the measurement value.

[0108] exist Figure 7 In the example shown in B, the band of light to be transmitted through the Fabry-Perot spectrometer 251 is band F between the green band G and the red band R, for example, the yellow band F.

[0109] like Figure 7 As shown in B, color reproducibility can be improved when the yellow band F is imaged by the narrowband imaging unit 120. By synthesizing the image captured by the broadband imaging unit 110 including RGB color filters and the image captured by the narrowband imaging unit 120 including the Fabry-Perot spectrometer 251, color reproduction can be improved and an image with improved image quality can be obtained.

[0110] The narrowband imaging unit 120 can image colors with reduced color reproducibility in the image captured by the wideband imaging unit 110. In other words, colors can be supplemented by the narrowband imaging unit 120. Figure 7 In B, taking the case where the narrowband imaging unit 120 supplements the color of band F between the green band G and the red band R as an example, band F can be changed and other colors can be supplemented.

[0111] For example, by capturing and supplementing colors with reduced color reproducibility in images captured by the broadband imaging unit 110 through the narrowband imaging unit 120, the reproducibility of human skin color can be further improved, and people can be captured more realistically.

[0112] Furthermore, this technology can also be applied to situations where skin color recognition capabilities are used for person identification and detection of changes in health status. Additionally, this technology can be applied to situations where the imaging device 100 is installed on medical smart glasses to assist medical examinations and improve diagnostic systems. For example, an image obtained by superimposing a color image with an image captured in band F, suitable for imaging shadows around the eyes, jaundice, etc., can be presented to a doctor.

[0113] like Figure 7 As shown in Figure C, when the narrowband imaging unit 120 is adapted to image the infrared light band F, objects or foreign objects can be detected, for example. For example, different objects of the same color that are difficult to distinguish in the visible light region can be identified by analyzing the image captured by the narrowband imaging unit 120 in the infrared light band F.

[0114] Furthermore, this technology can also be applied to the quantity of contents in paper or plastic packaging, the damage status of food, and the health status of plants, among other things. Additionally, the imaging device 100 can be used in smart glasses or smart homes, and for applications such as detecting high-temperature objects and issuing an alarm.

[0115] <First Processing of Imaging Device 100>

[0116] Reference Figure 8 The flowchart describes the first processing of the imaging device 100.

[0117] In step S11, the narrowband imaging unit 120 performs imaging with the wavelength of the light to be transmitted by the Fabry-Perot spectrometer 251 set to a predetermined band (appropriately described as a first wavelength). Simultaneously with the imaging by the narrowband imaging unit 120, the broadband imaging unit 110 also performs imaging (step S12).

[0118] In step S13, processing unit 131 performs analysis. Processing unit 131 performs analysis suitable for the imaging purpose. For example, in reference... Figure 7 In the case of analyzing the skin condition described in A, an analysis suitable for the purpose of analyzing the skin condition is performed. This analysis suitable for the purpose of analyzing the skin condition includes an analysis of whether imaging has already been performed in a suitable wavelength band for imaging the pigmentation.

[0119] Furthermore, when the narrowband imaging unit 120 performs imaging for supplementary colors in order to improve the reproducibility of skin colors, it includes an analysis of whether the supplementary colors are appropriate, which is an analysis of whether to set the band of the supplementary colors.

[0120] In performing such analysis, in other words, in analyzing (determining) whether imaging is performed in a state set to the optimal band, a configuration can be adopted in which the desired image is provided to the processing unit 131.

[0121] For example, a configuration can be adopted that uses only the image from the narrowband imaging unit 120 to perform the analysis. In this configuration, a processing flow that omits the imaging processing in the broadband imaging unit 110 in step S12 can be obtained.

[0122] Furthermore, for example, a configuration can be adopted that uses only the image from the broadband imaging unit 110 to perform the analysis. In this configuration, a processing flow that omits the imaging processing in the narrowband imaging unit 120 in step S11 can be obtained.

[0123] Furthermore, for example, a configuration can be adopted in which analysis is performed using images from narrowband imaging unit 120 and images from wideband imaging unit 110. In this configuration, processing of synthesizing images from narrowband imaging unit 120 and wideband imaging unit 110 is performed by processing unit 131, and analysis is performed using the synthesized image.

[0124] Whether to perform analysis by using a composite image obtained by combining an image from narrowband imaging unit 120 and an image from wideband imaging unit 110, or by using only an image from narrowband imaging unit 120 or only an image from wideband imaging unit 110, only needs to be set to use the image that allows the most accurate analysis, depending on the content of the analysis.

[0125] Furthermore, the analysis can be switched between using synthetic images, using only images from the narrowband imaging unit 120, or using only images from the wideband imaging unit 110, depending on the analysis content.

[0126] Using the analysis results from step S13, proceed to step S14 to determine whether the analysis results are satisfactory.

[0127] For example, in analyzing skin conditions, it is determined whether skin conditions such as pigmentation have been successfully imaged. Furthermore, for example, in analyzing skin color, it is determined whether imaging was successfully performed with high reproducibility of the skin color. In other words, this determination is whether imaging was performed within the appropriate wavelength band.

[0128] If the analysis results are determined to be unsatisfactory in step S14, in other words, if it is determined that imaging was not performed in the appropriate band, the process proceeds to step S15.

[0129] In step S15, information about the optimal wavelength is obtained. If the analysis result is determined to be unsatisfactory, the processing unit 131 retrieves information from the memory 134. Figure 2 The processing unit 131 reads information about the optimal band from the data stored in the server 151. Alternatively, the processing unit 131 controls the communication unit 135 to access the server 151 and read information about the optimal band from the data stored in the server 151.

[0130] For example, the wavelengths suitable for imaging skin conditions (such as the presence of pigmentation) and the wavelengths suitable for improving the reproducibility of skin color vary depending on race, age, gender, etc. For example, in steps S11 to S13, by performing imaging in a pre-set first wavelength and analyzing the image captured in the first wavelength, the race, age, gender, etc. of the object can be specified, and information about the optimal wavelength based on the specified information can be read.

[0131] In this configuration, memory 134 and server 151 store data relating the race, age, gender, and optimal wavelength of the objects. This data can be learned and can be updated.

[0132] When information about the optimal wavelength is obtained in step S15, the band of the Fabry-Perot spectrometer 251 of the narrowband photoelectric conversion unit 122 (appropriately described as the second band) is set based on this information. Then, the process returns to step S11 and imaging is performed in the set second band, repeating the processing in and after step S11.

[0133] However, if the analysis results are determined to be good in step S14, in other words, if it is determined that imaging has been performed in the optimal band, the process proceeds to step S16.

[0134] In step S16, the processing unit 131 synthesizes the image from the narrowband imaging unit 120 and the image from the wideband imaging unit 110, and outputs the image to the display unit, recording unit, etc. (not shown) via the image output unit 132.

[0135] In step S13, if analysis using the synthesized image is performed, a synthesized image that is also used for analysis can be output. Furthermore, if the analysis itself is performed using only images from the narrowband imaging unit 120 or the wideband imaging unit 110, a synthesized image is output after the process of generating the synthesized image is performed.

[0136] Furthermore, images from the narrowband imaging unit 120 and images from the broadband imaging unit 110 can be acquired by performing imaging again using the narrowband imaging unit 120 in the optimal band and simultaneously performing imaging using the broadband imaging unit 110, and a composite image can be generated and output.

[0137] When generating a composite image, color correction based on the analysis results can be performed, and the color-corrected composite image can be output. For example, when improving the reproducibility of skin tones, color correction for improving skin tones can be performed based on information from the analysis. Furthermore, for example, when information about the optimal wavelength is read in step S15, control data for performing color correction can be read simultaneously from memory 134 or server 151.

[0138] <Second processing of imaging device 100>

[0139] Reference Figure 9 The flowchart describes the second processing of the imaging device 100.

[0140] In step S31, the narrowband imaging unit 120 sequentially changes the wavelength of the light to be transmitted by the Fabry-Perot spectrometer 251 and performs multiple imaging operations. Simultaneously with the imaging by the narrowband imaging unit 120, the broadband imaging unit 110 also performs multiple imaging operations (step S32).

[0141] In step S33, the object is estimated. The object can be estimated using a composite image obtained by combining an image from the narrowband imaging unit 120 and an image from the wideband imaging unit 110, or using only one of the images from the narrowband imaging unit 120 and the wideband imaging unit 110.

[0142] The determination in step S34 is performed using the object estimation result from step S33. That is, in step S34, it is determined whether the object has been successfully estimated. If it is determined in step S34 that the object has been successfully estimated, the process proceeds to step S35.

[0143] In step S35, information about the optimal wavelength is obtained. The information obtained in step S35 is about the wavelength suitable for imaging the estimated object, which corresponds to the wavelength of the color to be interpolated to improve image quality.

[0144] For example, as a result of analyzing captured images, in cases where a human face occupies a large area of ​​the image, data is retrieved from memory 134 or server 151. Figure 2 Information about the wavelength of a person's skin color is read. By comparing the read information about the wavelength, the color of the person's face in the image from the narrowband imaging unit 120, and the color in the image from the broadband imaging unit 110, the wavelength to be imaged by the narrowband imaging unit 120 is set.

[0145] Since human skin color varies according to race, gender, etc., multiple pieces of information are read. Then, since the narrowband imaging unit 120 performs imaging in different bands, and the broadband imaging unit 110 also performs multiple imaging operations synchronously with the imaging of the narrowband imaging unit 120, multiple images are obtained from the narrowband imaging unit 120 and the broadband imaging unit 110.

[0146] For example, by comparing the color of a person's facial features in an image from narrowband imaging unit 120 with the color in an image from broadband imaging unit 110, a rough skin tone (race) is determined, and information about a suitable band for imaging the skin tone is selected.

[0147] The object can be estimated by obtaining data from server 151 in the cloud and using that data. Furthermore, such data can be stored in memory 134 in advance or at a predetermined time, and the data stored in memory 134 can be used.

[0148] Furthermore, the object can be set by the user. For example, in the case of analyzing pigmentation, the object is a person (face). Therefore, in the case of analyzing skin condition, a person (face) can be assumed to be set as the object for processing. Additionally, processing related to the estimation of the object can be performed based on AI functions.

[0149] In step S36, the band of the Fabry-Perot spectrometer 251 is set based on the information about the optimal wavelength obtained in the processing of step S35, and imaging in the optimal band is performed by the narrowband imaging unit 120. Furthermore, imaging is also performed synchronously with this imaging in the broadband imaging unit 110.

[0150] In step S37, the image captured in the optimal band from the narrowband imaging unit 120 and the image captured synchronously with the imaging of the narrowband imaging unit 120 from the broadband imaging unit 110 are synthesized and output.

[0151] However, if the object estimation fails in step S34, the process proceeds to step S37, where a composite image is generated and output. In this case, since multiple images have been captured in steps S31 and S32, multiple composite images can be generated. All of the multiple composite images can be output, or the image determined to be the best captured can be selected and output.

[0152] In this way, when estimating an object, imaging is performed in the band of the color to be interpolated in the image of the best captured object.

[0153] Note that, for example, the processes in steps S31 to S36 can be repeated multiple times. For example, an object is specified by performing the processes in steps S31 to S36 for the first time. If the object is specified as a person, for example, in the second process in steps S31 to S36, the skin color of the object is specified by performing multiple imaging operations while changing the band used for imaging in the bands for human skin color. Then, the optimal band for the specified skin color of the object is set, and the final imaging is performed.

[0154] By repeating the process multiple times in this way, a processing flow can be obtained that allows for more precise setting of the appropriate frequency band for the object.

[0155] Similar to the first process, in the second process, when generating the composite image, color correction based on the analysis results can be performed, and the color-corrected composite image can be output. For example, when improving the reproducibility of skin tones, color correction for improving skin tones can be performed based on information from the analysis. Furthermore, for example, when information about the optimal wavelength is read in step S35, control data for performing color correction can be read together from memory 134 or server 151.

[0156] <Third processing of imaging device 100>

[0157] Reference Figure 10 The flowchart further describes the third processing of the imaging device 100.

[0158] The difference between the third processing of the imaging device 100 and the second processing is that the step from the reference is omitted. Figure 9 The second process in the flowchart again executes the process of step S36, which sets the imaging wavelength to the optimal wavelength. Other points are similar to those in the second process, and therefore, due to their overlapping descriptions, their descriptions are appropriately omitted.

[0159] In step S51, the narrowband imaging unit 120 sequentially changes the wavelength of the light to be transmitted by the Fabry-Perot spectrometer 251 and performs multiple imaging operations. Simultaneously with the imaging by the narrowband imaging unit 120, the broadband imaging unit 110 also performs multiple imaging operations (step S52).

[0160] In step S53, the object is estimated. The determination in step S54 is performed using the object estimation result from step S53. That is, in step S54, it is determined whether the object has been successfully estimated. If it is determined in step S54 that the object has been successfully estimated, the process proceeds to step S55.

[0161] In step S55, information about the optimal wavelength is obtained. The information obtained in step S55 is about the wavelength suitable for imaging the estimated object, which corresponds to the wavelength of the color to be interpolated to improve image quality.

[0162] In step S56, based on the information about the optimal wavelength obtained in the processing of step S55, an image captured by the narrowband imaging unit 120 in the band closest to the optimal band is selected. Then, when the selected image is captured, the selected image is combined with the image captured by the broadband imaging unit 110 and output.

[0163] However, if the object estimation fails in step S54, the process proceeds to step S56, where a composite image is generated and output. In this case, since multiple images have been captured in steps S51 and S52, multiple composite images can be generated. All of the multiple composite images can be output, or the image determined to be the best captured can be selected and output.

[0164] In this way, when estimating an object, the image is selected to be captured in the band of the color to be interpolated in the image that best captures the object.

[0165] Similar to the first and second processes, in the third process, when generating the composite image, color correction based on the analysis results can be performed, and the color-corrected composite image can be output. For example, when improving the reproducibility of skin tones, color correction for improving skin tones can be performed based on information from the analysis. Furthermore, for example, when information about the optimal wavelength is read in step S55, control data for performing color correction can be read together from memory 134 or server 151.

[0166] <Other processing>

[0167] In the processing of the imaging device 100 described above, the narrowband imaging unit 120 and the wideband imaging unit 110 perform imaging synchronously, and a composite image is generated by synthesizing the synchronously captured images. In this way, when the narrowband imaging unit 120 and the wideband imaging unit 110 perform imaging synchronously, such as... Figure 11 As shown, even for moving objects, composite images can be generated without causing positional deviations.

[0168] Figure 11 The upper part of the diagram shows the object at time T1 and the object at time T2. Figure 11 As an example, the case of imaging an object that is a person (face) and has a color spot 302 on the person's face 301 will be described. Furthermore, the case of imaging an object whose face 301 is facing forward at time T1 and whose face 301 is slightly tilted at time T2 will also be described.

[0169] Figure 11 The diagram in the middle section illustrates an example of an image captured by the narrowband imaging unit 120. At time T1, the narrowband imaging unit 120 performs imaging in a band suitable for imaging the face 301 and acquires image 311. It is assumed that the discoloration 302 is not imaged in this band. At time T2, the narrowband imaging unit 120 performs imaging in a band suitable for imaging the discoloration 302 and acquires image 312. In image 312, the discoloration 302 is imaged, and the face 301 is not imaged.

[0170] In the case of synthesizing only the image obtained by the narrowband imaging unit 120, that is, if images 311 and 312 are synthesized, image 313 is obtained. In image 313, the face 301 and the freckle 302 are imaged. When referring to image 313, the freckle 302 is located above the eye. Figure 11 When the upper part of the image is shown, the actual location of the pigment spot 302 is next to the eye.

[0171] From time T1 to time T2, since the face 301 has shifted to a tilted neck position, the position of the discoloration 302 also moves in the image. When the discoloration 302 imaged by the narrowband imaging unit 120 at time T2 is combined with the face 301 imaged by the narrowband imaging unit 120 at time T1, there is a possibility of generating a composite image, such as image 313, where the discoloration 302 is located above the eyes.

[0172] In this technology, since the narrowband imaging unit 120 and the broadband imaging unit 110 perform imaging synchronously, they acquire images such as... Figure 11 The lower part of the image is shown.

[0173] At time T1, face 301 is imaged by broadband imaging unit 110. Furthermore, with narrowband imaging unit 120 configured with a band suitable for imaging spot 302, face 301 and the spot 302 next to the eyes are imaged, as shown in image 321. Therefore, as a composite image, an image of spot 302 located in the correct position is obtained, as shown in image 321.

[0174] If the narrowband imaging unit 120 is configured with a band that is not suitable for imaging the color spot 302, then as a composite image, an image (e.g., a color image) that only images the face 301 is acquired instead of an image such as image 311.

[0175] Furthermore, also at time T2, similarly, face 301 is imaged by broadband imaging unit 110, and pigmented spot 302 is imaged by narrowband imaging unit 120. At time T2, the position of pigmented spot 302 changes because the neck is in a tilted position. However, the changed face 301 is imaged by broadband imaging unit 110, and the changed pigmented spot 302 is imaged by narrowband imaging unit 120.

[0176] Therefore, as shown in image 322, a composite image 323 can be obtained in which the face 301 and the freckles 302 next to the eyes are imaged.

[0177] As described above, by synchronizing the imaging of the narrowband imaging unit 120 and the broadband imaging unit 110, imaging with reduced motion effects can be performed even if the object moves.

[0178] Furthermore, the following situations can be handled: Multiple images can be acquired by sequentially changing the bands of the narrowband imaging unit 120 and performing imaging. The case of generating a composite image obtained by synthesizing multiple previous images will also be considered.

[0179] The broadband imaging unit 110 also captures multiple images synchronously with the imaging of the narrowband imaging unit 120. The images captured by the broadband imaging unit 110 are used to detect the movement of the object.

[0180] If we don't consider the movement of objects, as in the reference... Figure 11 The middle part describes performing compositing in a state where movement is ignored even if the object is moved. Therefore, for example, there is a possibility of generating an image with a color patch 302 at an incorrect location.

[0181] By detecting the movement of the object using the image captured by the broadband imaging unit 110 and generating a synthetic image taking the movement into account, it is possible to prevent the generation of a synthetic image in which the color spot 302 is located in the wrong position. That is, by detecting the movement of the object using the image captured by the broadband imaging unit 110, for example, correcting the position of the color spot 302 in the image according to the movement, and performing synthesis, even if the object moves, it is possible to prevent the generation of a synthetic image in which the color spot 302 is located in the wrong position.

[0182] For example, when the narrowband imaging unit 120 synthesizes image 311 captured at time T1 and image 312 captured at time T2, an image is generated by moving the position of the color spot 302 in image 312 to a position corrected by the tilt amount of the object. Then, by synthesizing the image and image 311, for example, a synthesized image can be generated in which, as in image 321, the color spot 302 is located next to the eye of the face 301.

[0183] In this way, the image captured by the broadband imaging unit 110 can be used to detect object movement, and the detection result can be used to correct the image captured by the narrowband imaging unit 120. In other words, the output result from the broadband imaging unit 110 can be used to correct the output result between frames captured by the narrowband imaging unit 120. That is, according to this technology, moving object correction can be performed on the result of the narrowband imaging unit 120 by using the output obtained from the broadband imaging unit 110.

[0184] <Regarding calibration>

[0185] In the above embodiments, as an example, the narrowband photoelectric conversion unit 122 of the narrowband imaging unit 120 has been described as including a Fabry-Perot spectrometer 251, and the Fabry-Perot spectrometer 251 can change the configuration of the wavelength band of the light to be transmitted.

[0186] For reference Figure 5 As described, the Fabry-Perot spectrometer 251 has a configuration in which two semi-transparent mirrors 252 and 253 are arranged in parallel at a predetermined interval. The interval between the semi-transparent mirrors 252 and 253 is preferably uniform. However, typically, the interval between the semi-transparent mirrors 252 and 253 tends to be non-uniform.

[0187] When the spacing is non-uniform, the wavelength of light transmitted through the Fabry-Perot spectrometer 251 can be non-uniform. According to this technique, the non-uniformity of the Fabry-Perot spectrometer 251 can be corrected using an image captured by the broadband imaging unit 110 to make it appear uniform.

[0188] Reference Figure 12 The following description is provided. For example, the broadband imaging unit 110 and the narrowband imaging unit 120 image the monochromatic wall 401. The image captured by the broadband imaging unit 110 is defined as image 411. Furthermore, the image captured by the narrowband imaging unit 120 is defined as image 412.

[0189] By capturing the monochromatic wall 401, the image 411 obtained from the broadband imaging unit 110 is essentially an image with the same color as the monochromatic wall 401. However, in cases where the Fabry-Perot spectrometer 251 is non-uniform, for example, the image 412 obtained from the narrowband imaging unit 120 has a different color than the image 401. Figure 12 Image 412 shows the color non-uniformity.

[0190] When image 411 is considered the corrected image, image 412 is corrected so that image 412 becomes image 413, which is the same as image 411. For example, shadow detection is performed for each wavelength, and the image acquired by the narrowband imaging unit 120 is corrected based on the result.

[0191] Reference Figure 13 The flowchart describes the correction of the image captured by the narrowband imaging unit 120.

[0192] In step S71, the narrowband imaging unit 120 performs imaging by setting the wavelength of the light transmitted by the Fabry-Perot spectrometer 251 to a predetermined wavelength. Simultaneously with the imaging by the narrowband imaging unit 120, the broadband imaging unit 110 also performs imaging (step S72).

[0193] In step S73, the processing unit 131 processes the image captured by the narrowband imaging unit 120 (corresponding to...) Figure 12 Image 412 in the image) and the image captured by the broadband imaging unit 110 (corresponding to) Figure 12 The color uniformity in the image captured by the narrowband imaging unit 120 is compared with the color uniformity in the image captured by the wideband imaging unit 110.

[0194] In step S74, the comparison result from step S73 is used to determine whether the color uniformity in the image is different. If it is determined in step S74 that the color uniformity in the image is different, the process proceeds to step S75.

[0195] In step S75, a correction amount is calculated. The correction amount is calculated by shifting the colors of the image 412 captured by the narrowband imaging unit 120 in order to match the color uniformity of the image 411 captured by the broadband imaging unit 110.

[0196] After calculating the correction amount, the process proceeds to step S76. In step S76, imaging is performed again. For imaging, the narrowband imaging unit 120 and the broadband imaging unit 110 can be synchronized with each other, and imaging can be performed by each imaging unit. Furthermore, the image obtained in the process of step S72 can be used without imaging by the broadband imaging unit 110.

[0197] Furthermore, the narrowband imaging unit 120 performs imaging in a band different from the band set during the processing in step S71. In step S77, the image obtained by performing imaging using the narrowband imaging unit 120 in step S76 is corrected. This correction is performed using the correction amount calculated in step S75.

[0198] When the corrected image is generated in step S77, the process returns to step S73 and the subsequent processing is repeated. In step S73, the corrected image is compared with the image 411 captured by the broadband imaging unit 110.

[0199] In this way, correction parameters are set by capturing multiple images while changing the band of the narrowband imaging unit 120 and comparing them with images from the broadband imaging unit 110.

[0200] In step S74, if it is determined that the image resulting from the correction has the same uniformity as the image from the broadband imaging unit 110 (uniformity falls within a predetermined range), the process proceeds to step S78. In step S78, for example, the set correction parameters are stored in memory 134. Figure 2 )middle.

[0201] Subsequently, the image captured by the narrowband imaging unit 120 is corrected using the correction parameters stored in the memory 134, and the corrected image is used in each of the above processes.

[0202] As described above, since the imaging device 100 includes a broadband imaging unit 110 and a narrowband imaging unit 120, the image captured by the broadband imaging unit 110 can be used to correct the image captured by the narrowband imaging unit 120. Therefore, even if the Fabry-Perot spectrometer 251 is non-uniform, it can be corrected and processed to appear uniform.

[0203] According to this technology, an image can be acquired using a Fabry-Perot spectrometer 251, in which in-plane uniformity is improved in the image captured by the narrowband imaging unit 120.

[0204] <Another configuration for the broadband photoelectric conversion unit>

[0205] The configuration of the broadband photoelectric conversion unit 112 including an RGB color filter has been described as an example. The broadband photoelectric conversion unit 112 can also be configured to include filters other than color filters. For example, as... Figure 14 As shown, a configuration can be adopted in which pixels are configured to receive three colors individually and arranged in a two-dimensional array with one pixel.

[0206] Figure 14 An example cross-sectional configuration of the pixel units of the broadband photoelectric conversion unit 112 is shown. The pixels 520 arranged in the pixel array units of the broadband photoelectric conversion unit 112 include organic photoelectric conversion units 539 stacked in the depth direction within the same pixel (i.e., one pixel) and inorganic photoelectric conversion units PD1 and PD2 having two pn junctions. More specifically, the pixels 520 of the broadband photoelectric conversion unit 112 include a semiconductor substrate (silicon substrate) 522 forming the inorganic photoelectric conversion units described later. A light-receiving surface for light incidence is formed on the back side 523 side of the substrate 522, and a circuit including so-called readout circuits is formed on the front side 524 side of the substrate 522. That is, the pixel 520 has a light-receiving surface 525 on the back side 523 side of the substrate 522, and a circuit forming surface 526 on the front side 524 side of the substrate, which is the opposite side of the light-receiving surface 525. The semiconductor substrate 522 is made of a semiconductor substrate of a first conductivity type (e.g., n-type).

[0207] In the semiconductor substrate 522, a photoelectric conversion unit having two pn junctions, namely a first photodiode PD1 and a second photodiode PD2, is formed, stacked in the depth direction from the back side 523 side. In the semiconductor substrate 522, a p-type semiconductor region 528 serving as a hole accumulation layer, an n-type semiconductor region 529 serving as a charge accumulation layer, a p-type semiconductor region 531, an n-type semiconductor region 532 serving as a charge accumulation layer, and a p-type semiconductor region 533 are formed in the depth direction from the back side 523 side. The first photodiode PD1, having the n-type semiconductor region 529 as a charge accumulation layer, and the second photodiode PD2, having the n-type semiconductor region 532 as a charge accumulation layer, are formed.

[0208] In this example, the first photodiode PD1 is blue, and the second photodiode PD2 is red. Each of the n-type semiconductor regions 529 and 532 is formed to extend such that a portion of it reaches the front surface 524 of the substrate 522. Corresponding extensions 529a and 532a extend from the ends of the respective n-type semiconductor regions 529 and 532 on their opposite sides. Furthermore, a p-type semiconductor region 528, serving as a hole accumulation layer, is connected to a p-type semiconductor well region on the front surface side. Additionally, in each of the n-type semiconductor regions 529 of the first photodiode PD1 and 532 of the second photodiode PD2, a p-type semiconductor region serving as a hole accumulation layer is formed at least at the interface between the insulating films opposite to the front surface 524 of the substrate.

[0209] However, on the upper layer of the substrate back side in the region where the first photodiode PD1 and the second photodiode PD2 are formed, an organic photoelectric conversion unit 539 for the first color is stacked via an insulating film 534, configured such that the upper and lower surfaces of the organic photoelectric conversion unit 536 are sandwiched between the upper electrode 537 and the lower electrode 538a. In this example, the organic photoelectric conversion unit 536 is used for green. For example, the upper electrode 537 and the lower electrode 538a are formed from a transparent conductive film such as an indium tin oxide (ITO) film or an indium zinc oxide film.

[0210] In the example above, it was assumed that the organic photoelectric conversion unit 539 was used for green, the first photodiode PD1 for blue, and the second photodiode PD2 for red, as a color combination. However, other color combinations are possible. For example, the organic photoelectric conversion unit 539 could be set to red or blue, and the first photodiode PD1 and the second photodiode PD2 could be set to other corresponding colors. In this case, the positions of the first photodiode PD1 and the second photodiode PD2 in the depth direction are set according to the colors.

[0211] For example, organic photoelectric conversion films that utilize green wavelength light for photoelectric conversion can use organic photoelectric conversion materials containing rhodamine dyes, phthalocyanine dyes, quinacridones, etc. For organic photoelectric conversion films that utilize red wavelength light for photoelectric conversion, organic photoelectric conversion materials containing phthalocyanine dyes can be used. For organic photoelectric conversion films that utilize blue wavelength light for photoelectric conversion, organic photoelectric conversion materials containing coumarin dyes, tri-8-hydroxyquinoline Al(Alq3), phthalocyanine dyes, etc., can be used.

[0212] In the organic photoelectric conversion unit 539, transparent lower electrodes 538a and 538b, divided into two parts, are formed on an insulating film 534, and an insulating film 541 is formed to insulate and separate the lower electrodes 538a and 538b from each other. Then, an organic photoelectric conversion unit 536 and a transparent upper electrode 537 are formed on one of the lower electrodes 538a. An insulating film 542 is formed, protecting the patterned end faces of the upper electrode 537 and the organic photoelectric conversion unit 536, as well as the patterned end faces formed by etching. In this state, the upper electrode 537 is connected to the other lower electrode 538b through a contact metal layer 543 formed from another conductive film.

[0213] By forming an insulating film for protection, the end face of the organic photoelectric conversion film is protected, and contact between the organic photoelectric conversion film and the electrode can be prevented. The upper electrode 537 is selected based on its work function. Therefore, when different electrode materials come into contact with the end face (i.e., the sidewall of the organic photoelectric conversion film), dark current may be generated on the sidewall of the organic photoelectric conversion film. Furthermore, because the organic photoelectric conversion unit 536 and the upper electrode 537 are formed uniformly, a clean interface is formed. However, the sidewall of the organic photoelectric conversion unit 536, after patterning by dry etching, is not a clean surface, and contact with different electrode materials may degrade the interface and lead to an increase in dark current.

[0214] In a semiconductor substrate 522 within a pixel 520, a pair of conductive plugs 545 and 546 are formed penetrating the semiconductor substrate 522. The lower electrode 538a of the organic photoelectric conversion unit 539 is connected to one conductive plug 545, and the lower electrode 538b, connected to the upper electrode 537, is connected to the other conductive plug 546. Since one conductive plug 545 is sufficient for the lower electrode, at least one conductive plug is required throughout the entire pixel area, unless the upper electrode is separate for each pixel.

[0215] For example, conductive plugs 545 and 546 can be formed from W plugs having a SiO2 or SiN insulating layer on the periphery, semiconductor layers formed by ion implantation, etc., to suppress short circuits with Si. In this example, since the signal charge is electrons, conductive plug 545 will be an n-type semiconductor layer when formed together with the semiconductor layer by ion implantation. Preferably, the upper electrode is p-type because holes are extracted.

[0216] In this example, an n-type region 547 for charge accumulation is formed on the front side of the substrate 522 so that electrons, which are signal charges, are accumulated between electron-hole pairs photoelectrically converted by the organic photoelectric conversion unit 536 via the upper electrode 537 and the conductive plug.

[0217] The insulating film 534 on the back side 523 of the semiconductor substrate 522 is preferably a film with a negative fixed charge. For example, a hafnium oxide film can be used as the film with a negative fixed charge. That is, the insulating film 534 is formed as a three-layer structure obtained by forming a silicon oxide film, a hafnium oxide film, and a silicon oxide film from the back side 523. Because the hafnium oxide film has a negative fixed charge, the hole accumulation state at the interface between the silicon of the p-type semiconductor region (silicon) 528 and the insulating film 534 is enhanced, which is beneficial for suppressing the generation of dark current.

[0218] On the circuit forming surface 526 on the front side of the substrate 522, a plurality of pixel transistors are formed, each corresponding to the organic photoelectric conversion unit 536, the first photodiode PD1, and the second photodiode PD2. A four-transistor configuration or a three-transistor configuration can be used as the plurality of pixel transistors. Alternatively, a configuration sharing the aforementioned pixel transistors can also be used. In the organic photoelectric conversion unit 536, an n-type semiconductor region 547 for charge accumulation is connected to an n-type semiconductor region 548 serving as a floating diffusion unit and a transfer transistor Tr511 having a transfer gate electrode 549. In the first photodiode PD1, an extension 529a of the n-type semiconductor region 529 serving as a charge accumulation layer is connected to an n-type semiconductor region serving as a floating diffusion portion and a transfer transistor Tr512 having a transfer gate electrode 552. In the second photodiode PD2, an extension 532a of the n-type semiconductor region 532 serving as a charge accumulation layer is connected to the n-type semiconductor region serving as a floating diffusion portion and is connected to a transfer transistor Tr513 having a transfer gate electrode 554.

[0219] Then, a p-type semiconductor region 550 serving as a hole accumulation layer is formed at least at the interface between the substrate front side 524 and the insulating film facing the n-type semiconductor regions 529a to 532a constituting the first photodiode PD1 and the second photodiode PD2. The p-type semiconductor region 550 serving as a hole accumulation layer includes the interface between the p-type semiconductor region 533 and the insulating film. Furthermore, a p-type semiconductor region 550 serving as a hole accumulation layer is formed at the interface between the substrate front side 524 and the insulating film facing the n-type semiconductor region 547 for charge accumulation in the organic photoelectric conversion unit 536. Pixel transistors, including transfer transistors Tr511 to Tr513, are formed in the p-type semiconductor well region on the front side of the substrate.

[0220] Note that, although not shown, the pixel transistors of the pixel unit are formed on the front side of the semiconductor substrate 522, and peripheral circuits such as logic circuits are formed in the peripheral circuit unit.

[0221] On the front side of the semiconductor substrate 522, a multilayer wiring layer 558 with multilayer wiring 557 is formed via an interlayer insulating film 556. A support substrate 559 is attached to the multilayer wiring layer 558.

[0222] On the back side of the semiconductor substrate 522, more specifically, the surface of the upper electrode 537 of the organic photoelectric conversion unit 539 serves as a light-receiving surface 525. Then, an on-chip lens 562 is formed on the organic photoelectric conversion unit 539 via a planarization film 561. In this example, no color filter is formed.

[0223] Pixel 520, which does not have a color filter formed on it, can also be used as a pixel of the broadband imaging unit 110.

[0224] In addition, instead of a filter, it has Figure 15 The color arrangement of the filter shown can be used in the broadband imaging unit 110. Furthermore, as a filter, a filter called a plasma filter, which utilizes plasma resonance to perform optical control, can be used. Figure 15 This is a diagram illustrating an example configuration of adding G pixels to multispectral pixels. Figure 15 In this context, "G" refers to a G pixel, and "MS" refers to an MS pixel. A G pixel is a pixel whose color is green in the color filter layer. An MS pixel is a multispectral pixel, and it is a pixel that receives light (of a predetermined frequency band and color).

[0225] Figure 15 The diagram shows 16 pixels arranged in a 4×4 pattern within pixel region 203, and this array of pixel groups repeats within pixel region 203. (For distinction...) Figure 15 The 16 pixels shown are individually numbered. For example, in the 16 pixels, the pixel in the top left corner is pixel G1, and the pixel to the right of pixel G1 is pixel MS1.

[0226] exist Figure 15 The color arrangement shown illustrates an example where the same number of G pixels and MS pixels are arranged. That is, in 16 pixels, G1 to G8 are G pixels, and MS1 to MS8 are MS pixels. Furthermore, G pixels and MS pixels are arranged alternately in each of the horizontal and vertical directions.

[0227] Note that the description will continue here using an example of alternating G-pixels and MS-pixels, but the arrangement can be different. For example, the color arrangement could be such that two G-pixels and two MS-pixels are arranged alternately, or one G-pixel and two MS-pixels are arranged alternately.

[0228] G-pixels are pixels that receive green light, such as light in the 500nm to 550nm frequency band. Figure 15In the diagram, each of pixels G1 to G8 is a pixel that receives light in that frequency band.

[0229] MS pixels are pixels that receive light of the frequency band to be extracted. Figure 15 In this configuration, pixels MS1 through MS8 receive light from different frequency bands. That is, in this case, pixels MS1 through MS8 are a sensor capable of processing light from eight frequency bands.

[0230] Note that here, we will continue to describe the assumption that all pixels MS1 to MS8 receive light from different frequency bands, but they could also be pixels receiving light from the same frequency band. In the case of processing, for example, four frequency bands depending on the number of bands to be processed, the following configuration can also be used: for example, pixels MS1 and MS2 are pixels receiving light from the first frequency band, pixels MS3 and MS4 are pixels receiving light from the second frequency band, pixels MS5 and MS6 are pixels receiving light from the third frequency band, and pixels MS7 and MS8 are pixels receiving light from the fourth frequency band.

[0231] Assuming that all pixels MS1 to MS8 receive light from different frequency bands, for example, obtaining the following from pixels MS1 to MS8: Figure 16 The signal shown.

[0232] Pixel MS1 receives light from band M1. Similarly, pixel MS2 receives light from band M2, pixel MS3 receives light from band M3, pixel MS4 receives light from band M4, pixel MS5 receives light from band M5, pixel MS6 receives light from band M6, pixel MS7 receives light from band M7, and pixel MS8 receives light from band M8.

[0233] In this way, pixels MS1 to MS8 can be pixels that receive light from different frequency bands M1 to M8, respectively. Furthermore, pixels G1 to G8 can each receive light from the green frequency band G.

[0234] Therefore, in this configuration 1 case, for Figure 15 The 16 pixels shown are used to obtain information about green from pixel G, information about color M1 in band M1 from pixel MS1, information about color M2 in band M2 from pixel MS2, information about color M3 in band M3 from pixel MS3, information about color M4 in band M4 from pixel MS4, information about color M5 in band M5 from pixel MS5, information about color M6 in band M6 from pixel MS6, information about color M7 in band M7 from pixel MS7, and information about color M8 in band M8 from pixel MS8.

[0235] This technology can also be applied to the following situations: using reference Figure 15 and Figure 16 The plasma filter described is configured as a filter for the broadband photoelectric conversion unit 112 to power the multispectral sensor and capture multispectral images.

[0236] Note that multispectral sensors can be implemented using filters other than plasma filters; for example, they can be implemented by making the color filter multicolor. Furthermore, in addition to the visible light region, pixels that process light from the ultraviolet, infrared, and other regions can also be included.

[0237] <Examples applied to AR, VR, etc.>

[0238] The imaging device 100 using this technology can be applied to devices that provide augmented reality (AR), virtual reality (VR), mixed reality (MR), etc. Here, the application of the imaging device 100 to a device that provides AR will be described as an example.

[0239] Figure 17 This is a diagram illustrating a configuration example of an information processing system including the AR-HMD 701 that provides AR. Figure 17 The information processing system is configured by connecting AR-HMD 701 and information processing device 702 via a network 703, such as a local area network (LAN) or the Internet.

[0240] like Figure 17 As shown, the AR-HMD 701 is a glasses-type wearable terminal that includes a transmissive display unit. Under the control of an information processing device 702 executed via a network 703, the AR-HMD 701 displays video images of various objects, including people, on its display unit. The user will see objects superimposed on a landscape in front of them.

[0241] The projection method for the video image of the object can be a virtual image projection method, or it can be a retinal projection method in which the image is directly formed on the retina of the user's eye.

[0242] Information processing device 702 reproduces AR content and transmits the video data obtained through reproduction to AR-HMD 701, so that the video image of the AR content is displayed on AR-HMD 701. Information processing device 702 is configured, for example, by a personal computer (PC). Alternatively, information processing device 702 may be a server 151. Figure 2 And, for example, it can be configured to provide information about the estimation of the object and information about the band suitable for the object to the AR-HMD 701.

[0243] Instead of AR-HMD 701, such as Figure 18 The AR-HMD 701A, or the video transmission HMD shown in Figure A, is a video transmission HMD. Figure 18 The mobile terminal of the smartphone 701B shown in Figure B can be used as a display device for AR content.

[0244] When using the AR-HMD 701A as a display device, the video image of the AR content reproduced by the information processing device 702 is displayed as an overlay on an image of the landscape in front of the AR-HMD 701A captured by a camera installed in the AR-HMD 701A. A display showing the AR content overlaid on the image captured by the camera is located in front of the eyes of the user wearing the AR-HMD 701A.

[0245] Furthermore, when using the smartphone 701B, video images of AR content reproduced by the information processing device 702 are displayed as overlays on an image of the landscape in front of the smartphone 701B captured by a camera mounted on the back of the smartphone 701B. A display showing various images is provided on the front of the smartphone 701B.

[0246] The imaging device 100 described above can be applied to cameras included in AR-HMD 701, AR-HMD 701A and smartphone 701B.

[0247] As a display device for AR content, a projector that projects video images onto the front surface of an object existing in a real-world scene can be used. Various devices, such as tablet terminals and television receivers, can be used as display devices for AR content.

[0248] The display device and information processing equipment 702 can be connected wirelessly via a wired connection instead of via a network 703.

[0249] In the following description, such as Figure 19 As shown, a wearable terminal in the form of glasses will be described as an example of an information processing device that applies this technology.

[0250] Figure 19 The AR-HMD 701 shown is generally shaped like glasses and includes a display unit 711 and a camera 712. The display unit 711 corresponds to the lens portion of the glasses and is configured, for example, entirely as a transmissive display. Therefore, the display unit 711 transmissively overlays and displays annotations (virtual objects) on an image (real object) in the real world that the user is directly viewing.

[0251] Camera 712 is positioned at the end of display unit 711 corresponding to the left eye of the user wearing AR-HMD 701, and captures images of the real space including the user's field of vision. Imaging device 100 ( Figure 2 It can be applied to camera 712.

[0252] Images acquired by camera 712 can be displayed on display unit 711, and annotations can also be overlaid and displayed on the image. Furthermore, although not shown, the housing corresponding to the frame of the glasses in AR-HMD 701 can accommodate or mount various sensors, buttons, speakers, etc.

[0253] Note that the shape of AR-HMD 701 is not limited to... Figure 19 The shape shown is valid, and various shapes are possible, such as a hat shape, a strap shape that is fastened around the user's head, and a helmet shape that covers the entire user's head. That is, the technology according to this disclosure is generally applicable to HMDs.

[0254] Figure 20 This is a block diagram showing a configuration example of the AR-HMD 701. Figure 20 The AR-HMD 701 includes a central processing unit (CPU) 731, a memory 732, a sensor unit 733, an input unit 734, an output unit 735, and a communication unit 736. These are interconnected via a bus 737.

[0255] The CPU 731 performs processing to implement various functions of the AR-HMD 701 based on the programs and data stored in the memory 732. The memory 732 is configured with a storage medium such as a semiconductor memory or a hard disk, and stores the programs and data to be processed by the CPU 731.

[0256] In addition to sensor unit 733 Figure 19 In addition to the camera 712, various sensors such as a microphone, gyroscope sensor, and accelerometer sensor are included. The CPU 731 processes the various sensor information acquired by the sensor unit 733. The input unit 734 is configured with buttons, keys, touch panels, etc. The output unit 735 consists of… Figure 19 The system includes a display unit 711, speakers, and other components. The communication unit 736 is configured as a communication interface for mediating various types of communication.

[0257] When users use, such as Figure 21 When the AR-HMD 701 shown makes a gesture to touch and display information 751 in virtual space, for example, information about the information is displayed.

[0258] For example, if the information 751 displayed in the virtual space is a tree, the Fabry-Perot spectrometer 251 of the narrowband imaging unit 120 is set to have a band suitable for analyzing the state of the tree, performs imaging, and presents the image obtained by imaging and the information obtained by analyzing the image to the user.

[0259] The information 751 displayed in the virtual space can be information provided as AR content, or it can be a real-world object imaged by the camera 712. For example, the tree that is the aforementioned information 751 can be a tree that grows in the real world and is captured by the camera 712.

[0260] In addition to the gestures of touch information 751, for example, sensor unit 733 can detect the direction the user is facing, the direction of their gaze, the direction their head is facing, etc. In other words, user operations other than gestures that represent direct commands such as touch information 751 can also be included as part of the gestures used to issue commands, and such gestures (user operations) can be detected.

[0261] In addition, user movement can be detected and corresponding processing can be performed, such as processing to obtain information about objects in the user's line of sight.

[0262] The AR-HMD 701 can be used to analyze human skin conditions, image affected areas of patients as a medical device, and detect predetermined objects. Furthermore, information needed to perform this analysis, such as information about appropriate wavelengths, can be obtained from cloud servers.

[0263] Furthermore, the analysis can be performed by a server in the cloud, and the AR-HMD 701 receives the analysis results and presents them to the user. That is, the above analysis can be performed by an electronic device other than the imaging device 100 (including a data analysis unit in the electronic device), and the analysis can be performed via such an electronic device.

[0264] <About Recording Media>

[0265] The aforementioned series of processes can be performed by hardware or software. In the case where the processes are performed by software, a program for configuring the software is installed in the computer. Examples of computers include, for instance, computers built into dedicated hardware, and general-purpose personal computers that can perform various functions by installing various programs.

[0266] Figure 22 This is a block diagram illustrating an example hardware configuration of a computer performing the aforementioned series of processes according to a program. In the computer, a central processing unit (CPU) 1001, a read-only memory (ROM) 1002, and a random access memory (RAM) 1003 are interconnected via a bus 1004. The bus 1004 is further connected to an input / output interface 1005. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a driver 1010 are connected to the input / output interface 1005.

[0267] Input unit 1006 includes a keyboard, mouse, microphone, imaging element, etc. Output unit 1007 includes a display, speaker, etc. Storage unit 1008 includes a hard disk, non-volatile memory, etc. Communication unit 1009 includes a network interface, etc. Driver 1010 drives removable media 1011, such as a hard disk, optical disk, magneto-optical disk, or semiconductor memory.

[0268] In a computer configured as described above, for example, the CPU 1001 loads the program recorded in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004, and performs the series of processes described above.

[0269] The program executed by the computer (CPU 1001) can be provided by recording it on a removable medium 1011, such as a packaging medium. Alternatively, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0270] In a computer, by attaching the removable medium 1011 to the drive 1010, a program can be installed in the storage unit 1008 via the input / output interface 1005. Alternatively, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Furthermore, the program can be pre-installed in the ROM 1002 and the storage unit 1008.

[0271] Note that a program executed by a computer may be a program that performs processing in chronological order according to the order described in this specification, or it may be a program that performs processing in parallel or at necessary time intervals (such as when a call is made).

[0272] <Examples of Applications of Endoscopic Surgical Systems>

[0273] The technology disclosed herein (the technology) can be applied to various products. For example, the technology disclosed herein can be applied to endoscopic surgical systems.

[0274] Figure 23 This is a diagram illustrating an example of a schematic configuration of an endoscopic surgical system to which the technology (the technology) according to this disclosure can be applied.

[0275] Figure 23The illustration shows a surgeon (physician) 11131 performing surgery on a patient 11132 on a bed 11133 using an endoscopic surgical system 11000. As shown, the endoscopic surgical system 11000 includes an endoscope 11100, other surgical instruments 11110 (such as a pneumoperitoneum tube 11111 and an energy therapy device 11112), a support arm device 11120 supporting the endoscope 11100 thereon, and a trolley 11200 on which various devices for endoscopic surgery are mounted.

[0276] Endoscope 11100 includes a lens barrel 11101, the portion of which is inserted into a body cavity of patient 11132 at a predetermined length, and a camera 11102 connected to the proximal end of the lens barrel 11101. In the illustrated example, endoscope 11100 is shown comprising a rigid lens barrel 11101 as a rigid endoscope. However, endoscope 11100 may additionally comprise a flexible endoscope having a flexible lens barrel 11101.

[0277] The lens tube 11101 has an opening at its distal end for mounting the objective lens. The endoscope 11100 is connected to a light source device 11203, and the light generated by the light source device 11203 is guided by a light guide extending inside the lens tube 11101 to the distal end of the lens tube, and then illuminates the target for observation within the body cavity of the patient 11132 through the objective lens. Note that the endoscope 11100 can be a direct-viewing endoscope, an oblique-viewing endoscope, or a lateral-viewing endoscope.

[0278] An optical system and imaging element are housed inside the camera 11102, such that reflected light (observation light) from the observed target is converged onto the imaging element via the optical system. The imaging element performs photoelectric conversion on the observation light and generates an electrical signal corresponding to the observation light; in other words, an image signal corresponding to the observed image. The image signal is transmitted as RAW data to the camera control unit (CCU) 11201.

[0279] The CCU 11201 includes a central processing unit (CPU), a graphics processing unit (GPU), etc., and controls the operation of the endoscope 11100 and the display device 11202 as a whole. Furthermore, for example, the CCU 11201 receives image signals from the camera 11102 and performs various image processing operations on the image signals for displaying images based on the image signals, such as image processing (de-mosaic processing).

[0280] The display device 11202 displays an image based on an image signal under the control of the CCU 11201, and the CCU 11201 performs image processing on the image signal.

[0281] The light source device 11203 includes, for example, a light source such as a light-emitting diode (LED) and provides illumination light to the endoscope 11100 when imaging the surgical area.

[0282] Input device 11204 is the input interface of endoscopic surgery system 11000. Users can input various types of information and input commands into endoscopic surgery system 11000 via input device 11204. For example, users can input commands to change the imaging conditions of endoscope 11100 (type of illumination light, magnification, focal length, etc.).

[0283] Treatment tool control device 11205 controls the drive of energy treatment tool 11112 to perform tissue ablation or cutting, vascular sealing, etc. To ensure a clear view by endoscope 11100 and to ensure the surgeon's working space, inflator 11206 delivers gas into the patient's body cavity 11132 through insulated tubing 11111 to inflate the cavity. Recorder 11207 is a device capable of recording various types of information about the surgery. Printer 11208 is a device capable of printing various information related to the surgery in various formats, such as text, images, or graphics.

[0284] Note that the light source device 11203 supplying illumination light when imaging the surgical area onto the endoscope 11100 may, for example, include a white light source comprising LEDs, laser light sources, or combinations thereof. In the case where the white light source comprises a combination of red, green, and blue (RGB) laser light sources, the white balance adjustment of the captured image can be performed by the light source device 11203 because the output intensity and timing can be controlled with high precision for each color (each wavelength). Furthermore, in this case, images corresponding to each of the RGB light sources can be captured in a time-division manner by illuminating the target with laser light from each of the RGB laser light sources in a time-division manner, and by controlling the driving of the imaging element of the camera 11102 in synchronization with the illumination timing. According to this method, color images can be obtained even without setting a color filter for the imaging element.

[0285] Furthermore, the light source device 11203 can be controlled to change the intensity of the light to be output at predetermined intervals. By controlling the driving of the imaging element of the camera 11102 in time-division multiplexing to acquire and synthesize images in sync with the timing of the light intensity changes, high dynamic range images can be produced without underexposed thick shadows and overexposed bright areas.

[0286] Additionally, the light source device 11203 can also be configured to enhance light of a predetermined wavelength band prepared for special light observation. In special light observation, narrow-band imaging (narrow-band imaging) is performed by irradiating a predetermined tissue, such as blood vessels in the superficial portion of a mucosa, with high contrast by utilizing the wavelength dependence of light absorption by body tissue to illuminate a narrow-band light compared to the illumination light used in normal observation (i.e., white light). Alternatively, fluorescence observation, which obtains an image from fluorescence generated by irradiating excitation light, can also be performed in special light observation. In fluorescence observation, this can include irradiating body tissue with excitation light and observing fluorescence from the body tissue (autofluorescence observation), locally injecting a reagent such as indocyanine green (ICG) into body tissue and irradiating the body tissue with excitation light corresponding to the fluorescence wavelength of the reagent to obtain a fluorescence image, etc. The light source device 11203 can be configured to provide narrow-band light and / or excitation light suitable for special light observation as described above.

[0287] Figure 24 It is shown Figure 23 A block diagram showing an example of the functional configuration of camera 11102 and CCU 11201.

[0288] Camera 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera control unit 11405. CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. Camera 11102 and CCU 11201 are connected via a transmission cable 11400 for communication with each other.

[0289] Lens unit 11401 is an optical system disposed at the connection position with lens barrel 11101. Observation light collected from the distal end of lens barrel 11101 is guided to camera 11102 and incident on lens unit 11401. Lens unit 11401 includes a combination of multiple lenses, including zoom lenses and focusing lenses.

[0290] The imaging unit 11402 may include one (single-plate type) or multiple (multi-plate type) imaging elements. When the imaging unit 11402 is configured as a multi-plate type, for example, each imaging element can generate an image signal corresponding to its respective RGB values, and a color image can be obtained by synthesizing the image signals. Alternatively, the imaging unit 11402 may have a pair of imaging elements for acquiring image signals corresponding to the right and left eyes in a three-dimensional (3D) display, respectively. By performing 3D display, the surgeon 11131 can more accurately determine the depth of biological tissue in the surgical area. It should be noted that when the imaging unit 11402 is configured as a stereoscopic imaging unit, multiple lens units 11401 are provided corresponding to each imaging element.

[0291] Furthermore, the imaging unit 11402 need not be mounted on the camera 11102. For example, the imaging unit 11402 can be mounted inside the lens barrel 11101, immediately behind the objective lens.

[0292] The drive unit 11403 includes an actuator and, under the control of the camera control unit 11405, moves the zoom lens and focusing lens of the lens unit 11401 a predetermined distance along the optical axis. Therefore, the magnification and focus of the image captured by the imaging unit 11402 can be appropriately adjusted.

[0293] The communication unit 11404 includes communication means for transmitting various information to and receiving various information from the CCU 11201. The communication unit 11404 transmits the image signal acquired from the imaging unit 11402 as RAW data to the CCU 11201 via the transmission cable 11400.

[0294] In addition, the communication unit 11404 receives control signals from the CCU 11201 for controlling the camera 11102 and supplies the control signals to the camera control unit 11405. The control signals include information related to imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value during imaging, and / or information specifying the magnification and focus of the captured image.

[0295] It should be noted that imaging conditions such as the frame rate, exposure value, magnification, or focus described above can be specified by the user or can be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. In the latter case, the automatic exposure (AE) function, automatic focus (AF) function, and automatic white balance (AWB) function are combined in the endoscope 11100.

[0296] The camera control unit 11405 controls the driving of the camera 11102 based on the control signals received from the CCU 11201 via the communication unit 11404.

[0297] The communication unit 11411 includes a communication device for transmitting various information to and receiving various information from the camera 11102. The communication unit 11411 receives image signals transmitted from the camera 11102 via a transmission cable 11400.

[0298] In addition, the communication unit 11411 transmits control signals for controlling the camera 11102 to the camera 11102. Image signals and control signals can be transmitted via electrical communication, optical communication, etc.

[0299] The image processing unit 11412 performs various image processing operations on the image signal in RAW data form transmitted from the camera 11102.

[0300] The control unit 11413 performs various controls related to imaging of the surgical area, etc., and displaying the captured images obtained through imaging of the surgical area, etc. via the endoscope 11100. For example, the control unit 11413 generates control signals for controlling the drive of the camera 11102.

[0301] Furthermore, the control unit 11413 controls the display device 11202 to display captured images of the surgical area and other objects based on image signals processed by the image processing unit 11412. At this time, the control unit 11413 identifies various objects in the captured image using various image recognition technologies. For example, by detecting the shape, color, etc., of the edges of objects included in the captured image, the control unit 11413 can identify surgical tools such as forceps, specific living sites, bleeding, and aerosols using energy therapy tools such as the energy therapy tool 11112. When the display device 11202 displays the captured image, the control unit 11413 can use the recognition results to overlay and display various types of surgical support information on the image of the surgical area. When surgical support information is displayed and presented to the surgeon 11131 in an overlaid manner, the burden on the surgeon 11131 can be reduced, and the surgeon 11131 can perform the surgery reliably.

[0302] The transmission cable 11400 connecting the camera 11102 and CCU11201 to each other is an electrical signal cable prepared for electrical signal communication, an optical fiber prepared for optical communication, or a composite cable prepared for both electrical and optical communication.

[0303] Here, although communication is performed via wired communication using transmission cable 11400 in the example shown, communication between camera 11102 and CCU 11201 can be performed wirelessly.

[0304] <Application Examples of Moving Objects>

[0305] The technology disclosed herein (the technology) can be applied to a variety of products. For example, the technology disclosed herein can be implemented as a device mounted on any type of mobile body (such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, robots, etc.).

[0306] Figure 25 This is a block diagram illustrating an example configuration of a vehicle control system, which is an example of a mobile body control system to which the technology according to embodiments of the present disclosure can be applied.

[0307] The vehicle control system 12000 includes multiple electronic control units interconnected via a communication network 12001. Figure 25 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, a vehicle external information detection unit 12030, a vehicle internal information detection unit 12040, and an integrated control unit 12050. Furthermore, a microcomputer 12051, an audio / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as functional configurations of the integrated control unit 12050.

[0308] The drive system control unit 12010 controls the operation of devices related to the vehicle's drive system according to various programs. For example, the drive system control unit 12010 is used as a control device for drive force generating devices (such as internal combustion engines, drive motors, etc.) that generate drive force for the vehicle, drive force transmission mechanisms that transmit drive force to the wheels, steering mechanisms that adjust the vehicle's steering angle, and braking devices that generate braking force for the vehicle.

[0309] The body system control unit 12020 controls the operation of various devices installed on the vehicle body according to various programs. For example, the body system control unit 12020 acts as a control device for keyless entry systems, smart key systems, power windows, or various lights such as headlights, reversing lights, brake lights, turn indicators, or fog lights. In this case, radio waves or signals from a portable device that replaces the key can be input to the body system control unit 2020 for various switches. The body system control unit 12020 receives these input radio waves or signals and controls the vehicle's door locking devices, power windows, lights, etc.

[0310] The vehicle exterior information detection unit 12030 detects information about the exterior of the vehicle, including the vehicle control system 12000. For example, an imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture images of the vehicle exterior and receives the captured images. The vehicle exterior information detection unit 12030 can perform processing for detecting objects (such as people, vehicles, obstacles, signs, symbols, etc. on the road surface) or processing for determining the distance to the detected objects based on the received images.

[0311] Imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. Imaging unit 12031 can output the electrical signal as an image or as distance measurement information. Furthermore, the light received by imaging unit 12031 can be visible light or invisible light such as infrared light.

[0312] The vehicle interior information detection unit 12040 detects information about the vehicle interior. The vehicle interior information detection unit 12040 is connected, for example, to a driver state detection unit 12041 that detects the driver's state. The driver state detection unit 12041 includes, for example, a camera that images the driver. Based on the detection information input from the driver state detection unit 12041, the vehicle interior information detection unit 12040 can calculate the driver's fatigue level or concentration level, or determine whether the driver is drowsy.

[0313] The microcomputer 12051 can calculate control target values ​​for the drive force generation device, steering mechanism, or braking device based on information about the vehicle's interior or exterior obtained by the vehicle external information detection unit 12030 or the vehicle internal information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control designed to implement functions of an advanced driver assistance system (ADAS), including collision avoidance or shock absorption for the vehicle, following driving based on following distance, maintaining vehicle speed, collision warning, lane departure warning, etc.

[0314] Furthermore, the microcomputer 12051 can control the drive force generation device, steering mechanism, braking device, etc., based on information about the exterior or interior of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, thereby performing cooperative control for autonomous driving, which enables the vehicle to drive automatically without relying on the driver's operation.

[0315] Furthermore, the microcomputer 12051 can output control commands to the body system control unit 12020 based on information about the vehicle's exterior obtained by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare by controlling the headlights to change from high beam to low beam based on the position of the vehicle in front or oncoming vehicles detected by the vehicle exterior information detection unit 12030.

[0316] The sound / image output unit 12052 transmits at least one of sound and image output signals to an output device capable of visually or audibly notifying vehicle occupants or the outside of the vehicle. Figure 25 In the example, audio speaker 12061, display unit 12062, and instrument panel 12063 are shown as output devices. Display unit 12062 may include, for example, at least one of an in-vehicle display and a head-up display.

[0317] Figure 26 This is a diagram showing an example of the mounting location of the imaging unit 12031.

[0318] exist Figure 26 In the imaging unit 12031, imaging units 12101, 12102, 12103, 12104 and 12105 are included.

[0319] Imaging units 12101, 12102, 12103, 12104, and 12105 are, for example, located on the front nose, side mirrors, rear bumper, and rear door of vehicle 12100, as well as on the upper part of the windshield inside the vehicle compartment. Imaging unit 12101 on the front nose and imaging unit 12105 on the upper part of the windshield inside the vehicle compartment primarily acquire images of the front of vehicle 12100. Imaging units 12102 and 12103 on the side mirrors primarily acquire images of the sides of vehicle 12100. Imaging unit 12104 on the rear bumper or rear door primarily acquires images of the rear of vehicle 12100. Imaging unit 12105 on the upper part of the windshield inside the vehicle compartment is mainly used to detect vehicles, pedestrians, obstacles, signals, traffic signs, lanes, etc., ahead.

[0320] Notice, Figure 26Examples of the imaging ranges of imaging units 12101 to 12104 are shown. Imaging range 12111 represents the imaging range of imaging unit 12101 located in the front nose. Imaging ranges 12112 and 12113 represent the imaging ranges of imaging units 12102 and 12103 located in the side mirrors, respectively. Imaging range 12114 represents the imaging range of imaging unit 12104 located in the rear bumper or rear door. For example, a bird's-eye view of the vehicle 12100 viewed from above is obtained by overlaying image data captured by imaging units 12101 to 12104.

[0321] At least one of the imaging units 12101 to 12104 may have the function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of multiple imaging elements, or may be an imaging element having pixels for phase difference detection.

[0322] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging range 12111 to 12114 and the time change of that distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and thereby extract the nearest three-dimensional object existing on the driving path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., equal to or greater than 0 km / h) as the vehicle ahead. Furthermore, the microcomputer 12051 can preset the vehicle-to-vehicle distance to be ensured and execute automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. Therefore, cooperative control for autonomous driving, which enables the vehicle to drive automatically without relying on driver operation, can be performed.

[0323] For example, microcomputer 12051 can classify 3D object data of 3D objects into 3D object data of two-wheeled vehicles, standard vehicles, large vehicles, pedestrians, utility poles, and other 3D objects based on distance information obtained from imaging units 12101 to 12104, extract the classified 3D object data, and use the extracted 3D object data for automatic obstacle avoidance. For example, microcomputer 12051 distinguishes obstacles around vehicle 12100 into obstacles visible to the driver of vehicle 12100 and obstacles that are difficult to see. Then, microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle. If the collision risk is equal to or higher than a set value and therefore there is a possibility of collision, microcomputer 12051 outputs an alarm to the driver via audio speaker 12061 or display unit 12062, and executes forced deceleration or evasive steering via driving system control unit 12010. Microcomputer 12051 can thereby assist driving to avoid collisions.

[0324] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 can identify a pedestrian by determining whether a pedestrian exists in the captured images of the imaging units 12101 to 12104. This pedestrian identification is performed, for example, by extracting feature points from the captured images of the imaging units 12101 to 12104, which are infrared cameras, and performing pattern matching processing on a series of feature points representing the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the captured images of the imaging units 12101 to 12104 and identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 to overlay and display a rectangular outline for emphasizing the identified pedestrian. Furthermore, the sound / image output unit 12052 may also control the display unit 12062 to display an icon or the like representing a pedestrian at a desired location.

[0325] Furthermore, in this specification, "system" refers to an entire device comprising multiple devices.

[0326] Note that the effects described in this manual are merely examples and are not limited thereto; other effects may also exist.

[0327] Note that the embodiments of this technology are not limited to the above embodiments, and various modifications can be made without departing from the scope of this technology.

[0328] Note that this technology can also have the following configurations. (1)

[0330] An imaging device, comprising:

[0331] Two or more imaging units are capable of imaging or sensing the same object, wherein

[0332] At least one of two or more imaging units, a first imaging unit, includes a first filter configured to transmit multiple wavelength bands, and

[0333] At least one of the two or more imaging units, in addition to the first imaging unit, includes a second filter capable of changing the band. (2)

[0335] According to the imaging device of (1), wherein,

[0336] The bandwidth of the second imaging unit is narrower than that of the first imaging unit. (3)

[0338] According to the imaging device of (1) or (2), wherein,

[0339] The second filter is a Fabry-Perot spectrometer. (4)

[0341] According to the imaging device in (3), wherein,

[0342] The Fabry-Perot spectrometer is formed by a microelectromechanical system (MEMS) that can be driven by voltage. (5)

[0344] An imaging apparatus according to any one of (1) to (4), wherein,

[0345] The first filter is a color filter, and

[0346] The band of the second filter is set to the band of the color used for interpolating the color obtained by the color filter. (6)

[0348] According to the imaging device of (1), wherein,

[0349] The first imaging unit has four or more bands, and

[0350] Optical control using plasma resonance is used for four or more bands. (7)

[0352] An imaging apparatus according to any one of (1) to (6), wherein,

[0353] Organic photoelectric conversion films are used in at least one of two or more imaging units. (8)

[0355] An imaging apparatus according to any one of (1) to (7), wherein,

[0356] The object is estimated by using the image captured by the first imaging unit, and

[0357] The second filter's band is set to a band suitable for imaging the estimated object. (9)

[0359] According to the imaging device of (8), wherein,

[0360] The object is estimated using a synthetic image obtained by combining an image captured by a first imaging unit and an image captured by a second imaging unit. (10)

[0362] An imaging apparatus according to any one of (1) to (9), wherein,

[0363] The color information obtained by the second imaging unit is used to correct the image captured by the first imaging unit. (11)

[0365] An imaging apparatus according to any one of (1) to (10), wherein,

[0366] The distribution of color information in the image captured by the second imaging unit is compared with the distribution of color information in the image captured by the first imaging unit, and a correction amount is set to correct the image captured by the second imaging unit. (12)

[0368] An imaging apparatus according to any one of (1) to (11), wherein,

[0369] Moving object correction is performed on the results of the second imaging unit using the output obtained from the first imaging unit. (13)

[0371] An imaging apparatus according to any one of (1) to (12), wherein,

[0372] Object analysis and state analysis are performed using results obtained from two or more imaging units. (14)

[0374] According to the imaging device of (13), wherein,

[0375] Analysis is performed on the results obtained from any one of two or more imaging units, or on the results obtained by synthesizing the results obtained from two or more imaging units, via another electronic device at the communication destination. (15)

[0377] According to the imaging device of (14), wherein,

[0378] The band of the second filter is specified by another electronic device. (16)

[0380] An imaging apparatus according to any one of (1) to (15), wherein,

[0381] The first filter operates in the visible light band, and

[0382] The second filter can be used in the ultraviolet, visible, or infrared light bands. (17)

[0384] An imaging apparatus according to any one of (1) to (16), wherein,

[0385] Imaging devices are included in any device that provides augmented reality (AR), virtual reality (VR), and mixed reality (MR). (18)

[0387] An imaging apparatus according to any one of (1) to (17), wherein,

[0388] The first imaging unit and the second imaging unit perform imaging synchronously with each other. (19)

[0390] An imaging method, wherein,

[0391] The imaging device includes:

[0392] Two or more imaging units are capable of imaging or sensing the same object, wherein,

[0393] At least one of two or more imaging units, a first imaging unit, includes a first filter configured to transmit multiple wavelength bands, and

[0394] At least one second imaging unit, other than the first imaging unit, in one of two or more imaging units includes a second filter capable of changing the spectral band, and the imaging method includes:

[0395] The object is estimated by using a synthetic image obtained by combining an image captured by a first imaging unit and an image captured by a second imaging unit. (20)

[0397] An electronic device comprising:

[0398] Imaging device, including:

[0399] Two or more imaging units are capable of imaging or sensing the same object, wherein,

[0400] At least one of two or more imaging units, a first imaging unit, includes a first filter configured to transmit multiple wavelength bands, and

[0401] At least one second imaging unit, in addition to the first imaging unit, in one of two or more imaging units includes a second filter capable of changing the band; and

[0402] The processing unit is configured to process signals from the imaging device.

[0403] Reference tag list

[0404] 10 Compound Eye Camera Module

[0405] 21. SLR camera module

[0406] 22 Connecting components

[0407] 100 Imaging Device

[0408] 110 broadband imaging units

[0409] 111 Lens

[0410] 112 Broadband photoelectric conversion unit

[0411] 113 A / D Conversion Unit

[0412] 114 Clamping Units

[0413] 115 Color-Specific Output Units

[0414] 116 Defect Correction Unit

[0415] 117 Linear Matrix Units

[0416] 120 narrowband imaging units

[0417] 121 Lens

[0418] 122 Narrowband Photoelectric Conversion Unit

[0419] 123 A / D Conversion Unit

[0420] 131 Processing Unit

[0421] 132 Image Output Unit

[0422] 134 memory

[0423] 135 Communication Unit

[0424] 151 Server

[0425] 202 pixels

[0426] 203 pixel area

[0427] 204 Vertical Drive Circuit

[0428] 205-column signal processing circuit

[0429] 206 Horizontal Drive Circuit

[0430] 207 Output Circuit

[0431] 208 Control Circuit

[0432] 209 Vertical signal lines

[0433] 210 Horizontal Signal Line

[0434] 212 Input / Output Terminals

[0435] 251 Fabry-Perot Spectrometer

[0436] 252 Semi-transparent mirror

[0437] 253 Semi-transparent mirror

[0438] Images 311, 312, and 313

[0439] Images 321 and 322

[0440] 323 Composite Image

[0441] 401 wall

[0442] Images 411 and 412.

Claims

1. An imaging device, comprising: Two or more imaging units are capable of imaging or sensing the same object, wherein, At least one of the two or more imaging units includes a first imaging unit configured to transmit multiple wavelength bands, and At least one of the two or more imaging units, in addition to the first imaging unit, includes a second filter capable of changing the wavelength band. The type or features of the object are estimated by using the image captured by the first imaging unit, and The band of the second filter is set to a band determined based on information about a wavelength suitable for imaging the estimated object, the wavelength being the wavelength corresponding to the color to be interpolated for improving image quality.

2. The imaging device according to claim 1, wherein, The bandwidth of the second imaging unit is narrower than that of the first imaging unit.

3. The imaging device according to claim 1, wherein, The second filter is a Fabry-Perot spectrometer.

4. The imaging device according to claim 3, wherein, The Fabry-Perot spectrometer is formed by a microelectromechanical system that can be driven by voltage.

5. The imaging apparatus according to claim 1, wherein, The first filter is a color filter, and The band of the second filter is set to the band of the color used for interpolating the color obtained by the color filter.

6. The imaging apparatus according to claim 1, wherein, The first imaging unit has four or more bands, and Optical control using plasma resonance is employed for the four or more bands.

7. The imaging apparatus according to claim 1, wherein, An organic photoelectric conversion film is used in at least one of the two or more imaging units.

8. The imaging apparatus according to claim 1, wherein, The color information obtained by the second imaging unit is used to correct the image captured by the first imaging unit.

9. The imaging apparatus according to claim 1, wherein, The distribution of color information of the image captured by the second imaging unit is compared with the distribution of color information of the image captured by the first imaging unit, and a correction amount is set to correct the image captured by the second imaging unit.

10. The imaging apparatus according to claim 1, wherein, Moving object correction is performed on the results of the second imaging unit using the output obtained from the first imaging unit.

11. The imaging apparatus according to claim 1, wherein, Object analysis and state analysis are performed using the results obtained from the two or more imaging units.

12. The imaging apparatus according to claim 11, wherein, Analysis is performed on the results obtained from any one of the two or more imaging units, or on the results obtained by synthesizing the results obtained from the two or more imaging units, via another electronic device at the communication destination.

13. The imaging apparatus according to claim 12, wherein, The band of the second filter is specified by the other electronic device.

14. The imaging apparatus according to claim 1, wherein, The first filter operates in the visible light frequency band, and The second filter has a wavelength band of ultraviolet light, visible light, or infrared light.

15. The imaging apparatus according to claim 1, wherein, The imaging device is included in any device that provides augmented reality, virtual reality, and mixed reality.

16. The imaging apparatus according to claim 1, wherein, The first imaging unit and the second imaging unit perform imaging synchronously with each other.

17. An imaging device, comprising: Two or more imaging units are capable of imaging or sensing the same object, wherein, At least one of the two or more imaging units includes a first imaging unit configured to transmit multiple wavelength bands, and At least one of the two or more imaging units, in addition to the first imaging unit, includes a second filter capable of changing the wavelength band. The type or characteristics of the object are estimated using a synthesized image obtained by combining an image captured by the first imaging unit and an image captured by the second imaging unit, and The band of the second filter is set to a band determined based on information about a wavelength suitable for imaging the estimated object, the wavelength being the wavelength corresponding to the color to be interpolated for improving image quality.

18. An imaging method, wherein, The imaging device includes: Two or more imaging units are capable of imaging or sensing the same object, wherein, At least one of the two or more imaging units includes a first imaging unit configured to transmit multiple wavelength bands, and At least one second imaging unit, other than the first imaging unit, in one of the two or more imaging units includes a second filter capable of changing the spectral band, and the imaging method includes: The type or features of the object are estimated by using the image captured by the first imaging unit, wherein, The band of the second filter is set to a band determined based on information about a wavelength suitable for imaging the estimated object, the wavelength being the wavelength corresponding to the color to be interpolated for improving image quality.

19. An electronic device comprising: Imaging device, including: Two or more imaging units are capable of imaging or sensing the same object, wherein, At least one of the two or more imaging units includes a first imaging unit configured to transmit multiple wavelength bands, and At least one second imaging unit, in addition to the first imaging unit, in one of the two or more imaging units includes a second filter capable of changing the wavelength; and The processing unit is configured to estimate the type or characteristics of the object by using an image captured by the first imaging unit, and to set the band of the second filter to a band determined based on information about a wavelength suitable for imaging the estimated object, the wavelength suitable for imaging the estimated object being a wavelength corresponding to a color to be interpolated for improving image quality.

Images