Optical apparatus and imaging method
The electrically switchable array filter adjusts the refractive index under low light conditions and captures color and monochromatic images, solving the problem of insufficient light capture capability of image sensors in low light, improving the quality and frame rate of images and videos. A higher signal-to-noise ratio and fewer image defects are achieved.
Patent Information
- Application Number
- CN202410548640.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-09-06
- Filing Date
- 2024-05-06
- Publication Date
- 2024-11-12
AI Technical Summary
Existing image sensors lack light capture capabilities in low light conditions, resulting in low signal-to-noise ratio (SNR) and poor image quality. The commonly used Bayer filters reduce light sensitivity, limiting the shooting of low-light photos and videos. application.
An electrically switchable array filter is used, which consists of alternating polymer layers and polymer dispersed liquid crystal layers. The refractive index contrast can be adjusted by applying an electric field, thereby acting as a strong reflector at zero voltage, capturing color images at greater than At zero voltage, monochromatic images are captured and SNR is improved by combining RGB and monochromatic images.
Improve image and video capture quality and frame rate in low-light conditions, significantly improve SNR for optical filters and CMOS sensors, reduce blur and artifacts, and is faster and more stable in combination with computational photography methods method.
Smart Images

Figure CN118938537A_ABST
Abstract
Description
[0001] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 465,807, entitled “Electrically switchable optical materials and methods of forming the same,” filed in the U.S. Patent and Trademark Office on May 11, 2023, and U.S. Patent Application No. 18 / 243,093, filed in the U.S. Patent and Trademark Office on September 6, 2023, the disclosures of which are incorporated herein by reference in their entirety. Technical Field
[0002] Apparatus and methods related to the disclosure relate to the field of imaging devices, and more particularly to imaging sensors. Background Art
[0003] Image sensors are used in cameras and other imaging devices to capture images. In a typical imaging device, light enters through an opening (hole) at one end of the imaging device and is directed to the image sensor through an optical element (such as a lens). In most imaging devices, one or more layers of optical elements are arranged between the aperture and the image sensor to focus the light onto the image sensor. The image sensor includes pixels that generate signals via the optical elements when light is received. Commonly used image sensors include CCD (charge coupled device) image sensors and CMOS (complementary metal oxide semiconductor) sensors.
[0004] Filters are commonly used in image sensors to selectively transmit specific wavelengths of light onto pixels. A Bayer filter mosaic is commonly formed on an image sensor. A Bayer filter is a color filter array that places one of the RGB color filters on each of the detector pixels. The Bayer filter pattern includes a 50% green filter, a 25% red filter, and a 25% blue filter. Since each pixel generates a signal representing the intensity of a color component in the incident light (rather than a full range of colors), a demosaicing process is performed to insert a set of red, green, and blue values for each image pixel.
[0005] Image sensors are subject to various performance limitations. The performance limitations of image sensors include dynamic range, signal-to-noise ratio (SNR), and low light sensitivity. Dynamic range is defined as the ratio of the maximum possible signal that can be captured by a pixel to the total noise signal. Typically, the well capacity of the image sensor limits the maximum possible signal that can be captured by the image sensor. The maximum possible signal in turn depends on the intensity of the incident light and the exposure duration (e.g., integration time and shutter width).
[0006] The signal-to-noise ratio (SNR) of a captured image is largely a measure of image quality. Generally, the SNR is higher as the more light a pixel captures. The SNR of a captured image is generally related to the light gathering ability of the pixel.
[0007] Typically, Bayer filter sensors reduce the light sensitivity of the imaging system. At low light levels, the light gathering ability of each pixel is limited by the low signal level incident on each pixel. In addition, the color filter on the pixel also attenuates the signal level reaching the pixel. The IR (infrared) filter also reduces the light response from near IR signals that can carry valuable information.
[0008] Various methods for improving low-light photo and low-light video capture have been proposed to increase light capture, SNR, and image quality even at low signal levels. Within the smartphone market, camera performance has become a key metric. Even reasonable differences in image quality between smartphone models and brands can influence consumer purchasing decisions.
[0009] Due to the small size of smartphone image sensors, the amount of light they can capture is limited, limiting their ability to be used in low-light photo and low-light video applications. To compensate for this, the exposure time can be increased or multiple noisy images can be stacked. However, this often results in blurring and / or other image artifacts and makes the overall frame rate too slow for low-light video capture. As such, there is a continuing need for filters that maximize light capture at low light levels while also providing color information.
[0010] Although the photosensitivity of CMOS sensors has improved, this has repeatedly proven to be a complex, slow and expensive process. Another approach that has been used is to integrate larger CMOS sensors and lenses, however, there are strong limitations on the size of the final device. Finally, it has also been proposed to apply computational photography methods (such as capturing multiple lower quality images and combining them). However, this process is slow and susceptible to image defects.
[0011] The information disclosed in this background technology section is known to the inventor or derived by the inventor before or during the process of implementing the embodiments of the present application, or is technical information acquired during the process of implementing the embodiments. Therefore, it may include information that does not constitute prior art known to the public. Summary of the invention
[0012] Disclosed herein are apparatus and methods that are intended to address shortcomings in the art and may also provide additional or alternative advantages.
[0013] (A1) In one embodiment, an optical device is disclosed herein. The optical device includes: a filter including an electrically switchable array, wherein the electrically switchable array includes: alternating polymer layers and polymer dispersed liquid crystal layers, wherein the electrically switchable array is adjustable under an applied electric field.
[0014] (A2) An optical device according to (A1), wherein the alternating polymer layers and the polymer dispersed liquid crystal layers have a significantly different refractive index variation, wherein the refractive index variation is a distributed Bragg reflector.
[0015] (A3) An optical device according to (A2), wherein: (a) at zero applied voltage, the filter produces a high refractive index contrast between the polymer layer and the polymer dispersed liquid crystal layer, wherein, at zero applied voltage, the filter is strongly reflective within a wavelength range determined by the structural period of the alternating layers; and (b) when a voltage greater than zero is applied to the filter, the refractive index contrast is suppressed, wherein, at the voltage greater than zero, the filter is transparent within the wavelength range previously reflected.
[0016] (A4) The optical device according to (A3), wherein the filter is configured to: capture a color image when the filter is turned off; and capture a monochrome image when the filter is turned on.
[0017] (A5) An optical device according to (A1), wherein the process for manufacturing the filter comprises: exposing a uniform liquid mixture comprising a holographic polymer and a liquid crystal to a light source to generate a periodic light field, generating a liquid-liquid phase separation of a polymer dispersed liquid crystal from the periodic light field, and simultaneously generating crosslinking of the polymer layer.
[0018] (A6) An optical device according to (A1), wherein the process of manufacturing the filter includes: using spatially incoherent light to generate a standing wave pattern in a uniform liquid mixture, wherein the uniform liquid mixture is a precursor mixture including a monomer, a photoinitiator and a liquid crystal.
[0019] (B1) An imaging method, the method comprising: imaging an initial scene in a first frame using a Bayer filter to receive RGB values; imaging the initial scene in a second frame with the Bayer filter disconnected to receive light intensity values; converting the RGB values of the first frame into Y'CrCb values; removing the Y channel in the Y'CrCb frame to form a CrCb frame; subsampling the CrCb frame to form a subframe; and combining the CrCb subframe and the second frame to reconstruct the initial scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Example embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:
[0021] Figure 1A Depicted is a process for fabricating an electrically switchable array by exposure to projected light according to an example embodiment of the present disclosure.
[0022] Figure 1B Depicted are the intensity distribution in a standing wave, the approximate spatial refractive index variation, and the predicted reflection spectrum according to example embodiments of the present disclosure.
[0023] Figure 1C Is Figure 1B Approximate spatial refractive index distribution of the nanoscale refractive index variation of the standing wave patterned optical structure depicted in FIG.
[0024] Figure 1D Shown with Figure 1C The predicted reflectance spectrum of the optical structure having the refractive index distribution shown in Figure 1B Light field with light intensity distribution depicted in the same row.
[0025] FIG. 2A to FIG. 2D Depicted are an off and on electrically switchable optical filter and the corresponding transmission spectra according to example embodiments of the present disclosure.
[0026] Figure 3A A conventional image stacking method according to an example embodiment of the present disclosure is depicted.
[0027] Figure 3B An embodiment of an image stacking method of an electrically switchable filter according to an example embodiment of the present disclosure is depicted.
[0028] Figure 4 Depicted is an RGB image and a monochrome image for combination into a single image according to an example embodiment of the present disclosure.
[0029] FIG. 5A to FIG. 5B Depicting sensitivity curves for color images and monochrome images according to example embodiments of the present disclosure.
[0030] Figure 6 Depicted is the spectrum of a D65 light source and the improvement in photon capture that can be achieved with a monochrome image (rather than an image acquired through a Bayer filter) according to an example embodiment of the present disclosure.
[0031] Figure 7 Depicted is the spectrum of a 450 nm light source and the improvement in photon capture that can be achieved with a monochromatic image (rather than an image acquired through a Bayer filter) according to an example embodiment of the present disclosure.
[0032] Figure 8 A data processing and image reconstruction method for constructing a color image from a low quality / low SNR Bayer filtered color picture and a high SNR monochrome image is depicted according to an example embodiment of the present disclosure.
[0033] Fig. 9 is a functional block diagram of an imaging device according to an example embodiment of the present disclosure.
[0034] Fig.10 is a flowchart of an imaging method according to an example embodiment of the present disclosure. DETAILED DESCRIPTION
[0035] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of some example embodiments of systems and methods for image capture according to the present disclosure, and is not intended to represent the only form of constructing or using the present disclosure. The description sets forth the features of the present disclosure in conjunction with the illustrated embodiments. However, it should be understood that the same or equivalent functions and structures may be implemented by different embodiments that are also intended to be covered within the scope of the disclosure. As shown elsewhere herein, the same element numbers are intended to represent the same elements or features.
[0036] Among other things, the description herein discloses an RGB image sensor that captures more light by integrating a color Bayer filter that is structured to be electrically switched on and off. This enables recording a color image and a monochrome image pair that can be recombined in a low-light level capture scene into a monochrome image with a higher SNR than can be achieved with a conventional static Bayer filter. This will improve the quality and frame rate of low-light images and low-light video capture.
[0037] Among other things, as will be discussed in more detail, the advantages of our approach may be: a significant improvement in the SNR of the arrangement of optical filters and CMOS sensors, rather than a complex, slower and more expensive process of modification of the underlying CMOS materials. This is an approach that can be combined with computational photography methods, but is faster and less prone to image defects due to the higher SNR and the need for fewer images.
[0038] Described herein in greater detail are the design and fabrication of a color Bayer filter structured to be electrically switched on and off to capture color and monochrome image pairs. Also described herein are methods of combining the captured image pairs into a monochrome image. In some embodiments, the method provides an improvement of more than 50% in image acquisition time, reduces blur and artifacts, and allows for low-light video capture compared to prior methods.
[0039] Figure 1A Depicted is a process for fabricating an electrically switchable array by exposure to projected light according to an example embodiment of the present disclosure.
[0040] The precursor mixture 101 includes a monomer, a photoinitiator, and a liquid crystal. The photoinitiator can adjust the polymerization of the monomer and / or the liquid crystal when exposed to photons (e.g., light). For example, in some embodiments, a method of manufacturing a filter includes: exposing a uniform liquid mixture including a holographic polymer and a liquid crystal to a light source, generating a periodic light field, generating a liquid-liquid phase separation of a polymer dispersed liquid crystal from the periodic light field, and simultaneously generating crosslinking of the polymer layer, thereby generating alternating polymer layers and polymer dispersed liquid crystal layers.
[0041] As will be described in further detail, electrically switchable array 104 (also referred to herein as filter 104, optical array 104, optical structure 104, or filter structure 104) is fabricated by placing precursor mixture 101 on reflective surface 111 (also referred to herein as reflective layer 111) (with prism 102 above precursor mixture 101) and exposing precursor mixture 101 to incident light 103 (e.g., using a standard light projector). In some embodiments, a light source or source of electromagnetic radiation is located near the precursor mixture. For example, Figure 1A As shown in , a source of incident light 103 (eg, electromagnetic radiation 103 ) is positioned such that precursor mixture 101 is located between prism 102 and reflective layer 111 , and is directed through prism 102 to illuminate precursor mixture 101 at a particular angle.
[0042] When the incident light 103 is reflected by a specific part of the precursor mixture 101, an electromagnetic radiation standing wave (not shown) is generated. In the high intensity region of the standing light wave, the precursor mixture 101 is photopolymerized so that the liquid crystal phase and the polymerized monomer of the precursor mixture 101 form periodically changing nanoscale refractive index variations, and the period is defined by the standing light wave. This transforms the initial precursor mixture 101 into a nanostructured volume of two chemically different alternating layers, one type of which is rich in liquid crystal phase (107a, 107b) and the other type of which is rich in polymer (109a, 109b, 109c). These alternating polymer dispersed liquid crystal layers (107a, 107b) differ from the polymer layers in their average chemical composition and therefore have different refractive indices. Therefore, the structure acts as a distributed Bragg reflector and can also be formed in the form of a 2D or 3D optical architecture.
[0043] The generated alternating polymer dispersed liquid crystal regions 107a to 107b and the generated alternating polymer layers 109a to 109c are different from each other and have different refractive indices.
[0044] The above-mentioned refractive index variants may have any suitable size or arrangement. In some embodiments, the refractive index variants include arrays (e.g., 1-D arrays, 2-D arrays, 3-D arrays) with periodic and / or repeating features. The refractive index variants may have regular or periodic characteristic dimensions (e.g., the spacing between each variant, the period of the refractive index variant). In some embodiments, the spacing between the refractive index variants is less than or equal to 1 μm, less than or equal to 900 nm, less than or equal to 800 nm, less than or equal to 750 nm, less than or equal to 700 nm, less than or equal to 650 nm, less than or equal to 600 nm, less than or equal to 550 nm, less than or equal to 500 nm, less than or equal to 400 nm, less than or equal to 300 nm, less than or equal to 200 nm, or less than or equal to 100 nm. In some embodiments, the spacing between refractive index variations is greater than or equal to 100 nm, greater than or equal to 200 nm, greater than or equal to 300 nm, greater than or equal to 400 nm, greater than or equal to 500 nm, greater than or equal to 550 nm, greater than or equal to 600 nm, greater than or equal to 650 nm, greater than or equal to 700 nm, greater than or equal to 750 nm, greater than or equal to 800 nm, or greater than or equal to 900 nm. Combinations of the foregoing ranges are also possible (e.g., less than or equal to 1 μm and greater than or equal to 100 nm). Other ranges are possible as the present disclosure is not limited to the description provided herein.
[0045] In some embodiments, two or more groups of refractive index variations (e.g., nanoscale refractive index variations) may be present within the precursor mixture 101 (e.g., within the liquid crystal region of the precursor mixture). For example, the liquid crystal region of the precursor mixture 101 may include a first group of refractive index variations having a first characteristic size (e.g., within the first sublayer of the liquid crystal region 107a) and a second group of refractive index variations (e.g., within the second sublayer of the liquid crystal region 107b). In some embodiments, the liquid crystal region includes a plurality of refractive index variations. In some embodiments, the liquid crystal region includes 2, 3, 4, 5, 6, 7, 8, 9, or 10 groups of refractive index variations. In some embodiments, the liquid crystal region includes 20, 50, 100, or 1000 groups of refractive index variations, each of which may occupy a sublayer of the liquid crystal region (e.g., the 20th sublayer, the 50th sublayer, the 100th sublayer, the 1000th sublayer).
[0046] In some embodiments, a particular amount of incident light is reflected by the formed optical structure 104. In some embodiments, at least 50%, at least 60%, at least 70%, at least 80%, at least 85%, at least 90%, or at least 95% of the incident light is reflected by the formed optical structure 104. In some embodiments, no more than 95%, no more than 90%, no more than 85%, no more than 80%, or no more than 70% of the incident light is reflected by the formed optical structure 104. Combinations of the foregoing ranges are also possible (at least 50% and no more than 95%). Other ranges are possible. In some embodiments, the reflected light has a particular wavelength or has a maximum at a particular wavelength.
[0047] like Figure 1A As shown in , the electrically switchable array 104 may include a first polymer layer 109a, a second polymer layer 109b and a third polymer layer 109c, wherein a first polymer dispersed liquid crystal region 107a is embedded between the first polymer layer 109a and the second polymer layer 109b, and a second polymer dispersed liquid crystal region 107b is embedded between the second polymer layer 109b and the third polymer layer 109c.
[0048] As described above, in some embodiments, Figure 2C and Figure 2D As shown in , the polymer dispersed liquid crystal layer embedded in the optical array 104 can cause a shift in the position and amplitude of the spectral reflection peak due to an applied voltage.
[0049] The electrical response of the electrically switchable material is partly due to the optical properties of the polymer dispersed liquid crystal region. The molecules (i.e., liquid crystal elements) in the polymer dispersed liquid crystal region are rod-shaped. The optical properties of the liquid crystal phase depend on the direction of the rod-shaped molecules. The direction can be changed by applying an electric field that changes the optical properties (e.g., refractive index) of the liquid crystal phase. In some embodiments, the electrically switchable optical array 104 described herein can be integrated on a conventional CMOS sensor in a camera to capture images with improved SNR at low light levels. In other embodiments, the electrically switchable array 104 is used as existing hardware for the camera or as an additional element or "replacement" element of the camera. In this way, the electrically switchable array 104 can be designed to be compatible with conventional image sensors, although the resolution achieved after the integrated electrically switchable array 104 exceeds that of conventional image sensors. With improved performance, the electrically switchable array 104 according to an embodiment of the invention can include fewer pixels to obtain images of equal or better quality than conventional image sensors. Optionally, the size of the captured pixels can be reduced compared to pixels in conventional image sensors, while achieving better results.
[0050] The optical manufacturing time of the electrically switchable optical array 104 can be between seconds and minutes, depending on the lighting conditions and the size of the exposure area in which the image is recorded. Since the exposure time is inversely proportional to the exposure level, it is expected that the exposure time will be reduced to the sub-second level using a high-power projection system. Various technical benefits that will be shown in more detail are that the parameters that determine the optical properties of the final electrically switchable array 104 are easier to control.
[0051] First, during fabrication of the array from the precursor mixture 101, the spatial color distribution of the incident light defines the spatial distribution of the periodicity of the refractive index variations in the resulting filter structure 104 (also referred to herein as the electrically switchable array 104). Figure 1B , in the predicted reflectance spectrum diagram, a high peak reflectance of up to 90% of the incident light in the primary color region is shown matching the red, green, and blue channels of the incident light.
[0052] A key parameter may be the structure of the liquid crystal domains, which determines the electrical adjustability induced by an applied voltage.
[0053] A person skilled in the art will not choose the combination presented here. For example, in holography, spatially coherent light is usually necessary. Regarding Lippman photography, the method usually works with a smooth mirror surface (for example, Lippman uses a pool of liquid mercury to get the smoothest possible surface).
[0054] In some embodiments of the present invention, spatially incoherent light 103 is combined with a rough back reflector 111 (or reflective layer 111) to generate a standing wave pattern in the precursor mixture 101. It is not known whether this has been done before, and such a combination is not easily inferred by one skilled in the art. In addition, the materials described herein are beyond the scope of what exists with respect to optical control, electrical control, and achievable dynamics.
[0055] Various embodiments described herein may include refractive index variations within the optical array 104. Without wishing to be bound by any particular theory, one or more surfaces of a refractive index variation (or periodic variation) may cause partial reflection of an incident light wave, and for light waves with wavelengths relatively close to the size of the variation, the reflections may combine by constructive interference, such that the refractive index variation (or periodic variation) functions as a distributed Bragg reflector (DBR). The refractive index variation may function as an optical filter by reflecting specific light waves and not reflecting other specific light waves. As described in more detail below, the wavelength of light that is reflected or not reflected may be adjusted by the spacing of the refractive index variations. In other embodiments, they may also function as optical waveguides.
[0056] Figure 1BDepicted are the intensity distribution in a standing wave used to generate the optical array 104, the resulting approximate spatial refractive index variation in the optical array 104, and the predicted reflection spectrum according to an example embodiment of the present disclosure.
[0057] When incident light 103 passes through precursor mixture 101 and is reflected by reflective layer 111 (i.e., reflected light 105), a standing wave of electromagnetic radiation is formed. The intensity distribution in the standing wave is as follows: Figure 1B The standing wave of electromagnetic radiation (i.e., photons of light from the standing wave) interacts (e.g., causes a reaction) within the precursor mixture 101, causing it to Figure 1C The precursor mixture 101 (described above) is optically patterned with the nanoscale refractive index variations shown in FIG. In some embodiments, the precursor mixture 101 is polymerized (e.g., by electromagnetic radiation emitted from an electromagnetic radiation source) to form a Figure 1A , shown in FIG. 1 , includes a polymer matrix (eg, optical array 104 ) that includes nanoscale refractive index variations. Figure 1D Shown with Figure 1C The predicted reflectance spectrum of the optical structure having the refractive index distribution shown in Figure 1B The light intensity distribution is depicted in the same row in the light field.
[0058] FIG. 2A to FIG. 2D Depicts the process of switching an electrically switchable filter off and on.
[0059] Specifically, Figure 2A Depicted in "disconnected" mode Figure 1A The electrically switchable array 104 described in FIG. 1 is a liquid crystal array having a plurality of electrically switchable arrays 104. When no charge passes through the array, the rod-shaped molecules (e.g., Figure 2A 120a, 120b) in the polymer dispersed liquid crystal region may be randomly arranged, and the refractive index (Δn) in the polymer dispersed liquid crystal region may be very high (about 1.63, given by (n e +n o ) / 2 estimate, where n e =1.72, and n o =1.53). The refractive index in the polymer phase (109) may be about 1.5. This results in an effective refractive index difference between the layer containing the liquid crystal phase and the polymer layer of about 0.1 to 0.15. Figure 2C As shown in the corresponding graph in , when no voltage is applied, the optical array 104 reflects the maximum amount of light in the spectral band defined by the period and refractive index contrast of the optical structure. Figure 2C As shown in , the optical array 104 transmits the minimum amount of light in this spectral band to the underlying detector (T% represents transmittance). In other words, when no voltage is applied (e.g., Figure 2C V offstate), the optical array 104 can be used as a conventional Bayer filter, that is, the Bayer filter is "switched on".
[0060] Figure 2B Depicts the "on" mode Figure 1A The electrically switchable optical array 104 described in FIG. 1 is a liquid crystal array having a plurality of electrically switchable optical arrays. When an electric field is applied to the array, the rod-shaped molecules (e.g., Figure 2B 122a, 122b) in the figure are aligned with the field; the vertically incident light now experiences a refractive index of n o =1.53, which greatly reduces the refractive index difference with the adjacent polymer layer and suppresses the reflection band of the optical structure. Figure 2D As shown in , the optical array (104) now transmits all incident light to the detector below. In some embodiments, the rod-shaped molecules exhibit an "ordered arrangement" in the "on" mode. In other words, when a voltage greater than zero is applied (e.g., Figure 2D V on state), the optical array 104 can be used as a monochromatic filter, that is, the Bayer filter is "off".
[0061] Figure 3A A conventional image stacking method according to an example embodiment of the present disclosure is depicted. Five RGB images 302a to 302e are depicted, specifically, a first image 302a, a second image 302b, a third image 302c, a fourth image 302d, and a fifth image 302e. Although five images are shown, several images (e.g., more than five) may be captured in a conventional method for image acquisition. Capturing several RGB images and combining them may be cumbersome and time consuming. This conventional method may result in blurring and / or image artifacts, as well as making the overall frame rate too slow for low-light video. Methods are needed to increase the amount of light transmitted by the camera filter and the amount of light that can be captured by the sensor. Although the method described herein is not limited to low-light photography and low-light video, it is useful under such conditions.
[0062] Figure 3BA first embodiment of an image stacking method for an electrically switchable filter according to an example embodiment of the present disclosure is depicted. By integrating the electrically switchable filter 104 in a conventional RGB image sensor, the RGB image sensor can capture more light by switching between "RGB filter array when the electric field is off" and "transparent element when the electric field is on." When the electrically switchable array is off (i.e., the electrically switchable array is used as a Bayer filter), the RGB image sensor can capture an RGB image for true color data. When the electrically switchable array is on (i.e., the electrically switchable array is used as a monochrome filter and the Bayer filter is off), the RGB image sensor can capture a monochrome image for brightness information with increased SNR. In this way, the RGB image 304 can be combined with the monochrome image 306 to generate a final image with color information and improved SNR. In Figure 8 The process and corresponding algorithm for combining image 304 and image 306 are discussed in more detail in FIG. Figure 3B The embodiments depicted in can reduce image acquisition time by more than 50%.
[0063] Figure 4 Depicts an RGB image and a monochrome image for combining into a single image according to an example embodiment of the present disclosure. A series of pixels may form an RGB image 400. For example, the RGB image 400 may include blue pixels 404a, green pixels 404b, and red pixels 404c, and the blue pixels 404a, green pixels 404b, and red pixels 404b in addition to a series of other pixels may be combined to form the RGB image 400. The monochrome image 402 may include only brightness information 402, which includes a series of white pixels (e.g., white pixels 406) and a series of black pixels (e.g., black pixels 408). In this way, the method described herein for forming a final frame may only require one RGB image and one monochrome image.
[0064] FIG. 5A to FIG. 5B Plotting sensitivity curves for a CMOS sensor with a Bayer RGB filter (e.g., when an electric field is not applied) and a CMOS sensor without a Bayer RGB filter (e.g., when an electric field is applied). As shown in the figure, much of the incident light intensity never reaches the CMOS sensor with the Bayer filter because the filter absorbs the incident light intensity. As previously described, employing the methods described herein that provide adjustability of "turning the filter off and capturing an RGB image and turning the filter on and capturing a monochrome image" can provide maximum light capture capabilities for shorter acquisition times.
[0065] Figure 6The spectrum of a D65 illuminant and the ratio of light collected from the D65 illuminant by a sensor with a Bayer RGB filter and by a monochrome sensor (without a Bayer RGB filter) are depicted. According to an example embodiment of the present disclosure, the monochrome sensor collects 2.4 times more light than the sensor with a Bayer filter. For the D65 illuminant, the ratio between the intensity detected by the monochrome sensor and the sensor with the Bayer filter is given by:
[0066]
[0067] Figure 7 Depicts the spectrum of a 450nm LED light source and the ratio of light collected from this LED by a sensor with a Bayer RGB filter and by a monochrome sensor without a Bayer RGB filter. According to an example embodiment of the present disclosure, for this illumination spectrum, the monochrome sensor collects 3.8 times more light than the sensor with the Bayer filter. For a 450nm LED, the ratio between the intensity detected by the monochrome sensor and the sensor with the Bayer filter is given by:
[0068]
[0069] Figure 8 Depicting the process of image formation according to an example embodiment of the present disclosure.For example, activation of exposure periods of imagers in a camera may be staggered in an overlapping manner so that the scene is sampled sequentially.
[0070] In this method, a scene can be captured by the imager in a first image (e.g., image 802) when the electrically switchable array 104 is not subjected to an electric field. Image 802 can provide RGB values (e.g., red image 804a, green image 804b, and blue image 804c). The same scene as that captured in image 802 can also be captured with the monochrome filter turned on (i.e., the electrically switchable array 104 is subjected to an electric field) (e.g., image 803) to receive the light intensity value in frame 805.
[0071] The RGB values captured by the first image 802 (e.g., red image 804a, green image 804b, and blue image 804c) may be used to generate an image in the Y'CrCb color space (i.e., a Y'CrCb frame). Next, the Y channel in the Y'CrCb frame may be removed to form a Cr frame 806a and a Cb frame 806b. The Cr frame 806a and the Cb frame 806b may be subsampled to generate a denoised Cr frame 808a and a denoised Cb frame 708b.
[0072] Finally, Cr frame 808a and Cb frame 808b may be combined with frame 805 (e.g., light intensity values) to reconstruct the original scene captured by the imager in image 802 and image 803 to generate a final image 810 having a higher SNR than a standard RGB image 802 acquired using a standard Bayer filter.
[0073] Fig. 9 is a functional block diagram of an imaging device according to an example embodiment of the present disclosure.
[0074] The imaging system 900 may include, among other components, a camera array 910 (e.g., an electrically switchable optical filter array 104 integrated on a CMOS sensor), an image processing pipeline module (or image processing module) 920, and a controller 940. The camera array 910 includes imagers. An image 912 is captured by the imagers in the camera array 910.
[0075] Controller 940 is hardware, software, firmware, or a combination thereof for controlling various operating parameters of camera array 910. Controller 940 receives input 946 from a user or other external components and sends operating signals 942 to control camera array 910. Controller 940 may also send information 944 to image processing pipeline module 920 to assist in processing of image 912.
[0076] The image processing pipeline module 920 is hardware, firmware, software, or a combination thereof for processing images received from the camera array 910. For example, the image processing pipeline module 920 processes a plurality of images 912. The processed images 922 are then sent for display, storage, transmission, or further processing.
[0077] The image processing pipeline module 920 may include, among other components, an upstream pipeline processing module, an image pixel association module, a disparity confirmation and measurement module, a disparity compensation module, a super-resolution module, an address conversion module, an address and phase offset calibration module, and a downstream color processing module. The address and phase offset calibration module is a storage device for storing calibration data generated during camera array characterization in a manufacturing process or a subsequent recalibration process. In several embodiments, the calibration data may indicate a mapping between addresses of physical pixels in the imager and logical addresses of the image. In other embodiments, various calibration data suitable for a particular application may be utilized in the address and phase offset calibration module.
[0078] The address conversion module performs normalization based on the calibration data stored in the address and phase offset calibration module. Specifically, the address conversion module converts the "physical" address of each pixel in the image into the "logical" address of each pixel in the imager, and vice versa. In order to perform super-resolution processing to generate an image with enhanced resolution, it is necessary to resolve the phase difference between the corresponding pixels in each imager. Super-resolution processing may assume that for each pixel in the resulting image, a set of input pixels from each imager are mapped consistently, and it is assumed that the phase offset of the image captured by each imager is known to the position of the pixel in the resulting image. Optionally, the pixel offset can be estimated before super-resolution processing. The address conversion module resolves such phase differences by converting the physical address in the image into the logical address of the resulting image for subsequent processing.
[0079] The image captured by the imager is provided to an upstream pipeline processing module. The upstream pipeline processing module may perform one or more of normalization of color planes, dark level calculation and adjustment, fixed noise compensation, optical PSF (point spread function) deconvolution, noise reduction, lateral color correction, and crosstalk reduction.
[0080] In one embodiment, the upstream pipeline processing module also performs temperature normalization. Temperature normalization corrects for changes in the refractive index of the optical components through which the imager receives light caused by temperature changes of the camera during use. In several embodiments, the temperature normalization process involves determining the temperature of the camera array by measuring the dark current of one of the imagers of the plurality of camera arrays or an average thereof. Using this measurement, refractive index normalization is performed by selecting the correct point spread function from the temperature calibration data. Different point spread functions can be obtained during manufacturing during the temperature-affected refractive index shaping of the camera and stored in the imaging system for use in the temperature normalization process.
[0081] Fig.10 is a flowchart of an imaging method according to an example embodiment of the present disclosure.
[0082] Reference Fig.10First, an initial scene may be imaged in a first frame using a Bayer filter to receive RGB values (1002). For example, the initial scene may be imaged in the first frame when no electric field is applied to the electrically switchable array. In this case, the electrically switchable array may be used as a Bayer filter. Then, the initial scene may be imaged in a second frame with the Bayer filter turned off to receive light intensity values (1004). For example, the initial scene may be imaged in the second frame when an electric field is applied to the electrically switchable array. In this case, the electrically switchable array may be used as a monochrome filter. Next, the first frame in a first color space may be converted into a third frame in a second color space. The first color space may be an RGB space, and the second color space may be a YCrCb space. For example, the RGB values of the first frame may be converted into Y'CrCb values to obtain a Y'CrCb frame (1006). Then, the luminance channel in the third frame may be removed to obtain a fourth frame having chrominance channels. For example, the Y channel in the Y'CrCb frame may be removed to form a CrCb frame (1008). Then, the fourth frame may be subsampled to form a fifth frame having chroma channels. For example, the CrCb frame may be subsampled to form a CrCb subframe (1010). Finally, the fifth frame may be combined with the second frame to reconstruct the initial scene. For example, the CrCb subframe and the second frame may be combined to reconstruct the initial scene (1012).
[0083] It will be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish an element, a component, a region, a layer or a part from another element, another component, another region, another layer or another part. Therefore, without departing from the scope of the present disclosure, the first element, the first component, the first region, the first layer or the first part discussed herein may be referred to as the second element, the second component, the second region, the second layer or the second part.
[0084] For ease of description, spatially relative terms such as "below", "below", "lower", "under...", "above", "upper", etc. may be used herein to describe the relationship of one element or feature to another element or feature as shown in the figure. It will be understood that such spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation shown in the figure. For example, if the device in the accompanying drawings is turned over, the elements described as "below" or "below" or "under other elements" will be oriented to be "above" other elements or features. Therefore, the example terms "below" and "under..." may include both "above..." and "under..." orientations. The device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein should be interpreted accordingly. In addition, it will be understood that when a layer is referred to as being "between" two layers, it may be the only layer between the two layers, or one or more intermediate layers may also be present.
[0085] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used herein, the terms "substantially," "approximately," and similar terms are used as terms of approximation rather than terms of degree, and are intended to account for the inherent deviations in measurements or calculations that one of ordinary skill in the art would recognize.
[0086] As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that when used in this specification, the terms "include" and / or "comprise" illustrate the existence of the features, wholes, steps, operations, elements and / or components of the description, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. When a statement such as "at least one of..." is after a column of elements, the statement modifies the entire column of elements without modifying the individual elements in the column. In addition, when describing an embodiment of the present disclosure, the use of "may" represents "one or more embodiments of the present disclosure". In addition, the term "exemplary" is intended to represent an example or description. As used herein, the term "use" may be considered to be synonymous with the term "utilize".
[0087] It will be understood that when an element or layer is referred to as being “on,” “connected to,” “coupled to,” or “adjacent to” another element or layer, it may be directly on, connected to, coupled to, or adjacent to the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly on,” “directly connected to,” “directly coupled to,” or “immediately adjacent to” another element or layer, there are no intervening elements or layers.
[0088] Among other things, the advantages of the apparatus and methods described herein are improvements in the photosensitivity of CMOS sensors, rather than complex, slow and expensive processes for improving the underlying materials. In some embodiments, this is combined with computational photography methods, but is faster and less prone to image artifacts because fewer images are required.
[0089] The embodiments described herein are all example embodiments, so that the present disclosure is not limited thereto, and can be implemented in various other forms.Each embodiment provided in the above description does not exclude being associated with another example or another embodiment also provided herein or another example or another embodiment not provided herein but one or more features consistent with the disclosure or another embodiment.For example, unless otherwise mentioned in its description, even if the content described in a specific example or embodiment is not described in an example or embodiment different from it, these contents may also be understood to be related to or combined with different examples or embodiments.In addition, it should be understood that all descriptions of the disclosed principles, aspects, examples and embodiments are intended to cover their structural and functional equivalents.In addition, these equivalents should be understood to include not only currently known equivalents, but also equivalents to be developed in the future (that is, all devices invented to perform the same function, regardless of their structure).
[0090] As used herein, when a statement such as "at least one of..." follows a list of elements, the statement modifies the entire list of elements and does not modify the individual elements in the list. For example, the statement "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c. Herein, when the term "same" is used to compare the size of two or more elements, the term can cover "substantially the same" sizes.
[0091] It will be understood that although the terms first, second, third, fourth, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Therefore, without departing from the disclosed teachings, the first element discussed above may be referred to as the second element.
[0092] It should also be understood that even if a particular step or operation of manufacturing a device or structure is described later than other steps or operations, unless the other steps or operations are described as being performed after the step or operation, the step or operation may be performed later than the other steps or operations.
[0093] Many embodiments are described herein with reference to cross-sectional views as schematic diagrams of embodiments (and intermediate structures). As such, variations in the illustrated shapes due to, for example, manufacturing techniques and / or tolerances will be expected. Therefore, the embodiments should not be construed as limited to the specific shapes of the regions shown herein, but will include shape deviations, for example, caused by manufacturing. Therefore, the regions shown in the figures are schematic in nature, and their shapes are not intended to illustrate the actual shapes of regions of the device, and are not intended to limit the scope of the inventive concept. In addition, in the accompanying drawings, the sizes and relative sizes of layers and regions may be exaggerated for clarity.
[0094] For the sake of brevity, conventional elements, structures, or layers of an optical device may or may not be described in detail herein.
Claims
1. An optical device, include: Optical filters, including electrically switchable arrays, The electrically switchable array comprises alternating polymer layers and polymer dispersed liquid crystal layers, Among them, the electrically switchable array can be adjusted under the applied electric field.
2. The optical device according to claim 1, in, The alternating polymer layers and polymer dispersed liquid crystal layers have different refractive indices, and the alternating polymer layers and polymer dispersed liquid crystal layers form a refractive index variation, wherein the refractive index variation serves as a distributed Bragg reflector.
3. The optical device according to claim 1, in, In the absence of an applied voltage, the optical filter exhibits a refractive index contrast between the polymer layer and the polymer dispersed liquid crystal layer, so that light within a wavelength range determined by the periodic structure of the polymer layer and the polymer dispersed liquid crystal layer is reflected by the optical filter; and When a voltage greater than zero is applied to the filter, the refractive index contrast is suppressed, so that light in the wavelength range is transmitted by the filter.
4. The optical device according to claim 1, in, The filters are configured as: capturing a color image when no electric field is applied to the electrically switchable array; and A monochrome image is captured when an electric field is applied to the electrically switchable array.
5. A method for manufacturing an optical filter, the method comprising: include: exposing a homogeneous liquid mixture including a holographic polymer and a liquid crystal to a light source; generating a periodic light field; Liquid-liquid phase separation of polymer dispersed liquid crystals is generated from the periodic light field, and cross-linking of the polymer layers is simultaneously generated, thereby generating alternating polymer layers and polymer dispersed liquid crystal layers.
6. The method according to claim 5, in, The step of exposing the liquid mixture to a light source comprises: A standing wave pattern is generated in a homogeneous liquid mixture using spatially incoherent light, wherein the homogeneous liquid mixture is a precursor mixture including a monomer, a photoinitiator, and a liquid crystal.
7. The method according to claim 5, in, The alternating polymer layers and polymer dispersed liquid crystal layers have different refractive indices, and the alternating polymer layers and polymer dispersed liquid crystal layers form a refractive index variation, wherein the refractive index variation serves as a distributed Bragg reflector.
8. An imaging method, the method comprising: include: Imaging an initial scene in a first frame using a Bayer filter to receive RGB values; imaging the initial scene in a second frame with the Bayer filter turned off to receive light intensity values; Converting a first frame in a first color space into a third frame in a second color space, wherein the first color space is an RGB space and the second color space is a YCrCb space; The luminance channel in the third frame is removed to obtain a fourth frame having chrominance channels; subsampling the fourth frame to obtain a fifth frame having chroma channels; The fifth frame is combined with the second frame to reconstruct the original scene.
9. The imaging method according to claim 8, in, The first frame is a color image and the second frame is a monochrome image.
10. The imaging method according to claim 8, in, The Bayer filter comprises an electrically switchable array comprising alternating polymer layers and polymer dispersed liquid crystal layers having different refractive indices, wherein the alternating polymer layers and polymer dispersed liquid crystal layers form a refractive index variation and the refractive index variation acts as a distributed Bragg reflector.
11. The imaging method according to claim 8, in, The Bayer filter comprises an electrically switchable array, wherein an initial scene is imaged in a first frame without an electric field applied to the electrically switchable array, Therein, the initial scene is imaged in a second frame when an electric field is applied to the electrically switchable array.
