Multispectral optical sensor and system
By combining multispectral optical sensors and processing resources, the problem of insufficient color consistency in scene images under different ambient lighting conditions is solved, and more accurate image color adjustment is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AMS SENSORS GERMANY GMBH
- Filing Date
- 2021-10-06
- Publication Date
- 2026-06-26
Smart Images
Figure CN116457641B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a multispectral optical sensor, a multispectral optical system including a multispectral optical sensor, an image sensing system including a multispectral optical system, and a method using a multispectral optical sensor, specifically but not exclusively for adjusting captured images of a scene in response to the effects of ambient lighting on different parts of the scene. Background Technology
[0002] Color constancy is a desirable property of image sensing devices, such as cameras. Color constancy refers to the ability of a feature or object to appear with a relatively constant color under different lighting conditions. In other words, the appearance of an image captured by a camera can be affected by ambient lighting.
[0003] As an example, if the color temperature of the ambient light source is relatively low, such as in the 3000 Kelvin range, as in the case of incandescent light, the image of a white object exposed to the ambient light will include a slightly reddish tint. In contrast, for an ambient light source with a high color temperature, such as in the 6000 Kelvin range, as in the case of cloudy daylight, the image of a white object will include a slight blue tint. That is, the object will be perceived by the camera as including a color that depends on the ambient light source's illumination of the object.
[0004] It is known that this effect can be compensated for by measuring scene-related spectral information using a multispectral ambient light sensor (ALS). For example, firstly, referencing... Figure 1A The image shows a smartphone 1, which includes a multispectral ALS arrangement 3, a camera 4, and a cover glass 8 covering the multispectral ALS arrangement 3 and the camera 4, wherein the multispectral ALS arrangement 3 is configured to measure the spectral distribution of light incident from the scene onto the camera 4. Figure 1B A detailed cross-sectional view of the multispectral ALS arrangement 3 and camera 4 is shown. The multispectral ALS arrangement 3 includes a multispectral ALS sensor 2 with multiple optical detector regions 11. The multispectral ALS 2 is configured such that each optical detector region 11 detects a different range of wavelengths, for example, because the multispectral ALS 2 includes multiple different optical filters (…). Figure 1B (Not explicitly shown), where each optical filter is configured to transmit only the corresponding wavelength of a different range to the corresponding one in the optical detector region 11. For clarity, Figure 1B Only three optical detector regions 11 are shown in the diagram. However, those skilled in the art will understand that the multispectral ALS sensor 2 may have more than three optical detector regions 11 or fewer than three optical detector regions 11.
[0005] The multispectral ALS arrangement 3 includes a housing 20 that accommodates the multispectral ALS sensor 2. The multispectral ALS arrangement 3 also includes a diffuser 30 and an IR cutoff filter 32 located between the cover glass 8 and the housing 20.
[0006] The housing 20 defines an aperture or window 22 to allow light to enter the housing 20 via the cover glass 8, diffuser 30, and IR cutoff filter 32. The multispectral ALS arrangement 3 has an optical axis 40 perpendicular to the front surface of the multispectral ALS 2. Furthermore, as those skilled in the art will understand, the use of diffuser 30 provides a field of view (FOV) 42 for the multispectral ALS arrangement 3, which defines a large solid angle around the optical axis 40. Each optical detector region 11 detects different wavelength ranges incident on the optical detector region 11 from all different incident directions across the entire FOV 42 of the multispectral ALS arrangement 3.
[0007] Camera 4 has an optical axis 50, which is perpendicular to the front surface of the image sensor (not shown) of camera 4 and parallel to the optical axis 40 of multispectral ALS arrangement 3. Camera 4 has a field of view (FOV) 52, which defines a solid angle around the optical axis 50 of camera 4, wherein the solid angle of the FOV 52 of camera 4 is equal to or smaller than the solid angle of the FOV 42 of multispectral ALS arrangement 3.
[0008] Smartphone 1 uses white balance, and preferably automatic white balance (AWB), to adjust the coloration of images captured under different lighting conditions. For example, smartphone 1 may have predefined settings for typical lighting conditions (such as daylight, fluorescent lighting, or incandescent lighting), where in some cases, the predefined settings can be automatically selected.
[0009] Existing techniques for white balance include image processing using algorithms based on the "grey world theory" or "white patch theory." The gray world theory is based on the assumption that the average reflectance in a captured image is achromatic. That is, the average values of the three color channels—red, green, and blue—should be approximately equal. The white patch theory is based on the assumption that the brightest pixel in a captured image corresponds to the reflection from ambient light, and therefore the brightest pixel can correspond to the spectrum of ambient lighting. Both methods have known limitations, and notably, both tend to produce substantially different results. Therefore, it is desirable to be able to correct the captured image of a scene for the influence of ambient lighting on the scene without incurring the drawbacks of existing AWB methods.
[0010] Furthermore, different parts of a scene may experience different ambient lighting conditions. For example, depending on the corresponding ambient lighting conditions of different parts of a uniformly colored object, different parts of a uniformly colored object in a scene may even appear different. Therefore, it is desirable to be able to correct the captured image of the scene for the effects of different ambient lighting conditions on different parts of the scene without causing the drawbacks of existing AWB methods. Summary of the Invention
[0011] According to one aspect of this disclosure, a multispectral optical sensor is provided, comprising:
[0012] A monolithic semiconductor chip, comprising multiple subarrays defining an optical detector region;
[0013] Multiple optical filters; and
[0014] Multiple lens elements,
[0015] Each subarray of the optical detector region includes multiple corresponding optical detector regions.
[0016] Each subarray of the optical detector region has the same relative spatial arrangement of the optical detector regions as each other subarray of the optical detector region, and
[0017] Each optical filter is aligned between a corresponding subarray of the corresponding lens element and the corresponding optical detector region, such that light incident on any of the lens elements along the incident direction is converged by the corresponding optical filter to a corresponding optical detector region of the corresponding subarray of the optical detector region, the corresponding optical detector region depending on the incident direction.
[0018] This multispectral optical sensor can be used to measure spectral information associated with different parts or sectors of a scene captured by an image sensor or camera. This allows for the use of gradient white balance to adjust the coloration of scene images, for example, to more accurately reproduce the scene image as perceived by a human observer. This multispectral optical sensor can be particularly useful when different parts of a scene are illuminated by different ambient light sources.
[0019] Since light detected by each subarray of the optical detector region is transmitted through a corresponding optical filter, each subarray of the optical detector region can be considered as a monochromatic subarray of the optical detector region. The multispectral optical sensor can be fabricated, at least partially, using on-chip integration to achieve wafer-level packaging. The spatial arrangement of the optical detector regions, optical filters, and lens elements of each subarray defines the sectoration of the field of view of the multispectral optical sensor. Due to the symmetrical design of the monochromatic subarrays and corresponding filters and lens elements, each corresponding optical detector region of the different subarrays of the optical detector region detects light from the same sector of the scene, which is necessary for colorimetric analysis. Such a multispectral optical sensor can be used to generate sectorized color and spectral information for each different region of the scene (e.g., the center of the scene, the boundaries of the scene, and the outer regions of the scene). The sectorized color and spectral information can be used to achieve gradient white balance of the captured image of the scene relative to different ambient light conditions in the same scene.
[0020] Multiple subarrays in the optical detector region can be arranged as 1D or 2D arrays of subarrays, such as uniform 1D or 2D arrays of subarrays.
[0021] Each subarray of the optical detector region has its own optical filter. Optical filters can be processed, for example, formed or deposited, on all optical detector regions of the same subarray. Therefore, the spacing between optical detector regions within the same subarray is limited only by the design rules of a monolithic semiconductor chip.
[0022] Multiple optical detector regions in each subarray of the optical detector region can be arranged as a 1D or 2D array of optical detector regions, such as a uniform 1D or 2D array of optical detector regions.
[0023] Each subarray of the optical detector region may include a central optical detector region and one or more peripheral optical detector regions arranged around the central optical detector region.
[0024] One or more of the peripheral optical detector regions may be arc-shaped and may be arranged circumferentially around the central optical detector region.
[0025] One or more of the peripheral optical detector regions may be annular in shape and may be arranged concentrically with the central optical detector region.
[0026] Multiple optical filters can be set or formed on the front surface of a monolithic semiconductor chip.
[0027] Multiple lens elements may include microlens arrays (MLAs) or microFresnel lens arrays.
[0028] Multiple lens elements can be defined by an optical substrate or formed on an optical substrate.
[0029] Multispectral optical sensors may include spacers located between a monolithic semiconductor chip and an optical substrate.
[0030] Monolithic semiconductor chips and optical substrates can be attached to spacers.
[0031] The spacer can define multiple holes, each of which is aligned with a corresponding subarray of a corresponding lens element, corresponding optical filter, and optical detector region.
[0032] Spacers can define one or more opaque separators or opaque walls, where each separator or wall separates two adjacent apertures. Such spacers can block optical crosstalk between different subarrays in the optical detector region.
[0033] Holes can be formed by at least one of vertical etching, deep lithography, or injection molding.
[0034] Spacers may include or be formed of opaque materials.
[0035] The spacer may be made of or formed of a plastic material, such as a thermosetting polymer or a thermoplastic polymer.
[0036] Each optical filter may include an optical interference filter or an optical absorption filter.
[0037] A multispectral optical sensor may include multiple transmissive optical elements. Each transmissive optical element may be aligned between a corresponding lens element and a corresponding optical filter such that light incident on any of the lens elements is converged through the corresponding transmissive optical element and the corresponding optical filter onto one of the optical detector regions of a corresponding subarray of the optical detector region. Each transmissive optical element may receive converged light propagating along an initial propagation direction from the corresponding lens element and may convert the received converged light into transmitted converged light that propagates away from the transmissive optical element along a final propagation direction parallel to the optical axis of the corresponding optical filter, or the final propagation direction relative to the optical axis of the corresponding optical filter may define a smaller angle than the initial propagation direction of the received converged light.
[0038] Using multiple such transmissive optical elements can ensure that convergent light received by any one of the transmissive optical elements along an initial propagation direction inclined relative to the optical axis of the corresponding optical filter is converted by the transmissive optical element to propagate towards the corresponding optical filter along a direction parallel to the optical axis of the corresponding optical filter, or along a direction defined at a smaller angle relative to the optical axis of the corresponding optical filter than the initial propagation direction of the received convergent light. This can be advantageous when the optical transmission spectrum of the optical filter depends on the angle of incidence of the light incident on the optical filter (e.g., when the optical filter is an interferometric filter), ensuring that the light received by the optical filter undergoes the known fixed optical transmission spectrum of the optical filter, regardless of the initial propagation direction along which the convergent light is received by the corresponding transmissive optical element.
[0039] Multiple transmission optical elements may include multiple additional lens elements.
[0040] Multiple additional lens elements may include microlens arrays (MLAs) or microFresnel lens arrays.
[0041] Multiple transmission optical elements may be defined by an additional optical substrate or formed on an additional optical substrate.
[0042] An additional optical substrate can be attached to the front surface of a monolithic semiconductor chip.
[0043] The spacer can be attached to the front surface of the additional optical substrate.
[0044] Each transmission optical element can be defined by or formed on a corresponding optical filter.
[0045] Each of the multiple optical filters can have a corresponding optical transmission spectrum, such as a passband optical transmission spectrum.
[0046] The passband optical transmission spectra of multiple optical filters can span a predefined wavelength range. The difference between the first sum of the optical transmission values of the multiple optical filters at a first wavelength within the predefined wavelength range and the second sum of the optical transmission values of the multiple optical filters at a second wavelength within the predefined wavelength range can be less than a predetermined threshold.
[0047] The first sum of optical transmittance values can be equal to the second sum of optical transmittance values.
[0048] The sum of the optical transmission values of multiple optical filters can be the same at all wavelengths within a predefined wavelength range.
[0049] At least three optical transmission spectra from the optical filters can be selected for trichromatic stimulus detection.
[0050] The optical transmission spectra of at least three of the optical filters can correspond to the corresponding coordinates in the CIE color space.
[0051] The optical transmission spectra of at least three of the optical filters can correspond to the corresponding components of the XYZ color space.
[0052] Monolithic semiconductor chips can include CCD and / or CMOS monolithic semiconductor chips.
[0053] Each optical detector region may include CCD and / or CMOS optical detector regions.
[0054] According to one aspect of this disclosure, a multispectral optical system is provided, comprising:
[0055] The multispectral optical sensor as described above; and
[0056] Resource management
[0057] The multispectral optical sensors and processing resources are configured to communicate with each other.
[0058] The processing resources are configured to associate different electrical signals generated by different optical detector regions of the same subarray of the optical detector region with light incident on the multispectral optical sensor from the scene along the corresponding different incident directions, and to associate different electrical signals generated by corresponding optical detector regions of different subarrays of the optical detector region with light incident on the multispectral optical sensor from the scene along the same incident direction.
[0059] Different electrical signal values measured by corresponding optical detector regions of different subarrays of the optical detector region represent the spectrum of light incident on the multispectral optical sensor from the scene along the same incident direction associated with the corresponding optical detector regions of different subarrays of the optical detector region.
[0060] The processing resources can be configured to correlate the electrical signal generated by the optical detector region with the optical transmission spectrum of the corresponding optical filter.
[0061] The processing resources can be configured to determine the classification of ambient light sources for each of a number of different incident directions based on a comparison between the electrical signal value corresponding to each incident direction and predefined spectral data.
[0062] Predefined spectral data can include multiple discrete spectra, each corresponding to a different known type or species of ambient light source.
[0063] The processing resources can be configured to adjust the electrical signal values generated by different optical detector regions of each subarray of the optical detector region to compensate for any differences in the optical transmission spectrum of the corresponding optical filter caused by the convergent light propagating through the corresponding optical filter in different propagation directions along different optical detector regions of the same subarray of the optical detector region. In cases where the optical transmission spectrum of the optical filter depends on the incident angle of the light incident on the optical filter, such as when the optical filter is an interferometer filter, compensating for any differences in the optical transmission spectrum of the corresponding optical filter in this way can be advantageous.
[0064] According to one aspect of this disclosure, an image sensing system is provided, comprising:
[0065] The multispectral optical system as described above; and
[0066] Image sensors with known spatial relationships relative to multispectral optical sensors.
[0067] The image sensor and processing resources are configured to communicate with each other, and
[0068] The processing resources are configured to adapt to the images sensed by the image sensor based on the classification of ambient light sources for each incident direction.
[0069] The processing resources can be configured to adapt the image to white balance using one or more parameters based on the ambient light source classification for each incident direction.
[0070] The processing resources can be configured to adapt the image by performing gradient white balance on the image based on one or more parameters classified according to the ambient light source for each incident direction.
[0071] The processing resources can be configured to determine the ambient light source classification for each direction by identifying the closest match between the electrical signal value corresponding to each incident direction and predefined spectral data.
[0072] The processing resources can be configured to reconstruct the spectrum of the ambient light source for each incident direction from the electrical signal values corresponding to each incident direction.
[0073] Ambient light sources can be classified by color temperature or color coordinates.
[0074] According to one aspect of this disclosure, a monolithic semiconductor chip for a multispectral optical sensor is provided, wherein the monolithic semiconductor chip defines a plurality of subarrays of an optical detector region, wherein each subarray of the optical detector region includes a corresponding plurality of optical detector regions, and wherein the optical detector regions of each subarray of the optical detector region have the same spatial arrangement as the optical detector regions of each other subarray of the optical detector region.
[0075] A single semiconductor chip may include multiple optical filters, each of which is positioned in front of a corresponding subarray in the optical detector region.
[0076] Multiple optical filters can be set or formed on the front surface of a monolithic semiconductor chip.
[0077] According to one aspect of this disclosure, an electronic device is provided, comprising at least one of the following: the monolithic semiconductor chip, the multispectral optical sensor, the multispectral optical system, or the image sensing system described above.
[0078] Electronic devices may include mobile electronic devices, such as mobile phones, cellular phones, smartphones, tablet computers, or laptop computers.
[0079] According to one aspect of this disclosure, a method for using a multispectral optical sensor as described above is provided, the method comprising:
[0080] Correlating different electrical signals generated by different optical detector regions of the same subarray of the optical detector region with light incident on the multispectral optical sensor from the scene along corresponding different incident directions; and
[0081] Different electrical signals generated by corresponding optical detector regions of different subarrays of the optical detector region are associated with light incident on the multispectral optical sensor from the scene along the same incident direction.
[0082] The method may include associating an electrical signal generated by each optical detector region of a multispectral optical sensor with the optical transmission spectrum of a corresponding optical filter.
[0083] The electrical signal values measured by the corresponding optical detector regions of different subarrays of the optical detector region represent the spectrum of light incident on the multispectral optical sensor from the scene along the same incident direction associated with the corresponding optical detector regions of different subarrays of the optical detector region.
[0084] The method may include determining the classification of ambient light sources in each of a plurality of different directions based on a comparison between electrical signal values corresponding to each incident direction and predefined spectral data.
[0085] Predefined spectral data can include multiple discrete spectra, each corresponding to a different type or kind of ambient light source.
[0086] The method may include:
[0087] Images are sensed using an image sensor that has a known spatial relationship with respect to a multispectral optical sensor; and
[0088] The sensing image is adapted based on the classification of ambient light sources from each incident direction.
[0089] This method may include adapting the image by performing white balance on the image using one or more parameters based on the classification of ambient light sources for each incident direction. Alternatively, this method may include adapting the image by performing gradient white balance on the image using one or more parameters based on the classification of ambient light sources for each incident direction.
[0090] The method may include determining the ambient light source classification for each incident direction by identifying the closest match between the electrical signal value corresponding to each incident direction and predefined spectral data.
[0091] The method may include reconstructing the spectrum of the ambient light source for each incident direction from electrical signal values corresponding to each incident direction.
[0092] Ambient light sources can be classified by color temperature or color coordinates.
[0093] It should be understood that any one or more features of any of the foregoing aspects of this disclosure may be combined with any one or more features of any of the other foregoing aspects of this disclosure. Attached Figure Description
[0094] Multispectral optical sensors, multispectral optical systems, image sensing systems, and associated methods will now be described by way of non-limiting example only, with reference to the accompanying drawings, in which:
[0095] Figure 1A This is a schematic diagram of the rear side of a prior art electronic device in the form of a prior art smartphone, which has a prior art multispectral ambient light sensor (ALS) arrangement and a camera.
[0096] Figure 1B yes Figure 1A A schematic cross-section of existing technologies in smartphones, including multispectral ALS and cameras;
[0097] Figure 2A This is a schematic diagram of the rear of an electronic device in the form of a smartphone, which has a multispectral ALS array and a camera.
[0098] Figure 2B yes Figure 2A A schematic cross-section of the multispectral ALS arrangement and camera of a smartphone.
[0099] Figure 3 yes Figure 2A A schematic diagram of a multispectral ALS array;
[0100] Figure 4A yes Figure 3A schematic diagram of a monolithic multispectral ALS semiconductor chip for a multispectral ambient light sensor (ALS).
[0101] Figure 4B This is a schematic diagram of an alternative monolithic multispectral ALS semiconductor chip;
[0102] Figure 5 The operation of an image sensing system is illustrated, the image sensing system including... Figure 2A Multispectral ALS setup and camera for smartphones;
[0103] Figure 6A This is a schematic diagram of the first alternative multispectral ALS; and
[0104] Figure 6B This is a schematic diagram of the second alternative multispectral ALS. Detailed Implementation
[0105] First refer to Figure 2A The image shows a smartphone 101, which includes a multispectral optical sensor arrangement in the form of a multispectral ALS arrangement 103, a camera 104 having a known spatial relationship with respect to the ALS arrangement 103, and a cover glass 108 covering the multispectral ALS arrangement 103 and the camera 104.
[0106] Figure 2B A detailed cross-sectional view of the multispectral ALS arrangement 103 and camera 104 is shown. The multispectral ALS arrangement 103 includes a multispectral ALS 102 with multiple optical detector regions 111. For clarity, Figure 2B Only three optical detector regions 111 are shown in the diagram. However, as will be described in more detail below, the multispectral ALS 102 actually defines more than three optical detector regions 111.
[0107] The multispectral ALS arrangement 103 includes a housing 120 for accommodating a multispectral ALS 102. The multispectral ALS arrangement 103 also includes an IR cutoff filter 132 located between a cover glass 108 and the housing 120. The housing 120 defines an aperture or window 122 for allowing light to enter the housing 120 via the cover glass 108 and the IR cutoff filter 132. The multispectral ALS arrangement 103 has an optical axis 140 perpendicular to the front surface of the multispectral ALS 102.
[0108] As described below, the multispectral ALS arrangement 103 is configured to distinguish light incident on the multispectral ALS arrangement 103 from the scene along different incident directions, and to measure the spectral distribution of the light incident on the multispectral ALS arrangement 103 with respect to different incident directions on a field of view (FOV) 142, which defines a solid angle around the optical axis 140 of the multispectral ALS arrangement 103. Specifically, the multispectral ALS arrangement 103 is configured to distinguish light incident on the multispectral ALS arrangement 103 from different sectors 142a, 142b, ... 142i of the FOV 142, and to measure the spectral distribution of light incident on the multispectral ALS arrangement 103 from each sector 142a, 142b, ... 142i. The camera 104 also has an optical axis 150, which is perpendicular to the front surface of the image sensor chip (not shown) of the camera 104 and parallel to the optical axis 140 of the multispectral ALS arrangement 103. Camera 104 has a field of view (FOV) 152, which defines a solid angle around the optical axis 150 of camera 104, wherein the solid angle of the FOV 152 of camera 104 is comparable to the solid angle of the FOV 142 of multispectral ALS arrangement 103.
[0109] like Figure 3 As shown, the multispectral ALS 102 includes Figure 4A The monolithic multispectral ALS semiconductor chip 110 is shown in more detail below. The monolithic multispectral ALS semiconductor chip 110 defines a plurality of subarrays 112 defining optical detector regions, in the form of twelve subarrays 112 arranged in a 3×4 array, wherein the optical detector regions of each subarray 112 have the same relative spatial arrangement as the optical detector regions of each of the other subarrays 112. Specifically, each of the subarrays 112 defines a 3×3 array of optical detector regions 111a, 111b, 111c, ..., 111i.
[0110] The monolithic multispectral ALS semiconductor chip 110 includes a plurality of optical filters 160, each optical filter 160 having a corresponding optical transmission spectrum. Each optical filter 160 is a passband optical interference filter, which defines a corresponding spectral passband. Two or more of the optical filters 160 may define different spectral passbands. Furthermore, each optical filter 160 is formed on or attached to the monolithic multispectral ALS semiconductor chip 110 in front of a corresponding subarray 112 of optical detector regions 111a, 111b, 111c, ..., 111i.
[0111] The multispectral ALS 102 also includes a plurality of lens elements 162 in the form of a microlens array (MLA) defined or formed on the optical substrate 164. The multispectral ALS 102 also includes a spacer 166 located between the monolithic semiconductor chip 110 and the optical substrate 164 of the MLA. The monolithic semiconductor chip 110 and the optical substrate 164 are attached to opposite sides of the spacer 166. Furthermore, the spacer 166 defines a plurality of apertures 168, wherein each aperture 168 is aligned with a corresponding subarray 112 of a corresponding lens element 162, a corresponding optical filter 160, and an optical detector region 111a, 111b, 111c, ... 111i.
[0112] Each optical filter 160 is aligned between the corresponding lens element 162 and the corresponding subarray 112 of the optical detector regions 111a, 111b, 111c, ... 111i, such that in use, any light incident on any of the lens elements 162 along any given incident direction is converged by the corresponding optical filter 160 to one of the corresponding optical detector regions 111a, 111b, 111c, ... 111i of the corresponding subarray 112 of the optical detector regions 111a, 111b, 111c, ... 111i, which depends on the given incident direction. For example, light incident on any of the lens elements 162 along an incident direction parallel to the optical axis 140 of the multispectral ALS 102 (represented by the solid lines in Figure 4) is focused by the lens element 162 through the corresponding optical filter 160 onto the central optical detector region 111e of the corresponding subarray 112. Similarly, light incident on any of the lens elements 162 along an incident direction inclined relative to the optical axis 140 of the multispectral ALS 102 (represented by the dashed or dotted lines in Figure 4) is focused by the lens element 162 through the corresponding optical filter 160 onto one of the peripheral optical detector regions 111a, 111b, 111c, 111d, 111f, 111g, 111h, 111i of the corresponding subarray 112, depending on the specific incident direction.
[0113] Return to reference Figure 2A The smartphone 101 includes processing resources 180, which are configured to receive data from the image sensors (not shown) of the multispectral ALS 102 and the camera 104. Figure 5As shown, the processing resource 180 is configured to associate different electrical signals generated by different optical detector regions 111a, 111b, 111c, ... 111i of the same subarray 112 of the optical detector region with light incident on the multispectral ALS 102 from different regions 183a, 183b, 183c, ... 183i of the scene, which is generally designated as 182, along corresponding different incident directions 184a, 184b, 184c, ... 184i.
[0114] Conversely, processing resource 180 is configured to associate different electrical signals generated by corresponding optical detector regions of different subarrays 112 with light incident on the multispectral ALS 102 from the same region of scene 182 along the same incident direction. For example, processing resource 180 is configured to associate different electrical signals generated by corresponding optical detector regions 111a of different subarrays 112 with light incident on the multispectral ALS 102 from region 183a of scene 182 along the same incident direction 184a.
[0115] Furthermore, processing resource 180 is configured to associate electrical signals generated by any optical detector region 111a, 111b, 111c, ... 111i of any given subarray 112 with the optical transmission spectrum of the corresponding optical filter 160. Since each optical filter 160 has a different passband, different electrical signal values measured by corresponding optical detector regions of different subarrays 112 represent the spectrum of light incident on the multispectral ALS 102 from scene 182 along the same incident direction associated with the corresponding optical detector region of the different subarray 112. For example, different electrical signal values measured by corresponding optical detector region 111a of different subarrays 112 represent the spectrum of light incident on the multispectral ALS 102 from scene 182 along the same incident direction 184a associated with the corresponding optical detector region 111a of the different subarray 112.
[0116] Processing resource 180 is configured to determine the classification of ambient light sources for each of the multiple different incident directions 184a, 184b, 184c, ... 184i based on a comparison between electrical signal values corresponding to each incident direction 184a, 184b, 184c, ... 184i and predefined spectral data. The predefined spectral data may, for example, include multiple discrete spectra, each corresponding to a different known type or category of ambient light source.
[0117] Furthermore, the processing resource 180 is configured to adapt the image sensed by the camera 104 based on the ambient light source classification for each incident direction 184a, 184b, 184c, ..., 184i. Specifically, the processing resource 180 is configured to adapt the sensed image by performing white balance on the image using one or more parameters based on the ambient light source classification for each direction, for example, by performing gradient white balance on the image using one or more parameters based on the ambient light source classification for each incident direction.
[0118] Those skilled in the art will understand that various modifications to the above-described multispectral ALS 102 are possible. For example, Figure 6A A first alternative multispectral ALS 202 for use with multispectral ALS arrangement 103 is shown. Similar to... Figure 3 Multispectral ALS 102, Figure 6A The first alternative multispectral ALS 202 includes a monolithic multispectral ALS semiconductor chip 210, which is identical to the monolithic multispectral ALS semiconductor chip 110 of the multispectral ALS 102. Figure 3 Similar to the multispectral ALS 102, Figure 6A The first alternative multispectral ALS 202 also includes a plurality of passband optical interference filters 260, wherein each optical filter 260 is formed on or attached to a monolithic multispectral ALS semiconductor chip 210 in front of a corresponding subarray 212 of optical detector regions 211a, 211b, 211c, ... 211i.
[0119] The multispectral ALS 202 also includes a plurality of lens elements 262 in the form of a microlens array (MLA) defined or formed on the optical substrate 264. The multispectral ALS 202 also includes a spacer 266 located between the monolithic semiconductor chip 210 and the optical substrate 264 of the MLA. Furthermore, the spacer 266 defines a plurality of apertures 268, wherein each aperture 268 is aligned with a corresponding subarray 212 of a corresponding lens element 262, a corresponding optical filter 260, and optical detector regions 211a, 211b, 211c, ... 211i.
[0120] However, with Figure 3 The multispectral ALS 102 is different. Figure 6A The first alternative multispectral ALS 202 includes multiple transmission optical elements in the form of multiple additional lens elements 290, provided as an additional microlens array (MLA) on an additional optical substrate 292. The additional optical substrate 292 is attached to the front surface of a monolithic multispectral ALS semiconductor chip 210. The rear surface of an optical substrate 264 is attached to the front side of a spacer 266, and the front surface of the additional optical substrate 292 is attached to the rear side of the spacer 266.
[0121] Each additional lens element 290 is aligned between a corresponding lens element 262 and a corresponding optical filter 260 such that light incident on any of the lens elements 262 is converged by the corresponding additional lens element 290 and the corresponding optical filter 260 onto one of the optical detector regions 211a, 211b, 211c, ..., 211i of the corresponding subarray 212 of the optical detector region. Each additional lens element 290 receives converged light from the corresponding lens element 262 propagating along an initial propagation direction and converts the received converged light into transmitted converged light that propagates away from the additional lens element 290 along a final propagation direction parallel to the optical axis of the corresponding optical filter 260, or with a smaller angle relative to the optical axis of the corresponding optical filter 260 than the initial propagation direction of the received converged light.
[0122] The use of this additional microlens array (MLA) can be used to ensure that converged light received by any of the additional lens elements 290 along an initial propagation direction inclined relative to the optical axis of the corresponding optical filter 260 is converted by the additional lens element 290 to propagate towards the corresponding optical filter 260 along a direction parallel to the optical axis of the corresponding optical filter 260 or in a direction with an angle smaller than the initial propagation direction of the received converged light relative to the optical axis of the corresponding optical filter 260. This can be advantageous when the optical transmission spectrum of the optical filter 260 depends on the incident angle of the light incident on the optical filter 260, for example, when the optical filter 260 is an interference filter, to ensure that the light received by the optical filter 260 undergoes the known optical transmission spectrum of the optical filter 260, regardless of the initial propagation direction along which the converged light is received by the corresponding additional lens element 290.
[0123] exist Figure 6A In a first alternative multispectral ALS 202 variant, each additional lens element may be defined by or formed on the corresponding optical filter 260.
[0124] Figure 6B A second alternative multispectral ALS 302 for use with the multispectral ALS arrangement 103 is shown. In addition... Figure 6B The second alternative multispectral ALS 302 includes multiple transmission optical elements in the form of multiple Fresnel lens elements 390 provided as a micro Fresnel lens array, instead of including multiple transmission optical elements in the form of multiple additional lens elements 290 provided as an additional microlens array (MLA) on an additional optical substrate 292. The second alternative multispectral ALS 302 is identical in all respects to... Figure 6AThe first alternative multispectral ALS 202 is the same, wherein each Fresnel lens element 390 is defined by or formed on the corresponding optical filter 360 of the multispectral ALS 302.
[0125] exist Figure 6B In a second alternative variant of the multispectral ALS 302, each Fresnel lens element 390 may be defined by or formed on an additional optical substrate, wherein the additional optical substrate is attached to the front surface of the monolithic multispectral ALS semiconductor chip 310 of the multispectral ALS 302.
[0126] Those skilled in the art will understand that, Figure 6A The first alternative multispectral ALS 202 and Figure 6B In the second alternative multispectral ALS 302, multiple transmission optical elements effectively mean that the light is focused and propagates through each optical interference filter in a direction parallel to or nearly parallel to the optical axis of the optical interference filter, thus ensuring that the transmitted light undergoes the known optical transmission spectrum of the optical interference filter under normal incidence. This is achieved by using multiple transmission optical elements (such as...) Figure 6A The first alternative to the multispectral ALS 202 transmission optical element 290 or Figure 6B As an alternative to the second alternative multispectral ALS 302 (transmission optical element 390), the processing resources 180 of the smartphone 101 can be configured to adjust the electrical signal values generated by different optical detector regions 111a, 111b, 111c, ... 111i of the same subarray 112 of the optical detector region to compensate for any differences in the optical transmission spectrum of the corresponding optical filter 160 caused by the convergent light propagating through the corresponding optical filter 160 in different propagation directions along the different optical detector regions 111a, 111b, 111c, ... 111i of the same subarray 112 of the optical detector region.
[0127] Figure 4B It shows the relationship with Figure 2B and Figure 3 Multispectral ALS 102 Figure 6A The first alternative multispectral ALS 202 or Figure 6BAn alternative monolithic multispectral ALS semiconductor chip 410 is used in conjunction with the second alternative multispectral ALS 302. The alternative monolithic multispectral ALS semiconductor chip 410 defines a plurality of subarrays 412 of optical detector regions, in the form of twelve subarrays 412 arranged in a 3×4 array, wherein the optical detector regions of each subarray 412 have the same relative spatial arrangement as the optical detector regions of each of the other subarrays 412. Specifically, each of the subarrays 412 defines a central optical detector region 411a surrounded by four arc-shaped optical detector regions 411b, 411c, 411d, and 411e. The monolithic multispectral ALS semiconductor chip 410 includes a plurality of optical filters 460, each optical filter 460 having a corresponding optical transmission spectrum. Each optical filter 460 may be a passband optical interference filter, defining a corresponding spectral passband. Two or more of the optical filters 460 may define different spectral passbands. In addition, each optical filter 460 is formed on or attached to a monolithic multispectral ALS semiconductor chip 410 in front of a corresponding subarray 412 in the optical detector regions 411a, 411b, 411c, 411d and 411e.
[0128] Those skilled in the art will understand that other arrangements of the optical detector regions are possible within each subarray. For example, each subarray may define a central optical detector region surrounded by one or more concentrically arranged annular optical detector regions, each annular optical detector region having a different radius. Each subarray may define a 1D or 2D array of optical detector regions of any size. The optical detector regions of each subarray may be arranged as a non-rectangular 2D array.
[0129] Those skilled in the art will also understand that other arrangements of the subarrays are possible. For example, the subarrays can be arranged as 1D or 2D arrays of any size. The subarrays can be arranged as non-rectangular 2D patterns.
[0130] Although various multispectral ALS 102, 202, 302 for use with camera 104 of smartphone 101 have been described, it should be understood that any of multispectral ALS 102, 202, 302 can be used with cameras of any kind of electronic device. For example, any of multispectral ALS 102, 202, 302 can be used with cameras of mobile phones, cellular phones, tablet computers, or laptop computers.
[0131] Although this disclosure has been described with reference to preferred embodiments as described above, it should be understood that these embodiments are merely illustrative and the claims are not limited to these embodiments. In view of this disclosure, those skilled in the art will be able to make modifications and substitutions to the described embodiments, which are considered to fall within the scope of the appended claims. Each feature disclosed or shown in this specification may be incorporated into any embodiment, either alone or in any suitable combination with any other feature disclosed or shown herein. Specifically, those skilled in the art will understand that one or more features of the embodiments of this disclosure described above with reference to the accompanying drawings may have an effect or provide an advantage when used separately from one or more other features of the embodiments of this disclosure, and different combinations of features are possible beyond the specific combinations of features of the embodiments of this disclosure described above.
[0132] Those skilled in the art will understand that positional terms such as “above,” “along,” and “side” in the foregoing specification and appended claims are made with reference to conceptual illustrations (such as those shown in the accompanying drawings). These terms are used for ease of reference and are not intended to be restrictive. Therefore, these terms should be understood to refer to objects when they are in the orientation shown in the accompanying drawings.
[0133] When used with respect to features of embodiments of this disclosure, the use of the term "comprising" does not exclude other features or steps. When used with respect to features of embodiments of this disclosure, the use of the terms "a" or "an" does not exclude the possibility that an embodiment may include multiple such features.
[0134] The use of reference numerals in the claims should not be construed as limiting the scope of the claims.
[0135] List of reference numerals
[0136] 1. Smartphone;
[0137] 2. Multispectral ALS sensor;
[0138] 3. Multispectral ALS arrangement;
[0139] 4 cameras;
[0140] 8. Cover glass;
[0141] 11 Optical detector area;
[0142] 20. Housing;
[0143] 22 holes;
[0144] 30 diffusers;
[0145] 32IR cutoff filter;
[0146] The optical axis of a 40-spectral ALS array;
[0147] Field of view of a 42-multispectral ALS array;
[0148] The optical axis of the 50 camera;
[0149] 52 camera field of view;
[0150] 101 smartphone;
[0151] 102 multispectral ALS sensor;
[0152] 103 multispectral ALS arrangement;
[0153] 104 camera;
[0154] 108 cover glass;
[0155] 110 monolithic multispectral ALS semiconductor chip;
[0156] 111 Optical detector area;
[0157] 111a-111i optical detector region;
[0158] A subarray of 112 optical detector regions;
[0159] 120 housing;
[0160] 122 housing bore;
[0161] 132IR cutoff filter;
[0162] The optical axis of the 140 multispectral ALS arrangement;
[0163] Field of view of 142 multispectral ALS array;
[0164] The sectors of the field of view of the 142a-142i multispectral ALS arrangement;
[0165] The optical axis of a 150 camera;
[0166] Field of view of a 152 camera;
[0167] 160 optical filter;
[0168] 162 lens element;
[0169] 164 optical substrate;
[0170] 166 spacers;
[0171] 168 spacer holes;
[0172] 182 scenes;
[0173] The area of the 183a-183i scene;
[0174] 184a-184i incident direction;
[0175] 202 multispectral ALS sensor;
[0176] 210 monolithic multispectral ALS semiconductor chip;
[0177] 211a-211i optical detector region;
[0178] 212 subarrays of optical detector regions;
[0179] 260 optical filter;
[0180] 262 lens element;
[0181] 264 optical substrate;
[0182] 266 spacers;
[0183] 268 spacer hole;
[0184] 290 Additional lens elements;
[0185] 292 Additional optical substrate;
[0186] 302 multispectral ALS sensor;
[0187] 310 monolithic multispectral ALS semiconductor chip;
[0188] 360° optical filter;
[0189] 390 Additional lens elements;
[0190] 410 monolithic multispectral ALS semiconductor chip;
[0191] 411a-411i optical detector region;
[0192] 412 subarrays of optical detector regions; and
[0193] 460 optical filter.
Claims
1. A multispectral optical system, comprising: Multispectral optical sensors, including: A monolithic semiconductor chip, wherein the monolithic semiconductor chip defines a plurality of subarrays of an optical detector region; Multiple optical filters; Multiple lens elements; and Resource management Each subarray of the optical detector region includes multiple corresponding optical detector regions. Each subarray of the optical detector region has the same relative spatial arrangement of the optical detector regions as each other subarray of the optical detector region, and Each optical filter is aligned between a corresponding subarray of a corresponding lens element and an optical detector region, such that light incident from the scene onto each lens element along the incident direction is transmitted by each lens element and converged through the corresponding optical filter onto a corresponding optical detector region within the corresponding subarray of the optical detector region. The corresponding optical detector region depends on the incident direction, such that the corresponding optical detector region of each subarray of the optical detector region detects light incident on the multispectral optical sensor along the same incident direction. The processing resources are configured to associate different electrical signals generated by different optical detector regions of the same subarray of the optical detector region with light incident on the multispectral optical sensor from the scene along corresponding different incident directions, and to associate different electrical signals generated by corresponding optical detector regions of different subarrays of the optical detector region with light incident on the multispectral optical sensor from the scene along the same incident direction. The processing resources are configured to adjust the electrical signal values generated by the different optical detector regions of the same subarray of the optical detector region to compensate for any differences in the optical transmission spectra of the corresponding optical filters caused by the convergent light propagating through the corresponding optical filters along different propagation directions of the different optical detector regions of the same subarray of the optical detector region. The plurality of lens elements include microlens arrays or microFresnel lens arrays. The multispectral optical system includes a plurality of transmissive optical elements, each of which is aligned between a corresponding lens element and a corresponding optical filter such that light incident on any of the lens elements is converged through the corresponding transmissive optical element and the corresponding optical filter to one of the optical detector regions of a corresponding subarray of the optical detector region. Each transmissive optical element receives converged light from the corresponding lens element propagating along an initial propagation direction and converts the received converged light into transmitted converged light, which propagates away from the transmissive optical element along a final propagation direction parallel to the optical axis of the corresponding optical filter, or the final propagation direction relative to the optical axis of the corresponding optical filter defines a smaller angle than the initial propagation direction of the received converged light.
2. The multispectral optical system of claim 1, wherein the plurality of subarrays in the optical detector region are arranged as a 1D or 2D array of subarrays.
3. The multispectral optical system of claim 2, wherein the plurality of subarrays in the optical detector region are arranged as a uniform 1D or 2D array of subarrays.
4. The multispectral optical system according to claim 1 or 2, wherein the plurality of optical detector regions of each subarray of the optical detector region are arranged as a 1D or 2D array of optical detector regions.
5. The multispectral optical system of claim 4, wherein the plurality of optical detector regions of each subarray of the optical detector region are arranged as a uniform 1D or 2D array of optical detector regions.
6. The multispectral optical system according to claim 1 or 2, wherein each subarray of the optical detector region comprises a central optical detector region and one or more peripheral optical detector regions arranged around the central optical detector region.
7. The multispectral optical system of claim 6, wherein one or more of the peripheral optical detector regions are arc-shaped and arranged circumferentially around the central optical detector region, or wherein one or more of the peripheral optical detector regions are annular in shape and arranged concentrically with the central optical detector region.
8. The multispectral optical system according to claim 1 or 2, wherein the plurality of optical filters are disposed or formed on the front surface of the monolithic semiconductor chip.
9. The multispectral optical system according to claim 1 or 2, wherein the plurality of lens elements are defined by or formed on an optical substrate.
10. The multispectral optical system of claim 9, comprising a spacer located between the monolithic semiconductor chip and the optical substrate.
11. The multispectral optical system of claim 10, wherein the monolithic semiconductor chip and the optical substrate are attached to the spacer.
12. The multispectral optical system of claim 10, wherein the spacer defines a plurality of apertures, wherein each aperture is aligned with a corresponding subarray of a corresponding lens element, a corresponding optical filter, and an optical detector region.
13. The multispectral optical system of claim 1, wherein the plurality of transmission optical elements comprises a plurality of additional lens elements.
14. The multispectral optical system of claim 13, wherein the plurality of additional lens elements comprises a microlens array (MLA) or a microFresnel lens array.
15. The multispectral optical system of claim 13, wherein the plurality of transmission optical elements are defined by or formed on an additional optical substrate.
16. The multispectral optical system of claim 15, wherein the additional optical substrate is attached to the front surface of the monolithic semiconductor chip.
17. The multispectral optical system of claim 15, wherein the spacer is attached to the front surface of the additional optical substrate.
18. The multispectral optical system of claim 13, wherein each transmission optical element is defined by or formed on a corresponding optical filter.
19. The multispectral optical system according to claim 1 or 2, wherein at least one of the following: Each optical filter includes an optical interference filter; Each of the plurality of optical filters has a corresponding passband optical transmission spectrum; The sum of the optical transmission values of the plurality of optical filters is the same at all wavelengths within a predefined wavelength range; The optical transmission spectra of at least three of the optical filters are selected for trichromatic stimulus detection; The optical transmission spectra of at least three of the optical filters correspond to the corresponding coordinates in the CIE color space; The optical transmission spectra of at least three of the optical filters correspond to the respective components of the XYZ color space.
20. The multispectral optical system of claim 1, wherein the processing resources are configured to associate the electrical signal generated by the optical detector region with the optical transmission spectrum of the corresponding optical filter.
21. The multispectral optical system of claim 1 or 20, wherein the processing resources are configured to determine the ambient light source classification for each of the plurality of different incident directions based on a comparison between the electrical signal value corresponding to each incident direction of each subarray for each optical detector region and predefined spectral data, and optionally, wherein the predefined spectral data comprises a plurality of discrete spectra, each spectrum corresponding to a different known type or kind of ambient light source.
22. An image sensing system, comprising: The multispectral optical system according to any one of claims 1 to 21; as well as An image sensor having a known spatial relationship relative to the multispectral optical sensor. The image sensor and the processing resources are configured to communicate with each other, and The processing resources are configured to adjust the image sensed by the image sensor based on the ambient light source classification for each incident direction.
23. The image sensing system of claim 22, wherein the processing resources are configured to adapt the image by performing white balance on the image based on one or more parameters classifying the ambient light source in each direction.
24. The image sensing system of claim 23, wherein the processing resources are configured to adapt the image by performing gradient white balance on the image based on one or more parameters classifying the ambient light source for each incident direction.
25. An electronic device comprising at least one of the following: a multispectral optical system according to any one of claims 1 to 21 or an image sensing system according to any one of claims 22 to 24.
26. The electronic device of claim 25, including a mobile electronic device.
27. The electronic device of claim 26, wherein the mobile electronic device includes a mobile phone, a cellular phone, a smartphone, a tablet computer, or a laptop computer.