Compensation for problems in the optical alignment of multispectral optical sensors with sectoral view

Abstract

An optical multispectral sensor comprises a plurality of apertures, a plurality of lens arrangements, a plurality of optical filters, and a monolithic semiconductor chip defining a plurality of subarrangements of optical detector areas, wherein each subarrangement comprises the same number and relative spatial arrangement of optical detector areas, and the optical detector areas of each subarrangement are arranged at a predetermined distance in a direction parallel to a surface of the semiconductor chip.The multispectral optical sensor is configured such that light from a scene incident on one of the apertures along any direction of incidence is transmitted through the appropriate lens arrangement and optical filter to the appropriate subset of optical detector areas to produce a blurred image in a plane of the optical detector areas of the appropriate subset of optical detector areas, such that a ratio of a full width at half maximum (FWHM) of an optical intensity distribution of the blurred image to the predetermined spacing of the optical detector areas in the direction parallel to the surface of the semiconductor chip is greater than or equal to 2.0 and less than or equal to 4.0.

Description

AREA

[0001] The present disclosure relates to an optical multispectral sensor configured to enable spectral reconstruction of light incident on the optical multispectral sensor from different directions in a scene when one or more of the optical components of the optical multispectral sensor are misaligned, an optical multispectral system including the optical multispectral sensor, and an image acquisition system including the optical multispectral system. BACKGROUND

[0002] Color constancy is a desirable property of image capture devices, such as cameras. Color constancy refers to the ability to see a feature or object in a relatively constant color under different lighting conditions. That is, the appearance of an image captured by a camera can be affected by the ambient lighting. For example, if the color temperature of an ambient light source is relatively low, such as in the range of 3000 Kelvin, as might be the case with an incandescent light bulb, an image of a white object exposed to the ambient light source will have a reddish tint. Conversely, if the ambient light source has a high color temperature, such as in the range of 6000 Kelvin, as might be found in daylight on a cloudy day, the image of the white object will have a slight blue tint.This means that the object is perceived by a camera as a color that depends on the illumination of the object by the ambient light source.

[0003] It is known to compensate for such effects by using a multispectral ambient light sensor (ALS) to measure spectral information related to a scene. For example, Fig. 1A illustrates a smartphone 1 including a multispectral ALS 2, a camera 4, and a cover glass 8 covering the multispectral ALS 2 and the camera 4, wherein the multispectral ALS 2 is configured to measure the spectral distribution of light from a scene incident on the camera 4.

[0004] Fig. shows a detailed cross-sectional view of the multispectral ALS 2 and the camera 4. The multispectral ALS 2 has a plurality of optical detector areas 11. The multispectral ALS 2 is configured so that each optical detector area 11 detects a different wavelength range, e.g. because the multispectral ALS 2 comprises a plurality of different optical filters (in Fig. 1B not explicitly shown), wherein each optical filter is configured to transmit only a corresponding different wavelength range to a corresponding one of the optical detector areas 11. For the sake of clarity, Fig. only three optical detector areas 11 are shown. However, one skilled in the art will understand that the multispectral ALS sensor 2 may have more than three optical detector areas 11 or fewer than three optical detector areas 11.

[0005] The multispectral ALS 2 also comprises a housing 20, a diffuser 30 and an IR cut filter 32 located between the cover glass 8 and the housing 20.

[0006] The housing 20 has an aperture or window 22 through which light passes into the housing 20 via the cover glass 8, the diffuser 30, and the IR-cut filter 32. The multispectral ALS 2 has an optical axis 40 that is perpendicular to a front surface of the multispectral ALS 2. As one skilled in the art will appreciate, the use of the diffuser 30 provides the multispectral ALS 2 with a field of view (FOV) 42 that defines a large solid angle around the optical axis 40. Each optical detector region 11 detects a different range of wavelengths incident on the optical detector region 11 from all different incident directions across the entire FOV 42 of the multispectral ALS 2.

[0007] The camera 4 has an optical axis 50 that is perpendicular to a front surface of an image sensor (not shown) of the camera 4 and parallel to the optical axis 40 of the multispectral ALS 2. The camera 4 has a FOV 52 that defines a solid angle around the optical axis 50 of the camera 4, wherein the solid angle of the FOV 52 of the camera 4 is comparable to or smaller than the solid angle of the FOV 42 of the multispectral ALS 2.

[0008] Smartphone 1 uses white balance, preferably automatic white balance (AWB), to adjust the color of images captured under different lighting conditions. For example, smartphone 1 may have predefined settings for typical lighting conditions such as daylight, fluorescent light, or incandescent light; in some cases, the predefined settings may be selected automatically.

[0009] Existing techniques for white balance include image processing by applying an algorithm based on the "gray world theory" or "white patch theory." Gray world theory is based on the assumption that the average reflectance in a captured image is achromatic. That is, the average of the three color channels—red, green, and blue—should be approximately equal. White patch theory is based on the assumption that the brightest pixel in a captured image corresponds to a reflection of the ambient light source, and therefore the brightest pixel may correspond to a spectrum of ambient illumination. Both approaches have known limitations, and importantly, both approaches produce very different results.

[0010] Furthermore, different parts of a scene may be exposed to different ambient lighting conditions. For example, even different parts of a same-colored object in a scene may appear different depending on the corresponding ambient lighting conditions of the different parts of the same-colored object. Accordingly, it is desirable to be able to correct a captured image of a scene for the effects of different ambient lighting conditions on different parts of the scene without having to accept the shortcomings of state-of-the-art AWB methods.

[0011] It is known to capture the optical power in different spectral ranges incident on a multispectral ALS from different directions of a scene and to determine the spectrum of the light incident on the multispectral ALS from different directions of the scene through spectral reconstruction. However, sharp changes in the illumination conditions of a scene can lead to inaccuracies when using AWB to adjust the coloration of images acquired under different illuminations. This problem is further exacerbated when the light source or the sudden change is dynamic and moving in the scene.

[0012] Furthermore, it has been shown that the accuracy with which the spectrum of light incident on the multispectral ALS from different directions of the scene can be determined using spectral reconstruction is limited by misalignment between one or more of the optical components of the multispectral ALS. Therefore, it is desirable to improve the accuracy with which the spectrum of light incident on the multispectral ALS from different directions of the scene can be determined. SUMMARY

[0013] According to one aspect of the present disclosure, there is provided an optical multispectral sensor comprising: a variety of apertures; a variety of lens arrangements; several optical filters; and a monolithic semiconductor chip defining a plurality of sub-arrays of optical detector regions, each sub-array comprising the same number and relative spatial arrangement of optical detector regions, and the optical detector regions of each sub-array being arranged at a predetermined distance in a direction parallel to a surface of the semiconductor chip, wherein each subset of optical detector regions is aligned with a corresponding optical filter, a corresponding lens arrangement and a corresponding aperture, and wherein the multispectral optical sensor is configured such that light from a scene incident on one of the apertures along any incident direction is transmitted through the one of the apertures, the corresponding lens arrangement and the corresponding optical filter to the corresponding subarray of optical detector regions to generate a blurred image in a plane of the optical detector regions of the corresponding subarray of optical detector regions such that a ratio of a full width at half maximum (FWHM) of an optical intensity distribution of the blurred image to the predetermined spacing of the optical detector regions in the direction parallel to the surface of the semiconductor chip is greater than or equal to 2.0 and less than or equal to 4.0.

[0014] In use, the optical detector regions of the same subarray of optical detector regions can detect light incident on the multispectral optical sensor from different directions in the same spectral range, and corresponding optical detector regions of different subarrays of optical detector regions can detect light incident on the multispectral optical sensor from the same direction in different spectral ranges. The detected signals can be used to reconstruct the spectrum of light incident on the multispectral optical sensor from different directions in a scene. The aperture, the corresponding lens arrangement, the corresponding optical filter, and the corresponding subset of optical detector regions can together form a spectral channel.Any misalignment of the aperture, the corresponding lens array, the corresponding optical filter, and the corresponding subset of the optical detector areas can lead to inaccuracies in the reconstructed spectrum. Such inaccuracies can be at least partially compensated for by measuring the degree of misalignment of each spectral channel and adjusting the acquired signals for each spectral channel based on the measured degree of misalignment.It has been found that a ratio of the full width at half maximum (FWHM) of the optical intensity distribution of the out-of-focus image to the specified distance of the optical detector areas in the direction parallel to the surface of the semiconductor chip greater than or equal to 2.0 and less than or equal to 4.0 makes it possible to sufficiently compensate for the misalignment of each spectral channel to enable accurate spectral reconstruction without excessively increasing the number of optical detector areas for a given field of view (FOV) of the multispectral optical sensor.

[0015] Optionally, the ratio of the full width at half maximum (FWHM) of the optical intensity distribution of the blurred image to the specified distance of the optical detector regions in the direction parallel to the surface of the semiconductor chip is greater than or equal to 2.5 and less than or equal to 3.5 or substantially equal to 3.

[0016] Optionally, each optical filter is placed between the corresponding lens array and the corresponding subset of optical detector areas, so that the light from the scene passes through the corresponding lens array before passing through the corresponding optical filter.

[0017] Optionally, each lens assembly is positioned between the corresponding aperture and the corresponding optical filter so that the light from the scene passes through the corresponding aperture before the light hits the corresponding lens assembly.

[0018] Optionally, each aperture is placed between the corresponding lens assembly and the corresponding optical filter so that the light from the scene passes through the corresponding lens assembly before the light hits the corresponding aperture.

[0019] Optionally, each lens arrangement defines the corresponding aperture.

[0020] Optionally, each lens array is aligned to focus the incident light onto a focal plane adjacent to the plane of the optical detector regions to produce the blurred image in the plane of the optical detector regions.

[0021] Optionally, the focal plane of each lens array is arranged at a predetermined distance from the plane of the optical detector areas.

[0022] Optionally, the optical multispectral sensor comprises a plurality of scattering elements, each scattering element configured to scatter the incident light to produce the blurred image in the plane of the optical detector regions.

[0023] Optionally, the plurality of lens arrays includes one or more microlens arrays (MLA).

[0024] Optionally, at least some of the lens arrangements are defined by or formed on an optical substrate.

[0025] Optionally, each lens arrangement comprises one lens element or more than one lens element, e.g., two lens elements.

[0026] Optionally, each lens arrangement may include one or more Fresnel lenses.

[0027] According to one aspect of the present disclosure, there is provided a multispectral optical detection system comprising: the multispectral optical sensor as described above; and a processing resource, wherein the optical multispectral sensor and the processing resource are configured to communicate with each other, and where the processing resource is configured to: associating different electrical signals generated by different optical detector regions of the same subarray of optical detector regions with light incident on the multispectral optical sensor from a scene along correspondingly different directions of incidence, and different electrical signals generated by corresponding optical detector areas of different subgroups of optical detector areas are associated with light incident from the scene onto the optical multispectral sensor in the same direction of incidence.

[0028] Optionally, the processing resource is configured to combine the electrical signal generated by an optical detector region with the optical transmission spectrum of the corresponding optical filter.

[0029] Optionally, the processing resource is configured to determine a classification of the ambient light source for each of a plurality of different incident directions based on a comparison between the electrical signal values ​​corresponding to each incident direction and predefined spectral data.

[0030] Optionally, the predefined spectral data includes a plurality of spectra, each spectrum corresponding to a different known type or kind of ambient light source.

[0031] According to one aspect of the present disclosure, an image scanning system is provided, comprising: the multispectral optical detection system described above; and an image sensor with a known spatial relationship to the optical multispectral sensor, wherein the image sensor and the processing resource are configured to communicate with each other, and wherein the processing resource is configured to adjust an image captured by the image sensor based on the classification of the ambient light source for each incident direction.

[0032] Optionally, the processing resource is configured to adjust the image by white balancing the image based on one or more ambient light source classification parameters for each direction.

[0033] Optionally, the adjustment includes a gradient white balance of the image based on one or more parameters of the ambient light source classification for each direction of incidence.

[0034] According to one aspect of the present disclosure, an electronic device is provided comprising the multispectral optical system, the multispectral optical system and / or the image sensor system as described above.

[0035] It is understood that any one or more of the features of any of the above aspects of the present disclosure may be combined with any one or more of the features of any of the other above aspects of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] An optical multispectral sensor, an optical multispectral system, an image sensor system and associated methods will now be described, by way of non-limiting example only, with reference to the drawings in which they are illustrated: Fig. 1A is a schematic diagram of the back of a prior art electronic device in the form of a prior art smartphone with a prior art multispectral ambient light sensor (ALS) array and a camera; Fig. Figure 1B is a schematic cross-section of the prior art multispectral ALS and the prior art smartphone camera of Fig. 1A; Fig. Figure 2A is a schematic diagram of the back of an electronic device in the form of a smartphone with a multispectral ALS array and a camera; Fig. Figure 2B is a schematic cross-section through the multispectral ALS array and the camera of the smartphone from Fig. 2A; Fig. Figure 3A is a schematic representation of a multispectral ALS of the multispectral ALS array of Fig. 2A; Fig. Figure 3B is a schematic representation of a lens arrangement of the multispectral ALS of Fig. 3A; Fig. 4 is a schematic plan view of a monolithic multispectral ambient light sensor (ALS) semiconductor chip of the multispectral ALS of Fig. 3A; Fig. 5A is a schematic plan view of a sub-array of optical detector regions of the monolithic multispectral ambient light sensor (ALS) semiconductor chip of Fig. 4; Fig. Figure 5B shows the simulated projected angular distributions of the contours of the optical detector regions of different subarrays of differently misaligned spectral channels of the multispectral ALS of Fig. 3A, projected into a FOV of the multispectral ALS of Fig. 3A; Fig. Figure 6A shows the simulated projected position distributions of the contours of the optical detector regions of different subarrays of differently misaligned spectral channels of the multispectral ALS of Fig. 3A, which is integrated into a FOV of the multispectral ALS of Fig. 3A and a white line object are projected in the FOV, for optical detector regions at a distance of 80 µm; Fig. Figure 6B shows the simulated detected signal distributions and the simulated detected signals for each optical detector area in a middle row of optical detector areas and for each misaligned spectral channel corresponding to Fig. 6A; Fig. Figure 7A shows a spatially modulated optical intensity distribution of a first blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 5x5 array with a pitch of 80 µm; Fig. Figure 7B shows simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 7A were detected; Fig. Figure 7C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 7A corresponds; Fig. Figure 8A shows a compensated, spatially modulated optical intensity distribution of the first blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 5x5 array with a pitch of 80 µm; Fig. 8B simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 8A were detected; Fig. Figure 8C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 8A corresponds; Fig. Figure 9A shows a spatially modulated optical intensity distribution of the first blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in an 11x11 array with a pitch of 36 µm; Fig. Figure 9B shows simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 9A were detected; Fig. Figure 9C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 9A corresponds; Fig. Figure 10A shows a compensated, spatially modulated optical intensity distribution of the first blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in an 11x11 array with a pitch of 36 µm; Fig. 10B simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 10A were detected; Fig. Figure 10C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 10A corresponds; Fig. Figure 11A shows a spatially modulated optical intensity distribution of the first blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 15x15 array with a pitch of 26 µm; Fig. Figure 11B shows simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 11A were detected; Fig. Figure 11C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 11A corresponds; Fig. Figure 12A shows a compensated, spatially modulated optical intensity distribution of the first blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 15x15 array with a pitch of 26 µm; Fig. 12B simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 12A were detected; Fig. Figure 12C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 12A corresponds; Fig. Figure 13A shows a spatially modulated optical intensity distribution of a second blurred scene for three differently aligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 5x5 array with a pitch of 80 µm; Fig. Figure 13B shows simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 13A were detected; Fig. Figure 13C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 13A; Fig. Figure 14A shows a compensated, spatially modulated optical intensity distribution of the second blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 5x5 array with a pitch of 80 µm; Fig. 14B simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 14A were recorded; Fig. Figure 14C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 14A; Fig. Figure 15A shows a spatially modulated optical intensity distribution of the second blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in an 11x11 array with a pitch of 36 µm; Fig. Figure 15B shows simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 15A were detected; Fig. Figure 15C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 15A corresponds; Fig. Figure 16A shows a compensated spatially modulated optical intensity distribution of the second blurred scene for three differently aligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in an 11x11 array with a pitch of 36 µm; Fig. 16B simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 16A were detected; Fig. Figure 16C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 16A corresponds; Fig. Figure 17A shows a spatially modulated optical intensity distribution of the second blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 15x15 array with a pitch of 26 µm; Fig. Figure 17B shows simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 17A were detected; Fig. Figure 17C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 17A; Fig. Figure 18A shows a compensated spatially modulated optical intensity distribution of the second blurred scene for three differently aligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 15x15 array with a pitch of 26 µm; Fig. 18B simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 18A were detected; Fig. Figure 18C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 18A corresponds; Fig. Figure 19A shows a spatially modulated optical intensity distribution of a third blurred scene for three differently oriented spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 5x5 array with a pitch of 80 µm; Fig. Figure 19B shows simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 19A were detected; Fig. Figure 19C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 19A; Fig. Figure 20A shows a compensated, spatially modulated optical intensity distribution of the third blurred scene for three differently misaligned spectral channels at a FWHM intensity blur of 110 µm and optical detector areas arranged in a 5x5 array with a pitch of 80 µm; Fig. 20B simulated detected signal distributions and simulated detected signals generated by the optical detector areas of each misaligned spectral channel according to the spatially modulated optical intensity distribution of Fig. 20A were detected; Fig. Figure 20C shows the percentage error in the signal detected by the optical detector areas of each misaligned spectral channel corresponding to the spatially modulated optical intensity distribution of Fig. 20A corresponds; Fig. Figure 21 shows the effects of a ±1% signal error due to misalignment of the spectral channel on the reconstruction of the spectra of 16 different types of light sources; Fig. Figure 22 illustrates the effects of a signal error of ±2.5% due to misalignment of the spectral channel on the reconstruction of the spectra of 16 different types of light sources; and Fig. Figure 23 illustrates the effect of a ±5% signal error due to misalignment of the spectral channel on the reconstruction of the spectra of 16 different types of light sources. DETAILED DESCRIPTION OF THE DRAWINGS

[0037] In Fig. 2A, a smartphone 101 is shown including an optical multispectral sensor in the form of a sector view multispectral ALS 102, a camera 104 having a known spatial relationship to the multispectral ALS 102, and a cover glass 108 covering the multispectral ALS 102 and the camera 104.

[0038] Fig. Figure 2B shows a detailed cross-sectional view of the multispectral ALS 102 and camera 104. The multispectral ALS 102 includes an IR cut filter 132 located behind or below the cover glass 108.

[0039] As will be described in more detail below, the multispectral ALS 102 defines a plurality of subarrays of square optical detector regions 111, each subarray comprising 5x5 optical detector regions 111. However, in the interest of clarity, it should be understood that in Fig. 2B, only the optical detector regions 111 of a single subarray of 5x5 optical detector regions 111 are shown. The multispectral ALS 102 further includes a housing 120 defining an aperture or window 122 for admitting light into the housing 120 via the cover glass 108 and the IR-cut filter 132. The multispectral ALS 102 has an optical axis 140 that is perpendicular to a front surface of the multispectral ALS 102.

[0040] As described below, the multispectral ALS 102 is configured to distinguish between light incident on the multispectral ALS 102 along different directions of incidence and to measure the spectral distribution of the light incident on the multispectral ALS 102 for each different direction of incidence across a FOV 142 that defines a solid angle about the optical axis 140 of the multispectral ALS 102. In particular, the multispectral ALS 102 is configured to distinguish between light incident on the multispectral ALS 102 from different sectors of the FOV 142, as indicated by the dashed lines in Fig. 2B, and that it can measure the spectral distribution of light incident on the multispectral ALS 102 from each sector of the FOV 142. The camera 104 also has an optical axis 150 that is perpendicular to a front surface of an image sensor chip (not shown) of the camera 104 and parallel to the optical axis 140 of the multispectral ALS 102. The camera 104 has an FOV 152 that defines a solid angle about the optical axis 150 of the camera 104, wherein the solid angle of the FOV 152 of the camera 104 is comparable to the solid angle of the FOV 142 of the multispectral ALS 102.

[0041] As in Fig. 3A, the multispectral ALS 102 includes an aperture element 160 defining a plurality of apertures 160a, a first microlens array (MLA) 161a, a second microlens array (MLA) 161b, a monolithic multispectral ALS semiconductor chip 110, a circuit board 162, and a frame element 164. The monolithic multispectral ALS semiconductor chip 110 and the frame element 164 are mounted on the circuit board 162, with the frame element 164 disposed around the monolithic multispectral ALS semiconductor chip 110. The circuit board 162 and the frame part 164 together form the housing 120, which holds the aperture element 160 and the first and second MLAs 161a, 161b in alignment with the monolithic multispectral ALS semiconductor chip 110.

[0042] As in Fig. 4, the monolithic multispectral ALS semiconductor chip 110 defines a plurality of sub-arrays 112 of square optical detector regions 111 in the form of a 3x4 array of sub-arrays 112 of square optical detector regions 111, each sub-array 112 having the same number and relative spatial arrangement of optical detector regions 111 as each of the other sub-arrays 112. In particular, each of the sub-arrays 112 defines a 5x5 sub-array of optical detector regions 111.

[0043] The monolithic multispectral ALS semiconductor chip 110 includes a plurality of optical filters 166 formed on or attached to the monolithic multispectral ALS semiconductor chip 110 in front of a corresponding subarray 112 of optical detector regions 111. Each optical filter 166 has a corresponding optical transmission spectrum. In particular, each optical filter 166 is a passband optical interference filter defining a corresponding spectral passband. The optical filters 166 may, for example, define different spectral passbands.

[0044] As in Fig. 3B, the first and second MLAs 161a, 161b define a plurality of lens arrays 170, each lens array 170 having two convex lens elements 170a and 170b disposed opposite each other.

[0045] Each optical filter 166 is aligned between a corresponding lens array 170 and a corresponding subset 112 of optical detector regions 111.

[0046] Furthermore, the aperture element 160, the first and second MLAs 161a, 161b, and the monolithic multispectral ALS semiconductor chip 110 should be aligned such that each sub-array 112 of the optical detector regions 111 and the corresponding optical filter 166 are aligned in x and y with a corresponding lens array 170 and a corresponding aperture 160a to form a spectral channel 176.

[0047] In addition, as in Fig. 3A, each spectral channel 176 is configured such that, in use, any light incident on one of the apertures 160a of the multispectral ALS 102 along any direction of incidence is focused by the corresponding lens assembly 170 to a focal plane adjacent, for example, in front of, above, or in front of the plane of the corresponding sub-array 112 of optical detector regions 111 before passing through the corresponding optical filter 166 and forming a blurred image in a plane of the optical detector regions 111 of the corresponding sub-array of optical detector regions 112.For example, for a given configuration of the lens array 170, the distance between the aperture 160a and the lens array 170 and the distance between the lens array 170 and the monolithic multispectral ALS semiconductor chip 110 can be selected such that any light incident on one of the apertures 160a of the multispectral ALS 102 along a given direction of incidence is focused by the corresponding lens array 170 onto a focal plane adjacent to, above, or above the focal plane, for example, in front of, above, or above the plane of the corresponding subarray 112 of the optical detector regions 111, before passing through the corresponding optical filter 166 and forming a blurred image in a plane of the optical detector regions 111 of the corresponding subarray of the optical detector regions 112.

[0048] By focusing the incident light to a focal point in a plane adjacent to (i.e., not coincident with) the plane of the optical detector regions 111, rather than a plane coincident with the plane of the optical detector regions 111, the image formed by the light when it reaches the optical detector regions 111 is out of focus and therefore out of focus when captured. This provides several significant advantages, which are described below.

[0049] Fig. Figure 5A shows the subarray 112 of optical detector regions 111 of one of the spectral detection channels 176 in detail. The subarray 112 of optical detector regions 111 comprises a uniform 5x5 array of optical detector regions 111 arranged at a pitch of 56 µm, with adjacent optical detector regions 111 separated by a pitch of 3 µm. As described below with reference to Fig. As described in more detail in Figures 7A to 23, the optical detector regions 111 can be arranged in a 2D array other than a 5x5 array. For example, the optical detector regions 111 can be arranged in an 11x11 array or a 15x15 array. The pitch of the optical detector regions 111 can also differ from 56 µm. For example, the pitch of the optical detector regions 111 can be 80 µm, 36 µm, or 26 µm.

[0050] In use, any light incident on one of the apertures 160a of the multispectral ALS 102 along a direction parallel to the optical axis 140 is focused by the corresponding lens arrangement 170 to a focal position in a plane offset in z above the plane of the corresponding sub-array 112 of the optical detector regions 111 before passing through the corresponding optical filter 166, the focal position in x and y corresponding to a central position of the corresponding sub-array 112 of the optical detector regions 111, which coincides with the central optical detector region 111 located in Fig. 5A with the designation "12". 5A. Similarly, any light incident on one of the apertures 160a of the multispectral ALS 102 along a direction oblique to the optical axis 140 of the multispectral ALS 102 is focused by the lens array 170 to a focal position in a plane offset in z across the plane of the corresponding sub-array 112 of optical detector regions 111 before passing through the corresponding optical filter 166, wherein the focal position in x and y corresponds to a peripheral position of the corresponding sub-array 112 of optical detector regions 111, which may correspond to one of the peripheral optical detector regions 111 designated by the designations "0" to "11" and "13" to "24" in Fig. 5A.

[0051] Dating back to Fig. 2A, the smartphone 101 includes a processing resource 180 configured to receive data from the multispectral ALS 102 and the image sensor (not shown) of the camera 104. The processing resource 180 is configured to associate different electrical signals generated by different optical detector areas 111 of the same sub-array 112 of optical detector areas 111 with light incident on the multispectral ALS 102 from different sectors of a scene in the FOV 142 of the multispectral ALS 102. Conversely, the processing resource 180 is configured to associate different electrical signals generated by corresponding optical detector areas 111 of different sub-arrays 112 with light incident on the multispectral ALS 102 from the same sector of the scene in the FOV 142 of the multispectral ALS 102.

[0052] Furthermore, the processing resource 180 is configured to link the electrical signal generated by any optical detector region 111 of any subarray 112 to the optical transmission spectrum of the corresponding optical filter 166. Since each optical filter 166 has a different passband, the different electrical signal values ​​measured by corresponding optical detector regions 111 of different subarrays 112 of optical detector regions 111 are representative of an optical spectrum of the light incident on the multispectral ALS 102 from the same sector of the scene.

[0053] Processing resource 180 is configured to determine a classification of the ambient light source for each sector of the scene based on a comparison between the electrical signal values ​​corresponding to each sector of the scene and predefined spectral data. For example, the predefined spectral data may include a plurality of different spectra, each spectrum corresponding to a different known type or kind of ambient light source.

[0054] Furthermore, the processing resource 180 is configured to adjust an image captured by the camera 104 based on the ambient light source classification for each sector of the scene. Specifically, the processing resource 180 is configured to adjust the captured image by white balancing the image based on one or more ambient light source classification parameters for each sector of the scene, for example, by gradient white balancing the image based on one or more ambient light source classification parameters for each sector of the scene.

[0055] During the manufacturing of the multispectral ALS 102, the alignment tolerance between the aperture element 160 and the monolithic multispectral ALS semiconductor chip 110, and the alignment tolerance between the MLAs 161a, 161b and the monolithic multispectral ALS semiconductor chip 110, may each be 3-5 µm. This means that the cumulative alignment error for one or more of the spectral channels may be 10-15 µm, resulting in differences between the signals acquired by different spectral channels 176.It should be understood that these signal differences are related to misalignment between the aperture element 160 and the monolithic multispectral ALS semiconductor chip 110 and to misalignment between the MLAs 161a, 161b and the monolithic multispectral ALS semiconductor chip 110 for different spectral channels 176 and are independent of the signal differences between the spectral channels 176 as a result of the spectral distribution of the light incident on the multispectral ALS 102 from the scene. Chromatic aberration from spectral channel 176 to spectral channel 176 may also contribute to the differences between the signals acquired by the different spectral channels 176.The resulting differences between the signals acquired by the different spectral channels 176 caused by these effects reduce the accuracy of the spectral reconstruction method used to classify the spectrum of light received from the different sectors of the scene in the FOV 142 of the multispectral ALS 102.

[0056] As explained below with reference to the Fig. 5B to 23, the degree of blurring of the image formed by the light when it reaches the optical detector areas 111 of each sub-array 112 of the optical detector areas 111 is selected such that the differences between the signals detected by different spectral channels 176 caused by different alignment errors between the different spectral channels 176 can be reduced using a compensation method to such an extent that the spectral reconstruction can be used to accurately classify the spectrum of light received from the different sectors of a scene in the FOV 142 of the multispectral ALS 102 without excessively increasing the number of optical detector areas 111 of each sub-array 112.

[0057] Fig. Figure 5B shows simulated projected angular distributions of the contours of the optical detector areas 111 of various sub-arrays 112, projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102 using a mathematical model of each spectral channel 176. The different sectors of the scene are labeled "0" to "24." Distortion means that the simulated projected angular distribution of the contours of the different optical detector areas 111 of the different sub-arrays 112 is non-square.

[0058] The black dashed line shows a simulated projected angular distribution of the outlines of the optical detector regions 111 of a subfield 112 of an aligned spectral channel 176 projected into the FOV 142 of the multispectral ALS 102, for the ideal case that the aperture 160a and the lens arrangement 170 of the first spectral channel 176 are aligned with the corresponding subfield 112 of the optical detector regions 111 in x and y, such that the center of the projection of the subfield 112 of the optical detector regions 111 is aligned with the center of the FOV 142.

[0059] The red line shows a simulated projected angular distribution of the contours of the optical detector regions 111 of a subarray 112 of a first misaligned spectral channel 176, projected into the FOV 142 of the multispectral ALS 102 in the case that the aperture 160a and the lens arrangement 170 of the first misaligned spectral channel 176 are together misaligned by 10 µm in the positive x-direction, but aligned in y, relative to the corresponding subarray 112 of the optical detector regions 111.

[0060] The green line shows a simulated projected angular distribution of the contours of the optical detector regions 111 of a subarray 112 of a second misaligned spectral channel 176 projected into the FOV 142 of the multispectral ALS 102, in the case that the aperture 160a and the lens arrangement 170 of the second misaligned spectral channel 176 are together misaligned by 10 µm in the positive y-direction, but aligned in the x-direction, relative to the corresponding subarray 112 of the optical detector regions 111.

[0061] The blue line shows a simulated projected angular distribution of the contours of the optical detector regions 111 of a subarray 112 of a third misaligned spectral channel 176 projected into the FOV 142 of the multispectral ALS 102, in the case that the aperture 160a and the lens arrangement 170 of the third misaligned spectral channel 176 are together misaligned by 10 µm in the negative x-direction, but aligned in y, relative to the corresponding subarray 112 of the optical detector regions 111.

[0062] Fig. 6A shows simulated projected position distributions of the outlines of the optical detector areas 111 of three differently misaligned spectral channels 176, wherein the optical detector areas 111 of the three differently misaligned spectral channels 176 are projected onto corresponding sectors of the scene in the FOV 142 of the multispectral ALS 102 using a mathematical model of each misaligned spectral channel 176, for an optical detector area 111 with a pitch of 80 µm and for three different misalignments between the aperture 160a and the lens arrangement 170 of each misaligned spectral channel 176 relative to the corresponding subfield 112 of the optical detector areas 111, namely misalignments of +10 µm in x, +10 µm in y and -10 µm in x. The different sectors of the scene are labeled “0” to “24”.

[0063] Fig. Figure 6A also shows the optical intensity distribution in the scene in the form of a central, vertically oriented stripe of constant intensity on a background of lower intensity.

[0064] The solid lines in Fig. 6B show the simulated acquired signals for each misaligned spectral channel 176 as a function of x at y=172 in Fig. 6A. It should be clear that each simulated acquired signal for each sector of the scene is obtained by integrating the intensity distribution for each misaligned spectral channel 176 as a function of x at y=172 in Fig. 6A across the sector of the scene. One skilled in the art will understand that determining the simulated detected signals in this manner is mathematically equivalent to determining the detected signals when light from the scene is incident on the aperture 160a and imaged by the corresponding lens assembly 170 through the corresponding optical filter 166 onto the corresponding optical detector regions 111 of each misaligned spectral channel 176 when there is no blur, i.e., when the focal plane coincides with the plane of the optical detector regions 111.

[0065] The dashed lines in Fig. 6B show the simulated distributions of the detected signals for each misaligned spectral channel 176 as a function of x at y=172 in Fig. 6A, when a blur factor is applied to the position distribution of the optical intensity in the scene.

[0066] Fig. Figure 7A shows a spatially modulated optical intensity distribution of an out-of-focus scene formed by applying a FWHM intensity blur of 110 µm to an initial scene comprising a central, vertically aligned brighter stripe region of constant intensity on a lower intensity background to obtain an out-of-focus scene, and the spatial modulation of the out-of-focus scene by the simulated projected position distributions of the optical detector regions 111 of three differently misaligned spectral channels 176 projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102, using a mathematical model for each misaligned spectral channel 176, for optical detector regions 111 arranged in a 5x5 array with a pitch of 80 µm.and for three different misalignments between the aperture 160a and the lens array 170 of each misaligned spectral channel 176 relative to the corresponding sub-array 112 of the optical detector regions 111, namely misalignments of +10 µm in x, +10 µm in y, and -10 µm in x. The ratio between the FWHM intensity blur of 110 µm and the 80 µm spacing of the optical detector regions 111 is 1.375. The different sectors of the scene are labeled "0" to "24." It should be understood that the FWHM intensity blur used to obtain the blurred scene corresponds to one FWHM of the optical intensity distribution in the plane of the optical detector regions 111, which corresponds to one sector of the FOV 142.

[0067] In Fig. 7B, the dashed lines represent the simulated distributions of the detected signals for each of the three differently misaligned spectral channels 176 as a function of x at y=172 in Fig. 7A. A person skilled in the art will understand that the determination of the simulated detected signal distributions in this way is mathematically equivalent to the determination of the detected signal distributions for the case where light from the output scene is incident on the aperture 160a and imaged by the corresponding lens arrangement 170 through the corresponding optical filter 166 onto the corresponding optical detector regions 111 for each misaligned spectral channel 176 in the presence of blur, i.e., when the focal plane in z is offset from the plane of the optical detector regions 111, as described above with reference to the Fig. 3A and Fig. 3B described.

[0068] In Fig. 7B, the solid lines represent the simulated acquired signals for each misaligned spectral channel 176 as a function of x at y=172 in Fig. 7A. It is to be understood that each simulated detected signal for each sector of the blurred scene is obtained by integrating the simulated detected signal distribution for each misaligned spectral channel 176 as a function of x at y=172 in Fig. 7A over the sector of the scene, ie by integrating each simulated detected signal distribution represented by each dotted line in Fig. 7B, across the sector of the scene. One skilled in the art will understand that determining the simulated detected signals in this manner is mathematically equivalent to determining the detected signals when light from the source scene is incident on the aperture 160a and imaged by the corresponding lens assembly 170 through the corresponding optical filter 166 onto the corresponding optical detector regions 111 of each misaligned spectral channel 176 in the presence of blur, i.e., when the focal plane is offset in z from the plane of the optical detector regions 111, as described above with reference to the Fig. 3A and Fig. 3B described.

[0069] Fig. Figure 7C shows the percentage error in the signal for each optical detector area 111 in the middle row of the optical detector areas 111 labeled “10” to “14”, which correspond to the solid lines in Fig. Figure 7C shows the percentage error in the signal for each of the three differently misaligned spectral channels 176, where the percentage errors in the signals are calculated relative to the ideal spectral channel 176 when the aperture 160a and the lens assembly 170 of the spectral channel 176 are aligned in x and y with the corresponding subarray 112 of the optical detector regions 111 such that the center of the projection of the corresponding subarray 112 of the optical detector regions 111 is aligned with the center of the FOV 142. The mean absolute error in % of the signals is 4.28%.

[0070] The misalignment of each spectral channel 176 can be measured using a calibration procedure that involves positioning a suitable reference pattern, such as a reference pattern. B. a predetermined black and white pattern with markings in the center and near the corners, in the FOV 142 of the multispectral ALS 102, using each optical detector area 111 of each spectral channel 176 of the multispectral ALS 102 to detect a signal value corresponding to the reference pattern and, for each spectral channel 176, using a difference between the detected signal values ​​for each optical detector area 111 and the expected signal values ​​for each optical detector area 111 for the ideal case when the aperture 160a and the lens arrangement 170 of the spectral channel 176 are aligned with the corresponding subarray 112 of the optical detector areas 111 in x and y to determine the misalignment of the spectral channel 176 in x and y.A person skilled in the art will appreciate that a variety of different reference patterns can be used for the calibration procedure.

[0071] After calibration, data interpolation can be used to convert the acquired signals from the Fig. 7A-7C so that the measured misalignment of each spectral channel 176 is at least partially compensated, resulting in the compensated spatially modulated optical intensity distribution of the out-of-focus scene as shown in Fig. 8A, the simulated acquired signal distributions for each of the three differently misaligned spectral channels 176, which are indicated by the dotted lines in Fig. 8B, the simulated detected signals for each of the three differently misaligned spectral channels 176, which are indicated by the solid lines in Fig. 8B, and the percentage error in the signal for each optical detector area 111 of the middle row of optical detector areas 111 for each of the three differently misaligned spectral channels 176, as shown in Fig. 8C. After compensation, the mean absolute % error in the acquired signals due to the different misalignments of the spectral channels 176 is reduced to 4.04%. As will be described in more detail below, such compensation can improve the accuracy of the spectral reconstruction for each sector of the FOV 142. However, as will be described below with reference to the Fig. As described in more detail, an average absolute % error in the signals of <1% may be required for accurate spectral reconstruction.

[0072] Fig. Figure 9A shows a spatially modulated optical intensity distribution of the same blurred scene obtained with respect to Fig. 7A and formed by applying a FWHM intensity blur of 110 µm to an output scene comprising a central vertically aligned brighter stripe region of constant intensity on a lower intensity background to obtain a blurred scene, and by spatially modulating the blurred scene by the simulated projected position distributions of the optical detector regions 111 of three differently misaligned spectral channels 176 projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102, using a mathematical model of each misaligned spectral channel 176, for optical detector regions 111 arranged in an 11x11 array with a pitch of 36 µm,and for three different misalignments between the aperture 160a and the lens arrangement 170 of each misaligned spectral channel 176 relative to the corresponding sub-array 112 of the optical detector regions 111, namely misalignments of +10 µm in x, +10 µm in y, and -10 µm in x. In , Fig. 9A, the ratio between the FWHM intensity blur of 110 µm and the 36 µm spacing of the optical detector areas 111 is approximately 3.

[0073] In Fig. 9B, the dashed lines represent the simulated detected signal distributions for each of the three differently misaligned spectral channels 176, which correspond to the spatially modulated optical intensity distribution of Fig. 9A, and the solid lines represent the simulated detected signals for each of the three differently misaligned spectral channels 176 corresponding to the spatially modulated optical intensity distribution of Fig. 9A.

[0074] Fig. Figure 9C shows the percentage error in the signal for each optical detector area 111 in the middle row of optical detector areas 111 according to the solid lines in Fig. 9B for each of the three differently misaligned spectral channels 176. The mean absolute % error in the signals is 4.76%.

[0075] After calibration, data interpolation can be used to convert the acquired signals from the Fig. 9A-9C so that the measured misalignment of each spectral channel 176 is at least partially compensated, resulting in the compensated spatially modulated optical intensity distribution of the out-of-focus scene as shown in Fig. 10A, the simulated acquired signal distributions for each of the three differently misaligned spectral channels 176, which are indicated by the dotted lines in Fig. 10B, the simulated detected signals for each of the three differently misaligned spectral channels 176, which are represented by the solid lines in Fig. 10B, and the percentage error in the signal for each optical detector area 111 of the middle row of optical detector areas 111 for each of the three differently misaligned spectral channels 176, as shown in Fig. 10C. After compensation, the mean absolute % error in the acquired signals due to the different misalignments of the spectral channels 176 is reduced to 0.49%. Such a mean absolute % error is less than the targeted mean absolute % error of 1% required for accurate spectral reconstruction.

[0076] Fig. Figure 11A shows a spatially modulated optical intensity distribution of the same blurred scene obtained with respect to Fig. 7A and by applying a FWHM intensity blur of 110 µm to an output scene comprising a central vertically aligned brighter stripe region of constant intensity on a background of lower intensity to obtain a blurred scene, and by spatially modulating the blurred scene by the simulated projected position distributions of the optical detector regions 111 of three differently misaligned spectral channels 176 projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102, using a mathematical model of each misaligned spectral channel 176, for optical detector regions 111 arranged in a 15x15 array with a pitch of 26 µm,and for three different misalignments between the aperture 160a and the lens arrangement 170 of each misaligned spectral channel 176 relative to the corresponding sub-array 112 of the optical detector regions 111, namely misalignments of +10 µm in x, +10 µm in y, and -10 µm in x. For , Fig. 11A, the ratio of the FWHM intensity blur of 110 µm to the spacing of 26 µm of the optical detector areas 111 is approximately 4.1.

[0077] In Fig. 11B, the dashed lines represent the simulated detected signal distributions for each of the three differently misaligned spectral channels 176, which correspond to the spatially modulated optical intensity distribution of Fig. 11A, and the solid lines represent the simulated detected signals for each of the three differently misaligned spectral channels 176 corresponding to the spatially modulated optical intensity distribution of Fig. 11A.

[0078] Fig. Figure 11C shows the percentage error in the signal for each optical detector area 111 in the middle row of optical detector areas 111 according to the solid lines in Fig. 11B for each of the three differently misaligned spectral channels 176. The mean absolute % error in the signals is 4.85%.

[0079] After calibration, data interpolation is used to convert the acquired signals from the Fig. 11A-11C so that the measured misalignment of each spectral channel 176 is at least partially compensated, resulting in the compensated spatially modulated optical intensity distribution of the out-of-focus scene as shown in Fig. 12A, the simulated acquired signal distributions for each of the three differently misaligned spectral channels 176, represented by the dotted lines in Fig. 12B, the simulated detected signals for each of the three differently misaligned spectral channels 176, indicated by the solid lines in Fig. 12B, and the percentage error in the signal for each optical detector area 111 of the middle row of optical detector areas 111 for each of the three differently misaligned spectral channels 176, as shown in Fig. 12C. After compensation, the mean absolute % error in the acquired signals due to the different misalignments of the spectral channels 176 is reduced to 0.37%. As can be seen from the following discussion of the Fig. 21-23, such a mean absolute % error is less than the targeted mean absolute % error of 1% required for accurate spectral reconstruction. However, this reduction in the mean percentage error is only achieved at the expense of reducing the spacing of the optical detector regions 111 to 26 µm, thereby increasing the number of optical detector regions 111 required for each spectral channel 176 for a given FOV 142 to 15. This increases the electronic resources required to implement the optical detector regions 111 and increases the size and / or cost of the multispectral ALS 102. Consequently, a ratio of the FWHM intensity blur to the spacing of the optical detector regions 111 of about 3, in accordance with the Fig. 9A-9C and 10A-10C represent a good compromise between the accuracy of the spectral reconstruction and the size and / or cost of the multispectral ALS 102.

[0080] In each of the Fig. 21-23, the measured optical spectra of 16 different types of light sources are shown in blue. The corresponding reconstructed optical spectra, obtained from the measured signal values ​​of the different spectral channels 176, are shown in each of the Fig. 21-23 are shown in red, for the ideal case where the spectral channels 176 are perfectly aligned and the average percentage error in the acquired signals is 0%. The reconstructed optical spectra obtained for different average percentage errors in the acquired signals due to the different misalignments of the spectral channels 176 are shown in each of the Fig. shown in black. In particular, the Fig. 21 black reconstructed optical spectra with an average % error of ±1%, which are shown in Fig. 22 black reconstructed optical spectra have an average % error of ±2.5% and the Fig. 23 reconstructed optical spectra shown in black have an average % error of ±5%. Fig. 21-23 it can be seen that the mean absolute % error for an accurate spectral reconstruction should be less than about 1%.

[0081] Furthermore, the deviation of the acquired signals between the differently aligned spectral channels 176 depends on the structures or objects in the scene. Small spots with high intensity or hard dynamic structures can lead to a deviation of up to 20% in the acquired signals. Such deviations in the detected signals can be significant and lead to errors in the spectral reconstruction. Consequently, the simulations of the Fig. 13A to 20A for different scenes and for different ratios of the FWHM intensity blur to the distance of the optical detector areas 111.

[0082] For example, Fig. 13A shows a spatially modulated optical intensity distribution of an out-of-focus scene formed by applying a FWHM intensity blur of 110 µm to an output scene comprising a peripheral, vertically aligned, brighter stripe area of ​​constant intensity on a lower intensity background to obtain an out-of-focus scene, and by spatially modulating the out-of-focus scene by the simulated projected position distributions of the optical detector regions 111 of three differently misaligned spectral channels 176 projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102, using a mathematical model for each misaligned spectral channel 176, for optical detector regions 111 arranged in a 5x5 array with a pitch of 80 µm.and for three different misalignments between the aperture 160a and the lens array 170 of each misaligned spectral channel 176 relative to the corresponding sub-array 112 of the optical detector regions 111, namely misalignments of +10 µm in x, +10 µm in y, and -10 µm in x. The ratio between the FWHM intensity blur of 110 µm and the 80 µm separation of the optical detector regions 111 is 1.375.

[0083] In Fig. 13B, the dashed lines represent the simulated detected signal distributions for each of the three differently misaligned spectral channels 176, which correspond to the spatially modulated optical intensity distribution of Fig. 13A, and the solid lines represent the simulated detected signals for each of the three differently misaligned spectral channels 176 corresponding to the spatially modulated optical intensity distribution of Fig. 13A.

[0084] Fig. Figure 13C shows the percentage error in the signal for each optical detector area 111 in the middle row of optical detector areas 111 according to the solid lines in Fig. 13B for each of the three differently misaligned spectral channels 176. The mean absolute % error in the signals is 5.27%.

[0085] After calibration, data interpolation is used to convert the acquired signals from Fig. 13A-13C so that the measured misalignment of each spectral channel 176 is at least partially compensated, resulting in the detected signals of Fig. 14A-14C and a reduced mean absolute % error of 3.01% in the acquired signals due to the different misalignments of the spectral channels 176.

[0086] Fig. Figure 15A shows a spatially modulated optical intensity distribution of the same blurred scene described above with respect to Fig. 13A, formed by applying a FWHM intensity blur of 110 µm to an initial scene having a peripheral, vertically aligned, brighter stripe region of constant intensity on a lower intensity background to obtain a blurred scene, and by spatially modulating the blurred scene by the simulated projected position distributions of the optical detector regions 111 of three differently misaligned spectral channels 176 projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102, using a mathematical model of each misaligned spectral channel 176, for optical detector regions 111 arranged in an 11x11 array with a pitch of 36 µm,and for three different misalignments between the aperture 160a and the lens array 170 of each misaligned spectral channel 176 relative to the corresponding subarray 112 of the optical detector regions 111, namely misalignments of +10 µm in x, +10 µm in y, and -10 µm in x. The ratio between the FWHM intensity blur of 110 µm and the 36 µm spacing of the optical detector regions 111 is approximately 3.

[0087] In Fig. 15B, the dashed lines represent the simulated detected signal distributions for each of the three differently misaligned spectral channels 176, which correspond to the spatially modulated optical intensity distribution of Fig. 15A, and the solid lines represent the simulated detected signals for each of the three differently misaligned spectral channels 176 corresponding to the spatially modulated optical intensity distribution of Fig. 15A.

[0088] Fig. Figure 15C shows the percentage error in the signal for each optical detector area 111 in the middle row of optical detector areas 111 according to the solid lines in Fig. 15B for each of the three differently misaligned spectral channels 176. The mean absolute % error in the signals is 5.30%.

[0089] After calibration, data interpolation is used to extract the acquired signals from the Fig. to adjust so that the measured misalignment of each spectral channel 176 is at least partially compensated, resulting in the detected signals from the Fig. and a reduced mean absolute % error of 0.59% in the acquired signals due to the different misalignments of the spectral channels 176.

[0090] Fig. Figure 17A shows a spatially modulated optical intensity distribution of the same blurred scene described above with respect to Fig. 13A, formed by applying a FWHM intensity blur of 110 µm to an initial scene having a peripheral, vertically aligned, brighter stripe region of constant intensity on a lower intensity background to obtain a blurred scene, and by spatially modulating the blurred scene by the simulated projected position distributions of the optical detector regions 111 of three differently misaligned spectral channels 176 projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102, using a mathematical model of each misaligned spectral channel 176, for optical detector regions 111 arranged in a 15x15 array with a pitch of 26 µm,and for three different misalignments between the aperture 160a and the lens array 170 of each misaligned spectral channel 176 relative to the corresponding sub-array 112 of the optical detector regions 111, namely misalignments of +10 µm in x, +10 µm in y, and -10 µm in x. The ratio between the FWHM intensity blur of 110 µm and the 26 µm spacing of the optical detector regions 111 is approximately 4.1.

[0091] In Fig. 17B, ​​the dashed lines represent the simulated detected signal distributions for each of the three differently misaligned spectral channels 176, which correspond to the spatially modulated optical intensity distribution of Fig. 17A, and the solid lines represent the simulated detected signals for each of the three differently misaligned spectral channels 176 corresponding to the spatially modulated optical intensity distribution of Fig. 17A.

[0092] Fig. Figure 17C shows the percentage error in the signal for each optical detector area 111 in the middle row of optical detector areas 111 according to the solid lines in Fig. 17B for each of the three differently misaligned spectral channels 176. The mean absolute % error in the signals is 5.35%.

[0093] After calibration, data interpolation is used to convert the acquired signals from Fig. 17A-17C so that the measured misalignment of each spectral channel 176 is at least partially compensated, resulting in the detected signals of Fig. 18A-18C and a reduced mean absolute % error of 0.34% in the acquired signals due to the different misalignments of the spectral channels 176.

[0094] Fig. 19A shows a spatially modulated optical intensity distribution of a blurred scene formed by applying a FWHM intensity blur of 110 µm to an initial scene with a uniform intensity background to obtain a blurred scene, and the spatial modulation of the blurred scene by the simulated projected position distributions of the optical detector areas 111 of three differently misaligned spectral channels 176 projected onto corresponding different sectors of the scene in the FOV 142 of the multispectral ALS 102, using a mathematical model for each misaligned spectral channel 176, for optical detector areas 111 arranged in a 5x5 array with a pitch of 80 µm,and for three different misalignments between the aperture 160a and the lens array 170 of each misaligned spectral channel 176 relative to the corresponding sub-array 112 of the optical detector regions 111, namely misalignments of +10 µm in x, +10 µm in y, and -10 µm in x. The ratio between the FWHM intensity blur of 110 µm and the 80 µm separation of the optical detector regions 111 is 1.375.

[0095] In Fig. 19B, the dashed lines represent the simulated detected signal distributions for each of the three differently misaligned spectral channels 176, which correspond to the spatially modulated optical intensity distribution of Fig. 19A, and the solid lines represent the simulated detected signals for each of the three differently misaligned spectral channels 176 corresponding to the spatially modulated optical intensity distribution of Fig. 19A.

[0096] Fig. Figure 19C shows the percentage error in the signal for each optical detector area 111 in the middle row of optical detector areas 111 according to the solid lines in Fig. 19B for each of the three differently misaligned spectral channels 176. The mean absolute % error in the signals is 2.35%.

[0097] After calibration, data interpolation is used to convert the acquired signals from Fig. 19A-19C so that the measured misalignment of each spectral channel 176 is at least partially compensated, resulting in the detected signals of Fig. 20A-20C and a reduced mean absolute % error of 0.98% in the acquired signals due to the different misalignments of the spectral channels 176.

[0098] As can be seen from the description of Fig. 7A-20C, a ratio of the FWHM intensity blur to the distance of the optical detector areas 111 of approximately 3 is considered optimal for a variety of different scenes, since such a ratio can allow sufficient compensation of the measured misalignments of differently oriented spectral channels 176 for accurate spectral reconstruction without excessively increasing the number of optical detector areas for a given FOV 142.

[0099] One skilled in the art will appreciate that various modifications to the multispectral ALS 102 described above are possible. For example, each lens array 170 may include fewer or more than two lens elements 170a, 170b. In particular, each lens array 170 may consist of one lens element.

[0100] One or more of the lens elements 170a, 170b may comprise a Fresnel lens. One or more of the lens elements 170a, 170b may be defined or formed on a surface of the monolithic multispectral ALS semiconductor chip 110 above the corresponding optical filter 166.

[0101] In an alternative lens arrangement (not shown), the first and second MLAs 161a, 161b may be arranged such that the focal plane coincides with the optical detector regions 111, but the blurring is achieved by introducing one or more optical diffuser elements (not shown) into the optical path. It is also envisaged that a combination of this blurring technique with the method described with reference to Fig. described blurring technique can be used.

[0102] In each of the aforementioned embodiments, each lens assembly is disposed between the corresponding aperture and the corresponding optical filter, such that the light from the scene passes through the corresponding aperture before striking the lens assembly. In other embodiments, each aperture may be disposed between the corresponding lens assembly and the corresponding optical filter, such that the light from the scene passes through the corresponding lens assembly before striking the corresponding aperture. In other embodiments, each lens assembly defines the corresponding aperture.

[0103] Each optical detector area 111 may be square, rectangular, hexagonal, circular, or another shape.

[0104] Other arrangements of the optical detector regions 111 are possible within each subarray 112. For example, each subarray 112 can form a 1D or 2D array of optical detector regions 111 of any size.

[0105] The optical detector regions 111 of each sub-array 112 may be arranged in a square, rectangular, or non-rectangular 2D array. Each sub-array 112 may define a central optical detector region 111 surrounded by a plurality of peripheral arcuate optical detector regions 111. Each sub-array 112 may define a central optical detector region 111 surrounded by one or more concentrically arranged annular optical detector regions 111, each annular optical detector region 111 having a different radius.

[0106] Those of ordinary skill in the art will also understand that other arrangements of the subarrays 112 are possible. For example, the subarrays 112 may be arranged in a 1D or 2D array of any size. The subarrays 112 may be arranged in a 2D array of a different size than that shown in Fig. 4. The subarrays 112 may be arranged in a 2D pattern that is square, rectangular, or non-rectangular.

[0107] Although a multispectral ALS 102 has been described for use with the camera 104 of a smartphone 101, the multispectral ALS 102 can be used with the camera of any type of electronic device. For example, the multispectral ALS 102 can be used with a camera of a mobile phone, a cell phone, a tablet, or a laptop.

[0108] Although the disclosure has been described in terms of preferred embodiments, as set forth above, it is to be understood that these embodiments are illustrative only, and that the claims are not limited to these embodiments. Those skilled in the art will be able to make modifications and alternatives to the described embodiments in light of the disclosure that fall within the scope of the appended claims. Any feature disclosed or illustrated in the present description may be incorporated into any embodiment, either alone or in suitable combination with any other feature disclosed or illustrated herein.In particular, one skilled in the art will understand that one or more of the features of the embodiments of the present disclosure described above with reference to the drawings may produce effects or provide advantages when used in isolation from one or more of the other features of the embodiments of the present disclosure, and that combinations of features other than the specific combinations of features of the embodiments of the present disclosure described above are possible.

[0109] Those skilled in the art will understand that in the foregoing description and the appended claims, positional terms such as "above," "along," "lateral," etc., are used with reference to conceptual illustrations as illustrated in the accompanying drawings. These terms are used for convenience but are not to be construed as limiting. These terms should therefore be understood to refer to an object when it is in an orientation as illustrated in the accompanying drawings.

[0110] The use of the term "comprising" with respect to a feature of an embodiment of the present disclosure does not exclude other features or steps. The use of the term "a" or "an" with respect to a feature of an embodiment of the present disclosure does not preclude the embodiment from including a plurality of such features.

[0111] The use of reference signs in the claims is not to be understood as a limitation of the scope of the claims. LIST OF REFERENCE SYMBOLS 1 smartphone 2 multispectral ALS 4 Camera 8 cover glass; 11 optical detector areas; 20 housings 22 windows 30 diffusers 32 IR cut filters; 40 optical axis of the multispectral ALS; 42 Field of view of multispectral ALS; 50 optical axis of the camera; 52 Camera field of view; 101 smartphone 102 Multispectral optical sensor 104 Camera 108 cover glass 110 monolithic multispectral ALS semiconductor chip; 111 optical detector areas 112 Subarray of optical detector areas 120 housings 122 windows 132 IR cut filter 140 optical axis of the multispectral ALS 142 Field of view of multispectral ALS 150 optical axis of the camera 152 Camera field of view 160 aperture element 160a aperture 161a first microlens array 161b second microlens array 162 PCB 164 frame element 166 optical filters 170 lens arrangement 170a convex lens element 170b convex lens element 176 spectral channel 180 processing resources

Claims

[1] Multispectral optical sensor (102) comprising: a variety of apertures (160a); a plurality of lens arrangements (170); a plurality of optical filters (166); and a monolithic semiconductor chip (110) defining a plurality of sub-arrays (112) of optical detector regions (111), each sub-array (112) comprising the same number and relative spatial arrangement of optical detector regions (111), and the optical detector regions (111) of each sub-array (112) being arranged at a predetermined distance in a direction parallel to a surface of the semiconductor chip (110), wherein each subset of optical detector regions (111) is aligned with a corresponding optical filter (166), a corresponding lens arrangement (170) and a corresponding aperture (160a), and wherein the multispectral optical sensor (102) is configured such that light from a scene incident on one of the apertures (160a) along any direction of incidence is transmitted through the one of the apertures (160a), the corresponding lens arrangement (170), and the corresponding optical filter (166) to the corresponding sub-array (112) of optical detector regions (111) to generate a blurred image in a plane of the optical detector regions (111) of the corresponding sub-array (112) of optical detector regions (111), such that a ratio of a full width at half maximum (FWHM) of an optical intensity distribution of the blurred image to the predetermined spacing of the optical detector regions (111) in the direction parallel to the surface of the semiconductor chip (110) is greater than or equal to 2.0 and less than or equal to 4.

0. [2] Multispectral optical sensor (102) according to claim 1, wherein the ratio of a full width at half maximum (FWHM) of the optical intensity distribution of the blurred image to the predetermined distance of the optical detector regions (111) in the direction parallel to the surface of the semiconductor chip (110) is greater than or equal to 2.5 and less than or equal to 3.5 or substantially equal to 3. [3] A multispectral optical sensor (102) according to claim 1 or 2, wherein each optical filter (166) is disposed between the corresponding lens array (170) and the corresponding subarray (112) of optical detector regions (111) such that light from the scene passes through the corresponding lens array (170) before passing through the corresponding optical filter (166). [4] A multispectral optical sensor (102) according to any one of the preceding claims, wherein each lens assembly (170) is disposed between the corresponding aperture (160a) and the corresponding optical filter (166) such that light from the scene passes through the corresponding aperture (160a) before the light is incident on the corresponding lens assembly (170). [5] A multispectral optical sensor (102) according to any preceding claim, wherein each lens assembly is oriented to focus the incident light onto a focal plane adjacent to the plane of the optical detector regions to produce the out-of-focus image in the plane of the optical detector regions. [6] A multispectral optical sensor (102) according to any one of the preceding claims, wherein the focal plane of each lens array is arranged at a predetermined distance from the plane of the optical detector regions. [7] A multispectral optical sensor (102) according to any one of the preceding claims, comprising a plurality of diffuser elements, each diffuser element being configured to scatter the incident light to produce the blurred image in the plane of the optical detector regions. [8] Multispectral optical sensor (102) according to one of the preceding claims, wherein the plurality of lens arrays comprises one or more microlens arrays (MLA). [9] Multispectral optical sensor (102) according to one of the preceding claims, wherein each lens arrangement comprises one lens element or more than one lens element, for example two lens elements. [10] A multispectral optical sensor (102) according to any one of the preceding claims, wherein each lens arrangement comprises one or more Fresnel lenses. [11] Multispectral optical sensor system comprising: the multispectral optical sensor (102) according to any one of the preceding claims; and a processing resource (180), wherein the multispectral optical sensor (102) and the processing resource (180) are configured to communicate with each other, and wherein the processing resource (180) is configured to: Associating different electrical signals generated by different optical detector regions (111) of the same sub-array (112) of optical detector regions (111) with light incident on the optical multispectral sensor (102) from a scene along corresponding different directions of incidence, and Associating different electrical signals generated by corresponding optical detector regions (111) of different sub-arrays (112) of optical detector regions (111) with light incident from the scene along the same direction of incidence onto the optical multispectral sensor (102). [12] The multispectral optical sensor system of claim 11, wherein the processing resource (180) is configured to combine the electrical signal generated by an optical detector region (111) with the optical transmission spectrum of the corresponding optical filter (166). [13] A multispectral optical sensor system according to claim 11 or 12, wherein the processing resource (180) is configured to determine a classification of the ambient light source for each of a plurality of different incident directions based on a comparison between the electrical signal values ​​corresponding to each incident direction and predefined spectral data. [14] The multispectral optical sensor system of claim 13, wherein the predefined spectral data comprises a plurality of spectra, each spectrum corresponding to a different known type or kind of ambient light source. [15] Image acquisition system comprising: the multispectral optical sensor system according to one of claims 11 to 14; and an image sensor (104) having a known spatial relationship to the multispectral optical sensor (102), wherein the image sensor (104) and the processing resource (180) are configured to communicate with each other, and wherein the processing resource (180) is configured to adjust an image captured by the image sensor (104) based on the classification of the ambient light source for each direction of incidence. [16] The image acquisition system of claim 15, wherein the processing resource (180) is configured to adjust the image by white balancing the image based on one or more ambient light source classification parameters for each direction. [17] The image acquisition system of claim 16, wherein adjusting comprises gradient white balancing of the image based on one or more ambient light source classification parameters for each incident direction. [18] Electronic device (101) comprising at least one of the following elements: the multispectral optical sensor (102) according to one of claims 1 to 10, the multispectral optical system according to one of claims 11 to 14 or the image sensor system according to one of claims 15 to 17.

Images