spectrometer

Abstract

The application relates to a spectrometer, comprising a substrate and a photosensitive module, the substrate has a plurality of slit units arranged in a period for modulating incident light, the slit units form a transmission spectrum curve corresponding thereto, the photosensitive module is located at the lower end of the screen and comprises an image sensor for receiving the modulated incident light to obtain spectral information of the incident light, and the substrate is arranged on an optical path of the image sensor.

Description

Technical Field

[0001] This application relates to the field of spectral imaging technology, and more specifically, to a spectrometer. Background Technology

[0002] Various types of biometric systems are increasingly being used to provide enhanced security and / or improved user convenience. For example, fingerprint sensing systems, due to their small size, high performance, and widespread user acceptance, are widely used in various terminal devices, such as consumer smartphones. Currently, several fingerprint sensing systems are available on the market, such as those based on capacitive fingerprint modules and those based on optical fingerprint modules. While these types of fingerprint sensing systems can unlock devices, their application in mobile terminal fingerprint recognition allows criminals to steal user fingerprints and create fake fingerprints to bypass the user's security system. This actually increases the probability of fingerprint passwords being compromised on mobile terminals, posing a significant threat to the information security of mobile devices.

[0003] Therefore, it is necessary to enhance the security of traditional biometric systems, for example, by protecting them from attacks that exploit deceptive body parts, such as spoofed fingerprints, through liveness detection. Many solutions exist for liveness detection, such as hardware-based methods for identifying material properties, pulse detection via blood oxygen quantification, and software-based methods for identifying forged artifacts in the acquired fingerprint image and examining fine-scale textures.

[0004] However, existing live fingerprint recognition solutions all have certain drawbacks, such as overly complex structures, complicated recognition operations, or high fingerprint costs. Therefore, there is an urgent need for a simple and reliable fingerprint recognition solution to achieve live fingerprint recognition. Summary of the Invention

[0005] One advantage of this application is that it provides a spectrometer that can acquire spectral information by utilizing the modulation effects of diffraction and / or interference caused by the slit.

[0006] According to one aspect of this application, a spectrometer is provided, comprising:

[0007] A substrate having a plurality of slit units arranged periodically for modulating incident light, wherein the slit units have corresponding transmission spectrum curves;

[0008] A photosensitive module is located at the lower end of the screen and includes: an image sensor for receiving modulated incident light to obtain spectral information of the incident light, and a substrate disposed on the optical path of the image sensor.

[0009] In the spectrometer according to this application, each slit unit includes at least one slit and / or aperture.

[0010] In the spectrometer according to this application, the substrate is a screen.

[0011] In the spectrometer according to this application, the screen includes a glass cover and a light-emitting unit located below the glass cover.

[0012] In the spectrometer according to this application, the spectrometer further includes a light source, which is the light-emitting unit.

[0013] In the spectrometer according to this application, the photosensitive module further includes an optical component, the optical component including an aperture and at least one lens, the optical component being located on the photosensitive path of the image sensor.

[0014] In the spectrometer according to this application, the substrate is a modulation cover plate.

[0015] In the spectrometer according to this application, the modulation cover plate includes a glass cover plate made of a transparent material and an opaque material covering the glass cover plate, wherein the slit unit is formed in the portion of the modulation cover plate not covered by the opaque material.

[0016] In the spectrometer according to this application, the opaque material includes opaque, parallel conductive materials arranged in parallel to form a capacitor structure.

[0017] In the spectrometer according to this application, the opaque material includes an opaque, non-conductive material.

[0018] In the spectrometer according to this application, the spectrometer further includes a circuit board electrically connected to the image sensor, the circuit board being adapted to conduct electricity to the capacitor structure.

[0019] In the spectrometer according to this application, the modulation cover plate is a mask.

[0020] In the spectrometer according to this application, the modulation cover is a protective cover for an electronic device, the protective cover having a light-transmitting area and a non-light-transmitting area, the light-transmitting area forming the slit unit.

[0021] In the spectrometer according to this application, the photosensitive module includes a filter structure and an image sensor, wherein the filter structure is located on the photosensitive path of the image sensor.

[0022] In the spectrometer according to this application, the spectrometer further includes a filter located on the photosensitive path of the image sensor.

[0023] In the spectrometer according to this application, any one of the slit units and its two adjacent slit units define two vectors and a region with an area equal to the dot product of the two vectors. After the pattern of the region is translated by an integer number of vector displacements along the vector directions corresponding to the two vectors within the periodic region, the slit of the region coincides with the slit of the translated region. The periodic region is a region formed by a plurality of slit units arranged periodically. Attached Figure Description

[0024] Various other advantages and benefits of this application will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0025] Figure 1 The illustration shows a schematic diagram of the reflectance spectral data corresponding to real human fingers and fingerprint materials, using silicone and human skin as examples.

[0026] Figure 2 The illustration shows a schematic diagram of the screen of the live fingerprint recognition system according to the present invention.

[0027] Figure 3A This is a schematic diagram of spectral information directly acquired by an image sensor.

[0028] Figure 3B This is a schematic diagram of the spectral information obtained by the image sensor after modulation through the OLED screen according to the present invention.

[0029] Figure 4A This is a light intensity information map of a portion of the image sensor corresponding to an incident light wavelength of 450nm.

[0030] Figure 4B This is a light intensity information map of the same area of ​​the image sensor corresponding to the incident light at the 580nm wavelength.

[0031] Figure 5 The diagram illustrates an OLED screen with distributed R, G, and B light-emitting units.

[0032] Figure 6A This is a schematic diagram of a first example of a slit and / or aperture in an OLED screen according to the present invention.

[0033] Figure 6BThis is a schematic diagram of a second example of a slit and / or aperture in an OLED screen according to the present invention.

[0034] Figure 7A and Figure 7B This is a schematic diagram of the imaging optical path of the OLED screen according to the present invention.

[0035] Figure 8A The illustration shows a schematic diagram of a first variant example of an OLED screen according to the present invention.

[0036] Figure 8B A schematic diagram illustrating a second variation example of an OLED screen according to the present invention is shown.

[0037] Figure 8C The illustration shows a schematic diagram of a third variation of an OLED screen according to the present invention.

[0038] Figure 9 The illustration shows a schematic diagram of a variant embodiment of the live fingerprint recognition system according to the present invention.

[0039] Figure 10 The illustration shows a working example of a filter in a live fingerprint recognition system.

[0040] Figure 11 The illustration shows a schematic diagram of pixel merging in the image sensor of the live fingerprint recognition system according to the present invention.

[0041] Figure 12 The illustration shows a flowchart of a first example of a liveness detection method according to the present invention.

[0042] Figure 13 This is a schematic diagram of the neural network model of the present invention.

[0043] Figure 14 A flowchart illustrating a second example of the liveness detection method according to the present invention is shown.

[0044] Figure 15 The diagram shows... Figure 14 The flowchart shows the liveness detection and object recognition steps in the method shown.

[0045] Figure 16 The illustration shows a schematic diagram of a first example of a spectral pixel array of an image sensor according to the present invention.

[0046] Figure 17 A schematic diagram of a second example of a spectral pixel array of an image sensor according to the present invention is illustrated.

[0047] Figure 18 The illustration shows a schematic diagram of the live fingerprint recognition process according to the present invention.

[0048] Figure 19 The illustration shows a schematic diagram of the region of interest according to the present invention.

[0049] Figure 20 The illustration shows a schematic diagram of a spectroscopic device according to an alternative embodiment of this application.

[0050] Figure 21 The diagram shows... Figure 20 A schematic diagram illustrating an example of the structure of the modulation cover plate of the spectral device shown.

[0051] Figure 22 The diagram shows... Figure 20 A schematic diagram of another example of the structure of the modulation cover plate of the spectral device shown.

[0052] Figure 23 The diagram shows... Figure 20 The diagram shows a schematic representation of the configuration of the spectral device, including the optical components.

[0053] Figure 24 This is a diagram of the back of an existing mobile phone.

[0054] Figure 25 This is a schematic diagram of the back of a mobile phone with a protective cover according to this embodiment. Detailed Implementation

[0055] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0056] Application Overview

[0057] Because human skin contains physiological features such as capillaries (blood) and sweat pores, it is relatively more difficult to forge than fingerprint patterns. Furthermore, these physiological features cause the skin to absorb and reflect different wavelengths of light differently. This means that liveness detection can be achieved by analyzing the spectral information reflected from the skin. Specifically, reflectance spectral tests on real fingers and fingerprint molds show a significant difference in the reflectance spectra of real fingers and fingerprint molds within the 300nm-1100nm wavelength range. Figure 1 The illustration shows a schematic diagram of the reflectance spectral data corresponding to real human fingers and fingerprint materials, using silicone and human skin as examples. Figure 1 As shown, the difference between the two is significant; therefore, it is feasible to determine the liveness of an organism by analyzing the received reflectance spectrum.

[0058] Exemplary System

[0059] Based on the above theory, this invention provides a live fingerprint recognition system. The system includes a screen and a photosensitive module located at the lower end of the screen. The screen has multiple slits (holes). When light from a light source is projected onto the finger to be tested and reflected, incident light is generated. This incident light undergoes diffraction and / or interference after passing through the slits of the screen, and is then received by the photosensitive module to acquire spectral information. By processing the spectral information received by the image sensor, liveness information and image information of the fingerprint can be obtained, thereby achieving liveness recognition and fingerprint pattern image recognition. The screen can be an LCD screen, OLED screen, microLED screen, ULED screen, etc. Traditional under-display fingerprint recognition systems typically require algorithmic correction to avoid screen interference, including but not limited to diffraction and / or interference phenomena, so that the image information received by the photosensitive module cannot be used for fingerprint recognition. Generally, the screen, especially the corresponding slit, the photosensitive module, and the correspondence between the screen and the module are designed and adjusted to suppress diffraction and / or interference as much as possible. However, this invention can utilize the modulation effect of diffraction and / or interference brought about by the slit structure of the screen, which is equivalent to spatial dispersion modulation of the incident light. The photosensitive module then receives the modulated fingerprint image information to obtain spectral information, and the liveness detection is achieved by processing the spectral information. Figure 2 The illustration shows a schematic diagram of the screen of the live fingerprint recognition system according to the present invention.

[0060] Specifically, this invention uses an OLED screen as an example. This screen can be used for display and as a light source to project light onto the object under test (finger). Furthermore, because the OLED screen has slits, it can diffract and / or interfere with the incident light reflected from the object under test, thereby achieving spatial dispersion modulation of the incident light. The photosensitive module includes an image sensor, which can be implemented as an imaging chip such as a CMOS chip or a CCD chip. Preferably, the physical pixels on the image sensor are all black and white pixels (i.e., without a Bayer array). The incident light is modulated by the OLED screen and then received by the image sensor to obtain image information, and then spectral information. Processing the spectral information can determine if the subject is alive. It should be understood that the information received by the image sensor includes both information that can be used for fingerprint imaging and spectral information used for liveness detection.

[0061] Furthermore, to highlight the role of the slit in the OLED screen in this invention, the following is provided: Figure 3A and Figure 3B .in, Figure 3AThis is a schematic diagram of the spectral information directly acquired by the image sensor. As shown in the figure, the light projected from the light source reaches the finger being tested and is reflected to generate incident light. This incident light is directly received by the image sensor, which displays spectral information. The spectral information across different regions of the image sensor is relatively uniform; this means the incident light is not modulated by the OLED screen. Figure 3B This is a schematic diagram illustrating the spectral information acquired by the image sensor after modulation by an OLED screen according to the present invention. That is, the same incident light modulated by the OLED screen produces spectral information. A comparison reveals that the information displayed by the two is completely different; relatively speaking, the image information received by the image sensor after modulation by the OLED screen contains more information. Specifically, the spatial dispersion adjustment of the incident light by the screen results in the inclusion of the light's dispersion characteristics, i.e., spectral characteristics (spectral information), in the image information. Therefore, spectral information can be extracted from this information and used to achieve liveness detection.

[0062] To further illustrate the advantages of the slit, test images are provided. Different wavelengths of incident light, after passing through the OLED screen, will display different patterns on the image sensor, meaning that the OLED screen has different modulation effects on different bands of incident light. Figure 4A This is a light intensity map of a portion of the image sensor corresponding to an incident light wavelength of 450nm. Figure 4B The light intensity information diagrams for the same area of ​​the image sensor with incident light at the 580nm wavelength clearly show the difference between the two.

[0063] Specifically, this invention is based on the fact that OLED screens can achieve diffraction and / or interference. Preferably, the OLED screen described in this invention produces interference effects as much as possible. Therefore, the design of the screen or slit needs to be considered. It should be understood that OLED screens will have R, G, and B light-emitting units distributed according to demand patterns, for example, as... Figure 5As shown, conventionally, R, G, G, B light-emitting units are arranged in an array as a group. Slits are formed between these light-emitting units. Multiple slits are formed between a group of RGGB light-emitting units (as shown in the white box in the figure, which can be understood as a group of light-emitting units). Defining multiple slits as a slit unit means that the slit units need to be arranged with a fixed period, i.e., the distance (period) between adjacent slit units is equal (this characteristic only needs to be ensured for the screen area participating in the modulation of the incident light). This ensures that the interference effect is as obvious as possible, thereby allowing the image information received by the image sensor to contain spectral characteristics. It should be understood that this invention does not limit the screen to an RGGB array arrangement; the screen can be arranged in other ways, and the slit units can be adjusted accordingly. In this case, at least one slit contained in the smallest repeating light-emitting unit of the screen can be defined as a slit unit. Preferably, the slit arrangement of multiple slit units is consistent. It should be understood that different slit units do not need to be completely identical, but their differences should not be too large to avoid interference effects. Here, Figure 5 The diagram illustrates an OLED screen with distributed R, G, and B light-emitting units.

[0064] To better understand, let's further explain the slit unit. OLED screens have a pixel layer (light-emitting layer) with light-emitting units, a TFT structure in the circuit layer, and a reflective layer (generally removed in under-display module solutions). These layers prevent incident light from passing through. Slits and / or holes exist between pixels (light-emitting units) and between TFT structures, allowing incident light to pass through. These light-transmitting slits and holes are periodic within a certain range. For example, they could be periodic across the entire screen, or within the test area corresponding to the photosensitive module, or in other areas. That is, at least one slit and / or hole constitutes a slit unit, and any slit unit can define a vector a and a vector b with its two adjacent slit units. This means we can find vector a, vector b, and a region with an area equal to the dot product of a and b (the area of ​​the parallelogram formed by vectors a and b). Within this periodic region, after translating an integer number of vectors a and b along the corresponding vector directions, the slits and / or holes will essentially coincide. Each slit unit and its two adjacent slit units define two vectors and a region with an area equal to the area of ​​the two vectors. After the pattern of this region is translated by an integer number of vector displacements along the directions corresponding to the two vectors within a periodic region, the slit in this region coincides with the slit in the translated region. The periodic region is formed by multiple slit units arranged periodically. The periodic region has at least 25 slit units. Generally, the angle between vector a and vector b is 90 degrees. Specifically... Figure 6A , Figure 6BThe diagram shows two different OLED screen slits and / or pinholes. The bright areas represent the OLED slits and / or pinholes, while the outlined rectangular area represents the slit unit. Of course, the slit unit area will differ for different screens, but is limited to a rectangular area. It should be noted that the slits between slit units in this invention may be different from each other, but the shape, structure, and size of the slit units are basically consistent. That is, preferably, the shape and / or structure and / or size of multiple slit units are consistent. However, due to certain errors during processing, there may be some differences between slit units, which can also be understood as being consistent with the concept of this invention and covered by this invention. Here, Figure 6A This is a schematic diagram of a first example of a slit and / or aperture in an OLED screen according to the present invention, and Figure 6B This is a schematic diagram of a second example of a slit and / or aperture in an OLED screen according to the present invention.

[0065] Furthermore, let x(λ) denote the intensity signals of the incident light at different wavelengths λ, and let T(λ) denote the transmission spectrum curve formed by the slit units of the OLED screen, which can be denoted as Ti(λ) (i=1,2,3,…,m); then at least some physical pixels of the image sensor acquire the spectral information bi modulated by the OLED screen; then

[0066] bi=∫x(λ)*Ti(λ)*R(λ)dλ

[0067] Where R(λ) is the response of the image sensor;

[0068] Specifically, the light intensity information received by all physical pixels of an image sensor includes image information and spectral information. The spectral information can be processed to determine the presence of a living organism, while the corresponding image information is used for imaging.

[0069] It is important to note that, in order to ensure that the image sensor obtains as much spectral information as possible, the transmission spectrum curves Ti(λ) (i = 1, 2, 3, ..., m) should ideally satisfy the condition that at least two transmission spectrum curves have a correlation of less than or equal to 0.4. This correlation can be defined using the Pearson correlation coefficient. Furthermore, regarding the definition of the transmission spectrum curve in this invention, it should be understood that the existence of the transmission spectrum curve is mainly due to the presence of slits in the OLED screen. Incident light passing through the slits is modulated, and the transmission spectrum curve can be considered to determine the modulation effect on the incident light. Therefore, multiple slits forming a slit unit will each have a corresponding transmission spectrum curve. However, in this invention, the transmission spectrum curve is preferably not determined by a single slit unit; it may be influenced by surrounding slit units. That is, the transmission spectrum curve in this invention is preferably determined by at least two slit units. Furthermore, the number of transmission spectrum curves is equal to the number of effective light intensities bi. Effective light intensities bi refer to the light intensity information used for spectral recovery or spectral response judgment, and their number n is equal to the number of transmission spectrum curves. In general applications, the incident light is sampled discretely and uniformly, with a total of n sampling points. For example, in the 200-400nm band, the spectral resolution is 1nm, so the number of sampling points is 201. At this time, the transmission spectrum matrix formed by the transmission spectrum curves is an n*m matrix.

[0070] Furthermore, to facilitate understanding, a detailed example is provided, as shown in Figure 7. Assume that the thickness of the glass cover of the OLED screen is A, the refractive index is n, the photosensitive module has a beam-shrinking system (lens group) with a beam-shrinking ratio of N:1; the image sensor of the photosensitive module has an image plane pixel size of P, an LED array structure size of D, a distance from the array to the aperture of L, and a field of view of the image plane pixel of K.

[0071] It is important to note that the cover plate thickness A, the pixel field of view K, the distance L from the optical component (lens group) to the slit unit, and the beam-shrink ratio N of the lens group are intercoupled and will be affected by the parameters of the beam-shrinking system (lens group). All the following discussions are based on the assumption that the paraxial approximation holds true.

[0072] Based on the beam contraction ratio, the divergence angle of the reflected or transmitted light corresponding to the object under test is K / N.

[0073] The diameter of the incident light spot at the slit unit is d = (K / N) * A / n.

[0074] The period of the slit unit covered by the light spot is d / D, and it is speculated that this value needs to be at least 2-5.

[0075] If a pixel moves one grid, the incident light at the slit unit will move horizontally a distance N*P.

[0076] The angle deflection of the main ray is m = N*P / (L+A / n), and the position of the light spot at the LED array moves by v = m*L.

[0077] According to interference theory, the period of the interference fringes should be c = 2λ / D.

[0078] For spectral differences to appear between pixels, m must not be much smaller than c. It is speculated that m should be greater than c / 20, and preferably greater than c / 12.

[0079] v is generally not much smaller than D, such as v > D / 6.

[0080] Figure 7A and Figure 7B This is a schematic diagram of the imaging optical path of the OLED screen according to the present invention.

[0081] Furthermore, under-display fingerprint recognition systems also need to address issues such as stray light and improving fingerprint image resolution. Therefore, a typical under-display fingerprint recognition system also includes a filter. This filter filters the incident light, allowing specific wavelengths of light to enter or blocking them; for example, the filter can block light with wavelengths above 600nm. The filter can be located between the screen and the photosensitive module, or it can be placed within the photosensitive module. Under certain conditions, the filter can isolate stray light from ambient light and improve fingerprint image resolution.

[0082] The photosensitive module may further include an optical component, which includes at least one lens, and the optical component is located on the photosensitive path of the image sensor; the optical component may further include an aperture stop, which is used to limit the angle of incident light, thereby preventing stray light from entering the photosensitive module, such as... Figure 8A As shown. Figure 8A The illustration shows a schematic diagram of a first variant example of an OLED screen according to the present invention.

[0083] Furthermore, the OLED screen includes a glass cover and a light-emitting unit located at the lower end of the glass cover. During the recognition process, the object to be tested needs to be placed on the glass cover. The light-emitting unit projects a beam of light onto the object, which is then reflected by the object to generate incident light. The incident light is modulated by a slit in the OLED screen and then received by an image sensor to obtain spatially dispersed image information, thereby obtaining spectral information. However, it should be noted that some of the projected light A directly enters the slit and reaches the image sensor; some of the projected light B reaches the glass cover and is directly reflected into the slit before being received by the image sensor; some of the projected light C reaches the object to be tested (finger) and is reflected into the slit before being received by the image sensor; and some of the projected light D is absorbed by the object to be tested (finger). Liveness detection is based on the fact that fingers, due to the presence of capillaries and sweat glands, absorb different wavelengths of light differently. This results in inconsistent reflected light from the finger under the same light source, unlike the absorption of projected light by conventional silicone or fake fingers. This difference can be used to determine liveness. Therefore, the truly useful projected light should be projected light C and projected light D, while projected light A, projected light B, and ambient light are, to some extent, stray light. Figure 8B As shown. Figure 8B A schematic diagram illustrating a second variation example of an OLED screen according to the present invention is shown.

[0084] Therefore, in a dark room environment or by covering the test area with black light-absorbing elements, the light-emitting unit can project the same light. In this case, the incident light received by the image sensor can be essentially considered as projection light A and projection light B passing through the slit and being received by the image sensor. The spectral information acquired under this condition is recorded as the reference spectral information. Subtracting this reference spectral information from the spectral information obtained in subsequent tests on the object under test removes stray light from projection light A and projection light B. Figure 8C As shown. Figure 8C The illustration shows a schematic diagram of a third variation of an OLED screen according to the present invention.

[0085] Variant Implementation

[0086] Unlike the embodiments described above, the photosensitive module includes a filter structure and an image sensor. The filter structure is located on the photosensitive path of the image sensor and is a broadband filter structure in the frequency or wavelength domain. The pass spectra of different wavelengths of the filter structures are not entirely the same. The filter structure can be a metasurface, photonic crystal, nanopillar, multilayer film, dye, quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties. For example, in the embodiments of this application, the filter structure can be the light modulation layer in Chinese patent CN201921223201.2. Further, the spectral device includes an optical system located on the photosensitive path of the image sensor. Light is adjusted by the optical system and then modulated by the filter structure before being received by the image sensor to obtain the spectral response. The optical system may be a lens assembly, a light homogenizing assembly, or other optical systems. The image sensor can be a CMOS image sensor (CIS), a CCD, an array photodetector, or the like. In addition, the spectral device also includes a data processing unit, which may be a processing unit such as an MCU, CPU, GPU, FPGA, NPU, or ASIC, and can export the data generated by the image sensor to an external source for processing.

[0087] It should be noted that the filter structure also has a transmission spectrum matrix A. In this case, the transmission spectrum matrix T of the entire live fingerprint recognition system is determined by the transmission spectrum matrix A of the filter structure, the transmission spectrum matrix T of the OLED screen, and the response of the image sensor. The incident light will pass through the OLED screen and the filter structure to reach the image sensor and be modulated. Figure 9 The illustration shows a schematic diagram of a variant embodiment of the live fingerprint recognition system according to the present invention.

[0088] Identification scheme

[0089] It should be noted that, in principle, the OLED screen and the image sensor can constitute an optical system, that is, all the physical pixels on the image sensor can acquire image information and spectral information. That is, after the OLED screen modulates the incident light, it is received by the image sensor to obtain the corresponding light intensity information, which can be used to recover the spectral curve or to form an image; preferably, the optical path of the image sensor also includes an optical component, which is implemented as a lens group.

[0090] However, in live fingerprint recognition applications, it is necessary to extract physical pixels containing spectral information of strong liveness or material. Therefore, it is further necessary to determine spectral pixels, that is, among all physical pixels of the image sensor, based on the characteristics of the OLED screen, the corresponding physical pixels are selected and defined as spectral pixels. Spectral pixels can be understood as physical pixels that respond more significantly to a certain wavelength band. Specifically, this embodiment provides a method for determining spectral pixels (also called a calibration method). A first light-emitting unit emits light, and a test piece with low refractive index or high reflectivity, such as black paper, black rubber (low refractive index material), or white paper (high refractive index material), is placed on the glass cover of the OLED screen. The image sensor receives the first spectral response data of the first light-emitting unit after modulation by the slit unit of the OLED screen. Physical pixels with strong light intensity corresponding to the first spectral response data are extracted and recorded as the first position corresponding to the image sensor, for example, n points are extracted. Then, a second light-emitting unit is projected, and the second spectral response data is received. Physical pixels with strong spectral response are extracted and recorded as the second position corresponding to the image sensor. The first and second positions are compared, and points overlapping on the image sensor are removed from the first position. The remaining physical pixels are defined as spectral pixels. It can be understood that the transmission spectrum matrix corresponding to a spectral pixel has a higher transmittance for a specific wavelength band of light, i.e., the modulation effect is better. Furthermore, other lights can be projected to further filter the spectral pixels.

[0091] For example, fingerprint liveness detection systems typically incorporate filters that cut off wavelengths above 600nm. This means the system performs liveness detection in the 400-600nm range. Furthermore, since fingers are relatively more sensitive to blue light, green and blue light can be used for calibration. The first light-emitting unit emits blue light, and the second emits green light. By emitting green light from the second unit, physical pixels that receive both blue and green light strongly are removed. The remaining positions correspond to spectral pixels, which only have a high transmittance to blue light. This process can be understood as blue and green light calibration. Figure 10 The illustration shows a working example of a filter in a live fingerprint recognition system.

[0092] In the fingerprint liveness detection stage, only mixed light, such as white light (with R, G, and B light-emitting units emitting light simultaneously), is projected onto the finger. This mixed light is absorbed and reflected by the finger, and the reflected light enters the slit unit of the OLED screen, is modulated, and then received by the image sensor. The image sensor can be understood as consisting of non-spectral pixels and spectral pixels. The spectral pixels acquire spectral information, which is used to determine whether the person is alive. It's important to understand that although the OLED screen interferes with and / or diffracts the incident light, the non-spectral pixels still acquire strong texture information, enabling fingerprint imaging. To achieve both imaging and liveness detection, spectral pixels account for 10-25% of the physical pixels in the image sensor. The value acquired by the selected spectral pixel is then divided or subtracted from the average value of its eight neighboring physical pixels to extract the modulation difference (equivalent to spectral features), converting broad-spectrum information into narrow-spectrum information. The key comparison utilizes the narrow-spectrum information of the spectral pixels. Generally, the spectra that distinguish "living" materials are blue light information (narrowband information) in the 400-500nm range or green light information (narrowband information) in the 500-600nm range.

[0093] Accordingly, this application provides a liveness detection method, comprising: determining spectral pixels and non-spectral pixels of an image sensor; projecting mixed light onto a target object; acquiring image data of the target object through the non-spectral pixels and acquiring spectral data of the target object through the spectral pixels; and performing liveness detection and object recognition based on the spectral data.

[0094] In determining the spectral and non-spectral pixels of an image sensor, firstly, a low-refractive-index material or a high-reflectivity test piece is placed on the glass cover of the screen, and the first light-emitting unit of the screen is controlled to emit a first pre-calibrated light (e.g., blue light); then, the first spectral response data corresponding to the first light-emitting unit is received by the image sensor; next, physical pixels corresponding to response data exceeding a preset value in the first spectral response data (i.e., physical pixels with stronger light intensity corresponding to the first spectral response data, which are physical pixels with stronger spectral response to the reflected light corresponding to the first light-emitting unit) are extracted from the image sensor, and the position of the physical pixel corresponding to the response data exceeding the preset value in the first spectral response data is recorded as the first position; next, the second light-emitting unit of the screen is controlled to emit a second pre-calibrated light (e.g., green light); then, the image sensor... The system receives second spectral response data corresponding to the second light-emitting unit; then, it extracts physical pixels in the image sensor that correspond to response data exceeding a preset value in the second spectral response data (i.e., physical pixels with stronger light intensity corresponding to the second spectral response data, which are physical pixels with stronger spectral response to the reflected light corresponding to the second light-emitting unit), and records the position of the physical pixel corresponding to the response data exceeding the preset value in the second spectral response data as the second position; subsequently, it determines that the physical pixels overlapping the physical pixels at the first position and the second position in the image sensor are non-spectral pixels, and determines that the physical pixels other than the non-spectral pixels in the image sensor are spectral pixels, that is, it determines that the physical pixels other than the physical pixels overlapping the physical pixels at the first position and the second position in the image sensor are spectral pixels.

[0095] Offset compensation scheme

[0096] Because this invention utilizes the interference and / or diffraction of the screen's slits, the screen modulates the incident light, allowing the image sensor to acquire spectral and image information. Generally, the screen and image sensor need to remain relatively stable; otherwise, the parameters (or modulation effect) of the entire system will change, leading to inaccurate test results. Furthermore, a method for determining whether there is a shift is provided, including:

[0097] 1. Pre-store the position information of reference physical pixels. By placing white or black paper on the glass cover of the screen, and projecting light (the type of light used during operation), which can be mixed light, such as white light, or monochromatic light, the image sensor receives the light intensity information and selects the N points with the strongest light intensity (the brightest points), where N is greater than or equal to 2, for example, 100 points. Record the position information of these N points on the image sensor, for example, denote the N points as a reference position array ai(x,y), i = 1, 2, 3...N, such as N = 100.

[0098] 2. Determine if there is an offset. When the object to be tested is placed on the screen, the image sensor obtains the corresponding light intensity information and selects the position information of the N physical pixels with the strongest light intensity on the image sensor. Record the position array bi(x,y) of these N brightest physical pixels, i=1,2,3...N. Compare this position array with the pre-stored reference position array of physical pixels. If more than 80% of the position information changes, it is determined that an offset has occurred, and the offset self-calibration step is initiated.

[0099] 3. Offset Calibration Scheme 1: Compare the position array bi(x,y) with the reference position array ai(x,y). For example, the comparison method is subtraction, such as ci(x,y) = bi(x,y) - ai(x,y). Statistically analyze the array values ​​of ci(x,y). If a translation occurs, the adjusted values ​​X and Y can be fixed values ​​or statistically significant quantities. Apply X and Y to the selected points marked by the blue and green cursors. These newly compensated blue and green cursor-marked physical pixels replace the pre-stored physical pixels and are used for subsequent liveness detection.

[0100] 4. Offset calibration scheme 2: Prompt and guide the terminal user to recalibrate the blue and green cursors.

[0101] Merger Plan

[0102] In some solutions, physical pixels on the image sensor are binning to reduce computation and improve the signal-to-noise ratio. However, for this solution, if the physical pixels of the image sensor are binning, spectral information may not be extracted. Therefore, this invention further provides an image sensor in which each physical pixel individually receives and / or outputs light intensity information.

[0103] Preferably, the present invention also provides an image sensor, which is divided into a merged region and a non-merged region. The physical pixels corresponding to the merged region are merged, which can be done by merging 2*2, 3*3, or n*m pixels to output light intensity information. For the non-merged region, individual physical pixels receive and / or output light intensity information. Preferably, the non-merged region is located in the middle region of the image sensor. Figure 11The illustration shows a schematic diagram of pixel merging in the image sensor of the live fingerprint recognition system according to the present invention.

[0104] Furthermore, the location of physical pixels that function as spectral pixels on the image sensor can be determined, and light intensity information can be received and / or output separately for physical pixels at that location, while the remaining physical pixels are merged.

[0105] like Figure 18 As shown, the entire recognition process is as follows:

[0106] First, data acquisition is performed: acquiring the baseline fingerprint information, baseline image information, and identification information of the object to be detected; then, offset detection is performed: determining whether there is an offset between the identification information and the baseline image information based on the baseline fingerprint information, the baseline image information, and the identification information; next, identification data optimization processing is performed: in response to the absence of an offset between the identification information and the baseline image information, noise reduction is performed on the baseline fingerprint information and the identification information; optionally, the noise-reduced baseline information and the noise-reduced identification information are normalized; the baseline fingerprint information and the identification information can also be de-merged; subsequently, based on the correlation between the optimized baseline information and the optimized identification information and a preset threshold, it is determined whether the object to be detected is a live organism.

[0107] Data acquisition and offset detection process:

[0108] The recognition system pre-stores a base image information, also known as a reference image. A low-refractive-index or high-reflectivity test piece, such as black paper or black rubber (a low-refractive-index material), is placed on the glass cover of the screen. A light source, such as the light from an OLED screen, is then turned on. Part of the light emitted by the light source is absorbed by the test piece, while the unabsorbed light enters the slit unit of the screen, is modulated, and then received by the image sensor to obtain spectral response data. This spectral response data is recorded as the base image information. The light emitted by the light source needs to be consistent with the light projected during the actual recognition process; that is, the wavelength of the light emitted by the light source when acquiring the base image is essentially the same as the wavelength of the light projected during actual recognition. For example, if white light and / or blue light are used for fingerprint recognition in this example, then white light and / or blue light should also be used during the acquisition of the base image information. It is important to understand that in actual fingerprint recognition, the base image information is already burned into the recognition system.

[0109] Furthermore, if the light source needs to project different types of light twice during the recognition process, then at least two base images should be obtained, each acquired under different types of light. For example, in live fingerprint recognition, white light and monochromatic light, such as blue light, are projected twice. In the process of acquiring the base image information, the light source should also project white light and the corresponding blue light, and record the spectral response data under different types of light, denoted as white light base image information and blue light base image information.

[0110] It should be noted that the amount of data in the base image information can be approximately equal to the number of physical pixels of the image sensor; alternatively, a specific region on the image sensor can be selected as the base image information, such as the central region of the image sensor. This method can be called a ROI (region of interest) region. In some embodiments, the ROI region can be selected from the edge regions of the image sensor, for example... Figure 19 As shown, these are located in the four corner areas of the image sensor. Alternatively, physical pixels from different locations can be extracted to form the base image information based on requirements.

[0111] In the process, the user first needs to input the baseline information (baseline fingerprint information) of the object to be detected, such as the fingerprint pattern and the corresponding spectral response information. For greater accuracy, at least three inputs or at least three baseline information images are required. For example, in a valid input set (e.g., 10 inputs), the correlation coefficient R of each spectral feature parameter is calculated with the other nine. The lowest correlation coefficient R_min is taken and used to calculate the judgment threshold R_t for that input using a specific formula with the system-set parameter k. In actual use, the correlation coefficient between the parameter to be tested and the 10 input data is calculated separately and compared with its corresponding judgment threshold R_t (1~10). When nine or more are greater than the corresponding judgment threshold, the input is considered successful.

[0112] The reference information is compared with the base map information to detect any offset. If there is no offset, the reference information is burned into the recognition system. If there is an offset, it needs to be corrected. Optionally, the offset detection can be performed on the corrected information before the reference information is burned into the recognition system. In some embodiments, if an offset occurs, the base map information needs to be re-acquired.

[0113] Furthermore, during the unlocking process, the user places the object to be detected, such as a finger or palm, on the screen. The light source projects corresponding light, and the image sensor receives the light reflected by the object to be detected, which is modulated by the slit unit of the screen, to obtain corresponding identification information, such as the fingerprint pattern of the object to be detected and the spectral response information corresponding to the fingerprint.

[0114] The information to be identified is compared with the base image information to determine if there is any offset.

[0115] For offset identification and detection of baseline information and information to be identified based on base map information, the root mean square error (RMSE) can be used for evaluation. For example, the baseline information a... nm Information to be identified b nm and base graph information c nm The matrix is ​​subjected to root mean square error (RMSE) calculation. Specifically, first, the baseline RMSE value between the input information (reference information or information to be identified) and the base image information is calculated; then, the input information and the base image are offset relative to each other, and the offset RMSE value is calculated. If the baseline RMSE value is less than the offset RMSE value, it is determined that no offset has occurred; if the baseline RMSE value is greater than or equal to a certain offset RMSE value, it is determined that an offset has occurred. That is, in the process of determining whether there is an offset between the information to be identified and the reference image information based on the reference fingerprint information, the reference image information, and the information to be identified, first, the root mean square error value between the reference fingerprint information and the reference image information is calculated to obtain the baseline root mean square error value; then, the root mean square error value between the information to be identified and the reference image information is calculated to obtain the offset root mean square error value; in response to the baseline root mean square error value being less than the offset root mean square error value, it is determined that no offset has occurred between the information to be identified and the reference image information; in response to the baseline root mean square error value being greater than or equal to the offset root mean square error value, it is determined that an offset has occurred between the information to be identified and the reference image information. The offset of the above-mentioned input information and base image can be obtained by offsetting the input information or base image information vertically, horizontally, or left-right. For example, offsetting by one physical pixel in each of the four directions to obtain the offset input information, and then calculating the RMSE value with the base image information.

[0116] The formula for calculating the RMSE value corresponding to the baseline information is as follows.

[0117]

[0118] The formula for calculating the RMSE value corresponding to the information to be identified is as follows:

[0119]

[0120] Identification data processing process:

[0121] For the recorded information that has not shifted after detection, further processing is performed to facilitate liveness detection. This involves subtracting the baseline information and the information to be identified from the base image information to perform debase (noise reduction), thus removing background noise. For example, the baseline information and the information to be identified can be converted into vectors or matrices, and then subtracted from the corresponding base image information to obtain the denoised baseline information and the information to be identified.

[0122] Furthermore, optionally, the noise-reduced reference information and the information to be identified can be normalized, for example, by dividing each data point of the noise-reduced reference information and the information to be identified by the largest value in the corresponding information; or by dividing each data point of the noise-reduced reference information and the information to be identified by the average of the corresponding information values.

[0123] Secondly, the data undergoes de-binning, which involves extracting the baseline information and the information to be identified into n*m units. For example, in a 3*3 matrix, the value of the middle physical pixel is subtracted from the average value of the surrounding 8 physical pixels. The resulting data is then used to construct a new matrix or vector to obtain the processed baseline information and the processed information to be identified. De-binning removes noise from the data and reduces the amount of data, thus improving the efficiency of subsequent recognition. For example, in a 3*3 matrix, the original 9 data points corresponding to 9 physical pixels can be reduced to 1 data point, reducing the amount of data to be calculated by 9 times.

[0124] It is important to note that the area requiring de-binning during this process can be the entire photosensitive area of ​​the image sensor, a manually defined ROI region, or an area obtained through calibration methods. That is, this region is used in this invention to collect spectral information to determine whether the subject is alive; therefore, only this region needs to be de-binded.

[0125] Then, the processed baseline information and the processed information to be identified are used to determine the liveness of the subject.

[0126] Identification and comparison process:

[0127] All baseline information is processed to obtain corresponding processed baseline information. The correlation between the processed baseline information and the processed information to be identified is then calculated, for example, using the Pearson correlation coefficient. If the correlation R... 2 A correlation of 0.7 or higher than the preset threshold indicates a liveness. In some cases, the correlation can be required to be greater than or equal to 0.4, and in others, it can be greater than or equal to 0.9. In other words, the threshold corresponding to the correlation can be set manually, meaning that manufacturers or users can adjust it according to the recognition system and their needs.

[0128] In one modified embodiment, as the number of identifications increases, the system stores the correlation values ​​of successful liveness identifications over a continuous historical period, calculates the trend of these values, and adjusts the threshold value accordingly. For example, if the correlation value calculated for each identification gradually decreases, the threshold value should be appropriately increased to ensure that non-live individuals cannot bypass the identification system.

[0129] Liveness detection method

[0130] Furthermore, this invention provides a liveness detection method. The spectral information does not necessarily require reconstructing the spectral curve for liveness detection; instead, liveness detection can be performed directly based on the spectral response. Specifically, the spectral response of the incident light modulated by the slit of the OLED screen on the image sensor is obtained, and the reference spectral response of the object under test is obtained. To a certain extent, the spectral response can be understood as the de-binning data mentioned above. The obtained spectral response of the object under test is compared with the pre-stored reference spectral response; and based on the comparison result between the reference spectral response and the spectral response of the object under test, it is determined whether the object under test is a live body. The entire content of Chinese invention patent CN202110275126X is incorporated into this invention based on its structure. Preferably, the spectral information can be denoised by subtracting the reference spectral information before spectral response conversion for liveness detection.

[0131] Figure 12 The illustration shows a flowchart of a first example of a liveness detection method according to the present invention.

[0132] Liveness detection method

[0133] Liveness detection can be performed using neural networks.

[0134] Figure 13 This is a schematic diagram of the neural network model of the present invention. Specifically, Figure 13 The diagram illustrates a multilayer perceptron model. The number of nodes in the input layer is the same as the number of pixels in the target region. During computation, the data from each pixel is used as the data for each node in the input layer. Activation and fully connected operations are then performed between layers. The final output layer has only one node and is connected to the previous layer only by a fully connected layer. The output of this layer can be manipulated using the Logistic function; whether the output of the Logistic function is greater than 0.5 is used to determine if the subject is alive.

[0135] In actual use of the above networks, there are two steps: training and testing.

[0136] In the training step, the input is the image sensor detecting live or non-live objects, and the output is whether it is a live object (e.g., 1.0 for live, 0.0 for non-live). Then, the parameters in the network are trained through backpropagation.

[0137] In the detection step, the input value is the object to be detected, and the output value (e.g., whether it is greater than 0.5) is used to determine whether it is a living object.

[0138] Identification scheme

[0139] Furthermore, a fingerprint recognition method is also provided, comprising: projecting a first detection light onto a subject; receiving the first detection light reflected back from the subject and generating first spectral information and image information of the subject based on the first detection light; projecting a second detection light onto the subject; receiving the second detection light reflected back from the subject and generating second spectral information of the subject based on the second detection light; and performing liveness detection and object recognition based on the first spectral information, the image information, and the second spectral information. The first detection light is a mixed light, for example, at least two different light-emitting units in the R, G, and B light-emitting units of an OLED screen emit light, and the first detection light is preferably white light; the second detection light is preferably monochromatic light, such as green light and blue light. That is, the first detection light and the second detection light are two different types of light. After being reflected by the object under test, the first detection light and the second detection light enter the OLED screen and are modulated by the OLED screen, thereby being received by the image sensor to obtain the corresponding image information, first spectral information, and second spectral information. The first spectral information and the second spectral information are obtained from the spectral pixels of the image sensor.

[0140] Figure 14 A flowchart illustrating a second example of the liveness detection method according to the present invention is shown.

[0141] The liveness detection and object recognition based on the first spectral information, the image information, and the second spectral information includes: processing the first spectral information and the second spectral information to generate a first spectral response result and a second spectral response result; processing the image information to generate an image of the subject; comparing the image of the subject with a pre-stored reference image; and, in response to a successful match between the image of the subject and the reference image, determining whether the subject is a live object based on the first spectral response result and / or the second spectral response result. Figure 15 As shown. Figure 15 The diagram shows... Figure 14 The flowchart shows the liveness detection and object recognition steps in the method shown.

[0142] Live fingerprint recognition process

[0143] The finger to be tested is placed on the testing area of ​​the screen. The screen's light source begins to project light. When the light reaches the finger, part of it is absorbed and part is reflected to form incident light. The incident light enters the slit unit of the screen and is modulated. It is then received by the image sensor to obtain image information and spectral information. Imaging and liveness detection are then performed based on the image information and spectral information. Specifically, the image information is used to reconstruct the fingerprint image, which is then compared with a pre-stored reference fingerprint image. Simultaneously, the spectral information can be converted into a spectral response or spectral curve, which is compared with a pre-stored reference spectral response or spectral curve to determine whether the finger is alive.

[0144] It can also be based on image and spectral information, and use neural networks to identify and compare liveness and fingerprints.

[0145] Spectrometer Examples

[0146] The development of computational spectroscopy has made the miniaturization of spectrometers possible. Currently, computational spectroscopy requires specific filter structures paired with corresponding algorithms to achieve spectral reconstruction. Essentially, this can be understood as follows: after the image sensor measures the spectral response, it is input into the data processing unit for reconstruction calculations. This process is described in detail below:

[0147] The intensity signals of incident light at different wavelengths λ are denoted as x(λ), and the transmission spectrum curve of the filter structure is denoted as T(λ). The filter (filter structure) has m sets of structural units, and the transmission spectrum of each set of structural units is different. Overall, the filter structure can be denoted as Ti(λ) (i=1,2,3,…,m). Each set of structural units has a corresponding physical pixel below it, which detects the light intensity bi modulated by the filtered light structure. In a specific embodiment of this application, one physical pixel corresponds to one set of structural units, but it is not limited to this. In other embodiments, multiple physical pixels may be grouped together to correspond to one set of structural units. Therefore, in the computational spectral device according to the embodiments of this application, at least two sets of structural units constitute a "spectral pixel" (which can be understood as multiple sets of structural units and corresponding image sensors constituting a spectral pixel). It should be noted that the number of effective transmission spectra (the transmission spectrum used for spectral reconstruction is called the effective transmission spectrum) Ti(λ) of the filter structure may not be the same as the number of structural units. The transmission spectrum of the filter structure is manually set, tested, or calculated according to certain rules based on the needs of identification or reconstruction (for example, the transmission spectrum obtained by testing each structural unit is the effective transmission spectrum). Therefore, the number of effective transmission spectra of the filter structure may be less than or even more than the number of structural units. In this modified embodiment, a certain transmission spectrum curve is not necessarily determined by a group of structural units. Furthermore, the present invention can use at least one spectral pixel to reconstruct an image. That is, the spectral device in this application can reconstruct spectral curves or perform spectral imaging based on the spectral response.

[0148] The relationship between the spectral distribution of incident light and the measurements from the image sensor can be expressed by the following formula:

[0149] bi=∫x(λ)*Ti(λ)*R(λ)dλ

[0150] After discretization, we get

[0151] bi=Σ(x(λ)*Ti(λ)*R(λ))

[0152] Where R(λ) is the response of the image sensor, denoted as:

[0153] Ai(λ)=Ti(λ)*R(λ),

[0154] The above equation can then be extended into matrix form:

[0155]

[0156] Where bi (i = 1, 2, 3, ..., m) is the response of the image sensor after the light to be measured passes through the filter structure, corresponding to the light intensity measurement values ​​of the image sensor corresponding to each of the m structural units. When one physical pixel corresponds to one structural unit, it can be understood as the light intensity measurement values ​​corresponding to m 'physical pixels', which is a vector of length m. A is the system's response to light of different wavelengths, determined by the transmittance of the filter structure and the quantum efficiency of the image sensor. A is a matrix, where each row vector corresponds to a set of structural units' responses to incident light of different wavelengths. Here, the incident light is sampled discretely and uniformly, with a total of n sampling points. The number of columns in A is the same as the number of sampling points of the incident light. Here, x(λ) is the light intensity of the incident light at different wavelengths λ, which is the incident light spectrum to be measured.

[0157] This invention employs a substrate with periodic slit units, aperture units, or pillar units as a filter structure. The substrate is positioned along the optical path of an image sensor. Taking a slit unit as an example, the slit unit consists of at least one slit. Each slit unit has a corresponding transmission spectrum matrix T, which can modulate the incident light, thereby allowing the image sensor to receive and obtain a light intensity measurement value.

[0158] Taking the substrate as an OLED screen as an example, a spectrometer can be formed by placing a photosensitive module containing an image sensor below the OLED screen. Incident light passes through the slit unit of the OLED screen, is modulated by the slit unit, and is then received by the image sensor.

[0159] The spectrometer in this embodiment is based on an OLED screen that can achieve diffraction and / or interference. Preferably, the OLED screen described in this invention produces interference effects as much as possible. Therefore, the design of the screen or slits needs to be considered. It should be understood that the OLED screen will have R, G, B light-emitting units distributed according to the required pattern. For example, as shown in the figure, conventionally, R, G, G, B light-emitting units are arranged in an array as a group. The slits are formed between the light-emitting units. Multiple slits are formed between a group of RGGB light-emitting units. Multiple slits are defined as a slit unit. The slit units need to be arranged with a fixed period, that is, the distance (period) between adjacent slit units is equal (it is only necessary to ensure that the screen area involved in modulating the incident light has this characteristic). This can ensure that the interference effect is as obvious as possible, so that the image information received by the image sensor contains spectral characteristics. It should be understood that this invention does not limit the screen to be arranged in an RGGB array. The screen can be arranged in other ways. In this case, the slit units can also be adjusted accordingly. In this case, at least one slit contained in the smallest repeating light-emitting unit of the screen can be defined as a slit unit. It is important to understand that different slit units do not need to be completely identical, but their differences should not be too large to avoid interference.

[0160] To better understand, let's further explain the slit unit. An OLED screen has a pixel layer (light-emitting layer) and a circuit layer (TFT structure layer), which prevent incident light from passing through. Slits and / or holes exist between the pixels (light-emitting units) and between the TFT structures, allowing incident light to pass through. These light-transmitting slits and holes are periodic within a certain range; for example, they could be periodic across the entire screen, or within the test area corresponding to the photosensitive module, or other areas. At least one slit and / or hole constitutes a slit unit, and any slit unit can define a vector a and a vector b with its two adjacent slit units. That is, vectors a and b can be found, along with a region whose area is equal to the dot product of a and b (the area of ​​the parallelogram formed by vectors a and b). Within the periodic region, after translating an integer number of vectors a and b along the corresponding vector directions, the slits and / or holes will coincide. The periodic region has at least 25 slit units. Generally, the angle between vectors a and b is 90 degrees. Figures A and B show two different OLED screen slits and / or pinholes. The bright areas represent the OLED slits and / or pinholes, while the outlined rectangular areas represent slit units. Of course, the slit unit areas will differ for different screens, but are limited to rectangular areas. It should be noted that the slits between slit units within this invention may be different from each other, but the shape, structure, and size of the slit units are basically consistent. However, due to certain errors during manufacturing, there may be some differences between slit units, which can also be understood as being consistent with the concept of this invention and covered by this invention.

[0161] It is important to note that, in order to ensure that the image sensor obtains as much spectral information as possible, the transmission spectrum curves Ti(λ) (i = 1, 2, 3, ..., m) should ideally satisfy the condition that at least two transmission spectrum curves have a correlation of less than or equal to 0.4. This correlation can be defined using the Pearson correlation coefficient. Furthermore, regarding the definition of the transmission spectrum curve in this invention, it should be understood that the existence of the transmission spectrum curve is mainly due to the presence of slits in the OLED screen. Incident light passing through the slits is modulated, and the transmission spectrum curve can be considered to determine the modulation effect on the incident light. Therefore, multiple slits forming a slit unit will each have a corresponding transmission spectrum curve. However, in this invention, the transmission spectrum curve is preferably not determined by a single slit unit; it may be influenced by surrounding slit units. That is, the transmission spectrum curve in this invention is preferably determined by at least two slit units. Furthermore, the number of transmission spectrum curves is equal to the number of effective light intensities bi. Effective light intensities bi refer to the light intensity information used for spectral recovery or spectral response judgment, and their number n is equal to the number of transmission spectrum curves. In general applications, the incident light is sampled discretely and uniformly, with a total of n sampling points. For example, in the 200-400nm band, the spectral resolution is 1nm, so the number of sampling points is 201. At this time, the transmission spectrum matrix formed by the transmission spectrum curves is an n*m matrix.

[0162] Furthermore, the OLED screen includes a glass cover and a light-emitting unit located at the lower end of the glass cover. During the recognition process, the incident light to be measured needs to enter the glass cover. The incident light is modulated by the slit unit of the OLED screen and then received by the image sensor to obtain the spectral information modulated by spatial dispersion.

[0163] The photosensitive module may further include an optical component that adjusts the modulated light.

[0164] It is important to note that the transmission spectrum curve corresponding to the slit unit of the OLED screen is relatively sensitive to the angle of incident light; that is, changes in the angle of incident light will cause changes in the transmission spectrum curve. Therefore, when using the spectrometer, it is necessary to determine the angle of the incident light and select the corresponding transmission spectrum matrix for spectral reconstruction.

[0165] The spectrometer further includes a memory and a processing unit, which are communicatively connected to the image sensor or can be integrated into the image sensor. The transmission spectrum matrix can be digitized and stored in the memory, or the transmission spectrum matrix can be converted according to the requirements of the recovery algorithm before being digitized and stored in the memory.

[0166] Spectrometer Examples Requiring a Light Source

[0167] For example, when using the spectrometer for object recognition, the spectrometer may optionally also include a light source, preferably the light-emitting unit of the OLED screen. The light-emitting unit of the OLED screen emits light to the object under test. The object under test partially absorbs and partially reflects the light from the light source. The reflected light enters the slit unit of the OLED screen, is modulated, and then received by the image sensor to obtain a light intensity measurement value. The spectral information (spectral curve) is then calculated to determine the object. It should be noted that the object recognition system also includes a placement area for the object under test. The distance between the placement area and the OLED screen is preferably less than or equal to 6 cm, and more preferably less than or equal to 3 cm; this allows the incident angle of the incident light to better meet the modulation requirements, resulting in a better modulation effect.

[0168] Furthermore, the spectrometer can also be used to measure jaundice, color temperature, etc. It can recover the spectral curve based on the incident light, and then identify jaundice or measure color temperature based on the spectral curve.

[0169] It should be noted that the structure and principle of the spectrometer described in this invention are highly consistent with the above-mentioned live fingerprint recognition system embodiment. This embodiment focuses on referencing the content of the above embodiment and explains some differences and details.

[0170] Alternative embodiments

[0171] As mentioned above, this invention utilizes the interference and diffraction principles of slits or pinholes to modulate incident light. While ensuring the interference and diffraction effects are as pronounced as possible, the image information received by the image sensor can contain spectral characteristics. These spectral characteristics can then be used for applications such as spectral reconstruction, material identification, and authenticity verification. Therefore, in an alternative embodiment of this invention, the spectral device is no longer based on a screen. Figure 20 A schematic diagram of a spectroscopic device according to an alternative embodiment of this application is illustrated. Figure 20As shown, the spectral device in this embodiment includes a modulation cover plate and an image sensor, that is, the substrate is implemented as a modulation cover plate. The modulation cover plate is located above the image sensor, and the modulation cover plate has slit units. Specifically, the incident light is modulated through the slit units, i.e., interference and diffraction effects are generated. The modulated incident light is received by the image sensor to obtain spectral information. The modulation cover plate can be made of a transparent material, such as transparent plastic or transparent glass. Preferably, since glass has relatively high transmittance, a glass cover plate can be selected. Then, an opaque material is applied to the surface of the modulation cover plate, and the slit units of the present invention are formed where the opaque material is not applied. The opaque material can be formed on the modulation cover plate by processes such as vapor deposition and bonding. That is, the modulation cover plate includes a glass cover plate made of transparent material and an opaque material covering the glass cover plate, and the slit units are formed where the opaque material is not applied to the modulation cover plate. Figure 21 As shown, the slit unit includes at least one slit and / or a small hole, that is, the slit unit is composed of at least one slit or at least one small hole, thereby having interference and diffraction effects. Figure 21 The diagram shows... Figure 20 A schematic diagram illustrating an example of the structure of the modulation cover plate of the spectral device shown.

[0172] Preferably, in this invention, at least one slit and / or aperture unit constitutes a slit with a certain periodicity. Specifically, any slit unit can define a vector a and a vector b with its two adjacent slit units, i.e., vector a, vector b, and a region with an area equal to the dot product of a and b (the area of ​​the parallelogram formed by vectors a and b) can be found. Within the periodic region, the slit and / or aperture will coincide after translating an integer number of vectors a and b along the corresponding vector directions. The periodic region has at least 25 periodic units. Generally, the angle between vectors a and b is 90 degrees. It should be noted that the slits between slit units in this invention may be different from each other, but the shape, structure, and size of the slit units are basically consistent. That is, a slit unit can have slits or apertures with different structures, sizes, or shapes. However, due to certain errors in processing, there may be certain differences between slit units, which can also be understood as being consistent with the concept of this invention and covered by this invention.

[0173] Taking at least one slit constituting the slit unit as an example, the modulation cover plate processing technology of the present invention can be implemented by applying photoresist to a transparent cover plate, curing it, developing it, etching it, applying an opaque material to the etched area to form a light-shielding area, and then removing the remaining photoresist to form a slit. Relatively speaking, the slit structure and dimensional accuracy of the slit unit corresponding to this embodiment are higher than those corresponding to OLED screens, and it is also easier to process and obtain.

[0174] In another variation example, such as Figure 22 As shown, the modulation cover plate can be made of an opaque material, and the slit unit of the present invention includes at least one small hole, through which interference and diffraction effects are achieved. For example, the modulation cover plate can be implemented as a random mask, and a high-precision modulation cover plate can be formed using mature technology. Figure 22 The diagram shows... Figure 20 A schematic diagram of another example of the structure of the modulation cover plate of the spectral device shown.

[0175] Furthermore, such as Figure 23 As shown, the spectroscopic device in this embodiment further includes an optical component located on the photosensitive path of the image sensor. Preferably, the optical component is located between the modulation cover and the image sensor. The optical component can be a lens, a filter, or a combination thereof, primarily used to adjust the modulated light. Further, the spectroscopic device may also include a support assembly for fixing the optical component and the modulation cover. Further, the spectroscopic device includes a circuit board, to which the image sensor is electrically connected. The support assembly is preferably fixed to the circuit board. Figure 23 The diagram shows... Figure 20 The diagram shows a schematic representation of the configuration of the spectral device, including the optical components.

[0176] In this embodiment, the OLED screen is replaced with a specific modulation cover plate to achieve the acquisition of spectral information. Its working principle and application scenarios are similar to those of the OLED screen. Furthermore, in this embodiment, a light source can be set separately, and the function of the OLED screen can be achieved through the combination of the light source and the modulation cover plate.

[0177] Preferably, the spectroscopic device further includes a collimation system for collimating the incident light. The collimation system can be implemented as at least one lens or a microlens array. The collimation system is located at the upper end of the modulation cover plate, meaning that the incident light passes through the collimation system before entering the modulation cover plate for modulation.

[0178] In some embodiments, the opaque material includes conductive materials such as metals. A capacitor can be formed by parallel arrangement of metal materials. Note that two parallel metal materials cannot conduct electricity. A slit unit can be formed with the assistance of a non-conductive, opaque material. That is, in this embodiment, the opaque material is divided into conductive and non-conductive materials. A capacitor is formed by parallel arrangement of conductive materials, and a corresponding slit is formed with the assistance of non-conductive materials, thus constituting a slit unit. Connecting the conductive material to the circuit board allows the slits formed to function as capacitors. In other words, the circuit board is suitable for conducting electricity through the capacitor structure. Record the reference capacitance value under normal conditions. During use, if dust or dirt enters the slit, it will cause a change in the capacitance value. A threshold can be set. When the difference between the capacitance value and the reference capacitance value exceeds the threshold, the user is reminded to clean the surface of the modulation cover.

[0179] Alternative embodiments

[0180] Existing consumer electronics and wearable devices typically include at least one camera module for taking pictures. Specifically, the consumer electronic device comprises a main body and a camera module, with the camera module mounted on the main body. Furthermore, the consumer electronic device includes a protective cover plate, which is disposed on the main body, forming an enclosed space with the main body. The camera module is located within this enclosed space, thereby preventing dust from adhering to the lens surface of the camera module and affecting image formation.

[0181] This embodiment uses a mobile phone as an example. Figure 24 This is a schematic diagram of the back of an existing mobile phone, which typically has at least one rear camera module; Figure 25 This is a schematic diagram of the back of a mobile phone with a protective cover according to this embodiment. The consumer electronic device includes a device body, an image sensor, and a protective cover. The image sensor and the protective cover are disposed on the device body, wherein the protective cover is located on the light-sensing path of the image sensor. Further, the protective cover has a light-transmitting area and a light-blocking area, wherein the light-transmitting area is composed of multiple slits. That is, in this embodiment, by forming light-transmitting and light-blocking areas on the protective cover, the protective cover achieves the modulation cover function of the previous embodiment. In other words, the modulation cover is the protective cover of the electronic device, and the protective cover has light-transmitting and light-blocking areas, with the light-transmitting area forming the slit unit.

[0182] Preferably, an opaque material can be applied to the surface of the protective cover, thereby creating a non-transparent area in the area with the opaque material and a light-transmitting slit in the area without the opaque material. At least one slit constitutes a slit unit for modulating the incident light (achieving broadband modulation using interference and diffraction effects). Preferably, the opaque material is located on the inner surface of the protective cover, thereby preventing dust and particles from falling between the slits and affecting the modulation effect. It should be noted that changes in the slit units of the present invention will affect the corresponding modulation effect to some extent, making it impossible for the built-in recovery and recognition algorithms to accurately achieve spectral recovery or substance identification. Therefore, the opaque material being located on the inner surface of the protective cover ensures that the slits are not affected by the environment.

[0183] The consumer electronic device may further include optical components, a circuit board, and a bracket. These components, along with the image sensor, form a camera module, which is fixed within a sealed space formed by the device body and the protective cover. The optical components may be a lens and / or a filter. Furthermore, it may include a focusing mechanism, such as a voice coil motor or SMA, to drive the lens movement for focusing.

[0184] This embodiment focuses on implementing the protective cover of consumer electronics as a light filter structure with modulation effect (or equivalent to the OLED screen or modulation cover in the previous embodiment).

[0185] Alternative embodiments

[0186] Unlike the embodiments described above, the photosensitive module includes a filter structure and an image sensor. The filter structure is located on the photosensitive path of the image sensor and is a broadband filter structure in the frequency or wavelength domain. The pass spectra of different wavelengths of the filter structures are not entirely the same. The filter structure can be a metasurface, photonic crystal, nanopillar, multilayer film, dye, quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties. For example, in the embodiments of this application, the filter structure can be the light modulation layer in Chinese patent CN201921223201.2. Further, the spectral device includes an optical system located on the photosensitive path of the image sensor. Light is adjusted by the optical system and then modulated by the filter structure before being received by the image sensor to obtain the spectral response. The optical system may be a lens assembly, a light homogenizing assembly, or other optical systems. The image sensor can be a CMOS image sensor (CIS), a CCD, an array photodetector, or the like. In addition, the spectral device also includes a data processing unit, which may be a processing unit such as an MCU, CPU, GPU, FPGA, NPU, or ASIC, and can export the data generated by the image sensor to an external source for processing.

[0187] However, it's important to note that some of the projected light A from the light-emitting unit directly enters the slit and reaches the image sensor; some projected light B reaches the glass cover and is reflected directly into the slit before being received by the image sensor; some projected light C reaches the object under test (finger) and is reflected into the slit before being received by the image sensor; and some projected light D is absorbed by the object under test (finger). Liveness detection is based on the fact that fingers, due to the presence of capillaries and sweat glands, absorb different wavelengths of light differently, differing from the absorption of projected light by conventional silicone or fake fingers. This difference can be used to determine liveness. Therefore, the truly useful projected light should be projected light C and projected light D, while projected light A, projected light B, and ambient light are, to some extent, stray light. Therefore, in a dark room environment, the light-emitting unit can project the same light. In this case, the incident light received by the image sensor can be essentially considered as projected light A and projected light B passing through the slit and being received by the image sensor. The spectral information collected under this condition is recorded as the reference spectral information. Subtracting this reference spectral information from the spectral information obtained in subsequent tests on the object under test removes stray light from projected light A and projected light B. This results in higher accuracy in spectral curve recovery.

[0188] Spectral curve recovery method

[0189] This invention further provides a spectral restoration method based on a neural network, comprising: acquiring sampled spectral data to be processed; and inputting the sampled spectral data to be processed into a neural network with predetermined parameters to output a spectral restoration result. Specifically, the entire contents of Chinese invention CN2021104180126 are incorporated herein by reference.

[0190] The neural network is trained using training spectral data. The training process of the neural network includes: acquiring a pair of training spectral data, wherein the pair of training spectral data includes pre-sampling spectral data and post-sampling spectral data, wherein the pre-sampling spectral data is based on the superposition of at least one Gaussian distribution and / or at least one Lorentz distribution of the spectral curve; and training the neural network for spectral recovery using the pre-sampling spectral data of the training spectral data pair as input data and the post-sampling spectral data of the training spectral data pair as labels until the parameters of the neural network converge.

[0191] Furthermore, acquiring the training spectral data pair includes: generating the pre-sampling spectral data with a first preset length based on the superposition of at least one Gaussian distribution and / or at least one Lorentz distribution; adding first noise spectral data to the pre-sampling spectral data to obtain noisy pre-sampling spectral data; and sampling the noisy pre-sampling spectral data to obtain the post-sampling spectral data with a second preset length.

[0192] Furthermore, another high-resolution spectral recovery method is provided, including:

[0193] Step 1: Obtain the dictionary of the transmission spectrum of the spectral chip after discrete cosine transform, the discrete cosine transform dictionary, and the measurement value vector of the image sensor of the spectral chip;

[0194] Step 2: The first layer of modeling based on Bayesian hierarchical modeling is to model the sparse vector corresponding to the spectral vector as a vector of normal multiplicative distribution to obtain the vector of the first normal distribution variable and the vector of the second normal distribution variable. The dot product of the vector of the first normal distribution variable and the vector of the second normal distribution variable is calculated to obtain the vector of the normal multiplicative distribution, and the dot product of the first covariance matrix of the vector of the first normal distribution variable and the second covariance matrix of the vector of the second normal distribution variable is calculated to obtain the covariance matrix of the vector of the normal multiplicative distribution.

[0195] Step 3: The second layer of modeling based on Bayesian hierarchical modeling is to model the reciprocal of the product of the variances at each position in the first covariance matrix of the vector of the first normal distribution variable and the second covariance matrix of the vector of the second normal distribution variable as a gamma distribution that follows the first hyperparameter and the second hyperparameter.

[0196] Step 4: Based on the Bayesian method, calculate the estimated vector of the first posterior probability density of the vector of the first normally distributed variable and the estimated vector of the second posterior probability density of the vector of the second normally distributed variable.

[0197] Step 5: Calculate the vector of the normal multiplicative distribution based on the dot product of the estimated vector of the first posterior probability density and the estimated vector of the second posterior probability density;

[0198] Step 6: Update the first expectation matrix and the second expectation matrix corresponding to the first covariance matrix and the second covariance matrix based on the first covariance matrix, the second covariance matrix, the estimated vector of the first posterior probability density, the estimated vector of the second posterior probability density, the first hyperparameter and the second hyperparameter;

[0199] Step 7: Repeat steps 4 to 6 until the iteration condition is met;

[0200] Step 8: Calculate the covariance matrix of the vector of the normal multiplication distribution based on the first expectation matrix and the second expectation matrix; and

[0201] Step 9: Obtain the spectral vector based on the vector of the normal multiplication distribution, its covariance matrix, and the discrete cosine transform dictionary.

[0202] For ease of understanding, the entire contents of Chinese invention CN2021109755685 are incorporated into this invention.

[0203] A further spectral recovery method is provided, comprising:

[0204] Step 1: Obtain the transmission spectrum matrix of the spectral chip and the measurement value vector of the image sensor of the spectral chip;

[0205] Step 2: Based on the improved regularized description model, an augmented matrix is ​​constructed from the transmission spectrum matrix. The augmented matrix includes a first sub-matrix in the upper left, a second sub-matrix in the upper right, a third sub-matrix in the lower left, and a fourth sub-matrix in the lower right.

[0206] Step 3: Set the first spectral vector;

[0207] Step 4: Determine the row with the maximum residual based on the transmission spectrum matrix, the measured value vector, and the first spectral vector;

[0208] Step 5: Determine the first iteration vector and the first spectral residual vector based on the first spectral vector;

[0209] Step 6: Update the first iteration vector based on the rows corresponding to the row with the largest residual in the first and second submatrices of the augmented matrix;

[0210] Step 7: Determine the rows to be iterated in the third and fourth submatrices of the augmented matrix;

[0211] Step 8: Update the first spectral vector and the first spectral residual vector based on the row to be iterated and the updated first iteration vector;

[0212] Step 9: Repeat steps 6 to 8 until the calculations are complete for all rows of the third and fourth submatrices of the augmented matrix; and

[0213] Step 10: Repeat steps 4 to 9 until the first spectral residual vector satisfies the predetermined condition.

[0214] For ease of understanding, all the contents of Chinese invention CN 2021108481584 are incorporated into this invention.

[0215] Furthermore, a high-resolution spectral recovery method is provided, comprising:

[0216] Step 1: Obtain the transmission spectrum matrix of the spectral chip and the measurement value vector of the image sensor of the spectral chip;

[0217] Step 2: Set a predetermined selection probability for each row of the transmission spectrum matrix. The predetermined selection probability is the quotient of the square of the L2 norm of a certain row of the transmission spectrum matrix and the square of the Frobenius norm of the transmission spectrum matrix.

[0218] Step 3: Select a predetermined row of the transmission spectrum matrix based on the predetermined selection probability;

[0219] Step 4: Based on the inner product of the spectral vector before iteration and the predetermined row, the measured value vector and the value at the corresponding position of the predetermined row, the L2 norm of the predetermined row, and the predetermined row, an update vector is obtained;

[0220] Step 5: Subtract the updated vector from the spectral vector before iteration to obtain the spectral vector after iteration; and,

[0221] Step 6: Repeat steps 3 to 5 until the iterated spectral vector satisfies the termination condition, which is based on the iterated spectral vector and its L2 norm, the transmission spectrum matrix and its Frobenius norm, and the measured value vector.

[0222] The spectrometer includes an OLED screen, an image sensor, a memory, and a processing unit. The memory and processing unit can optionally be integrated into the image sensor, or they can be communicatively connected. The memory samples, quantizes, and stores the transmission spectrum matrix of the slit cells of the OLED screen in digital format. In some embodiments, the transmission spectrum matrix can be calculated before storage. The processing unit is configured to recover the spectral curve based on the spectral response data generated on the image sensor according to the transmission spectrum matrix and the incident light. Preferably, the transmission spectrum matrix can be stored in the memory.

[0223] Furthermore, a method for providing spectral resolution is provided, comprising: (a) obtaining spectral response data by receiving incident light modulated by slit cells of an OLED screen by a receiving image sensor; (b) digitizing the transmission spectrum matrix; and (c) improving the spectral resolution using at least one of the following operations: least squares estimation process, matrix inversion, equalization, or pseudo-inverse matrix manipulation; wherein the slit cells of the OLED screen are implemented as broadband filters; wherein the spectral responses of different slit cells are independent of different peaks and troughs, distributed across the entire target spectral range, and partially overlapping with multiple slit cells of the OLED screen. The digitization includes steps involving sampling and quantization.

[0224] A spectral reconstruction method is provided. Based on the transmission spectrum matrix of an OLED screen and spectral response data acquired by an image sensor, a regularization parameter is selected. Preferably, this regularization parameter can be selected using parameter estimation methods such as generalized maximum likelihood estimation, leave-one-out cross-validation, and generalized moment estimation. Preferably, the dimensionality of the OLED screen's transmission spectrum matrix can be reduced to decrease computational complexity. Based on the selected regularization parameter, a non-negative least squares solution is performed using a processor. Solution methods include, but are not limited to, preprocessing conjugate gradient method, trust region reflection method, and bounded variable least squares method, to complete the spectral reconstruction.

[0225] Spectral Imaging Examples

[0226] It should be noted that the principle of spectral imaging is to denote the intensity signals of incident light at different wavelengths λ as f(λ), and the transmission spectrum curve of the filter structure as T(λ). The filter has m sets of filter structures, each with a different transmission spectrum, also called a "structural unit," which can be denoted as Ti(λ) (i = 1, 2, 3, ..., m). Each set of filter structures has a corresponding physical pixel below it, which detects the light intensity Ii modulated by the filter structure. In a specific embodiment of this application, one physical pixel corresponding to one set of structural units is used as an example for illustration, but it is not limited to this. In other embodiments, multiple physical pixels can also form a set corresponding to a set of structural units.

[0227] The relationship between the spectral distribution of the incident light and the measurements of the photodetector array can be expressed by the following formula:

[0228] Ii=Σ(f(λ)·Ti(λ)·R(λ))

[0229] Where R(λ) is the detector response, denoted as:

[0230] Si(λ)=Ti(λ)·R(λ)

[0231] The above equation can then be extended into matrix form:

[0232]

[0233] Where Ii (i = 1, 2, 3, ..., m) is the response of the photodetector after the light to be measured passes through the broadband filter unit, corresponding to the light intensity measurements of m photodetector units, also known as m "physical pixels," which is a vector of length m. S is the system's response to light of different wavelengths, determined by the transmittance of the filter structure and the quantum efficiency of the photodetector response. S is a matrix, where each row vector corresponds to the response of a broadband filter unit to incident light of different wavelengths. Here, the incident light is sampled discretely and uniformly, with a total of n sampling points. The number of columns in S is the same as the number of sampling points of the incident light. Here, f(λ) is the light intensity of the incident light at different wavelengths λ, which is the incident light spectrum to be measured.

[0234] In practical applications, the system's response parameter S is known. By using the light intensity reading I of the detector, the spectrum f of the input light can be obtained by using an algorithm. The process can adopt different data processing methods depending on the specific situation, including but not limited to: least squares, pseudo-inverse, equalization, least 2 norm, artificial neural network, etc.

[0235] The above example, using one physical pixel corresponding to a set of structural units, illustrates how to recover spectral information, also known as a "spectral pixel," using m sets of physical pixels (i.e., pixels on an image sensor) and their corresponding m sets of structural units (identical structures on the modulation layer are defined as structural units). It is worth noting that in this embodiment, multiple physical pixels can also correspond to a set of structural units. Further, a set of structural units and at least one corresponding physical pixel constitute a unit pixel; in principle, at least one unit pixel constitutes a spectral pixel.

[0236] Based on the above implementation method, arraying the spectral pixels can realize a snapshot-type spectral imaging device.

[0237] For example, such as Figure 16 As shown, an image sensor with 1896*1200 pixels is used. Figure 16 (A portion of the image sensor area is shown.) Simultaneously, m=4 is selected, meaning a 4x4 pixel unit is chosen to form a spectral pixel. This results in 474x300 independent spectral pixels, each of which can have its spectral result calculated individually using the method described above. By combining this image sensor with lens groups and other components, snapshot-style spectral imaging of the object under test can be performed, enabling the acquisition of spectral information for every point on the object in a single exposure. Figure 16 The illustration shows a schematic diagram of a first example of a spectral pixel array of an image sensor according to the present invention.

[0238] Based on this, the selection method of spectral pixels can be rearranged according to actual needs, without making any adjustments to the image sensor, to improve spatial resolution. For example... Figure 17 As shown, you can select a close arrangement of solid and dashed boxes to increase the spatial resolution in the example above from 474*300 to nearly 1896*1200. Figure 17 A schematic diagram of a second example of a spectral pixel array of an image sensor according to the present invention is illustrated.

[0239] Furthermore, for the same image sensor, the spatial resolution and spectral resolution can be rearranged as needed. For example, in the above example, when the spectral resolution requirement is high, 8*8 unit pixels can be used to form a spectral pixel; when the spatial resolution requirement is high, 3*3 physical pixels can be used to form a spectral pixel.

[0240] The spectral imaging system is structurally identical to the spectrometer system, but their recovery algorithms differ. Specifically, the spectral imaging algorithm is provided based on the structure of the spectrometer embodiment.

[0241] A spectral recovery method is provided, comprising:

[0242] The process involves: acquiring the light energy response signal matrix and standard spectrum output by the photosensitive chip of the spectral imaging device; determining a basic element recovery function and its response signal vector based on the light energy response signal matrix, wherein the basic element recovery function uses predetermined pixel values ​​of the photosensitive chip and nearby pixel values ​​to recover the spectral image value of its corresponding predetermined channel; acquiring a recovery tensor, the product of the recovery tensor and the response signal vector being equal to the output of the basic element recovery function based on the response signal vector; and obtaining the recovered spectral image based on the product of the recovery tensor and the response signal vector.

[0243] The light energy response signal matrix is ​​represented as a matrix B comprising two dimensions: image width w and image height h. The standard spectrum has a dimension of 1 and is set to minimize the distance between the product of the true value tensor of the spectral image received by the spectral imaging device and the standard spectrum and the tensor of the spectral image to be recovered.

[0244] Furthermore, the standard spectrum is denoted as s, and the channel standard spectrum corresponding to the k-th channel of the standard spectrum is denoted as s. k , so that:

[0245] x k →O(i,j)s k

[0246] Among them, s k O(i,j) is the spectral image value of the k-th channel of a certain spectral pixel, and O(i,j) is the tensor of the true value of the spectral curve of a certain spectral pixel, where → indicates that the Euclidean distance between tensors is minimized.

[0247] The process of obtaining the transmission spectrum matrix of the spectral chip and the measurement vector of the image sensor of the spectral chip includes: obtaining the initial transmission spectrum matrix A of the spectral chip and the initial measurement vector b of the image sensor of the spectral chip; and obtaining the matrix A' and measurement vector b' of the overdetermined system from the initial transmission spectrum matrix A and the initial measurement vector b' by extracting coefficients from the spectral vector based on a regularized description model, wherein the regularized description model is:

[0248]

[0249] Where λ>0 are the regularization coefficients, D is a tridiagonal Toeplitz matrix, and ||·|| denotes the L2 norm. The matrix A' and the measurement vector b' of the overdetermined system are respectively:

[0250]

[0251]

[0252] Furthermore, the matrix A' and the measurement vector b' of the overdetermined system are respectively used as the transmission spectrum matrix and the measurement vector of the spectral chip.

[0253] For ease of understanding, the entire contents of Chinese invention CN2021111546565 are included here.

[0254] Furthermore, a spectral image reconstruction method is provided, comprising: acquiring transmission spectrum data and output signal data of a spectral imaging chip; acquiring local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on pixels used for reconstructing the spectral image; inputting the local output signal data into an attention model to obtain attention local data; and inputting the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels used for reconstructing the spectral image.

[0255] The process of obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on the pixels used to reconstruct the spectral image includes: obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data with a side length of a predetermined number of pixels in the region near the pixel used to reconstruct the spectral image.

[0256] Furthermore, inputting the local output signal data into the attention model to obtain attention local data includes: dividing the local output signal data into multiple predetermined regions, each predetermined region including output signal data corresponding to multiple pixels of the spectral imaging chip; and performing matrix multiplication for each predetermined region to obtain the attention local data.

[0257] For ease of understanding, the entire contents of Chinese invention CN2021111516729 are included here.

[0258] The basic principles of this application have been described with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0259] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0260] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0261] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0262] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A spectrometer, characterized in that, include: A substrate having a plurality of slit units arranged periodically for modulating incident light, wherein the slit units have corresponding transmission spectrum curves; The substrate includes a screen or a modulation cover plate; A photosensitive module, located at the lower end of the substrate, includes: an image sensor for receiving modulated incident light to obtain spectral information of the incident light, wherein the substrate is disposed on the optical path of the image sensor; A certain distance is provided between the substrate and the image sensor; The spectrometer further includes a light source, which is a light-emitting unit located at the lower end of the substrate; During identification, the object to be identified is placed on the substrate, and the light-emitting unit projects a light onto the object. After being reflected by the object, the incident light is generated. The incident light is modulated by the multiple slit units of the substrate and then received by the image sensor to obtain spatially dispersed image information, thereby obtaining the spectral information of the incident light.

2. The spectrometer according to claim 1, wherein, Each slit unit includes at least one slit and / or aperture.

3. The spectrometer according to claim 1, wherein, The screen includes a glass cover and a light-emitting unit located below the glass cover.

4. The spectrometer according to claim 1, wherein, The photosensitive module further includes an optical component, which includes an aperture and at least one lens, and the optical component is located on the photosensitive path of the image sensor.

5. The spectrometer according to claim 1, wherein, The modulation cover plate includes a glass cover plate made of transparent material and an opaque material covering the glass cover plate, wherein the slit unit is formed in the part of the modulation cover plate not covered by the opaque material.

6. The spectrometer according to claim 5, wherein, The opaque material includes parallel conductive materials that are opaque, and the parallel conductive materials form a capacitor structure.

7. The spectrometer according to claim 6, wherein, The opaque material includes opaque and non-conductive materials.

8. The spectrometer according to claim 7, wherein, The spectrometer further includes a circuit board electrically connected to the image sensor, the circuit board being adapted to conduct electricity to the capacitor structure.

9. The spectrometer according to claim 1, wherein, The modulation cover plate is a mask.

10. The spectrometer according to claim 1, wherein, The modulation cover is a protective cover for electronic devices. The protective cover has a light-transmitting area and a non-light-transmitting area, and the light-transmitting area forms the slit unit.

11. The spectrometer according to claim 1, wherein, The photosensitive module includes a filter structure and an image sensor, with the filter structure located on the photosensitive path of the image sensor.

12. The spectrometer according to claim 1, wherein, The spectrometer further includes a filter located on the light-sensitive path of the image sensor.

13. The spectrometer according to claim 1, wherein, Each of the slit cells and its two adjacent slit cells define two vectors and a region with an area equal to the dot product of the two vectors. After the pattern of this region is translated by an integer number of vector displacements along the vector directions corresponding to the two vectors within the periodic region, the slit of this region coincides with the slit of the translated region. The periodic region is a region formed by multiple slit cells arranged periodically.

Images