Multispectral module
By adding filters to the filter units of the multi-spectral module and optimizing the transmittance curve, the problem of low color coordinate recovery accuracy is solved, and higher color coordinate and spectral recovery accuracy is achieved.
Patent Information
- Application Number
- CN202411001248.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-25
AI Technical Summary
There are errors in the color coordinate recovery process of existing multi-spectral modules, which affect the accuracy, mainly due to silicon-based substrate performance, filter unit material and lens transmittance.
Add filters to the filter units of the multi-spectral module to optimize the transmittance curve, and adjust the cutoff wavelength and cutoff range through the local searchable gradient optimization algorithm to improve the color coordinate recovery accuracy.
It significantly improves the chromatic coordinate recovery accuracy and spectral recovery accuracy, reduces errors, and improves the accuracy of multi-spectral modules in applications.
Smart Images

Figure CN118571895B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of computational spectral imaging, and more specifically, to a multispectral module. Background Art
[0002] Color coordinates are one of the important contents of colorimetry and have a wide range of uses. Color coordinates x, y are used to describe the position of color in the color space. x, y can be obtained by dividing the corresponding three stimulus values X, Y, Z by their sum (X+Y+Z).
[0003] In 1931, CIE recommended three primary colors and their representative wavelengths, namely 700nm red light, 546.1nm green light and 435.8nm blue light. The three primary colors can be mixed to produce equal energy white light, which is recorded as E light source, and the three stimulus values of equal energy white light are equal. Then, under the observation condition of 2° field of view, visual instruments are used to match various spectral colors, and the three primary colors (R, G, B) components of the generated equal energy spectral colors are defined as the CIE1931 RGB system standard colorimetric observer spectral three stimulus values, recorded as r (λ), g (λ), b (λ).
[0004] Since negative values appear in the calculation process of the CIE1931 RGB colorimetric system, CIE uses imaginary tristimulus values X, Y, and Z to establish a new colorimetric system, namely the CIE1931 XYZ standard colorimetric system. By selecting appropriate three primary colors, the spectral tristimulus values X (x), Y (y), and Z (z) are not negative, and Y is proportional to the brightness value.
[0005] Multispectral module is a new technology that has gradually emerged in recent years. Spectral analysis, as an important means of natural science analysis, can be used to detect various indicators such as the physical structure, material and chemical composition of objects. Traditional spectral analysis is mostly single-point position in space. Multispectral module combines spectral analysis with imaging to achieve spatial surface spectral analysis. In addition, multispectral module has a smaller appearance and can realize more application scenarios and functions compared with traditional spectrometers. It is widely used in portable material identification instruments, more accurate color correction tools and other application directions.
[0006] The multi-spectral module is mainly composed of three parts: silicon-based photoelectric conversion substrate, filter unit, and upper lens system, and the number of filter units is greater than 3. The multi-spectral module realizes spectral recognition mainly through algorithm recovery and other processes.
[0007] The multispectral module can also calculate the color coordinates by restoring the spectrum or restoring the tristimulus values. More accurate color coordinate calculation can enable the multispectral module to realize functions such as auxiliary photography, color restoration, and brightness detection, which is of great significance to the application and popularization of multispectral modules.
[0008] In the actual application of multi-spectral modules, due to many factors such as silicon-based substrate performance, filter unit material, lens transmittance, etc., there are errors in the recovery of color coordinates, which affects the final accuracy of color coordinate recovery.
[0009] Therefore, it is desirable to provide an improved multi-spectral module solution for color coordinate restoration or spectrum restoration. Summary of the invention
[0010] The embodiment of the present application provides a multi-spectral module, which optimizes the overall transmittance of the multi-spectral module by adding a filter element above the filter unit, so as to significantly improve the color coordinate restoration accuracy.
[0011] According to one aspect of the present application, a multi-spectral module is provided, comprising: a photoelectric conversion sensor based on silicon-based materials; a filter unit array, located on the photosensitive path of the photoelectric conversion sensor and formed by n types of filter units, wherein n is greater than 4; and a filter element, located on the photosensitive path of the photoelectric conversion sensor and having a transmittance curve, wherein the transmittance curve has a cutoff wavelength of 380-430nm and a cutoff range of 0-70nm in the wavelength band range of 380nm to 500nm; wherein, there is at least one maximum transmittance value in the wavelength band range of 380nm to 500nm, the cutoff wavelength is the wavelength corresponding to 50% of the maximum transmittance value, and the cutoff range is the absolute value of the difference between the wavelengths corresponding to 20% and 80% of the maximum transmittance value.
[0012] The multi-spectral module further includes: a lens system, which includes at least one optical lens located on the light sensing path of the photoelectric conversion sensor and on the filter unit array, for controlling the direction and intensity distribution of incident light.
[0013] In the above multi-spectral module, the filter element and the lens system are integrated into one body, or are respectively arranged on the light sensing path of the photoelectric conversion sensor.
[0014] In the above-mentioned multi-spectral module, the filter element is formed as a filter film plated on the at least one optical lens, or the filter element is combined with the at least one optical lens to form an optical lens with a filtering function.
[0015] The multi-spectral module further comprises: a micro-lens array formed on the surface of the filter unit array or the photoelectric conversion sensor.
[0016] In the above-mentioned multi-spectral module, the cut-off wavelength is 380nm, and the optimized cut-off range is 10nm, 30nm, 50nm or 70nm; or, the cut-off wavelength is 395nm, and the optimized cut-off range is 10nm, 30nm, 50nm or 70nm; or, the cut-off wavelength is 410nm, and the optimized cut-off range is 10nm, 30nm, 50nm or 70nm; or, the cut-off wavelength is 430nm, and the optimized cut-off range is 10nm, 30nm, 50nm or 70nm.
[0017] In the above multi-spectral module, the transmittance curve design of the filter element includes:
[0018] The initial transmittance curve of the filter is multiplied by the transmittance curve of the multi-spectral module, and the restoration error is calculated based on the following formula:
[0019]
[0020] is the matrix product of the conversion matrix from multi-spectrum to color space and the transmittance matrix of the multi-spectral module, XYZ is the spectral tristimulus value curve of the standard chromaticity observer of the 1931 CIE-XYZ system, To restore the error result, mean refers to the mean value. refers to the square root; and
[0021] The restoration error is optimized to a minimum value by a local search non-gradient optimization algorithm, so as to return the optimal cut-off wavelength and cut-off range as the cut-off wavelength and cut-off range of the optical filter.
[0022] In the above multi-spectral module, the optimized transmittance curve of the filter further has a second cutoff wavelength between 610 nm and 700 nm in the wavelength band of 560 nm to 780 nm and a second cutoff range of 20 nm to 90 nm.
[0023] In the above multi-spectrum module, the second cut-off wavelength and the second cut-off range are based on 50% of the maximum transmittance in the 560nm-780nm band, and 20% to 80% of the maximum transmittance.
[0024] According to another aspect of the present application, a method for calculating a color coordinate restoration error is provided, which is used to calculate the error of color coordinate restoration of the multi-spectral module as described above, including: obtaining transmittance matrices of different channels of the multi-spectral module; calculating corresponding original values according to the transmittance matrices of the different channels and a reflectance spectrum data set; adding noise to the original values; calculating a transformation matrix by a least squares method according to the noisy original values corresponding to a training set divided from the reflectance spectrum data set; restoring three stimulus values from the noisy original values corresponding to a test set divided from the reflectance spectrum data set using the transformation matrix; calculating restored color coordinates according to the three stimulus values; and comparing the errors of the restored color coordinates and true color coordinates.
[0025] The multi-spectral module provided in the embodiment of the present application can optimize the overall transmittance of the multi-spectral module by adding a filter element above the filter unit, so as to significantly improve the color coordinate restoration accuracy or the spectrum restoration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] By reading the detailed description of the preferred specific embodiments below, various other advantages and benefits of the present application will become clear to those of ordinary skill in the art. The drawings in the specification are only used for the purpose of illustrating the preferred embodiments and are not considered to be limitations of the present application. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work. Moreover, the same reference numerals are used to represent the same components throughout the drawings.
[0027] Figure 1 The figure shows a schematic diagram of a multi-spectral module according to an embodiment of the present application.
[0028] Figure 2 The diagram illustrates an example of optimizing the transmittance curve of an optical filter in a multi-spectral module according to an embodiment of the present application.
[0029] Figure 3A to Figure 3P The diagrams illustrate first to sixteenth examples of optimized transmittance curves of the optical filter in the multi-spectral module according to an embodiment of the present application.
[0030] Figures 4A to 4P The diagrams illustrate the first to sixteenth examples of optimized transmittance curves of the optical filter element in the multi-spectral module for infrared band filtering according to an embodiment of the present application.
[0031] Figure 5 FIG. 1 is a schematic diagram showing improvement of color coordinate restoration error of a multi-spectral module according to an embodiment of the present application.
[0032] Fig. 6AA schematic diagram showing the comparison between the restored results without filters and the true values of the tristimulus values is shown.
[0033] Figure 6B The figure shows a schematic diagram comparing the restored results with filters and the true values of tristimulus values. DETAILED DESCRIPTION
[0034] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0035] As mentioned above, the existing multi-spectral module is mainly composed of three parts: a silicon-based photoelectric conversion substrate, that is, a photoelectric conversion sensor as a silicon-based substrate for realizing photoelectric conversion; a filter unit array for controlling spectral modulation differences, and the filter unit array is formed by n types of filter units; a lens system for controlling the direction and intensity distribution of incident light. Generally speaking, the image sensors used in the prior art are implemented as single-channel black-and-white image sensors, three-channel RGB image sensors, and RGB-IR\RGBW image sensors for image restoration, while the present invention is applied to the field of multi-spectral imaging, so the number of filter units corresponding to the present invention is greater than 4, that is, n is greater than 4.
[0036] Taking a multi-spectral module as an example, assuming that the spectral modulation matrix of the multi-spectral module is T, the number of types of filter units of the multi-spectral module is n, and the 1931 CIE-XYZ system standard chromaticity observer spectral tristimulus value curve is XYZ, then through a 3×n conversion matrix A, it is possible to fit the tristimulus value curve with the multi-spectral modulation curve, which is expressed as ,in Represents matrix multiplication. The closer it is to XYZ, the stronger the module's color coordinate recovery ability is, that is, the higher the color coordinate recovery accuracy is. It is understandable that a similar situation also exists in spectrum recovery. Here, the transformation matrix A can be obtained by calculating the product of XYZ and the pseudo-inverse of T.
[0037] Due to the limitations of process and materials, the existing multi-spectral modules have some defects. Most of the photoelectric conversion sensors on silicon-based substrates on the market have relatively close photoelectric conversion efficiencies in the blue light band (e.g., 380-500nm) and small changes in the QE slope due to the limitations of the silicon material itself. The filter materials currently used as filter units are implemented as dyes or pigments. Due to the limitations of the filter materials themselves, most filter materials have relatively close transmittances near the blue light band, and the transmittance slope changes slightly. For example, the existing filter materials in the blue band have a relatively high transmittance in the blue light band. Therefore, in general, the transmittance of the multi-spectral module in the blue light band is relatively consistent and the transmittance is relatively high, which is quite different from the performance of the three stimulus values in the blue light band, and the difference cannot be reduced by selecting and optimizing the filter materials and photoelectric conversion sensors. Therefore, the error of fitting the three stimulus value curve in the blue light band is large.
[0038] Based on this, an embodiment of the present application provides a multi-spectral module, including: a photoelectric conversion sensor based on silicon-based materials; a filter unit array, located on the photosensitive path of the photoelectric conversion sensor and formed by n types of filter units, wherein n is greater than 4; and a filter element, located on the photosensitive path of the photoelectric conversion sensor and having a transmittance curve, wherein the transmittance curve has a cut-off wavelength of 380-430nm in the wavelength band of 380nm to 500nm and an optimized cut-off range of 0-70nm; wherein, there is at least one maximum transmittance value in the wavelength band of 380nm to 500nm, the cut-off wavelength is the wavelength corresponding to 50% of the maximum transmittance value, and the cut-off range is the absolute value of the difference between the wavelengths corresponding to 20% and 80% of the maximum transmittance value.
[0039] The photoelectric conversion sensor may be implemented as a photosensitive array sensor such as a CMOS chip, a CCD chip, etc., which is generally implemented as a silicon-based material, but those skilled in the art will appreciate that it may also be formed of other materials.
[0040] The filter unit array is formed at the upper end of the photoelectric conversion sensor. The filter unit array can preferably be directly formed on the surface of the photoelectric conversion sensor to form a multi-spectral chip; it can also be located above the photoelectric conversion sensor (not in contact with the photoelectric conversion sensor), that is, the filter unit array and the photoelectric conversion sensor are composed of two components. The filter unit array is used to modulate the incident light. In the embodiment of the present application, the filter unit array can be composed of structures or materials such as metasurfaces, filter materials (filter materials are dyes or pigments), quantum dots, photonic crystals, etc. Generally, its transmittance near blue light is relatively close, and the transmittance slope changes less. It should be noted that the present invention refers to the type of filter unit array due to the characteristics of structure, material, etc., so that the transmittance of the same type of filter units (such as filter units composed of filter materials) near blue light is close.
[0041] The examples are not limiting. The embodiments of the present application are described by taking the filter unit array composed of different types of filter materials as an example. The filter unit array can be directly formed on the surface of the photoelectric conversion sensor to form a multi-spectral chip. In the embodiments of the present application, the multi-spectral chip can include n types of filter materials, that is, the n types of filter materials respectively constitute n filter units, each filter unit corresponds to a transmittance curve T1, T2, …, Tn, and the transmission spectrum matrix Tcf of the filter unit array can be considered to be composed of T1, T2, …, Tn; and then the array is arranged to obtain the filter unit array. Among them, in order to better restore the color coordinates, in the embodiment of the present application, the multi-spectral module requires more channels, so preferably the type of filter units n of the multi-spectral module is greater than 4, that is, the filter units of the filter unit array of the multi-spectral module are greater than 4; it should be noted that, generally speaking, the more the number of filter units, the more conducive to improving the restoration accuracy, regardless of the process and the restoration effect. Theoretically speaking, in order to achieve a better restoration effect, more filter units are required, more preferably, greater than 6 filter units, 7 filter units, more preferably greater than 8 filter units, for example, 9 filter units, 16 filter units, etc.
[0042] Usually, the filter unit is arranged corresponding to the pixel of the photoelectric conversion sensor, for example, one filter unit corresponds to one pixel, or multiple pixels (2, 4, etc.). In some cases, at least one filter material may correspond to one or more pixels to form a filter unit, for example, two or more filter materials are stacked or arrayed on the top of at least one pixel to form a filter unit. By way of example but not limitation, at least 9 filter materials may be used to form 9 filter units, which are arranged in a 3×3 manner to form a filter unit group, and the filter unit group is then arrayed to obtain a filter unit array. It can be understood that the filter unit array is composed of more than 4 filter units that can achieve spectral curve recovery; of course, the filter unit array may be any form of array arrangement, i.e., n×m, n≥1, m≥1, and m may be equal to n, such as 4×4; wherein the number of filter materials in the filter unit array may also be less than the number of filter units in the filter unit group, for example, 3×3 corresponds to 9 filter units, but the number of filter materials is less than 9, that is, there may be repeated filter units, or a new filter unit may be formed by the joint action of at least two filter materials; in some examples, the number of filter material types may also be greater than the number of filter units in the filter unit group, that is, there are at least two filter materials that work together to form a filter unit, such as a filter unit formed by two filter materials stacked, arranged in parallel, or mixed. At the same time, in the embodiment of the present application, the filter unit array can be obtained by arranging the filter units according to the requirements, and is not limited to the periodic arrangement of the filter unit group, and can also be a non-periodic arrangement, that is, it is designed according to the requirements, and the embodiment of the present application has no clear limitation on the arrangement of the filter units in the filter unit array. It should be noted that in the embodiment of the present application, a blank pixel can also be listed as a filter unit, that is, when there is a pixel of the photoelectric conversion sensor without a filter unit, the pixel is a blank pixel. For example, a 3×3 filter unit group can be composed of 8 filter materials, with a blank pixel in the middle.
[0043] In addition, in an embodiment of the present application, the multi-spectral module may further include a lens system, which includes at least one optical lens located on the light-sensitive path of the photoelectric conversion sensor and on the filter unit array, for controlling the direction and intensity distribution of the incident light.
[0044] In the embodiment of the present application, the optical filter and the lens system can together constitute an optical system, wherein the optical filter and the lens system can be integrated, for example, a filter film is plated on the at least one optical lens to form the lens system; the optical filter and at least one optical lens can also be combined to form an optical lens with a filtering function. Alternatively, the lens system and the optical filter are respectively arranged on the light-sensitive path of the photoelectric conversion sensor, and the optical filter can be implemented as a filter. Of course, in the embodiment of the present application, the optical system may also include only the optical filter but not the lens system.
[0045] That is, in the multi-spectral module according to the embodiment of the present application, the filter element and the lens system are integrated into one, or are separately arranged on the light sensing path of the photoelectric conversion sensor.
[0046] Furthermore, in the above-mentioned multi-spectral module, the optical filter is formed as a filter film plated on the at least one optical lens, or the optical filter is combined with the at least one optical lens to form an optical lens with a filtering function.
[0047] Further, in the embodiment of the present application, the multi-spectral module may further include a microlens array, and the microlens array may be formed on the surface of the filter unit array or on the surface of the photoelectric conversion sensor. And, for the blank pixels as described above, it is preferred that the microlens array is not provided. Of course, according to the requirements, in some examples, a microlens array may also be provided on the blank pixels.
[0048] Therefore, in the multi-spectral module according to the embodiment of the present application, it further includes a microlens array formed on the surface of the filter unit array or the photoelectric conversion sensor.
[0049] In an embodiment of the present application, the transmittance of the existing multi-spectral module in the wavelength band range of 380nm to 500nm does not meet the target. Therefore, for the purpose of optimizing the transmittance in the wavelength band range of 380nm to 500nm, the filter element is defined, for example, as a filter, with a predetermined transmittance of a predetermined percentage of the maximum transmittance in the wavelength band range of 380nm to 500nm, for example, 50% of the maximum transmittance is the cut-off wavelength, and the transmittance range corresponding to the predetermined transmittance, for example, the wavelength range corresponding to 20% and 80% of the maximum transmittance is the cut-off range.
[0050] That is, in the embodiment of the present application, for the design and optimization of the optical filter, it is necessary to use the initial transmittance curve, that is, the transmittance curve of the main optical filter available on the market, and define the initial cut-off wavelength and initial cut-off range of the optical filter. Specifically, within the wavelength range of 380nm to 500nm, there is at least one maximum transmittance, and the wavelength corresponding to 50% of the maximum transmittance is taken as the cut-off wavelength, and the cut-off range refers to the absolute value of the difference between the wavelengths corresponding to 20% and 80% of the maximum transmittance within the wavelength range of 380 to 500nm. At the same time, it should be noted that if the transmittance curve of the filter has a high overall transmittance in the range of 380nm-500nm, there may be a transmittance value at 380nm that is greater than 50% of the maximum transmittance, and the wavelength corresponding to 50% of the maximum transmittance does not fall within the 380-500nm band. In this case, the cut-off wavelength can be considered to be 380nm; similarly, when calculating the cut-off range, if within the 380 to 500nm band, the wavelength corresponding to 20% of the maximum transmittance is less than the transmittance value of 380nm, that is, the wavelength corresponding to 20% of the maximum transmittance does not fall within 380nm-500nm, then the wavelength of 20% of the maximum transmittance is calculated as 380nm; and in the case where there are multiple values, there only needs to be an absolute value of the difference between the wavelengths corresponding to 20% and 80% of the maximum transmittance that falls within a certain range, and the range is 0nm-70nm.
[0051] Therefore, in the multi-spectral module for color coordinate restoration according to the embodiment of the present application, the predetermined transmittance corresponding to the initial cut-off wavelength designed for the transmittance curve of the filter is 50% of the maximum transmittance in the 380nm to 500nm band, and the transmittance range corresponding to the initial cut-off range is 20% to 80% of the maximum transmittance.
[0052] Then, based on the initial cutoff wavelength and the initial cutoff range, the transmittance curve of the filter is modified: first, the initial transmittance curve of the filter is multiplied by the transmittance curve of the multi-spectral module, and the result is brought into the following formula to calculate the restoration error:
[0053]
[0054] Here, as mentioned above, is the matrix product of the conversion matrix A from multi-spectrum to color space and the transmittance matrix T of the multi-spectral module, wherein the conversion matrix A is a 3×n matrix, XYZ is the spectral tristimulus value curve of the standard chromaticity observer of the 1931 CIE-XYZ system, To restore the error result, mean refers to the mean value. Refers to the square root.
[0055] Afterwards, the restoration error is optimized to a minimum value by a local search non-gradient optimization algorithm, such as Nelder-Mead, so as to return the optimized cut-off wavelength and cut-off range as the cut-off wavelength and cut-off range of the filter. Here, Figure 2 The diagram illustrates an example of optimizing the transmittance curve of an optical filter in a multi-spectral module according to an embodiment of the present application.
[0056] Some examples of optimized transmittance curves of the optical filter in the multi-spectral module according to the embodiments of the present application are described below.
[0057] Figure 3A to Figure 3P The diagram shows the first to sixteenth examples of the optimized transmittance curve of the filter in the multi-spectral module according to the embodiment of the present application. In the first to sixteenth examples, the optimized transmittance curve of the filter satisfies the following conditions: within the wavelength range of 380 to 500 nm, the cut-off wavelength is between 380-430 nm, and the cut-off range is between 0-70 nm.
[0058] like Figure 3A As shown, in the first example, the cutoff wavelength is 380nm at 50% of the maximum value; at the same time, 20% of the maximum transmittance is not within the range, so 380nm is taken for calculation, and the cutoff range is 10nm, which is the wavelength at 80% of 390nm minus 380nm.
[0059] like Figure 3B As shown, in the second example, the cutoff wavelength is 380nm at 50% of the maximum value; at the same time, 20% of the maximum transmittance is not within the range, so 380nm is taken for calculation, that is, 410nm minus 380nm, and the cutoff range is 30nm.
[0060] like Figure 3C As shown, in the third example, 50% of the maximum transmittance does not fall within 380nm-500nm, and the transmittance value at 380nm is greater than 50% of the maximum transmittance, then the cutoff wavelength is 380nm; at the same time, 20% of the maximum transmittance is not within the range, then 380nm is taken for calculation, that is, 430nm minus 380nm, and the cutoff range is 50nm.
[0061] like Figure 3D As shown, in the fourth example, 50% of the maximum transmittance does not fall within 380nm-500nm, and the transmittance value at 380nm is greater than 50% of the maximum transmittance, then the cutoff wavelength is 380nm; at the same time, 20% of the maximum transmittance is not within the range, then 380nm is taken for calculation, that is, 450nm minus 380nm, and the cutoff range is 70nm.
[0062] like Figure 3EAs shown, in the fifth example, the cut-off wavelength is about 395 nm, and the cut-off range 400 nm minus 390 nm is 10 nm.
[0063] like Figure 3F As shown, in the sixth example, the cut-off wavelength is about 395 nm; at the same time, 20% of the maximum transmittance is not within the range, so 380 nm is taken for calculation, that is, 410 nm minus 380 nm, and the cut-off range is 30 nm.
[0064] like Figure 3G As shown, in the seventh example, the cut-off wavelength is about 395 nm; at the same time, 20% of the maximum transmittance is not within the range, so 380 nm is taken for calculation, that is, 430 nm minus 380 nm, and the cut-off range is 50 nm.
[0065] like Figure 3H As shown, in the eighth example, the cut-off wavelength is about 395 nm; at the same time, 20% of the maximum transmittance is not within the range, so 380 nm is taken for calculation, that is, 450 nm minus 380 nm, and the cut-off range is 70 nm.
[0066] like Fig. 3I As shown, in the ninth example, the cut-off wavelength is about 410 nm; the cut-off range is 415 nm minus 405 nm, which is 10 nm.
[0067] like Figure 3J As shown, in the tenth example, the cut-off wavelength is about 410 nm; the cut-off range is 424 nm minus 394 nm, which is 30 nm.
[0068] like Figure 3K As shown, in the eleventh example, the cut-off wavelength is about 410 nm; the cut-off range is 433 nm minus 383 nm, which is 50 nm.
[0069] like Figure 3L As shown, in the twelfth example, the cut-off wavelength is about 410 nm; at the same time, 20% of the maximum transmittance is not within the range, so 380 nm is taken for calculation, that is, 450 nm minus 380 nm, and the cut-off range is 70 nm.
[0070] like Figure 3M As shown, in the thirteenth example, the cut-off wavelength is about 430 nm; the cut-off range is 435 nm minus 425 nm, which is 10 nm.
[0071] like Figure 3N As shown, in the fourteenth example, the cut-off wavelength is about 430 nm; the cut-off range is 444 nm minus 414 nm, which is 30 nm.
[0072] like Fig.3OAs shown, in the fifteenth example, the cut-off wavelength is about 430 nm; the cut-off range is 453 nm minus 403 nm, which is 50 nm.
[0073] like Figure 3P As shown, in the sixteenth example, the cut-off wavelength is about 430nm; the cut-off range is 70nm, which is 462nm minus 392nm. It should be noted that the transmittance of the filter in other bands will be designed according to the needs, and this application does not constitute a limitation. Figure 3A to Figure 3P Here, 100% transmittance is only an example and does not constitute a limitation.
[0074] Furthermore, the applicant of the present application has found that if the filter further adds infrared band filtering, the color coordinate recovery accuracy can be further increased, wherein the main index range of the infrared band filtering of the filter is: between the 560-780nm band, there is a second cutoff wavelength between 610nm-700nm, and there is a second cutoff range between 20-90nm. It should be noted that if there are multiple 50% of the maximum transmittance, only one wavelength needs to fall between 610nm-700nm; if there are multiple 20% and / or 80% of the maximum transmittance, only the absolute value of the difference between the wavelengths corresponding to 20% and 80% of the maximum transmittance needs to fall between 20-90nm. At the same time, if there is a cutoff range, one of the values of 20% and 80% of the maximum transmittance does not exist between 560nm and 780nm, then 560nm or 780nm can be taken for calculation.
[0075] Therefore, in the multi-spectral module according to the embodiment of the present application, the optimized transmittance curve of the filter further has a second cutoff wavelength between 610 nm and 700 nm within the wavelength band of 560 nm to 780 nm and a second cutoff range of 20 nm to 90 nm.
[0076] Furthermore, in the above multi-spectrum module, the second cut-off wavelength and the second cut-off range are respectively based on 50% of the maximum transmittance within the 560nm-780nm band and the absolute value of the difference between 20% and 80% of the maximum transmittance.
[0077] Figures 4A to 4P The diagrams illustrate the first to sixteenth examples of optimized transmittance curves of the optical filter element in the multi-spectral module for infrared band filtering according to an embodiment of the present application.
[0078] like Figure 4A As shown, in the first example, the cut-off wavelength is about 610 nm; the cut-off range is 621 nm minus 601 nm, which takes an absolute value of 20 nm.
[0079] like Figure 4B As shown, in the second example, the cut-off wavelength is about 610 nm; the cut-off range is 632 nm minus 592 nm, which takes an absolute value of 40 nm.
[0080] like Figure 4C As shown, in the third example, the cut-off wavelength is about 610 nm; the cut-off range is 643 nm minus 583 nm, which takes an absolute value of 60 nm.
[0081] like Figure 4D As shown, in the fourth example, the cut-off wavelength is about 610 nm; the cut-off range is 659 nm minus 569 nm, which takes an absolute value of 90 nm.
[0082] like Figure 4E As shown, in the fifth example, the cut-off wavelength is about 640 nm; the cut-off range is 651 nm minus 631 nm, which takes an absolute value of 20 nm.
[0083] like Figure 4F As shown, in the sixth example, the cut-off wavelength is about 640 nm; the cut-off range is 662 nm minus 622 nm, which takes an absolute value of 40 nm.
[0084] like Figure 4G As shown, in the seventh example, the cut-off wavelength is about 640 nm; the cut-off range is 673 nm minus 613 nm, which takes an absolute value of 60 nm.
[0085] like Figure 4H As shown, in the eighth example, the cut-off wavelength is about 640 nm; the cut-off range is 689 nm minus 599 nm, which takes an absolute value of 90 nm.
[0086] like Fig. 4I As shown, in the ninth example, the cut-off wavelength is about 670 nm; the cut-off range is 681 nm minus 661 nm, which takes an absolute value of 20 nm.
[0087] like Figure 4J As shown, in the tenth example, the cut-off wavelength is about 670 nm; the cut-off range is 692 nm minus 652 nm, which takes an absolute value of 40 nm.
[0088] like Figure 4K As shown, in the eleventh example, the cut-off wavelength is about 670 nm; the cut-off range is 703 nm minus 643 nm, which takes an absolute value of 60 nm.
[0089] like Figure 4L As shown, in the twelfth example, the cut-off wavelength is about 670 nm; the cut-off range is 719 nm minus 629 nm, which takes an absolute value of 90 nm.
[0090] like Figure 4MAs shown, in the thirteenth example, the cut-off wavelength is about 700 nm; the cut-off range is 711 nm minus 691 nm, which takes an absolute value of 20 nm.
[0091] like Figure 4N As shown, in the fourteenth example, the cut-off wavelength is about 700 nm; the cut-off range is 722 nm minus 682 nm, which takes an absolute value of 40 nm.
[0092] like Fig.4O As shown, in the fifteenth example, the cut-off wavelength is about 700 nm; the cut-off range is 733 nm minus 673 nm, which takes an absolute value of 60 nm.
[0093] like Figure 4P As shown, in the sixteenth example, the cut-off wavelength is about 700 nm; the cut-off range is 749 nm minus 659 nm, which takes an absolute value of 90 nm.
[0094] In the embodiment of the present application, the filter unit may be formed on the surface of the filter element. For example, the filter element may be first formed on the surface of the photoelectric conversion sensor, and then the filter unit may be formed on the surface of the filter element.
[0095] In the embodiment of the present application, the filter element may also be formed on the surface of the filter unit. For example, the filter unit may be first formed on the surface of the photoelectric conversion sensor, and then the filter element may be formed on the surface of the filter unit.
[0096] Therefore, in the embodiment of the present application, by adding a filter with an optimized transmittance curve, the calculated color coordinate restoration error is significantly reduced, such as Figure 5 shown. Figure 5 FIG. 1 is a schematic diagram showing the improvement of the color coordinate restoration error of the multi-spectral module according to an embodiment of the present application. Figure 5 In the figure, the horizontal axis is the signal-to-noise ratio, and the vertical axis is the sum of the x and y color coordinate errors (the smaller the better). It can be seen that with the increase of the signal-to-noise ratio, the color coordinate recovery accuracy is improved, and the signal-to-noise ratio tends to be stable after being optimized to a certain extent.
[0097] In another example, the improvement of the color coordinate recovery error can be judged from the tristimulus values, that is, the tristimulus value curve recovery is significantly improved, thereby improving the color coordinate recovery accuracy, such as Fig. 6A and Figure 6B shown. Fig. 6A A schematic diagram showing the comparison between the restored results without filters and the true values of the tristimulus values is shown. Figure 6B The figure shows a comparison between the restored results with filters and the true values of the three stimulus values. Fig. 6A and Figure 6B In the figure, the horizontal axis is the wavelength and the vertical axis is the relative intensity.
[0098] In an embodiment of the present application, a calculation method for improving the color coordinate restoration error can be provided. The method is used for a multi-spectral module, wherein the multi-spectral module includes a photoelectric conversion sensor of a common silicon-based substrate, and is implemented as a 9-channel filter unit array for example, that is, 9 types of filter units are formed by 9 different filter materials (filter units with or without filter materials may exist, that is, blank pixels), and then arranged in an array to obtain the filter unit array, and a lens system corresponding to the photoelectric conversion sensor.
[0099] The calculation method includes:
[0100] 1. Select an appropriate light source database, such as a natural broadband light source reflectance spectrum dataset, where the corresponding sample size can be set according to demand, for example, 50,000-250,000, which can be 50,000, 80,000, 100,000, 120,000, 160,000, etc.; randomly divide the reflectance spectrum dataset into a training set and a test set according to a certain ratio (such as 9:1).
[0101] 2. Input the transmittance matrix T of different channels of the multispectral module.
[0102] 3. Calculate the corresponding raw values based on the transmittance matrix T and reflectance spectrum data set of different channels.
[0103] 4. Add noise to the original value.
[0104] 5. According to the original values after adding noise corresponding to the training set, the transformation matrix A is calculated by the least squares method, and then the transformation matrix A is used to restore the three stimulus values from the original values after adding noise corresponding to the test set.
[0105] 6. Based on the tristimulus values recovered from the test set, calculate the corresponding color coordinates.
[0106] 7. Compare the error between the restored color coordinates and the true color coordinates.
[0107] Calculation of the transformation matrix A:
[0108] In linear regression, or any form of linear model, the goal is to find a parameter vector (or matrix) that minimizes the difference (usually the sum of squared errors) between the model's predictions and the actual observed values. Mathematically, this can be formulated as solving the following optimization problem:
[0109]
[0110] XYZ is the incident light tristimulus value, the original value ( ) is the pixel response value of the sensor corresponding to the n filter units of the multispectral module. β is the corresponding conversion matrix, which is responsible for mapping the pixel response values of the n filter units to the three stimulus values.
[0111] In order to obtain the parameter β with the minimum sum of squared errors, we set the derivative to zero to solve, so:
[0112]
[0113] when When it is not reversible, use the pseudo-inverse:
[0114]
[0115] This solution is the optimal solution in the least squares sense.
[0116] Calculation of the original value:
[0117] After solving the calculation problem of the transformation matrix A, another issue that needs to be considered is the original value ( ), since the transmittance matrix T corresponding to the corresponding channel of the multi-spectral module can be measured, when the reflection spectrum training data of natural light is further measured, for example, it can be expressed as Then, we can get by integrating along the wavelength .Right now:
[0118]
[0119] in, is the pixel response value vector corresponding to the n filter units of the multispectral module, T is the transmittance matrix corresponding to the n channels of the multispectral module, It is a training dataset of the collected natural broadband light source reflectance spectrum.
[0120] Here, in the embodiment of the present application, the transmittance matrix T of the multi-spectral module can be understood as:
[0121]
[0122] Transmittance The transmittance of the lens system , the transmittance of the filter element array , the transmittance of the filter (filter) and sensor response If the lens is not included, it can be understood as , the transmittance of the filter element array , the transmittance of the filter (filter) and sensor response It should be noted that, since there may be a microlens array, the transmittance of the multi-spectral module can be calculated by default. middle.
[0123] Addition of original value noise:
[0124] Since there are various noises in the pixel response values in the actual process, in order to make the simulation results close to the real values, noise is added to the above original value results. The process is:
[0125]
[0126] in is the original value after adding noise, is the theoretical original value without noise, is the noise value calculated according to the actual noise model.
[0127] According to the noisy Calculate the transformation matrix:
[0128]
[0129] Calculation of recovery error:
[0130] Preparing a test spectrum set , and calculate the original value of noise corresponding to each spectrum in the spectrum set, combined with the conversion matrix obtained by the above calculation , get the predicted XYZ value, and then get the predicted color coordinates , calculated and theoretical color coordinates The error:
[0131]
[0132]
[0133] On this basis, the color coordinate error can be obtained .
[0134] Therefore, an embodiment of the present application further provides a method for calculating a color coordinate restoration error, which is used to calculate the error of color coordinate restoration of a multi-spectral module used for color coordinate restoration as described above, including: obtaining transmittance matrices of different channels of the multi-spectral module; calculating corresponding original values based on the transmittance matrices of the different channels and a reflectance spectrum data set; adding noise to the original values; calculating a transformation matrix by a least squares method based on the noisy original values corresponding to a training set divided from the reflectance spectrum data set; restoring three stimulus values from the noisy original values corresponding to a test set divided from the reflectance spectrum data set using the transformation matrix; calculating restored color coordinates based on the three stimulus values; and comparing the errors between the restored color coordinates and true color coordinates.
[0135] In addition, in an embodiment of the present application, a calculation method for improving the error of a three-stimulus value curve can be provided. The method is used for a multi-spectral module. The multi-spectral module includes a photoelectric conversion sensor on a common silicon-based substrate, which is implemented as a 9-channel filter unit array, that is, 9 types of filter units are formed by 9 different filter materials, and then arranged in an array to obtain the filter unit array, and a lens system corresponding to the photoelectric conversion sensor.
[0136] The method includes:
[0137] 1. Extract the transmittance matrix T corresponding to different channels of the multispectral module and calculate the conversion matrix A.
[0138] 2. Use the transmittance matrix T and the conversion matrix A to fit the tristimulus value curve.
[0139] 3. Calculate the error between the fitting result and the true tristimulus value curve.
[0140] 4. Draw the fitting results and the true three stimulus value curves for intuitive comparison.
[0141] Therefore, an embodiment of the present application further provides a method for calculating the error of a three-stimulus value curve, which is used for calculating the color coordinate restoration error of the multi-spectral module used for color coordinate restoration as described above, including: obtaining the transmittance matrix of different channels of the multi-spectral module; calculating a conversion matrix according to the transmittance matrix of the different channels; fitting a three-stimulus value curve according to the transmittance matrix and the conversion matrix; and calculating the error between the fitted three-stimulus value curve and the true three-stimulus value curve.
[0142] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.
[0143] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.
[0144] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.
[0145] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0146] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A multi-spectral module, include: Photoelectric conversion sensors based on silicon-based materials; a filter unit array, located on the light sensing path of the photoelectric conversion sensor and formed by n types of filter units, wherein n is greater than 4; and A filter, located on the light sensing path of the photoelectric conversion sensor, and having a transmittance curve, wherein the transmittance curve has a cutoff wavelength of 380-430 nm and a cutoff range of 0-70 nm in a wavelength band of 380 nm to 500 nm; There is at least one maximum transmittance in the wavelength range of 380nm to 500nm, the cutoff wavelength is the wavelength corresponding to 50% of the maximum transmittance, and the cutoff range is the absolute value of the difference between the wavelengths corresponding to 20% and 80% of the maximum transmittance.
2. The multi-spectral module according to claim 1, further comprising: include: A lens system, wherein the lens system comprises at least one optical lens located on the light sensing path of the photoelectric conversion sensor and on the filter unit array, and is used to control the direction and intensity distribution of incident light.
3. The multi-spectral module according to claim 2, in, The filter element and the lens system are integrated into one body, or are respectively arranged on the light sensing path of the photoelectric conversion sensor.
4. The multi-spectral module according to claim 3, in, The optical filter is formed as a filter film plated on the at least one optical lens, or the optical filter is combined with the at least one optical lens to form an optical lens with a filtering function.
5. The multi-spectral module according to claim 1, further comprising: include: A microlens array is formed on the surface of the filter unit array or the photoelectric conversion sensor.
6. The multi-spectral module according to claim 1, in, The cut-off wavelength is 380 nm, and the cut-off range is 10 nm, 30 nm, 50 nm or 70 nm; or The cut-off wavelength is 395 nm, and the cut-off range is 10 nm, 30 nm, 50 nm or 70 nm; or The cut-off wavelength is 410 nm, and the cut-off range is 10 nm, 30 nm, 50 nm or 70 nm; or The cut-off wavelength is 430 nm, and the cut-off range is 10 nm, 30 nm, 50 nm or 70 nm.
7. The multi-spectral module according to claim 6, in, The light transmission curve design of the filter element includes: The initial transmittance curve of the filter is multiplied by the transmittance curve of the multi-spectral module, and the restoration error is calculated based on the following formula: ; XYZ fit is the matrix product of the conversion matrix from multi-spectrum to color space and the transmittance matrix of the multi-spectral module, XYZ is the 1931 CIE-XYZ system standard chromaticity observer spectrum tristimulus value curve, error is the recovery error result, mean refers to the mean, and sqrt refers to the square root; and The restoration error is optimized to a minimum value by a local search non-gradient optimization algorithm, so as to return the optimized cut-off wavelength and cut-off range as the cut-off wavelength and cut-off range of the optical filter.
8. The multi-spectral module according to claim 1, in, The transmittance curve of the optical filter further has a second cutoff wavelength between 610 nm and 700 nm within the wavelength band of 560 nm to 780 nm and a second cutoff range of 20 nm to 90 nm.
9. The multi-spectral module according to claim 8, in, The second cut-off wavelength and the second cut-off range are based on 50% of the maximum transmittance in the 560nm-780nm band and an absolute value of a difference between 20% and 80% of the maximum transmittance.
10. A method for calculating a color coordinate restoration error, for calculating the color coordinate restoration error of the multi-spectral module according to any one of claims 1 to 9, include: Obtaining a transmittance matrix of different channels of the multispectral module; Calculate the corresponding original values according to the transmittance matrices and reflectance spectrum data sets of the different channels; Add noise to the original value; Calculating a conversion matrix by least square method according to the original values after adding noise corresponding to the training set divided from the reflectance spectrum data set; Using the conversion matrix, recover tristimulus values from the original values after adding noise corresponding to the test set divided from the reflectance spectrum data set; Calculate the restored color coordinates according to the tristimulus values; as well as, Compare the errors between the restored color coordinates and the true color coordinates.
