Method for acquiring parameters of ambient light and projection surface and spectral imaging equipment
By using multi-color blocks to set the image in the projection device, and combining the calculation method, the problem of ambient light and projection surface parameters is solved, the projection effect is improved, and it is suitable for a variety of terminal devices.
Patent Information
- Application Number
- CN202310762415.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-26
- Publication Date
- 2024-12-27
AI Technical Summary
Existing projection devices are affected by ambient light and projection surface parameters in the projection effect, and it is difficult to accurately obtain these parameters to improve the projection effect.
By using a set image composed of multiple color blocks of different brightness and colors, it is projected to the projection surface and obtained the reflection spectrum, and combined with least squares method or neural network modeling, the parameters of the ambient light and the projection surface are calculated.
It realizes accurate acquisition of ambient light and projection surface parameters, improves the projection effect of projection equipment, and is suitable for various terminal equipment, including mobile phones, home appliances and automotive electronics fields.
Smart Images
Figure CN119211494A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of optical processing technology, and more specifically, to a method for acquiring parameters of ambient light and a projection surface and a spectral imaging device. Background Art
[0002] A projection device, also known as a projector or a projector, is an electronic device that can project image content onto a target area (e.g., a projection screen area, a blank area on a wall, etc.). Currently, before projecting image content onto a target area through a projection device, the projection device needs to be dimmed to ensure accurate display.
[0003] Currently, there are two main external factors that affect the projection effect of the projection device: ambient light and the condition of the projection surface of the target area to be projected. Therefore, if the parameters of the ambient light and the projection surface can be accurately obtained, the projection effect of the projection device can be significantly improved.
[0004] Therefore, it is desirable to provide a solution for obtaining parameters of ambient light and projection surface. Summary of the invention
[0005] An embodiment of the present application provides a method for obtaining parameters of ambient light and a projection surface and a spectral imaging device, which can calculate the parameters of ambient light and the projection surface based on the reflectance spectrum of a set image including multiple color blocks in different areas of the projection surface, thereby conveniently and accurately obtaining the parameters of ambient light and the projection surface.
[0006] According to one aspect of the present application, a method for obtaining parameters of ambient light and a projection surface is provided, comprising: determining a base light spectrum of a set image, the set image comprising a plurality of color blocks of arbitrary colors with different brightnesses; projecting the set image onto a target area of a projection surface, and obtaining a reflection spectrum of the set image reflected from the target area of the projection surface; and, based on the reflection spectra corresponding to the plurality of color blocks of the set image, obtaining an ambient light spectrum and a reflection spectrum of the projection surface.
[0007] In the above-mentioned method for obtaining parameters of ambient light and projection surface, the set image is a single image including multiple color blocks of arbitrary colors with different brightness, or multiple images each containing multiple color blocks of arbitrary colors with different brightness, each of the multiple images contains one or more color blocks, and the one or more color blocks are the same or different brightness and the same or different colors.
[0008] In the above method for obtaining parameters of ambient light and projection surface, determining the base light color of at least one set image includes: projecting base light of known spectrum onto a target area of the projection surface; and obtaining a reflection spectrum of the base light in the target area of the projection surface as the base light spectrum.
[0009] In the above method for obtaining parameters of ambient light and projection surface, the RGB color of the base light of the known spectrum is (0, 0, 0).
[0010] In the above method for obtaining parameters of ambient light and projection surface, the RGB color of the base light of the known spectrum is adjustable.
[0011] In the above method for acquiring parameters of ambient light and projection surface, the known spectrum of the base light is a spectrum information parameter obtained by color space mapping.
[0012] In the above method for acquiring parameters of ambient light and projection surface, the reflection spectrum of the setting image includes the reflection spectrum of the base light, the reflection spectrum of the plurality of color blocks in the setting image and the reflection spectrum of the ambient light.
[0013] In the above-mentioned method for obtaining parameters of ambient light and projection surface, obtaining the ambient light spectrum and the reflection spectrum of the projection surface based on the reflection spectrum corresponding to the multiple color blocks of the setting image includes: based on the reflection spectrum of the setting image, defining the reflection spectrum of each area corresponding to each color block in the multiple color blocks.
[0014] In the above method for acquiring parameters of ambient light and projection surface, the reflection spectrum of each color block is acquired by a spectral imaging sensor, or is a preset spectrum.
[0015] In the above method for obtaining parameters of ambient light and projection surface, obtaining the ambient light spectrum and the reflection spectrum of the projection surface based on the reflection spectra corresponding to the multiple color blocks of the setting image includes:
[0016] Determine the reflectance spectrum relationship of the area corresponding to each color block as follows:
[0017] Reflection spectrum n = (base light spectrum + ambient light spectrum) * reflectivity;
[0018] Reflectance spectrum 1 = (region 1 spectrum + ambient light spectrum) * reflectivity;
[0019] Reflectance spectrum 2 = (region 2 spectrum + ambient light spectrum) * reflectivity;
[0020] …
[0021] Reflection spectrum n-1 = (region n-1 spectrum + ambient light spectrum) * reflectivity.
[0022] In the above method for obtaining parameters of ambient light and projection surface, obtaining the ambient light spectrum and the reflection spectrum of the projection surface based on the reflection spectrum corresponding to the multiple color blocks of the setting image further includes:
[0023] Define the loss function Loss as:
[0024] Loss = ||reflection spectrum 1 - (region 2 spectrum + ambient light spectrum) * reflectivity || + ||reflection spectrum 2 - (region 2 spectrum + ambient light spectrum) * reflectivity || + ... + ||reflection spectrum n-1 - (region n-1 spectrum + ambient light spectrum) * reflectivity || + ||reflection spectrum n - (base light spectrum + ambient light spectrum) * reflectivity ||, where ||·|| represents the norm of a vector or matrix;
[0025] The loss function Loss is minimized by using the least square method to calculate the best estimated values of the ambient light spectrum and the reflectivity of the target area of the projection surface.
[0026] In the above method for obtaining parameters of ambient light and projection surface, the ambient light spectrum and the reflectivity of the target area of the projection surface are solved from the relationship by a neural network modeling method.
[0027] According to another aspect of the present application, a spectral imaging device is provided, which is applied to the method for obtaining parameters of ambient light and projection surface as described above, and is used to obtain a base light spectrum of a set image and / or a reflection spectrum of the set image.
[0028] In the spectral imaging device as described above, obtaining the base light spectrum of a set image and / or the reflectance spectrum of the set image includes:
[0029] Determine the relationship:
[0030] C k =∫S(λ)ρ(λ)τ(λ)f k (λ)α(λ)dλ;
[0031] Where: C k represents the pixel value of the kth channel of the spectral imaging device; S(λ) represents the ambient light spectrum; ρ(λ) represents the reflectivity of the target area of the projection surface; τ(λ) represents the transmittance of the lens of the spectral imaging device; f k (λ) represents the transmittance of the kth structure of the spectral imaging sensor of the spectral imaging device; α(λ) represents the spectral sensitivity of the spectral imaging sensor of the spectral imaging device itself;
[0032] The above parameters satisfy the relationship:
[0033] S(λ)ρ(λ)=X;
[0034] τ(λ)f k (λ)α(λ)=T;
[0035] C k =Y;
[0036] Among them, X kis the spectral image value of the input spectrum at the kth structure, and Y is the output response corresponding to the structure, and we get:
[0037] Y=TX.
[0038] In the spectral imaging device as described above, the known multiple color blocks of the set image are calibrated in advance to calculate the input light spectrum X through T of the known spectral imaging device and obtain the output response Y, where the input light is the base light and / or the reflected light.
[0039] According to another aspect of the present application, a terminal device is provided, comprising the spectral imaging device as described above.
[0040] The method for obtaining parameters of ambient light and projection surface and the spectral imaging device provided in the embodiments of the present application can calculate the parameters of ambient light and projection surface based on the reflection spectrum of a set image including multiple color blocks in different areas of the projection surface, thereby conveniently and accurately obtaining the parameters of ambient light and projection surface. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] By reading the detailed description of the preferred specific embodiments below, various other advantages and benefits of the present application will become clear to those of ordinary skill in the art. The drawings in the specification are only used for the purpose of illustrating the preferred embodiments and are not considered to be limitations of the present application. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work. Moreover, the same reference numerals are used to represent the same components throughout the drawings.
[0042] Figure 1 A schematic flowchart of a method for obtaining parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated.
[0043] Figure 2 It is a schematic diagram of different brightness color blocks of a setting image in a method for obtaining parameters of ambient light and a projection surface according to an embodiment of the present application.
[0044] Figure 3 A schematic diagram of a first example of setting a reflectance spectrum of an image in a method for acquiring parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated.
[0045] Figure 4 A schematic diagram of a second example of setting a reflectance spectrum of an image in a method for acquiring parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated.
[0046] Figure 5A schematic diagram of various regions defined in a reflection spectrum of a setting image in a method for acquiring parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated.
[0047] Figure 6 Schematic diagram showing the band responses of cone cells to spectral images.
[0048] Figure 7 The figure shows a schematic configuration diagram of a spectral imaging device according to an embodiment of the present application.
[0049] Figure 8 The figure illustrates a schematic diagram of the principle of spectral imaging of a spectral imaging device according to an embodiment of the present application.
[0050] Fig. 9 The diagram shows the spectral information of the image (210, 250, 230) emitted by the projector.
[0051] Fig.10 The diagram shows a schematic diagram of adjusting the projection display effect by using the spectrum information of the ambient light and the reflectivity information of the target area of the projection surface.
[0052] Fig.11 A schematic diagram of a display applied to a spectral imaging device according to an embodiment of the present application is illustrated.
[0053] Fig.12 The figure shows a schematic diagram of a spectral imaging device according to an embodiment of the present application being applied in the field of security.
[0054] Fig.13 The diagram shows a schematic diagram of a spectral imaging device according to an embodiment of the present application being applied in the military field.
[0055] Fig.14 The figure shows a schematic diagram of a spectral imaging device according to an embodiment of the present application applied in the field of medical health.
[0056] Fig.15 The figure shows a schematic diagram of a spectral imaging device according to an embodiment of the present application being applied in the field of automotive electronics. DETAILED DESCRIPTION
[0057] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0058] Schematic method
[0059] Figure 1 A schematic flowchart of a method for obtaining parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated.
[0060] like Figure 1 As shown, the method for obtaining parameters of ambient light and projection surface according to an embodiment of the present application includes the following steps.
[0061] S110, determining a base light spectrum of a set image, wherein the set image includes a plurality of color blocks of arbitrary colors with different brightness.
[0062] Here, the set image may be a single image including a plurality of color blocks of arbitrary colors with different brightness, or a plurality of images including a plurality of color blocks of arbitrary colors with different brightness. Furthermore, each of the plurality of images may include one or more color blocks of arbitrary brightness and arbitrary color. For example, the plurality of color blocks may have the same brightness or different brightness, or the plurality of color blocks may have the same color or different colors, and the plurality of color blocks may also have the same brightness and different colors, or the same color and different brightness, and so on. That is, in the embodiment of the present application, it is only required that the set image include a plurality of color blocks of arbitrary colors with different brightness, without limiting the number of images of the set image, and without limiting the brightness and color of the color blocks contained in each image.
[0063] The base light spectrum of the setting image can be obtained, for example, when the projection device is not turned on, and can be presented as, for example Figure 2 The background shown is black, here, Figure 2 It is a schematic diagram of different brightness color blocks of a set image in the method for obtaining parameters of ambient light and projection surface according to an embodiment of the present application. It is worth noting that, in fact, when the projection device projects the set image onto the target area of the projection surface (e.g., a wall or cloth surface, etc.), due to the base brightness of the projection device and the presence of ambient light (e.g., in a non-total black environment), the black area will also have a certain brightness, and at the same time, each square area corresponding to multiple color blocks of different brightness in the set image will also have a color offset.
[0064] Therefore, in the embodiment of the present application, it is necessary to first obtain the base light spectrum of the set image in advance, which can be obtained by pre-calibration. For example, the specific calibration is: use the light with the RGB color of the known spectrum (0, 0, 0) to project to the target area of the projection surface, and obtain the reflection spectrum at this time, that is, the reflectivity of the base light can be obtained. Of course, other known spectrum lights can also be preset for calculation.
[0065] Here, those skilled in the art can understand that in the embodiment of the present application, it is only necessary that the spectrum of the base light is known, and the color coordinate value of the color space of the base light is not limited, for example, it can be any color coordinate value of the RGB color space or other color spaces. In addition, in the embodiment of the present application, the color coordinates of the known spectrum of the base light, for example, the RGB coordinates are adjustable, and the known spectrum of the base light can also be a spectrum information parameter mapped from the color space by other means.
[0066] Therefore, in the method for obtaining parameters of ambient light and a projection surface according to an embodiment of the present application, determining the base light color of at least one set image includes: projecting base light of a known spectrum onto a target area of the projection surface; and obtaining a reflection spectrum of the base light in the target area of the projection surface as the base light spectrum.
[0067] S120: Project the set image onto a target area of a projection surface, and obtain a reflection spectrum of the set image reflected from the target area of the projection surface.
[0068] Specifically, when the set image is projected onto the target area of the projection surface, since the set image includes multiple color blocks of arbitrary colors with different brightness, the reflection spectrum of the set image includes the reflection spectrum of the base light and the reflection spectrum of each color block, and the ambient light also has a reflection spectrum for the target area of the projection surface.
[0069] Therefore, in the method for obtaining parameters of ambient light and projection surface according to an embodiment of the present application, the reflection spectrum of the setting image includes the reflection spectrum of the base light, the reflection spectrum of multiple color blocks in the setting image and the reflection spectrum of the ambient light.
[0070] Specifically, in the embodiment of the present application, the measured reflectance spectrum image of the setting image can be as follows: Figure 3 As shown, for example, the test area can be square, circular, or any other shape, and can be a certain interval or a continuous area; and the projected colors can be white, or a combination of various colors, such as four rows of color blocks, with the intensity decreasing from left to right, and the various colors can be white, RGB, or other layouts. Here, Figure 3 A schematic diagram of a first example of setting a reflectance spectrum of an image in a method for acquiring parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated, and Figure 4 A schematic diagram of a second example of setting a reflectance spectrum of an image in a method for acquiring parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated.
[0071] S130: Acquire an ambient light spectrum and a reflection spectrum of a projection surface based on reflection spectra corresponding to the plurality of color blocks of the setting image.
[0072] First, for example Figure 3 and Figure 4 The reflectance spectrum of the setting image shown in FIG. 1 defines the reflectance spectrum of each area corresponding to each color block in the plurality of color blocks, such as Figure 5 As shown, 1, 2, ..., n-1 are main test color blocks, and n is a base color block. The reflectance spectrum of each color block can be obtained by a spectral imaging sensor, or the spectrum of the test light of each color block can also use a preset spectrum. Here, Figure 5 A schematic diagram of various regions defined in a reflection spectrum of a setting image in a method for acquiring parameters of ambient light and a projection surface according to an embodiment of the present application is illustrated.
[0073] Then, the reflectance spectra of each area corresponding to each color block are obtained, wherein the reflectance spectra of the area corresponding to each color block have the following relationship:
[0074] Reflectance spectrum n = (base light spectrum + ambient light spectrum) * reflectivity
[0075] Reflection spectrum 1 = (region 1 spectrum + ambient light spectrum) * reflectivity
[0076] Reflectance spectrum 2 = (region 2 spectrum + ambient light spectrum) * reflectivity
[0077] …
[0078] Reflectance spectrum n-1 = (spectrum of region n-1 + ambient light spectrum) * reflectivity
[0079] As described above, in the embodiment of the present application, since the ambient light spectrum and reflectivity are parameters that change every time, they are also the main objects to be solved according to the solution of the embodiment of the present application.
[0080] It can be seen that the above problem is a problem of solving a noisy linear process, which can be solved by various solving algorithms. The method here is not limited to a certain method of solving the matrix equation to obtain the spectrum to be measured, and generally adopts the least squares method or neural network modeling.
[0081] For example, the loss function Loss can be defined as:
[0082] Loss = || reflection spectrum 1 - (region 2 spectrum + ambient light spectrum) * reflectivity || + || reflection spectrum 2 - (region 2 spectrum + ambient light spectrum) * reflectivity || + ... + || reflection spectrum n-1 - (region n-1 spectrum + ambient light spectrum) * reflectivity || + || reflection spectrum n - (base light spectrum + ambient light spectrum) * reflectivity ||, where ||·|| represents the norm of a vector or matrix.
[0083] Then, by minimizing the Loss using the least square method, the best estimated values of the ambient light spectrum and the reflectivity of the target area of the projection surface can be calculated.
[0084] Alternatively, it can be solved by neural network modeling. Since neural network modeling can realize spectral calculation on NPU, it has the characteristics of low power consumption while ensuring high spectral accuracy, which is very suitable for mobile terminals such as mobile phones. In addition, it can also be used for the calculation of ambient light spectrum and reflectivity of target area of projection surface in spectral imaging in the fields of home appliances, automobiles, etc.
[0085] Here, considering that the environment used by the user is not the same each time, the method for obtaining the parameters of the ambient light and the projection surface according to the embodiment of the present application can be performed each time the projection device is turned on, so as to calculate the ambient light spectrum and the reflectivity of the target area of the projection surface. Of course, in this process, if the measured spectral information of the ambient light is consistent with the last record, and the obtained spectral information of the base light is also consistent, it can be further confirmed that the current environment and target area have not changed, and the calculation process can be reduced.
[0086] Of course, in order to ensure accuracy, further calculations may be performed according to the method for obtaining parameters of ambient light and projection surface according to the embodiment of the present application to obtain the current ambient light spectrum and the reflectance spectrum of the target area of the projection surface.
[0087] Therefore, the method for obtaining parameters of ambient light and projection surface according to the embodiment of the present application takes into account that the projected image of the projection device is affected by the ambient light and the target area of the projection, including but not limited to the materials of the wall cloth, etc., that is, the reflectivity of the projected target area of different materials is different, so the projection light actually seen by the human eye also has various color cast problems after reflection. The parameters of the ambient light and the projection surface are calculated based on the reflection spectrum of the set image including multiple color blocks in different areas of the projection surface, so as to conveniently and accurately obtain the parameters of the ambient light and the projection surface.
[0088] Schematic spectral imaging equipment
[0089] As described above, in the method for obtaining the parameters of the ambient light and the projection surface according to the embodiment of the present application, a spectral imaging sensor is needed to obtain the spectrum of the base light and the reflectance spectrum information of the set image. Here, since the set image includes multiple color blocks of arbitrary colors with different brightness, the spectral imaging sensor needs to obtain spectral information of different bands to make the accuracy more accurate. In the case of using RGB, RGBIR, RGBW cameras, etc., even if the number of channels is small, the relatively rough spectral information of the ambient light and the reflectivity information of the target area of the projection surface can be obtained by the method for obtaining the parameters of the ambient light and the projection surface according to the embodiment of the present application.
[0090] However, since the spectral range of light that RGB, RGBIR, and RGBW cameras can perceive is limited, they cannot provide full-band responses, such as Figure 6 Here, Figure 6 Schematic diagram showing the band responses of cone cells to spectral images.
[0091] That is, since the projector device needs to calculate the ambient light spectrum and reflectivity in order to achieve a better projection display effect, although the traditional RGB camera, RGBIR, and RGBW camera can also calculate the ambient light spectrum and reflectivity according to the method described above, if the RGB camera is used, then the calculation of the ambient light spectrum and reflectivity will be limited to the QE interval (perception interval) of the RGB camera, and the ambient light outside the RGB perception interval will not be perceived. And these ambient lights will also affect human perception. Therefore, the lack of a complete visible light spectrum makes the solution according to the embodiment of the present application less effective. For example, Figure 6 As shown in the figure, if the S cell responds most strongly and the M and L cells respond less strongly when light of about 450nm is received, the human brain will "mark" this type of light as blue after post-processing the perceived information. It can be seen that cone cells can only respond to light within about 400-700nm, so the human eye can only see the color of light in this band, which is often called visible light, but the perception range provided by the RGB camera is much smaller than the perception range of the human eye. In addition, if the QE curve of the three perception ranges of the RGB camera is limited, the accuracy of the above solution process will also be reduced.
[0092] Therefore, in the embodiment of the present application, a spectral imaging device is used to implement the method for acquiring parameters of ambient light and a projection surface according to the embodiment of the present application.
[0093] here, Figure 7 FIG. 1 is a schematic configuration diagram of a spectral imaging device according to an embodiment of the present application. Figure 7As shown, in the spectral imaging device according to the embodiment of the present application, the optical system is optional, which may be an optical system such as a lens assembly, a homogenizing assembly, etc. The filter structure is a broadband filter structure in the frequency domain or wavelength domain. The pass spectra of different wavelengths of the filter structures at different locations are not exactly the same. The filter structure may be a structure or material with filter characteristics such as a metasurface, a photonic crystal, a nanocolumn, a multilayer film, a dye, a quantum dot, a MEMS (micro-electromechanical system), an FP etalon (FP standard tool), a cavity layer (resonant cavity layer), a waveguide layer (waveguide layer), a diffraction element, etc. For example, in an embodiment of the present application, the filter structure may be the light modulation layer in the Chinese patent CN201921223201.2, and the image sensor (i.e., the light detector array) may be a CMOS image sensor (CIS), a CCD, an array light detector, etc. In addition, the optional data processing unit may be a processing unit such as an MCU, a CPU, a GPU, an FPGA, an NPU, an ASIC, etc., which may export the data generated by the image sensor to the outside for processing.
[0094] Specifically, after the image sensor measures the light intensity information, it is transmitted to the data processing unit for recovery calculation. The specific process of this process is described as follows:
[0095] The intensity signal of the incident light at different wavelengths λ is recorded as x(λ), and the transmission spectrum curve of the filter structure is recorded as T(λ). There are m groups of structural units on the filter (filter structure), and the transmission spectrum of each group of structural units is different. Generally speaking, the filter structure can be recorded as Ti(λ) (i=1,2,3,…,m). There are corresponding physical pixels under each group of structural units to detect the light intensity bi modulated by the filter structure. In the embodiment of the present application, for example, one physical pixel, that is, one physical pixel corresponds to a group of structural units, but it is not limited to this, and multiple physical pixels can also correspond to a group of structural units. Therefore, in the spectral imaging device according to the embodiment of the present application, multiple groups of structural units constitute a "spectral pixel". Further, at least one spectral pixel can be used to restore the image. It should be noted that the number of effective transmission spectra (the transmission spectrum used for spectral restoration is called the effective transmission spectrum) Ti(λ) of the filter structure may be inconsistent with the number of structural units. The transmission spectrum of the filter structure is artificially set, tested, or calculated according to certain rules based on the needs of identification or restoration (for example, the transmission spectrum of each structural unit tested above is the effective transmission spectrum). Therefore, the number of effective transmission spectra of the filter structure may be less than the number of structural units, or even more than the number of structural units; for example, a certain transmission spectrum curve is not necessarily determined by a group of structural units.
[0096] The relationship between the spectral distribution of the incident light and the measured value of the image sensor can be expressed as follows:
[0097] Yi=∫x(λ)*Ti(λ)*T(λ)dλ
[0098] Discretize it again and get:
[0099] Yi=Σ(x(λ)*Ti(λ)*T(λ))
[0100] Where T(λ) is the response of the image sensor, expressed as:
[0101] Yi(λ)=Xi(λ)*T(λ),
[0102] The above formula can be expanded into matrix form:
[0103] Among them, Ti (i = 1, 2, 3, ..., m) is the response of the image sensor after the light to be measured passes through the filter structure, which corresponds to the light intensity measurement value of the image sensor corresponding to m structural units. When a physical pixel corresponds to a structural unit, it can be understood as the light intensity measurement value corresponding to m "physical pixels", which is a vector of length m. X is the system's response to light of different wavelengths, which is determined by two factors: the transmittance of the filter structure and the quantum efficiency of the image sensor. X is a matrix, and each row vector corresponds to the response of a group of structural units to incident light of different wavelengths. Here, the incident light is discretely and uniformly sampled, with a total of n sampling points. The number of columns of X is the same as the number of sampling points of the incident light. Here, X(λ) is the light intensity of the incident light at different wavelengths λ, that is, the incident light spectrum to be measured.
[0104] In some examples, the difference from the above is that the filter structure can be directly formed on the upper surface of the image sensor, such as quantum dots, nanowires, etc., which directly form a filter structure or material (nanowires, quantum dots, etc.) in the photosensitive area of the sensor. Taking the filter structure as an example, at this time, it can be understood that when the raw material of the image sensor is processed to form the image sensor, the filter structure is formed on the upper surface of the raw material. The transmission spectrum and the response of the image sensor are integrated, that is, it can be understood that the response of the detector and the transmission spectrum are the same curve. At this time, the relationship between the spectral distribution of the incident light and the light intensity measurement value of the image sensor can be expressed by the following formula:
[0105] Yi=Σ(x(λ)*Ti(λ))
[0106] Furthermore, at least one filter structure for modulating incident light is provided on the image sensor with the filter structure. It can be understood that the image sensor (i.e., the light detector array) in the first embodiment, which may be a CMOS image sensor (CIS), CCD, array light detector, etc., is replaced with the image sensor with the filter structure integrated in the second embodiment.
[0107] At this time, the relationship between the spectral distribution of the incident light and the light intensity measurement value of the image sensor can be expressed by the following formula:
[0108] Yi=∫x(λ)*Ti(λ)*Ri(λ)dλ
[0109] Discretize it again and get:
[0110] Yi=Σ(x(λ)*Ti(λ)*Ri(λ))
[0111] That is, in this embodiment, Yi(λ)=Ti(λ)*Xi(λ)
[0112] Since each spectral imaging device needs to be provided with an optical system in addition to the spectral chip, including various lenses, etc., further, specifically, Figure 8 As shown in the figure, the principle of obtaining spectral imaging is as follows: after the target is acquired by the spectral imaging sensor, it is converted into an electrical signal for processing through the camera lens, filter and spectral chip, and finally the spectral information of the corresponding object is obtained. Here, Figure 8 The figure illustrates a schematic diagram of the principle of spectral imaging of a spectral imaging device according to an embodiment of the present application.
[0113] Specifically, if Figure 8 When the projector emits (210, 250, 230), the spectral information of the image obtained is as follows: Fig. 9 As shown, the measured projection shows R = 0.66992968, G = 1.0, B = 0.70530216. Here, Fig. 9 The diagram shows the spectral information of the image (210, 250, 230) emitted by the projector.
[0114] Therefore, the display effect of the projection can be adjusted by continuously obtaining the spectrum information of the ambient light and the reflectivity information of the target area of the projection surface, such as Fig.10 Here, Fig.10 The projection display effect after adjustment is shown by the spectrum information of the ambient light and the reflectivity information of the target area of the projection surface, wherein the display effect can also include the optimization of information such as chromaticity, brightness, and color temperature. Here, Fig.10 The diagram shows a schematic diagram of adjusting the projection display effect by using the spectrum information of the ambient light and the reflectivity information of the target area of the projection surface.
[0115] When using a spectral imaging device to calculate the ambient light spectrum and reflectivity in the method for obtaining the parameters of the ambient light and the projection surface as described above, the following formula is used:
[0116] C k =∫S(λ)ρ(λ)τ(λ)fk (λ)α(λ)dλ
[0117] Where: C k represents the pixel value of the kth channel; S(λ) represents the spectrum of ambient light; ρ(λ) represents the reflectivity of the target area of the projection surface; τ(λ) represents the transmittance of the lens; f k (λ) represents the transmittance of the kth structure; α(λ) represents the spectral sensitivity of the spectral imaging sensor of the spectral imaging device itself.
[0118] Corresponding to the above spectrum recovery calculation formula:
[0119] S(λ)ρ(λ)=X;
[0120] τ(λ)f k (λ)α(λ)=T;
[0121] C k =Y
[0122] Among them, X k is the spectral image value of the input spectrum at the kth structure. Y is the output response corresponding to the structure, and the integration yields:
[0123] Y=TX
[0124] In the embodiment of the present application, the known multiple color blocks of the set image can be calibrated in advance, and the input light X, that is, the reflection spectrum of each corresponding color block is calculated through the known spectral imaging device T and the output response Y.
[0125] Here, when spectral imaging equipment is used, the spectral reconstruction accuracy of the traditional RGB camera mentioned above is at least 7-8 times higher. The method of solving the matrix equation to obtain the spectrum to be measured generally adopts the least square method or neural network modeling. The neural network modeling method can realize spectral calculation on the NPU, which ensures high spectral accuracy while having the characteristics of low power consumption, so it can be applied to various terminal devices.
[0126] For example, Fig.11 A schematic diagram of a display applied to a spectral imaging device according to an embodiment of the present application is illustrated.
[0127] In addition, the spectral imaging device according to the embodiment of the present application can be applied to the security field, such as Fig.12 As shown; the spectral imaging device according to the embodiment of the present application can be applied to the military field, such as Fig.13 As shown; the spectral imaging device according to the embodiment of the present application can be applied to the field of medical health, such as Fig.14 As shown; the spectral imaging device according to the embodiment of the present application can be applied to the field of automotive electronics, such as Fig.15 shown.
[0128] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.
[0129] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.
[0130] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.
[0131] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0132] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A method for obtaining parameters of ambient light and projection surface, characterized in that: include: Determine a base light spectrum of a set image, wherein the set image includes a plurality of color blocks of arbitrary colors with different brightness; Projecting the set image onto a target area of a projection surface, and acquiring a reflection spectrum of the set image reflected from the target area of the projection surface; and The ambient light spectrum and the reflection spectrum of the projection surface are acquired based on the reflection spectrum corresponding to the multiple color blocks of the setting image.
2. The method according to claim 1, wherein: The set image is a single image including multiple color blocks of arbitrary colors with different brightness, or multiple images each including multiple color blocks of arbitrary colors with different brightness, each of the multiple images includes one or more color blocks, and the one or more color blocks are of the same or different brightness and the same or different colors.
3. The method according to claim 1, wherein: Determining the base light color of at least one setting image includes: Projecting a base light of known spectrum onto a target area of the projection surface; and, A reflection spectrum of the base light in the target area of the projection surface is obtained as the base light spectrum.
4. The method according to claim 3, wherein: The RGB color of the base light of the known spectrum is (0, 0, 0).
5. The method according to claim 3, wherein: The RGB color of the base light of the known spectrum is adjustable.
6. The method according to claim 3, wherein: The known spectrum of the base light is a spectrum information parameter obtained through color space mapping.
7. The method according to claim 1, wherein: The reflection spectrum of the setting image includes the reflection spectrum of the base light, the reflection spectrum of a plurality of color blocks in the setting image, and the reflection spectrum of the ambient light.
8. The method according to claim 1, wherein: Acquiring the ambient light spectrum and the reflection spectrum of the projection surface based on the reflection spectrum corresponding to the multiple color blocks of the setting image includes: Based on the reflectance spectrum of the setting image, a reflectance spectrum of each area corresponding to each of the plurality of color blocks is defined.
9. The method according to claim 7, wherein: The reflectance spectrum of each color block is acquired through a spectral imaging sensor, or is a preset spectrum.
10. The method according to claim 7, wherein: Acquiring the ambient light spectrum and the reflection spectrum of the projection surface based on the reflection spectrum corresponding to the multiple color blocks of the setting image includes: Determine the reflectance spectrum relationship of the area corresponding to each color block as follows: Reflection spectrum n = (base light spectrum + ambient light spectrum) * reflectivity; Reflectance spectrum 1 = (region 1 spectrum + ambient light spectrum) * reflectivity; Reflectance spectrum 2 = (region 2 spectrum + ambient light spectrum) * reflectivity; …… Reflection spectrum n-1 = (region n-1 spectrum + ambient light spectrum) * reflectivity.
11. The method according to claim 10, wherein: Acquiring the ambient light spectrum and the reflection spectrum of the projection surface based on the reflection spectrum corresponding to the plurality of color blocks of the setting image further includes: Define the loss function Loss as: Loss = ||reflection spectrum 1 - (region 2 spectrum + ambient light spectrum) * reflectivity || + ||reflection spectrum 2 - (region 2 spectrum + ambient light spectrum) * reflectivity || + ... + ||reflection spectrum n-1 - (region n-1 spectrum + ambient light spectrum) * reflectivity || + ||reflection spectrum n - (base light spectrum + ambient light spectrum) * reflectivity ||, where ||·|| represents the norm of a vector or matrix; The loss function Loss is minimized by using the least square method to calculate the best estimated values of the ambient light spectrum and the reflectivity of the target area of the projection surface.
12. The method according to claim 11, wherein: The ambient light spectrum and the reflectivity of the target area of the projection surface are solved from the relationship by a neural network modeling method.
13. A spectral imaging device, applied to the method for obtaining parameters of ambient light and projection surface as claimed in any one of claims 1 to 12, for obtaining a base light spectrum of a set image and / or a reflection spectrum of the set image.
14. The spectral imaging device according to claim 13, wherein: Acquiring a base light spectrum of a setting image and / or a reflectance spectrum of the setting image includes: Determine the relationship: C k =∫S(λ)ρ(λ)τ(λ)f k (λ)a(λ)dλ; Where: C k represents the pixel value of the kth channel of the spectral imaging device; S(λ) represents the ambient light spectrum; ρ(λ) represents the reflectivity of the target area of the projection surface; τ(λ) represents the transmittance of the lens of the spectral imaging device; f k (λ) represents the transmittance of the kth structure of the spectral imaging sensor of the spectral imaging device; α(λ) represents the spectral sensitivity of the spectral imaging sensor of the spectral imaging device itself; The above parameters satisfy the relationship: S(λ)ρ(λ)=X; τ(λ)f k (λ)a(λ)=T; C k =Y; Among them, X k is the spectral image value of the input spectrum at the kth structure, and Y is the output response corresponding to the structure, and we get: Y=TX.
15. The spectral imaging device according to claim 14, wherein: The known multiple color blocks of the set image are calibrated in advance, so as to calculate the input light spectrum X through T of a known spectral imaging device and obtain the output response Y, where the input light is the base light and / or the reflected light.
16. A terminal device, comprising the spectral imaging device according to any one of claims 13 to 15.