Spectral imaging restoration method and apparatus

Abstract

The application relates to a spectral imaging recovery method. The spectral imaging recovery method comprises: receiving incident light by a light modulation layer of a spectral imaging chip, the light modulation layer comprising a plurality of modulation units arranged periodically, and each modulation unit comprising a plurality of structure units; acquiring a response signal of the incident light modulated by the light modulation layer of the spectral imaging chip by an image sensor of the spectral imaging chip, each structure unit corresponding to one or more physical pixels of the image sensor; and recovering spectral information of the incident light based on spectral pixels corresponding to each modulation unit, wherein the modulation units corresponding to each spectral pixel are spaced apart from each other by no less than one structure unit. In this way, the calculation efficiency of spectral recovery is improved by performing spectral recovery calculation in units of modulation units containing structure unit arrays and spaced apart from each other.

Description

[0001] This application is a divisional application of Chinese invention application No. 202211022771.1, filed on August 25, 2022, entitled "Spectral Imaging Recovery Method and Apparatus". Technical Field

[0002] This application relates to the field of spectral technology, and more specifically, to a spectral imaging recovery method and apparatus. Background Technology

[0003] Spectral imaging technology is a technique that organically combines spectral detection and imaging. It can image an object under different spectra, simultaneously obtaining the geometric shape information and spectral characteristics of the object being detected. Spectral imaging technology has become an important tool for Earth observation and deep space exploration, and is widely used in fields such as agricultural and forestry production, mineral resource exploration, cultural relic detection, marine remote sensing, environmental monitoring, disaster prevention and mitigation, and military reconnaissance.

[0004] Spectrometers employing spectral imaging technology have become one of the most commonly used measurement tools in scientific research and industry. Currently, traditional imaging chips and imaging devices, such as cameras, contain image sensors to acquire image information of the object's RGB colors. Spectral imaging technology is mainly based on a spectrometer combined with a mechanical scanning structure. This approach requires precise control of the mechanical scanning and a trade-off between scanning step size, which leads to increased costs and reduced temporal resolution.

[0005] In addition, spectrometers that utilize filters and photodetector arrays have the advantage of a natural two-dimensional photosensitive structure, which allows for direct spectral imaging through the arraying of spectrometers. This approach has irreplaceable advantages in terms of cost, time resolution, and integration.

[0006] Therefore, it is desirable to provide a spectral recovery solution suitable for spectrometers. Summary of the Invention

[0007] This application provides a spectral imaging restoration method that improves the computational efficiency of spectral restoration by performing spectral restoration calculations on a unit basis, consisting of modulation units that are spaced apart from each other and contain arrays of structural units.

[0008] According to one aspect of this application, a spectral imaging recovery method is provided, comprising: receiving incident light by an optical modulation layer of a spectral imaging chip, the optical modulation layer comprising a plurality of modulation units arranged periodically, and each modulation unit comprising a plurality of structural units; acquiring a response signal of an image sensor of the spectral imaging chip to the incident light modulated by the optical modulation layer, each structural unit corresponding to one or more physical pixels of the image sensor; and recovering spectral information of the incident light based on the spectral pixels corresponding to each modulation unit, wherein the modulation units corresponding to each spectral pixel are spaced apart from each other by at least one structural unit.

[0009] In the above-described spectral imaging restoration method, the number of structural units in the modulation unit interval is equal to the square root of the number of structural units included in the modulation unit.

[0010] In the above-described spectral imaging restoration method, the number of physical pixels between the modulation units is not less than the number of preset modulation structure units.

[0011] In the above-described spectral imaging restoration method, each spectral pixel corresponds to a first number of structural units, each modulation unit includes a second number of structural units, and the second number is not less than the first number.

[0012] In the above-described spectral imaging restoration method, the optical modulation layer includes a modulation unit with a filter structure and a non-modulation unit without a filter structure.

[0013] In the above-described spectral imaging restoration method, restoring the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit includes: obtaining the light intensity information corresponding to each structural unit of the spectral pixel to obtain a response vector; obtaining the modulation matrix and noise vector of the optical modulation layer; and determining the spectral vector of the incident light based on the response vector, the modulation matrix, and the noise vector to obtain the spectral information of the incident light.

[0014] In the above-described spectral imaging restoration method, the step of restoring the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit further includes: obtaining the number and position coordinates of physical pixels and structural units in the spectral pixel unit; and, when it is determined that the number of physical pixels is greater than the number of structures in the structural unit, determining the spectral vector of the incident light based on the response vector, the modulation matrix, and the noise vector to obtain the spectral information of the incident light.

[0015] The above-described spectral imaging recovery method further includes: acquiring the modulated transmission spectrum of each structure in each structural unit; determining the correlation of the transmission spectra based on the acquired transmission spectra; determining the position coordinates of the uncorrelated transmission spectra under the structural unit; and determining the hole convolution rate based on the position coordinates.

[0016] In the above spectral imaging restoration method, the spectral vector of the incident light is X = [λ1, λ2, ..., λ]. N ] T The response vector is Y = [Y1, Y2, ..., Y... M ] T The modulation matrix and the noise vector are respectively D:

[0017]

[0018] And W = [W1, W2, ..., W n ] T Then Y = DX + W.

[0019] In the above-described spectral imaging restoration method, after acquiring the response signal of the incident light modulated by the light modulation layer by the image sensor of the spectral imaging chip, the method further includes: correcting the response vector Y to obtain a corrected response vector Y', denoted as:

[0020] Y'={exp[(R1Y)*(R2Y) T ]}*(P*H)

[0021] Where R1 and R2 are weight matrices, P is a weight vector, and H is the distance matrix of the response vector Y, where each position corresponds to the distance value between two corresponding positions of the response vector Y.

[0022] In the above-described spectral imaging restoration method, the parameters of the weight matrices R1 and R2 and the parameters of the weight vector P are obtained through calibration or training.

[0023] According to another aspect of this application, a spectral imaging recovery method is provided, comprising: receiving incident light by an optical modulation layer of a spectral imaging chip, the optical modulation layer including a plurality of modulation units arranged periodically, and each modulation unit including a plurality of structural units; acquiring a response signal of an image sensor of the spectral imaging chip to the incident light modulated by the optical modulation layer, each structural unit corresponding to one or more physical pixels of the image sensor; and recovering spectral information of the incident light based on the spectral pixels corresponding to each modulation unit, wherein each spectral pixel corresponds to a first number of structural units, each modulation unit includes a second number of structural units, and the second number is not less than the first number.

[0024] In the above spectral imaging restoration method, the modulation units corresponding to each spectral pixel are spaced apart by no less than one structural unit.

[0025] The spectral imaging restoration method provided in this application improves the computational efficiency of spectral restoration by performing spectral restoration calculations on a unit basis, which consists of modulation units that are spaced apart from each other and contain arrays of structural units. Attached Figure Description

[0026] Various other advantages and benefits of this application will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0027] Figure 1 The figure shows a schematic diagram of the structure of a spectral imaging chip according to an embodiment of this application.

[0028] Figure 2 The illustration shows an example of the correspondence between physical pixels, structural units, and modulation units of a spectral imaging chip according to an embodiment of this application.

[0029] Figure 3 An example of an exemplary configuration of the light modulation layer of a spectral imaging chip according to an embodiment of this application is illustrated.

[0030] Figure 4 The illustration shows a schematic array arrangement of structural units in the spectral pixels of a spectral imaging chip according to an embodiment of the present application.

[0031] Figure 5 The illustration shows a calculation method for existing spectral imaging restoration methods and a calculation method for spectral imaging restoration methods according to embodiments of this application.

[0032] Figure 6 The illustration shows a schematic flowchart of an example of a spectral imaging recovery method according to an embodiment of this application.

[0033] Figure 7 The illustration shows a schematic diagram of an existing hole-free window and an example of a window with holes in the spectral imaging restoration method according to an embodiment of this application.

[0034] Figure 8 The figure shows a schematic diagram of the structural unit of the sensor of the spectral imaging chip in the spectral imaging recovery method according to an embodiment of the present application.

[0035] Figure 9 The illustration shows a flowchart of the process for determining the hole convolution rate in the spectral imaging recovery method according to an embodiment of this application.

[0036] Figure 10 The illustration shows a flowchart of another example of a spectral imaging recovery method according to an embodiment of this application. Detailed Implementation

[0037] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0038] Exemplary methods

[0039] Figure 1 The figure shows a schematic diagram of the structure of a spectral imaging chip according to an embodiment of this application.

[0040] like Figure 1 As shown, the spectral imaging chip 100 according to an embodiment of this application may include: a light modulation layer 110, an image sensing layer 120, and a signal processing circuit layer 130, wherein the light modulation layer 110, the image sensing layer 120, and the signal processing circuit layer 130 are sequentially stacked along the thickness direction. The light modulation layer 110 has at least one modulation unit 1101 distributed along its surface. The image sensing layer 120 has multiple sensing units 1201 distributed along its surface, and each modulation unit 1101 corresponds to at least one sensing unit 1201 along the thickness direction. Each modulation unit 1101 and its corresponding sensing unit 120 constitute a spectral pixel of the spectral imaging chip, and each spectral pixel may correspond to multiple physical pixels of an image sensor. That is, each physical pixel of the image sensing layer may correspond to a modulation unit of the light modulation layer, or multiple pixels may correspond to one modulation unit.

[0041] The signal processing circuit layer 130 is electrically connected to the sensing unit 1201. The signal processing circuit layer 130 is used to execute an information processing method to determine the image information and spectral information of the object to be imaged. In this embodiment, the modulation structure can be the optical modulation layer in Chinese Patent CN201921223201.2, and the filtering characteristics at each position do not need to be known in the technical solution according to the embodiment of this application. In addition, in the technical solution according to the embodiment of this application, there may be no filter structure, so that each point of the image sensor array has a different spectral response. For example, quantum dots, nanowires, etc. can all realize this structure. The image sensing layer 120 can specifically be a CIS wafer, and each sensing unit in the image sensing layer 120 corresponds to a pixel in the CIS wafer, used to detect the light beam passing through the optical modulation layer. For example, but not limited to, the micro-nano structure optical modulation layer can be monolithically integrated on the CIS wafer directly at the wafer level, and the spectral imaging chip can be fabricated in one fabrication using CMOS technology.

[0042] Furthermore, in the spectral imaging chip according to the embodiments of this application, each modulation unit may have multiple structural units, and each structural unit may also correspond to one or more physical pixels. Figure 2 The illustration shows an example of the correspondence between physical pixels, structural units, and modulation units of a spectral imaging chip according to an embodiment of this application. Figure 2 As shown, each 3*3 physical pixel corresponds to one structural unit, each modulation unit has 20*20 structural units, and each modulation unit is periodically set.

[0043] Furthermore, in the spectral imaging chip according to embodiments of this application, the optical modulation layer may have modulation units and non-modulation units. Figure 3 The illustration shows an example of an exemplary configuration of the light modulation layer of a spectral imaging chip according to an embodiment of this application. For example... Figure 3 As shown, to obtain spectral information of a portion of the imaging region, the optical modulation layer is configured to include at least one modulation unit and at least one non-modulation unit, wherein the modulation and non-modulation units can be arranged alternately. Since the spectral information of the pixel corresponding to each modulation unit may not be sufficient to determine the complete image information of the object to be imaged, it is necessary to determine the image information of the object to be imaged based on the light intensity information of the pixel corresponding to each non-modulation unit after illumination by the target beam. Specifically, one approach is to ignore the light intensity information of the pixel corresponding to each modulation unit and only use the light intensity information of the pixels corresponding to all non-modulation units to determine the image information of the object to be imaged; alternatively, one can first determine the light intensity information of the pixel corresponding to each modulation unit using the light intensity information of the pixel corresponding to each non-modulation unit, and then combine this information with the light intensity information of the pixels corresponding to all non-modulation units to jointly determine the image information of the object to be imaged.

[0044] In this embodiment, the spectral imaging chip uses the modulation unit of the optical modulation layer to modulate the incident light to obtain a modulated spectrum; an image sensor receives the modulated spectrum and provides a differential response to it; and a signal processing circuit layer reconstructs the differential response to obtain the original spectrum of the incident light. Its working principle can be understood as follows: the spectral vector of the incident light signal is X = [λ1, λ2, ..., λ...]. N ] T Where N is the number of wavelengths of the spectral information to be recovered, and the response vector of the image sensor is Y = [Y1, Y2, ..., Y...]. M ] T Where M is the number of structural units included in each spectral pixel. The modulation matrix of the optical modulation layer is an M×N matrix, represented as:

[0045]

[0046] And the noise vector W = [W1, W2, ..., W n ] T Then the relationship between the spectral vector X, the response vector Y, the modulation matrix D, and the noise vector W is: Y = DX + W.

[0047] In actual products, it is generally necessary to first calibrate the product to obtain the modulation matrix D, then measure the object to be measured, and use the known modulation matrix D and the response vector Y obtained by the image sensor to calculate the spectral vector X of the object to be measured.

[0048] As mentioned above, in spectral imaging chips, it is typically necessary to distinguish between physical pixels, structural units, and spectral pixels (corresponding to modulation units). That is, a physical pixel refers to the smallest photosensitive unit on the image sensor (photodetector array). A structural unit refers to the smallest unit of the filter structure used in the image sensor for filtering, as spectral imaging chips require such a structure. One structural unit can cover one or more physical pixels.

[0049] Spectral pixels typically correspond to multiple structural units arranged in an array. Figure 4 The illustration shows a schematic array arrangement of structural units in the spectral pixels of a spectral imaging chip according to an embodiment of this application. Figure 4 As shown, in the array arrangement of structural units in spectral pixels, structural units with different filtering properties are usually arranged periodically with the same period in the X and Y directions, for example, as... Figure 4 The diagram shows an arrangement of structural units with 25 different filter structures, each with three cycles in the X and Y directions.

[0050] As shown in the diagram above, each spectral pixel consists of 25 structural units (5x5), each with a different filter structure. Each group of 25 filter structures constitutes one spectral pixel. For example, Figure 4 The three regions enclosed by boxes A, B, and C in the diagram all completely contain all 25 filter structures. This allows for the calculation of the spectral information of the region using the signals generated by these 25 filter structures. In this embodiment, the physical pixel region used to calculate the spectral information of the spectral pixels is referred to as a "computation window." Figure 4 In the diagram, A, B, and C are all calculation windows.

[0051] Furthermore, to maximize the spatial resolution of the spectral image, the computation window typically slides along the X and Y coordinates, performing calculations one structural unit at a time. Based on this window sliding method, the final spatial resolution of the spectral image is (X-2a, Y-2a), where a is the floor function (w / 2), and w is the window side length. This means that O(X*Y) spectral calculations are required. Each spectral calculation requires w... 2 Information about each physical pixel. Assume the computational cost of spectral calculation is O(w). 2 (In reality, depending on the algorithm, the computational requirements may be even higher, for example, O(w) 4 In this embodiment of the application, O(w) is uniformly used. 2 If we discuss this further, then the computational cost of reconstructing the entire spectral image is O(X*Y*w). 2 ).

[0052] However, the applicant of this application discovered that when multiple structural units correspond to one array arrangement, the obtained modulation spectra are basically similar. Therefore, if the above calculation method is used when multiple structural units correspond to one array arrangement, the computational load will be enormous, and the calculation speed will be significantly affected. Therefore, when the product application has stringent requirements for computational load (e.g., limited device computing power, or a need for high-speed, real-time imaging), O(X*Y*w) becomes insufficient. 2 The computational load may be too large. In this case, it is necessary to appropriately reduce the computational load.

[0053] In this embodiment, a window jumping approach can be used, where the calculation window is slid to obtain spectral signals at different locations, and each movement is greater than one structural unit. Figure 5 As shown. Figure 5 The illustrations show the calculation methods of existing spectral imaging restoration methods and the calculation methods of the spectral imaging restoration method according to embodiments of this application. Wherein, Figure 5Figure (a) illustrates how the computation window typically moves in existing spectral imaging restoration methods. Figure 5 (b) shows the movement method when the calculation window jump is 2 in the spectral imaging restoration method according to the embodiment of this application (note that the usual movement method is the movement method when the window jump is 1). Of course, since the range of the structural unit corresponding to the spectral pixel also has a certain matching relationship with the step size, it is not suitable to set too large a step size in order to better reduce the amount of calculation and ensure spatial resolution.

[0054] In addition, when there is a periodic array arrangement of structural units, the calculation step size can be repeatedly applied to make corresponding jumps during the calculation.

[0055] Based on this Figure 6 A schematic flowchart illustrating an example of a spectral imaging restoration method according to an embodiment of this application is shown. Figure 6 As shown, the spectral imaging recovery method according to an embodiment of this application includes: S110, receiving incident light by an optical modulation layer of a spectral imaging chip, the optical modulation layer including a plurality of modulation units arranged periodically, and each modulation unit including a plurality of structural units; S120, acquiring a response signal of the incident light modulated by the optical modulation layer by an image sensor of the spectral imaging chip, each structural unit corresponding to one or more physical pixels of the image sensor; S130, recovering the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit, wherein the modulation units corresponding to each spectral pixel are spaced apart from each other by not less than one structural unit.

[0056] In this embodiment of the application, before obtaining the response signal of the incident light modulated by the light modulation layer by the image sensor of the spectral imaging chip in step S120, the method further includes: obtaining the number and position coordinates of physical pixels and structural units in the spectral pixel unit; and, when it is determined that the number of physical pixels is greater than the number of structures in the structural unit, determining the spectral vector of the incident light based on the response vector, the modulation matrix and the noise vector to obtain the spectral information of the incident light.

[0057] As can be seen, by increasing window skipping, the number of spectral reconstruction calculations required is reduced. It is worth noting that when the number of structural unit intervals s = w, the spectral imaging restoration method according to the embodiments of this application can be expected to achieve better results. This is because each window does not share structural units with other windows, thus minimizing crosstalk between adjacent spectral pixels. In other words, when the number of structural unit intervals s = w, the number of spectral reconstruction calculations required is minimized while utilizing all filter structures in the calculation. In this case, the computational cost of reconstructing the entire spectral image is O((X / w)*(Y / w)*w). 2= O(X*Y). Here, the window jumping method according to the embodiments of this application already has mature applications that utilize GPUs for computation. Therefore, even algorithms that could originally utilize GPUs for computational acceleration can still utilize GPUs for computational acceleration after applying window jumping.

[0058] That is, in the above-mentioned spectral imaging recovery method, the number of structural units in the modulation unit interval is equal to the square root of the number of structural units included in the modulation unit.

[0059] Furthermore, as described above, in the above-described spectral imaging restoration method, the optical modulation layer includes the modulation unit having a filter structure and the non-modulation unit not having a filter structure.

[0060] In the above-described spectral imaging restoration method, restoring the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit includes: obtaining the light intensity information corresponding to each structural unit of the spectral pixel to obtain a response vector; obtaining the modulation matrix and noise vector of the optical modulation layer; and determining the spectral vector of the incident light based on the response vector, the modulation matrix, and the noise vector to obtain the spectral information of the incident light.

[0061] In the above spectral imaging restoration method, the spectral vector of the incident light is X = [λ1, λ2, ..., λ]. N ] T The response vector is Y = [Y1, Y2, ..., Y... M ] T The modulation matrix and the noise vector are respectively D:

[0062]

[0063] And W = [W1, W2, ..., W n ] T Then Y = DX + W.

[0064] Furthermore, if you want to reduce the computational load on this basis, or do not want to reduce the spatial resolution too much, you can reduce the computational load by using window holes.

[0065] Figure 7 The illustration shows a schematic diagram of an existing hole-free window and an example of a hole in the window of the spectral imaging restoration method according to an embodiment of this application. Figure 7 As shown, Figure 7 (a) represents the typical calculation window, with a total of w. 2 A filter structure; Figure 7 (b) is the calculation window for the number of voids d = 1 (note that although...) Figure 7(b) shows nine boxes, but they need to be combined for calculation, so it is considered one window instead of nine (the usual calculation window is the case when d=0). By selectively ignoring some structures, the computational cost can be reduced by decreasing the amount of data to be considered for each spectral reconstruction while keeping the field of view of each window unchanged.

[0066] Furthermore, although window holes may render some filter structures unusable, by appropriately combining the number of structural unit spacings (s) and the number of window holes (d), as many filter structures as possible can participate in the computation. Here, by setting s and d, the computational complexity can be reduced to O(X*Y*w). 2 / (s 2 *(d+1) 2 For example, when s = w and d = 1, the computational cost can be reduced to O(X*Y / 4). Similar to window jumping, window holes can also be conveniently applied to situations using GPU acceleration.

[0067] Therefore, in the above-described spectral imaging restoration method, each spectral pixel corresponds to a first number of structural units, each modulation unit includes a second number of structural units, and the second number is not less than the first number.

[0068] Furthermore, those skilled in the art will understand that, in addition to being combined with the aforementioned interval window settings, window holes can also be used independently in spectral recovery algorithms.

[0069] Furthermore, the window hole is not limited to the shape shown in the figure. Theoretically, any combination of pixels at any position within a square area can form a window with a hole (if all pixels are taken, there will be no hole), thus making it suitable for specific situations.

[0070] Figure 8 The figure shows a schematic diagram of the structural unit of the sensor of the spectral imaging chip in the spectral imaging recovery method according to an embodiment of the present application.

[0071] like Figure 8 As shown, the transmission spectrum information of each structure in structural unit A and structural unit B can be obtained. In structural unit A, the transmission spectrum at coordinate a is uncorrelated with the transmission spectrum at coordinate b in structural unit B, so the hole convolution rate is 3. Similarly, the transmission spectrum information of each structure in structural unit C and structural unit D can be obtained. In structural unit C, the transmission spectrum at coordinate c is uncorrelated with the transmission spectrum at coordinate d in structural unit D, so the hole convolution rate is 4.

[0072] Therefore, as Figure 9As shown, the spectral imaging recovery method according to the embodiments of this application may further include the following steps: S210, acquiring the modulated transmission spectrum of each structural unit; S220, determining the correlation of the transmission spectrum based on the acquired transmission spectrum; S230, determining the position coordinates of the uncorrelated transmission spectrum in the structural unit; and S240, determining the hole convolution rate based on the position coordinates.

[0073] Figure 9 The illustration shows a flowchart of the process for determining the hole convolution rate in the spectral imaging recovery method according to an embodiment of this application.

[0074] Figure 10 A flowchart illustrating another example of the spectral imaging recovery method according to an embodiment of this application is shown. Figure 10 The spectral imaging recovery method of this example includes: S310, receiving incident light by an optical modulation layer of a spectral imaging chip, the optical modulation layer including a plurality of modulation units arranged periodically, and each modulation unit including a plurality of structural units; S320, acquiring a response signal of the incident light modulated by the optical modulation layer by an image sensor of the spectral imaging chip, each structural unit corresponding to one or more physical pixels of the image sensor; S330, recovering the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit, wherein each spectral pixel corresponds to a first number of structural units, each modulation unit includes a second number of structural units, and the second number is not less than the first number.

[0075] As mentioned above, when using window holes alone, the computational cost can be reduced to O(X*Y*w). 2 / (d+1) 2 For example, when d=1, the computational complexity can be reduced to O(X*Y*w). 2 / 4).

[0076] Here, as Figure 10 Other details of the spectral imaging recovery method according to the other example have been described above and will not be repeated here to avoid redundancy.

[0077] Furthermore, while reducing computational load through interval windows or hole windows, the number of samples is reduced compared to existing spectral imaging restoration methods. This makes the detection accuracy more susceptible to external environmental factors, such as increased temperature drift. Therefore, for the response vector Y = [Y1, Y2, ..., Y...] obtained from the image sensor... M ] TThe overall distribution of intensity values ​​at each location is also affected by the greater fluctuations in the individual intensity values ​​at each location. Therefore, it is desirable to provide a correction mechanism to correct the response vector in order to improve the accuracy of the recovered spectral information.

[0078] Specifically, considering that the influence of external environmental factors on the intensity values ​​at various positions of the response vector is holistic, that is, it will have a basically consistent impact on the intensity values ​​at various positions of the response vector, by capturing the trend characteristics between the intensity values ​​at various positions of the response vector, the dependence of spectral recovery on intensity values ​​can be alleviated, thereby enhancing the robustness to the influence of environmental factors.

[0079] First, the intensity values ​​at each position of the response vector are weighted using a weighting matrix. Then, the correlation information between these positions is obtained through an correlation matrix to capture the overall trend characteristics from a numerical perspective. Specifically, a first weighting matrix R1 and a second weighting matrix R2 are obtained, and each is multiplied by the response vector Y to obtain a first weighted vector V1 and a second weighted vector V2. The transposes of V1 and V2 are then multiplied to obtain the trend matrix M. Here, since V1 and V2 are both column vectors, their multiplication results in a matrix. Furthermore, the parameters of the first weighting matrix R1 and the second weighting matrix R2 can be obtained by calibration based on the intensity values ​​of known environmental factors.

[0080] Next, the distance between every two locations in the response vector Y is calculated to obtain the distance matrix H. i,j =d(Y i ,Y j Here, the distance value can be an absolute distance, i.e., Y. i -Y j The absolute value of the distance can also be the mean square distance. Then, the weight vector P is multiplied by the distance matrix H as the query vector, thereby collapsing the distance matrix in terms of dimension to obtain a trend vector L of the same dimension as the response vector Y, representing the spatial relationship of the intensity values ​​at each position of the response vector in the high-dimensional space represented by the vector. Furthermore, the parameters of the weight vector P can also be obtained by calibration based on the intensity values ​​influenced by known environmental factors.

[0081] Finally, the trend matrix M is scaled by an exponential function and multiplied by the trend vector L to obtain the final corrected response vector Y', thereby fusing the expression of trend features based on the self-attention mechanism.

[0082] Therefore, the correction of the response vector can be expressed as:

[0083] Y'={exp[(R1Y)*(R2Y) T ]}*(P*H)

[0084] Where H i,j =d(Y i ,Y j ).

[0085] Furthermore, the parameters of the first weight matrix R1, the second weight matrix R2, and the weight vector can also be obtained through training. That is, training data is acquired, including the initial response vector Y used for training and the actual response vector Y obtained after correcting for the influence of environmental factors. G Then, the predicted response vector Y' is calculated using the above method, and its relationship with the true response vector Y' is calculated. G The parameters of the first weight matrix R1, the second weight matrix R2, and the weight vector are obtained by updating the parameters through backpropagation of the gradient using loss functions such as cross-entropy loss function or mean squared error loss function.

[0086] Exemplary computer program products and computer-readable storage media

[0087] In addition to the methods described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the spectral imaging restoration methods according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0088] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0089] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the spectral imaging restoration methods according to various embodiments of this application described in the "Exemplary Methods" section of this specification.

[0090] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0091] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0092] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0093] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0094] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0095] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for spectral imaging reconstruction, characterized in that, include: Incident light is received by the optical modulation layer of the spectral imaging chip. The optical modulation layer includes a plurality of modulation units arranged periodically, and each modulation unit includes a plurality of structural units. The response signal of the image sensor of the spectral imaging chip to the incident light modulated by the light modulation layer is acquired, and each structural unit corresponds to one or more physical pixels of the image sensor; as well as, The spectral information of the incident light is recovered based on the spectral pixels corresponding to each modulation unit, wherein the modulation units corresponding to each spectral pixel are spaced apart from each other by at least one structural unit. The process of recovering the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit includes: The light intensity information corresponding to each structural unit of the spectral pixel is obtained to obtain the response vector; Obtain the modulation matrix and noise vector of the optical modulation layer; and, The spectral vector of the incident light is determined based on the response vector, the modulation matrix, and the noise vector to obtain the spectral information of the incident light.

2. The spectral imaging restoration method according to claim 1, wherein, The number of structural units in the modulation unit interval is equal to the square root of the number of structural units included in the modulation unit.

3. The spectral imaging restoration method according to claim 1, wherein, The number of physical pixels between the modulation units is not less than the number of preset modulation structure units.

4. The spectral imaging restoration method according to claim 1, wherein, Each spectral pixel corresponds to a first number of structural units, and each modulation unit includes a second number of structural units, wherein the second number is not less than the first number.

5. The spectral imaging restoration method according to claim 1, wherein, The optical modulation layer includes a modulation unit with a filter structure and a non-modulation unit without a filter structure.

6. The spectral imaging restoration method according to claim 1, wherein, The step of recovering the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit further includes: Obtain the number and position coordinates of physical pixels and structural units in the spectral pixel unit; When it is determined that the number of physical pixels is greater than the number of structures in the structural unit, the spectral vector of the incident light is determined based on the response vector, the modulation matrix, and the noise vector to obtain the spectral information of the incident light.

7. The spectral imaging restoration method according to claim 1, further comprising: Obtain the modulated transmission spectrum of each structure in each structural unit; The correlation of the transmission spectra is determined based on the obtained transmission spectra; Determine the position coordinates of the uncorrelated transmission spectrum under the structural unit; and The hole convolution rate is determined based on the location coordinates.

8. The spectral imaging restoration method according to claim 1, wherein, The spectral vector of the incident light is X=[λ1,λ2, ……,λ N ] T The response vector is Y = [Y1, Y2, ..., YM]. T The modulation matrix is ​​D: The noise vector W is: W=[W1, W2, ……, W n ] T , The relationship between the spectral vector X, the response vector Y, the modulation matrix D, and the noise vector W is: Y = DX + W.

9. A method for spectral imaging reconstruction, characterized in that, include: Incident light is received by the optical modulation layer of the spectral imaging chip. The optical modulation layer includes a plurality of modulation units arranged periodically, and each modulation unit includes a plurality of structural units. The response signal of the image sensor of the spectral imaging chip to the incident light modulated by the light modulation layer is acquired, and each structural unit corresponds to one or more physical pixels of the image sensor; The spectral information of the incident light is recovered based on the spectral pixels corresponding to each modulation unit, wherein each spectral pixel corresponds to a first number of structural units, each modulation unit includes a second number of structural units, and the second number is not less than the first number; The process of recovering the spectral information of the incident light based on the spectral pixels corresponding to each modulation unit includes: The light intensity information corresponding to each structural unit of the spectral pixel is obtained to obtain the response vector; Obtain the modulation matrix and noise vector of the optical modulation layer; and, The spectral vector of the incident light is determined based on the response vector, the modulation matrix, and the noise vector to obtain the spectral information of the incident light.

10. The spectral imaging restoration method according to claim 9, wherein, The modulation units corresponding to each spectral pixel are spaced apart by at least one structural unit.

11. A spectral imaging restoration device, characterized in that, The apparatus includes the application of the spectral imaging recovery method according to any one of claims 1-8.

12. A spectral imaging restoration device, characterized in that, The apparatus includes the application of the spectral imaging restoration method according to any one of claims 9-10.

Images