Spectrum recovery method, spectrum recovery device and electronic equipment

Abstract

The invention relates to a spectrum recovery method and device and electronic equipment. The spectrum recovery method comprises the following steps: acquiring a light energy response signal matrix and a standard spectrum output by a photosensitive chip of the spectral imaging equipment; determining a basic element recovery function and a response signal vector of the basic element recovery function based on the light energy response signal matrix, wherein the basic element recovery function recovers a spectral image value of a corresponding predetermined channel by using a predetermined pixel value of the photosensitive chip and a pixel value nearby the predetermined pixel value; obtaining a recovery tensor, wherein the product of the recovery tensor and the response signal vector is equal to the output of the basic element recovery function based on the response signal vector; and obtaining a recovered spectral image based on the product of the recovery tensor and the response signal vector. In this way, spectrum recovery which is high in speed, easy in parallel operation and high in recovery precision is achieved.

Description

[0001] This case is a divisional application of the application filed on September 29, 2021, entitled "Spectral Recovery Method, Spectral Recovery Device and Electronic Equipment", with application number 2021111546565. Technical Field

[0002] This application relates to the field of spectral chip technology, and more specifically, to a spectral recovery method, apparatus, and electronic device. Background Technology

[0003] Spectral imaging technology is a technique that organically combines spectral detection and imaging. It can image an object under different spectra, simultaneously obtaining the geometric shape information and spectral characteristics of the object being detected. Spectral imaging technology has become an important tool for Earth observation and deep space exploration, and is widely used in fields such as agricultural and forestry production, mineral resource exploration, cultural relic detection, marine remote sensing, environmental monitoring, disaster prevention and mitigation, and military reconnaissance.

[0004] In practical applications, as spectral resolution increases, analytical methods are used for spectral reconstruction in computational spectral chips. The advantage of analytical methods is that they can directly perform inverse problem calculations. The disadvantage is the difficulty in matrix inversion when high reconstruction resolution is required. In computational spectral chips, the number of structural units often reaches tens or even hundreds of thousands. The matrix elements to be solved represent the contribution of pixels to the spectral wavenumber, thus forming a large matrix unsuitable for inversion operations.

[0005] Therefore, there is a need to provide an improved spectral recovery method. Summary of the Invention

[0006] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a spectral restoration method, a spectral restoration apparatus, and an electronic device, which restore spectral images by introducing a standard spectrum and a restoration tensor, thereby achieving fast, easily parallelizable, and highly accurate spectral restoration.

[0007] According to one aspect of this application, a spectral restoration method is provided, comprising: acquiring a light energy response signal matrix and a standard spectrum output by a photosensitive chip of a spectral imaging device; determining a basic element recovery function and a response signal vector of the basic element recovery function based on the light energy response signal matrix, wherein the basic element recovery function uses predetermined pixel values ​​of the photosensitive chip and nearby pixel values ​​to restore the spectral image value of a corresponding predetermined channel; acquiring a restoration tensor, wherein the product of the restoration tensor and the response signal vector is equal to the output of the basic element recovery function based on the response signal vector; and obtaining a restored spectral image based on the product of the restoration tensor and the response signal vector.

[0008] In the above spectral restoration method, the light energy response signal matrix is ​​represented as including the image width. and image height Two-dimensional matrix The dimension of the standard spectrum is And it is set to minimize the distance between the product of the true value tensor of the spectral image received by the spectral imaging device and the standard spectrum and the tensor of the spectral image to be recovered.

[0009] In the above spectral recovery method, the standard spectrum is represented as: And the corresponding first standard spectrum The standard spectrum of each channel is expressed as follows: , so that:

[0010] in, It is the first spectral pixel of a certain spectrum. Spectral image values ​​of the channel, It is the tensor of the true value of the spectral curve of a certain spectral pixel, and This indicates that the Euclidean distance between tensors is minimized.

[0011] In the above spectral restoration method, the basic element restoration function is used at a predetermined threshold distance from the predetermined pixel in both width and height. The pixel value, and the response signal vector of the predetermined pixel is denoted as , is represented as:

[0012] In the above spectral restoration method, the restored tensor is denoted as... The recovery tensor used to recover the predetermined pixel is the first... The channel recovery vector of the values ​​of each channel is denoted as . ,have:

[0013] in, The basic element recovery function is based on the response signal vector. The output.

[0014] In the above spectral restoration method, the process of solving the restoration tensor includes: establishing a first constraint equation based on the restoration tensor, the spectral response tensor block corresponding to the basic element restoration function of the spectral imaging device, and the standard spectrum; establishing a second constraint equation based on the restoration tensor and the spectral response tensor block; and obtaining the restoration tensor based on the first constraint equation and the second constraint equation.

[0015] In the above spectral restoration method, the first constraint equation is the channel restoration vector. The spectral response tensor block in the spectral response tensor that corresponds to the fundamental element recovery function. and the standard spectrum of the channel The product of these is equal to one, expressed as:

[0016] in, For the recovery of the spectral response tensor of the spectral imaging device Spectral response tensor block The first and second orders of the tensor block are rearranged to the same order, forming a shape of The matrix, and the spectral response tensor is represented as including the image width. Image height and calibrated resolution Three-dimensional tensors .

[0017] In the above spectral restoration method, the second constraint equation is the channel restoration vector. The spectral response tensor block in the spectral response tensor that corresponds to the fundamental element recovery function. The product of a unit vector and a vector is constrained to 0, which is expressed as:

[0018] in It is a unit vector.

[0019] In the above spectral restoration method, obtaining the restored tensor based on the first and second constraint equations includes: obtaining the restored tensor based on the first, second, and third constraint equations, wherein the third constraint equation is the Tikhonov matrix and the channel restoration vector. The L2 norm of the product and the Lagrange multiplier of the regular term The product constraint is 0, which is expressed as:

[0020] In the above spectral restoration method, solving the first, second, and third constraint equations to obtain the restored tensor includes: multiplying the first constraint equation by a sensitivity coefficient and adding it to the second and third constraint equations to obtain a joint equation, expressed as:

[0021] Differentiating and finding the zeros of the joint equation, it can be expressed as:

[0022] The channel recovery vector is obtained as follows:

[0023] Iterate through the above steps to obtain the complete recovery tensor as follows:

[0024] In the above spectral restoration method, the second constraint equation is the channel restoration vector. The spectral response tensor block corresponding to the fundamental element recovery function in the spectral response tensor The product of is constrained to have a L2 norm of 0, expressed as:

[0025] In the above spectral restoration method, obtaining the restored tensor based on the first and second constraint equations includes obtaining the restored tensor based on the first, second, and third constraint equations, and includes: multiplying the first constraint equation by a sensitivity coefficient and adding it to the second and third constraint equations to obtain a joint equation, expressed as:

[0026] Differentiating and finding the zeros of the joint equation, it can be expressed as:

[0027] The channel recovery vector is obtained as follows:

[0028] In the above spectral recovery method, the first constraint equation is expressed as:

[0029] in, It is a neural network consisting of connection layers and activation layers, and It is the block of spectral response tensors in the spectral response tensor that corresponds to the fundamental element recovery function. The first and second orders of the tensor block are rearranged to the same order, forming a shape of The matrix.

[0030] In the above spectral recovery method, the second constraint equation is expressed as:

[0031] In the above spectral recovery method, obtaining the recovered tensor based on the first constraint equation and the second constraint equation includes: The recovered tensor is obtained based on the first constraint equation, the second constraint equation, and the third constraint equation, wherein the third constraint equation is expressed as:

[0032]

[0033] in, The noise particle is a Gaussian distribution with expectation of 0 and shape as follows: A random number matrix.

[0034] In the above spectral recovery method, a fourth constraint equation is established:

[0035] In the above-described spectral restoration method, solving the first constraint equation, the second constraint equation, and the third constraint equation to obtain the restored tensor includes: training the neural network based on the dataset to perform spectral restoration through the trained neural network.

[0036] According to another aspect of this application, a spectral restoration apparatus is provided, comprising: a data acquisition unit for acquiring a light energy response signal matrix and a standard spectrum output by a photosensitive chip of a spectral imaging device; a response signal unit for determining a basic element recovery function and a response signal vector of the basic element recovery function based on the light energy response signal matrix, wherein the basic element recovery function uses predetermined pixel values ​​of the photosensitive chip and nearby pixel values ​​to restore the spectral image value of a corresponding predetermined channel; a recovery tensor unit for acquiring a recovery tensor, wherein the product of the recovery tensor and the response signal vector is equal to the output of the basic element recovery function based on the response signal vector; and a spectral restoration unit for obtaining a restored spectral image based on the product of the recovery tensor and the response signal vector.

[0037] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions that, when the processor is executed, cause the processor to perform the spectral recovery method as described above.

[0038] According to another aspect of this application, a computer-readable storage medium is provided, wherein computer program instructions are stored thereon, which, when executed by a computing device, are operable to perform the spectral recovery method as described above.

[0039] The spectral restoration method, spectral restoration device, and electronic device provided in this application can restore spectral images by introducing standard spectra and restoration tensors, thereby achieving fast, easy-to-parallel-computation, and high-precision spectral restoration. Attached Figure Description

[0040] Various other advantages and benefits of this application will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0041] Figure 1 The illustration shows a schematic configuration diagram of a spectral imaging device according to an embodiment of this application; Figure 2 The illustration shows a schematic diagram of establishing a standard spectrum of the RGB color gamut in the spectral recovery method according to an embodiment of this application; Figure 3 The illustration shows a flowchart of a spectral recovery method according to an embodiment of this application; Figure 4 The figure shows a block diagram of a spectral recovery apparatus according to an embodiment of this application; Figure 5 A block diagram of an electronic device according to an embodiment of this application is illustrated. Detailed Implementation

[0042] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0043] Application Overview The spectral restoration method according to the embodiments of this application is applied to a spectral imaging device. Figure 1 The illustration shows a schematic configuration diagram of a spectral imaging device according to an embodiment of this application. Figure 1As shown, in the spectral imaging device according to the embodiments of this application, the optical system is optional, and may be an optical system such as a lens assembly or a homogenizing assembly. The filter structure is a broadband filter structure in the frequency domain or wavelength domain. The pass spectra of different wavelengths of the filter structure are not completely the same at different locations. The filter structure can be a metasurface, photonic crystal, nanopillar, multilayer film, dye, quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties. For example, in the embodiments of this application, the filter structure can be the light modulation layer in Chinese Patent CN201921223201.2. The image sensor (i.e., photodetector array) can be a CMOS image sensor (CIS), CCD, array photodetector, etc. Additionally, optional data processing units can be MCUs, CPUs, GPUs, FPGAs, NPUs, ASICs, etc., which can export the data generated by the image sensor for external processing.

[0044] For example, after the image sensor measures the light intensity information, it is transmitted to the data processing unit for reconstruction calculation. This process is described in detail below: Incident light at different wavelengths λ The intensity signal below is denoted as x(λ) The transmission spectrum curve of the filter structure is denoted as T(λ) The filter (filter structure) has m The filter structure consists of several structural units, each with a different transmission spectrum. Overall, the filter structure can be denoted as... T i (λ) (i=1,2,3,…,m) Each group of structural units has a corresponding physical pixel below it, which detects the light intensity bi modulated by the filtered light structure. In a specific embodiment of this application, one physical pixel corresponds to one group of structural units, but this is not a limitation. In other embodiments, multiple physical pixels may be grouped together to correspond to one group of structural units. Therefore, in the spectral imaging device according to the embodiments of this application, multiple groups of structural units constitute a "spectral pixel". Furthermore, the present invention can use at least one spectral pixel to reconstruct an image. It should be noted that the effective transmission spectrum of the filter structure (the transmission spectrum used for spectral reconstruction is called the effective transmission spectrum) is... T i (λ)The number of structural units may not be the same as the number of structural units. The transmission spectrum of the filter structure is set, tested, or calculated manually according to certain rules based on the needs of identification or recovery (for example, the transmission spectrum obtained by each structural unit through testing is the effective transmission spectrum). Therefore, the number of effective transmission spectra of the filter structure may be less than the number of structural units, or even more than the number of structural units. In this modified embodiment, a certain transmission spectrum curve is not necessarily determined by a group of structural units.

[0045] The relationship between the spectral distribution of incident light and the measurements from the image sensor can be expressed by the following formula: b i =∫x(λ)*T i (λ)*R(λ)dλ After discretization, we get: b i =Σ(x(λ)*T i (λ)*R(λ)) Where R(λ) is the response of the image sensor, denoted as: A i (λ)=T i (λ)*R(λ), The above equation can then be extended into matrix form:

[0046] in, b i (i=1,2,3,…,m) These are the responses of the image sensor after the light under test passes through the filter structure, respectively corresponding to... m The light intensity measurement value of the image sensor corresponding to each structural unit. When one physical pixel corresponds to one structural unit, it can be understood as: m The light intensity measurement value corresponding to each "physical pixel" is a value of length . m The vector. A The system's response to light of different wavelengths is determined by two factors: the transmittance of the filter structure and the quantum efficiency of the image sensor. A It is a matrix, where each row vector corresponds to a set of structural units responding to incident light of different wavelengths. Here, the incident light is sampled discretely and uniformly, resulting in a total of [number missing]. n One sampling point. A The number of columns is the same as the number of sampling points for the incident light. Here, x(λ) That is, incident light at different wavelengths λ The light intensity is the incident light spectrum that is to be measured.

[0047] In some embodiments, unlike the above embodiments, the filter structure can be directly formed on the upper surface of the image sensor, such as quantum dots, nanowires, etc., which directly form the filter structure or material (nanowires, quantum dots, etc.) in the photosensitive area of ​​the sensor. Taking the filter structure as an example, it can be understood that when the raw materials of the image sensor are processed to form the image sensor, the filter structure is formed on the upper surface of the raw materials. The transmission spectrum and the response of the image sensor are integrated, that is, it can be understood that the response of the detector and the transmission spectrum are the same curve. At this time, the relationship between the spectral distribution of the incident light and the light intensity measurement value of the image sensor can be expressed by the following formula: b i =Σ(x(λ) *R i (λ)) That is, in this embodiment, the transmission spectrum A i (λ)= R i (λ) Furthermore, it can also be a combination of the two embodiments described above, that is, at least one filter structure for modulating incident light is provided on the image sensor with the filter structure. It can be understood that the image sensor (i.e., photodetector array) in the first embodiment, which can be a CMOS image sensor (CIS), CCD, array photodetector, etc., can be replaced with an image sensor with an integrated filter structure in the second embodiment.

[0048] At this point, the relationship between the spectral distribution of the incident light and the light intensity measurement value of the image sensor can be expressed by the following formula: b i =∫x(λ)*T i (λ)*R i (λ)dλ After discretization, we get: b i =Σ(x(λ)*T i (λ)*R i (λ)) That is, in this embodiment, A i (λ)=T i (λ)*R i (λ) Exemplary methods If we consider multi-channel spectral imaging, the imaging principle of each spectral pixel in a spectral imaging device, such as a snapshot spectral camera, can be expressed by the following equation:

[0049] here, The tensor representing the spectral image that the algorithm wants to recover is generally composed of... , , It consists of three orders, among which, Image width, in pixels; The height of the image, in pixels; and The number of spectral image channels output by the spectral imaging device (e.g., for RGB images, ...). ).and, The spectral response tensor of the spectral imaging device, pre-calibrated by... , , It consists of three orders, of which the dimension To determine the calibration resolution, that is, the number of spectral data channels for the target spectral band provided by the calibration device. This tensor This characterizes the transmission capability of photosensitive chips (i.e., spectral chips), such as the structures on image sensors as described above, for monochromatic light of different wavelengths. Representation tensor The line, number Column, No. The value of the layer number, i.e., pixels. The transmittance of the o-th monochromatic light, and Indicates targeting only the first The transmittance of a single monochromatic light-emitting diode (SCDED) chip. Furthermore... It is the light energy response signal matrix given by the photosensitive chip of the spectral imaging device, which is composed of... , It consists of two orders.

[0050] Therefore, in this embodiment of the application, spectral recovery is performed within a known... Under the given conditions, find , so that:

[0051] in To represent a certain error function, in image processing, SSIM (Structural Similarity) is commonly used as a function to measure error. This is the spectral recovery function.

[0052] To obtain spectral image tensors using spectral reconstruction methods In this application embodiment, a new concept in the field of spectral reconstruction, "standard spectrum", is introduced, denoted as... Its dimension is And corresponding to the first The standard spectrum of each channel is This refers to the standard spectral vector. Furthermore, the standard spectral vectors are stacked together as column vectors to form a matrix. ,Depend on , It consists of two orders. In spectral imaging devices, Each channel Each corresponds to a standard spectrum .

[0053] here, Figure 2 The illustration shows a schematic diagram of establishing a standard spectrum of the RGB color gamut in the spectral recovery method according to an embodiment of this application.

[0054] like Figure 2 As shown, it illustrates the reference to the CIE 1931 standard. , , The curve (i.e., the tristimulus value curve, which can also be written as X, Y, Z).

[0055] Assume the observation matrix has an observation range of the spectrum as follows: nm~ nm, observation precision is nm, then the standard spectrum has a total of There are 1 element. Therefore, the three standard spectra can be calculated as follows:

[0056]

[0057]

[0058] in, .

[0059] Note that the following conditions must be met when establishing a standard spectrum:

[0060]

[0061] In addition, the tensor of the true value of the spectral image received by the spectral imaging device is denoted as... , its origin , , Composed of three orders, representing the spectrum of the initial incident light at each physical pixel, we have:

[0062] in, It can also be written as Generally refers to the first spectral pixel of a certain spectrum. The spectral image values ​​of the channel, i.e. ,in , It is a general term. Furthermore, Generally speaking, this refers to the tensor of the true value of the spectral curve of a specific spectral pixel. , Also used in a general sense. The symbol " "" indicates that it should be as close as possible to, and if used between tensors, it means that their Euclidean distance should be as small as possible.

[0063] As mentioned above, and Obtaining this requires the use of the spectral reconstructive function, namely:

[0064] therefore:

[0065] Based on this, the objective of the spectral recovery method according to the embodiments of this application is transformed into finding a function. ,make:

[0066] Specifically, as described above, the main functional structure of the spectral imaging device according to the embodiments of this application is a photosensitive chip (i.e., a spectral chip) covered with a modulation layer. The photosensitive chip has a wide... Individual, tall Each photosensitive unit can independently respond to light illumination, and it can be assumed that the response of a photosensitive unit to light is positively linearly correlated with the light energy. A modulation layer composed of various structures is applied to the photosensitive chip, which is defined as a spectral chip. This modulation layer causes different photosensitive units on the chip to have different responses to light of the same energy but different spectra. This variation in response is caused by… It is determined by:

[0067] The meaning of the above formula is: order-pair tensor The accumulation. Here, This represents generalized multiplication. When used between tensors, it takes the smallest tensor order and multiplies its elements element by element. For higher orders, it uses a broadcast strategy to multiply elements by element.

[0068] Here, tensor It is determined by the modulation layer on the photosensitive chip. That is, in the photosensitive unit The transmission spectrum on the image. In the following text, the term "physical pixel" is used to refer to the smallest imaging unit on the photosensitive chip, which can be composed of one or more physical pixels; the present invention is preferably described using a one-to-one example.

[0069] The spectral restoration method according to the embodiments of this application finally outputs a multispectral image. In the middle, for each These are all defined as "spectral pixels." As the name suggests, a spectral pixel is a geometric pixel in a multispectral image. Further, it can be understood that the modulation layer has at least one modulation unit, each modulation unit corresponding to at least one physical pixel. The modulation unit and the physical pixel constitute a structured pixel, and at least one structured pixel constitutes a spectral pixel. Each modulation unit has a corresponding transmission spectrum curve, that is, the transmission spectrum curve is determined by the modulation unit, and the transmission spectrum curve further constitutes a transmission spectrum tensor. It should be noted that the transmission spectral tensor A can be obtained through calibration, calculated artificially, or obtained through other methods. Furthermore, the spectral chip of the spectral imaging device in this invention has at least two spectral pixels, and at least two spectral pixels have different structures or different light modulation effects. The value of a channel within a spectral pixel can be defined as a "basic element". A basic element is the smallest element required by the spectral recovery method according to the embodiments of this application. In parallel computing, the recovery operation of each basic element corresponds to one thread.

[0070] To obtain the value of a basic element, we need to use the value of its corresponding physical pixel and the values ​​of its nearby physical pixels, that is:

[0071] in, The selection should be determined based on the actual situation. Defined as the basic primitive recovery function. Once determined... Then the function described above can be determined. This allows for spectral recovery. Therefore, in the following text, we will use... When used to represent a specific channel of a specific spectral pixel ,use This refers to the photosensitive chip response signal vector corresponding to a specific spectral pixel, that is, It is used to determine the photosensitive chip response signal vector of a specific spectral pixel, which is transformed into a vector. When referring to a specific spectral pixel, it can be used Therefore, the above formula is to find... To satisfy:

[0072] This function It can be any differentiable form, and when it is a linear function, it takes the following form:

[0073] Here, in the spectral recovery method according to the embodiments of this application, a new concept, "recovery tensor," is introduced, for example, denoted as... , its origin , , , It consists of four orders, among which, It is the side length of the physical pixel array used to recover any single spectral pixel, measured in pixels. Furthermore, That is, recover the tensor The first pixel used to recover a certain pixel A vector of values ​​for each channel, i.e. ;in , It is a general term. Furthermore, It represents the product of vectors or matrices. When used between vectors, it is called the inner product; when used between matrices, it is called the matrix multiplication. When used between tensors of different orders, the smallest first or second order is used for the vector inner product or matrix multiplication. For higher orders, a broadcast strategy is used.

[0074] As mentioned above, in spectral imaging devices, Each channel Each corresponds to a standard spectrum Assuming the total light energy remains at a unit energy level, if the spectral imaging device achieves this at a certain spectral pixel... The actual spectrum received above With channel When the corresponding standard spectra are completely identical, the spectral pixel value of the corresponding channel of the image output by the imaging device should be 1. Of course, those skilled in the art will understand that a spectral pixel value of 1 is not a limitation and can be other constants. Thus, the following limiting equation (1) is obtained:

[0075] here, For recovery Spectral response tensor block The first and second orders of this tensor block should be rearranged to the same order to form a shape of The matrix, where each pixel Defined as .

[0076] It can be understood that a standard spectrum is a standard filter on each channel. When the actual spectrum completely overlaps with the transmission spectrum of this filter, the value on that channel should be 1. This standard spectrum can be a Gaussian filter with a specific wavelength at its center and a fixed peak width; or it can be any defined shape (such as RGB tristimulus values).

[0077] In addition to this, for For all other spectra, the response should be as small as possible. Therefore, the constraint equation (2) is obtained:

[0078] in It is a vector in which all elements are 1.

[0079] In addition, to prevent overfitting, the Tikhonov method can be used to adjust the parameter vector. By applying constraints, we obtain the constraint equation (3):

[0080] in It is a Tikhonov matrix, a diagonal matrix, which is usually taken as the identity matrix. is the Lagrange multiplier for the regularization term. Furthermore, if overfitting prevention is not required, then... It can also be set to 0.

[0081] By using a solver to solve the above three equations, we can obtain the results. The value of .

[0082] Considering the limited computational speed of the solver, in the spectral recovery method according to the embodiments of this application, a more efficient method can be proposed to solve the above set of equations, that is, to integrate the three constraint equations into one:

[0083] In order to allow the constraint equation (1) to also be included In this process, a sensitivity multiplier is introduced. This is used to characterize the spectral recovery sensitivity requirement. Higher values ​​result in higher sensitivity, but the corresponding model robustness and spectral accuracy will decrease. This needs to be confirmed in practical applications. The methods for determining the values ​​include, but are not limited to, using a leave-one-out strategy to train from a known spectral dataset; or empirically setting the values ​​based on the known error characteristics of the calibration data and actual needs.

[0084] Furthermore, as mentioned above, in In this case, the integrated constraint equations may also include only the first and second constraint equations as described above.

[0085] In order to To minimize it, we should find its zeros after differentiating it, that is:

[0086] To simplify calculations, the integrated formula can be adjusted as follows:

[0087] Solving the equation, we get Method for finding:

[0088] Extending this algorithm to the entire image yields:

[0089] Finally, the recovery matrix is ​​obtained. The hyperspectral image can then be obtained using the following formula:

[0090] The spectral restoration method according to the embodiments of this application has the advantages of high speed, easy parallel computation, and high restoration accuracy, and it is feasible as a spectral imaging restoration algorithm.

[0091] In a modified embodiment, considering that the constraint equation (2) is required to be more stringent in the following form, but model fitting will also be more difficult:

[0092] The final constraint equations are:

[0093] Take the derivative, and set the derivative to zero:

[0094] To facilitate the calculation of modulation constraint equations

[0095] Solving for:

[0096] Figure 3The illustration shows a flowchart of a spectral recovery method according to an embodiment of this application.

[0097] like Figure 3 As shown, the spectral recovery method according to an embodiment of this application includes the following steps.

[0098] Step S110: Obtain the light energy response signal matrix and standard spectrum output by the photosensitive chip of the spectral imaging device.

[0099] That is, to obtain the light energy response signal matrix as described above. ,and Including image width and image height Two dimensions.

[0100] That is, in the spectral restoration method according to the embodiments of this application, the light energy response signal matrix is ​​represented as including the image width. and image height Two-dimensional matrix .

[0101] Furthermore, the dimension of the standard spectrum is the same as the calibration resolution of the spectral response tensor of the pre-calibrated spectral imaging device, that is, it is also... Furthermore, the standard spectrum is set such that the distance between the product of the true value tensor of the spectral image received by the spectral imaging device and the standard spectrum is minimized and the spectral image tensor to be recovered is minimized, i.e.:

[0102] Here, the standard spectrum is denoted as Its corresponding number The standard spectrum of each channel is denoted as , It can also be written as Generally refers to the first spectral pixel of a certain spectrum. The spectral image values ​​of the channel, i.e. ,in , It is a general term. Furthermore, Generally speaking, this refers to the tensor of the true value of the spectral image for a specific spectral pixel. , Also used in a general sense. The symbol " "" indicates that it should be as close as possible to, and if used between tensors, it means that their Euclidean distance should be as small as possible.

[0103] That is, in the spectral recovery method according to the embodiments of this application, the standard spectrum is represented as... And the corresponding first standard spectrum The standard spectrum of each channel is expressed as follows: , so that:

[0104] in, It is the first pixel of a certain pixel. Spectral image values ​​of the channel, It is the tensor of the true spectral image value of a certain pixel. This indicates that the Euclidean distance between tensors is minimized.

[0105] Step S120: Determine the basic element recovery function and the response signal vector of the basic element recovery function based on the light energy response signal matrix. The basic element recovery function uses the predetermined pixel value of the photosensitive chip and the pixel value of its vicinity to recover the spectral image value of the corresponding predetermined channel.

[0106] That is, based on the formula described above:

[0107] Obtain the response signal vector .

[0108] That is, in the spectral restoration method according to the embodiments of this application, the basic element restoration function uses pixel values ​​that are at a predetermined distance of a predetermined threshold from the predetermined pixel in both width and height, and the response signal vector of the predetermined pixel is denoted as... .

[0109] Step S130: Obtain the recovery tensor, the product of the recovery tensor and the response signal vector is equal to the output of the basic element recovery function based on the response signal vector.

[0110] That is, in the spectral recovery method according to the embodiments of this application, the recovery tensor is denoted as... The recovery tensor used to recover the predetermined pixel is the first... The channel recovery vector of the values ​​of each channel is denoted as . Then we have:

[0111] in, The basic element recovery function is based on the response signal vector. The output.

[0112] Step S140: The recovered spectral image is obtained based on the product of the recovery tensor and the response signal vector.

[0113] That is, in the spectral restoration method according to the embodiments of this application, the pixels of the spectral image are:

[0114] As described above, in this embodiment of the application, the recovery tensor can be solved by establishing constraint equations. The specific process includes: establishing a first constraint equation based on the recovery tensor, the spectral response tensor block corresponding to the basic element recovery function of the spectral imaging device, and the standard spectrum; establishing a second constraint equation based on the recovery tensor and the spectral response tensor block; and obtaining the recovery tensor based on the first constraint equation and the second constraint equation.

[0115] The first constraint equation is the constraint equation (1) as described above:

[0116] in, For recovery Spectral response tensor block The first and second orders of this tensor block should be rearranged to the same order to form a shape of The matrix, where each pixel Defined as .

[0117] Therefore, in the spectral recovery method according to the embodiments of this application, the first limiting equation is the channel recovery vector. The spectral response tensor block in the spectral response tensor that corresponds to the fundamental element recovery function. and the standard spectrum of the channel The product of is equal to one.

[0118] Furthermore, the second constraint equation is the constraint equation (2) as described above:

[0119] in It is a vector in which all elements are 1, i.e., a unit vector.

[0120] Therefore, in the spectral recovery method according to the embodiments of this application, the second limiting equation is the channel recovery vector. The spectral response tensor block in the spectral response tensor that corresponds to the fundamental element recovery function. The product constraint with the unit vector is 0.

[0121] Thus, based on the first constraint equation and the second constraint equation, the recovered tensor is obtained.

[0122] Furthermore, as described above, preferably, in the spectral recovery method according to the embodiments of this application, the solution can be obtained by integrating the first constraint equation and the second constraint equation.

[0123] Furthermore, preferably, when solving the recovery tensor, the recovery tensor is further constrained to prevent overfitting to obtain a third constraint equation.

[0124] The third constraint equation is the constraint equation (3) as described above:

[0125] Therefore, in the spectral recovery method according to the embodiments of this application, the third limiting equation is the Tikhonov matrix and the channel recovery vector. The L2 norm of the product and the Lagrange multiplier of the regular term The product constraint is 0.

[0126] Thus, obtaining the recovered tensor based on the first and second constraint equations includes solving the first, second, and third constraint equations to obtain the recovered tensor.

[0127] Specifically, the first constraint equation is first multiplied by the sensitivity coefficient, and then added to the second and third constraint equations to obtain the joint equation. Next, the channel recovery vector is obtained by solving for the zeros of the derivative of the joint equation. Finally, the above steps are iteratively repeated to obtain the entire recovery tensor.

[0128] Therefore, in the spectral recovery method according to the embodiments of this application, solving the first constraint equation, the second constraint equation, and the third constraint equation to obtain the recovered tensor includes: Multiplying the first constraint equation by the sensitivity coefficient and adding it to the second and third constraint equations yields the joint equation, expressed as:

[0129] Differentiating and finding the zeros of the joint equation, it can be expressed as:

[0130] The channel recovery vector is obtained as follows:

[0131] Iterate through the above steps to obtain the complete recovery tensor as follows:

[0132] Furthermore, in a variant embodiment, the second limiting equation is the channel recovery vector. The spectral response tensor block corresponding to the fundamental element recovery function in the spectral response tensor The product of is constrained to have a L2 norm of 0, expressed as:

[0133] Then, multiplying the first constraint equation by the sensitivity coefficient and adding it to the second and third constraint equations yields the joint equation, expressed as:

[0134] The channel recovery vector is obtained as follows:

[0135] Here, we can see that the value of the channel recovery vector is based on the spectral response tensor block corresponding to the fundamental element recovery function. and standard spectrum Therefore, for a given spectral imaging device, the spectral response tensor can be pre-calibrated and a standard spectrum can be acquired to determine the channel recovery vector. Thus, when performing spectral imaging, the response signal vector can be determined based on the light energy response signal matrix, thereby obtaining the recovered spectral image.

[0136] Therefore, by introducing a standard spectrum and a recovery tensor to recover the spectral image, a spectral recovery method that is fast, easy to parallelize, and has high recovery accuracy according to the embodiments of this application is realized.

[0137] Furthermore, in the embodiments of this application, a method based on a neural network model can also be used to restore the spectral image.

[0138] Specifically, when using a neural network for recovery, that is:

[0139] in This is a neural network composed of connection layers and activation layers. To allow the neural network to acquire as much information as possible, information from the response matrix needs to be further incorporated, i.e.:

[0140] According to the definition of standard spectrum as described above, when , The area covered All were spectralized as When the light shines, That is, the first limiting equation is:

[0141] At the same time, considering the need to minimize errors, the second constraint equation should be satisfied:

[0142] Furthermore, for neural networks, regularization terms can no longer be used to control overfitting. Therefore, noise factors are introduced. The data is strengthened to obtain the third constraint equation:

[0143]

[0144] here, Represents a noise particle, with the shape of is a random number matrix with an expected value of 0 that follows a Gaussian distribution. The farther the elements are from the center of the matrix, the greater the variance of the random numbers. The specific parameters can be selected according to the actual training situation.

[0145] In addition, other common data augmentation methods can be used to control model overfitting. However, these two constraints alone are insufficient for more complex neural network models to learn effectively. Therefore, in this embodiment, data from a hyperspectral database can be used to further train the model, i.e., a fourth constraint equation can be established:

[0146] in, This represents the true value of the spectral image.

[0147] Based on the above constraint equations, dataset pairs are created. Training can then be performed to achieve spectral imaging restoration. In the embodiments of this application, the neural network includes, but is not limited to: FC (fully connected), CNN (convolutional neural network), RNN (recurrent neural network), ResNet, attention (attention neural network), transformer (transformer neural network), and their variations.

[0148] That is, in the spectral recovery method according to the embodiments of this application, the first limiting equation is expressed as:

[0149] in, It is a neural network consisting of connection layers and activation layers.

[0150] Furthermore, in the above-described spectral recovery method, the second constraint equation is expressed as:

[0151] Furthermore, in the above-described spectral recovery method, obtaining the recovered tensor based on the first and second constraint equations includes: obtaining the recovered tensor based on the first, second, and third constraint equations, wherein the third constraint equation is expressed as:

[0152]

[0153] in, The noise particle is a Gaussian distribution with expectation of 0 and shape as follows: A random number matrix.

[0154] Furthermore, the above-mentioned spectral recovery method further includes establishing a fourth constraint equation:

[0155] Accordingly, in the above-described spectral restoration method, solving the first constraint equation, the second constraint equation, and the third constraint equation to obtain the restoration tensor includes: training the neural network based on the dataset to perform spectral restoration through the trained neural network.

[0156] Indicative device Figure 4 A block diagram of a spectral recovery apparatus according to an embodiment of this application is shown.

[0157] like Figure 4 As shown, the spectral restoration apparatus 200 according to an embodiment of this application includes: a data acquisition unit 210, configured to acquire the light energy response signal matrix and standard spectrum output by the photosensitive chip of the spectral imaging device; a response signal unit 220, configured to determine a basic element recovery function and a response signal vector of the basic element recovery function based on the light energy response signal matrix, wherein the basic element recovery function uses predetermined pixel values ​​of the photosensitive chip and nearby pixel values ​​to restore the spectral image value of its corresponding predetermined channel; a recovery tensor unit 230, configured to acquire a recovery tensor, wherein the product of the recovery tensor and the response signal vector is equal to the output of the basic element recovery function based on the response signal vector; and a spectral restoration unit 240, configured to obtain the restored spectral image based on the product of the recovery tensor and the response signal vector.

[0158] Here, those skilled in the art will understand that the specific functions and operations of each unit and module in the above-described spectral recovery device 200 have been referenced above. Figure 3 The spectral recovery method is described in detail here, and therefore, its repeated description will be omitted.

[0159] As described above, the spectral recovery device 200 according to the embodiments of this application can be implemented in various terminal devices, such as servers for high-resolution spectral recovery, or various spectrometers and spectral imaging devices. In one example, the spectral recovery device 200 according to the embodiments of this application can be integrated into the terminal device as a software module and / or a hardware module. For example, the spectral recovery device 200 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the spectral recovery device 200 can also be one of many hardware modules of the terminal device.

[0160] Alternatively, in another example, the spectral recovery device 200 and the terminal device can also be separate devices, and the spectral recovery device 200 can be connected to the terminal device via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0161] Exemplary electronic devices Below, for reference Figure 5 This describes an electronic device according to embodiments of the present application.

[0162] Figure 5 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0163] like Figure 5 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0164] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0165] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the spectral recovery methods of the various embodiments of this application described above and / or other desired functions. Various contents such as response signal data and standard spectral data may also be stored in the computer-readable storage medium.

[0166] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0167] For example, the input device 13 can be, for example, a keyboard, a mouse, etc.

[0168] The output device 14 can output various information to the outside, such as spectral reconstruction results. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0169] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0170] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the spectral recovery methods according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0171] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0172] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the spectral recovery methods according to various embodiments of this application described in the "Exemplary Methods" section of this specification.

[0173] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0174] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0175] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0176] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0177] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0178] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A spectral restoration method, characterized by, The method comprises: acquiring a light energy response signal matrix and a standard spectrum output by a photosensitive chip of the spectral imaging device, wherein the photosensitive chip is composed of a light detector array and a light filtering structure arranged above the light detector array; determining a basic element recovery function and a response signal vector of the basic element recovery function based on the light energy response signal matrix, wherein the basic element recovery function uses a predetermined pixel value of the photosensitive chip and pixel values around the predetermined pixel value to recover a spectral image value of a predetermined channel corresponding to the predetermined pixel value; acquiring a recovery tensor, wherein a product of the recovery tensor and the response signal vector is equal to an output of the basic element recovery function based on the response signal vector; and obtaining a recovered spectral image based on the product of the recovery tensor and the response signal vector.

2. The spectral restoration method of claim 1, wherein, The light energy response signal matrix is expressed as a matrix including image width and image height two dimensions , the dimension of the standard spectrum is , and is set to the distance between the product of the true value tensor of the spectral image received by the spectral imaging device and the standard spectrum and the spectral image tensor to be recovered is minimum.

3. The spectral restoration method of claim 2, wherein, The standard spectrum is represented as and the channel standard spectrum of the corresponding jth channel of the standard spectrum is represented as such that: wherein is the true value of the spectral curve of a certain spectral pixel and is the spectral image value of the channel of a certain spectral pixel, is the true value of the spectral curve of a certain spectral pixel and denotes the Euclidean distance between the tensors.

4. The spectral restoration method of claim 1 or 3, wherein, The basic element recovery function uses pixel values that are a predetermined threshold distance in width and height from the predetermined pixel, and the response signal vector of the predetermined pixel is denoted by , and is expressed as: ​ 。 5. The spectral restoration method of claim 4, wherein, The recovery tensor is denoted by The channel recovery vector for recovering the value of the i-th channel of the predetermined pixel in the recovery tensor is denoted by The channel recovery vector has the form where the channel recovery vector is determined by wherein, is the output of the basic meta-recovery function based on the response signal vector .

6. The spectral restoration method of claim 1 or 5, wherein, The solving process of the recovery tensor comprises: establishing a first constraint equation based on the recovery tensor, a spectral response tensor block corresponding to the basic element recovery function of a spectral response tensor of the spectral imaging device, and the standard spectrum; establishing a second constraint equation based on the recovery tensor and the spectral response tensor block; and obtaining the recovery tensor based on the first constraint equation and the second constraint equation.

7. The spectral restoration method of claim 6, wherein, said first restriction equation is that the product of the channel recovery vector a spectral response tensor block of the spectral response tensor corresponding to the basis element recovery function and the channel standard spectrum equals one, denoted as: wherein, a spectral response tensor of the spectral imaging device for recovering a spectral response tensor block , the first and second order of the tensor block are rearranged to the same order, forming a matrix of shape , and the spectral response tensor is represented as a tensor comprising three dimensions of image width , image height , and nominal resolution .​ 8. The spectral restoration method of claim 6 or 7, wherein, The second restriction equation is that the channel recovery vector The product of the spectral response tensor block corresponding to the basis element recovery function in the spectral response tensor and the unit vector is constrained to be 0, denoted as: wherein is a unit vector.

9. The spectral restoration method of claim 8, wherein, The obtaining of the recovery tensor based on the first constraint equation and the second constraint equation comprises: obtaining the recovery tensor based on the first constraint equation, the second constraint equation, and a third constraint equation, the third constraint equation being a product of a Tikhonov matrix and a channel recovery vector constrained to be zero, expressed as: ​ 。 10. The spectral restoration method of claim 9, wherein, solving the first constraint equation, the second constraint equation and a third constraint equation to obtain the recovery tensor, which comprises: multiplying the first constraint equation by a sensitivity coefficient, and adding the second constraint equation and the third constraint equation to obtain a joint equation, which is represented as: deriving the joint equation and finding a zero point, which is represented as: obtaining the channel recovery vector as: iterating the above steps to obtain the entire recovery tensor as: 。 11. The spectral restoration method of claim 7, wherein, the second constraint equation is a two-norm constraint on the product of the channel recovery vector and a spectral response tensor block in the spectral response tensor corresponding to the elemental recovery function is zero, denoted as: 。 12. The spectral restoration method of claim 11, wherein, The obtaining of the recovery tensor based on the first constraint equation and the second constraint equation comprises the obtaining of the recovery tensor based on the first constraint equation, the second constraint equation and the third constraint equation, and the obtaining comprises: multiplying the first constraint equation by a sensitivity coefficient, and adding the second constraint equation and the third constraint equation to obtain a joint equation, which is represented as: deriving the joint equation and finding a zero point, which is represented as: obtaining the channel recovery vector as: 。 13. The spectral restoration method of claim 6, wherein, the first constraint equation is represented as: wherein, is a neural network composed of a connection layer and an activation layer, and is a spectral response tensor block in the spectral response tensor corresponding to the elementary meta-recovery function , the first and second orders of the tensor block are rearranged to the same order to form a matrix with a shape of .

14. The spectral restoration method of claim 13, wherein, the second constraint equation is represented as: 。 15. The spectral restoration method of claim 14, wherein, The obtaining of the recovery tensor based on the first constraint equation and the second constraint equation comprises: the obtaining of the recovery tensor based on the first constraint equation, the second constraint equation and the third constraint equation, and the third constraint equation is represented as: where, is a noise sub-matrix, which is a matrix of random numbers with a shape of and is expected to be 0, following a Gaussian distribution.

16. The spectral recovery method according to claim 15, further comprising establishing a fourth constraint equation: 。 17. The spectral restoration method of claim 16, wherein, The solving of the first constraint equation, the second constraint equation and the third constraint equation to obtain the recovery tensor comprises: training the neural network based on the data set, so as to perform spectral recovery through the trained neural network.

18. A spectral restoration apparatus, characterized by, The method comprises: a data acquisition unit configured to acquire a spectral response tensor of a pre-calibrated spectral imaging device, a light energy response signal matrix output by a photosensitive chip of the spectral imaging device, and a standard spectrum; a response signal unit configured to determine a basis element recovery function and a response signal vector of the basis element recovery function based on a light energy response signal matrix, the basis element recovery function using a predetermined pixel value of the photosensitive chip and pixel values around the predetermined pixel value to recover a spectral image value of a predetermined channel corresponding to the predetermined pixel value; a recovery tensor unit configured to obtain a recovery tensor, a product of the recovery tensor and the response signal vector being equal to an output of the basis element recovery function based on the response signal vector; and a spectral recovery unit configured to obtain a recovered spectral image based on a product of the recovery tensor and the response signal vector. ​

Images