A neural network-based narrow-band spectral imaging method

Abstract

The application discloses a narrow-band spectral imaging method based on a neural network, which comprises the following steps: a target object is encoded, imaged and detected by a broadband encoding filter array to obtain corresponding full-band light intensity information, then the obtained full-band light intensity information is input into a constructed truncated network to obtain each narrow-band image; the truncated network is obtained by training a full network, and the full network is composed of a front prediction network, a truncated network and a full connection layer connected between the two. Compared with a traditional narrow-band spectral imaging technology based on a dispersion element, the application can greatly improve the speed of acquiring a narrow-band light intensity image, quantitatively evaluate the optimality of the reconstructed narrow-band light intensity and improve the signal-to-noise ratio of the acquired narrow-band light intensity. Meanwhile, the design result is more targeted, and the manufacturing cost of a narrow-band spectral imaging device can be greatly reduced.

Description

Technical Field

[0001] This invention relates to the fields of spectral sensing and photoelectric imaging, and specifically to a narrow-band spectral imaging method based on neural networks, which can be widely used in the fields of material identification, agricultural product monitoring, and medical diagnosis. Background Technology

[0002] Image information, as a fundamental medium for understanding the world, is crucial in our daily lives. Currently, most imaging technologies record scene information based on the three visible light bands (RGB), suitable for characterizing objects based on their shape and color. However, due to the limited segmentation into only three visible bands, the recognition capability is quite limited. Furthermore, imaging actual objects involves a large amount of reflectance / transmission spectral information, making it unsuitable for a wide range of spectral analysis applications. By measuring the emission / reflection / transmission / radiation characteristics of specific narrow-band light intensities from different calibrated light sources or objects, we can identify substances and determine their composition, finding wide applications in astronomical observation, chemical analysis, food safety, medical diagnosis, and many other fields.

[0003] Traditional imaging spectrometers mainly consist of dispersive spectroscopic elements (gratings, prisms, and narrowband tunable filters) and detectors. They can sequentially decompose a continuous wide-band spectrum into energy outputs of specific wavelengths to achieve scanning and acquisition of the entire spectrum. This approach can achieve high spectral resolution and a wide spectral range, but it has disadvantages such as large size, slow scanning speed, and low signal throughput.

[0004] With the continuous development of artificial intelligence and machine learning algorithms, some methods for jointly optimizing hardware and software design using deep learning have emerged in recent years. Object or scene information is modulated by encoders such as metasurfaces, nanowires, and thin-film filters. Sensors acquire the raw signals, and complex software algorithms are used to deconstruct the original image signal, achieving snapshot imaging. After processing the obtained cubic spectral data, a light intensity image of a narrow band of interest can be obtained. However, obtaining a narrow band light intensity image in this way remains complex, and the narrow band light intensity image obtained from the reconstructed spectral data cube will increase the reconstruction error compared to the actual image. Therefore, obtaining an accurate narrow band image is often difficult. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a narrow-band spectral imaging method based on neural networks, enabling the calculation of light intensity from the full-band spectrum to the narrow-band spectrum.

[0006] A narrow-band spectral imaging method based on neural networks includes: encoding the target object through a broadband coded filter array, imaging, and detection by a detector to obtain corresponding full-band light intensity information; then inputting the obtained full-band light intensity information into a constructed truncated network to obtain images of various narrow bands; the truncated network is trained from a full network, which consists of a pre-prediction network, a truncated network, and a fully connected layer connecting the two.

[0007] Preferably, the pre-prediction network adopts a neural network structure, with narrow-band light intensity as input and wide-band spectrum as output. The number of input neurons is equal to the number of narrow-band channels, and the number of output neurons is equal to the number of wide-band spectrum channels, with several hidden layers in between. The structure and specific number of hidden layers can be selected according to actual conditions and optimized based on optimization results to ultimately obtain a structure that meets the target requirements.

[0008] Preferably, the truncated network adopts a neural network structure, with the input being the full-band light intensity and the output being the narrow-band light intensity. The number of input neurons is equal to the number of coded filters, and the number of output neurons is equal to the number of narrow-band light intensity channels. Several hidden layers are set in the middle.

[0009] Preferably, in the fully connected layer, the connection weight between the i-th input neuron and the j-th output neuron is W. ij , representing the spectral response of the j-th coded filter in the i-th spectral channel, where i = 1, 2, ..., M, j = 1, 2, ..., N, M is the number of broadband spectral channels, and N is the number of coded filters. It should be noted that theoretically, W... ij , representing the spectral response of the j-th coded filter in the i-th spectral channel. In actual modeling, we also consider the influence of the detector; that is, as a preferred approach, W is used in modeling. ij W can be obtained by combining the known parameters of the filter and the detector. ij This corresponds to the actual transmittance after the manufactured coded filter is combined with the detector response.

[0010] Preferably, the pre-prediction network is trained separately using narrow-band light intensity and the corresponding wide-band spectrum. After obtaining the trained pre-prediction network, the entire network is then constructed and subsequently trained.

[0011] As a preferred option, during the full network training process, the pre-prediction network is set to not update parameters; the weights of the fully connected layers are set to encode the actual transmittance of the filter; and all bias terms are set to 0, thus not updating parameters.

[0012] Preferably, during the training of the entire network, both the input and output are narrow-band light intensity values.

[0013] Preferably, during the full network training process, the loss function is determined as the mean square error between the reconstructed narrowband light intensity value and the input narrowband light intensity value.

[0014] As a preferred option, the training sets used in both the pre-prediction network training process and the full network training process are from CAVE, ICVL Multispectral Image Database.

[0015] As a preferred option, the training sets used in the pre-prediction network training process and the full network training process are both composed of light intensity data from CAVE and ICVL Multispectral Image Database and several randomly synthesized Gaussian superposition curves.

[0016] A modeling method for narrow-band spectral imaging algorithms based on neural networks includes the following steps:

[0017] (1) Determine the spectral interval Δλ1 of each narrow band channel of the target, the number of imaging spectral channels (number of narrow band light intensity channels) M, and the number of coded filters N.

[0018] (2) Construct a light intensity to spectrum prediction network; the number of input units of the prediction network is the number of narrow band channels L, the number of output units is the number of wide band spectral channels M, and several hidden layers are set in the middle so that the entire prediction network can realize the function of narrow band light intensity to spectrum reconstruction.

[0019] (3) Train the light intensity to spectrum prediction network, select a narrow band light intensity dataset for training, so as to achieve the function of predicting the output spectrum based on the input narrow band light intensity.

[0020] (4) Next, based on the prediction network, the light intensity sensing network is constructed. The first layer added after the prediction network is a fully connected layer (i.e., the encoding layer). The connection weight between the i-th (i = 1, 2, ..., M) input neuron and the j-th (j = 1, 2, ..., N) output neuron in the encoding layer is W. ij , representing the spectral response of the j-th coded filter in the i-th spectral channel; several hidden layers are set after the coding layer to further process the output value of the coding layer; the number of output neurons is the number of narrow band channels L, so that the entire network can realize functions such as reconstructing the intensity of narrow band light.

[0021] (5) Train the entire neural network; set the structural parameters of the preceding prediction network section to not be updated during training. Set the weights W of the encoding layer... ij (i = 1, 2, ..., M; j = 1, 2, ..., N) is set as the actual transmittance of the manufactured coded filter combined with the detector response, and the bias term b is... j(j = 1, 2, ..., N) are all set to 0 to prevent parameter updates, and a narrow-band light intensity dataset is selected for training. The entire network is trained, and the network parameters are continuously updated.

[0022] (6) After training is completed, the entire network is truncated from the coding layer to obtain the truncated network. The N encoded light intensity values ​​obtained by inputting the experiment can be used to reconstruct L narrow band light intensity values.

[0023] The truncated network described above is the network we are actually using.

[0024] Furthermore, in step (1), the spectral spacing Δλ1 of each narrow-band channel and the calculation formula for the number of wide-band spectral channels M are as follows:

[0025] Δλ1=(λ max -λ min ) / L, M=(λ max -λ min ) / Δλ2+1

[0026] Where, λ min , λ max λ1 and λ2 are the minimum and maximum wavelengths of the target spectral band, respectively; L is the number of narrow-band light intensity channels; and Δλ2 is the wide-band spectral resolution.

[0027] Furthermore, in steps (2) and (4), the structure and unit parameters of the hidden layer are determined according to the actual scene to realize the reconstruction of object light intensity images of different narrow bands; the loss function used to train the prediction network and the reconstruction network is determined according to the corresponding function.

[0028] In the modeling process of this invention, the entire neural network of the algorithm consists of a pre-prediction network and a light intensity sensing network for practical application. The pre-prediction network is used to predict the light intensity from a narrow band to a wide band spectrum. The physical meaning of the number of input neurons is the number of narrow band light intensity channels, and the physical meaning of the output neurons is the number of wide band spectrum channels. There are several hidden layers in between. The light intensity sensing network for practical application is placed after the pre-prediction network to reconstruct the light intensity from the full band to the narrow band. It consists of a fully connected layer (i.e., the coding layer) and several hidden layers. The fully connected layer is used to simulate the modulation process of the spectrum by the coding filter, and the physical meaning of the output neurons is the number of narrow band light intensity channels. By training the pre-prediction network and the complete neural network of the entire algorithm (prediction network-light intensity sensing network) separately, the reconstruction of the light intensity from the narrow band to the wide band spectrum and the joint optimization of the light intensity from the narrow band to the narrow band can be achieved respectively. After training the entire neural network, the imaging module is used to image the target encoded by the thin-film filter. Utilizing the parallel data processing capabilities of the GPU platform, real-time reconstruction of images across various narrow bands can be achieved. Because the training process of the neural network clearly defines the training objective, it is easier to evaluate the optimality of the optimization results. Furthermore, the entire process directly trains on the light intensity of narrow bands, improving design accuracy. This invention is expected to have significant applications in fields such as material identification, agricultural product monitoring, and medical diagnosis.

[0029] Compared with traditional narrow-band spectral imaging techniques based on dispersive elements (gratings, prisms, and narrowband tunable filters), this invention has the following beneficial technical effects:

[0030] (1) It can significantly improve the speed of acquiring narrow-band light intensity images. Narrow-band spectral imaging based on dispersive elements often relies on multiple narrow-band filters for beam splitting, which results in a slow scanning speed. However, by utilizing the parallel data processing capabilities of deep learning, the acquisition and reconstruction process of each narrow-band image can be shortened to the millisecond level, achieving snapshot imaging in the narrow band.

[0031] (2) The optimality of the reconstructed narrow-band light intensity can be quantitatively evaluated. Since the training process has a definite loss function as the optimization objective, the optimality of the reconstructed narrow-band light intensity curve can be quantitatively evaluated, which solves the problem that the narrow-band light intensity response obtained by the traditional design method cannot be quantitatively evaluated.

[0032] (3) It can improve the signal-to-noise ratio of the narrow-band light intensity. This invention introduces a deep learning method, using a smaller number of broadband filter blocks, and uses a neural network to reconstruct a larger number of narrow-band light intensities with high precision. Compared with the traditional method of using a larger number of narrow-band filters to detect light intensity signals separately, it improves the light throughput and thus the signal-to-noise ratio.

[0033] (4) The design results are more targeted. Since the actual responsivity of the detector is different at the corresponding wavelength, the present invention takes into account the magnitude of the filter spectral transmittance S(λ) and the actual responsivity of the detector D(λ) during neural network training, which can better realize the collaborative design of hardware and software.

[0034] (5) It can significantly reduce the manufacturing cost of narrow-band spectral imaging equipment. The present invention uses a smaller number of thin-film encoded N light intensity values ​​to encode and modulate the target, reducing the size of the equipment and thus ensuring the repeatability of the imaging equipment processing and mass production. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the neural network structure of the narrow-band spectral imaging method based on neural networks of the present invention;

[0036] Figure 2 This is a flowchart of the narrow-band spectral imaging method based on neural networks of the present invention;

[0037] Figure 3 This is a schematic diagram of the workflow for conventional computational narrowband spectral imaging.

[0038] Figure 4 This is a schematic diagram of the narrow-band spectral imaging process of the present invention;

[0039] Figures 5(a), (b), (c), and (d) show the full-band light intensity information of some test sets after processing. Figure 3 and Figure 4 Comparison of narrow-band light intensity errors reconstructed by two narrow-band spectral imaging methods (after normalization);

[0040] Figure 6 This is a schematic diagram of the narrow-band spectral imaging system of the present invention;

[0041] Figure 7 The narrow-band light intensity image is reconstructed from an actual standard color chart using the neural network of this invention. Detailed Implementation

[0042] The present invention will now be described in detail with reference to the embodiments and accompanying drawings, but the present invention is not limited thereto.

[0043] like Figure 1As shown, during the model training phase, the entire algorithm's neural network consists of a pre-prediction network and a light intensity sensing network for practical application. The pre-prediction network is used to predict the light intensity from a narrow band to a wide band spectrum. The physical meaning of the number of input neurons is the number of narrow band light intensity channels L, and the physical meaning of the output neurons is the number of wide band spectrum channels M. Several hidden layers are connected in between. During training, the pre-prediction network is trained separately first. After training, the light intensity sensing network for practical application is placed after the pre-prediction network to reconstruct the light intensity from the full band to the narrow band. It consists of a fully connected layer (i.e., the coding layer) and several hidden layers. The coding layer simulates the modulation process of the spectrum by the coding filter, and the physical meaning of the number of output neurons is the number of narrow band light intensity channels L. By training the pre-prediction network and the complete neural network of the entire algorithm (prediction network-light intensity sensing network) separately, the reconstruction of the light intensity from the narrow band to the wide band spectrum and the joint optimization of the light intensity from the narrow band to the narrow band can be achieved respectively.

[0044] The modeling steps of the implemented neural network-based narrow-band spectral imaging method are as follows: Figure 2 As shown, it includes:

[0045] (1) Determine the spectral range λ of the target min , λ max And given the number of light intensity channels L and the spectral resolution Δλ², then λ min , λ max The spectral spacing Δλ1=(λ) between narrow band channels max -λ min ) / L, number of spectral channels M=(λ max -λ min ) / Δλ2+1. The number of spectral coding filters is determined to be N.

[0046] (2) Construct a neural network, which is called the (front-end) prediction network. The number of input units (input neurons) in this network is equal to the number of narrow-band light intensity channels L, and the number of output units (output neurons) is equal to the number of wide-band spectral channels M. Several hidden layers are set in between. A suitable training set is selected to train the prediction network to achieve the function of predicting the spectral response of the output based on the input narrow-band light intensity.

[0047] (3) Next, based on the prediction network, the light intensity sensing network is constructed. The first layer added after the prediction network is a fully connected layer (i.e., the encoding layer). The weight W between the i-th (i = 1, 2, ..., M) input neuron and the j-th (j = 1, 2, ..., N) output neuron of the encoding layer is... ijThis represents the spectral response of the j-th coded filter in the i-th spectral channel; several hidden layers are set after the coding layer. The number of output units is the number of narrow-band light intensities L. The structure and number of units in each hidden layer can be adjusted according to specific application scenarios to obtain a light intensity sensing network that meets the requirements, enabling the reconstruction, analysis, and other processing functions of narrow-band light intensity. The loss function is determined according to the corresponding functions.

[0048] (4) Train the entire neural network; set the structural parameters of the previous prediction network to not be updated during training, and set the weights W of the encoding layer to be... ij (i = 1, 2, ..., M; j = 1, 2, ..., N) is set to the actual transmittance of the fabricated coded filter combined with the detector response, and the bias term b is... j (j = 1, 2, ..., N) are all set to 0 to prevent parameter updates, and the light intensity dataset is selected for training. The entire network is trained, and the network parameters are continuously updated.

[0049] (5) After training is completed, the entire network is truncated from the coding layer to obtain the truncated network.

[0050] In practical applications, by using a truncated network, L narrow-band light intensity values ​​can be reconstructed from the N encoded light intensity values ​​obtained from the input experiment.

[0051] The typical computational narrow-band spectral imaging workflow is as follows: Figure 3 As shown, assuming the spectrum of the target object is S(λ), and the transmittance of the N coded filters is W1(λ), ​​W2(λ), ..., W... N (λ), λ min To λ max This refers to the spectral range that the spectral detection equipment can detect. Based on... Figure 3 In relation 1, after modulation by the j-th (j = 1, 2, ..., N) coded filter, the detector obtains the intensity value I of the spectrum at that point. j It can be represented as:

[0052]

[0053] In the formula, S(λ) is the reflectance spectrum of the target object, and W... j (λ) represents the transmittance of the coded filter.

[0054] We can use deep learning, compressed sensing, and other methods to reconstruct the object's spectrum S'(λ) based on the light intensity obtained from the detector. Figure 3 Relation 2, considering that the detector's responsivity at various wavelengths is not always 1, the transmittance of each narrow-band filter combined with the detector's response is W'1(λ), ​​W'2(λ), ..., W' t (λ), the intensity value I' of each narrow bandt It can be represented as:

[0055]

[0056] This method is rather cumbersome, and obtaining the narrow-band light intensity by summing or integrating the reconstructed spectrum undoubtedly increases the reconstruction error. Therefore, we can utilize deep neural networks and, by selecting appropriate loss functions (such as MAE, MSE, and RMSE), directly train the relationship between the full-band light intensity and the narrow-band light intensity. The specific process is as follows: Figure 4 As shown.

[0057] This method not only enables the reconstruction of light intensities across different narrow bands but also reduces reconstruction errors while taking into account the influence of hardware factors such as detectors. By inputting the intensity values ​​of the light reflected from the object, modulated by each coded filter, into a trained neural network (truncated network), images of various narrow bands can be obtained, enabling wide applications in fields such as material identification, agricultural product monitoring, and medical diagnosis.

[0058] Example

[0059] The proposed reconstruction method for a narrowband optical intensity network based on a broadband coded filter is as follows:

[0060] (1) Determine the spectral range λ of the target min =400nm, λ max =700nm, and given the number of light intensity channels L=30 and the spectral resolution Δλ2=1nm, then λ min , λ max The spectral spacing Δλ1=(λ) between narrow band channels max -λ min ) / L=10nm, number of spectral channels M=(λ max -λ min ) / Δλ2+1=301. The spectral coding filter is determined to be an optical thin-film filter, with a number of N=16.

[0061] (2) Construct a neural network, referred to as the prediction network. The network has 30 input units (narrow-band light intensity channels) and 301 output units (wide-band spectral channels), with two hidden layers. Batch normalization is applied to neurons in each layer. The light intensity dataset used for training consists of 300,000 CAVE images, an ICVL Multispectral Image Database, and 100,000 randomly synthesized Guass curves. Train the prediction network to predict the spectral response based on narrow-band light intensity.

[0062] (3) Next, construct the prediction network-light intensity sensing network. Based on the prediction network, the first layer added is a fully connected layer (i.e., the encoding layer). The weight W between the i-th (i = 1, 2, ..., 300) input unit and the j-th (j = 1, 2, ..., 16) output unit of the encoding layer is... ij This represents the spectral response value of the j-th coded filter in the i-th spectral channel. Two hidden layers are set after the coding layer, and the neurons in each layer also undergo batch normalization to further nonlinearly process the output value of the coding layer. The entire network outputs a narrow-band light intensity curve. Its loss function is determined as the mean square error between the reconstructed narrow-band light intensity value (reconstructed value) and the input narrow-band light intensity value (true value).

[0063] (4) Train the entire neural network; set the structural parameters of the preceding prediction network part to not be updated during training, and set the weights W of the hardware layer (coding layer) to be... ij (i = 1, 2, ..., 300; j = 1, 2, ..., 16) is set as the actual transmittance of the fabricated coded filter combined with the detector response, and the bias term b is the actual transmittance. j (j = 1, 2, ..., 16) are all set to 0 to prevent parameter updates. The light intensity dataset selected for training consists of 300,000 CAVE images, an ICVL Multispectral ImageDatabase, and 100,000 randomly synthesized Guass curves. The entire network is trained, and the network parameters are continuously updated.

[0064] (5) After training, the entire network is truncated from the coding layer (the network nodes of the coding layer are retained) to obtain the truncated network, namely the narrow band light intensity network. The 16 encoded light intensity values ​​obtained by inputting the experiment can be used to reconstruct 30 narrow band light intensity values.

[0065] To verify the accuracy of the obtained truncated network, test sets with different spectral shapes were selected to test the reconstruction error of the narrow-band intensity network (GR network) obtained in this invention. The error was compared with that of the existing intensity data obtained by reconstructing the spectrum using a spectral sensing network (SED network) (Deeplylearned broadband encoding stochastic hyperspectral imaging, Wenyi Zhang, Hongya Song, Xin He, Longqian Huang, Xiyue Zhang, Junyan Zheng, Weidong Shen, Xiang Hao and Xu Liu, Official journal of the CIOMP 2047-7538) (all curves have been normalized). It was found that the reconstruction error was reduced by an order of magnitude, and the error for various spectral shapes was low, showing strong universality. The specific results are shown in Figures 5(a), (b), (c), and (d).

[0066] Based on the truncated network obtained in the previous step, the parallel data processing capability of deep learning can be used to reconstruct a narrow-band light intensity image. The specific optical system we built is as follows: Figure 6 As shown, the spectrum of the target object is modulated by a thin-film filter and then imaged onto the detector by the focusing effect of the lens. We can use the previously obtained truncated network to reconstruct the narrow-band spectral data cube of the object. Figure 7 This is a partial reconstruction result of the narrow-band light intensity image of the actual color card using this system and neural network.

[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A narrow-band spectral imaging method based on neural networks, characterized in that, include: The target object is encoded by a broadband coded filter array, imaged, and detected by a detector to obtain the corresponding full-band light intensity information. Then, the obtained full-band light intensity information is input into the constructed truncated network to obtain images of each narrow band. The truncated network is trained from the full network, which consists of a pre-prediction network, a truncated network, and a fully connected layer connecting the two. The pre-prediction network adopts a neural network structure. The input is narrow-band light intensity, and the output is wide-band spectrum. The number of input neurons is the number of narrow-band channels, and the number of output neurons is the number of wide-band spectrum channels. Several hidden layers are set in the middle. The truncated network adopts a neural network structure, with the input being the full-band light intensity and the output being the narrow-band light intensity. The number of input neurons is the number of coded filters, and the number of output neurons is the number of narrow-band light intensity channels. Several hidden layers are set in the middle. In the fully connected layer, the connection weight between the i-th input neuron and the j-th output neuron is W. ij , represents the spectral response of the j-th coded filter in the i-th spectral channel, i=1,2,…,M, j=1,2,…,N, M is the number of broadband spectral channels, and N is the number of coded filters; During the training of the entire network, the pre-prediction network is set to not update parameters; the weights of the fully connected layers are set to encode the actual transmittance of the filter; all bias terms are set to 0, and parameters are not updated; after training is completed, the entire network is truncated after the fully connected layers to obtain the truncated network.

2. The narrow-band spectral imaging method based on neural networks according to claim 1, characterized in that, The pre-prediction network is trained separately using narrow-band light intensity and the corresponding wide-band spectrum.

3. The narrow-band spectral imaging method based on neural networks according to claim 1, characterized in that, During the training of the entire network, both the input and output are narrow-band light intensity values.

4. The narrow-band spectral imaging method based on neural networks according to claim 1, characterized in that, During the training of the entire network, the loss function is determined to be the mean square error between the reconstructed narrowband light intensity value and the input narrowband light intensity value.

5. The narrow-band spectral imaging method based on neural networks according to claim 2, characterized in that, The training sets used in both the pre-prediction network training process and the full network training process are from CAVE, ICVL Multispectral Image Database.

6. The narrow-band spectral imaging method based on neural networks according to claim 2, characterized in that, The training sets used in the pre-prediction network training process and the full network training process are both composed of light intensity data from CAVE and ICVL Multispectral Image Database and several randomly synthesized Gaussian superposition curves.

Images