Spectral image reconstruction method, spectral image reconstruction apparatus, and electronic device

Abstract

The present application relates to a kind of spectral image reconstruction method, device and electronic equipment.The spectral image reconstruction method includes: obtaining the transmission spectrum data of spectral imaging chip and the output signal data of spectral imaging chip;Based on the pixel for reconstructing spectral image, obtain the local transmission spectrum data of the transmission spectrum data and the local output signal data of the output signal data;The local output signal data is input into attention model to obtain attention local data;And the local transmission spectrum data, the local output signal data and the attention local data are input into neural network model to obtain the pixel for reconstructing spectral image.It is thus realized to reconstruct high-quality spectral image at low cost.

Description

Technical Field

[0001] This application relates to the field of spectral chip technology, and more specifically, to a spectral image reconstruction method, a spectral image reconstruction device, and an electronic device. Background Technology

[0002] Spectral imaging technology is a technique that organically combines spectral detection and imaging. It can image an object under different spectra, simultaneously obtaining the geometric shape information and spectral characteristics of the object being detected. Spectral imaging technology has become an important tool for Earth observation and deep space exploration, and is widely used in fields such as agricultural and forestry production, mineral resource exploration, cultural relic detection, marine remote sensing, environmental monitoring, disaster prevention and mitigation, and military reconnaissance.

[0003] Typically, hyperspectral image reconstruction methods utilize transmission spectrum data from calibrated spectral imaging chips to calculate a reconstruction function, which is then used to reconstruct each captured hyperspectral image. While this method is highly efficient during reconstruction, obtaining the reconstruction function requires significant computational resources for each snapshot-type spectral imaging chip. These computational resources account for a considerable portion of production costs.

[0004] Therefore, there is a need to provide an improved method for spectral image reconstruction. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a spectral image reconstruction method, a spectral image reconstruction apparatus, and an electronic device, which directly outputs a spectral image from a reconstruction function and a response signal by training a deep learning network, thereby achieving high-quality spectral image reconstruction at low cost.

[0006] According to one aspect of this application, a spectral image reconstruction method is provided, comprising: acquiring transmission spectrum data and output signal data of a spectral imaging chip; acquiring local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on pixels for reconstructing the spectral image; inputting the local output signal data into an attention model to obtain attention local data; and inputting the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels for reconstructing the spectral image.

[0007] In the above-described spectral restoration method, obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on the pixels used to reconstruct the spectral image includes: obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data with a side length of a predetermined number of pixels in the region near the pixel used to reconstruct the spectral image.

[0008] In the above-described spectral restoration method, inputting the local output signal data into an attention model to obtain attention local data includes: dividing the local output signal data into multiple predetermined regions, each predetermined region including output signal data corresponding to multiple pixels of the spectral imaging chip; and performing matrix multiplication for each predetermined region to obtain the attention local data.

[0009] In the above spectral restoration method, inputting the local output signal data into the attention model to obtain attention local data includes: normalizing the transmission spectrum data and the output signal data to... Within the range.

[0010] In the above-described spectral restoration method, inputting the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels for reconstructing the spectral image includes: concatenating the local transmission spectrum data, the local output signal data, and the attention local data at each spectral resolution in the local transmission spectrum data in the thickness dimension to obtain a cascaded tensor; concatenating multiple cascaded tensors corresponding to multiple spectral resolutions of the local transmission spectrum data in the thickness dimension to obtain an input tensor; inputting the input tensor into a 3D convolutional neural network to obtain an output feature map, wherein the thickness dimension of the output feature map is the same as the number of channels of the spectral image to be reconstructed; and inputting each feature map of the output feature map in the thickness dimension into a fully connected network to obtain the pixels for reconstructing the spectral image.

[0011] In the above-mentioned spectral restoration method, the input and output data of the 3D convolutional neural network both include four dimensions: data length, data width, data thickness, and number of channels. The data thickness is equal to the number of iterations of the 3D convolutional neural network, and the number of channels of the input data is one.

[0012] In the above-described spectral restoration method, the 3D convolutional neural network includes a convolutional layer, an activation layer, and a pooling layer; the first convolutional layer has... The step size, where The number of regions in the attention model is the square root of the number of regions, and the remaining convolutional layers have a stride of 1; the first convolutional layer uses the Sigmoid activation function, and the remaining layers use the Leaky ReLU activation function; the pooling layers are all max pooling with a length and width of 2 and a thickness that is variable; and the number of channels in each convolutional network layer is twice the number of channels in the layer above it.

[0013] In the above-described spectral restoration method, the training data for the 3D convolutional neural network and the fully connected network includes at least one of the following: the transmission spectrum data and the output signal data obtained based on still hyperspectral photographs acquired using high-precision spectral cameras with different modulation layers; the output signal data obtained by simulating imaging of hyperspectral images in a publicly available hyperspectral dataset using multiple transmission spectrum data from a product database; and the transmission spectrum data and output signal data obtained by adding noise to the above-described transmission spectrum data and output signal data.

[0014] In the above-mentioned spectral restoration method, the training process of the 3D convolutional neural network and the fully connected network includes: splitting the entire image into small blocks with a predetermined side length using random center coordinates; hashing and reordering the ground truth spectral values ​​corresponding to the small blocks; and inputting the shuffled small blocks and their corresponding spectra into a training package for model training.

[0015] In the above-described spectral restoration method, inputting the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels for reconstructing the spectral image includes: cascading the local transmission spectrum data, the local output signal data, and the attention local data to obtain input data; and inputting the input data into a fully convolutional network to obtain the pixels for reconstructing the spectral image.

[0016] In the above-described spectral restoration method, the fully convolutional network includes a predetermined number of downsampling convolutional neural networks and the predetermined number of upsampling deconvolutional neural networks; each convolutional layer in the downsampling convolutional neural network has double max pooling and its number of channels is twice the number of channels in the previous convolutional layer; each convolutional layer in the upsampling deconvolutional neural network has double unpooling and its number of channels is half the number of channels in the previous convolutional layer; the output tensor of each convolutional layer in the downsampling convolutional neural network is concatenated with the output tensor of each convolutional layer in the upsampling deconvolutional neural network; and the number of channels in the output layer of the fully convolutional network is the number of channels in the spectral image to be reconstructed.

[0017] In the above-mentioned spectral restoration method, the training process of the fully convolutional network includes: obtaining an RGB three-channel pre-trained model; freezing all pre-trained parameters and performing preliminary training on the input and output layers; and gradually and symmetrically unfreezing the pre-trained parameters and reducing the learning rate to achieve training of the entire fully convolutional network.

[0018] According to another aspect of this application, a spectral image reconstruction apparatus is provided, comprising: a data acquisition unit for acquiring transmission spectrum data and output signal data of a spectral imaging chip; a data partitioning unit for acquiring local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on pixels for reconstructing a spectral image; an attention unit for inputting the local output signal data into an attention model to obtain attention local data; and a neural network unit for inputting the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels for reconstructing the spectral image.

[0019] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions that, when the processor is executed, cause the processor to perform the spectral image reconstruction method as described above.

[0020] According to another aspect of this application, a computer-readable storage medium is provided, wherein computer program instructions are stored thereon, which, when executed by a computing device, are operable to perform the spectral image reconstruction method as described above.

[0021] The spectral image reconstruction method, spectral image reconstruction apparatus, and electronic device provided in this application can directly output spectral images from reconstruction functions and response signals by training a deep learning network, thereby achieving the reconstruction of high-quality spectral images at low cost. Attached Figure Description

[0022] Various other advantages and benefits of this application will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0023] Figure 1 The illustration shows a schematic configuration diagram of a spectral imaging device according to an embodiment of this application; Figure 2 A schematic flowchart of a spectral image reconstruction method according to an embodiment of this application is shown; Figure 3 The illustration shows a schematic diagram of the implementation of the attention model in the spectral image reconstruction method according to an embodiment of this application; Figure 4The illustration shows a schematic diagram of a first example of a neural network model in a spectral image reconstruction method according to an embodiment of this application; Figure 5 The illustration shows a schematic diagram of a second example of a neural network model in a spectral image reconstruction method according to an embodiment of this application; Figure 6 The figure shows a schematic block diagram of a spectral image reconstruction apparatus according to an embodiment of the present application; Figure 7 A schematic block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0024] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0025] Application Overview The spectral image reconstruction method according to the embodiments of this application is applied to a spectral imaging device. Figure 1 The illustration shows a schematic configuration diagram of a spectral imaging device according to an embodiment of this application. Figure 1 As shown, in the spectral imaging device according to the embodiments of this application, the optical system is optional, and may be an optical system such as a lens assembly or a homogenizing assembly. The filter structure is a broadband filter structure in the frequency domain or wavelength domain. The pass spectra of different wavelengths of the filter structure are not completely the same at different locations. The filter structure can be a metasurface, photonic crystal, nanopillar, multilayer film, dye, quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties. For example, in the embodiments of this application, the filter structure can be the light modulation layer in Chinese Patent CN201921223201.2. The image sensor (i.e., photodetector array) can be a CMOS image sensor (CIS), CCD, array photodetector, etc. Additionally, optional data processing units can be MCUs, CPUs, GPUs, FPGAs, NPUs, ASICs, etc., which can export the data generated by the image sensor for external processing.

[0026] For example, after the image sensor measures the light intensity information, it is transmitted to the data processing unit for reconstruction calculation. This process is described in detail below: Incident light at different wavelengths λ The intensity signal below is denoted as x(λ) The transmission spectrum curve of the filter structure is denoted as T(λ)The filter (filter structure) has m The filter structure consists of several structural units, each with a different transmission spectrum. Overall, the filter structure can be denoted as... T i (λ) (i=1,2,3,…,m) Each group of structural units has a corresponding physical pixel below it, which detects the light intensity bi modulated by the filtered light structure. In a specific embodiment of this application, one physical pixel corresponds to one group of structural units, but this is not a limitation. In other embodiments, multiple physical pixels may be grouped together to correspond to one group of structural units. Therefore, in the spectral imaging device according to the embodiments of this application, multiple groups of structural units constitute a "spectral pixel". Furthermore, the present invention can use at least one spectral pixel to reconstruct an image. It should be noted that the effective transmission spectrum of the filter structure (the transmission spectrum used for spectral reconstruction is called the effective transmission spectrum) is... T i (λ) The number of structural units may not be the same as the number of structural units. The transmission spectrum of the filter structure is set, tested, or calculated manually according to certain rules based on the needs of identification or recovery (for example, the transmission spectrum obtained by each structural unit through testing is the effective transmission spectrum). Therefore, the number of effective transmission spectra of the filter structure may be less than the number of structural units, or even more than the number of structural units. In this modified embodiment, a certain transmission spectrum curve is not necessarily determined by a group of structural units.

[0027] The relationship between the spectral distribution of incident light and the measurements from the image sensor can be expressed by the following formula: b i =∫x(λ)*T i (λ)*R(λ)dλ After discretization, we get: b i =Σ(x(λ)*T i (λ)*R(λ)) Where R(λ) is the response of the image sensor, denoted as: A i (λ)=T i (λ)*R(λ), The above equation can then be extended into matrix form: in, b i (i=1,2,3,…,m) These are the responses of the image sensor after the light under test passes through the filter structure, respectively corresponding to...m The light intensity measurement value of the image sensor corresponding to each structural unit. When one physical pixel corresponds to one structural unit, it can be understood as: m The light intensity measurement value corresponding to each "physical pixel" is a value of length . m The vector. A The system's response to light of different wavelengths is determined by two factors: the transmittance of the filter structure and the quantum efficiency of the image sensor. A It is a matrix, where each row vector corresponds to a set of structural units responding to incident light of different wavelengths. Here, the incident light is sampled discretely and uniformly, resulting in a total of [number missing]. n One sampling point. A The number of columns is the same as the number of sampling points for the incident light. Here, x(λ) That is, incident light at different wavelengths λ The light intensity is the incident light spectrum that is to be measured.

[0028] In some embodiments, unlike the above embodiments, the filter structure can be directly formed on the upper surface of the image sensor, such as quantum dots, nanowires, etc., which directly form the filter structure or material (nanowires, quantum dots, etc.) in the photosensitive area of ​​the sensor. Taking the filter structure as an example, it can be understood that when the raw materials of the image sensor are processed to form the image sensor, the filter structure is formed on the upper surface of the raw materials. The transmission spectrum and the response of the image sensor are integrated, that is, it can be understood that the response of the detector and the transmission spectrum are the same curve. At this time, the relationship between the spectral distribution of the incident light and the light intensity measurement value of the image sensor can be expressed by the following formula: b i =Σ(x(λ) *R i (λ)) That is, in this embodiment, the transmission spectrum A i (λ)= R i (λ) Furthermore, it can also be a combination of the two embodiments described above, that is, at least one filter structure for modulating incident light is provided on the image sensor with the filter structure. It can be understood that the image sensor (i.e., photodetector array) in the first embodiment, which can be a CMOS image sensor (CIS), CCD, array photodetector, etc., can be replaced with an image sensor with an integrated filter structure in the second embodiment.

[0029] At this point, the relationship between the spectral distribution of the incident light and the light intensity measurement value of the image sensor can be expressed by the following formula: b i =∫x(λ)*T i (λ)*R i (λ)dλ After discretization, we get: b i =Σ(x(λ)*T i (λ)*R i (λ)) That is, in this embodiment, A i (λ)=T i (λ)*R i (λ) Exemplary methods Figure 2 A schematic flowchart of a spectral image reconstruction method according to an embodiment of this application is shown.

[0030] like Figure 2 As shown, the spectral image reconstruction method according to an embodiment of this application includes the following steps.

[0031] Step S110: Obtain the transmission spectrum data and output signal data of the spectral imaging chip.

[0032] Specifically, in the embodiments of this application, the transmission spectrum data of the spectral imaging chip is a third-order transmission spectrum tensor, for example, represented as... Its shape is , representing the calibrated spectral resolution, width resolution, and height resolution of the image, respectively. The transmission spectrum data A of the spectral imaging chip can be obtained through calibration, calculation, or other methods. Furthermore, during spectral imaging, the output signal data of the spectral imaging chip, such as light intensity data, is a matrix, for example, represented as... Its shape is Furthermore, if the spectral image to be reconstructed is represented as Then we have: Therefore, the purpose of the spectral image reconstruction method according to the embodiments of this application is to reconstruct a spectral image from the transmission spectrum data and output data of the spectral imaging chip.

[0033] Step S120: Based on the pixels used to reconstruct the spectral image, obtain the local transmission spectrum data of the transmission spectrum data and the local output signal data of the output signal data.

[0034] That is, in the embodiments of this application, the reconstruction of the spectral image is performed pixel by pixel, and the recovered pixels are... When this is done, it is necessary to take the area in the vicinity with a side length of a predetermined number of pixels, for example... Transmission spectral tensor of each pixel and signal data matrix The local part is denoted as and Here, in the embodiments of this application, where it is not necessary to particularly emphasize the pixel position, it is abbreviated as... and Therefore, in this embodiment, the region used to reconstruct the spectrum of a single pixel can also be called a "reconstruction kernel." That is, the model for reconstructing spectral images proposed in this embodiment is built based on a reconstruction kernel, and each reconstruction kernel reconstructs the spectrum of only one pixel. To recover the entire spectral image, the reconstruction kernel is applied to the entire image (filled with padding values ​​if necessary).

[0035] In this way, since each multispectral image training data can be split into multiple pixels for training the model, it is suitable for situations where the training dataset is small.

[0036] Therefore, in the spectral image reconstruction method according to the embodiments of this application, obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on the pixels used to reconstruct the spectral image includes: obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data with a side length of a predetermined number of pixels in the region near the pixel used to reconstruct the spectral image.

[0037] Step S130: Input the local output signal data into the attention model to obtain local attention data. Here, the local output signal data, as described above... Before inputting the neural network model, an attention model is first input to enhance the subsequent model's ability to identify and filter boundaries. Furthermore, considering the high computational demands of the attention model, it is preferable to further input... Attention calculation is performed by region, as follows: In the formula The number of edges in the segment. , The ontology coordinates of the attention kernel , The coordinates corresponding to the background of attention. .here, Figure 3 The figure illustrates a schematic diagram of the implementation of the attention model in the spectral image reconstruction method according to an embodiment of this application.

[0038] like Figure 3 As shown, although The side length might originally be tens of pixels, but it was divided into smaller segments before attention calculation. The region is limited, therefore the calculated attention tensor only has 9 layers. Because it requires less computational power and storage, this attention model can also be called a mini-attention model. It is worth noting that the segmentation of the mini-attention model is not limited to... The layout can be modified according to the actual situation. , , wait.

[0039] Therefore, in the spectral image reconstruction method according to the embodiments of this application, inputting the local output signal data into an attention model to obtain attention local data includes: dividing the local output signal data into multiple predetermined regions, each predetermined region including output signal data corresponding to multiple pixels of the spectral imaging chip; and performing matrix multiplication for each predetermined region to obtain the attention local data.

[0040] Furthermore, before inputting the local output signal data into the attention model, the local output signal data, i.e., the matrix described above, can be first processed. Normalization to Within the interval. Furthermore, for the transmission spectrum data, i.e., the third-order tensor as described above. It also normalizes it to Within the range.

[0041] Therefore, in the spectral image reconstruction method according to the embodiments of this application, inputting the local output signal data into the attention model to obtain attention local data includes: normalizing the transmission spectrum data and the output signal data to... Within the range.

[0042] Step S140: Input the local transmission spectrum data, the local output signal data, and the local attention data into the neural network model to obtain the pixels used for reconstructing the spectral image. Here, in this embodiment, depending on the training dataset, different neural network models can be used to reconstruct the spectral image.

[0043] Figure 4 The illustration shows a schematic diagram of a first example of a neural network model in a spectral image reconstruction method according to an embodiment of this application.

[0044] like Figure 4As shown, after obtaining the attention-based local data processed using a simplified attention model, it can be combined with the local transmission spectrum data to perform spectral reconstruction using a 3D convolutional neural network, as represented by: in, The 3D convolutional neural network includes convolutional layers, activation layers, and pooling layers. This represents the number of iterations in the 3D convolutional neural network. The input data of the 3D convolutional neural network has four orders: ,in For the length and width of the data (for example, as mentioned above, when taking...) and At the same time, keep its length and width as ); Represents the thickness of the data. The number of channels representing the data. All data is stitched together at the thickness level, and the input data has only one channel level. The output of the 3D convolutional depth network... For shape ,in The number of channels must match the required number of spectral images so that subsequent FC networks can perform independent reconstruction.

[0045] Specifically, the parameters of the 3D convolutional neural network according to the embodiments of this application can be as follows. In the first layer of the 3D convolutional network, the following parameters are applied: The stride is set to a minimum to avoid unnecessary computations between different attention layers and simplify the model; a stride of 1 is used in all other convolutional layers. The first convolutional layer uses the Sigmoid activation function, while the remaining layers use the Leaky ReLU activation function. All pooling layers are max pooling layers with a stride of 2 in both width and height, and a thickness that is variable (depending on the number of spectral channels required for the final output). Furthermore, as the network deepens, each convolutional layer can double the number of channels in the channel dimension compared to the previous layer. During training, the 3D convolutional neural network iterates until its thickness matches the required spectral dimension before stopping.

[0046] Furthermore, in the aforementioned 3D convolutional neural network, the thickness order information is effectively preserved, and the thickness order corresponds to the various frequency bands in the input. Therefore, a unified fully connected (FC) network can be further used to reconstruct the spectrum, and the reconstruction method is as follows: in, It is a fully connected network, which can consist of one or more layers, the number of which can vary depending on different requirements. Since the spectral signal reconstruction in each frequency band uses a unified FC network, it can be called a unified FC reconstruction network.

[0047] Therefore, in the spectral image reconstruction method according to the embodiments of this application, inputting the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels for reconstructing the spectral image includes: concatenating the local transmission spectrum data, the local output signal data, and the attention local data at each spectral resolution in the local transmission spectrum data in the thickness dimension to obtain a cascaded tensor; concatenating multiple cascaded tensors corresponding to multiple spectral resolutions of the local transmission spectrum data in the thickness dimension to obtain an input tensor; and inputting the input tensor into a 3D convolutional neural network to obtain an output feature map, wherein the thickness dimension of the output feature map is the same as the number of channels of the spectral image to be reconstructed; and inputting each feature map of the output feature map in the thickness dimension into a fully connected network to obtain the pixels for reconstructing the spectral image.

[0048] Furthermore, in the above-mentioned spectral image reconstruction method, the input and output data of the 3D convolutional neural network both include four dimensions: data length, data width, data thickness, and number of channels. The data thickness is equal to the number of iterations of the 3D convolutional neural network, and the number of channels of the input data is one.

[0049] Furthermore, in the aforementioned spectral image reconstruction method, the 3D convolutional neural network includes convolutional layers, activation layers, and pooling layers; the first convolutional layer has... The step size, where The number of regions in the attention model is the square root of the number of regions, and the remaining convolutional layers have a stride of 1; the first convolutional layer uses the Sigmoid activation function, and the remaining layers use the Leaky ReLU activation function; the pooling layers are all max pooling with a length and width of 2 and a thickness that is variable; and the number of channels in each convolutional network layer is twice the number of channels in the layer above it.

[0050] Here, the first example of the neural network model is a deep learning-based neural network model, therefore a dataset is required for training. In this application embodiment, the methods for obtaining the dataset mainly include the following: 1. Use a high-precision spectrophotometer to acquire still life hyperspectral images as the true values. Under the same conditions and with the same photographic parameters, use spectrophotometers with different modulation layers to take pictures and obtain their transmission spectral tensors. and signal data matrix This method is difficult to obtain data for as input, but it is highly accurate and can be used as validation and test data.

[0051] 2. Download publicly available hyperspectral datasets and utilize multiple datasets from the product database. The signal data matrix is ​​obtained by simulating imaging of the hyperspectral images in the dataset. Then, the original hyperspectral image is used as the ground truth for training. This method can obtain a large amount of data, making it suitable for creating training datasets.

[0052] 3. The above two methods Add noise to perform data augmentation.

[0053] Furthermore, during the model training process, although the training dataset contains images, the aforementioned model should not be trained by directly inputting the entire image. This is because hyperspectral images exhibit strong spatial continuity; directly inputting them for training may lead to model convergence difficulties or overfitting. Therefore, the entire image needs to be divided into portions with random center coordinates and side lengths of [missing information]. small pieces and The true spectral values ​​corresponding to these small blocks are hashed and reordered. The shuffled small blocks and their corresponding spectra are then combined into a training batch and input into the model for training.

[0054] Therefore, in the spectral image reconstruction method according to the embodiments of this application, the training data of the 3D convolutional neural network and the fully connected network includes at least one of the following: the transmission spectrum data and the output signal data obtained based on still hyperspectral photographs acquired using high-precision spectral cameras with different modulation layers; the output signal data obtained by simulating imaging of hyperspectral images in a publicly available hyperspectral dataset using multiple transmission spectrum data from a product database; and the transmission spectrum data and output signal data obtained by adding noise to the above-mentioned transmission spectrum data and output signal data.

[0055] Furthermore, in the above-mentioned spectral image reconstruction method, during the training process of the 3D convolutional neural network and the fully connected network, the entire image is divided into small blocks with a predetermined side length using random center coordinates. The spectral ground values ​​corresponding to these small blocks are then hashed and reordered. The shuffled small blocks and their corresponding spectra are then combined to form a training package and input into the model for training.

[0056] In the aforementioned neural network model, since each multispectral image training data point can be split into multiple pixels for model training, it is suitable for situations with small training datasets. However, because this neural network model reconstructs only one pixel of the hyperspectral image per use, it is inefficient when reconstructing high-resolution hyperspectral images. When a large dataset is available and there is a need to reconstruct high-resolution hyperspectral images, the following neural network model can be used.

[0057] Figure 5 The illustration shows a second example of a neural network model in a spectral image reconstruction method according to an embodiment of this application.

[0058] Specifically, when the available training dataset is large, one can use methods such as Figure 5 The FCN (Fully Convolutional Networks) shown replaces the aforementioned 3D convolutional neural network, reconstructing the entire spectral image in one go.

[0059] It is worth noting that FCN is often used for semantic segmentation, but its ability to reconstruct details is relatively poor. Using only FCN can lead to distortion of details in the reconstructed hyperspectral image. Therefore, in this embodiment, an attention model is also required before the input to the FCN network. Considering the special nature of spectral restoration, the restoration of details should only refer to the information of the pixels near the target pixel. Therefore, the scope of attention is also controlled to be within the vicinity of each pixel. That is, each pixel uses the attention calculation method of the simplified attention model described above, and finally all attention layers are superimposed together to participate in the input of FCN, i.e.: in, The shape of the local transmission spectrum data is... , representing the calibrated spectral resolution, width resolution, and height resolution of the image, respectively; The local output signal data has the following shape: ; The attention local data has a shape of (s, w, h). , where is the number of attention layers; This is an FCN network.

[0060] The specific hyperparameter values ​​of the FCN network can be adjusted and determined through actual training results. In one example, a classic implementation of the FCN network can be adopted: a 5-layer downsampling convolutional neural network structure and a 5-layer upsampling deconvolutional neural network (CTNN) structure. In the convolutional neural network structure, each convolutional layer uses double max pooling and doubles the number of channels; in the deconvolutional neural network structure, each deconvolutional layer uses double unpooling and halves the number of channels. Residual connections are added to the corresponding CNN and CTNN layers, directly concatenating the tensors on the CNN channels to the CTNN channels, which is the implementation method of u-Net (a commonly used FCN pre-trained model), such as... Figure 5 As shown.

[0061] Furthermore, in typical FCN applications, the FCN output layer has only one channel. However, in this embodiment, the output layer is a multispectral image, meaning it should have c channels. Therefore, the output channels of the last CTNN layer should be c.

[0062] Therefore, in the spectral image reconstruction method according to the embodiments of this application, inputting the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels for reconstructing the spectral image includes: cascading the local transmission spectrum data, the local output signal data, and the attention local data to obtain input data; and inputting the input data into a fully convolutional network to obtain the pixels for reconstructing the spectral image.

[0063] Furthermore, in the aforementioned spectral image reconstruction method, the fully convolutional network includes a predetermined number of downsampling convolutional neural networks and the predetermined number of upsampling deconvolutional neural networks; each convolutional layer in the downsampling convolutional neural network has 2x max pooling, and its number of channels is twice the number of channels in the previous convolutional layer; each convolutional layer in the upsampling deconvolutional neural network has 2x unpooling, and its number of channels is half the number of channels in the previous convolutional layer; the output tensor of each convolutional layer in the downsampling convolutional neural network is concatenated with the output tensor of each convolutional layer in the upsampling deconvolutional neural network; and the number of channels in the output layer of the fully convolutional network is the number of channels in the spectral image to be reconstructed.

[0064] During the training of the fully convolutional network, the backpropagation method used for training general neural networks can be followed. Since the method of reconstructing spectral images using FCN requires a large amount of data for training, various data augmentation methods can be used to expand the size of the training set before training. Alternatively, the dataset can be augmented by downloading a large number of RGB three-channel images and converting them into multi-channel hyperspectral images using existing publicly available algorithms.

[0065] In addition, to accelerate model convergence and reduce reliance on data, transfer learning can be performed by loading an existing RGB three-channel pre-trained model. For example, the transfer learning method involves applying the intermediate multi-layer parameters of the pre-trained model to the aforementioned fully convolutional network model. Specifically, all pre-trained parameters are first frozen, and initial training is performed on the input and output layers. Then, the pre-trained parameters are gradually and symmetrically unfrozen, and the learning rate is reduced, ultimately training the entire network. This avoids getting trapped in local minima and accelerates training convergence.

[0066] Therefore, in the above-mentioned spectral image reconstruction method, the training process of the fully convolutional network includes: pre-training the RGB three-channel model; freezing all pre-training parameters and performing preliminary training on the input and output layers; gradually and symmetrically unfreezing the pre-training parameters and reducing the learning rate to achieve training of the entire fully convolutional network.

[0067] Indicative device Figure 6 A schematic block diagram of a spectral image reconstruction apparatus according to an embodiment of this application is shown.

[0068] like Figure 6 As shown, the spectral image reconstruction apparatus 200 according to an embodiment of this application includes: a data acquisition unit 210, configured to acquire transmission spectrum data and output signal data of a spectral imaging chip; a data partitioning unit 220, configured to acquire local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on pixels for reconstructing the spectral image; an attention unit 230, configured to input the local output signal data into an attention model to obtain attention local data; and a neural network unit 240, configured to input the local transmission spectrum data, the local output signal data, and the attention local data into a neural network model to obtain the pixels for reconstructing the spectral image.

[0069] Here, those skilled in the art will understand that the specific functions and operations of each unit and module in the above-described spectral image reconstruction apparatus 200 have been referenced above. Figures 1 to 5 The spectral image reconstruction method is described in detail, and therefore, its repeated description will be omitted.

[0070] As described above, the spectral image reconstruction apparatus 200 according to embodiments of this application can be implemented in various terminal devices, such as servers for high-resolution spectral image reconstruction, or various spectrometers and spectral imaging devices. In one example, the spectral image reconstruction apparatus 200 according to embodiments of this application can be integrated into the terminal device as a software module and / or a hardware module. For example, the spectral image reconstruction apparatus 200 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the spectral image reconstruction apparatus 200 can also be one of many hardware modules of the terminal device.

[0071] Alternatively, in another example, the spectral image reconstruction device 200 and the terminal device can also be separate devices, and the spectral image reconstruction device 200 can be connected to the terminal device via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0072] Exemplary electronic devices Below, for reference Figure 7 This describes an electronic device according to embodiments of the present application.

[0073] Figure 7 A schematic block diagram of an electronic device according to an embodiment of this application is shown.

[0074] like Figure 5 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0075] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0076] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the spectral image reconstruction methods of the various embodiments of this application described above and / or other desired functions. Various contents such as transmission spectrum data and output signal data may also be stored in the computer-readable storage medium.

[0077] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0078] For example, the input device 13 can be, for example, a keyboard, a mouse, etc.

[0079] The output device 14 can output various information to the outside, such as reconstructed spectral images. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0080] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0081] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the spectral image reconstruction methods according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0082] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0083] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the spectral image reconstruction methods according to various embodiments of this application described in the "Exemplary Methods" section of this specification.

[0084] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0085] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0086] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0087] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0088] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0089] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for spectral image reconstruction, characterized in that, include: Acquire the transmission spectrum data and output signal data of the spectral imaging chip; Based on the pixels used to reconstruct the spectral image, local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data are obtained; The local output signal data is input into the attention model to obtain local attention data; and... The local transmission spectrum data, the local output signal data, and the local attention data are input into a neural network model to obtain the pixels used to reconstruct the spectral image; The process of obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on the pixels used to reconstruct the spectral image includes: obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data with a side length of a predetermined number of pixels in the region near the pixel used to reconstruct the spectral image, based on the pixel's position. The step of inputting the local output signal data into the attention model to obtain attention local data includes: dividing the local output signal data into multiple predetermined regions, each predetermined region including output signal data corresponding to multiple pixels of the spectral imaging chip; and... The corresponding positional elements of each predetermined region are multiplied to obtain the attention local data.

2. The spectral image reconstruction method as described in claim 1, wherein, The process of inputting the local output signal data into the attention model to obtain local attention data includes: The transmission spectrum data and the output signal data are normalized to... Within the range.

3. The spectral image reconstruction method as described in claim 1, wherein, The local transmission spectrum data, the local output signal data, and the attention local data are input into the neural network model to obtain the pixels used to reconstruct the spectral image, including: The local transmission spectrum data at each spectral resolution in the local transmission spectrum data, the local output signal data, and the local attention data are concatenated in the thickness dimension to obtain a cascaded tensor. Multiple cascaded tensors corresponding to multiple spectral resolutions of the local transmission spectrum data are cascaded in the thickness dimension to obtain the input tensor. The input tensor is fed into a 3D convolutional neural network to obtain an output feature map, wherein the thickness dimension of the output feature map is the same as the number of channels of the spectral image to be reconstructed; and, Each feature map in the thickness dimension of the output feature map is input into a fully connected network to obtain the pixels used to reconstruct the spectral image.

4. The spectral image reconstruction method as described in claim 3, wherein, The input and output data of the 3D convolutional neural network include four dimensions: data length, data width, data thickness, and number of channels. The data thickness is equal to the number of iterations of the 3D convolutional neural network, and the number of channels of the input data is one.

5. The spectral image reconstruction method as described in claim 3, wherein, The 3D convolutional neural network includes convolutional layers, activation layers, and pooling layers; The first convolutional layer has The step size, where It is the square root of the number of predetermined regions in the attention model, and the remaining convolutional layers have a stride of 1; The first convolutional layer uses the Sigmoid activation function, and the remaining layers use the Leaky ReLU activation function; All pooling layers are max-pooling layers with a length and width order of 2, and variable thickness order, with the pooling parameter of the thickness order determined by the number of channels in the target spectral image; and, The number of channels in each layer of a convolutional network is twice the number of channels in the layer above it.

6. The spectral image reconstruction method as described in claim 3, wherein, The training data for the 3D convolutional neural network and the fully connected network includes at least one of the following: The transmission spectrum data and the output signal data are obtained based on still life hyperspectral images acquired using high-precision spectroscopic cameras with different modulation layers; The output signal data is obtained by simulating imaging hyperspectral images from a hyperspectral dataset obtained from public channels using multiple transmission spectrum data from the product database; as well as, The transmission spectrum data and output signal data obtained by adding noise to the above transmission spectrum data and output signal data.

7. The spectral image reconstruction method as described in claim 3, wherein, The training process for the 3D convolutional neural network and the fully connected network includes: The entire image is divided into small blocks with a predetermined side length using random center coordinates; Hash and reorder the spectral truth values ​​corresponding to the small blocks; and The shuffled small pieces and their corresponding spectra are combined to form a training package, which is then input into the model for training.

8. The spectral image reconstruction method as described in claim 1, wherein, The local transmission spectrum data, the local output signal data, and the attention local data are input into the neural network model to obtain the pixels used to reconstruct the spectral image, including: The local transmission spectrum data, the local output signal data, and the local attention data are concatenated to obtain input data; and... The input data is fed into a fully convolutional network to obtain the pixels used to reconstruct the spectral image.

9. The spectral image reconstruction method as described in claim 8, wherein, The fully convolutional network includes a predetermined number of downsampled convolutional neural networks and the predetermined number of upsampled deconvolutional neural networks; Each convolutional layer in the downsampling convolutional neural network has 2x max pooling and its number of channels is twice that of the previous convolutional layer; Each convolutional layer of the upsampled deconvolutional neural network has 2x unpooling, and its number of channels is half the number of channels of the previous convolutional layer; The output tensor of each convolutional layer in the downsampling convolutional neural network is concatenated with the output tensor of each convolutional layer in the upsampling deconvolutional neural network; as well as, The number of channels in the output layer of the fully convolutional network is the same as the number of channels in the spectral image to be reconstructed.

10. The spectral image reconstruction method as described in claim 8, wherein, The training process of the fully convolutional network includes: Obtain the RGB three-channel pre-trained model; Freeze all pre-trained parameters and perform initial training on the input and output layers; and, Gradually and symmetrically unfreeze the pre-trained parameters and reduce the learning rate to train the entire fully convolutional network.

11. A spectral image reconstruction apparatus, characterized in that, include: The data acquisition unit is used to acquire the transmission spectrum data and output signal data of the spectral imaging chip. A data partitioning unit is used to obtain local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on the pixels used to reconstruct the spectral image; An attention unit is configured to input the local output signal data into an attention model to obtain local attention data; and, A neural network unit is used to input the local transmission spectrum data, the local output signal data, and the attention local data into the neural network model to obtain the pixels used to reconstruct the spectral image; The process of obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data based on the pixels used to reconstruct the spectral image includes: obtaining local transmission spectrum data of the transmission spectrum data and local output signal data of the output signal data with a side length of a predetermined number of pixels in the region near the pixel used to reconstruct the spectral image, based on the pixel's position. The step of inputting the local output signal data into the attention model to obtain attention local data includes: dividing the local output signal data into multiple predetermined regions, each predetermined region including output signal data corresponding to multiple pixels of the spectral imaging chip; and... The corresponding positional elements of each predetermined region are multiplied to obtain the attention local data.

Images