Microscopic hyperspectral image high spatial resolution reconstruction method based on transmissivity correction

Abstract

The invention relates to the technical field of computational imaging, in particular to a microscopic hyperspectral image high-spatial-resolution reconstruction method based on transmittance correction, and the method comprises the steps: carrying out the LR-HSI and HR-RGB data transmittance correction at the same time in a data preprocessing stage; the problem of uneven light splitting possibly existing in a beam splitter in a snapshot microlens array microscopic hyperspectral imaging system can be eliminated, and fine adjustment is performed on an initialized model through real data so as to improve the prediction accuracy of a final image reconstruction model on the real data.

Description

Technical Field

[0001] The present invention belongs to the technical field of computational imaging, and in particular relates to a method for reconstructing a microscopic hyperspectral image with high spatial resolution based on transmittance correction. Background Art

[0002] The snapshot microlens array micro-hyperspectral imaging system uses a microlens array to cut the field of view and obtain micro-hyperspectral images without scanning. However, the spatial resolution of the data collected by the system is low, and it is impossible to achieve the unity of high spatial resolution, high spectral resolution and high temporal resolution. Applying deep learning technology to the field of image fusion can generate high spatial resolution and high spectral resolution images using low spatial resolution hyperspectral images and high spatial resolution multispectral images (both grayscale images and RGB image data are acceptable, and RGB images are used as an example below). This is a key method to solve the spatial resolution limitations of the snapshot microlens array micro-hyperspectral imaging system.

[0003] In the training phase, the existing image fusion methods perform spatial and spectral downsampling on high spatial resolution hyperspectral image (HR-HSI) data to obtain low spatial resolution hyperspectral image (LR-HSI) data and high spatial resolution RGB image (HR-RGB) data, and use LR-HSI and HR-RGB data as model data input for training to obtain an initialized image fusion model that can generate HR-HSI data. Finally, the initialized image fusion model is adjusted by actual data to obtain the final model.

[0004] The snapshot microlens array micro-hyperspectral imaging system divides the incident light into two parts through a beam splitter to ensure that the RGB camera and the hyperspectral camera can simultaneously obtain data with the same lighting conditions, so as to ensure high-quality data input for the image fusion model. However, if the beam splitting performance of the beam splitter is uneven, that is, the energy of the two beams of light incident on the RGB camera and the hyperspectral camera is different, it may cause a large difference between the model data input and the data input during training, thereby affecting the performance of the initialized image fusion model fusion, resulting in the phenomenon that the image fusion model performs well in simulated data but poorly in actual data. Summary of the invention

[0005] In view of this, the present invention aims to provide a high spatial resolution reconstruction method for microscopic hyperspectral images based on transmittance correction. By simultaneously performing transmittance correction on LR-HSI and HR-RGB data, the problem of uneven light splitting that may exist in the beam splitter can be eliminated, thereby improving the accuracy of the initialization model in fusing real data.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows: A method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction, comprising: S1: Use a scanning microscopic hyperspectral imaging system to collect HR-HSI data of samples and blanks, and perform spectral downsampling and spatial downsampling on the HR-HSI data to obtain corresponding first LR-HSI data and first HR-RGB data; calculate the transmittance of the first LR-HSI data, the first HR-RGB data, and the HR-HSI data, respectively, and obtain first LR-HSI transmittance data, first HR-RGB transmittance data, and HR-HSI transmittance data correspondingly; S2: repeat step S1, and use the obtained multiple sets of first LR-HSI transmittance data and multiple sets of first HR-RGB transmittance data as input, and the obtained multiple sets of HR-HSI data as output, and perform supervised training on the deep learning model to obtain an image reconstruction model; S3: using a snapshot microlens array micro-hyperspectral imaging system to collect the second LR-HSI data and the second HR-RGB data of the sample and the blank, respectively calculating the transmittance of the second LR-HSI data and the second HR-RGB data, and correspondingly obtaining the second LR-HSI transmittance data and the second HR-RGB transmittance data; S4: performing unsupervised training on the image reconstruction model obtained in step S2 using the second LR-HSI transmittance data and the second HR-RGB transmittance data obtained in step S3 to obtain a final image reconstruction model; S5: Calculate the transmittance of the LR-HSI data to be reconstructed and the corresponding HR-RGB data respectively; input the obtained transmittance data into the final image reconstruction model obtained in step S4 to obtain the corresponding HR-HSI transmittance data.

[0007] Further, in step S1, the first LR-HSI data and the first HR-RGB data are obtained by the following formulas: ; in, represents the first LR-HSI data, Y represents the first HR-RGB data, and Z represents the HR-HSI data; H represents the spatial downsampling operator, and P represents the spectral downsampling operator.

[0008] Further, in step S1, LR-HSI transmittance data, HR-RGB transmittance data and HR-HSI transmittance data are obtained respectively by the following formulas: ; in, , , Respectively represent HR-HSI transmittance data, first LR-HSI transmittance data and first HR-RGB transmittance data; Represents the HR-HSI data of the sample, represents the first LR-HSI data of the sample, represents the first HR-RGB data of the sample, , and They respectively represent the spectral averages of the HR-HSI data of the blank film, the first LR-HSI data, and the first HR-RGB data.

[0009] Furthermore, in the process of supervised training of the deep learning model in step S2, the supervised loss function used is: ; in, represents the supervised loss function, It represents the loss function of the HR-HSI transmittance data predicted by the deep learning model and the corresponding real HR-HSI transmittance data in the edge area. The loss function of the HR-HSI transmittance data predicted by the deep learning model and the corresponding real HR-HSI transmittance data in the smooth area, represents the loss weight.

[0010] Furthermore, the loss function of the edge area is for: ; in, represents the edge area of ​​the jth real HR-HSI transmittance data, represents the edge area of ​​the j-th predicted HR-HSI transmittance data, and M represents the number of real or predicted HR-HSI transmittance data; Furthermore, the loss function in the smooth region is for: ; in, represents the smoothing area of ​​the jth real HR-HSI transmittance data, Indicates the smoothing area of ​​the j-th predicted HR-HSI transmittance data.

[0011] Furthermore, the process of obtaining the edge area and smooth area of ​​the real or predicted HR-HSI transmittance data includes: converting the real or predicted HR-HSI transmittance data into a grayscale image, using an edge detection algorithm based on gradient changes to detect the edge area in the grayscale image, and the non-edge area in the grayscale image is the smooth area.

[0012] Furthermore, in the process of performing unsupervised training on the image reconstruction model in step S4, the unsupervised loss function used is: ; in, represents the unsupervised loss function, N represents the number of second LR-HSI transmittance data in each training, represents the i-th second LR-HSI transmittance data, represents the HR-HSI transmittance data predicted by the image reconstruction model based on the i-th second LR-HSI transmittance data, H represents the spatial downsampling operator, represents the loss weight, Represents the calculation factor of the first-order derivative of the spectral dimension.

[0013] Compared with the prior art, the invention can achieve the following beneficial effects: In the high spatial resolution reconstruction method of microscopic hyperspectral images based on transmittance correction created by the present invention, transmittance data is used as a training set to train a deep learning model to obtain an image reconstruction model, and transmittance correction of LR-HSI and HR-RGB data can be performed simultaneously, which can eliminate the influence of inconsistent conditions for obtaining LR-HSI and HR-RGB data caused by uneven spectral analysis, and improve the accuracy of the fusion model; in addition, since the snapshot microlens array microscopic hyperspectral imaging system cannot obtain HR-HSI data, and the data used for training the deep learning model comes from the scanning microscopic hyperspectral imaging system, in order to avoid the image reconstruction model obtained by training being inapplicable to the data of the snapshot microlens array microscopic hyperspectral imaging system due to different data sources, the present invention uses the data of the snapshot microlens array microscopic hyperspectral imaging system to perform unsupervised secondary training on the image reconstruction model, so as to further improve the model's prediction accuracy for real data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of a process for reconstructing a microscopic hyperspectral image with high spatial resolution based on transmittance correction according to an embodiment of the present invention; Figure 2 A flowchart of a method for reconstructing a microscopic hyperspectral image with high spatial resolution based on transmittance correction described in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0016] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0017] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0018] like Figure 1 to Figure 2 As shown, the method for reconstructing a microscopic hyperspectral image with high spatial resolution based on transmittance correction described in the embodiment of the present invention includes: S1: Use a scanning microscope hyperspectral imaging system to collect HR-HSI data of samples and blanks, and perform spectral downsampling and spatial downsampling on the HR-HSI data to obtain the corresponding first LR-HSI data and first HR-RGB data; calculate the transmittance of the first LR-HSI data, the first HR-RGB data, and the HR-HSI data, respectively, and obtain the first LR-HSI transmittance data, the first HR-RGB transmittance data, and the HR-HSI transmittance data correspondingly.

[0019] The sample is an empty slide containing the sample to be detected, and the empty slide after removing the sample is a blank slide. In some embodiments, the first LR-HSI data and the first HR-RGB data are obtained by the following formulas: ; in, represents the first LR-HSI data, Y represents the first HR-RGB data, and Z represents the HR-HSI data; H represents the spatial downsampling operator, and P represents the spectral downsampling operator.

[0020] Further, in step S1, LR-HSI transmittance data, HR-RGB transmittance data and HR-HSI transmittance data are obtained respectively by the following formulas: ; in, , , Respectively represent HR-HSI transmittance data, first LR-HSI transmittance data and first HR-RGB transmittance data; Represents the HR-HSI data of the sample, represents the first LR-HSI data of the sample, represents the first HR-RGB data of the sample, , and They respectively represent the spectral averages of the HR-HSI data of the blank film, the first LR-HSI data, and the first HR-RGB data.

[0021] S2: Repeat step S1, and use the obtained multiple sets of first LR-HSI transmittance data and multiple sets of first HR-RGB transmittance data as input, and the obtained multiple sets of HR-HSI data as output, and use the deep learning model for supervised training to obtain an image reconstruction model, which can determine the mapping relationship between the HR-HSI transmittance data and the first LR-HSI transmittance data and the first HR-RGB transmittance data, that is: ; in, represents the HR-HSI transmittance data predicted by the image reconstruction model, Represents the mapping relationship, Represents the parameters of the image reconstruction model.

[0022] In one embodiment, a deep learning model based on guided filtering and deep residual network disclosed in the paper "Research on Hyperspectral Image Fusion Method Based on Deep Learning" from Xidian University is used as a deep learning model. The deep learning model first enhances the edge detail information of the full-color image based on the CLAHE (Contrast-Limited Adaptive Histogram Equalization) algorithm, and then uses the guided filter to generate an initialized fused image. Next, the mapping relationship between the initialized fused image and the residual image is learned through the DRCNN network (Deep Residual Convolutional Neural Network). Finally, the initialized fusion result is added to the residual image output by the network to obtain the final fused image.

[0023] In some embodiments, during supervised training using a deep learning model, the supervised loss function used is: ; in, represents the supervised loss function, It represents the loss function of the HR-HSI transmittance data predicted by the deep learning model and the corresponding real HR-HSI transmittance data in the edge area. The loss function of the HR-HSI transmittance data predicted by the deep learning model and the corresponding real HR-HSI transmittance data in the smooth area, Represents the loss weight, and multiple experiments are needed to determine the optimal weight value.

[0024] In some embodiments, the process of obtaining the edge area and smooth area of ​​the real or predicted HR-HSI transmittance data includes: converting the real or predicted HR-HSI transmittance data into a grayscale image, and using an edge detection algorithm based on gradient changes to detect the edge area in the grayscale image, and the non-edge area in the grayscale image is the smooth area. The edge detection algorithm can use the Canny algorithm, sobel algorithm and other algorithms that can realize edge detection. The method of using the selected edge detection algorithm and its internal specific parameters can be adaptively selected and adjusted according to the actual situation of the image. The present invention does not limit this. In a certain embodiment, the Canny algorithm is used to perform edge detection on the grayscale image of the real or predicted HR-HSI transmittance data, so as to obtain the edge area and smooth area of ​​the real or predicted HR-HSI transmittance data.

[0025] Loss function for edge regions As follows: ; in, represents the edge area of ​​the jth real HR-HSI transmittance data, represents the edge area of ​​the j-th predicted HR-HSI transmittance data, and M represents the number of real or predicted HR-HSI transmittance data.

[0026] Loss function for smooth regions As follows: ; in, represents the smoothing area of ​​the jth real HR-HSI transmittance data, Indicates the smoothing area of ​​the j-th predicted HR-HSI transmittance data.

[0027] Because the snapshot microlens array micro-hyperspectral imaging system cannot obtain HR-HSI data, the training data of the image reconstruction model can only be obtained through the scanning micro-hyperspectral imaging system. However, there is a gap between the scanning micro-hyperspectral imaging system and the snapshot micro-hyperspectral imaging system. Therefore, the reconstruction performance of the reconstruction model trained with scanning data will be reduced when directly applied to the snapshot system. Therefore, it is necessary to use the real data collected by the snapshot microlens array micro-hyperspectral imaging system to adjust the reconstruction model so that the model can adapt to the snapshot system. Since the snapshot system cannot obtain HR-HSI data, it can only be adjusted using unsupervised learning methods.

[0028] S3: A snapshot microlens array micro-hyperspectral imaging system is used to collect the second LR-HSI data and the second HR-RGB data of the sample and the blank, and the transmittance of the second LR-HSI data and the second HR-RGB data are calculated respectively, so as to obtain the second LR-HSI transmittance data and the second HR-RGB transmittance data accordingly.

[0029] S4: Perform unsupervised training on the image reconstruction model obtained in step S2 using the second LR-HSI transmittance data and the second HR-RGB transmittance data obtained in step S3 to obtain a final image reconstruction model.

[0030] In some embodiments, during the unsupervised training of the image reconstruction model, the unsupervised loss function used is: ; in, represents the unsupervised loss function, N represents the number of second LR-HSI transmittance data in each training, represents the i-th second LR-HSI transmittance data, represents the HR-HSI transmittance data predicted by the image reconstruction model based on the i-th second LR-HSI transmittance data, It represents the calculation factor of the first-order derivative of the spectral dimension, that is, the first-order derivative of the hyperspectral data along the spectral dimension is calculated. It is a common data preprocessing method in hyperspectral data analysis, which can highlight the change characteristics of the spectral dimension of the data. Using this method as a loss function can ensure the prediction performance by constraining the change trend of the spectral dimension of the predicted data. Represents the loss weight, and multiple experiments are needed to determine the optimal weight value.

[0031] S5: Calculate the transmittance of the LR-HSI data to be reconstructed and the corresponding HR-RGB data respectively; input the obtained transmittance data into the final image reconstruction model obtained in step S4 to obtain the corresponding HR-HSI transmittance data.

[0032] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0033] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction, characterized in that: include: S1: using a scanning microscopic hyperspectral imaging system to collect HR-HSI data of a sample and a blank, and performing spectral downsampling and spatial downsampling on the HR-HSI data to obtain corresponding first LR-HSI data and first HR-RGB data; The transmittances of the first LR-HSI data, the first HR-RGB data, and the HR-HSI data are calculated respectively, and first LR-HSI transmittance data, first HR-RGB transmittance data, and HR-HSI transmittance data are obtained correspondingly; S2: repeat step S1, and use the obtained multiple groups of the first LR-HSI transmittance data and the obtained multiple groups of the first HR-RGB transmittance data as input, and the obtained multiple groups of the HR-HSI data as output, and perform supervised training on the deep learning model to obtain an image reconstruction model; S3: using a snapshot microlens array micro-hyperspectral imaging system to collect second LR-HSI data and second HR-RGB data of the sample and the blank, respectively calculating the transmittance of the second LR-HSI data and the second HR-RGB data, and correspondingly obtaining second LR-HSI transmittance data and second HR-RGB transmittance data; S4: performing unsupervised training on the image reconstruction model obtained in step S2 using the second LR-HSI transmittance data and the second HR-RGB transmittance data obtained in step S3 to obtain a final image reconstruction model; S5: Calculate the transmittance of the LR-HSI data to be reconstructed and the corresponding HR-RGB data respectively; The obtained transmittance data is input into the final image reconstruction model obtained in step S4 to obtain corresponding HR-HSI transmittance data.

2. The method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction according to claim 1 is characterized in that: In step S1, the first LR-HSI data and the first HR-RGB data are obtained by the following formulas: ； in, represents the first LR-HSI data, Y represents the first HR-RGB data, and Z represents the HR-HSI data; H represents a spatial downsampling operator, and P represents a spectral downsampling operator.

3. The method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction according to claim 1 is characterized in that: In step S1, the LR-HSI transmittance data, the HR-RGB transmittance data and the HR-HSI transmittance data are obtained respectively by the following formulas: ； in, , , respectively represent the HR-HSI transmittance data, the first LR-HSI transmittance data and the first HR-RGB transmittance data; represents the HR-HSI data of the sample, represents the first LR-HSI data of the sample, represents the first HR-RGB data of the sample, , and They respectively represent the spectral averages of the HR-HSI data, the first LR-HSI data and the first HR-RGB data of the blank film.

4. The method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction according to claim 1 or 3, characterized in that: In the process of supervised training of the deep learning model in step S2, the supervised loss function used is: ； in, represents the supervised loss function, represents the loss function of the HR-HSI transmittance data predicted by the deep learning model and the corresponding real HR-HSI transmittance data in the edge area, The loss function of the HR-HSI transmittance data predicted by the deep learning model and the corresponding real HR-HSI transmittance data in the smooth region, represents the loss weight.

5. The method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction according to claim 4 is characterized in that: Loss function for edge regions for: ； in, represents the edge area of ​​the jth real HR-HSI transmittance data, represents the edge area of ​​the j-th predicted HR-HSI transmittance data, and M represents the number of real or predicted HR-HSI transmittance data.

6. The method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction according to claim 4, characterized in that: Loss function for smooth regions for: ； in, represents the smoothing area of ​​the jth real HR-HSI transmittance data, Indicates the smoothing area of ​​the j-th predicted HR-HSI transmittance data.

7. The method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction according to any one of claims 4 to 6, characterized in that: The process of obtaining edge regions and smooth regions of real or predicted HR-HSI transmittance data includes: The real or predicted HR-HSI transmittance data is converted into a grayscale image, and the edge area in the grayscale image is detected using an edge detection algorithm based on gradient changes. The non-edge area in the grayscale image is the smooth area.

8. The method for high spatial resolution reconstruction of microscopic hyperspectral images based on transmittance correction according to claim 1 or 3, characterized in that: In the process of performing unsupervised training on the image reconstruction model in step S4, the unsupervised loss function used is: ； in, represents the unsupervised loss function, N represents the number of the second LR-HSI transmittance data in each training, represents the i-th second LR-HSI transmittance data, represents the HR-HSI transmittance data predicted by the image reconstruction model according to the i-th second LR-HSI transmittance data, H represents the spatial downsampling operator, represents the loss weight, Represents the calculation factor of the first-order derivative of the spectral dimension.

Images