Multiband hyperspectral image fusion method based on transfer learning

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a multiband hyperspectral image fusion method based on transfer learning. Comprising the following steps: S1, collecting a high-resolution hyperspectral image set of a target; s2, the high-resolution hyperspectral image set is processed; s3, dividing the preprocessed image group to obtain a first data set and a second data set; s4, obtaining a to-be-processed data set; s5, segmenting each piece of to-be-processed data; s6, training the pre-training model; s7, repeating the steps S4-S5 to obtain a second sub-data set; s8, on the basis of an unsupervised learning method, performing fine adjustment on the optimal model by using the training sub-data set to obtain an optimal prediction model; s9, inputting the to-be-predicted sub-data set into the optimal prediction model for prediction; and S10, connecting the prediction sub-data along the spectral dimension to obtain a final high-resolution hyperspectral image. On the premise of ensuring the image fusion performance, the method is suitable for data sets with different multi-band numbers.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multi-band hyperspectral image fusion method based on transfer learning. Background Art

[0002] Snapshot microlens array micro-hyperspectral imaging system ( Figure 1 ) Based on the principle of common optical path design, it is possible to simultaneously obtain low spatial resolution hyperspectral images (LR-HSI) and high spatial resolution grayscale images (HR-PAN) without scanning. However, since the system cannot directly obtain the corresponding high spatial resolution hyperspectral images (HR-HSI), a feasible method is to use a scanning microscopic hyperspectral imaging system to collect HR-HIS of the same target sample and use it as a reference image. By performing downsampling processing on the reference image in the spatial and spectral dimensions, the data characteristics collected by the snapshot microlens array microscopic hyperspectral imaging system can be simulated. This synthetic dataset is used to pre-train the model to obtain an initial model, and then the model is fine-tuned and optimized using a small amount of actual collected data and an unsupervised learning method to achieve efficient and accurate image fusion for the snapshot microlens array microscopic hyperspectral imaging system.

[0003] At present, the research on hyperspectral image fusion is mainly focused on public and accessible remote sensing datasets, while the research on microscopic hyperspectral image fusion is relatively small. There is a difference in the number of spectral bands between the snapshot microlens array microscopic hyperspectral imaging system and the scanning microscopic hyperspectral imaging system. When pre-training the data collected by the scanning microscopic hyperspectral imaging system, the spectral dimension must be adjusted to match the characteristics of the scanning microscopic hyperspectral imaging system. However, this adjustment may introduce problems such as spectral distortion or information loss, thereby affecting the final image fusion accuracy. At the same time, if the subsequent improved snapshot microlens array microscopic hyperspectral imaging system has different spectral bands, it needs to be retrained, which consumes computing resources. Summary of the invention

[0004] In view of this, the present invention aims to provide a multi-band hyperspectral image fusion method based on transfer learning to solve the problem of poor model performance of the snapshot microlens array microscopic hyperspectral imaging system in the image fusion process caused by the above-mentioned problems. While ensuring the image fusion performance, the present invention can adapt to data sets with different numbers of multi-bands, and can utilize existing training models to save computing resources.

[0005] To achieve the above object, the technical solution created by the present invention is implemented as follows: A multi-band hyperspectral image fusion method based on transfer learning specifically includes the following steps: S1: A set of high-resolution hyperspectral images of the target collected by a scanning microscopic hyperspectral imaging system; S2: performing spectral dimension reduction operation and spatial dimension reduction operation on the high-resolution hyperspectral image set to obtain a preprocessed image group; the preprocessed image group includes one-to-one corresponding first transmittance data and second transmittance data; S3: dividing the preprocessed image group in proportion to obtain a first data set and a second data set; S4: obtaining a data set to be processed based on the first transmittance data and the second transmittance data in the first data set; S5: Segment each to-be-processed data contained in the to-be-processed data set to obtain a first sub-data set; S6: Use the first sub-dataset to train the pre-trained model to obtain the best model; S7: Replace the first data set with the second data set, and repeat steps S4-S5 to obtain a second sub-data set; S8: dividing the second sub-dataset into a training sub-dataset and a sub-dataset to be predicted in proportion, and optimizing the best model using the training sub-dataset based on an unsupervised learning method to obtain the best prediction model; S9: inputting the sub-dataset to be predicted into the optimal prediction model for prediction, and obtaining the predicted sub-dataset; S10: Connect the prediction sub-data contained in the prediction sub-dataset along the spectral dimension to obtain a final high-resolution hyperspectral image.

[0006] Furthermore, in step S1, the scanning microscope hyperspectral imaging system uses a tungsten lamp as a light source and collects the target under the same objective lens magnification.

[0007] Furthermore, in step S2, a spectral dimension reduction operation is performed on the high-resolution hyperspectral image to obtain a grayscale image, and the grayscale image is subjected to transmittance conversion to obtain first transmittance data; a spatial dimension reduction operation is performed on the high-resolution hyperspectral image to obtain a low-resolution hyperspectral image, and the low-resolution hyperspectral image is subjected to transmittance conversion to obtain second transmittance data.

[0008] Further, in step S5, the specific steps of segmenting the data to be processed are: assuming that the number of bands contained in the data to be processed is N, the data to be processed is divided into a plurality of microscopic hyperspectral image data of M bands by the following formula, N>M, and the first sub-data set is obtained, and its grayscale image is calculated respectively; ; in, For the rounding operation, The number of sub-data contained in the first sub-data set.

[0009] Furthermore, the number of overlapping bands contained in two adjacent sub-data is at least : ; in, This is a floor operation.

[0010] Furthermore, in the connection process of step S10, the band values ​​of the overlapping areas of adjacent prediction sub-data are fitted in turn by the least squares method to obtain the optimal band value of each overlapping area, and the current band value of each overlapping area is replaced by the optimal band value of each overlapping area to obtain the final high-resolution hyperspectral image.

[0011] Compared with the prior art, the invention can achieve the following beneficial effects: (1) The multi-band hyperspectral image fusion method based on transfer learning created by the present invention improves the efficiency and accuracy of the microscopic hyperspectral image fusion of the snapshot microlens array microscopic hyperspectral imaging system, and is adaptable to data sets with different numbers of bands. It has wide applicability and does not require the pre-training data set to be readjusted each time, which significantly saves computing resources.

[0012] (2) The multi-band hyperspectral image fusion method based on transfer learning created by the present invention combines a variety of technical means to form a comprehensive solution, which can better cope with complex data fusion tasks and has higher generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of the process of the multi-band hyperspectral image fusion method based on transfer learning described in an embodiment of the present invention; Figure 2 Schematic diagram of the prediction results of the three processes described in the embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0015] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0016] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0017] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0018] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0019] like Figure 1As shown, the present invention provides a multi-band hyperspectral image fusion method based on transfer learning, which specifically includes the following steps: S1: collecting a high-resolution hyperspectral image set of a target based on a scanning microscopic hyperspectral imaging system; S2: performing spectral dimension reduction operation and spatial dimension reduction operation on the high-resolution hyperspectral image set to obtain a pre-processed image group; the pre-processed image group includes a one-to-one corresponding first transmittance data and a second transmittance data; S3: dividing the pre-processed image group according to a ratio to obtain a first data set and a second data set; S4: obtaining a to-be-processed data set based on the first transmittance data and the second transmittance data in the first data set; S5: performing a spectral dimension reduction operation and a spatial dimension reduction operation on the high-resolution hyperspectral image set to obtain a pre-processed image group; the pre-processed image group includes a one-to-one corresponding first transmittance data and a second transmittance data; S3: dividing the pre-processed image group according to a ratio to obtain a first data set and a second data set; S4: obtaining a to-be-processed data set based on the first transmittance data and the second transmittance data in the first data set; S5: performing a spectral dimension reduction operation on each to-be-processed data set included in the to-be-processed data set The first sub-dataset is segmented to obtain a first sub-dataset; S6: the pre-trained model is trained with the first sub-dataset to obtain an optimal model; S7: the first data set is replaced with the second data set, and steps S4-S5 are repeated to obtain a second sub-dataset; S8: the second sub-dataset is divided into a training sub-dataset and a sub-dataset to be predicted in proportion, and the optimal model is optimized with the training sub-dataset based on an unsupervised learning method to obtain an optimal prediction model; S9: the sub-dataset to be predicted is input into the optimal prediction model for prediction to obtain a prediction sub-dataset; S10: the prediction sub-data contained in the prediction sub-dataset are connected along the spectral dimension to obtain a final high-resolution hyperspectral image.

[0020] The present invention uses joint transfer learning, unsupervised learning, least squares fitting and other methods, based on a trained low spectral dimension image fusion model, pre-trains it with data, divides the real data into several sub-datasets with the number of bands equal to the number of low spectral dimension images, and selects a small part of the data to fine-tune the pre-trained model. Finally, the fine-tuned model is used to train each sub-dataset, and the sub-datasets are spliced ​​based on the least squares method to complete the data fusion of the snapshot microlens array microscopic hyperspectral imaging system.

[0021] It should be noted that the unsupervised learning method adjusts the pre-trained model by using a small amount of data to be predicted. The core of this method is to design a set of loss functions that do not depend on the reference image data. By adjusting the predicted data to optimize the new loss function, the pre-trained model can complete the prediction task more accurately while minimizing the loss. In the present invention, one of the loss functions that does not depend on the reference data is to reduce the dimension of the predicted high-resolution hyperspectral image data to low-resolution hyperspectral image data, and calculate the mean square error between the low-resolution hyperspectral image data and the input real low-resolution hyperspectral image data.

[0022] Furthermore, in step S8, according to actual conditions, the weights of some pre-trained models (such as early layers) are frozen to retain common features, and only high-level weights are trained to adapt to real data to improve the prediction accuracy of real data. The pre-trained model is selected according to user needs. If the image features of the images collected by the snapshot microlens array micro-hyperspectral imaging system change little, the corresponding model can be fine-tuned using an unsupervised learning method. Otherwise, the high-resolution hyperspectral image needs to be re-produced using the present invention.

[0023] In some embodiments, the above-mentioned proportional division is based on actual conditions, and a division ratio of 8:2 is usually adopted.

[0024] In some embodiments, in step S1, the scanning microscope hyperspectral imaging system uses a tungsten lamp as a light source and collects the target under the same objective lens magnification.

[0025] The snapshot microlens array microscope hyperspectral imaging system cannot obtain real high-resolution hyperspectral image data, so the training data can only be obtained through scanning microscope hyperspectral data; tungsten lamps are used as light sources because tungsten lamps have a continuous spectrum, and hyperspectral imaging experiments must use a light source with a continuous spectrum for illumination; and the same objective lens magnification is used to minimize the difference between the data of the snapshot microlens array microscope hyperspectral imaging system and the data of the scanning microscope hyperspectral imaging system.

[0026] In some embodiments, in step S2, a spectral dimension reduction operation is performed on the high-resolution hyperspectral image to obtain a grayscale image, and the grayscale image is subjected to transmittance conversion to obtain first transmittance data; a spatial dimension reduction operation is performed on the high-resolution hyperspectral image to obtain a low-resolution hyperspectral image, and the low-resolution hyperspectral image is subjected to transmittance conversion to obtain second transmittance data.

[0027] It should be noted that the transmittance data can map the digital (DN) value of the image to the interval [0,1], which not only helps to accelerate model convergence, enhance the stability of the training process, and ensure the standardization of input data, but also can more effectively retain the spectral information in the original data.

[0028] In some embodiments, in step S5, the specific steps of segmenting the data to be processed are: assuming that the number of bands contained in the data to be processed is N, the data to be processed is divided into a plurality of microscopic hyperspectral image data of M bands by the following formula, N>M, a first sub-data set is obtained, and its grayscale image is calculated respectively; ; in, For the rounding operation, The number of sub-data contained in the first sub-data set.

[0029] In some embodiments, two adjacent sub-data contain at least the number of overlapping bands : ; in, This is a floor operation.

[0030] It should be noted that through the above processing, a data set that meets the input requirements of the pre-trained model is generated (assuming that the model is trained on hyperspectral image data with 31 spectral dimensions). These data sets must be consistent with the requirements of the pre-trained model in terms of the number of bands, and other aspects must meet the requirements of the pre-trained model.

[0031] The following is an example of dividing several 31-band microscopic hyperspectral image data to further explain the segmentation operation: Assume that the number of bands contained in each data to be processed (HR-HSI data) in the data set to be processed is N (N>31), and each data to be processed (HR-HSI data) is divided into n 31-band microscopic hyperspectral image data (hereinafter referred to as sub-data), then n is obtained by the following formula: ; in, Represents a round-up operation. Obviously, the total number of bands of n sub-data is greater than or equal to N, so each sub-data may contain some repeated bands. The number of repeated bands specified in the present invention only calculates the number of repeated bands of adjacent sub-data. Therefore, the calculation formula for the number of overlapping bands at least contained in adjacent sub-data is: ; in, Represents the floor operation.

[0032] Therefore, the n sub-data partitioning scheme is: 1) The number of the first sub-data band is 31, without duplication.

[0033] 2) Starting from the second sub-data, the subsequent ( The number of bands of the sub-data is 31, and the number of repeated bands is ( )indivual.

[0034] 3) The number of remaining sub-data bands is 31, and the number of repeated bands is indivual.

[0035] In the prediction stage, since the model parameters are fixed, spectral information redundancy and overfitting are no longer the main risks. At this time, in order to ensure the continuity between sub-data segments and the smoothness of the prediction results, it is necessary to retain a certain amount of overlapping bands when segmenting. By setting appropriate band overlap, not only can the boundary effect that may occur during data reorganization be effectively alleviated, but also the stability and accuracy of the prediction can be improved by fusing the prediction information of the overlapping area. Since the input of the pre-trained model contains multiple spectral bands, the choice of the number of overlapping bands during segmentation is diverse. If the number of overlapping bands is too small, it may lead to significant boundary effects and destroy the continuity and accuracy of the prediction; while too many overlapping bands will increase computational redundancy and may introduce additional noise or deviation. Therefore, it is of great significance to reasonably determine the optimal number of overlapping channels to improve the overall stability and accuracy of model prediction.

[0036] To this end, the present invention proposes a method for determining the number of overlapping bands between adaptive spectral segments based on mean square error optimization. By comparing the mean square error between the predicted data and the reference data under different numbers of overlapping bands, the optimal number of overlapping bands is determined. The calculation process of this method is as follows: first, different numbers of overlapping bands are set, and corresponding segmented data are obtained respectively; secondly, the trained model is used to predict multiple batches of segmented data produced by different numbers of overlapping bands, and the data is reorganized based on the averaging method to obtain a fused image. ; Then calculate the different overlapping band numbers to obtain With reference data (high-resolution hyperspectral image); finally, the MSE values ​​of multiple groups of data are compared, and the number of overlapping bands corresponding to the minimum MSE value is selected as the optimal number of bands.

[0037] Since the training data and the test data are segmented in different ways and their spectral characteristics are similar, using the training data to determine the optimal number of overlapping bands not only avoids accidental results that may occur when the model predicts the training set, but also ensures the validity of the results on the test set.

[0038] In some embodiments, the band values ​​of the overlapping areas of each adjacent sub-data of each predicted sub-data are fitted in turn by the least squares method to obtain the optimal band value of each overlapping area, and the current band value of each overlapping area is replaced by the optimal band value of each overlapping area.

[0039] It should be noted that the number of bands in each sub-data is the same as the input image of the low spectral dimension model, so that the divided sub-data in the prediction stage can meet the data input requirements of the pre-trained model. A small amount of data is selected to fine-tune the pre-trained model to improve the performance of the model on specific tasks. In the process of splicing sub-data sets, the least squares method is used to fit the band values ​​of the overlapping areas instead of simply taking the average value, which can better retain and optimize the information of the overlapping areas and improve the quality of the final prediction results.

[0040] like Figure 2 As shown in the figure, from left to right are the first process, the second process and the third process. The blue box represents the data to be predicted, the yellow arrow represents the prediction model, and the green box represents the prediction result. Figure 2 It can be seen from the figure that the first process and the second process overlap at position 3, that is, there are two predicted values ​​at this position, which is obviously unreasonable because there can only be one value at a position. Similarly, the second process and the third process also overlap at position 5. When there are multiple observations at the same point, the present invention proposes to calculate the best matching value based on the least squares method for replacement. The replacement here is to use the best matching value calculated by the least squares method as the predicted data at the overlap.

[0041] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0042] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-band hyperspectral image fusion method based on transfer learning, characterized by: The specific steps include: S1: A set of high-resolution hyperspectral images of the target collected by a scanning microscopic hyperspectral imaging system; S2: performing spectral dimension reduction and spatial dimension reduction operations on the high-resolution hyperspectral image set to obtain a preprocessed image group; The pre-processed image group includes first transmittance data and second transmittance data corresponding to each other; S3: dividing the preprocessed image group in proportion to obtain a first data set and a second data set; S4: obtaining a data set to be processed based on the first transmittance data and the second transmittance data in the first data set; S5: Segment each to-be-processed data contained in the to-be-processed data set to obtain a first sub-data set; S6: Use the first sub-dataset to train the pre-trained model to obtain the best model; S7: Replace the first data set with the second data set, and repeat steps S4-S5 to obtain a second sub-data set; S8: dividing the second sub-dataset into a training sub-dataset and a sub-dataset to be predicted in proportion, and optimizing the best model using the training sub-dataset based on an unsupervised learning method to obtain the best prediction model; S9: inputting the sub-dataset to be predicted into the optimal prediction model for prediction, and obtaining the predicted sub-dataset; S10: Connect the prediction sub-data contained in the prediction sub-dataset along the spectral dimension to obtain a final high-resolution hyperspectral image.

2. The multi-band hyperspectral image fusion method based on transfer learning according to claim 1, characterized in that: In step S1, the scanning microscope hyperspectral imaging system uses a tungsten lamp as a light source and collects the target under the same objective lens magnification.

3. The multi-band hyperspectral image fusion method based on transfer learning according to claim 1, characterized in that: In step S2, a spectral dimension reduction operation is performed on the high-resolution hyperspectral image to obtain a grayscale image, and a transmittance conversion is performed on the grayscale image to obtain first transmittance data; A spatial dimension reduction operation is performed on the high-resolution hyperspectral image to obtain a low-resolution hyperspectral image, and a transmittance conversion is performed on the low-resolution hyperspectral image to obtain second transmittance data.

4. The multi-band hyperspectral image fusion method based on transfer learning according to claim 1, characterized in that: In step S5, the specific steps of segmenting the data to be processed are: assuming that the number of bands contained in the data to be processed is N, the data to be processed is divided into a plurality of microscopic hyperspectral image data of M bands by the following formula, N>M, and a first sub-data set is obtained; ； in, For the rounding operation, The number of sub-data contained in the first sub-data set.

5. The multi-band hyperspectral image fusion method based on transfer learning according to claim 4, characterized in that: The minimum number of overlapping bands contained in two adjacent sub-data : ； in, This is a floor operation.

6. The multi-band hyperspectral image fusion method based on transfer learning according to claim 1, characterized in that: In the connection process of step S10, the band values ​​of the overlapping areas of adjacent prediction sub-data are fitted in turn by the least squares method to obtain the optimal band value of each overlapping area, and the current band value of each overlapping area is replaced by the optimal band value of each overlapping area to obtain the final high-resolution hyperspectral image.

Images