Hyperspectral fusion imaging method and system driven by domain knowledge and adaptive spectrum tuning

Abstract

The present invention discloses a method and system for hyperspectral fusion imaging driven by domain knowledge and adaptive spectral tuning, the method comprising: establishing a domain knowledge base to provide background and target prior information in different environments; establishing an adaptive spectral tuning device, designing a spectral adaptive learning algorithm based on spectral autocorrelation guidance, learning spectral information from a specific target prior library, collecting multispectral images with significant spectral characteristic spectral information through scoring functions and liquid crystal tunable filtering; constructing a fusion imaging system, designing an unsupervised fusion algorithm to obtain a fused image from multispectral imaging information and hyperspectral imaging information in a tensor representation framework. The present invention obtains multispectral imaging bands through an adaptive spectral tuning network, improves the discriminative spectral characteristics of a specific area, and the intelligent imaging system realizes hyperspectral fusion imaging with high-altitude spectral resolution performance, which can be applied to lightweight and small-sized hyperspectral fusion imaging equipment.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image processing, and in particular relates to a hyperspectral fusion imaging method and system driven by domain knowledge and adaptively tuned by spectrum. Background Art

[0002] Hyperspectral images are image data that continuously collect band and high-resolution spectral information in the visible and infrared bands. Compared with traditional multispectral images, hyperspectral images can provide richer spectral characteristics and spatial distribution of objects. However, during the imaging acquisition process, affected by various factors such as atmospheric environment, light radiation, sensor noise, etc., the collected hyperspectral images often have low spatial resolution and noisy spectral resolution. Therefore, in order to improve the spectral resolution, rich spatial information can be extracted from the multispectral imaging results of the same scene and effectively fused with the spectral information in the hyperspectral imaging. Therefore, it is of great significance to study how to achieve the fusion of hyperspectral and multispectral images in the imaging process.

[0003] In recent years, the fusion of multispectral images with high spatial resolution and hyperspectral images with high spectral resolution to generate images with high spectral resolution has become the main solution for fusion imaging. Domestic and foreign scholars have proposed fusion methods from various perspectives such as degradation modeling, component replacement, tensor representation, and deep learning. These methods are mainly divided into three categories: model-driven, data-driven, and model-data dual-driven. The model-driven method constructs a degradation model for the physical imaging mechanism, builds a reasonable prior structure for the input data, and designs the corresponding iterative solution algorithm for solution. The main priors used include nuclear norm minimization constraints, total variation priors, spectral manifolds, etc. However, these artificial priors cannot reflect the deep features inside the data. The data-driven method designs a deep feature extraction network to realize the mining of deep spatial-spectral joint features, and trains the network with a large amount of data. However, hyperspectral imaging data cannot meet the construction of supervised training models. The model-data dual-driven method can effectively alleviate this problem. This type of method is a combination of the first two methods. It provides an interpretable iterative solution process by constructing a fusion model dominated by the imaging mechanism, and uses a deep network to approximate the solution factor. Common methods include deep proximal solver, iterative soft threshold network, deep image prior, etc.

[0004] At present, most of the mainstream fusion imaging methods are verified in simulation experiments, and the fusion performance on real data is still limited. Moreover, there is a lot of redundant information in some spectral bands of multispectral images, which makes the fused image unable to highlight the significant spectral characteristics of targets in specific fields, which is not conducive to subsequent high-precision target detection and other tasks. Therefore, it is of great significance to analyze the imaging mechanism and design a hyperspectral intelligent fusion imaging method and system with a customized spectral range for specific tasks and field targets. Summary of the invention

[0005] The purpose of the present invention is to provide a hyperspectral fusion imaging method and system driven by domain knowledge and adaptive spectral tuning, so as to realize hyperspectral fusion imaging with high spatial spectral resolution performance, which can be applicable to light and small hyperspectral fusion imaging equipment.

[0006] The technical solution to achieve the purpose of the present invention is: a hyperspectral fusion imaging method driven by domain knowledge and adaptive spectrum tuning, comprising the following steps:

[0007] S101, establish a domain knowledge base to provide background and target spectral prior information;

[0008] S102, establishing a multispectral imaging method based on spectral autocorrelation guidance and adaptive spectral band tuning, obtaining an imaging range through spectral autocorrelation calculation and spectral indexing, and collecting a multispectral image through liquid crystal tunable filtering;

[0009] S103, non-local sampling is performed on multispectral imaging in a specific spectral range and hyperspectral imaging in the same optical path; spectral and spatial downsampling is performed on the sampling information to obtain spectral subspace and high-order spatial coefficient representation, and fusion imaging of high-altitude spectral resolution images is obtained through an unsupervised fusion iterative network.

[0010] Based on the same inventive concept, the present invention also provides a domain knowledge driven and adaptive spectral tuned hyperspectral fusion imaging system, comprising a domain prior knowledge base, a multispectral imaging device with adaptive tunable spectrum, a hyperspectral imaging device with the same optical path, and an unsupervised hyperspectral fusion imaging device. The domain prior knowledge base is used to provide spectral and background prior information of a specific target, the multispectral imaging device with adaptive tunable spectrum is used to collect multispectral imaging of a specific target's discriminative spectral range, the hyperspectral imaging device with the same optical path is used to collect hyperspectral imaging of the same scene, and the input end of the unsupervised hyperspectral fusion imaging device is connected to the output end of the multispectral imaging device and the output end of the hyperspectral imaging device, so as to fuse the multispectral and hyperspectral imaging information and realize fusion imaging with high spatial spectral resolution.

[0011] In addition, the present invention also provides an electronic device, including an imaging device, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned hyperspectral intelligent imaging method based on domain knowledge driven and adaptive band tuning is implemented.

[0012] In addition, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the hyperspectral intelligent imaging method based on domain knowledge driving and adaptive band tuning is implemented.

[0013] In addition, the present invention also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the above-mentioned hyperspectral intelligent imaging method based on domain knowledge driving and adaptive band tuning is implemented.

[0014] Compared with the prior art, the present invention has the following significant advantages: (1) the present invention constructs a domain knowledge base to provide intelligent imaging with prior information on the spectra of specific targets under different environments; (2) the present invention constructs an adaptive spectrally tunable multispectral imaging method, uses domain knowledge as a priori guidance, extracts spectral autocorrelation features, calculates the spectral autocorrelation matrix of the extracted spectral features, obtains the weights of different spectra within the features, extracts the spectral range with the highest score by maximizing the scoring function, and collects multispectral imaging of a specific spectral range by liquid crystal tunable filtering; (3) the present invention constructs an unsupervised hyperspectral fusion imaging algorithm to fuse multispectral and hyperspectral imaging information under a specific spectrum, and uses a tensor fusion network and a joint training loss to train network parameters to achieve high-altitude spectral resolution hyperspectral imaging for specific targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a basic flow chart of the hyperspectral intelligent imaging method based on domain knowledge driven and adaptive band tuning of the present invention.

[0016] Figure 2 It is a flow chart of a hyperspectral intelligent imaging method based on tunable spectrum in an embodiment of the present invention.

[0017] Figure 3 Schematic diagram of a display kernel tensor fusion network constructed in an embodiment of the present invention.

[0018] Figure 4 It is a basic structural diagram of a hyperspectral intelligent imaging system based on domain knowledge drive and adaptive band tuning in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0020] like Figure 1 and Figure 2 As shown, a domain knowledge driven and adaptive spectrum tuned hyperspectral fusion imaging method of the present invention comprises:

[0021] S101, establish a domain knowledge base, including environments such as cities, forests, oceans, geology, deserts, rainforests, and specific targets such as vehicles, trees, buildings, minerals, jungles, sand, etc., to provide spectral prior information P of the background and targets;

[0022] S102: Establish a multispectral imaging method based on spectral autocorrelation guidance and adaptive spectral band tuning, obtain imaging range Bands through spectral autocorrelation calculation and spectral indexing, and collect multispectral images through liquid crystal tunable filtering

[0023] S103, multispectral imaging of a specific spectral range Hyperspectral imaging with the same optical path Perform non-local sampling; perform spectral and spatial downsampling on the sampling information to obtain the spectral subspace S and high-order spatial coefficient representation Obtaining fusion imaging of high-altitude spectral resolution images through unsupervised fusion iterative network

[0024] Optionally, in step S101, a domain knowledge base of the detection target is established, and denoising, multivariate scattering correction and background separation are performed on the hyperspectral data covering the targets and materials of interest collected in different scenarios. Pure spectral prior information is obtained through the VCA algorithm, and a target spectral library and a background spectral library are established.

[0025] In this embodiment, step S102 of establishing a multispectral imaging method based on spectral autocorrelation guidance and adaptive spectral band tuning includes:

[0026] S201, construct multispectral imaging spectrum index dictionary D S , used for subsequent screening of specific imaging spectral ranges;

[0027] S202, according to the selected specific target area, select the spectral prior information P of the background and the target from the domain knowledge base constructed in step S101, and use the trained channel upsampling network to extract the spectral feature matrix V; send the spectral feature matrix V to the spectral autocorrelation calculation module, calculate the autocorrelation coefficient between each channel, obtain the spectral autocorrelation matrix A, and obtain the discriminative imaging spectral range D by maximizing the scoring function and the spectral index set S-A ;

[0028] S203, adjusting the spectrum by liquid crystal tunable filtering to obtain multi-spectral imaging in a discriminative spectral range

[0029] In step S202 of this embodiment, the trained channel upsampling network is used to extract the spectral feature matrix V; the spectral feature matrix V is sent to the spectral autocorrelation calculation module, and the self-attention mechanism is used to calculate the autocorrelation coefficient between each channel to obtain the spectral autocorrelation matrix A, which can effectively mine the weight proportion within the spectral channel, and obtain the discriminative imaging spectral range D by maximizing the scoring function and the spectral index set. S-A The function expression of the process is:

[0030] V=Sigmoid(Linear 32 (S-MLP ×k-64 (P))),

[0031] V Q ,V K ,V V =LeakyReLU(V+Linear 16 (LeakyReLU(Linear 32 (V)))),

[0032] A=SSAN(V Q ,V K ,V V )=sum(softmax(V Q V K T / 4)V V ),

[0033] D S-A =D S [max{i=k|A1,A2,…,A N}]

[0034] In the above formula, P is the spectral prior information, V is the spectral upsampling feature, and V Q ,V K ,V V To calculate the encoding features of the spectral autocorrelation matrix, S-MLP ×k-64It consists of k layers of fully connected layers with a dimension of 64 and a LeakyReLU activation function. 16 and Linear 32 They are constructed by fully connected layers with dimensions of 16 and 32 respectively. Sigmoid represents the Sigmoid activation function. SSAN represents the spectral autocorrelation attention calculation network, which can well learn the relationship between channels and effectively enhance the spectral range of specific target spectral features. A is the calculated one-dimensional spectral autocorrelation matrix. A1, A2, …, A N Represents the autocorrelation coefficients of the 1st, 2nd, ..., Nth channels. The correlation coefficients are sorted in descending order by max{i=k|...}. The indexes corresponding to the first k largest autocorrelation coefficients are calculated and an index set is constructed. S Index to obtain imaging spectrum range D S-A .

[0035] In step S103 of this embodiment, the sampling information is spectrally and spatially downsampled to obtain a spectral subspace S and a high-order spatial coefficient representation. Obtaining fusion imaging of high-altitude spectral resolution images through unsupervised fusion iterative network include

[0036] S301, based on the sensor imaging mechanism, firstly model the degradation relationship and snapshot imaging: ×3R, where R is the sensor spectral response parameter, × i Represents the tensor multiplication operation along the i-th mode of the tensor; Where M is the sensor coding aperture, shift d is the band shift operation, that is, each band image is shifted downward by (i-1)×d unit lengths along the second dimension, and sum is the sum along the spectral channel. By analyzing the imaging mechanism, more accurate relationship modeling can be obtained, which is helpful for the subsequent realization of high-altitude spectral resolution fusion imaging.

[0037] S302, for the non-local sampling information of hyperspectral imaging, a trainable low-rank spectral feature network is used to learn the spectral subspace S from the non-local information of the hyperspectral imaging, and multiply it with the multi-spectral sampling data to obtain a high-order spatial coefficient representation

[0038] S303, based on degradation modeling, imaging model and non-local sampling, introduce sparse representation for spatial coefficients and low-rank constraints for spectral subspace to construct an unsupervised fusion imaging model:

[0039]

[0040] in, represents the sparse constraint on the spatial coefficient matrix, represents the low-rank constraint of the spectral subspace, μ and η are the parameters of the sparse constraint and the low-rank constraint respectively;

[0041] S304, perform a fourth-order kernel tensor decomposition on the subspace to learn relevant information of different modes, and transform the fusion problem as follows:

[0042]

[0043] in, is a core tensor, D1, D2, D3 and D4 represent the dictionary information corresponding to the four modes respectively. It represents the inverse sampling process, that is, the recovery representation from high-order to low-order. By performing non-local high-order modeling on the input data, not only can the spectral pattern and spatial pattern information be learned from the samples, but also the high-order correlation and spatial structure between local blocks can be learned through the factor matrix corresponding to the fourth pattern, which is conducive to improving the fusion performance.

[0044] In this embodiment, in step S304, the fusion model is solved by constructing a tensor fusion network and gradient descent to obtain a hyperspectral fusion imaging. include:

[0045] S401, construct a tensor network prototype based on sparse core tensor representation

[0046]

[0047]

[0048] in, As network input and reconstructed output, As the output of the intermediate layer of the tensor network, D1, D2, D3 and D4 are the network layer learning parameters, and LeakyReLU is the LeakyReLU activation function;

[0049] S402, the fusion model is decomposed into the following three steps using an alternating optimization method to solve:

[0050]

[0051]

[0052]

[0053]

[0054] The first step is to use the trainable low-rank spectral feature network in step S302 to learn the spectral subspace S. represents the first-order gradient of the objective function. The second step is to use the tensor scaling soft threshold iteration method, combined with the tensor network prototype proposed in step S401, to construct an m-layer display tensor fusion network for solving the sparse kernel tensor Represents an expression consisting of sparse constraints, where each layer performs spectral upsampling and band inverse drift operations on the kernel tensors obtained by multispectral and hyperspectral sampling, respectively. The spectrum is implemented through a channel upsampling layer, and the inverse drift operation is implemented by constructing a circulant matrix. The kernel tensor fusion network is as follows Figure 3 As shown, the third step is to obtain the reconstructed image by training parameters and

[0055] S403, construct an unsupervised training loss to update the network parameters in the solution process of S402, and the training loss function expression is:

[0056]

[0057] Wherein, MSE is mean square error loss, α is a hyperparameter of balanced training, and the iteration loop is set to E steps, and if it exceeds the loop, the step S404 is jumped;

[0058] S404, reconstructing the image according to the obtained kernel tensor, factor matrix and spectral subspace, and outputting a fused image

[0059] In addition, if Figure 4 As shown, the present invention also provides a domain knowledge driven and adaptive spectral tuned hyperspectral fusion imaging system, including a domain prior knowledge base, a multispectral imaging device with adaptive tunable spectrum, a hyperspectral imaging device with the same optical path, and an unsupervised hyperspectral fusion imaging device. The domain prior knowledge base is used to provide spectral and background prior information of a specific target, the multispectral imaging device with adaptive tunable spectrum is used to collect multispectral imaging of a specific target's discriminative spectral range, the hyperspectral imaging device with the same optical path is used to collect hyperspectral imaging of the same scene, and the input end of the unsupervised hyperspectral fusion imaging device is connected to the output end of the multispectral imaging device and the output end of the hyperspectral imaging device, so as to fuse the multispectral and hyperspectral imaging information and realize fusion imaging with high spatial spectral resolution.

[0060] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral fusion imaging method driven by domain knowledge and adaptive spectrum tuning, characterized in that: The following steps are involved: S101, establish a domain knowledge base to provide background and target spectral prior information; S102, establishing a multispectral imaging method based on spectral autocorrelation guidance and adaptive spectral band tuning, obtaining an imaging range through spectral autocorrelation calculation and spectral indexing, and collecting a multispectral image through liquid crystal tunable filtering; S103, performing non-local sampling on multispectral imaging in a specific spectral range and hyperspectral imaging in the same optical path; The sampling information is spectrally and spatially downsampled to obtain spectral subspace and high-order spatial coefficient representation, and the fusion imaging of high-altitude spectral resolution images is obtained through an unsupervised fusion iterative network; The multispectral imaging method based on spectral autocorrelation guidance and adaptive spectral band tuning is established, and the specific steps are as follows: S201, construct multispectral imaging spectrum index dictionary D S , used for subsequent screening of specific imaging spectral ranges; S202, according to the selected specific target area, select the spectral prior information P of the background and the target from the domain knowledge base constructed in step S101, and use the trained channel upsampling network to extract the spectral feature matrix V; The spectral feature matrix V is sent to the spectral autocorrelation calculation module to calculate the autocorrelation coefficient between each channel to obtain the spectral autocorrelation matrix A. The discriminative imaging spectral range D is obtained by maximizing the scoring function and the spectral index set. S-A ; S203, adjusting the spectrum through liquid crystal tunable filtering to obtain multi-spectral imaging Y in a discriminative spectral range.

2. The hyperspectral fusion imaging method driven by domain knowledge and adaptive spectrum tuning according to claim 1 is characterized in that: In step S101, a domain knowledge base of the detection target is established, and denoising, multivariate scattering correction and background separation operations are performed on the hyperspectral data covering the targets and materials of interest collected in different scenarios. Pure spectral prior information is obtained through the VCA algorithm, and the target spectral library and the background spectral library are established.

3. The hyperspectral fusion imaging method driven by domain knowledge and adaptive spectrum tuning according to claim 1 is characterized in that: Step S202 uses the trained channel upsampling network to extract the spectral feature matrix V; the spectral feature matrix V is sent to the spectral autocorrelation calculation module to calculate the autocorrelation coefficient between each channel to obtain the spectral autocorrelation matrix A, and the discriminative imaging spectral range D is obtained by maximizing the scoring function and the spectral index set. S-A The function expression of the process is: V=Sigmoid(Linear 32 (S-MLP ×k-64 (P))), In Q ,V K ,V V =LeakyReLU(V+Linear 16 (LeakyReLU(Linear 32 (V)))), A=SSAN(V Q ,V K ,V V )=sum(softmax(V Q V K T / 4)V V ), D S-A =D S [max{i=k|A1,A2,…,A N }] Among them, V is the spectral upsampling feature, V Q ,V K ,V V To calculate the encoding features of the spectral autocorrelation matrix, S-MLP ×k-64 It is composed of k layers of fully connected layers with a dimension of 64 and a LeakyReLU activation function. 16 and Linear 32 They are constructed by fully connected layers with dimensions of 16 and 32 respectively. LeakyReLU represents the LeakyReLU activation function, Sigmoid represents the Sigmoid activation function, softmax represents the softmax activation function, sum represents the summation operation along the channel dimension, and SSAN represents the spectral autocorrelation attention calculation network; A is the calculated one-dimensional spectral autocorrelation matrix, A1, A2, …, A N represents the autocorrelation coefficient of the 1st, 2nd, ..., Nth channel; D S-A is the imaging spectral range.

4. The hyperspectral fusion imaging method driven by domain knowledge and adaptive spectrum tuning according to claim 1 is characterized in that: In step S103, the sampled information is spectrally and spatially downsampled to obtain a spectral subspace S and a high-order spatial coefficient representation. Obtaining fusion imaging of high-altitude spectral resolution images through unsupervised fusion iterative network include: S301, based on the sensor imaging mechanism, firstly model the degradation relationship and snapshot imaging: Where R is the sensor spectral response parameter, × i Represents the tensor multiplication operation along the i-th mode of the tensor; Where M is the sensor coding aperture matrix, shift d is the band shift operation, that is, each band image is shifted downward by (i-1)×d unit lengths along the second dimension, and sum is the sum along the spectral channel; S302, for the non-local sampling information of hyperspectral imaging, a trainable low-rank spectral feature network is used to learn the spectral subspace S from the non-local information of the hyperspectral imaging, and multiply it with the multi-spectral sampling data to obtain a high-order spatial coefficient representation S303, based on degradation modeling, imaging model and non-local sampling, introduce sparse representation for high-order spatial coefficient representation, introduce low-rank constraint for spectral subspace, and construct an unsupervised fusion imaging model: in, represents the sparse constraint on the spatial coefficient matrix, represents the low-rank constraint of the spectral subspace, μ and η are the parameters of the sparse constraint and the low-rank constraint respectively; S304, perform a fourth-order kernel tensor decomposition on the subspace to learn relevant information of different modes, and transform the fusion problem as follows: in, is a core tensor, D1, D2, D3 and D4 represent the dictionary information corresponding to the four modes respectively. represents the inverse sampling process, i.e., the recovery representation from high order to low order; S305, solving the fusion model by constructing a tensor fusion network and gradient descent to obtain hyperspectral fusion imaging 5. The hyperspectral fusion imaging method driven by domain knowledge and adaptive spectrum tuning according to claim 4 is characterized in that: In step S305, the fusion model is solved by constructing a tensor fusion network and gradient descent to obtain hyperspectral fusion imaging. include: S401, construct a tensor network prototype based on sparse core tensor representation in, As network input and reconstructed output, As the output of the intermediate layer of the tensor network, D1, D2, D3 and D4 are the network layer learning parameters, and LeakyReLU is the LeakyReLU activation function; S402, the fusion model is decomposed into the following three steps using an alternating optimization method to solve: The first step is to use the trainable low-rank spectral feature network in step S302 to learn the spectral subspace S, ▽f represents the first-order gradient of the objective function, and the second step is to perform a tensor fusion network on the sparse kernel tensor to solve the kernel tensor Represents an expression consisting of sparse constraints. The third step is to obtain the reconstructed image by training parameters. and S403, construct an unsupervised training loss to update the network parameters in the solution process of S402, and the training loss function expression is: Wherein, MSE is mean square error loss, α is a hyperparameter of balanced training, and the iteration loop is set to E steps, and if it exceeds the loop, the step S404 is jumped; S404, reconstructing the image according to the obtained kernel tensor, factor matrix and spectral subspace, and outputting a fused image 6. A domain knowledge driven and adaptive spectrum tuned hyperspectral fusion imaging system, characterized in that: Used to implement any of the methods described in claims 1-5, the system includes a domain prior knowledge base, a multispectral imaging device with an adaptive tunable harmonic spectrum, a hyperspectral imaging device with the same optical path, and an unsupervised hyperspectral fusion imaging device, wherein the domain prior knowledge base is used to provide spectral and background prior information of a specific target, the multispectral imaging device with an adaptive tunable harmonic spectrum is used to collect multispectral imaging of a specific target's discriminative spectral range, the hyperspectral imaging device with the same optical path is used to collect hyperspectral imaging of the same scene, and the input end of the unsupervised hyperspectral fusion imaging device is connected to the output end of the multispectral imaging device and the output end of the hyperspectral imaging device, so as to fuse the multispectral and hyperspectral imaging information and realize fusion imaging with high spatial spectral resolution.

7. An electronic device comprising an imaging device, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images